Verify all code paths

Quote from Hes

I've been wondering what does everyone else use to verify that all code paths have been tested?

Depends on the code. Motion code needs to be tested rather differently than pure logic code. But both need to be tested thoroughly.

No function should have any more branches than it absolutely has to have, and should have specific halting states programmed for any "disallowed" branches. For example, any SWITCH statement should always have the DEFAULT filled in, even if it should "never happen":

; using a SWITCH as a "counter"
SWITCH Variable
  CASE 1
    RETURN Variable+1
  CASE 10
    RETURN 0
  DEFAULT
    ;FOLD If This happens, this function was called with an illegal value
    LOOP
      HALT
    ENDLOOP
    ;ENDFOLD
ENDSWITCH 

Writing test programs is sometimes called for. Write a program that will call a production program or function with every possible "valid" argument, plus deliberately invalid arguments. Functions should be kept as small and "tight" as possible, to make them easily testable in isolation. For tests that have a broad range of possibilities, sometimes writing logging functions becomes a good idea.

Checking each line of a program generally is overkill -- but you do need to check every possible branch. And every function should be incapable of returning bad values, or letting execution continue if the function encounters conditions that were not foreseen. That's why that HALT statement is inside a LOOP with a label on it -- just using a HALT by itself would let the program continue if someone hit the Start button again.

For a motion program with multiple path branches, write a test program that calls each motion subroutine in every possible combination, even the illegal ones -- if the robot manages to make an illegal move, then your protections against bad branches need improvement.

For a function that performs math, generate a list of random values and fire them at the function. If you want to get fancy, it's possible to write log files of the incoming and outgoing results so as to perform truly massive test patterns.

Test programs also need to test what happens if a function is called with missing arguments, not just "bad" or "out of bounds" arguments.

run them all and then you will know... programming is not about creating may possible states with many inputs and outputs but systematically limiting any diversity to a necessary minimum but so that system can still function. it us unclear what your programs are or how they are organized. i place checks in everything to account for any scenario, and test them all. for example common application is machine tending where robot moves between different stations (CNC machines, storage units etc). each of them can have multiple locations that need to be accessed by robot. For example one storage may have few hundred of locations and there may be more than one storage. Also each CNC may have several locations. but i do not let robot run random paths and connect all possible locations in each storage with every possible location in each CNC. that would be wild. what i do it create some common in-between points. And i may compute location of them too to get smoother and faster motions but there are constraints and checks everywhere and variety of combinations is limited - and fully tested. in addition to in-the-program checks, there are external permissive, workspaces, etc, that sense anything out of order and catch any potential runaway process. An example of that would not be something that code calls for but user may decide to intercept and move robot around, then select wrong program or does block selection and try to continue something that is way out of order - it still hits one of the traps. so test is not complete until someone has run them all. one of the hard things is automated recovery to bring robot to safe position from random situation. but it is a beautiful thing when complete. and since that is one example that has many possibilities, a lot of testing is done and to cover for unexpected, this is run at reduced speed with collision detection active. and EVERYTHING is logged... and i mean to stay in DB so i can see the messages later on too (archive or KrcDiag). unfortunately standard MsgLib does not log to DB, but i use my own library anyway so i know how things behave...

I would definitely recommend the DEBUGGING option available with the KUKA Work Visual. This gives pretty much the overview of the active function and the interpreted statement.

The next option could be a brute force option. This is something I used in one of the previous project to understand the whole process and algorithm. I developed internal logging which simply can be called anywhere in the program and logs the user defined messages and based on previous message, I modify in the next test run based on requirement. This gave me an overview of all the called functions and how the interpreter travelled for a program cycle.

I have seen programmers using kuka $ variables ( I assume $PRO_OP but not sure) to read the actual executing statement and function calls.

Apart from this, following some basic rules from high level programming languages like if loop ends with an else and switch always with a default, avoiding while and try using FOR loops can help avoiding unknown program executions.

Quote from Hes

This is all good but still relies on that you as a person do not miss anything. Using waypoints and the like for moves is of course the only way to do it properly and checking every move in t1 while watching the robot. But what if you believe you have seen all of the possible outcomes but still managed to miss something?

Then ask someone else for review or come up with some strategy to ensure all cases are covered.

Quote from Hes

That is what i would like to build a check for. E.g. you run a program, test it thoroughly and then check with something that you actually have run every line and hence covered all outcomes should have been covered. This would help make sure that you got everything covered (not considering input values).

And if the checking program misses something than its creator becomes liable. You really want to make something that would make you liable for other peoples mistakes?

Quote from Hes

I'm currently reading up on cwrite/cread. Is it possible to read a src file with cwrite/cread? Can you get the amount of lines in a src? Filtering out blank lines and the first &.. lines is what i am currently thinking about.

Yes. one can read (and write) any file. the problem is that KRL resources are pretty limited. Since this type of validation can be done offline too, why bother with KRL? Use Visual Studio or Python etc. and you can handle even largest and most complex KRL module or entire KRC:\ in case there are dependencies in some library files etc. Plus this is way easier to develop and test in some comfortable IDE rather than using KRC or OL.

Quote from Hes

I totally agree with that you should check all code, but what if the code isnt yours? What if you can see the whole program cycle in 3mins but how do you verify that what you actually saw/checked was everything there is to see? I believe this would be a helpful tool.

I have done that several times on home recovery programs created by others. Here the question is if you can really drive robot to a known point from ANY random robot position. Solving tasks like this in limited time requires strategy...

One obvious check i always start with is to see if every IF statement also includes ELSE case. If there is no ELSE case (or it is there but when executing ELSE case no motion to a known point), program is obviously not covering all cases.

Some people may structure code differently, for example instead of ELSE statement, they may use another IF block, there may be GOTOs, loops with EXIT case etc. And if there is a SWITCH statement, it must include DEFAULT case. There may be a complete coverage only when there is a mechanism to go to a safe point on EACH of the cases. Same applies if the statements are nested.

So creating program that will parse the code requires both careful thinking as well as plenty of code to test on - variants written by as many different people as possible... and they need to be independently verified as complete or partial solution.

And same strategy can be applied to other situations such as pick and place where motions are calculated. One can check if calculations are done in a way to always fit certain limits and then do a complete check on space in those limits. This means 100% check of all known positions: all fixed points (if some sort of grid like in palletize/depalletize), and all characteristic points.

An example of "characteristic points":

on some linear space (Z axis for example) would include extremes: min, max, mid point and additional pair of points (before/after) any point where S & T may have value change.

the same applies to 2D and 3D spaces - just treat each axis/dimension as another "linear space" .

This is just an example of ways allows you to reduce problem from infinite number of cases to finite number of checks. One can of course cover other scenarios (calculated tool or base, all tool states, workspace limits, singularities,...).

A good verification tool should be able to generate own report as well as create test program to validate all or some of the cases on the KRC (OL for example).

Hes, it seems like you want a Proof of Correctness. That's a part of computer science and you can find a lot of info about it if you search

I only have the faintest notion about how that works, but to make checking of the code practical, the code is usually structured in a way to make it easier to check. Getting a proof of correctness for any random codebase at all is a tall order.

I am not sure if that's actually what you meant, but you wrote several times about testing "if the program pointer has passed that line ever or not". That would test if all lines in the program were executed at least once, which would likely be a subset of that code's configuration space.

E.g., if a function "SmallFunc", that has 2 branches, is called by function "FuncA" and both of the branches in "SmallFunc" are executed, all lines in "SmallFunc" are counted as executed. But if there is also "FuncB", that might call "SmallFunc", those calls will be counted as checked, because all lines in "SmallFunc" have been executed already.

The same thing will happen with any sequential branching statements. One of possible approaches is to build a tree with every branch and function call, then check if every bottom branch was reached and you are done. Except that some of the branches are never reached, because it's god damn impossible to get there.

It's a long and arduous path and you can never be 100% sure that any complex system will always work as intended. But I sympathize with your wish for something more reliable than a spit and a prayer. Some tips from my meager experience so far:

Don't try and verify whole codebase at once. Put some code in libraries, test the hell out of them. It won't mean that the system made of perfect parts will be perfect, but it's an easier start.

Verify your inputs. E.g. KRL will happily silently drop bits of any values above what can be expressed in 16 bits. Sometimes there is no different algorithm (short of making a BiggerNum library) and it's unlikely to get unworkable inputs anyways. If some functions don't work with all values, but won't fail silently, they can still be fine.'

Some things fail with very specific inputs, such that randomly generated values are unlikely to trigger. E.g. Lin motion when there is a difference of exactly 180 degrees in one axis.