camera's coordinates to pick object

This is a typical issue in vision-based robotic systems.

Basically, you have 3 different coordinate systems which must be aligned somehow: The camera, the "table" (your chessboard, in this case), and the robot.

The camera's coordinate system consists of an XY grid of pixels, usually starting from one corner of the image(depends on your vision software). The safe assumption is that you will never get this perfectly aligned with any other coordinate frame by physical adjustment alone. Also, you have to create a conversion between pixels and mms, since that changes as the height and/or zoom of the camera is changed.

The robot has its own internal coordinate system that is part of it's physical construction: $WORLD. Again, it's safe to assume you can never make it perfectly aligned with either of the other two simply by physical adjustment. Fortuantely, the robot comes pre-equipped to be taught other coordinate systems.

Then there's the "table": your chessboard. Usually, the simple way is to treat one corner of the board origin of the coordinate system, and make X and Y aligned with the two edges that meet at that corner (which makes Z orthogonal to the surface of the board by default). Generally, the process is to mount your board in its ideal position, then teach the camera and robot the board's coordinate system.

For the camera, this means finding the location of that corner in the camera's pixel coordinate system, and some points along the X and Y edges, at a known mm distance from the origin. This will give you enough data to build a trig algorithm that can convert the location of an object at X and Y pixels from the camera image origin, into a location X and Y mm from the chessboard origin.

For the robot, it's quite simple. Temporarily mount a sharp-ish (pencil-sharp, not hypodermic-sharp) pointer firmly to the robot, and teach it as a TCP. Then, use the 3-Point Base function in the robot -- touch the pointer to the chessboard origin, then a point on the chessboard X axis, then a point in the chessboard's positive XY plane. That's it -- now the robot has a Base whose origin and orientation matches that of the chessboard, and you can program points in that base. This is just like the process of coming up with the conversion between the Camera and chessboard coordinate systems, except that the robot has all the math "canned" internally, while for the camera-to-chessboard, you'll need to create your own algorithm.

Once you've taught the robot it's relationship to the chessboard, and worked out the conversion formula between camera pixels and chessboard coordinates, that's about all there is to it. You can touch the pointer to the chessboard at certain points with the pointer Tool and the chessboard Base active, locate the pointer tip in the camera, convert from pixels to XY position, and check that against what the robot reports its position as being. Once you have the math correct, these should match up pretty closely (they'll never be perfect, but should be more than close enough).

TCP setup is detailed in the manual. Briefly, go under the SETUP or CONFIG menu (I don't have a KRC to check against), there should be a sub-menu for Tool&Base. You'll need to do Tool first, then Base.

Setting up the Tool, you'll need to use the 4-point XYZ method. This will tech the XYZ position of the tool, but not do anything for the ABC rotations. This should be enough, if you're careful.

4-Point TCP setup works by stepping through the menu prompts. You need to touch the tip of the pointer to the same point in space from 4 different orientations. Usually we attach a pointer to the table, and use the tip as the fixed spatial point. A pencil on the table, and a table on the robot, will serve, as long as their mountings are solid. If they move during your measurement, it will add so much error that you'll need to re-mount and start over.

Once you have the pointer TCP set up correctly, you can use the 3-point Base setup method, under the same menu tree.

programming manual for system integrator explain this and just about everything else. this is the #1 manual to have. even if it is for a different controller, workflow is the same.

Get the manuals here, but especially the Configuration manual: https://www.robot-forum.com/ro…uals-for-kss-version-5-2/

1. Misunderstanding.The 4point XYZ method only sets xyz coordinates of tool but not abc. During calibration you have to change orientation, in which coordinates you move does not matter.The robot calculates the coordinates correct.

2. The same as to 1. No matter in which coordinates you move the robot, but the tool must be selected correct at the beginning of base calibration.

As Hermann says, which coordinate system you jog in is irrelevant -- the only requirement is that the physical location be correct when you set the point.

Brief technical digression: the robot "knows" where the "Zero Tool" (the center of the Axis 6 mounting flange) is at all times, the same way that you know where your own hand is, even with your eyes closed -- that flange is a part of the robot, and it's relationship to all the axes cannot be changed.

However, the robot knows nothing about what you are attaching to the A6 flange, except for what you tell it (using $TOOL and/or the TOOL_DATA array). If you attach a tool (for example, a pencil) to the A6 flange, and touch the tip of the tool to the same point in space from 4 very different directions, the robot will inherently know the position of the Zero Tool for each of those four orientations. As a side effect, the Zero Tool will be define four points on the surface of a sphere, centered on the fixed reference point you are touching the tool to. As long as that reference point is fixed, the robot does not need to know where it is, but with 4 points on the surface of a sphere, the robot can mathematically reverse-engineer the XYZ dimensions of the tool, because there will only be one solution for the tool which will work for all 4 Zero Tool positions. Of course, this process is slightly "noisy," so the robot will do it's best to average out the error for each point. But if you are sloppy about touching the reference point at all 4 locations, you may generate more noise in the calculation than the algorithm can tolerate.

As for what the Tool does for you: in order to define a Base using the 3-point method, the robot must have an accurate tool TCP first. The Base is defined by where you move the Tool, so if the Tool is not defined, attempting to define the Base is useless.

In defining either Tool or Base, having a precise physical tool is important -- using a flat-tipped pencil will give you greater "noise" in the accuracy of the measurements than using a sharp pencil.

To relate the coordinate systems to each other, there are multiple methods, but all the frames must agree. If you set up the camera to use one corner of the chessboard as the origin, one edge as the X axis, and another edge as the Y axis, then you must teach the robot to use the same points and edges as the same origin and axes. That will get both the camera and robot using the "same map," so to speak.

Creating the pixel-to-mm conversion will take an additional step, but it shouldn't be too hard. Then it will depend on where you perform the conversion while in operation -- will you do it in the computer that the camera is connected to, or inside the robot? It works either way. You can do the math anywhere along the chain, as long as the conversion formula is accurate and the the conversion is performed before being assigned to a robot motion command.

Well, if you do something like this:

DECL E6POS PositionFromCamera
$BASE = BASE_DATA[1] ; activate common robot/vision base
$TOOL = TOOL_DATA[1] ; activate robot tool
PTP StartPosition ; hand-programmed position clear of camera FOV, with correct orientation
$OUT[1] = TRUE ; send "get image" signal to camera
WAIT FOR $IN[1] ; wait for camera to send "measurement taken"
$OUT[1] = FALSE ; reset "get image" signal
WAIT FOR NOT $IN[1] ; wait for camera to complete handshake
PositionFromCamera.X = XInputFromCamera ; Get X value from camera
PositionFromCamera.Y = YInputFromCamera ; Get Y value from camera
PTP PositionFromCamera ; move to vision measured position

In this example, the camera only controls X and Y. When the robot moves to PositionFromCamera, it retains whatever ZABC values StartPosition had.

In general, in KRL, if you give a LIN or PTP command a position (POS, E6POS, FRAME, AXIS, E6AXIS) that only has some members of the Structure variable filled in, the robot simply retains whatever the last position was for the members that are left undefined. So, in the above example, if StartPosition had a Z value of 50mm, then the robot would still be at 50mm in Z after moving to PositionFromCamera.

In this application, you're not going to bother with B or C, beyond setting them correctly at StartPosition, simply b/c your vision system has no way to measure for corrections around those axes. You may end up making A adjustments, since your vision system can make measurements of rotations around the Z axis. But just getting X and Y working first is the way to start.

The value of your Tool should be unchanging, once you've done the calibration. Your Base value should also be permanent, but you may find it useful to make adjustments to a copy on the fly. This example does the same thing the previous one does, but by changing $BASE, instead of making a point:

DECL FRAME AdjustedBase, ShiftFrame
ShiftFrame = $NULLFRAME ; set to all 0s to start
$BASE = BASE_DATA[1] ; activate common robot/vision base
$TOOL = TOOL_DATA[1] ; activate robot tool
PTP StartPosition ; hand-programmed position clear of camera FOV, with correct orientation
$OUT[1] = TRUE ; send "get image" signal to camera
WAIT FOR $IN[1] ; wait for camera to send "measurement taken"
$OUT[1] = FALSE ; reset "get image" signal
WAIT FOR NOT $IN[1] ; wait for camera to complete handshake
ShiftFrame.X = XValueFromCamera
ShiftFrame.Y = YValueFromCamera
$BASE = BASE_DATA[1] : ShiftFrame ; apply shift to $BASE -- BASE_DATA[1] is NOT altered, only the "working copy"
PTP PickupPosition

In this case, PickupPosition needs to be pre-programmed for the "zero position" pickup -- that is, the correct pickup position if/when the corrections from the camera are all 0 (or so tiny as to be practically 0). In this case, you teach a "perfect 0" position for the robot, and then shift that position around to follow the camera by shifting $BASE mathematically using the vision data. This shift works by taking BASE_DATA[1] as its reference origin (that's why it's on the left side of the Geometric Operator ":"), and applies the values in ShiftFrame before putting the shifted results into $BASE. It's important to ensure that BASE_DATA[1] never gets altered, as that's your permanent reference frame -- instead, you apply the shifts to the "working copy" in $BASE.

you could read messages that robot displays and remedy whatever they tell the problem is. for example program speed, approximation etc.

"Work space exceeded" isn't about a particular workspace -- instead, it means that the robot has been given a command to move to a location that is beyond the arm's physical reach.

This is separate from a "axis limit" error. There are positions which the robot can reach with a particular set of axis angles, but potentially not with another.

PTP motions are predictive for these errors, but LIN/CIRC motions are not. Basically, a LIN motion is made up of many, many, very very small PTP motions, and the "look ahead" for these errors is just the next PTP motion segment. So if you order the robot to an unworkable position using PTP, the error will be generated before the robot starts moving. Change the PTP to a LIN, and the robot will try to make the move, and keep moving until it physically hits a limit.

Axis limit errors are complicated b/c you can "wind up" the wrist axes during LIN motions until you hit a condition where a position that is entirely reachable, cannot be reached by LIN from the robot's current position. If you encounter cases like this, it's best to use PTP motions, perhaps with E6AXIS positions rather than E6POS, to "unwind" the axes before beginning a LIN motion. This is most often achieved by using the system HOME position at the beginning and end of each program run.

Another technique is to use PTP motions whenever possible. This usually allows the motion planner to pick out the least difficult path between Points A and B -- however, if the wrist has been "wound up," this can result in the wrist "unwinding" unpredictably when the path planner detects that it's needed. So you want to do this well clear of everything.
Often, how we do this is to make all the moves above the table PTP motions, and make only the "pick/drop" motions LIN. That is, generate your Pickup position from the vision, mathematically generate an "Above Pick" position from that, then make the Above->Pick->Above moves LINs, and all the others (say, HOME->Above) PTPs. This creates minimal opportunity for the LIN motions to "wind up" the wrist axes.