Vision calibration using User Frame

    I have an application where it is required that the Tool should reach to certain points. The points are going to be received from camera. The image size is 640 by 480 pixels. And the point received will have (X,Y) values in the form of pixel locations (obviously!).

    One solution was to put a 'User frame' at the physical location where the image starts with X,Y of frame aligned with X,Y of Image frame (pixel (0,0) location) and convert pixel values to actual location values by a conversion factor of pixel to mm.

    Is there any other, better, more elegant way to achieve this?

    Edit: We are using CRX 10iA cobot from Fanuc.

    Edited once, last by Maverick_OZ (February 2, 2024 at 10:04 AM).

  • Go to Best Answer
    Quote from asdf

    What camera/vision system are you using?

    Generally this is done by calibrating the camera with a checkerboard grid, then teaching a Uframe on that grid so that the camera and robot "agree" on a coordinate plane.

    Thank you for the reply. We are using Intel Realsense D456. Could you point me to a post/link where the steps are detailed?

    Quote from Maverick_OZ

    The image size is 640 by 480 pixels. And the point received will have (X,Y) values in the form of pixel locations (obviously!).

    Quote from Maverick_OZ

    We are using Intel Realsense D456.

    Really? You are using a 3d camera to get coordinates in a x/y plane? Why?

    With that camera you also can get Z coordinates.

    But for your question. You can search with your favorite search engine for "opencv camera calibration" and get many answers.

    We are using 3D camera to get depth of extracted X,Y points because we want to use the depth info later. Sorry if I was unclear but my question is related to the use of User Frames. I did google before coming to this forum and could not find anything directly applicable and hence the question.

    Basically, the way your system is setup you can do however you want.
    Either the hard way, which would be to put the camera at an arbitrary orientation and then calculating where pixel 0,0 will be in the robot's world frame and then for each new position use trigonometric functions to calculate the destination coordinates for the robot.

    OR

    As you your self stated, teach a user frame which will have its origo at the 0,0 pixel in the camera field of view.

    How you do to accomplish this is up to you.

    Given the circumstances I would probably put some sort of grid in the camera's field of view and align the grid to be (as close to) perfect aligned (perpendicular/normal to) the field of view and then teach the frame using four point method. First three to determine the orientation of the user frame, and then using the live view from the camera I would position the TCP right at the 0,0 pixel and then teach the forth point (which will transform the previously taught user frame orientation to that specific position).

    But as others have already pointed out, it would be far easier and straight forward to use the integrated FANUC iRVision instead.

    You shouldn't use 2D calibration and then try to set a depth detection on top of it. Think you will not get the result you expect.

    When you move a part only in depth direction from far to near, the x/y coordinates in camera perspective will also change although the x/y coordinate in real world don't change.

    You must follow a more universal way to solve your problem.

    Normal way is to calibrate the camera and robot system by moving robot tcp to some different positions that can be detected by camera system, give the coordinates to the camera system, then the camera system calculates a transformation matrix for transfering pixels to robot coordinates. (can also be done the other way round, camera sends pixels to robot, and robot calculate the transformation).

    In which user frame you perform that calibration is not that important, you just must use the same one in production while using camera recognition.

    During 2d calibration you only move the camera in a plane parallel to camera. During 3d calibration you move the camera in 3d space during calibration.

    The math behind 2d calibration is the same as behind 3d, but a special case, where some parts of calculations is constant 0 or 1.

    • Best Answer

    This is often referred to in some circles as "hand-eye calibration." This is distinct from "camera calibration", where the ratio of pixels/mm must be determined.

    Since the Realsense camera comes from the factory pre-calibrated, you don't need to worry about camera calibration. But the camera and robot are ignorant of each other. Hence, hand-eye calibration is still required.

    Is this a robot-carried camera, or a fixed-mount camera? This has major effects on how to go about hand-eye calibration.

    For a fixed camera, calibrating to a UFrame is reasonably straightforward. In an ideal situation, you would mount a sharp-tipped pointer to the robot, and set it up as an accurate UTool. Then, your calibration grid would have precise points (Origin, X+ axis, and XY+ plane) that could be both touched using the pointer, and identified clearly in the vision image.

    The grid would be placed under the camera, the UFrame would be taught using the pointer and the three-point method. The camera would need some sort of calibration to the same grid (note: the grid must not move during this entire process!). I'm not familiar with the Realsense cameras, but most vision systems can create a user reference frame for its output of coordinates for located objects. In the worst case, if the camera provides no means for this, it's possible to create an algorithm that converts between the camera data and the robot UFrame. There's multiple ways to do this, but it makes things much easier if you physically align your grid with the camera first -- get the grid orthogonal to the Z axis of the camera, and align the X+Y axes as closely as possible. Ideally, you can then just 3-point the UFrame and be close enough.

    An alternate method: Mount a target to the robot that can be precisely defined as a UTool, and can also be located precisely by the vision system. I've used a bullseye with a large black dot, and a tiny white dot exactly at the center, to do this. Move this target to multiple points in the camera's FOV, and record both the robot coordinates and the camera coordinates of the target. You can iteratively alter the UFrame until these coordinates match up to a reasonable degree (there will always be some degree of error), or create a best-fit transform between the camera coordinates and robot coordinates.

    Quote from SkyeFire

    The grid would be placed under the camera, the UFrame would be taught using the pointer and the three-point method. The camera would need some sort of calibration to the same grid (note: the grid must not move during this entire process!). I'm not familiar with the Realsense cameras, but most vision systems can create a user reference frame for its output of coordinates for located objects. In the worst case, if the camera provides no means for this, it's possible to create an algorithm that converts between the camera data and the robot UFrame. There's multiple ways to do this, but it makes things much easier if you physically align your grid with the camera first -- get the grid orthogonal to the Z axis of the camera, and align the X+Y axes as closely as possible. Ideally, you can then just 3-point the UFrame and be close enough.

    Can you please link me to a blog or a video wherein I can simulate this inside RoboGuide. I found this. But is not directly applicable.

    Quote from hermann

    Normal way is to calibrate the camera and robot system by moving robot tcp to some different positions that can be detected by camera system, give the coordinates to the camera system, then the camera system calculates a transformation matrix for transfering pixels to robot coordinates. (can also be done the other way round, camera sends pixels to robot, and robot calculate the transformation).

    Thank you for the reply. Are we talking about Hand to eye calibration here?

    Quote from Maverick_OZ

    Thank you for the reply. Are we talking about Hand to eye calibration here?

    You are now mixing up the complete thread with asking several question in a row, it will get a bit confused.

    My description was for a fixed camera setup. Don't know if it is officially called hand eye calibration, but can't remember a different or the correct name.

    May be the term 'Hand eye calibration' normally is used for calibration with camera mounted on robot flange.

    I would say that whatever it's called, a calibration is needed to translate what the camera sees (be it fixed mounted or robot mounted) in to positional data that the robot can interpret.

    For a fixed camera, the calibration will accomplish this translation:

    Camera pixel position (Vertical, Horizontal) ---> X- and Y-coordinate in the robot's coordinate system

    For a robot mounted cam, the camera's current position needs to be taken into account:

    Camera pixel position (V, H) + Camera position ---> X- and Y-coordinate in robot

Advertising from our partners