diff --git a/docs/core/examples/guides/.vscode/settings.json b/docs/core/examples/guides/.vscode/settings.json new file mode 100644 index 000000000..70ab33523 --- /dev/null +++ b/docs/core/examples/guides/.vscode/settings.json @@ -0,0 +1,3 @@ +{ + "cSpell.words": ["AICA", "Freedrive", "Orbbec", "servoing", "URDF"] +} diff --git a/docs/core/examples/guides/assets/UR-calibration-file.png b/docs/core/examples/guides/assets/UR-calibration-file.png new file mode 100644 index 000000000..93f00a741 Binary files /dev/null and b/docs/core/examples/guides/assets/UR-calibration-file.png differ diff --git a/docs/core/examples/guides/assets/frame-broadcaster-direct-filepath.png b/docs/core/examples/guides/assets/frame-broadcaster-direct-filepath.png new file mode 100644 index 000000000..a0ebda1b1 Binary files /dev/null and b/docs/core/examples/guides/assets/frame-broadcaster-direct-filepath.png differ diff --git a/docs/core/examples/guides/assets/frame-broadcaster-expected-file-format.png b/docs/core/examples/guides/assets/frame-broadcaster-expected-file-format.png new file mode 100644 index 000000000..9aec092a9 Binary files /dev/null and b/docs/core/examples/guides/assets/frame-broadcaster-expected-file-format.png differ diff --git a/docs/core/examples/guides/assets/frame-broadcaster-publish-manually.png b/docs/core/examples/guides/assets/frame-broadcaster-publish-manually.png new file mode 100644 index 000000000..cac26bc46 Binary files /dev/null and b/docs/core/examples/guides/assets/frame-broadcaster-publish-manually.png differ diff --git a/docs/core/examples/guides/assets/robot-calibration-configuration.png b/docs/core/examples/guides/assets/robot-calibration-configuration.png new file mode 100644 index 000000000..fcd644509 Binary files /dev/null and b/docs/core/examples/guides/assets/robot-calibration-configuration.png differ diff --git a/docs/core/examples/guides/assets/robot-camera-calibration-component.png b/docs/core/examples/guides/assets/robot-camera-calibration-component.png new file mode 100644 index 000000000..7de0dbc85 Binary files /dev/null and b/docs/core/examples/guides/assets/robot-camera-calibration-component.png differ diff --git a/docs/core/examples/guides/assets/stag-detector-example.png b/docs/core/examples/guides/assets/stag-detector-example.png new file mode 100644 index 000000000..1a4f7f36d Binary files /dev/null and b/docs/core/examples/guides/assets/stag-detector-example.png differ diff --git a/docs/core/examples/guides/assets/stag-marker-detection.webm b/docs/core/examples/guides/assets/stag-marker-detection.webm new file mode 100644 index 000000000..332591ce4 Binary files /dev/null and b/docs/core/examples/guides/assets/stag-marker-detection.webm differ diff --git a/docs/core/examples/guides/assets/stagDetector-predicates_0.png b/docs/core/examples/guides/assets/stagDetector-predicates_0.png new file mode 100644 index 000000000..2cd136b92 Binary files /dev/null and b/docs/core/examples/guides/assets/stagDetector-predicates_0.png differ diff --git a/docs/core/examples/guides/assets/stagDetector-predicates_1.png b/docs/core/examples/guides/assets/stagDetector-predicates_1.png new file mode 100644 index 000000000..a07fefb48 Binary files /dev/null and b/docs/core/examples/guides/assets/stagDetector-predicates_1.png differ diff --git a/docs/core/examples/guides/fiducial-markers.md b/docs/core/examples/guides/fiducial-markers.md new file mode 100644 index 000000000..d1dcb121d --- /dev/null +++ b/docs/core/examples/guides/fiducial-markers.md @@ -0,0 +1,233 @@ +--- +sidebar_position: 14 +title: Fiducial Markers +--- + +import stagDetectorExample from './assets/stag-detector-example.png' import stagMarkerDetection from +'./assets/stag-marker-detection.webm' import stagMarkerNumOne from './assets/stagDetector-predicates_1.png' import +stagMarkerNumZero from './assets/stagDetector-predicates_0.png' + +# Fiducial Markers + +Different types of fiducial markers are used in robotics to provide precise 3D pose estimation and identification for +cameras, enabling or improving robotic calibration and object manipulation. + +AICA's `core-vision` package gives you the choice between using two commonly used markers, the STag and ArUco. + +:::tip Performing the [intrinsic calibration](./camera-calibration.md) of the camera improves the precision for fiducial +marker detection and tracking. ::: + +This guide provides an example of STag marker detection. Using the ArUco marker follows a very similar process. + +## Preparing fiducial markers + +A fiducial marker is an object placed in the field of view of an image for use as a point of reference or a measure. +STag and ArUco markers are two of the common types of fiducial marker systems used for real-time 6D pose estimation. +This section explains how to obtain, download and print these markers. + +### Obtaining markers + +- **ArUco marker**: ArUco markers can be generate online (e.g., from [here](https://chev.me/arucogen/)), which permits + choosing the dictionary, marker ID, and marker size. It can be then exported as PDF or SVG for printing. + +- **STag marker**: STag marker set can be either downloaded from + [public Google Drive](https://drive.google.com/drive/folders/0ByNTNYCAhWbIV1RqdU9vRnd2Vnc?resourcekey=0-9ipvecbezW8EWUva5GBQTQ) + or obtained from the [ROS2 STag project repository](https://github.com/usrl-uofsc/stag_ros/tree/ros2-devel) or the + generator/reference files linked by the project. In practice, you’ll want to obtain the marker PDF/SVG or generate the + markers from the project’s reference generator, then print them at true size. + +### Printing markers + +After choosing the marker family, selecting the library/dictionary, and the marker ID, download it as PDF or SVG. Use +the actual size of the marker (100% scale) for printing, so the black border and marker geometry are not resized. Also +it is recommended to print with high contrast and avoid compression artifacts. + +If possible, print on a rigid and flat sheet of paper to reduce warping, since fiducial detection is sensitive to +distortion. As another solution, you can fix the printed marker on a rigid surface, such as a piece of wood or +cardboard. + +After printing, measure the marker’s outer dimensions and compare them with the intended size from the generator. This +matters because calibration will be wrong if the marker size in the software does not match the physical print. + +## Using the STag detector + +Launch AICA Studio with a configuration that contains the `core-vision` package and create a new application. + +1. Remove the hardware interface that is included in new applications by default. +2. From the `Scene` menu, use the `Add Component` tab and look for the **Camera Streamer** and **STag Detector** + components, either by searching or by manually going under the `Core Vision Components` menu. Add both of them to the + graph. +3. Next, connect both components to the start block. Moreover, connect the outputs of the Camera Streamer to the + relevant inputs of the STag Detector. +4. Enable **auto-configure** and **auto-activate** on both components. +5. By selecting any of the components, you can find all the available component parameters in the right panel under + Settings. +6. If an intrinsic camera calibration is performed prior to this, add the file path of the camera configuration file as + a parameter to the **Camera Streamer** component. + +By this point, you should have something like the following: + +
+ CameraStreamer configuration alongside STagDetector component +
+ +:::info The Camera Streamer parameters are explained in the [CameraStreamer component guide](./camera-streamer.md). ::: + +## STag Detector parameters + +- **Rate**: The rate parameter doesn't affect the behavior of the component as the detection process occurs on reception + of a new image. +- **Bundle file**: The filepath to a predefined marker bundle configuration. This additional feature is described in a + separate guide (coming soon). +- **Marker selection**: The name(s) of the marker(s) that we want to recognize. If any of these markers enters the + camera frame, the `is_any_selected_marker_detected` predicate is set to **True**. Also if a decision needs to be made + based on the existence of a specific STag marker in the camera frame, its name should be indicated in this parameter. + The markers name should always be prepended with the value of `Prefix`. +- **Marker size**: Determines the side length of a square that specifies the marker in meters. +- **Library**: This is the ID number of the HD library utilized by STag markers. The allowed numbers are + `[11, 13, 15, 17, 19, 21, 23]`. +- **Error correction**: This parameter sets how permissive the detector is. Meaning it controls how many bit errors the + STag detector is allowed to tolerate when decoding a marker ID. Lower values mean stricter matching, but detection + would be less robust to blur, noise, bad lighting, or partial image degradation. Higher values mean more tolerant + matching. The detector can still recognize a marker even if some bits are read incorrectly, so detection is more + robust, but the risk of wrong matches increases. The lower and upper limits for this parameter are 0 and 11 + respectively (inclusive). +- **Prefix**: This is the prefix used for marker names. + +## STag Detector predicates + +- **Is any marker detected**: This predicate will be set to **True** if any marker is detected in the camera frame, even + though its name is not indicated in the 'Marker Selection' parameter. As you can see in the screenshot below, the + marker name specified in the `Marker selection` parameter is stag_1, but the marker recognized in the camera frame is + stag_0, yet `Is any marker detected` predicate is set to **True**. + +
+ Is any marker detected at all +
+ +- **Is any selected marker detected**: If one or more marker names are indicated in the `Marker selection` parameter, + and if any of them appears in the camera frame, this predicate with be set to **True**. If the names of none of the + markers present in the camera frame is indicated as a parameter, this predicate will remain **False**. In the + screenshot below the name of the marker appearing in the camera frame matches the name indicated in the + `Marker selection` parameter. + +
+ Is any of the selected markers detected +
+ +- **Is a marker bundle detected**: If a registered group of markers is detected by the camera, this parameter will be + set to **True**. Otherwise it will remain **False**. + +## Running the application + +After setting up the proper parameters for Camera Streamer and STag Detector: + +1. Press **Start** to start the application. +2. To see the live camera feed, select **Launch RViz** from the Launcher settings +3. In RViz, select _Add > By topic > /stag_detector/annotated_image > Image_. This adds a panel that shows the live + image. The marker should be detected in the camera. + +
+ +
+ +The following YAML snippet contains the full application of the STag Detector above: + +
+ Example application, STag Detector + +```yaml +schema: 2-0-6 +dependencies: + core: v5.1.0 +on_start: + load: + - component: stag_detector + - component: camera_streamer +components: + stag_detector: + component: core_vision_components::pose_detection::STagDetector + display_name: STag Detector + events: + transitions: + on_configure: + lifecycle: + component: stag_detector + transition: activate + on_load: + lifecycle: + component: stag_detector + transition: configure + parameters: + marker_selection: + value: + - stag_0 + type: string_array + inputs: + image: /camera_streamer/image + camera_info: /camera_streamer/camera_info + camera_streamer: + component: core_vision_components::image_streaming::CameraStreamer + display_name: Camera Streamer + events: + transitions: + on_configure: + lifecycle: + component: camera_streamer + transition: activate + on_load: + lifecycle: + component: camera_streamer + transition: configure + parameters: + camera_info_path: + value: /data/ost.yaml + type: string + undistorted_image_cropping: + value: false + type: bool + outputs: + image: /camera_streamer/image + camera_info: /camera_streamer/camera_info +graph: + positions: + on_start: + x: -60 + y: 0 + stop: + x: -60 + y: 100 + components: + stag_detector: + x: 820 + y: 0 + camera_streamer: + x: 200 + y: 60 + edges: + on_start_on_start_camera_streamer_camera_streamer: + path: + - x: 100 + y: 60 + - x: 100 + y: 120 + camera_streamer_image_stag_detector_image: + path: + - x: 700 + y: 280 + - x: 700 + y: 260 + camera_streamer_camera_info_stag_detector_camera_info: + path: + - x: 660 + y: 360 + - x: 660 + y: 300 +``` + +
+ +:::info The process for using ArUco markers follows a similar process. ::: diff --git a/docs/core/examples/guides/hand-eye-calibration.md b/docs/core/examples/guides/hand-eye-calibration.md new file mode 100644 index 000000000..a448c7334 --- /dev/null +++ b/docs/core/examples/guides/hand-eye-calibration.md @@ -0,0 +1,325 @@ +--- +sidebar_position: 15 +title: Hand-Eye calibration +--- + +import RobotCalibrationConfiguration from './assets/robot-calibration-configuration.png' import RobotCameraCalibration +from './assets/robot-camera-calibration-component.png' import URCalibrationFile from './assets/UR-calibration-file.png' +import FrameBroadcasterExpectedFileFormat from './assets/frame-broadcaster-expected-file-format.png' import +FrameBroadcasterPublishManually from './assets/frame-broadcaster-publish-manually.png' import +FrameBroadcasterDirectFilepath from './assets/frame-broadcaster-direct-filepath.png' + +# Hand-Eye calibration + +Robot calibration is a fundamental prerequisite for any robotic system that relies on precise coordination between a +vision sensor and a manipulator. It establishes the spatial transformation between the robot’s end-effector and the +camera frame, enabling accurate mapping between observed features and actionable robot coordinates. Without proper +calibration, even high-quality perception or motion planning algorithms can yield significant positioning errors. + +The AICA's `core-vision` package provides a structured workflow for performing hand–eye calibration efficiently and +reproducibly. + +Accurate hand–eye calibration is critical in tasks such as visual servoing, object manipulation, inspection, and +assembly. This tool is designed to minimize discrepancies and provide reliable calibration outputs suitable for +industrial environments. + +## Robot Camera Calibration component + +The Robot Camera Calibration component is the component that does the main job of calculating the transformation between +the `camera` and the `robot end-effector`. + +Click on the Robot Camera Calibration component block to view and edit the available parameters. + +
+ Robot Camera Calibration parameters +
+ +The parameters of the Robot Camera Calibration component are defined as follows: + +- **Rate**: Determines the frequency at which transformations are acquired. This parameter does not affect the + component’s behavior. +- **Bundle file**: The filepath to a predefined marker bundle configuration. This additional feature is described in a + separate guide (coming soon). +- **Camera frame**: The name of the camera frame used in the application. This can be retrieved from the list of frames + in RViz. +- **Marker frame**: The name of the marker detected by the camera. It is indicated in the STag or ArUco marker detector + component used in the application. +- **Robot base frame**: The name of the robot's base frame, which can be found in RViz. +- **Robot end-effector frame**: The name of the robot’s end-effector frame. This is also available in the RViz frame + list. +- **Number of recorded points**: The number of transformations from `camera` to `robot end-effector` that are recorded. + These are computed from multiple pairs of `robot end-effector -> robot base` and `camera -> marker` transformations. +- **Distance between points**: The minimum spatial difference required between two consecutive transformations for them + to be recorded. +- **Epsilon time**: +- **Calibration folder path**: The directory where the final calibration file will be stored. +- **Calibration file**: The name of the output file generated after calibration, containing the computed calibration + data. +- **Points dataset file**: A file containing previously recorded transformations of the robot end-effector or marker, + which can be reused if available. +- **Reset calibration**: Setting this component to `True` resets the calibration. Meaning it starts the calibration + process again, even if the calibration matrices are already set. +- **Reset dataset**: If set to `True`, the recording process will re-start, even if the dataset of points already + exists. +- **Is camera attached**: This parameter should be set to `True` if the camera is attached to the robot end-effector. + +## Robot Calibration using AICA Studio and a marker + +After completing the camera calibration as described in the [Camera Calibration example](./camera-calibration.md), and +verifying marker detection as outlined in the [Fiducial Markers](./fiducial-markers.md) section, you can proceed with +the hand–eye calibration process. + +This example demonstrates the eye-in-hand configuration (camera mounted on the robot arm). The procedure for the +eye-to-hand configuration (static camera) follows a similar workflow. + +- Ensure that the camera is properly configured and operational. +- Connect all required components and controllers in the AICA application. Refer to the system setup illustration below + for guidance. +- Place the marker within the robot workspace, ensuring it is fully visible to the camera. To monitor the live camera + feed, enable `Launch RViz` from the Launcher settings, as described in the [marker detection](./marker-detection.md) + guide. +- Run the program and move the robot TCP (Tool Center Point) to capture images of the marker from multiple perspectives. + The application starts capturing images automatically. +- The robot TCP can be moved by jogging or Freedrive mode using the robot's teach pendant. In the case of using a + Universal Robot, AICA Studio offers the option of `Hand Guiding Controller` which facilitates and accelerates the + process. This controller is described in the [Hand Guiding Controller](./ur-harware-interface.md) page. +- Ensure sufficient variation in position and orientation to improve calibration accuracy. +- Once the number of captured images reaches the `Number of recorded points` indicated in the Robot Camera Calibration + component parameters, the system automatically generates a calibration file in YAML format in the following directory: + +```bash +/tmp/calibration/camera_calibration.yaml +``` + +An example of the calibration file: + +
+ An example showing a final calibration file +
+ +:::info The order of the data in the calibration file is `w, x, y, z` for the Orientation and `x, y, z` for the +Position. ::: + +In the screenshot below you can see an example of the components configuration for the Hand-Eye calibration, that +resulted to the output above. Notice that the Camera and the marker detector components might differ based on the type +of hardware being used. + +
+ The configuration required for hand-eye calibration +
+ +The following YAML snippet contains the full application of the image above: + +
+ Example application, Hand-Eye calibration + +```yaml +schema: 2-0-6 +dependencies: + core: v5.1.0 +on_start: + load: + - component: orbbec_camera + - component: robot_camera_calibration + - component: stag_detector + - hardware: hardware +components: + orbbec_camera: + component: orbbec_camera::OBCameraNodeDriver + display_name: Orbbec Camera + outputs: + color_image: /orbbec_camera/color_image + color_camera_info: /orbbec_camera/color_camera_info + robot_camera_calibration: + component: core_vision_components::calibration::RobotCameraCalibration + display_name: Robot Camera Calibration + events: + transitions: + on_load: + lifecycle: + component: robot_camera_calibration + transition: configure + on_configure: + lifecycle: + component: robot_camera_calibration + transition: activate + parameters: + camera_frame: + value: orbbec_camera_link + type: string + marker_frame: + value: stag_0 + type: string + robot_base_frame: + value: world + type: string + robot_ee_frame: + value: ur_tool0 + type: string + is_camera_attached: + value: true + type: bool + stag_detector: + component: core_vision_components::pose_detection::STagDetector + display_name: STag Detector + events: + transitions: + on_load: + lifecycle: + component: stag_detector + transition: configure + on_configure: + lifecycle: + component: stag_detector + transition: activate + parameters: + marker_selection: + value: + - stag_0 + type: string_array + inputs: + image: /orbbec_camera/color_image + camera_info: /orbbec_camera/color_camera_info +hardware: + hardware: + display_name: Hardware Interface + urdf: Universal Robots 5e + rate: 500 + events: + transitions: + on_load: + load: + - controller: robot_state_broadcaster + hardware: hardware + - controller: ur_hand_guiding_controller + hardware: hardware + parameters: + robot_ip: 192.168.42.20 + controllers: + robot_state_broadcaster: + plugin: aica_core_controllers/RobotStateBroadcaster + events: + transitions: + on_load: + switch_controllers: + hardware: hardware + activate: robot_state_broadcaster + ur_hand_guiding_controller: + plugin: aica_ur_controllers/URHandGuidingController + parameters: + ft_sensor_name: + value: ur_tcp_fts_sensor + type: string + ft_sensor_reference_frame: + value: ur_tool0 + type: string + force_limit: + value: + - 20 + - 20 + - 20 + - 2 + - 2 + - 2 + type: vector + events: + transitions: + on_load: + switch_controllers: + hardware: hardware + activate: ur_hand_guiding_controller +graph: + positions: + on_start: + x: 120 + y: 0 + stop: + x: 120 + y: 100 + components: + orbbec_camera: + x: 340 + y: 0 + robot_camera_calibration: + x: 340 + y: 320 + stag_detector: + x: 800 + y: -80 + hardware: + hardware: + x: 1260 + y: -120 + edges: + on_start_on_start_robot_camera_calibration_robot_camera_calibration: + path: + - x: 260 + y: 60 + - x: 260 + y: 380 + on_start_on_start_stag_detector_stag_detector: + path: + - x: 260 + y: 60 + - x: 260 + y: -20 + on_start_on_start_hardware_hardware: + path: + - x: 260 + y: 60 + - x: 260 + y: -60 +``` + +
+ +:::tip Don't forget to modify the `robot_ip` according to the ip of the robot that you are using. + +::: + +There are several ways to use the transformation information obtained in the calibration file: + +1. Adding the camera link to the URDF. +2. Publish the transformation with a `FrameBroadcaster` manually. +3. Using the calibration file path directly as a Frame Broadcaster component parameter to publish the transformations. + +:::tip In order to use the calibration file path as a `Frame Broadcaster` component parameter, the format of the +calibration file needs to be slightly modified. In the next part the expected format is explained in more detail. ::: + +## Frame Broadcaster component + +The **Frame Broadcaster** is a component that parses a frame from the parameters and/or frames from the provided file +and broadcasts all available frames to TF. + +The parameters of the Robot Camera Calibration component are defined as follows: + +- **Rate**: Determines the frequency of all periodic callbacks. This parameter does not affect the component’s behavior. +- **Filepath**: This shows the path to the file containing a frame in YAML format. An example of the expected file + format is shown in the screenshot below. + +
+ The expected file format for the Frame Broadcaster +
+ +- **Frame**: Name of the frame that we wish to broadcast. +- **reference frame**: The name of the reference frame that the frame is defined according to it. +- **Pose values**: The values of Position (meters) and Orientation (quaternions) concatenated as a vector, as in x, y, + z, qw, qx, qy, qz. +- **Broadcast periodically**: If this parameter is set to True, the frames are broadcasted periodically. Otherwise it + publishes once immediately. + +:::info You either need to fill in the Filepath, or provide the `Frame Broadcaster` component with Frame, Reference +frame, and the Pose values parameters. ::: + +In the first screenshot below, you see an example of the `Frame broadcaster` component, and a transformation which is +being published manually. + +
+ Inserting the transformation into the frame Broadcaster manually. +
+ +In this second example, you see how the calibration filepath is given as a parameter to publish the transformations. + +
+ Using the calibration file path directly as a Frame Broadcaster component parameter to publish the transformations. +
diff --git a/docs/core/examples/guides/yolo-example.md b/docs/core/examples/guides/yolo-example.md index a91c6f39f..4f78c0662 100644 --- a/docs/core/examples/guides/yolo-example.md +++ b/docs/core/examples/guides/yolo-example.md @@ -1,5 +1,5 @@ --- -sidebar_position: 14 +sidebar_position: 16 title: Using YOLO to track objects ---