diff --git a/docs/core/examples/guides/.vscode/settings.json b/docs/core/examples/guides/.vscode/settings.json new file mode 100644 index 000000000..70ab33523 --- /dev/null +++ b/docs/core/examples/guides/.vscode/settings.json @@ -0,0 +1,3 @@ +{ + "cSpell.words": ["AICA", "Freedrive", "Orbbec", "servoing", "URDF"] +} diff --git a/docs/core/examples/guides/assets/stag-detector-example.png b/docs/core/examples/guides/assets/stag-detector-example.png new file mode 100644 index 000000000..1a4f7f36d Binary files /dev/null and b/docs/core/examples/guides/assets/stag-detector-example.png differ diff --git a/docs/core/examples/guides/assets/stag-marker-detection.webm b/docs/core/examples/guides/assets/stag-marker-detection.webm new file mode 100644 index 000000000..332591ce4 Binary files /dev/null and b/docs/core/examples/guides/assets/stag-marker-detection.webm differ diff --git a/docs/core/examples/guides/assets/stagDetector-predicates_0.png b/docs/core/examples/guides/assets/stagDetector-predicates_0.png new file mode 100644 index 000000000..2cd136b92 Binary files /dev/null and b/docs/core/examples/guides/assets/stagDetector-predicates_0.png differ diff --git a/docs/core/examples/guides/assets/stagDetector-predicates_1.png b/docs/core/examples/guides/assets/stagDetector-predicates_1.png new file mode 100644 index 000000000..a07fefb48 Binary files /dev/null and b/docs/core/examples/guides/assets/stagDetector-predicates_1.png differ diff --git a/docs/core/examples/guides/fiducial-markers.md b/docs/core/examples/guides/fiducial-markers.md new file mode 100644 index 000000000..2028fb208 --- /dev/null +++ b/docs/core/examples/guides/fiducial-markers.md @@ -0,0 +1,240 @@ +--- +sidebar_position: 14 +title: Fiducial Markers +--- + +import stagDetectorExample from './assets/stag-detector-example.png' +import stagMarkerDetection from './assets/stag-marker-detection.webm' +import stagMarkerNumOne from './assets/stagDetector-predicates_1.png' +import stagMarkerNumZero from './assets/stagDetector-predicates_0.png' + +# Fiducial Markers + +Different types of fiducial markers are used in robotics to provide precise 3D pose estimation and identification for +cameras, enabling or improving robotic calibration and object manipulation. + +AICA's `core-vision` package gives you the choice between using two commonly used markers, the STag and ArUco. + +:::tip +Performing the [intrinsic calibration](./camera-calibration.md) of the camera improves the precision for fiducial +marker detection and tracking. +::: + +This guide provides an example of STag marker detection. Using the ArUco marker follows a very similar process. + +## Preparing fiducial markers + +A fiducial marker is an object placed in the field of view of an image for use as a point of reference or a measure. +STag and ArUco markers are two of the common types of fiducial marker systems used for real-time 6D pose estimation. +This section explains how to obtain, download and print these markers. + +### Obtaining markers + +- **ArUco marker**: ArUco markers can be generate online (e.g., from [here](https://chev.me/arucogen/)), which permits + choosing the dictionary, marker ID, and marker size. It can be then exported as PDF or SVG for printing. + +- **STag marker**: STag marker set can be either downloaded from + [public Google Drive](https://drive.google.com/drive/folders/0ByNTNYCAhWbIV1RqdU9vRnd2Vnc?resourcekey=0-9ipvecbezW8EWUva5GBQTQ) + or obtained from the [ROS2 STag project repository](https://github.com/usrl-uofsc/stag_ros/tree/ros2-devel) or the + generator/reference files linked by the project. In practice, you’ll want to obtain the marker PDF/SVG or generate the + markers from the project’s reference generator, then print them at true size. + +### Printing markers + +After choosing the marker family, selecting the library/dictionary, and the marker ID, download it as PDF or SVG. Use +the actual size of the marker (100% scale) for printing, so the black border and marker geometry are not resized. Also +it is recommended to print with high contrast and avoid compression artifacts. + +If possible, print on a rigid and flat sheet of paper to reduce warping, since fiducial detection is sensitive to +distortion. As another solution, you can fix the printed marker on a rigid surface, such as a piece of wood or +cardboard. + +After printing, measure the marker’s outer dimensions and compare them with the intended size from the generator. This +matters because calibration will be wrong if the marker size in the software does not match the physical print. + +## Using the STag detector + +Launch AICA Studio with a configuration that contains the `core-vision` package and create a new application. + +1. Remove the hardware interface that is included in new applications by default. +2. From the `Scene` menu, use the `Add Component` tab and look for the **Camera Streamer** and **STag Detector** + components, either by searching or by manually going under the `Core Vision Components` menu. Add both of them to the + graph. +3. Next, connect both components to the start block. Moreover, connect the outputs of the Camera Streamer to the + relevant inputs of the STag Detector. +4. Enable **auto-configure** and **auto-activate** on both components. +5. By selecting any of the components, you can find all the available component parameters in the right panel under + Settings. +6. If an intrinsic camera calibration is performed prior to this, add the file path of the camera configuration file as + a parameter to the **Camera Streamer** component. + +By this point, you should have something like the following: + +
+ CameraStreamer configuration alongside STagDetector component +
+ +:::info +The Camera Streamer parameters are explained in the [CameraStreamer component guide](./camera-streamer.md). +::: + +## STag Detector parameters + +- **Rate**: The rate parameter doesn't affect the behavior of the component as the detection process occurs on reception + of a new image. +- **Bundle file**: The filepath to a predefined marker bundle configuration. This additional feature is described in a + separate guide (coming soon). +- **Marker selection**: The name(s) of the marker(s) that we want to recognize. If any of these markers enters the + camera frame, the `is_any_selected_marker_detected` predicate will switch to **True** (see more in the section below). + The marker names should always be prepended with the value of `Prefix`. +- **Marker size**: The measured side length of the marker in meters. +- **Library**: This is the ID number of the HD library utilized by STag markers. Allowed options are + `[11, 13, 15, 17, 19, 21, 23]`. +- **Error correction**: This parameter sets how permissive the detector is. It controls how many bit errors the STag + detector is allowed to tolerate when decoding a marker ID. Lower values mean stricter matching, but detection would be + less robust to blur, noise, bad lighting, or partial image degradation. Higher values mean more tolerant matching. The + detector can still recognize a marker even if some bits are read incorrectly, so detection is more robust, but the + risk of wrong matches increases. The lower and upper limits for this parameter are 0 and 11 respectively. +- **Prefix**: This is the prefix used for marker names. + +## STag Detector predicates + +- **Is any marker detected**: This predicate will be set to **True** if any marker is detected in the camera frame, even + though its name is not indicated in the 'Marker Selection' parameter. As you can see in the screenshot below, the + marker name specified in the `Marker selection` parameter is stag_1, but the marker recognized in the camera frame is + stag_0, yet `Is any marker detected` predicate is set to **True**. + +
+ Is any marker detected at all +
+ +- **Is any selected marker detected**: If one or more marker names are indicated in the `Marker selection` parameter, + and if any of them appears in the camera frame, this predicate with be set to **True**. If the names of none of the + markers present in the camera frame is indicated as a parameter, this predicate will remain **False**. In the + screenshot below the name of the marker appearing in the camera frame matches the name indicated in the + `Marker selection` parameter. + +
+ Is any of the selected markers detected +
+ +- **Is a marker bundle detected**: If a registered group of markers is detected by the camera, this parameter will be + set to **True**. Otherwise it will remain **False**. + +## Running the application + +After setting up the proper parameters for Camera Streamer and STag Detector: + +1. Press **Start** to start the application. +2. To see the live camera feed, select **Launch RViz** from the Launcher settings +3. In RViz, select _Add > By topic > /stag_detector/annotated_image > Image_. This adds a panel that shows the live + image. The marker should be detected in the camera. + +
+ +
+ +

+ +The following YAML snippet contains the full application of the STag Detector above: + +
+ Example application, STag Detector + +```yaml +schema: 2-0-6 +dependencies: + core: v5.1.0 +on_start: + load: + - component: stag_detector + - component: camera_streamer +components: + stag_detector: + component: core_vision_components::pose_detection::STagDetector + display_name: STag Detector + events: + transitions: + on_configure: + lifecycle: + component: stag_detector + transition: activate + on_load: + lifecycle: + component: stag_detector + transition: configure + parameters: + marker_selection: + value: + - stag_0 + type: string_array + inputs: + image: /camera_streamer/image + camera_info: /camera_streamer/camera_info + camera_streamer: + component: core_vision_components::image_streaming::CameraStreamer + display_name: Camera Streamer + events: + transitions: + on_configure: + lifecycle: + component: camera_streamer + transition: activate + on_load: + lifecycle: + component: camera_streamer + transition: configure + parameters: + camera_info_path: + value: /data/ost.yaml + type: string + undistorted_image_cropping: + value: false + type: bool + outputs: + image: /camera_streamer/image + camera_info: /camera_streamer/camera_info +graph: + positions: + on_start: + x: -60 + y: 0 + stop: + x: -60 + y: 100 + components: + stag_detector: + x: 820 + y: 0 + camera_streamer: + x: 200 + y: 60 + edges: + on_start_on_start_camera_streamer_camera_streamer: + path: + - x: 100 + y: 60 + - x: 100 + y: 120 + camera_streamer_image_stag_detector_image: + path: + - x: 700 + y: 280 + - x: 700 + y: 260 + camera_streamer_camera_info_stag_detector_camera_info: + path: + - x: 660 + y: 360 + - x: 660 + y: 300 +``` + +
+ +:::info +The process for using ArUco markers follows a similar process. +::: diff --git a/docs/core/examples/guides/yolo-example.md b/docs/core/examples/guides/yolo-example.md index a91c6f39f..4f78c0662 100644 --- a/docs/core/examples/guides/yolo-example.md +++ b/docs/core/examples/guides/yolo-example.md @@ -1,5 +1,5 @@ --- -sidebar_position: 14 +sidebar_position: 16 title: Using YOLO to track objects ---