feat(sdk): builtin field tables, unprojectPixel and ImageAnnotations additions (0.36.0)

.claude/skills/plotjuggler-plugin/references/builtin-objects.md

@@ -114,10 +114,8 @@ exactly once. Do not keep a raw pointer into anything the closure captured.
 - `ImageAnnotations.image_topic` explicitly names the base image topic; camera
   calibration correlates by **name convention** (`<ns>/camera_info` ↔
   `<ns>/image_raw`), not by a reference.
-- **Association fields ride *outside* the serialized bytes.** The annotations
-  codec deliberately does not encode `image_topic` (or a timestamp) into the
-  payload — the store keys entries by (topic, timestamp), and association is a
-  topic/metadata convention. Don't expect a codec round-trip to carry them.
+- The annotations codec carries the top-level `timestamp` (only when non-zero) and
+  `image_topic` (only when non-empty); see `docs/image_annotations_format.md`.
 
 ## Codec round-trips are not bit-exact
 

CHANGELOG.md

@@ -3,6 +3,38 @@
 All notable changes to `plotjuggler_sdk` are recorded here. Versioning policy is in
 [`CLAUDE.md`](./CLAUDE.md) → "Release Versioning".
 
+## [0.36.0]
+
+Host contract: unchanged (no floor impact) — client-side helpers and an additive codec
+change, no new host surface.
+
+Builtin objects (client-side):
+
+- Add compile-time field tables (`pj_base/builtin/field_table.hpp`, one
+  `<type>_fields.hpp` per tabled struct, `describe(BuiltinObjectType)` in
+  `field_table_registry.hpp`) for `FrameTransforms`, `ImageAnnotations`, `PointCloud`,
+  `SceneEntities`, `Image`, `DepthImage`, `CameraInfo` and `VideoFrame`: a generic
+  binder reads and writes any described field by name. Packed payloads are `kBuffer`
+  descriptors; nullable numbers are `kOptionalNumber`; `std::array` members are
+  fixed-size `kList`s written through `list_replace`. A `kBuffer` layout never promises
+  more records than its bytes hold (`record_count` is 0 otherwise), and `set_number`
+  saturates to the target integer type (NaN becomes 0) instead of an undefined
+  float-to-integer conversion.
+- Add `sdk::PinholeIntrinsics` and `sdk::unprojectPixel` (`depth_image_utils.hpp`):
+  unproject a pixel with positive metric depth through K^-1, validating K once per
+  image. Skew and projective terms are honoured (a float32-rounded K is accepted);
+  only a non-finite or singular K is rejected. Distortion correction stays the
+  caller's job.
+- Add the object-topic metadata keys `pj_snapshot`
+  (`ObjectTopicMetadataBuilder::snapshot`: every entry is a complete clear-and-replace
+  snapshot) and `pj_derived` (`kDerivedMetadataKey`; set by a host on topics produced
+  by a host data processor, the value names the processor kind, an open set:
+  `kDerivedOnDemandValue` is `"on_demand"`).
+- The ImageAnnotations wire carries the top-level `timestamp` and `image_topic`;
+  additive, old readers skip them.
+- Fix the shared Timestamp decoder rejecting `INT64_MIN` nanoseconds (and anything within
+  one second of it): a payload our own encoder wrote failed to decode.
+
 ## [0.35.0]
 
 Host contract: unchanged (no floor impact) — a client-side codec helper, no new

VERSION

@@ -1 +1 @@
-0.35.0
+0.36.0

docs/builtin_type.md

@@ -673,6 +673,53 @@ splice form and decodes with an empty span; the host runs `validateGridMap()`
 right after attaching the spliced bytes, and consumers that index cells run it
 again defensively.
 
+## Field tables
+
+`pj_base/builtin/field_table.hpp` describes a builtin struct's members at
+compile time — name, shape (`FieldKind`), and type-erased get/set accessors —
+so a generic consumer (e.g. a script binder) can read or write any described
+field by name without hand-written per-type glue. A struct opts in with a
+`FieldTable<T>` specialization built from `field<&T::member>("member")` calls
+(a `kBuffer` factory such as `pointCloudDataField()` for a packed record buffer
+like `PointCloud::data`); nested
+structs and lists of described structs link to their own table through
+`FieldDescriptor::nested`, so a consumer walks an arbitrarily deep struct tree
+with one generic recursive routine. `field_table_registry.hpp` exposes
+`describe(BuiltinObjectType)` to look up a table from the runtime tag a
+`BuiltinObject` carries.
+
+Each tabled builtin has a `<type>_fields.hpp` header; `describe()` in
+`field_table_registry.hpp` is the authoritative list. `Image::data`,
+`DepthImage::data`, and `VideoFrame::data` each expose their packed pixel /
+bitstream bytes through a `kBuffer` descriptor the same way `PointCloud::data`
+does, with the per-record layout derived from the encoding/format string
+rather than a channel list (see each header's doc comment for how a raw vs.
+compressed encoding differs). `CameraInfo`'s `K`/`R`/`P` (and `DepthImage`'s
+`K`) are `std::array<double, N>` members, described as a fixed-size `kList`
+written through `list_replace` rather than `list_emplace`/`list_clear` (the
+element count never changes); `Image::compressed_depth_min`/
+`compressed_depth_max` are `kOptionalNumber` — a nullable number read/written
+like `kNumber` but guarded by `has_value()`. A field table is hand-maintained
+alongside its struct: adding, renaming, or retyping a member requires updating the matching
+`FieldTable<T>` specialization in the same change, or the table silently
+drifts from the struct it claims to describe. `pj_base/tests/field_table_test.cpp`
+enforces this — its generic copy-through-table round trip is compared against
+the struct's own `operator==` and its canonical wire-codec bytes, so a member
+the table forgot to list shows up as a round-trip mismatch rather than a
+silent gap.
+
+An object topic carrying a field-tabled type can mark itself with the
+`pj_snapshot` object-topic metadata key (`PJ::sdk::kSnapshotMetadataKey`,
+`ObjectTopicMetadataBuilder::snapshot()`, SDK 0.36.0). Value `"true"` means
+every entry on the topic is a complete clear-and-replace snapshot — a
+generic field-table consumer may render each entry alone, without replaying
+the topic's history to reconstruct state first. This matters for
+SceneEntities/ImageAnnotations producers in particular, whose entries can
+otherwise be incremental (add/update/delete deltas); a producer that instead
+republishes its whole set each time sets this flag so a stateless consumer
+(and `set_object_topic_retention(topic, 1)`, which keeps only the latest
+snapshot) can treat each entry as the full picture.
+
 ## Conversion Examples
 
 | Source type | Canonical builtin type | Conversion intent |

docs/image_annotations_format.md

@@ -66,16 +66,17 @@ the renderable annotation fields and ignores the rest:
 | `ImageAnnotations.circles` | Mapped to `ImageAnnotations::circles`. |
 | `ImageAnnotations.points` | Mapped to `ImageAnnotations::points`. |
 | `ImageAnnotations.texts` | Mapped to `ImageAnnotations::texts`. |
-| Top-level `timestamp` | Not serialized or decoded today. |
+| Top-level `timestamp` | Serialized when non-zero, decoded into `ImageAnnotations::timestamp`; a timestamp of 0 is indistinguishable from an absent one. |
+| `image_topic` (field 6, string) | Serialized when non-empty, decoded into `ImageAnnotations::image_topic`. |
 | Top-level `metadata` | Not serialized or decoded today. |
 | Per-annotation `timestamp` | Not serialized or decoded today. |
 | Per-annotation `metadata` | Not serialized or decoded today. |
 | `TextAnnotation.background_color` | Not represented by the SDK type; skipped on decode and not emitted on encode. |
 
-`PJ::sdk::ImageAnnotations::image_topic` is also not part of this payload. It is
-runtime association metadata used by PlotJuggler to attach overlays to an image
-stream. Adapters that need to preserve it across storage or transport must store
-it outside the `PJ.ImageAnnotations` bytes.
+`PJ::sdk::ImageAnnotations::image_topic` is part of the payload: it is written
+as field 6 when non-empty, so runtime association between an annotation set
+and its image stream can be carried on the wire. A reader predating this
+addition ignores the unknown field.
 
 ## Codec Rules
 
@@ -100,9 +101,10 @@ The writer never emits `UNKNOWN`/`0`. The reader maps unknown or zero topology
 values to `kPoints`, so incomplete or forward-compatible payloads still decode
 to a renderable annotation.
 
-An annotation value with no circles, points, or texts serializes to an empty byte
-buffer. Decoding an empty buffer is treated as invalid input by the current
-reader.
+An annotation value serializes to an empty byte buffer only when it has no circles,
+points, or texts, a zero `timestamp`, and an empty `image_topic`; one carrying just a
+non-zero `timestamp` and/or an `image_topic` produces a non-empty payload. Decoding an
+empty buffer is treated as invalid input by the current reader.
 
 The reader accepts protobuf wire types `VARINT`, `I64`, `LEN`, and `I32`. It
 decodes the mapped fields and skips unknown fields, including unknown nested

docs/sdk-utilities.md

@@ -60,6 +60,8 @@ CMake helpers become available through `find_package(plotjuggler_sdk)`.
 | `PJ::sdk::MediaMetadataBuilder, ObjectTopicMetadataBuilder` | Build media hints and canonical renderer metadata | `pj_base/sdk/media_metadata.hpp; pj_base/sdk/object_topic_metadata.hpp` |
 | `PJ::sdk::ObjectBytes` | Own object-read bytes through move-only RAII | `pj_base/sdk/object_bytes.hpp` |
 | `PJ::sdk::ArrowSchemaHolder, ArrowArrayHolder, ArrowStreamHolder` | Own Arrow out-params; successful writes transfer ownership, failed writes retain it | `pj_base/sdk/arrow.hpp` |
+| `PJ::sdk::FieldTable<T>, field<&T::member>, describe` | Read/write builtin struct members by name in a generic binder; lists are element-addressable | `pj_base/builtin/field_table.hpp; pj_base/builtin/field_table_registry.hpp` |
+| `PJ::sdk::PinholeIntrinsics, unprojectPixel` | Unproject rectified metric depth pixels; validate K once per image | `pj_base/builtin/depth_image_utils.hpp` |
 | `pj_embed_file, pj_configure_plugin` | Embed UI/manifest assets and configure plugin exports/build rules | `cmake/PjPlugin.cmake` |
 
 Outside this SDK, reuse pj-official-plugins common/:

pj_base/CLAUDE.md

@@ -18,7 +18,8 @@ build details. It must not depend on Qt, `pj_datastore` or `pj_plugins`.
 - `include/pj_base/builtin/` — 18 builtin struct headers (`*.hpp`) and all 18
   wire codecs (`*_codec.hpp`). Numeric tags are stable; values 2 and 12 are
   permanently reserved. Also contains the tagged, type-erased `BuiltinObject`
-  holder and type-erased codec dispatcher.
+  holder and type-erased codec dispatcher, and the compile-time field tables
+  (`field_table.hpp`, one `<type>_fields.hpp` per tabled struct).
 - `include/pj_base/sdk/` — C++ SDK over the ABI: DataSource + Toolbox `*_plugin_base.hpp`, `service_registry.hpp`/`service_traits.hpp`, host views, Arrow RAII holders, `testing/`.
 - `include/pj_base/*_protocol.h`, `plugin_data_api.h`, `builtin_object_abi.h`, `plugin_abi_export.hpp` — the stable C-ABI surface for DataSource/MessageParser/Toolbox (the Dialog protocol header lives in `pj_plugins/dialog_protocol/`).
 - `proto/pj/` — canonical `.proto` wire contracts for the builtin types (see its README).

pj_base/CMakeLists.txt

@@ -186,6 +186,7 @@ if(PJ_BUILD_TESTS)
     tests/frame_transforms_codec_test.cpp
     tests/image_annotations_codec_test.cpp
     tests/image_annotations_decoder_test.cpp
+    tests/field_table_test.cpp
     tests/media_metadata_test.cpp
     tests/object_topic_metadata_test.cpp
     tests/push_message_test.cpp

pj_base/include/pj_base/builtin/camera_info_fields.hpp

@@ -0,0 +1,39 @@
+/**
+ * @file camera_info_fields.hpp
+ * @brief `FieldTable` specialization for `camera_info.hpp`'s `CameraInfo`.
+ */
+// Copyright 2026 Davide Faconti
+// SPDX-License-Identifier: Apache-2.0
+
+#pragma once
+
+#include <array>
+
+#include "pj_base/builtin/camera_info.hpp"
+#include "pj_base/builtin/field_table.hpp"
+
+namespace PJ::sdk {
+
+/// Fields of `CameraInfo`. `K` and `R` are fixed-size lists of 9 numbers,
+/// `P` a fixed-size list of 12 (`std::array<double, N>` — see
+/// `field_table.hpp`'s `kList` fixed-size case, written through
+/// `list_replace` rather than `list_emplace`/`list_clear`); `D` is a
+/// growable list of numbers (`std::vector<double>`, size depends on
+/// `distortion_model`).
+template <>
+struct FieldTable<CameraInfo> {
+  static constexpr std::array fields{
+      field<&CameraInfo::timestamp_ns>("timestamp_ns"),
+      field<&CameraInfo::frame_id>("frame_id"),
+      field<&CameraInfo::width>("width"),
+      field<&CameraInfo::height>("height"),
+      field<&CameraInfo::distortion_model>("distortion_model"),
+      field<&CameraInfo::D>("D"),
+      field<&CameraInfo::K>("K"),
+      field<&CameraInfo::R>("R"),
+      field<&CameraInfo::P>("P"),
+  };
+  static constexpr FieldTableView view{"CameraInfo", Span<const FieldDescriptor>(fields)};
+};
+
+}  // namespace PJ::sdk

pj_base/include/pj_base/builtin/depth_image_fields.hpp

@@ -0,0 +1,95 @@
+/**
+ * @file depth_image_fields.hpp
+ * @brief `FieldTable` specialization for `depth_image.hpp`'s `DepthImage`.
+ */
+// Copyright 2026 Davide Faconti
+// SPDX-License-Identifier: Apache-2.0
+
+#pragma once
+
+#include <array>
+#include <cstdint>
+
+#include "pj_base/builtin/depth_image.hpp"
+#include "pj_base/builtin/field_table.hpp"
+#include "pj_base/builtin/point_cloud.hpp"
+
+namespace PJ::sdk {
+
+namespace detail {
+
+/// Pixel datatype of a `DepthImage::encoding` value, or `kUnknown` when the
+/// string is not one of the two encodings `depth_image.hpp`'s doc comment
+/// documents ("16UC1" — millimeters as uint16 — or "32FC1" — meters as
+/// float32). `DepthImage::encoding` is an open string (like `Image::encoding`),
+/// so an unrecognized value is expected, not an error.
+[[nodiscard]] constexpr PointField::Datatype depthImageDatatype(std::string_view encoding) noexcept {
+  if (encoding == "16UC1") {
+    return PointField::Datatype::kUint16;
+  }
+  if (encoding == "32FC1") {
+    return PointField::Datatype::kFloat32;
+  }
+  return PointField::Datatype::kUnknown;
+}
+
+}  // namespace detail
+
+/// `kBuffer` descriptor for `DepthImage::data`. Unlike `Image`, `DepthImage`
+/// has no `row_step` member, so `buffer()` derives it as `width * bpp` (no
+/// row padding representable) whenever `bpp` (from
+/// `bytesPerElement(detail::depthImageDatatype(encoding))`) is known; an unrecognized
+/// encoding reports `record_step = row_step = 0` and `bytes` alone (its full
+/// span) is the payload — same convention as `imageDataField()`
+/// (`image_fields.hpp`); a declared extent larger than `data` also reports 0
+/// records. `channels` always carries exactly one entry naming
+/// the encoding string. `buffer_assign()` is `detail::assignPayloadBytes`.
+[[nodiscard]] constexpr FieldDescriptor depthImageDataField(std::string_view name) {
+  FieldDescriptor d{};
+  d.name = name;
+  d.kind = FieldKind::kBuffer;
+  d.buffer = [](const void* p) -> BufferLayout {
+    const auto& image = *static_cast<const DepthImage*>(p);
+    const PointField::Datatype datatype = detail::depthImageDatatype(image.encoding);
+    const uint32_t bpp = bytesPerElement(datatype);
+    BufferLayout layout;
+    layout.bytes = image.data;
+    layout.record_step = bpp;
+    const bool fits = detail::rowsFitIn(image.height, static_cast<uint64_t>(image.width) * bpp, 0, image.data.size());
+    layout.record_count = bpp == 0 || !fits ? 0 : static_cast<uint64_t>(image.width) * image.height;
+    layout.row_step = bpp == 0 ? 0 : image.width * bpp;
+    layout.is_bigendian = false;  // DepthImage carries no endianness member; native in-memory doubles/ints.
+    layout.channels.push_back(
+        BufferLayout::Channel{
+            .name = image.encoding,
+            .offset = 0,
+            .datatype = static_cast<uint8_t>(datatype),
+            .count = bpp == 0 ? 0 : uint32_t{1}});
+    return layout;
+  };
+  d.buffer_assign = &detail::assignPayloadBytes<DepthImage>;
+  return d;
+}
+
+/// Fields of `DepthImage`: every member is an ordinary field except `data`
+/// (`depthImageDataField`); `anchor` is written through that descriptor's
+/// `buffer_assign()` and is not listed. `K` is a fixed-size list of 9
+/// numbers (`std::array<double, 9>` — see `field_table.hpp`'s `kList`
+/// fixed-size case); `D` is a growable list of numbers
+/// (`std::vector<double>`, size depends on `distortion_model`).
+template <>
+struct FieldTable<DepthImage> {
+  static constexpr std::array fields{
+      field<&DepthImage::width>("width"),
+      field<&DepthImage::height>("height"),
+      field<&DepthImage::encoding>("encoding"),
+      depthImageDataField("data"),
+      field<&DepthImage::K>("K"),
+      field<&DepthImage::distortion_model>("distortion_model"),
+      field<&DepthImage::D>("D"),
+      field<&DepthImage::timestamp_ns>("timestamp_ns"),
+  };
+  static constexpr FieldTableView view{"DepthImage", Span<const FieldDescriptor>(fields)};
+};
+
+}  // namespace PJ::sdk

pj_base/include/pj_base/builtin/depth_image_utils.hpp

@@ -15,12 +15,81 @@
 #pragma once
 
 #include <array>
+#include <cmath>
+#include <optional>
 
 #include "pj_base/builtin/depth_image.hpp"
 
 namespace PJ {
 namespace sdk {
 
+/// Camera matrix K reduced to its inverse, built once per image so a per-pixel loop
+/// checks only its own inputs.
+class PinholeIntrinsics {
+ public:
+  /// Rejects a non-finite or singular K and nothing else: skew and projective terms
+  /// are honoured, so a K that went through float32 is accepted. Distortion
+  /// correction belongs to the caller.
+  [[nodiscard]] static std::optional<PinholeIntrinsics> fromK(const std::array<double, 9>& k) noexcept {
+    for (double value : k) {
+      if (!std::isfinite(value)) {
+        return std::nullopt;
+      }
+    }
+    const double cofactor0 = k[4] * k[8] - k[5] * k[7];
+    const double cofactor1 = k[5] * k[6] - k[3] * k[8];
+    const double cofactor2 = k[3] * k[7] - k[4] * k[6];
+    const double det = k[0] * cofactor0 + k[1] * cofactor1 + k[2] * cofactor2;
+    if (!std::isfinite(det) || det == 0) {
+      return std::nullopt;
+    }
+    const std::array<double, 9> inverse{
+        cofactor0 / det, (k[2] * k[7] - k[1] * k[8]) / det, (k[1] * k[5] - k[2] * k[4]) / det,
+        cofactor1 / det, (k[0] * k[8] - k[2] * k[6]) / det, (k[2] * k[3] - k[0] * k[5]) / det,
+        cofactor2 / det, (k[1] * k[6] - k[0] * k[7]) / det, (k[0] * k[4] - k[1] * k[3]) / det};
+    for (double value : inverse) {
+      if (!std::isfinite(value)) {
+        return std::nullopt;
+      }
+    }
+    return PinholeIntrinsics(inverse);
+  }
+
+  /// Unproject a rectified pixel with positive metric depth: the ray K^-1 [u, v, 1]
+  /// scaled so the returned point's z equals `depth_m`. Rejects a non-finite result
+  /// or a ray with z <= 0.
+  [[nodiscard]] std::optional<std::array<double, 3>> unproject(double u, double v, double depth_m) const noexcept {
+    if (!std::isfinite(u) || !std::isfinite(v) || !std::isfinite(depth_m) || depth_m <= 0) {
+      return std::nullopt;
+    }
+    const double ray_z = k_inverse_[6] * u + k_inverse_[7] * v + k_inverse_[8];
+    if (!std::isfinite(ray_z) || ray_z <= 0) {
+      return std::nullopt;
+    }
+    const double scale = depth_m / ray_z;
+    const double point_x = (k_inverse_[0] * u + k_inverse_[1] * v + k_inverse_[2]) * scale;
+    const double point_y = (k_inverse_[3] * u + k_inverse_[4] * v + k_inverse_[5]) * scale;
+    if (!std::isfinite(point_x) || !std::isfinite(point_y)) {
+      return std::nullopt;
+    }
+    return std::array<double, 3>{point_x, point_y, depth_m};
+  }
+
+ private:
+  explicit PinholeIntrinsics(const std::array<double, 9>& k_inverse) noexcept : k_inverse_(k_inverse) {}
+
+  /// Row-major K^-1; only `fromK` builds it, so every entry is finite.
+  std::array<double, 9> k_inverse_;
+};
+
+/// One-pixel form of `PinholeIntrinsics::fromK(k)->unproject(u, v, depth_m)`; a loop
+/// over an image should call `fromK` once instead.
+[[nodiscard]] inline std::optional<std::array<double, 3>> unprojectPixel(
+    const std::array<double, 9>& k, double u, double v, double depth_m) noexcept {
+  const auto intrinsics = PinholeIntrinsics::fromK(k);
+  return intrinsics ? intrinsics->unproject(u, v, depth_m) : std::nullopt;
+}
+
 /// Rectification rotation. For a DepthImage with empty distortion_model
 /// (i.e. rectified) this returns the identity rotation. Unrectified
 /// depth has no canonical rectification rotation — the caller has the