From 615a7bf66f9d57483d703c236c941af2947bb826 Mon Sep 17 00:00:00 2001 From: Adam Kinniburgh Date: Thu, 22 Oct 2026 13:55:48 +0100 Subject: [PATCH 1/2] feat: document the array shape --- .../build-plugin/references/data-streams.md | 48 ++++++++++++++++++- .../build-plugin/references/index-defs.md | 2 + .../build-plugin/references/test-agent.md | 2 +- .../skills/build-plugin/references/testing.md | 8 ++++ REVIEW.md | 1 + 5 files changed, 58 insertions(+), 3 deletions(-) diff --git a/.claude/skills/build-plugin/references/data-streams.md b/.claude/skills/build-plugin/references/data-streams.md index 450eb269..ba9fbf80 100644 --- a/.claude/skills/build-plugin/references/data-streams.md +++ b/.claude/skills/build-plugin/references/data-streams.md @@ -578,7 +578,7 @@ Always use `displayName` — column names in scripts are often terse API field n - **Explicit set only**: list each column; unlisted columns are hidden. - **Mix**: list specific columns with shapes/roles, then add the pattern for the rest. -**Type inference**: SquaredUp infers column types from the JS primitive returned. Return the correct primitive type from your script — don't coerce to string. Declared `shape` overrides inference. +**Type inference**: SquaredUp infers column types from the JS primitive returned. Return the correct primitive type from your script — don't coerce to string. Declared `shape` overrides inference. A list is the exception — it is never inferred; an array is serialised to a JSON string unless the column declares the `array` shape (see [List columns](#list-columns-array-shape)). ### Shapes @@ -626,7 +626,13 @@ _Special_ | `"guid"` | GUID | | `"customunit"` | Custom unit label. Options: `prefix`, `separator` | -**Array form** — use when you need formatting options: +_List_ + +| Shape | Notes | +| --------- | ------------------------------------------------------------------------------------------------------ | +| `"array"` | A list of values. Options: `itemShape` (any shape above). See [List columns](#list-columns-array-shape) | + +**With options** — use the `[name, options]` form when you need formatting options: ```json { "name": "price", "shape": ["number", { "decimalPlaces": 2 }] } @@ -636,6 +642,38 @@ _Special_ { "name": "cost", "shape": ["currency", { "code": "jpy" }] } ``` +### List columns (`array` shape) + +When a field holds several values — tags, topics, IP addresses, environments — and should stay a list, declare the `array` shape: + +```json +{ "name": "topics", "displayName": "Topics", "shape": ["array", { "itemShape": "string" }] } +{ "name": "ports", "displayName": "Ports", "shape": ["array", { "itemShape": ["number", { "thousandsSeparator": false }] }] } +{ "name": "seenAt", "displayName": "Seen at", "shape": ["array", { "itemShape": "date" }] } +``` + +Without it the platform serialises the array to a JSON string (`"[\"devops\",\"cli\"]"`). Don't work around that by joining to a comma-separated string or adding a display twin of the column: one list column feeds table display, tile filtering, the import and `objectPropertyPath` read-back. + +**`itemShape`** takes any shape from the tables above, in either form: a name (`"string"`, `"number"`, `"boolean"`, `"date"`, `"url"`, `"state"`, `"bytes"`, `"seconds"`, `"percent"`, …) or `[name, options]`. Each member is converted to and formatted as that shape. `"shape": "array"` with no `itemShape` infers it from the first non-null member. Declare it anyway, so the column's type doesn't depend on whichever row came first. + +**The value** is a real JS array of primitives: the raw API array via `pathToData`, or an array your script builds (split a delimited string here, rather than in the tile). A serialised JSON array string, for example from a `valueExpression`, is parsed back into members too. + +| Member | Result | +| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------ | +| `null` | Dropped | +| An object, alongside primitives | Dropped | +| Every member an object (`[{ "name": "a" }]`) | The whole cell stays a JSON string. Map to primitives in the script first: `tags.map((t) => t.name)` | +| Not convertible (`"abc"` with a `number` item) | The whole cell becomes no data, not a shorter list | +| Empty list | An empty list, rendered blank | + +**Downstream behaviour:** + +- **Display:** the members are joined with `, `. +- **Tile filters** test each member. `contains` and `equals` keep a row if any member matches, and a negated filter drops a row if any member matches. Numeric and date comparisons need a `number` or `date` `itemShape`. They are refused on a list of strings, just as on a string column. +- **Not a scalar:** a list column is never picked as a tile's value column, so scalar, gauge and bar tiles still need a separate numeric column. +- **Import:** a list column mapped into `objectMapping.properties` is written as a multi-value graph property. See [index-defs.md](index-defs.md). +- **Read-back:** an `objectPropertyPath` column that reads a list property must declare the `array` shape as well. Without it you get only the first value. See [Object property lookup](#object-property-lookup). + ### Roles | Role | Description | @@ -738,8 +776,14 @@ Replace a raw ID column with a human-readable property from a related indexed ob // Combine properties { "name": "AgentLabel", "sourceId": "AgentID", "sourceType": "my-agent", "objectPropertyPath": "name", "valueExpression": "{{ object.name }} ({{ object.company }})" } + +// Read back a multi-value property — the array shape returns every value, not just the first +{ "name": "repoTopics", "displayName": "Repository topics", "sourceId": "repoId", "sourceType": "Repository", + "objectPropertyPath": "topics", "shape": ["array", { "itemShape": "string" }] } ``` +Appending a related object's list property this way lets tiles filter and group a stream's rows by it (e.g. issues filtered by their repository's topics). See [List columns](#list-columns-array-shape). + --- ## Post-request scripts diff --git a/.claude/skills/build-plugin/references/index-defs.md b/.claude/skills/build-plugin/references/index-defs.md index fab10aa2..b30bbb26 100644 --- a/.claude/skills/build-plugin/references/index-defs.md +++ b/.claude/skills/build-plugin/references/index-defs.md @@ -59,10 +59,12 @@ Defines what gets imported into the SquaredUp graph. - `objectMapping.type` maps to the `sourceType` column. It can also be a fixed string: `{ "value": "My Device" }` — use when all rows are the same type, rather than a computed column. - `objectMapping.properties` are extra fields stored on the graph node, accessible in scripts as `object.propName`. - Use `{ "targetProp": "sourceProp" }` syntax when the column name differs from the desired property name. +- **A list-valued property needs a list column.** Map a column that declares `"shape": ["array", { "itemShape": "string" }]` and the import writes a real multi-value property, one value per member. A list without that shape arrives as a JSON string and is stored as one text value. See [List columns](data-streams.md#list-columns-array-shape). - **Never map a column into `properties` that is already mapped as `id`, `name`, or `type`.** The id's raw value is always available on every object as `rawId` (`{{object.rawId}}` in templates, `context.objects[N].rawId` in scripts — and as a scalar, unlike user-defined properties, which arrive as arrays), and the name as `name`. Adding e.g. `{ "projectId": "id" }` to `properties` when `"id": "id"` already exists creates a duplicate that has to be re-indexed to take effect and otherwise sits as dead config — use `rawId`/`name` instead. - **Never use a reserved name as a property name.** The property name — a string entry, or the *key* in `{ "targetProp": "sourceProp" }` — becomes a property on the graph object, and the graph owns some names. Using one deploys cleanly, then fails the import with a raw schema error that doesn't mention your mapping: - `id`, and any `__`-prefixed name — always rejected: `Payload failed schema validation` … `/vertices/0/id` … `boolean schema is false`. You don't need it — the object's id is already `rawId` (above). - `state` — reserved by the platform; an API status (`"active"`) or a postal state (`"OH"`) fails with `must be equal to one of the allowed values`. Rename it: `{ "status": "state" }`, or `{ "region": "state" }` for an address. + - `tags` — reserved by the graph, with its own schema; a plugin's tag list fails with `/vertices/0/tags` … `must be array`. Prefix it: `{ "checkTags": "tags" }`. - `rawId`, `sourceId`, `sourceType`, `name`, `label` — silently overwritten by the platform, so the mapping does nothing. A reserved name is fine as the *source column* (`{ "status": "state" }` reads a column called `state`). `squaredup validate` rejects `id`, `state` and `__` names; it can't catch the silent overwrites. diff --git a/.claude/skills/build-plugin/references/test-agent.md b/.claude/skills/build-plugin/references/test-agent.md index a3bd2694..18bc7a9f 100644 --- a/.claude/skills/build-plugin/references/test-agent.md +++ b/.claude/skills/build-plugin/references/test-agent.md @@ -31,7 +31,7 @@ For **build-mode**, also pass the **build spec** distilled from the Phase 2 plan - Endpoint path + HTTP method, and scoping → `baseDataSourceName`: single-object → `httpRequestScopedSingle`, multi-object → `httpRequestScoped`, unscoped/global → `httpRequestUnscoped`. - Candidate `pathToData` (where the row array lives in the response). -- Planned columns with `displayName`, `shape`, and any `role` (label/value/timestamp/sourceId…). +- Planned columns with `displayName`, `shape`, and any `role` (label/value/timestamp/sourceId…). Flag any list-valued field as `["array", { "itemShape": … }]` (see [List columns](data-streams.md#list-columns-array-shape)). Otherwise the sub-agent sees a JSON string in the shaped rows and "fixes" it by joining. - Any `ui` parameters, `timeframes` setting, paging mode, and the `matches` selector. - The endpoint's **data granularity** (from Phase 1) and, when the stream supports a time range coarser than the default (`timeframes` is an array or `true` at daily/monthly granularity), a **first-test `--timeframe` hint**. `squaredup test` defaults to `last1hour`, which 404s against an endpoint aggregated to daily/monthly (e.g. billing, usage), so tell the sub-agent the timeframe to pass on its first test — e.g. "daily granularity: first test with `--timeframe last7days`". This stops every sub-agent from independently rediscovering the default-timeframe 404. Omit the hint for streams with no time-range control (`timeframes: false`) — there is no timeframe to pass. diff --git a/.claude/skills/build-plugin/references/testing.md b/.claude/skills/build-plugin/references/testing.md index 10a7e78a..785a3c69 100644 --- a/.claude/skills/build-plugin/references/testing.md +++ b/.claude/skills/build-plugin/references/testing.md @@ -101,6 +101,14 @@ Example cell (a shaped currency value): { "raw": "1234.567", "value": 1234.57, "formatted": "$1,234.57" } ``` +Example cell (a [list column](data-streams.md#list-columns-array-shape), `"shape": ["array", { "itemShape": "string" }]`): + +```json +{ "raw": ["devops", "cli"], "value": ["devops", "cli"], "formatted": "devops, cli" } +``` + +For a list column, check `value`, not `formatted`. `formatted` is always the members joined with `, ` for display, so a plain string there is expected. `value` must be an array (its column's `shapeName` is `shape_array`). If `value` is the JSON string `"[\"devops\",\"cli\"]"`, the column is missing the `array` shape, or every member is an object that needs mapping to a primitive first. Fix the shape. Don't join the members in the script. + `data` is **absent** when shaping produced nothing (e.g. the request errored). `test` sets a non-zero exit code when any request errored — inspect the error and the `[Response] Body`, fix the stream, and re-test. ### Zooming in diff --git a/REVIEW.md b/REVIEW.md index d53f0b38..f72ee4a8 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -95,6 +95,7 @@ When suggesting changes: - timeframe - Data streams that don't support time-based filtering must explicitly declare `"none"` as an available timeframe. Omitting this causes UI warnings and incorrect default behaviour in OOB dashboards. - Numeric values - Return raw numeric values rather than pre-formatted strings (e.g. return `1234.56` not `"£1,234.56"`). Use column format expressions in dashboards for display formatting. - Column shapes - Use the `bytes` shape for columns representing byte sizes; SquaredUp will automatically display the most appropriate unit (e.g. 4.2 GB). Check for other applicable semantic shapes such as `timestamp` and `duration`. +- List values - A column holding several values (tags, topics, addresses) uses `"shape": ["array", { "itemShape": "" }]`, not a comma-joined string, and not a second display column alongside it. An `objectPropertyPath` column reading a list property declares the same shape. - Timestamps - SquaredUp expects ISO 8601 strings for timestamp columns. If the upstream API returns Unix timestamps, the script must convert them. - Expressions - Where a transformation can be expressed using `map`, prefer it over a mustache-style value expression, or complex processing in a script for performance. - Deduplication - If multiple data streams share the same API endpoint and differ only by a filter, consider merging them into a single stream with a UI parameter to control the filter. From 34126019d63352188a2832206d1c5c8b122b4452 Mon Sep 17 00:00:00 2001 From: Adam Kinniburgh Date: Thu, 22 Oct 2026 14:28:37 +0100 Subject: [PATCH 2/2] docs: address review on list-column guidance Separate the two causes of a JSON-string list value in testing.md, and add tags to the reserved-property checklist in REVIEW.md. Co-Authored-By: Claude Opus 5.5 --- .claude/skills/build-plugin/references/testing.md | 2 +- REVIEW.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.claude/skills/build-plugin/references/testing.md b/.claude/skills/build-plugin/references/testing.md index 785a3c69..26991490 100644 --- a/.claude/skills/build-plugin/references/testing.md +++ b/.claude/skills/build-plugin/references/testing.md @@ -107,7 +107,7 @@ Example cell (a [list column](data-streams.md#list-columns-array-shape), `"shape { "raw": ["devops", "cli"], "value": ["devops", "cli"], "formatted": "devops, cli" } ``` -For a list column, check `value`, not `formatted`. `formatted` is always the members joined with `, ` for display, so a plain string there is expected. `value` must be an array (its column's `shapeName` is `shape_array`). If `value` is the JSON string `"[\"devops\",\"cli\"]"`, the column is missing the `array` shape, or every member is an object that needs mapping to a primitive first. Fix the shape. Don't join the members in the script. +For a list column, check `value`, not `formatted`. `formatted` is always the members joined with `, ` for display, so a plain string there is expected. `value` must be an array (its column's `shapeName` is `shape_array`). If `value` is a JSON string such as `"[\"devops\",\"cli\"]"`, there are two causes. If the column is missing the `array` shape, add it. If every member is an object, the shape alone won't fix it: map the members to primitives in the script first. Either way, don't join the members in the script. `data` is **absent** when shaping produced nothing (e.g. the request errored). `test` sets a non-zero exit code when any request errored — inspect the error and the `[Response] Body`, fix the stream, and re-test. diff --git a/REVIEW.md b/REVIEW.md index f72ee4a8..ce1ee25d 100644 --- a/REVIEW.md +++ b/REVIEW.md @@ -106,7 +106,7 @@ When suggesting changes: ### Source types (indexDefinitions/) - Name source types after how they are referred to in the upstream product or API (e.g. `agent`, `device`). Do not prefix them with the plugin name (e.g. avoid `NinjaOne Device`). A separate friendly display name can be configured if needed (via custom_types.json). -- objectMapping.properties - Never use `id`, `state`, or a `__`-prefixed name as a property name; they fail the import at graph write. Rename instead, e.g. `{ "status": "state" }`. `rawId`, `sourceId`, `sourceType`, `name` and `label` are silently overwritten, so mapping them does nothing. +- objectMapping.properties - Never use `id`, `state`, `tags`, or a `__`-prefixed name as a property name; they fail the import at graph write. Rename instead, e.g. `{ "status": "state" }` or `{ "checkTags": "tags" }`. `rawId`, `sourceId`, `sourceType`, `name` and `label` are silently overwritten, so mapping them does nothing. - pageSize - For APIs where a tenant can hold tens of thousands of objects, check the import stream's `pageSize` against both import limits: ~6MB per page and ~220 pages per import run. See the build-plugin skill's index-defs.md, "Large imports". ### Documentation - (docs/README.md)