You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f2e00a2
Browse filesBrowse the repository at this point in the historyBrowse files
CoPA's CrowdStrike import (~92,700 hosts) hit two limits the skill
never mentioned. At 1,000 hosts per page one page overflowed the
~6MB Lambda response cap, and dropping to 250 per page would have
needed ~371 calls against a ~220 call-chain cap per import run. The
skill also claimed SquaredUp "supports as many paged requests as
needed", which steered authors towards small pages.
Import steps run page by page, so the 6MB cap applies to each page
(after formatting) while every page adds a hop to a chain shared
with any step it depends on. The two limits push pageSize in
opposite directions, so the new section explains how to size it for
the largest expected environment, keep import rows lean so a bigger
page fits, and fall back to importing less when nothing fits.
Phase 1 now asks for an object-count estimate up front, and
checkpoints decode the raw import errors these limits produce.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: .claude/skills/build-plugin/SKILL.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -62,7 +62,7 @@ Before writing a single file, understand and explore the API. **Use `AskUserQues
62
62
63
63
1.**Find the docs** — Gather URLs or spec files from the user, then fetch and read them.
64
64
2.**Identify the object model** — What are the core entities? (e.g. installations, devices, sites). These become the **indexed objects** in SquaredUp — available for drilldown, search, scoping dashboards, and use as variables.
65
-
3.**Find the list endpoints** — Used to import objects. Prefer fetching **50–250 records per page** across multiple requests — SquaredUp has a per-page timeout but supports as many paged requests as needed.
65
+
3.**Find the list endpoints** — Used to import objects. Prefer fetching **50–250 records per page** across multiple requests — SquaredUp has a per-page timeout. Note the page-size parameter's maximum, and estimate how many objects the largest environment holds (ask the user if unsure): an import run is capped at ~220 pages and ~6MB per page, so tens of thousands of objects constrain `pageSize` — see [index-defs.md](references/index-defs.md#large-imports--choosing-pagesize).
66
66
4.**Find the data endpoints** — These power data streams. For each, record three things; Phase 2 turns the last two into the stream's `timeframes`:
67
67
-**Scoping** — scoped to a single object, multiple objects, or global (no object context).
68
68
-**Time-range control** — does the endpoint accept a queryable time range at all (a `from`/`to`, `start`/`end`, or `period` parameter), or only return a fixed snapshot / current values?
Copy file name to clipboardExpand all lines: .claude/skills/build-plugin/references/checkpoints.md
+6-1Lines changed: 6 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -56,7 +56,12 @@ squaredup objects --matches '{"sourceType":{"type":"equals","value":"<Object Typ
56
56
Notes:
57
57
58
58
-`index` / `index-status` are **folder-independent** — they need only `--datasource-id <id>` (not `--plugin-id`), since the re-index endpoint is keyed purely on the datasource.
59
-
-**Branch on the JSON, not the exit code.**`index-status` exits `0` on every clean read regardless of import state. A failed import is `done: true, succeeded: false` — *not* a non-zero exit. Read the run-level `message` and the per-step `steps[]` (each step has `name`, `status`, `errorReason`, `totalObjectsReceived`, `totalObjectsWritten`) to see **which** step broke and why, fix that import stream, and re-trigger.
59
+
-**Branch on the JSON, not the exit code.**`index-status` exits `0` on every clean read regardless of import state. A failed import is `done: true, succeeded: false` — *not* a non-zero exit. Read the run-level `message` and the per-step `steps[]` (each step has `name`, `status`, `errorReason`, `totalObjectsReceived`, `totalObjectsWritten`) to see **which** step broke and why, fix that import stream, and re-trigger. Three platform errors read as noise but have plugin-side causes:
60
+
-`Payload failed schema validation` with `/vertices/N/<prop>` — a [reserved property name](index-defs.md) (`id`, `state`, `__…`) in `objectMapping.properties`.
61
+
-`Function.ResponseSizeTooLarge` — one page is over ~6MB; lower `pageSize` or trim the row.
62
+
-`'call-chain-length' reached threshold of 220` — too many pages in the run; raise `pageSize`.
63
+
64
+
For the last two, see [Large imports](index-defs.md#large-imports--choosing-pagesize).
60
65
-**`status` is the run's lifecycle, not the outcome.** While running it's `ready`/`inProgress`; once `done` it's one of `succeeded`, `failed`, `warning`, or `cancelled`. `succeeded` and `warning` both report `succeeded: true` (a `warning` run finished but a step emitted warnings — check the `steps[]`); `failed` and `cancelled` report `succeeded: false`. A datasource that has never imported reports `status: "notRun"`, `done: false`.
61
66
-`--since` is **exclusive** (`scheduledStart > since`): always pass the `since` from `index` so `done` can't latch on a stale previous run.
62
67
- If `index` reports `alreadyRunning: true`, it adopted the in-flight run — poll with the `since` it returned. Imports can take several minutes (object import allows up to ~10 min); use a generous overall timeout.
Copy file name to clipboardExpand all lines: .claude/skills/build-plugin/references/index-defs.md
+24Lines changed: 24 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -85,6 +85,30 @@ Building and testing a dependent step needs objects from its dependency to alrea
85
85
86
86
---
87
87
88
+
## Large imports — choosing `pageSize`
89
+
90
+
An import step fetches one page at a time and writes each page's objects to the graph before asking for the next, so a step has no object-count limit of its own. Two limits bound the stream's [`paging.pageSize`](data-streams.md#pagination) from opposite sides instead:
91
+
92
+
| Limit | What it counts |`errorReason` when hit | Pushes `pageSize`|
93
+
| --- | --- | --- | --- |
94
+
|**~6MB per page**| One page, measured after SquaredUp formats it — larger than the raw API body. That page's objects are then written to the graph in a single request. |`Function.ResponseSizeTooLarge`| down |
95
+
|**~220 calls per import run**| Each page is one more call in a chain. A dependent step continues the chain of the step it `dependsOn`. |`'call-chain-length' reached threshold of 220`| up |
96
+
97
+
Each page must also return within the ~28s per-page timeout, another reason not to max out `pageSize`.
98
+
99
+
Size it for the **largest environment the plugin will meet**, not your test tenant — a 500-object test account passes at any page size:
100
+
101
+
1.**Pages** — `objects ÷ pageSize`, plus the pages of every step this one transitively `dependsOn`, must stay comfortably under ~200.
102
+
2.**Bytes** — `pageSize × bytes per row` should stay under ~2MB of raw API response, leaving headroom for formatting. Measure a real page during exploration, e.g. `curl … | wc -c`.
103
+
104
+
A worked example: a CrowdStrike import of ~92,700 hosts overflowed 6MB at 1,000 hosts per page. 250 per page would have needed ~371 calls, so 500 per page with a lean row (~186 calls) was the size that fit both limits.
105
+
106
+
**Keep import rows lean.** An import stream exists to feed `objectMapping` — return only the id, name, type and mapped properties. If the API has a field-selection parameter (`fields`, `select`, `$select`), use it, and leave nested blobs and anything only a dashboard needs to a separate data stream. Smaller rows let a larger `pageSize` fit under 6MB, which is what lets a big inventory fit under the call cap.
107
+
108
+
**If no `pageSize` satisfies both**, import less rather than fighting the limits: filter at the API (e.g. only active or managed hosts), or import a subset and query the rest live through unscoped data streams. Raise it with the user — the call cap is a platform setting, not something a plugin can change.
109
+
110
+
---
111
+
88
112
## Stream-level requirements
89
113
90
114
Import streams are ordinary data stream files in `dataStreams/` — they need `name`, `displayName`, `description`, and `tags` like any other stream. Mark import-only streams as hidden so they don't clutter the tile editor:
Copy file name to clipboardExpand all lines: REVIEW.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -105,6 +105,7 @@ When suggesting changes:
105
105
106
106
- Name source types after how they are referred to in the upstream product or API (e.g. `agent`, `device`). Do not prefix them with the plugin name (e.g. avoid `NinjaOne Device`). A separate friendly display name can be configured if needed (via custom_types.json).
107
107
- objectMapping.properties - Never use `id`, `state`, or a `__`-prefixed name as a property name; they fail the import at graph write. Rename instead, e.g. `{ "status": "state" }`. `rawId`, `sourceId`, `sourceType`, `name` and `label` are silently overwritten, so mapping them does nothing.
108
+
- pageSize - For APIs where a tenant can hold tens of thousands of objects, check the import stream's `pageSize` against both import limits: ~6MB per page and ~220 pages per import run. See the build-plugin skill's index-defs.md, "Large imports".
0 commit comments