Skip to content

Commit f2e00a2

Browse files
clarkdclaude
andcommitted
Size import pageSize against both import limits
CoPA's CrowdStrike import (~92,700 hosts) hit two limits the skill never mentioned. At 1,000 hosts per page one page overflowed the ~6MB Lambda response cap, and dropping to 250 per page would have needed ~371 calls against a ~220 call-chain cap per import run. The skill also claimed SquaredUp "supports as many paged requests as needed", which steered authors towards small pages. Import steps run page by page, so the 6MB cap applies to each page (after formatting) while every page adds a hop to a chain shared with any step it depends on. The two limits push pageSize in opposite directions, so the new section explains how to size it for the largest expected environment, keep import rows lean so a bigger page fits, and fall back to importing less when nothing fits. Phase 1 now asks for an object-count estimate up front, and checkpoints decode the raw import errors these limits produce. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 39e2a37 commit f2e00a2

4 files changed

Lines changed: 32 additions & 2 deletions

File tree

‎.claude/skills/build-plugin/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ Before writing a single file, understand and explore the API. **Use `AskUserQues
6262

6363
1. **Find the docs** — Gather URLs or spec files from the user, then fetch and read them.
6464
2. **Identify the object model** — What are the core entities? (e.g. installations, devices, sites). These become the **indexed objects** in SquaredUp — available for drilldown, search, scoping dashboards, and use as variables.
65-
3. **Find the list endpoints** — Used to import objects. Prefer fetching **50–250 records per page** across multiple requests — SquaredUp has a per-page timeout but supports as many paged requests as needed.
65+
3. **Find the list endpoints** — Used to import objects. Prefer fetching **50–250 records per page** across multiple requests — SquaredUp has a per-page timeout. Note the page-size parameter's maximum, and estimate how many objects the largest environment holds (ask the user if unsure): an import run is capped at ~220 pages and ~6MB per page, so tens of thousands of objects constrain `pageSize` — see [index-defs.md](references/index-defs.md#large-imports--choosing-pagesize).
6666
4. **Find the data endpoints** — These power data streams. For each, record three things; Phase 2 turns the last two into the stream's `timeframes`:
6767
- **Scoping** — scoped to a single object, multiple objects, or global (no object context).
6868
- **Time-range control** — does the endpoint accept a queryable time range at all (a `from`/`to`, `start`/`end`, or `period` parameter), or only return a fixed snapshot / current values?

‎.claude/skills/build-plugin/references/checkpoints.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,12 @@ squaredup objects --matches '{"sourceType":{"type":"equals","value":"<Object Typ
5656
Notes:
5757

5858
- `index` / `index-status` are **folder-independent** — they need only `--datasource-id <id>` (not `--plugin-id`), since the re-index endpoint is keyed purely on the datasource.
59-
- **Branch on the JSON, not the exit code.** `index-status` exits `0` on every clean read regardless of import state. A failed import is `done: true, succeeded: false` — *not* a non-zero exit. Read the run-level `message` and the per-step `steps[]` (each step has `name`, `status`, `errorReason`, `totalObjectsReceived`, `totalObjectsWritten`) to see **which** step broke and why, fix that import stream, and re-trigger.
59+
- **Branch on the JSON, not the exit code.** `index-status` exits `0` on every clean read regardless of import state. A failed import is `done: true, succeeded: false` — *not* a non-zero exit. Read the run-level `message` and the per-step `steps[]` (each step has `name`, `status`, `errorReason`, `totalObjectsReceived`, `totalObjectsWritten`) to see **which** step broke and why, fix that import stream, and re-trigger. Three platform errors read as noise but have plugin-side causes:
60+
- `Payload failed schema validation` with `/vertices/N/<prop>` — a [reserved property name](index-defs.md) (`id`, `state`, `__…`) in `objectMapping.properties`.
61+
- `Function.ResponseSizeTooLarge` — one page is over ~6MB; lower `pageSize` or trim the row.
62+
- `'call-chain-length' reached threshold of 220` — too many pages in the run; raise `pageSize`.
63+
64+
For the last two, see [Large imports](index-defs.md#large-imports--choosing-pagesize).
6065
- **`status` is the run's lifecycle, not the outcome.** While running it's `ready`/`inProgress`; once `done` it's one of `succeeded`, `failed`, `warning`, or `cancelled`. `succeeded` and `warning` both report `succeeded: true` (a `warning` run finished but a step emitted warnings — check the `steps[]`); `failed` and `cancelled` report `succeeded: false`. A datasource that has never imported reports `status: "notRun"`, `done: false`.
6166
- `--since` is **exclusive** (`scheduledStart > since`): always pass the `since` from `index` so `done` can't latch on a stale previous run.
6267
- If `index` reports `alreadyRunning: true`, it adopted the in-flight run — poll with the `since` it returned. Imports can take several minutes (object import allows up to ~10 min); use a generous overall timeout.

‎.claude/skills/build-plugin/references/index-defs.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,30 @@ Building and testing a dependent step needs objects from its dependency to alrea
8585

8686
---
8787

88+
## Large imports — choosing `pageSize`
89+
90+
An import step fetches one page at a time and writes each page's objects to the graph before asking for the next, so a step has no object-count limit of its own. Two limits bound the stream's [`paging.pageSize`](data-streams.md#pagination) from opposite sides instead:
91+
92+
| Limit | What it counts | `errorReason` when hit | Pushes `pageSize` |
93+
| --- | --- | --- | --- |
94+
| **~6MB per page** | One page, measured after SquaredUp formats it — larger than the raw API body. That page's objects are then written to the graph in a single request. | `Function.ResponseSizeTooLarge` | down |
95+
| **~220 calls per import run** | Each page is one more call in a chain. A dependent step continues the chain of the step it `dependsOn`. | `'call-chain-length' reached threshold of 220` | up |
96+
97+
Each page must also return within the ~28s per-page timeout, another reason not to max out `pageSize`.
98+
99+
Size it for the **largest environment the plugin will meet**, not your test tenant — a 500-object test account passes at any page size:
100+
101+
1. **Pages** — `objects ÷ pageSize`, plus the pages of every step this one transitively `dependsOn`, must stay comfortably under ~200.
102+
2. **Bytes** — `pageSize × bytes per row` should stay under ~2MB of raw API response, leaving headroom for formatting. Measure a real page during exploration, e.g. `curl … | wc -c`.
103+
104+
A worked example: a CrowdStrike import of ~92,700 hosts overflowed 6MB at 1,000 hosts per page. 250 per page would have needed ~371 calls, so 500 per page with a lean row (~186 calls) was the size that fit both limits.
105+
106+
**Keep import rows lean.** An import stream exists to feed `objectMapping` — return only the id, name, type and mapped properties. If the API has a field-selection parameter (`fields`, `select`, `$select`), use it, and leave nested blobs and anything only a dashboard needs to a separate data stream. Smaller rows let a larger `pageSize` fit under 6MB, which is what lets a big inventory fit under the call cap.
107+
108+
**If no `pageSize` satisfies both**, import less rather than fighting the limits: filter at the API (e.g. only active or managed hosts), or import a subset and query the rest live through unscoped data streams. Raise it with the user — the call cap is a platform setting, not something a plugin can change.
109+
110+
---
111+
88112
## Stream-level requirements
89113

90114
Import streams are ordinary data stream files in `dataStreams/` — they need `name`, `displayName`, `description`, and `tags` like any other stream. Mark import-only streams as hidden so they don't clutter the tile editor:

‎REVIEW.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@ When suggesting changes:
105105

106106
- Name source types after how they are referred to in the upstream product or API (e.g. `agent`, `device`). Do not prefix them with the plugin name (e.g. avoid `NinjaOne Device`). A separate friendly display name can be configured if needed (via custom_types.json).
107107
- objectMapping.properties - Never use `id`, `state`, or a `__`-prefixed name as a property name; they fail the import at graph write. Rename instead, e.g. `{ "status": "state" }`. `rawId`, `sourceId`, `sourceType`, `name` and `label` are silently overwritten, so mapping them does nothing.
108+
- pageSize - For APIs where a tenant can hold tens of thousands of objects, check the import stream's `pageSize` against both import limits: ~6MB per page and ~220 pages per import run. See the build-plugin skill's index-defs.md, "Large imports".
108109

109110
### Documentation - (docs/README.md)
110111

0 commit comments

Comments
 (0)