Skip to content

History: paging, span, and not sleeping through the budget - #46

Merged
cubehouse merged 2 commits into
mainfrom
feat/history-ergonomics
Sep 23, 2026
Merged

cubehouse merged 2 commits into
mainfrom
feat/history-ergonomics

Conversation

@cubehouse

Copy link
Copy Markdown
Member

Builds on #45, which exposed the three history calls. This adds the loop above
them, and fixes two things in the layer underneath.

What a backfill needs that one call does not give it

const history = tp.entity(DISNEYLAND).history;
const span = await history.span();

for await (const { entityId, row } of history.days({
  from: span.archiveFrom,
  to: span.retrievableThrough,
})) {
  write(entityId, row);
}

span() returns archiveFrom, recordedTo and retrievableThrough in
one shape. The coverage documents do not: a park nests them under summary, an
entity carries them at the top level under different names, so every caller
writes that branch before their first question. retrievableThrough is the end
date to bound a backfill by, because it is what the key may read rather than
what the archive holds, and those differ on every plan below the top one.
Asking past the entitlement is how a long run ends in 403s.

days() pages until the server stops offering a next, follows that URL
verbatim rather than re-deriving it, and yields { entityId, row } as rows
arrive instead of collecting them. A park's daily call is the one paged call in
the family, so before this a park backfill stopped at the first 31 days without
saying anything. changeRows() is the same treatment for changes(). Both
flatten a park envelope and an entity envelope to the same stream, so the
caller writes one loop and does not branch on 'entities' in res.

BudgetExhaustedError carries retryAfterMs, so a run that exhausts the
hourly budget can write down where it got to and come back.

Two fixes underneath

A 429 could park the client for hours. The transport honoured any
Retry-After up to retry.max times. Right for a REST 429, which asks for
seconds; wrong for a history one, because that budget is hourly, so a spent one
can ask for most of an hour and three of those is roughly two and a half hours
of a silent process. RetryConfig gains maxRetryAfterMs (120000 by default):
past it the client does not sleep at all and throws RateLimitError with
retryAfterMs set. A REST 429 is still ridden out unchanged.

EntityHistoryCoverage was missing the park shape. /history/coverage
answers a PARK with HistoryParkCoverageDocument, the same way /history and
/history/daily do. The two shapes do not overlap where it counts: a park
carries summary and fields, an entity carries firstRecordedAt,
lastRecordedAt and kinds. So a user read .kinds off a park's coverage,
got undefined at runtime, and the compiler said nothing.

The fixture that covered it was hand-written in the entity shape and named
after a park, so it agreed with the code for the same reason the code was
wrong, and the live smoke test asserting typeof c.kinds === 'object' for
Magic Kingdom would have failed on the next drift run. Widening the type
immediately produced two compiler errors at exactly those two places. Both
coverage fixtures are now captured from production, and there is one test per
shape.

Also in here: the user agent announced 7.0.0-alpha.0 from a package at
8.0.0, and the schema is regenerated against the current spec, which
supersedes #41.

Verification

examples/backfill.mjs is the whole thing end to end, with checkpoint/resume
and NDJSON or CSV output. Run against production it pulled Disneyland Resort's
entire daily archive, 98,452 rows from 2021-07-03, in one run. Both output
formats and the resume path were exercised.

That output was then compared against the Python SDK's independent
implementation of the same backfill. They matched row for row on every
completed day. The only rows that differed were today's, where
operatingMinutes had grown by the elapsed time between the two runs, which is
what the spec says today's partial counts do.

99 unit tests, 12 of them new, covering paging, park flattening, both coverage
shapes, and the budget with the shipped retry policy rather than a test-only
one. Reverting the retry cap fails two of them. tsc, eslint, prettier and
the build are clean.

🤖 Generated with Claude Code

#45 exposed the three history endpoints. This adds what a backfill needs on
top of them, and fixes two things in the layer underneath.

span() returns archiveFrom, recordedTo and retrievableThrough in one shape.
The coverage documents do not: a park nests them under summary, an entity
carries them at the top level under different names, so every caller writes
that branch before their first question. retrievableThrough is the end date to
bound a backfill by, because it is what the key may read rather than what the
archive holds, and those differ on every plan below the top one.

days() pages until the server stops offering a next, follows that URL verbatim
rather than re-deriving it, and yields { entityId, row } as rows arrive instead
of collecting them. A park's daily call is the one paged call in the family, so
a park backfill previously stopped at the first 31 days without saying so.
changeRows() does the same for changes(). Both flatten a park envelope and an
entity envelope to the same stream, so the caller writes one loop.

BudgetExhaustedError carries retryAfterMs, so a run that hits the hourly budget
can checkpoint and come back.

Two fixes underneath:

- A 429 could park the client for hours. The transport honoured any
  Retry-After up to retry.max times, which is right for a REST 429 asking for
  seconds and wrong for a history one: that budget is hourly, so three waits is
  about two and a half hours of a silent process. RetryConfig gains
  maxRetryAfterMs, 120000 by default; past it the client does not sleep and
  throws RateLimitError with retryAfterMs set.

- EntityHistoryCoverage was missing the park shape. /history/coverage answers a
  PARK with HistoryParkCoverageDocument, exactly as /history and /history/daily
  do. A TypeScript user read .kinds off a park's coverage, got undefined, and
  the compiler said nothing. The fixture covering it was hand-written in the
  entity shape and named after a park, so it agreed with the code for the same
  reason the code was wrong; both coverage fixtures are now captured from
  production, and the live smoke test asserts the shape it actually gets.

Also: the user agent announced 7.0.0-alpha.0 from a package at 8.0.0, and the
schema is regenerated against the current spec, which supersedes #41.

examples/backfill.mjs is the whole thing end to end, with resume and CSV. Run
against production it pulled Disneyland Resort's entire daily archive, 98,452
rows, and matched the Python SDK's output row for row on every completed day;
the only rows that differed were today's, where operatingMinutes had grown by
the elapsed time between the two runs, which is what the spec says it does.

99 unit tests, 12 of them new. tsc, eslint, prettier and the build are clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The announced version was a literal and it had drifted: 7.0.0-alpha.0 in a
package at 8.0.0, so every request the SDK made named a version a major old and
nothing anywhere failed. The Python sibling had the same bug and was two majors
out, which is what made it worth fixing structurally rather than by hand.

A gate test now asserts the User-Agent the server actually receives carries the
version package.json declares. Checking the header rather than the constant
means a correct constant wired up wrongly fails too.

8.1.0 rather than 9.0.0: the only shape that changed, EntityHistoryCoverage
widening to a union, has never been published. 8.0.0 shipped before #45 merged,
so no consumer can be relying on the narrower type.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cubehouse

Copy link
Copy Markdown
Member Author

Second commit: releasing this as 8.1.0, and fixing the version drift structurally.

PACKAGE_VERSION was a literal reading 7.0.0-alpha.0 in a package at 8.0.0, so every request announced a version a major old and nothing failed. The Python sibling had the same bug and was two majors out, which is why this is now a gate test rather than a one-time correction: it asserts the User-Agent the server actually receives carries the version package.json declares. Checking the header rather than the constant means a correct constant wired up wrongly fails too.

8.1.0, not 9.0.0. The only shape that changed is EntityHistoryCoverage widening to a union, and that type has never been published: 8.0.0 shipped before #45 merged. No consumer can be relying on the narrower version, so this is additive.

@cubehouse
cubehouse merged commit 2068594 into main Sep 23, 2026
3 checks passed
@cubehouse
cubehouse deleted the feat/history-ergonomics branch September 23, 2026 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant