History: paging, span, and not sleeping through the budget - #46
Conversation
#45 exposed the three history endpoints. This adds what a backfill needs on top of them, and fixes two things in the layer underneath. span() returns archiveFrom, recordedTo and retrievableThrough in one shape. The coverage documents do not: a park nests them under summary, an entity carries them at the top level under different names, so every caller writes that branch before their first question. retrievableThrough is the end date to bound a backfill by, because it is what the key may read rather than what the archive holds, and those differ on every plan below the top one. days() pages until the server stops offering a next, follows that URL verbatim rather than re-deriving it, and yields { entityId, row } as rows arrive instead of collecting them. A park's daily call is the one paged call in the family, so a park backfill previously stopped at the first 31 days without saying so. changeRows() does the same for changes(). Both flatten a park envelope and an entity envelope to the same stream, so the caller writes one loop. BudgetExhaustedError carries retryAfterMs, so a run that hits the hourly budget can checkpoint and come back. Two fixes underneath: - A 429 could park the client for hours. The transport honoured any Retry-After up to retry.max times, which is right for a REST 429 asking for seconds and wrong for a history one: that budget is hourly, so three waits is about two and a half hours of a silent process. RetryConfig gains maxRetryAfterMs, 120000 by default; past it the client does not sleep and throws RateLimitError with retryAfterMs set. - EntityHistoryCoverage was missing the park shape. /history/coverage answers a PARK with HistoryParkCoverageDocument, exactly as /history and /history/daily do. A TypeScript user read .kinds off a park's coverage, got undefined, and the compiler said nothing. The fixture covering it was hand-written in the entity shape and named after a park, so it agreed with the code for the same reason the code was wrong; both coverage fixtures are now captured from production, and the live smoke test asserts the shape it actually gets. Also: the user agent announced 7.0.0-alpha.0 from a package at 8.0.0, and the schema is regenerated against the current spec, which supersedes #41. examples/backfill.mjs is the whole thing end to end, with resume and CSV. Run against production it pulled Disneyland Resort's entire daily archive, 98,452 rows, and matched the Python SDK's output row for row on every completed day; the only rows that differed were today's, where operatingMinutes had grown by the elapsed time between the two runs, which is what the spec says it does. 99 unit tests, 12 of them new. tsc, eslint, prettier and the build are clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The announced version was a literal and it had drifted: 7.0.0-alpha.0 in a package at 8.0.0, so every request the SDK made named a version a major old and nothing anywhere failed. The Python sibling had the same bug and was two majors out, which is what made it worth fixing structurally rather than by hand. A gate test now asserts the User-Agent the server actually receives carries the version package.json declares. Checking the header rather than the constant means a correct constant wired up wrongly fails too. 8.1.0 rather than 9.0.0: the only shape that changed, EntityHistoryCoverage widening to a union, has never been published. 8.0.0 shipped before #45 merged, so no consumer can be relying on the narrower type. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Second commit: releasing this as 8.1.0, and fixing the version drift structurally.
8.1.0, not 9.0.0. The only shape that changed is |
Builds on #45, which exposed the three history calls. This adds the loop above
them, and fixes two things in the layer underneath.
What a backfill needs that one call does not give it
span()returnsarchiveFrom,recordedToandretrievableThroughinone shape. The coverage documents do not: a park nests them under
summary, anentity carries them at the top level under different names, so every caller
writes that branch before their first question.
retrievableThroughis the enddate to bound a backfill by, because it is what the key may read rather than
what the archive holds, and those differ on every plan below the top one.
Asking past the entitlement is how a long run ends in 403s.
days()pages until the server stops offering anext, follows that URLverbatim rather than re-deriving it, and yields
{ entityId, row }as rowsarrive instead of collecting them. A park's daily call is the one paged call in
the family, so before this a park backfill stopped at the first 31 days without
saying anything.
changeRows()is the same treatment forchanges(). Bothflatten a park envelope and an entity envelope to the same stream, so the
caller writes one loop and does not branch on
'entities' in res.BudgetExhaustedErrorcarriesretryAfterMs, so a run that exhausts thehourly budget can write down where it got to and come back.
Two fixes underneath
A 429 could park the client for hours. The transport honoured any
Retry-Afterup toretry.maxtimes. Right for a REST 429, which asks forseconds; wrong for a history one, because that budget is hourly, so a spent one
can ask for most of an hour and three of those is roughly two and a half hours
of a silent process.
RetryConfiggainsmaxRetryAfterMs(120000 by default):past it the client does not sleep at all and throws
RateLimitErrorwithretryAfterMsset. A REST 429 is still ridden out unchanged.EntityHistoryCoveragewas missing the park shape./history/coverageanswers a PARK with
HistoryParkCoverageDocument, the same way/historyand/history/dailydo. The two shapes do not overlap where it counts: a parkcarries
summaryandfields, an entity carriesfirstRecordedAt,lastRecordedAtandkinds. So a user read.kindsoff a park's coverage,got
undefinedat runtime, and the compiler said nothing.The fixture that covered it was hand-written in the entity shape and named
after a park, so it agreed with the code for the same reason the code was
wrong, and the live smoke test asserting
typeof c.kinds === 'object'forMagic Kingdom would have failed on the next drift run. Widening the type
immediately produced two compiler errors at exactly those two places. Both
coverage fixtures are now captured from production, and there is one test per
shape.
Also in here: the user agent announced
7.0.0-alpha.0from a package at8.0.0, and the schema is regenerated against the current spec, whichsupersedes #41.
Verification
examples/backfill.mjsis the whole thing end to end, with checkpoint/resumeand NDJSON or CSV output. Run against production it pulled Disneyland Resort's
entire daily archive, 98,452 rows from 2021-07-03, in one run. Both output
formats and the resume path were exercised.
That output was then compared against the Python SDK's independent
implementation of the same backfill. They matched row for row on every
completed day. The only rows that differed were today's, where
operatingMinuteshad grown by the elapsed time between the two runs, which iswhat the spec says today's partial counts do.
99 unit tests, 12 of them new, covering paging, park flattening, both coverage
shapes, and the budget with the shipped retry policy rather than a test-only
one. Reverting the retry cap fails two of them.
tsc,eslint,prettierandthe build are clean.
🤖 Generated with Claude Code