Skip to content

ci(TC-959): read the full node usage inventory in quota restore - #321

Merged
samgbafa merged 2 commits into
mainfrom
skgbafa/tc-959-quota-usage-cap
Oct 17, 2026
Merged

samgbafa merged 2 commits into
mainfrom
skgbafa/tc-959-quota-usage-cap

Conversation

@samgbafa

@samgbafa samgbafa commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Run 39005748999's "Restore storage quota overrides" step failed at its first request with quota API returned an invalid response, before any PUT.

GET /admin/usage (tinycloud-node-server/src/routes/admin.rs::get_usage) returns every row of the space table: no pagination, no filter, no query parameters. It returns a JSON object {"spaces":[{"space_id","usage_bytes"}...],"count":n} as compact serde JSON with a Content-Length. A row is about 107 bytes, so the body passes 64 KiB at about 610 spaces. Production had about 1,034 spaces on 2026-10-12. request_json read at most 65,537 bytes and rejected anything over 65,536, so prod could never pass this step. The test fakes used inventories of two rows.

GET /admin/quota/<space> does return current_usage_bytes, but it echoes any parseable ID and does not show that the space is in the space table. Only the inventory proves that each space exists exactly once, so the script still reads it.

Change

  • reconcile-quota.py: read_bounded() streams the body in chunks of at most 64 KiB and stops at cap+1. /admin/usage gets USAGE_RESPONSE_CAP = 16 MiB, about 150,000 spaces. Every other quota or billing body keeps the 64 KiB cap. A body over the cap, or any body that is not a JSON object, still fails closed. Paths, verified TLS, no-redirect, bounded retries, no body logging, and the exact-target checks are unchanged.
  • reconcile-quota.py (second commit, from review): every response is now decoded as strict UTF-8 before json.loads. Until now json.loads(bytes) auto-detected the encoding, so a UTF-8 BOM, UTF-16 or UTF-32 body was accepted (this predates the PR). The node sends plain UTF-8 only, so any other encoding now fails closed.
  • test-quota-authority.py: the fake response now streams like http.client.HTTPResponse.read(amt). A 3,000-row inventory (~310 KiB) in the node's real row schema drives the full restore in both authority modes. A missing, duplicated or null-usage target inside that inventory still blocks every PUT. A body of exactly 16 MiB is accepted. At 16 MiB + 1 it fails after reading at most cap+1 bytes. Array, truncated, empty, string, HTML, invalid UTF-8, UTF-8 BOM + valid inventory, UTF-16, UTF-16-LE, UTF-32-LE and gzip-compressed bodies all fail closed with no PUT, and the same inventory as plain UTF-8 still passes.
  • test-deploy-recovery.py: the fake streams and serves a 3,000-row inventory.
  • docs/releasing.md: one sentence on the bound.

No changes to docker.yml, guards, locks or hashes.

Verification

  • With the new tests and the unchanged c601b52 script, test-quota-authority.py had 6 failures and test-deploy-recovery.py had 10. Each failure came from quota API returned an invalid response. After the fix both pass: 25/25 and 17/17.
  • Encoding cases: on 7f73e4b4 the UTF-8 BOM, UTF-16, UTF-16-LE and UTF-32-LE cases failed (4 failures: the script accepted them). Invalid UTF-8 and gzip already failed closed. After strict decoding, all pass: 25/25.
  • test-docker-tags.py, test-check-deploy-ref.sh and reconcile-quota.py --validate-only pass.
  • I also ran the real node binary locally with 1,100 seeded spaces. Its /admin/usage answered 200 application/json with Content-Length 118,249, no Content-Encoding, no chunking, no BOM, and an object at the top level. Against it, the c601b52 script failed with the same message as prod. The fixed script applied both overrides and read them back. http.client sends Accept-Encoding: identity, so a gzip-capable proxy would not compress this response.

GET /admin/usage lists every space on the node, unpaginated (~107 bytes a
row). With ~1,034 production spaces the body is ~110 KiB, above the 64 KiB
cap request_json applied to every response, so the restore failed closed
with "quota API returned an invalid response" before any PUT.

Read /admin/usage through a bounded streaming read capped at 16 MiB (stops
at cap+1, fails closed above it); other quota bodies keep the 64 KiB cap.
Test fakes now stream like http.client and serve a prod-sized inventory in
the node's real row schema.
@samgbafa

Copy link
Copy Markdown
Contributor Author

REQUEST_CHANGES: see TC-959

json.loads(bytes) auto-detects UTF-8-sig, UTF-16 and UTF-32, so a BOM or a
wide encoding was accepted. The node sends plain UTF-8 only; decode it
strictly and fail closed otherwise.
@samgbafa

Copy link
Copy Markdown
Contributor Author

APPROVE: see TC-959 (reviewed exact head fc9701d)

@samgbafa
samgbafa merged commit 6d9d3b4 into main Oct 17, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant