Repository navigation
ci(TC-959): read the full node usage inventory in quota restore - #321
Merged
Merged
Conversation
GET /admin/usage lists every space on the node, unpaginated (~107 bytes a row). With ~1,034 production spaces the body is ~110 KiB, above the 64 KiB cap request_json applied to every response, so the restore failed closed with "quota API returned an invalid response" before any PUT. Read /admin/usage through a bounded streaming read capped at 16 MiB (stops at cap+1, fails closed above it); other quota bodies keep the 64 KiB cap. Test fakes now stream like http.client and serve a prod-sized inventory in the node's real row schema.
Contributor
Author
|
REQUEST_CHANGES: see TC-959 |
json.loads(bytes) auto-detects UTF-8-sig, UTF-16 and UTF-32, so a BOM or a wide encoding was accepted. The node sends plain UTF-8 only; decode it strictly and fail closed otherwise.
Contributor
Author
|
APPROVE: see TC-959 (reviewed exact head fc9701d) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Run 39005748999's "Restore storage quota overrides" step failed at its first request with
quota API returned an invalid response, before any PUT.GET /admin/usage(tinycloud-node-server/src/routes/admin.rs::get_usage) returns every row of thespacetable: no pagination, no filter, no query parameters. It returns a JSON object{"spaces":[{"space_id","usage_bytes"}...],"count":n}as compact serde JSON with a Content-Length. A row is about 107 bytes, so the body passes 64 KiB at about 610 spaces. Production had about 1,034 spaces on 2026-10-12.request_jsonread at most 65,537 bytes and rejected anything over 65,536, so prod could never pass this step. The test fakes used inventories of two rows.GET /admin/quota/<space>does returncurrent_usage_bytes, but it echoes any parseable ID and does not show that the space is in thespacetable. Only the inventory proves that each space exists exactly once, so the script still reads it.Change
reconcile-quota.py:read_bounded()streams the body in chunks of at most 64 KiB and stops at cap+1./admin/usagegetsUSAGE_RESPONSE_CAP = 16 MiB, about 150,000 spaces. Every other quota or billing body keeps the 64 KiB cap. A body over the cap, or any body that is not a JSON object, still fails closed. Paths, verified TLS, no-redirect, bounded retries, no body logging, and the exact-target checks are unchanged.reconcile-quota.py(second commit, from review): every response is now decoded as strict UTF-8 beforejson.loads. Until nowjson.loads(bytes)auto-detected the encoding, so a UTF-8 BOM, UTF-16 or UTF-32 body was accepted (this predates the PR). The node sends plain UTF-8 only, so any other encoding now fails closed.test-quota-authority.py: the fake response now streams likehttp.client.HTTPResponse.read(amt). A 3,000-row inventory (~310 KiB) in the node's real row schema drives the full restore in both authority modes. A missing, duplicated or null-usage target inside that inventory still blocks every PUT. A body of exactly 16 MiB is accepted. At 16 MiB + 1 it fails after reading at most cap+1 bytes. Array, truncated, empty, string, HTML, invalid UTF-8, UTF-8 BOM + valid inventory, UTF-16, UTF-16-LE, UTF-32-LE and gzip-compressed bodies all fail closed with no PUT, and the same inventory as plain UTF-8 still passes.test-deploy-recovery.py: the fake streams and serves a 3,000-row inventory.docs/releasing.md: one sentence on the bound.No changes to docker.yml, guards, locks or hashes.
Verification
c601b52script,test-quota-authority.pyhad 6 failures andtest-deploy-recovery.pyhad 10. Each failure came fromquota API returned an invalid response. After the fix both pass: 25/25 and 17/17.7f73e4b4the UTF-8 BOM, UTF-16, UTF-16-LE and UTF-32-LE cases failed (4 failures: the script accepted them). Invalid UTF-8 and gzip already failed closed. After strict decoding, all pass: 25/25.test-docker-tags.py,test-check-deploy-ref.shandreconcile-quota.py --validate-onlypass./admin/usageanswered200 application/jsonwith Content-Length 118,249, no Content-Encoding, no chunking, no BOM, and an object at the top level. Against it, thec601b52script failed with the same message as prod. The fixed script applied both overrides and read them back.http.clientsendsAccept-Encoding: identity, so a gzip-capable proxy would not compress this response.