Repository navigation
test(TC-1317): read-audit review follow-ups (F1-F4) - #361
Merged
Merged
Conversation
- F1: the stranger's read is denied (401/403) on /signed/kv and /encryption/networks while the read audit is held, not only on /invoke. - F2: export tinycloud_read_audit_stopped_total and the tinycloud_read_audit_drain_stopped gauge; test that a dead drain is counted and reported as a stalled drain. - F3: the slow-batch time includes the SQLite writer-lock wait; say so. - F4: the deadline test checks Retry-After: 1 and stage=read_audit.
Contributor
Author
|
REQUEST_CHANGES: see TC-1317 (review of head |
The route test's log capture depended on test order: tracing-core caches one call-site interest per process, computed from whichever thread's dispatcher reaches the call site first. Assert RequestCancel::stage() in a core test instead; the route test keeps the 503, Retry-After: 1 and the request_timeout body.
Contributor
Author
|
APPROVE: see TC-1317 (re-review of head |
samgbafa
changed the base branch from
skgbafa/tc-1317-read-audit-bound
to
main
October 19, 2026 11:12
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups from the TC-1317 review of #350 (Linear TC-1317 comment cdeb26c7). Stacked on #350; merge after it. No production behaviour changes except two new Prometheus series.
F1. Denials stay denials on every reclassified route.
reclassified_auth_routes_answer_busy_and_denials_stay_denialsnow sends the stranger's kv/get to/signed/kvand to/encryption/networks(audience: the node), not only to/invoke, while the read audit is held. Each must answer 401/403. Mutation check: making either route's_fallback answer503 read_audit_busynow fails the test (/signed/kv: a denial must not turn into busy: 503 …, and the same for/encryption/networks). Before this change the full node lib passed under both mutants.F2. Dead-drain visibility.
tinycloud_read_audit_stopped_total(reads that failed because the drain was gone) and the gaugetinycloud_read_audit_drain_stopped(1 once the drain is gone). Both come from the existingReadAuditHealth.stopped/drain_stoppedfields, updated on each sampler tick inobserve_read_audit.tc1317_a_dead_drain_is_counted_and_reported. On a pipeline whose drain is gone, the watchdog reports a stalled drain on its first look, before any read. Reads answerStoppedand are counted (stopped = 2,busy = 0). The warning is rate-limited and repeats after the interval while the drain stays dead. Mutation check: droppingdrain_stoppedfrom the watchdog's dead-drain condition fails the test.F3. Wording. Since #350, a batch's in-flight time includes the SQLite writer-lock wait. So
Read audit batch slowand itsmax_commit_msmeasure that wait as well as the commit, on SQLite only. The CHANGELOG and theSLOW_BATCH/slow_batchesdoc comments now say so. Field names are unchanged.F4. The deadline is reported against the read audit, and the reply carries
Retry-After.a_deadline_before_the_audit_wait_answers_request_timeoutattaches theRetry-Afterfairing and asserts503,Retry-After: 1and therequest_timeoutbody.tc1317_a_read_waiting_on_its_audit_is_in_the_read_audit_stage. A read runs under aRequestCancelcontext while the drain is held. The test assertscontext.stage() == ReadAuditwhile the read waits, andOtherafter it commits. A deadline that cuts the wait logs that same stage.Otherinenqueuefails the core test.Verification (local, native PG 18, blackhole)
cargo fmt --check.-Dwarnings, workspace-wide and with duckdb for node and core.a_deadline_before_the_audit_wait invoke_deadline --test-threads 810/10 (2 tests per run), androutes::tests --test-threads 643/3 (111 tests per run).read_audit request_cancel --test-threads 64: 10/10. core lib: 492 passed in 3 runs. One earlier run failedwal_reader_progresses_while_audit_waits_for_writer(TC-273, with a 100 ms read timeout) at load average 62.observe_read_auditwithstopped = 3, drain_stopped = trueexportstinycloud_read_audit_stopped_total 3andtinycloud_read_audit_drain_stopped 1.Not included: N5 (re-pin
log-shipper/canonical.jsonafter #147 and #154 land).