TC-735: persist KV blobs before the per-space sequence lock

[samgbafa]

Change

Content-addressed KV blobs persist before the per-space sequence lock. A short preparatory transaction checks authorization, hosted-space policy and write preconditions and durably registers one attempt per write. Ordinary uploads hold no pooled DB connection or transaction; a short commit transaction rechecks the fence/attempt, assigns the sequence and commits the reference. kv/sync cursors remain commit-ordered. Protected legacy meeting writes retain a bounded pre-upload freeze barrier (TC-1260); a freeze ACK waits for DB commit or rollback, and post-ACK cleanup only removes unreferenced bytes.

An active (space,hash) fence and per-attempt rows protect overlapping uploads. The last unreferenced attempt moves its fence to deleting; a writer cannot dedup or upload over deleting bytes. DELETE is idempotent, confirmed physical absence refunds the process-local SpaceSizes exactly once, and an old delete cannot hit a later generation. Failed uploads/preconditions/DB commits clean up known unreferenced bytes; crash leftovers are eligible after five minutes of DB time plus the 60-second GC pass when no same-hash writer remains active. Quota tracks physical rather than strictly committed bytes; multi-node shared accounting is a separate TC-1162 concern.

Filesystem memory staging publishes complete files atomically. A crash-leftover stage can be cleaned/refunded independently of a referenced final blob, retaining its recovery row through cleanup. An overlapping live same-hash writer protects its stage. Boot quota reconstruction tracks stage-file charges from the same listing as the total, then refunds a failed retry's reservation and the older stage independently on confirmed removal; known staged-upload failures refund immediately after unlink. Physical deletion and quota settlement continue even if the caller is cancelled. GC drains all expired attempts for one hash under the fence; each interval processes at most four 256-row pages and can reach later unrelated orphans. Attempt created_at is immutable; nullable retry_after in this PR's unshipped _020000 migration schedules retries only. A failed read delays that claim without making it look live. A writer's commit or abort makes deferred claims due at the DB statement time without resetting them to never-retried priority; PostgreSQL commit wakeup is in the existing statement, not another round trip. Open acceptance ruling: continuously overlapping same-hash uploads can defer a crashed stage refund beyond nominal ~6 minutes. TC-735 asks Naomi whether to accept that live-writer exception or require attempt-scoped stage/accounting. No unconditional six-minute claim for that case.

First-generation blobs use the old bare hash key (gen 0). Only a new upload after an orphan generation is deleted uses hash@N. An old 1.20.x binary cannot read an N>0 row after rollback. No known raw-blob production backup was found in Amos's records; provider-side config is unverified. Future bucket-level copy (including TC-950 S3→R2) should GC first or tolerate orphans.

Read boundary

The existing current_kv/kv_write lookup returns generation in the same query, and storage get/range derives the key. Gen-0 reads still use the bare key; GET remains 16 DB round trips without an extra allocation beyond the key string. read_audit.rs, invoke_read_only, authorization and audit are unchanged by this write fix. Separately reviewed #302 batches authorization snapshot reads and is required for this PR.

Verification at 2421150 (Sam Gbafa)

• Full local serialized PostgreSQL 16.15 + MySQL 8.0.46 + MinIO workspace suite: 1,152 passed, 9 ignored (CI unset only for runner-iptables-only cases); the isolated test bucket was created before the passing run. postgres_same_space_commits_are_seq_ordered, conditional-put concurrency, real MinIO storage/GC and 30 core orphan tests passed. Persistent oldest read failures (256), four live-blocked pages (1,024) and 256 failing DELETE fences each failed before and passed after bounded cross-tick queue selection; logical 0/60/120-second retry eligibility is controlled through DB timestamps. The further writer-departure regressions failed before and passed after waking at DB time instead of NULL: SQLite repeatedly replaces a live writer across 1,024 old claims; PostgreSQL commits a real reference with 256 deferred claims; MySQL's failure cleanup and independent orphan progress pass with its production SQL path. Filesystem stage + failed retry quota and immediate failed-upload refunds still pass; a contended size lock/cancel test was red 19/green 0 for a lost refund, and another was red with underflow/green 0 for an uncharged reservation. All 15 filesystem storage tests pass. The earlier freeze module 20×14 repetitions (~/scratch/tc-735/freeze-20-retryage.log) are unchanged by this collector-only follow-up. Strict all-target Clippy and formatting pass.
• Wire budget: sign-in 69, KV PUT 42, GET 16 (PG16; earlier exact output artifact://1190; this follow-up changes only collector/wakeup SQL and tests, without adding a PUT round trip). Against today's 37-round-trip PUT, +5 is ~0.33s at ~65ms RTT when Batch authorization graph snapshot in one SQL round trip (TC-735 dependency) #302 is present; GET and sign-in each save six trips.
• Isolated PostgreSQL 18.6 upgrade proof at parent 4cc4a4e (migration code unchanged in 2421150), upgraded 1.20.6 schema: db: page account session discovery #296 _000000 alone → TC-737 _010000 → TC-735 _020000, one migration per step; populated pre-current_kv BIGINT-generation backfill also passed. Synthetic integration 6698935 is not a TC-735: persist KV blobs before the per-space sequence lock #301 worktree edit. Fixture N>0 rows: kv_write=0, current_kv=0. Evidence: ~/scratch/tc-735/pg18-proof/{joint-upgrade-4cc4a4e,populated-old-kv-pg18-4cc4a4e}.log. On separate 100k-row synthetic fixtures, PG16 and PG18 use index scans (no temp spill) for both never-retried and due-retry GC selection (orphan-fair-index-plan-{pg16,pg18}.log). The second merger must rebase/rename its migration to sort after everything on main and repeat PG18 proof on the actual release head.

Earlier PG18.6+MinIO same-space testing measured ~70 ops/s against ~93/180 targets, not below comparable main steady state; the epoch transaction bottleneck is TC-1218. This revision has not been retimed. A matched warmed main+#302 versus candidate 8/32 same-space no-regression run needs fresh exact-head reviews and Naomi's new ≤15-minute exclusive-window approval. No performance-target acceptance claimed.

Gates

#302 merges/ships before #301; otherwise PUT costs 51, not the accepted 42 round trips. Release sequence on Sam's word: reviewed #292 revert → #296/1.20.8 → #295/1.20.9 → TC-1071/1.20.10 → TC-737 → #302 → TC-735. #291 also merges before #301: its test-only whole-second M1 minimum-TTL fixture correction is separate. Exact 2421150 CI passed workspace PostgreSQL-backed tests, M1 real-data, workspace build/Clippy/format and image build (runs 38855270124 and 38855270160); Astra's exact-head correctness rereview approved. Miller's independent security verdict, Naomi's overlapping-live-writer quota-bound ruling and approved final timing remain pending. TC-1176 holds node releases. No merge or deployment requested.

[samgbafa]

Correction on current head 5f34b4d: the PostgreSQL-backed workspace CI timeout is a TC-735 write-path freeze-ack regression, not a proven TC-1176 read-audit issue. legacy_freeze_ack_waits_for_actual_in_flight_kv_storage_and_commit (SQLite) timed out because a freeze acknowledged while its test upload was paused, the assertion panicked before releasing that upload, and teardown hung. Origin/main passed the single test and freeze module 20/20 each; a new PG16 delayed-upload RED test observed the early acknowledgement directly. A local, not-yet-pushed fix acquires the durable freeze row before upload, keeps sequence locking after upload, bounds freeze attempts to 30s, and rejects post-freeze writes before uploading. Added failed-upload refund/GC regression. Focused PG16 freeze module 13/13 and 20/20 repeated runs passed. Full PG-backed workspace suite and fresh CI/review are still required before this PR is ready; testing pauses for the TC-1149 20:35–21:05Z exclusive host window. No read_audit or nonmutating KV read changes, no merge/deploy. TC-735 root cause. The ~93/180 same-space throughput targets remain unmet (separate pre-existing epoch-head bottleneck TC-1218).

[samgbafa]

REQUEST_CHANGES: see TC-735

[samgbafa]

REQUEST_CHANGES — TC-735

[samgbafa]

REQUEST_CHANGES: see TC-735

[samgbafa]

REQUEST_CHANGES — TC-735

[samgbafa]

REQUEST_CHANGES: see TC-735

[samgbafa]

REQUEST_CHANGES — TC-735

[samgbafa]

REQUEST_CHANGES: see TC-735

[samgbafa]

REQUEST_CHANGES — TC-735

[samgbafa]

REQUEST_CHANGES: see TC-735

[samgbafa]

REQUEST_CHANGES — TC-735

[samgbafa]

APPROVE: see TC-735

[samgbafa]

REQUEST_CHANGES: see TC-735

[samgbafa]

REQUEST_CHANGES — TC-735

[samgbafa]

APPROVE: see TC-735

[samgbafa]

REQUEST_CHANGES — TC-735