TC-735: persist KV blobs before the per-space sequence lock

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 ## [Unreleased]
 
+- KV puts persist content-addressed blobs before taking the per-space sequence lock, while committed `kv/sync` cursors remain ordered. A short preparatory transaction registers one durable attempt per write; ordinary uploads hold no pooled DB connection, unlike the previous write path. Protected legacy meeting writes remain the documented exception: their pre-upload freeze barrier waits for commit or abort and fails after 30 seconds if stalled (TC-1260). Failed attempts transition their last unreferenced hash fence to `deleting`; a new writer cannot deduplicate or upload until confirmed physical absence removes that fence. Known failures refund once after deletion; a 60-second collector reclaims crash leftovers after a five-minute DB-time grace when no same-hash writer remains active, including lost DELETE responses. S3 startup reconstructs pending refunds from the same physical listing as its quota count. Quota remains conservative and per process, not a durable global ledger (TC-735).
+  Lost connections during candidate registration retry with the same attempt ID. Neither an upload nor a committed write is replayed on an uncertain outcome. Filesystem-staged S3 bodies now retain their temporary file through send and retry. First uploads keep the bare hash key (generation zero); reuploads after orphan deletion use a new generation-scoped key, so a delayed old delete cannot remove newly committed bytes. Reads obtain the generation in the existing KV lookup. Rolling back to an older binary cannot read generation-above-zero records.
+  Physical operations on the same `(space, hash, generation)` now serialize in-process from existence check through upload or deletion and quota settlement; overlapping writers cannot double-charge, and stale S3 HEADs cannot re-create a refunded charge. Filesystem memory staging writes a same-directory temporary file before atomically publishing the final hash key. GC reclaims a crash-leftover stage and its quota charge even if a later same-hash retry committed a reference to the final blob; failed stage cleanup retains the candidate for retry without deleting referenced content. Expired same-hash attempts drain as a group under their fence; a live sibling temporarily defers cleanup without hiding unrelated orphans. Registration age stays immutable while separate retry eligibility delays blocked claims, so a deferred crash claim cannot masquerade as a live uploader across collectors. Each collector tick has a bounded page count. SQLite reserves the database writer before preparatory authorization reads, and populated older PostgreSQL KV backfills decode generation as BIGINT.
+  Boot quota reconstruction tracks stage-file charges from the same directory listing as the total; a failed retry and an older crash-leftover stage each refund their own charge after confirmed removal. Known filesystem-stage failures refund their reservation immediately after the unpublished file is removed, and cancellation of a delete cannot interrupt its physical unlink and quota settlement.
+  Filesystem reservation and refund update pending charges atomically with `SpaceSizes` across cancellation. Each bounded collector page fairly admits never-retried and due-retry attempts through indexed queries, while a full deleting-fence page reserves capacity for expired attempts; persistent storage failures or live same-hash writers cannot starve unrelated orphans across ticks. When a writer departs, deferred siblings become due using DB time without regaining their original first-attempt priority, so repeated departures also cannot starve unrelated orphans.
+  PostgreSQL selection skips independently held hash fences before its page limit and releases selection locks before storage I/O. A live same-space, same-hash attempt can defer an older crashed stage's physical-byte refund beyond the usual five-minute grace plus 60-second pass. Its upload has a 240-second deadline; the stage becomes due for the next scheduled pass (within 60 seconds) after the last overlapping attempt commits or aborts; a backlog beyond the collector's bounded pages can require further passes. Repeated overlaps extend that delay without deleting committed content or undercounting bytes. The existing epoch-head anti-join still executes inside the per-space sequence-held transaction; its query and indexing are TC-1274, not this change.
+- Load each invocation's authorization graph in one SQL round trip on PostgreSQL, SQLite and MySQL rather than four. Revocations in that statement resolve through the persisted CID-alias index exactly like `revocation::effective_rows`, so a delegation revoked under another accepted encoding is still refused (TC-1071). Bound metadata lookups to 65 nodes for indexed access on large tables while the unchanged 64-node authorization limit still denies oversized chains. Sort abilities by `(resource, ability)` on every backend; authorization decisions are unchanged, but the diagnostic reason when multiple caveats all reject may normalize. No schema change (TC-735).
 - New migration `m20261017_000000_epoch_order_space_index` adds `idx_epoch_order_space_parent_child ON epoch_order (space, parent, child)` for the head-epoch anti-join that every write runs under its space lock (TC-1274). On a fresh database the build is instant. On PostgreSQL, if a valid index of that name and shape already exists, the migration issues no DDL and takes no lock on `epoch_order`. If the index of that name is INVALID or has a different shape, the boot fails, and nothing is applied, with an error naming the fix. On an existing database **without** the index, the migration builds it inside the boot's migration transaction. That holds a SHARE lock that blocks writes for the whole build, and the lock wait is bounded by the 30 s `lock_timeout` (TC-1176). **Runbook (PostgreSQL, before upgrading a node with history):** in autocommit, run `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_epoch_order_space_parent_child ON epoch_order (space, parent, child);`. Then confirm `SELECT indisvalid FROM pg_index WHERE indexrelid = 'idx_epoch_order_space_parent_child'::regclass;` returns `t`. If it returns `f`, run `DROP INDEX CONCURRENTLY idx_epoch_order_space_parent_child;` and build it again. `down` keeps the index. MySQL is covered only by the generated statement, because the full MySQL migration chain does not run (TC-1314).
 - Request events include validated/echoed UUID correlation, typed replacement flags, node-keyed target account/space pseudonyms and per-request database acquisition/execution timings and attempt counts. Driver/actor instrumentation includes failures and request-awaited audit batches; forwarding requires explicit first-party origin classification. A shared core `Pseudonymizer` preserves existing account/space hashes, adds the service/space-scoped database hash primitive for F6, and is derived once at startup for injection into consumers. See `docs/request-telemetry.md` for the wire contract, key custody, restart joinability and coverage/exclusions (TC-959 N2).
 - CACAO/SIWE delegation ingestion normalizes ECDSA recovery IDs 0/1 to 27/28 before hashing and persisting the canonical DAG-CBOR bytes (TC-1071). Equivalent low-s signatures now share one delegation CID, including the CID used for revocation lookup. High-s signatures are rejected during decoding, matching the existing EIP-191 verifier and EIP-2 low-s convention; UCAN/JWS bytes remain unchanged. Existing stored delegation IDs are not rewritten.

docs/kv-sync.md

@@ -129,3 +129,67 @@ cursor.
 
 Read audit records every poll. Poll on demand, or no more often than every
 60 seconds.
+
+## Write ordering and blob recovery
+
+KV puts stage and authorize content before upload. A short preparatory
+transaction checks authorization and hosting, then commits an active
+`(space, content hash, generation)` fence and a separate claim for each
+writer. The first generation is zero and retains the original bare-hash
+storage key; a generation above zero is allocated only after a prior
+generation's physical deletion is confirmed. Ordinary writes release every
+pooled connection and database lock before S3 or filesystem I/O. A
+four-minute upload deadline stays below the five-minute DB-clock GC grace.
+After upload, an ordinary writer takes the per-space sequence guard, starts
+a fresh commit transaction and takes
+the DB sequence lock; it verifies its own active claim, then rechecks
+authorization and conditional-write preconditions. The guards stay through
+commit, so `seq` and sync cursors remain commit-ordered even when same-space
+uploads finish out of order. An unauthorized write uploads nothing.
+
+For a single node process, each `(space, hash, generation)` serializes its
+physical existence check, upload or deletion, and quota settlement without
+holding a database connection. Filesystem memory staging writes into a
+same-directory temporary file and atomically renames only complete bytes to
+the hash key. A process crash can leave an incomplete stage; startup counts
+its physical bytes, and the hash candidate's GC pass removes and refunds it.
+SQLite preparatory writes reserve the database writer before authorization
+reads, so a separate collector connection cannot invalidate the read snapshot.
+
+Protected legacy meeting paths are the TC-1260 exception: the durable freeze
+row and its pooled transaction remain locked across upload until commit or
+abort. A freeze acknowledgement drains earlier uploads; a later protected
+write is refused before upload. A stalled writer causes the freeze attempt
+to fail after 30 seconds rather than acknowledge an incomplete drain.
+
+Before upload, a lost pooled connection retries the preparatory transaction
+with the same attempt ID. The registration is idempotent; neither an upload
+nor a committed write is replayed on an uncertain outcome. A missing claim
+or a fence already deleting returns a retryable error, never a reference to
+a potentially removed object.
+
+A failed upload, precondition or database write removes only its own claim.
+If another claim or a current or historical KV reference exists, it leaves
+the shared generation's blob alone. Otherwise it commits a `deleting` fence
+and byte entitlement, releases the DB connection, then deletes that
+generation's object. A writer seeing `deleting` cannot deduplicate or
+upload over it. Only confirmed physical absence refunds the per-process
+quota exactly once; only then is the fence retired and the next generation
+made available. Each delete targets its original generation's storage key,
+so even a delayed second delete cannot remove a later committed generation.
+A lost DELETE response or a crash mid-delete leaves the fence for
+idempotent GC retry. Reads get the generation from their existing KV lookup,
+without another database query; generation zero reads the bare key. A
+rolled-back binary cannot read records with generation above zero, which
+can occur after a failed upload is collected and its content is saved again.
+
+The collector runs every 60 seconds, aging attempts after five minutes by
+DB time and draining eligible work in batches of 256. On S3 startup the
+same object-listing snapshot reconstructs both `SpaceSizes` and which
+`deleting` fences counted physical bytes; a deletion before the listing
+needs no refund, while a deletion afterward refunds once. Orphans present
+at restart count until GC. Quota is conservative physical accounting,
+**not commit-only accounting**: pending uploads and crash leftovers can
+temporarily over-count. `SpaceSizes` remains in-memory and per process; other
+nodes' writes and deletes do not synchronize it. Globally exact multi-node
+accounting requires a separate shared byte ledger (TC-1162).

docs/meeting-publication-migration.md

@@ -48,7 +48,12 @@ An existing pause continues to block legacy writes under `xyz.tinycloud.tinychat
 
 Reads, chat keys, cursors, credentials, and other spaces retain their existing behavior. Frozen ordinary KV put/delete returns HTTP 409. The core guard also rejects internal legacy cleanup. The pause survives a node restart; only an exact generation-checked release removes it.
 
-The durable guard remains locked by protected KV mutation transactions through storage persistence and commit. PostgreSQL row locking and SQLite writer serialization retain their existing behavior. Failed guard reads fail closed. Existing snapshot keys remain protected after the legacy pause is released.
+The durable guard is acquired before a protected KV upload starts and stays locked through storage persistence and commit or abort. A freeze acknowledgement therefore drains writes already uploading; a later writer is refused before uploading. If a writer stalls, the freeze attempt fails after 30 seconds rather than acknowledging an incomplete drain. PostgreSQL row locking and SQLite writer serialization retain their existing behavior. Failed guard reads fail closed. Existing snapshot keys remain protected after the legacy pause is released.
+
+The acknowledgement covers DB commit or rollback and the end of any protected
+upload, not physical orphan reclamation. Cleanup of a rolled-back write may
+finish afterward; it can only delete and refund an unreferenced, hash-fenced
+blob, never change committed KV rows or upload new data.
 
 ## Deployment compatibility