diff --git a/CHANGELOG.md b/CHANGELOG.md index fb211081..60972656 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,15 @@ - SQL local durability and reservation before commit (`docs/sql-local-durability.md`). Postgres stays the master copy. File-backed SQL databases run `PRAGMA synchronous = FULL` (`[global.storage.sql] synchronous = "full"|"normal"`, env `TINYCLOUD_STORAGE__SQL__SYNCHRONOUS`; `normal` selects the earlier fsync behaviour), and the directories and renames that make a database file or a hydrated cache visible are fsynced. A write is checked before COMMIT and rolled back whole if it would pass the owner's remaining storage budget (`402`), and, when configured, the per-database cap (`max_database_bytes`, `507 sql_database_size_limit`) or the disk headroom (`min_free_bytes`, bounded by the WAL and checkpoint growth the transaction can cause and shared by all databases on the node, `507 sql_disk_headroom`). `max_database_bytes` and `min_free_bytes` default to `0` (off) and are set by config. A COMMIT that fails on disk answers `500 sql_commit_failed` and the node re-hydrates the database from Postgres. A write that fails because the disk is full answers `507 sql_disk_headroom`, whatever `min_free_bytes` is, not the owner's quota or the size limit. Every write request runs in one transaction (a single statement, a batch, a nested write (`WITH … INSERT`) and a request that carries a `PRAGMA` included), so such a request counts two more native SQL calls in request telemetry, and all of them get the same owner budget, size cap and disk gate. A request that wrote nothing is not gated and reserves nothing. The growth of the main file that the next checkpoint will cause stays held in the disk reservation from the commit until a checkpoint runs or the database's connection closes. No migration. Details: TC-1491. - `MemorySink` caps its size hint at 64 MiB and treats a failed reservation as no hint. Details: TC-1491. - A local SQL commit failure (answered 500) records the same diagnostic failure note as other SQL 500 responses. Details: TC-1491. +- KV puts persist content-addressed blobs before taking the per-space sequence lock, and committed `kv/sync` cursors stay ordered. A preparatory transaction authorizes the write and claims every distinct blob under a durable attempt, committing all claims or none; on PostgreSQL each claim statement also checks that the invocation's spaces are hosted. Uploads hold no pooled database connection, transaction or lock. Protected legacy meeting writes keep their pre-upload freeze barrier, which waits for commit or abort and fails after 30 seconds if stalled (TC-1260). + Failed attempts move their last unreferenced hash fence to `deleting`; a new writer cannot deduplicate or upload until confirmed physical absence removes the fence, and known failures refund once after deletion. A 60-second collector reclaims crash leftovers after a five-minute database-time grace when no same-hash writer remains, including lost delete responses; it closes, without deleting, a fence whose blob a committed write references. S3 startup reconstructs pending refunds from the same physical listing as its quota count. Quota stays conservative and per process, not a durable global ledger. A lost connection during registration retries the preparatory transaction with the same attempt ID; neither an upload nor a committed write is replayed on an uncertain outcome. + First uploads keep the bare hash key (generation zero); an upload after orphan deletion uses a generation-scoped key `hash@N`, so a delayed delete cannot remove newly committed bytes. Reads take the generation from the KV lookup. A binary without this change cannot read generation-above-zero records, so a fault after such records exist is recovered by a forward fix (`docs/kv-sync.md`). Migration `m20261023_010000_kv_orphan_candidates` adds the `generation` columns and the claim tables and can be applied again over its own tables on PostgreSQL and SQLite. + Physical operations on the same `(space, hash, generation)` serialize in-process from existence check through upload or deletion and quota settlement, so overlapping writers cannot double-charge and a stale S3 HEAD cannot re-create a refunded charge. Filesystem staging writes a same-directory temporary file and atomically publishes the final hash key; the collector reclaims a crash-leftover stage and its charge even if a later same-hash retry committed a reference to the final blob, and failed stage cleanup keeps the candidate for retry without deleting referenced content. Expired same-hash attempts drain as a group under their fence, and a live sibling defers cleanup without hiding unrelated orphans. Registration age stays immutable while a separate retry time delays blocked claims, each collector tick has a bounded page count, SQLite reserves the database writer before preparatory authorization reads, and PostgreSQL KV backfills from earlier schemas decode generation as BIGINT. + Boot quota reconstruction tracks stage-file charges from the same directory listing as the total; a failed retry and a crash-leftover stage from an earlier attempt each refund their own charge after confirmed removal. Known filesystem-stage failures refund their reservation immediately after the unpublished file is removed, and cancellation of a delete cannot interrupt its physical unlink and quota settlement. + Filesystem reservation and refund update pending charges atomically with `SpaceSizes` across cancellation. Each bounded collector page fairly admits never-retried and due-retry attempts through indexed queries, while a full deleting-fence page reserves capacity for expired attempts; persistent storage failures or live same-hash writers cannot starve unrelated orphans across ticks. When a writer departs, deferred siblings become due using DB time without regaining their first-attempt priority, so repeated departures also cannot starve unrelated orphans. + PostgreSQL selection skips independently held hash fences before its page limit and releases selection locks before storage I/O. A live same-space, same-hash attempt can defer an older crashed stage's physical-byte refund beyond the usual five-minute grace plus 60-second pass. Its upload has a 240-second deadline; the stage becomes due for the next scheduled pass (within 60 seconds) after the last overlapping attempt commits or aborts; a backlog beyond the collector's bounded pages can require further passes. Repeated overlaps extend that delay without deleting committed content or undercounting bytes. + Details: TC-735. +- Authorization is checked before a conditional KV write's precondition. Details: TC-735. ## [1.22.0] - 2026-10-23 diff --git a/docs/kv-sync.md b/docs/kv-sync.md index 7ed519bf..59178bf6 100644 --- a/docs/kv-sync.md +++ b/docs/kv-sync.md @@ -129,3 +129,145 @@ cursor. Read audit records every poll. Poll on demand, or no more often than every 60 seconds. + +## Write ordering and blob recovery + +KV puts stage and authorize content before upload. A short preparatory +transaction checks authorization and hosting, then commits an active +`(space, content hash, generation)` fence and a separate claim for each +writer. The first generation is zero and keeps the bare-hash +storage key; a generation above zero is allocated only after a prior +generation's physical deletion is confirmed. Ordinary writes release every +pooled connection and database lock before S3 or filesystem I/O. A +four-minute upload deadline stays below the five-minute DB-clock GC grace. +After upload, an ordinary writer takes the per-space sequence guard, starts +a fresh commit transaction and takes +the DB sequence lock; it verifies its own active claim, then rechecks +authorization and conditional-write preconditions. The guards stay through +commit, so `seq` and sync cursors remain commit-ordered even when same-space +uploads finish out of order. An unauthorized write uploads nothing. + +For a single node process, each `(space, hash, generation)` serializes its +physical existence check, upload or deletion, and quota settlement without +holding a database connection. Filesystem memory staging writes into a +same-directory temporary file and atomically renames only complete bytes to +the hash key. A process crash can leave an incomplete stage; startup counts +its physical bytes, and the hash candidate's GC pass removes and refunds it. +SQLite preparatory writes reserve the database writer before authorization +reads, so a separate collector connection cannot invalidate the read snapshot. + +Protected legacy meeting paths are the TC-1260 exception: the durable freeze +row and its pooled transaction remain locked across upload until commit or +abort. A freeze acknowledgement drains earlier uploads; a later protected +write is refused before upload. A stalled writer causes the freeze attempt +to fail after 30 seconds rather than acknowledge an incomplete drain. + +Before upload, a lost pooled connection retries the preparatory transaction +with the same attempt ID. The registration is idempotent; neither an upload +nor a committed write is replayed on an uncertain outcome. A missing claim +or a fence already deleting returns a retryable error, never a reference to +a potentially removed object. + +A lost connection during the preparatory transaction that survives its retry +answers 503 `unavailable_retryable`. The claims are idempotent per attempt ID +and hold only candidate metadata, so no committed KV write can be affected. + +On PostgreSQL each claim statement also checks that every space the invocation +names is hosted, so an unhosted space is refused after authorization and +before any upload without a separate query. The claims of all distinct blobs +in one put share that one preparatory transaction, which commits every claim +or none, so no partial claim is left for the collector. + +A failed upload, precondition or database write removes only its own claim. +If another claim or a current or historical KV reference exists, it leaves +the shared generation's blob alone. Otherwise it commits a `deleting` fence +and byte entitlement, releases the DB connection, then deletes that +generation's object. A writer seeing `deleting` cannot deduplicate or +upload over it. Only confirmed physical absence refunds the per-process +quota exactly once; only then is the fence retired and the next generation +made available. Each delete targets the storage key of its own generation, +so even a delayed second delete cannot remove a later committed generation. +A lost DELETE response or a crash mid-delete leaves the fence for +idempotent GC retry. Reads get the generation from their existing KV lookup, +without another database query; generation zero reads the bare key. A +rolled-back binary cannot read records with generation above zero, which +can occur after a failed upload is collected and its content is saved again. + +The collector runs every 60 seconds, aging attempts after five minutes by +DB time and draining eligible work in batches of 256. On S3 startup the +same object-listing snapshot reconstructs both `SpaceSizes` and which +`deleting` fences counted physical bytes; a deletion before the listing +needs no refund, while a deletion afterward refunds once. Orphans present +at restart count until GC. Quota is conservative physical accounting, +**not commit-only accounting**: pending uploads and crash leftovers can +temporarily over-count. `SpaceSizes` remains in-memory and per process; other +nodes' writes and deletes do not synchronize it. Globally exact multi-node +accounting requires a separate shared byte ledger (TC-1162). + +A put waits for the per-space sequence lock after its upload. The upload +deadline (240 seconds) does not cover that wait. If the wait outlasts the +five-minute grace, the collector can reclaim the claim and the commit answers +503 `unavailable_retryable`; nothing is stored for that attempt, and a retry +stores the content under the next generation. + +## Recovery after deploying generation-keyed blobs + +A forward fix is the default. An image rollback alone cannot boot: a binary +without this change refuses to start while the migration row +`m20261023_010000_kv_orphan_candidates` is recorded in `seaql_migrations`. +Take a named full database backup (a PlanetScale backup or branch) before the +deployment. Never run a binary without this change and one with it against the +same database at the same time. + +What a binary without this change does with the data this change writes, once +it is made to start: + +- It reads every record at the bare hash key. A record with `generation` above + zero is stored at `hash@N`, so it is reported as missing. +- When it overwrites a key whose `current_kv.generation` is above zero, it + does not update that column. After the return to this change the key reads + as missing until the column is repaired. +- It does not run the claim collector, so the claim tables stop being + maintained. + +Emergency path, available only while no record has `generation` above zero. +Stop the node and run it as one `psql` script with `-v ON_ERROR_STOP=1`. The +script raises, and changes nothing, when such a record exists: + +```sql +BEGIN; +DO $$ +BEGIN + IF EXISTS (SELECT 1 FROM current_kv WHERE generation > 0) + OR EXISTS (SELECT 1 FROM kv_write WHERE generation > 0) THEN + RAISE EXCEPTION 'generation > 0 rows exist: forward fix only'; + END IF; +END $$; +DELETE FROM seaql_migrations + WHERE version = 'm20261023_010000_kv_orphan_candidates'; +DROP TABLE kv_orphan_attempt, kv_orphan_candidate, kv_blob_generation; +COMMIT; +``` + +The `generation` columns stay. Dropping the tables abandons pending deletions: +the objects they named are not deleted, which leaks objects and loses no data. +Deleting only the history row and keeping the tables is also supported: the +binary without this change starts, and the return to this change applies the +migration again over the retained tables. A deleting fence retained across such +a window is closed by the collector without deleting a blob that a committed +write references. + +On the return to this change the migration is applied again, then run the +repair statement. It changes only rows whose generation differs from their +write and is safe to run every time: + +```sql +UPDATE current_kv c +SET generation = w.generation +FROM kv_write w +WHERE w.space = c.space AND w.key = c.key AND w.invocation = c.invocation + AND c.generation <> w.generation AND NOT c.deleted; +``` + +The migration can be applied again over its own tables and rows on PostgreSQL +and SQLite. On MySQL its indexes are created without a guard, so it cannot. diff --git a/docs/meeting-publication-migration.md b/docs/meeting-publication-migration.md index c665f242..6eca6582 100644 --- a/docs/meeting-publication-migration.md +++ b/docs/meeting-publication-migration.md @@ -48,7 +48,12 @@ An existing pause continues to block legacy writes under `xyz.tinycloud.tinychat Reads, chat keys, cursors, credentials, and other spaces retain their existing behavior. Frozen ordinary KV put/delete returns HTTP 409. The core guard also rejects internal legacy cleanup. The pause survives a node restart; only an exact generation-checked release removes it. -The durable guard remains locked by protected KV mutation transactions through storage persistence and commit. PostgreSQL row locking and SQLite writer serialization retain their existing behavior. Failed guard reads fail closed. Existing snapshot keys remain protected after the legacy pause is released. +The durable guard is acquired before a protected KV upload starts and stays locked through storage persistence and commit or abort. A freeze acknowledgement therefore drains writes already uploading; a later writer is refused before uploading. If a writer stalls, the freeze attempt fails after 30 seconds rather than acknowledging an incomplete drain. PostgreSQL row locking and SQLite writer serialization retain their existing behavior. Failed guard reads fail closed. Existing snapshot keys remain protected after the legacy pause is released. + +The acknowledgement covers DB commit or rollback and the end of any protected +upload, not physical orphan reclamation. Cleanup of a rolled-back write may +finish afterward; it can only delete and refund an unreferenced, hash-fenced +blob, never change committed KV rows or upload new data. ## Deployment compatibility diff --git a/docs/sql-block-store-contract.md b/docs/sql-block-store-contract.md index fa8a1a8f..abb31bdc 100644 --- a/docs/sql-block-store-contract.md +++ b/docs/sql-block-store-contract.md @@ -503,7 +503,7 @@ traits plus an in-memory implementation for tests): Migration `m20261021_000000_sql_store_manifest`. Other pending migrations: #374 adds `m20261020_000000_current_kv_key_c_index`; #301 (TC-735) carries -`m20261017_020000_kv_orphan_candidates`. SeaORM applies out-of-order +`m20261023_010000_kv_orphan_candidates`. SeaORM applies out-of-order versions; none touches these tables. Rules from #341 / #343 / #374: - `CREATE TABLE IF NOT EXISTS` for both tables; **no foreign keys** (an FK to diff --git a/tinycloud-core/src/db.rs b/tinycloud-core/src/db.rs index 6cbc4b40..e6f0b12d 100644 --- a/tinycloud-core/src/db.rs +++ b/tinycloud-core/src/db.rs @@ -12,8 +12,8 @@ use crate::read_audit::ReadAuditPipeline; use crate::relationships::*; use crate::sql_sizes::SqlSizes; use crate::storage::{ - either::EitherError, ByteRangeSpec, Content, HashBuffer, ImmutableReadStore, ImmutableStaging, - ImmutableWriteStore, RangeRead, StorageSetup, StoreSize, + either::EitherError, ByteRangeSpec, Content, HashBuffer, ImmutableDeleteStore, + ImmutableReadStore, ImmutableStaging, ImmutableWriteStore, RangeRead, StorageSetup, StoreSize, }; use crate::types::{ AccountDelegationRecord, CapabilitiesReadParams, DelegationQuery, DelegationQueryDirection, @@ -21,6 +21,7 @@ use crate::types::{ Resource, SpaceIdWrap, }; use crate::util::DelegationInfo; +use rand::{rngs::OsRng, RngCore}; use sea_orm::{ entity::prelude::*, error::{DbErr, RuntimeErr, SqlxError}, @@ -53,6 +54,12 @@ use tinycloud_auth::{ /// cap still does; the cap is what bounds HTTP/2 over TLS, whose header list /// may reach 16 MiB. pub const KV_MAX_MUTATIONS_PER_INVOCATION: usize = 4096; +/// One invocation's uploads finish within 240 seconds of preflight, before +/// the five-minute orphan age. A live same-hash retry defers an older crash +/// stage until that retry finishes; repeated overlaps can extend its charge. +const KV_UPLOAD_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(240); +const _: () = + assert!(KV_UPLOAD_TIMEOUT.as_secs() < crate::kv_orphans::ORPHAN_GRACE.whole_seconds() as u64); pub const HOOK_DELIVERY_STATUS_PENDING: &str = "pending"; pub const HOOK_DELIVERY_STATUS_RETRYING: &str = "retrying"; @@ -161,6 +168,7 @@ mod meeting_snapshot_fence_tests { #[derive(Debug, Clone)] pub struct SpaceDatabase { conn: C, + candidate_conn: RecoveringConnection, storage: B, secrets: S, encryption: Option, @@ -170,6 +178,9 @@ pub struct SpaceDatabase { revocation_chain_locks: ChainLockRegistry, kv_object_locks: KvObjectLockRegistry, space_sequence_locks: SpaceSequenceLockRegistry, + // SQLite lock order everywhere: per-space sequence guard, then writer + // mutex, then BEGIN. Freeze takes only the writer mutex. PostgreSQL has no + // writer mutex and takes its sequence guard after uploading. writer_lock: Option>>, read_audit: ReadAuditPipeline, database_fence_cache: Arc, @@ -289,6 +300,8 @@ where MeetingSnapshotProtected, #[error("legacy meeting artifacts are frozen")] LegacyMeetingFrozen, + #[error("KV blob candidate is unavailable; retry the invocation")] + KvCandidateUnavailable, #[error("KV precondition failed")] KvPreconditionFailed, #[error("KV response is {size} bytes, exceeding the requested limit of {limit} bytes")] @@ -325,6 +338,49 @@ where } } +/// Reject unauthorized or unhosted mutations before any content upload. +/// The commit transaction repeats authorization after the sequence lock. +/// `check_hosted` is false only where the caller's registration statement +/// checks that every space exists itself. +async fn authorize_mutation_preupload( + conn: &C, + invocation: &Invocation, + mode: InvokeMode, + now: OffsetDateTime, + spaces: &[SpaceId], + delegated_sql_caveats: bool, + check_hosted: bool, +) -> Result<(), TxError> { + match mode { + InvokeMode::Admitted => { + invocation::authorize_admitted(conn, &invocation.0, now, None, delegated_sql_caveats) + .await + .map_err(TxError::::from)?; + } + InvokeMode::Public => { + invocation::verify_and_authorize_with_setting( + conn, + &invocation.0, + now, + delegated_sql_caveats, + ) + .await + .map_err(TxError::::from)?; + } + InvokeMode::Internal | InvokeMode::MeetingPublication => {} + } + if check_hosted { + let hosted = space::Entity::find() + .filter(space::Column::Id.is_in(spaces.iter().cloned().map(SpaceIdWrap))) + .count(conn) + .await?; + if hosted != spaces.len() as u64 { + return Err(TxError::SpaceNotFound); + } + } + Ok(()) +} + /// How long a boot migration on PostgreSQL waits for any one lock before it /// fails (TC-1176). Schema changes need strong table locks, which queue /// behind any open transaction on the table and, while queued, block every @@ -418,8 +474,9 @@ where tx.commit().await?; let writer_lock = (pool.get_database_backend() == sea_orm::DatabaseBackend::Sqlite) .then(|| Arc::new(tokio::sync::Mutex::new(()))); - let read_audit = ReadAuditPipeline::start(pool, writer_lock.clone()); + let read_audit = ReadAuditPipeline::start(pool.clone(), writer_lock.clone()); Ok(Self { + candidate_conn: pool, conn, storage, secrets, @@ -620,7 +677,11 @@ impl SpaceDatabase where C: TransactionTrait, { - /// Change the legacy artifact barrier. Freeze commit is the drain acknowledgement. + /// Change the legacy artifact barrier. Freeze commit acknowledges that + /// in-flight protected writes have committed or rolled back in the DB and + /// no protected upload can still land afterward. Cleanup of an unreferenced, + /// hash-fenced blob from a rolled-back write may finish after the ACK; it + /// only deletes/refunds physical storage, never changes committed KV rows. pub async fn set_legacy_meeting_write_freeze( &self, space: &SpaceId, @@ -630,16 +691,27 @@ where where C: ConnectionTrait, { - let _writer = match &self.writer_lock { - Some(lock) => Some(lock.lock().await), - None => None, - }; - let tx = self.begin_graph_transaction().await?; - let status = - crate::meeting_legacy_guard::transition(&tx, space, frozen, expected_generation) - .await?; - tx.commit().await?; - Ok(status) + // Acknowledgement must never wait indefinitely for a stalled upload. + // On timeout the transaction is dropped (and rolled back), so the + // caller cannot treat a failed freeze as a drained write barrier. + tokio::time::timeout(std::time::Duration::from_secs(30), async { + let _writer = match &self.writer_lock { + Some(lock) => Some(lock.lock().await), + None => None, + }; + let tx = self.begin_graph_transaction().await?; + let status = + crate::meeting_legacy_guard::transition(&tx, space, frozen, expected_generation) + .await?; + tx.commit().await?; + Ok(status) + }) + .await + .unwrap_or_else(|_| { + Err(crate::meeting_legacy_guard::FreezeError::Db(DbErr::Custom( + "legacy meeting freeze timed out waiting for in-flight writes".into(), + ))) + }) } pub async fn legacy_meeting_freeze_status( @@ -1427,15 +1499,21 @@ where range: ByteRangeSpec, ) -> Result)>, EitherError> { - let Some((metadata, hash)) = metadata_with_hash(&self.conn, space_id, key) + let Some(entry) = get_kv_entity(&self.conn, space_id, key) .await .map_err(EitherError::A)? else { return Ok(None); }; + let current_kv::Model { + metadata, + value: hash, + generation, + .. + } = entry; let Some(content) = self .storage - .read_range(space_id, &hash, range) + .read_range(space_id, &hash, generation, range) .await .map_err(EitherError::B)? else { @@ -1721,7 +1799,7 @@ where precondition: Option, ) -> Result> where - B: ImmutableWriteStore + ImmutableReadStore, + B: ImmutableWriteStore + ImmutableReadStore + ImmutableDeleteStore, S: ImmutableStaging, S::Writable: 'static + Unpin, { @@ -1745,7 +1823,7 @@ where precondition: Option, ) -> Result> where - B: ImmutableWriteStore + ImmutableReadStore, + B: ImmutableWriteStore + ImmutableReadStore + ImmutableDeleteStore, S: ImmutableStaging, S::Writable: 'static + Unpin, { @@ -1769,7 +1847,7 @@ where path: Path, ) -> Result<(), TxStoreError> where - B: ImmutableWriteStore + ImmutableReadStore, + B: ImmutableWriteStore + ImmutableReadStore + ImmutableDeleteStore, S: ImmutableStaging, S::Writable: 'static + Unpin, { @@ -1803,7 +1881,7 @@ where mode: InvokeMode, ) -> Result, TxStoreError> where - B: ImmutableWriteStore + ImmutableReadStore, + B: ImmutableWriteStore + ImmutableReadStore + ImmutableDeleteStore, S: ImmutableStaging, S::Writable: 'static + Unpin, { @@ -2099,7 +2177,7 @@ where inputs: InvocationInputs, ) -> Result<(TransactResult, Vec>), TxStoreError> where - B: ImmutableWriteStore + ImmutableReadStore, + B: ImmutableWriteStore + ImmutableReadStore + ImmutableDeleteStore, S: ImmutableStaging, S::Writable: 'static + Unpin, { @@ -2120,7 +2198,7 @@ where inputs: InvocationInputs, ) -> Result<(TransactResult, Vec>), TxStoreError> where - B: ImmutableWriteStore + ImmutableReadStore, + B: ImmutableWriteStore + ImmutableReadStore + ImmutableDeleteStore, S: ImmutableStaging, S::Writable: 'static + Unpin, { @@ -2135,7 +2213,7 @@ where options: KvInvokeOptions, ) -> Result<(TransactResult, Vec>), TxStoreError> where - B: ImmutableWriteStore + ImmutableReadStore, + B: ImmutableWriteStore + ImmutableReadStore + ImmutableDeleteStore, S: ImmutableStaging, S::Writable: 'static + Unpin, { @@ -2155,7 +2233,7 @@ where options: KvInvokeOptions, ) -> Result<(TransactResult, Vec>), TxStoreError> where - B: ImmutableWriteStore + ImmutableReadStore, + B: ImmutableWriteStore + ImmutableReadStore + ImmutableDeleteStore, S: ImmutableStaging, S::Writable: 'static + Unpin, { @@ -2171,7 +2249,7 @@ where mode: InvokeMode, ) -> Result<(TransactResult, Vec>), TxStoreError> where - B: ImmutableWriteStore + ImmutableReadStore, + B: ImmutableWriteStore + ImmutableReadStore + ImmutableDeleteStore, S: ImmutableStaging, S::Writable: 'static + Unpin, { @@ -2323,6 +2401,7 @@ where key: path.clone(), metadata, value, + generation: 0, }); } // add delete for tx @@ -2337,134 +2416,470 @@ where } } - // TC-732: conditional writes no longer need SERIALIZABLE. The - // per-space sequence lock is taken right after BEGIN, before the - // precondition read, so under READ COMMITTED that read already sees - // every earlier same-space commit and no same-space writer can commit - // until this one does. A failed precondition is a 412, never a false - // success (see `postgres_conditional_put_is_atomic_under_concurrency`). - let sequence_spaces = invocation.0.spaces().cloned().collect::>(); - let _sequence_guards = self.acquire_space_sequence_guards(&sequence_spaces).await; - let _writer = match &self.writer_lock { - Some(lock) => Some(lock.lock().await), - None => None, - }; - // TC-1274: the write transaction begins here, so the request deadline - // may no longer cut this request; it completes and reports its outcome. - // The handler task is never polled after a cut, so this is not - // refused here. - let latched = crate::request_cancel::latch(); - debug_assert!(latched, "a cut handler is not polled again"); - let begin_start = Instant::now(); - let tx_result = self.begin_graph_transaction().await; - crate::telemetry::observe_stage( - crate::telemetry::InvocationStage::DbTxBegin, - crate::telemetry::StageOutcome::from(tx_result.is_ok()), - begin_start.elapsed(), - ); - let tx = tx_result?; - // TC-732: order this transaction's event sequence under the - // per-space lock. On PostgreSQL this is an advisory lock, so the - // meeting legacy guard lock below is still the first row lock. - crate::kv_sync::lock_space_sequences(&tx, &sequence_spaces).await?; - // On SQLite `begin_graph_transaction` already holds the write lock, - // so no read-to-write upgrade race remains. Distinct spaces are - // locked in a stable order to prevent cross-space deadlocks on PG. let mut legacy_spaces: Vec<_> = mutation_keys .iter() .filter(|(_, path)| crate::meeting_legacy_guard::protects(path.as_str())) .map(|(space, _)| space) .collect(); - legacy_spaces.sort_by_key(|space| space.to_string()); + legacy_spaces.sort_unstable(); legacy_spaces.dedup(); - for space in legacy_spaces { - if crate::meeting_legacy_guard::lock_writer(&tx, space).await? { + // Preserve frozen-write rejection before authorization and candidate + // registration. This read is only a preflight; the transaction takes + // the durable row lock before any upload below. + for space in &legacy_spaces { + if crate::meeting_legacy_guard::status(&self.conn, space) + .await + .map_err(|error| TxStoreError::Tx(TxError::Db(error)))? + .frozen + { return Err(TxStoreError::LegacyMeetingFrozen); } } - // DbTxBody spans post-begin to pre-commit. The guard defaults to an - // `error` outcome so any `?`/early return inside the transaction is - // recorded as a failure; it is disarmed to `ok` right before commit. - let tx_body_timer = - crate::telemetry::StageTimer::start(crate::telemetry::InvocationStage::DbTxBody); - let auth_graph = match crate::auth_graph::AuthGraphSnapshot::load(&tx, &roots).await { - Ok(graph) => graph, - Err(error) => { - crate::telemetry::observe_stage( - crate::telemetry::InvocationStage::AuthorizationGraphLoad, - crate::telemetry::StageOutcome::Error, - authz_start.elapsed(), - ); - return Err(TxStoreError::Tx(match error { - revocation::ChainTraversalError::Db(error) => TxError::Db(error), - revocation::ChainTraversalError::LimitExceeded => { - TxError::ChainTraversalLimitExceeded - } - })); + // Check the entire invocation before creating an object in the block + // store. The transaction below repeats authorization after acquiring + // the sequence lock, preserving revocation and time ordering. + let now = OffsetDateTime::now_utc(); + let mut sequence_spaces = invocation.0.spaces().cloned().collect::>(); + sequence_spaces.sort_unstable(); + sequence_spaces.dedup(); + let mut blobs = write_hashes + .iter() + .map(|((space, _), hash)| (space.clone(), *hash)) + .collect::>(); + blobs.sort_unstable(); + blobs.dedup(); + // GC ages attempts from registration, not from storage.persist. + // This deadline also covers preparatory retries and queueing behind + // legacy freeze locks, so no PUT may begin after its five-minute fence + // grace could expire. + let upload_deadline = tokio::time::Instant::now() + KV_UPLOAD_TIMEOUT; + let mut attempt_bytes = [0u8; 16]; + OsRng.fill_bytes(&mut attempt_bytes); + let attempt_id = hex::encode(attempt_bytes); + // Registration and authorization commit before any blob can be + // persisted. Retrying an uncertain preparatory COMMIT uses exactly the + // same attempt ID, so the claim is recorded once and no other + // writer's claim is affected. + // SQLite's preparatory write follows sequence→writer→BEGIN and + // releases every guard before an ordinary upload. + let mut retried = false; + let blobs = loop { + let _sequence = if self.writer_lock.is_some() { + self.acquire_space_sequence_guards(&sequence_spaces).await + } else { + Vec::new() + }; + let _writer = match &self.writer_lock { + Some(lock) => Some(lock.lock().await), + None => None, + }; + // TC-1274: the first data write (candidate registration, then the + // upload and the commit transaction) starts here, so the request + // deadline cannot cut this request after this point. The sequence and writer + // waits above stay cuttable. The handler task is never polled after + // a cut, so this is not refused here. + let latched = crate::request_cancel::latch(); + debug_assert!(latched, "a cut handler is not polled again"); + let prepared = Box::pin(self.register_preupload( + &invocation, + mode, + now, + &sequence_spaces, + &blobs, + &attempt_id, + )) + .await; + match prepared { + Ok(generations) => break generations, + Err(TxError::Db(error)) + if !retried && crate::db_pool::is_connection_error(&error) => + { + retried = true; + } + Err(TxError::Db(error)) if crate::kv_orphans::is_candidate_unavailable(&error) => { + return Err(TxStoreError::KvCandidateUnavailable); + } + Err(TxError::Db(error)) if crate::kv_orphans::is_space_not_hosted(&error) => { + return Err(TxError::::SpaceNotFound.into()); + } + Err(error) => return Err(error.into()), } }; - crate::telemetry::observe_stage( - crate::telemetry::InvocationStage::AuthorizationGraphLoad, - crate::telemetry::StageOutcome::Ok, - authz_start.elapsed(), - ); - let mut deleted_hashes = HashMap::new(); - for key @ (space, path) in &mutation_keys { - let current = get_kv_entity(&tx, space, path).await?; - if let Some(precondition) = options.preconditions.get(key) { - let matches = kv_precondition_matches( - *precondition, - current.as_ref().map(|entry| entry.value), - ); - if !matches { - return Err(TxStoreError::KvPreconditionFailed); + for op in &mut ops { + if let Operation::KvWrite { + space, + value, + generation, + .. + } = op + { + let index = blobs + .binary_search_by(|(s, v, _)| (s, v).cmp(&(&*space, &*value))) + .expect("registered generation for every staged write"); + *generation = blobs[index].2; + } + } + // Keep the write-only transaction future off caller/test thread stacks: + // its auth graph and commit state grew beyond the default debug stack. + // Read-only invocations return above without this allocation. + let result = Box::pin(async { + let begin_mutation = || { + let legacy_spaces = &legacy_spaces; + async move { + // Retrying a lost BEGIN or freeze lock is safe before + // any KV transaction body or COMMIT has run. + let mut retried = false; + loop { + let begin_start = Instant::now(); + let tx_result = self.begin_graph_transaction().await; + crate::telemetry::observe_stage( + crate::telemetry::InvocationStage::DbTxBegin, + crate::telemetry::StageOutcome::from(tx_result.is_ok()), + begin_start.elapsed(), + ); + let tx = match tx_result { + Ok(tx) => tx, + Err(error) + if !retried && crate::db_pool::is_connection_error(&error) => + { + retried = true; + continue; + } + Err(error) => return Err(TxStoreError::Tx(TxError::Db(error))), + }; + let locked: Result = async { + // Protected legacy writes take the durable freeze + // row before uploading and hold it through commit. + for space in legacy_spaces { + if crate::meeting_legacy_guard::lock_writer(&tx, space).await? { + return Ok(true); + } + } + Ok(false) + } + .await; + match locked { + Ok(false) => break Ok(tx), + Ok(true) => break Err(TxStoreError::LegacyMeetingFrozen), + Err(error) + if !retried && crate::db_pool::is_connection_error(&error) => + { + retried = true; + drop(tx); + } + Err(error) => break Err(error.into()), + } + } + } + }; + // TC-1260: protected legacy paths hold their `lock_writer` + // transaction across upload. Freeze ACK must wait + // until such an in-flight write commits or aborts. Ordinary paths + // never hold a transaction, pool connection, sequence or writer + // lock during upload. + let protected = !legacy_spaces.is_empty(); + let _legacy_sqlite_sequence = if protected && self.writer_lock.is_some() { + self.acquire_space_sequence_guards(&sequence_spaces).await + } else { + Vec::new() + }; + let _legacy_writer = if protected { + match &self.writer_lock { + Some(lock) => Some(lock.lock().await), + None => None, } + } else { + None + }; + let legacy_tx = if protected { + Some(begin_mutation().await?) + } else { + None + }; + if !blobs.is_empty() { + self.persist_registered_blobs::( + std::mem::take(&mut stages), + &write_hashes, + &blobs, + upload_deadline, + ) + .await?; } - if let Some(entry) = current { - deleted_hashes.insert(key.clone(), entry.value); + // PG sequence guards are taken after upload. SQLite takes them + // before its writer mutex and BEGIN. The protected SQLite writer + // already holds both guards above. + let _sequence_guards = if protected && self.writer_lock.is_some() { + Vec::new() + } else { + self.acquire_space_sequence_guards(&sequence_spaces).await + }; + let _ordinary_writer = if protected { + None + } else { + match &self.writer_lock { + Some(lock) => Some(lock.lock().await), + None => None, + } + }; + let tx = match legacy_tx { + Some(tx) => tx, + None => begin_mutation().await?, + }; + crate::kv_sync::lock_space_sequences(&tx, &sequence_spaces).await?; + // Legacy freeze alone holds its transaction across upload. Take + // the hash fence only now, after all external storage I/O, for + // both legacy and ordinary writes. + if !blobs.is_empty() { + crate::kv_orphans::lock_candidates(&tx, &blobs, &attempt_id) + .await + .map_err(|error| { + if crate::kv_orphans::is_candidate_unavailable(&error) { + TxStoreError::KvCandidateUnavailable + } else { + error.into() + } + })?; } - } - let caps = invocation.0.capabilities.clone(); - let invoker = invocation.0.invoker.clone(); - // Extract capabilities read params from UCAN facts field - // Facts is Vec, we look for an object with capabilitiesReadParams key - let caps_read_params: Option = invocation - .0 - .invocation - .payload() - .facts - .as_ref() - .and_then(|facts| { - facts.iter().find_map(|fact| { - fact.as_object() - .and_then(|obj| obj.get("capabilitiesReadParams")) - .and_then(|v| serde_json::from_value(v.clone()).ok()) - }) - }); - // verify and commit invocation and kv operations - let event = match mode { - InvokeMode::Internal | InvokeMode::MeetingPublication => { - Event::InternalInvocation(Box::new(invocation), ops) + // The sequence lock precedes every precondition read under READ + // COMMITTED, including when an upload was prepared concurrently. + // SQLite already holds its single-writer lock at BEGIN, preventing a + // read-to-write upgrade race. Multi-space PG locks are acquired in + // stable order to avoid cross-space deadlocks. + // DbTxBody spans post-begin to pre-commit. The guard defaults to an + // `error` outcome so any `?`/early return inside the transaction is + // recorded as a failure; it is disarmed to `ok` right before commit. + let tx_body_timer = + crate::telemetry::StageTimer::start(crate::telemetry::InvocationStage::DbTxBody); + let auth_graph = match crate::auth_graph::AuthGraphSnapshot::load(&tx, &roots).await { + Ok(graph) => graph, + Err(error) => { + crate::telemetry::observe_stage( + crate::telemetry::InvocationStage::AuthorizationGraphLoad, + crate::telemetry::StageOutcome::Error, + authz_start.elapsed(), + ); + return Err(TxStoreError::Tx(match error { + revocation::ChainTraversalError::Db(error) => TxError::Db(error), + revocation::ChainTraversalError::LimitExceeded => { + TxError::ChainTraversalLimitExceeded + } + })); + } + }; + crate::telemetry::observe_stage( + crate::telemetry::InvocationStage::AuthorizationGraphLoad, + crate::telemetry::StageOutcome::Ok, + authz_start.elapsed(), + ); + let mut deleted_hashes = HashMap::new(); + for key @ (space, path) in &mutation_keys { + let current = get_kv_entity(&tx, space, path).await?; + if let Some(precondition) = options.preconditions.get(key) { + let matches = kv_precondition_matches( + *precondition, + current.as_ref().map(|entry| entry.value), + ); + if !matches { + return Err(TxStoreError::KvPreconditionFailed); + } + } + if let Some(entry) = current { + deleted_hashes.insert(key.clone(), entry.value); + } } - InvokeMode::Admitted => Event::AdmittedInvocation(Box::new(invocation), ops), - InvokeMode::Public => Event::Invocation(Box::new(invocation), ops), - }; - let commit = transact( + let caps = invocation.0.capabilities.clone(); + let invoker = invocation.0.invoker.clone(); + // Extract capabilities read params from UCAN facts field + // Facts is Vec, we look for an object with capabilitiesReadParams key + let caps_read_params: Option = invocation + .0 + .invocation + .payload() + .facts + .as_ref() + .and_then(|facts| { + facts.iter().find_map(|fact| { + fact.as_object() + .and_then(|obj| obj.get("capabilitiesReadParams")) + .and_then(|v| serde_json::from_value(v.clone()).ok()) + }) + }); + // verify and commit invocation and kv operations + let event = match mode { + InvokeMode::Internal | InvokeMode::MeetingPublication => { + Event::InternalInvocation(Box::new(invocation), ops) + } + InvokeMode::Admitted => Event::AdmittedInvocation(Box::new(invocation), ops), + InvokeMode::Public => Event::Invocation(Box::new(invocation), ops), + }; + let commit = transact( + &tx, + &self.storage, + &self.secrets, + vec![event], + self.encryption.as_ref(), + Some(&auth_graph), + // Taken on this `tx` right after BEGIN, above. + &sequence_spaces, + self.delegated_sql_caveats, + self.named_keys, + ) + .await?; + + let results = self + .invocation_side_effects::( + &tx, + &caps, + &invoker, + &options, + &write_hashes, + &deleted_hashes, + caps_read_params.as_ref(), + ) + .await?; + if !blobs.is_empty() { + crate::kv_orphans::finish_candidates(&tx, &blobs, &attempt_id).await?; + } + + // Record the transaction body as successful (post-begin to + // pre-commit) before the commit itself; commit latency is tracked + // separately (EpochPersist on the delegate/revoke path). + tx_body_timer.observe_ok(); + // commit tx if all side effects worked + tx.commit().await?; + Ok((commit, results)) + }) + .await; + if result.is_err() && !blobs.is_empty() { + // The failed transaction (including any hash locks) has ended. + // A commit error may be outcome-unknown: the candidate fence and + // committed-reference check distinguish that from an orphan. + if let Err(error) = + crate::kv_orphans::refund_failed(&self.conn, &self.storage, &blobs, &attempt_id) + .await + { + tracing::warn!( + ?error, + "failed to refund orphaned KV blob; periodic collector will retry" + ); + } + } + result + } + + /// Pre-upload authorization and candidate registration (TC-735), before + /// any bytes reach the store. The commit transaction still repeats + /// authorization under the sequence lock. + /// + /// One preparatory transaction authorizes the invocation and claims every + /// distinct blob, committing all claims or none; a lost connection retries + /// the whole block with the same attempt ID. On PostgreSQL each claim + /// statement also checks that every space the invocation names exists, so + /// an unhosted space is refused with `SpaceNotFound`, after authorization + /// and before any upload. Without a blob to claim, and on other backends, + /// a separate COUNT makes that check. + /// + /// A separate function, not inline in the caller: see + /// [`Self::persist_registered_blobs`]. + async fn register_preupload( + &self, + invocation: &Invocation, + mode: InvokeMode, + now: OffsetDateTime, + spaces: &[SpaceId], + blobs: &[(SpaceId, Hash)], + attempt_id: &str, + ) -> Result, TxError> + where + B: StorageSetup, + { + let tx = self.candidate_conn.begin_preparatory().await?; + let claim_checks_hosting = + !blobs.is_empty() && tx.get_database_backend() == DbBackend::Postgres; + authorize_mutation_preupload::<_, B, K>( &tx, - &self.storage, - &self.secrets, - vec![event], - self.encryption.as_ref(), - Some(&auth_graph), - // Taken on this `tx` right after BEGIN, above. - &sequence_spaces, + invocation, + mode, + now, + spaces, self.delegated_sql_caveats, - self.named_keys, + !claim_checks_hosting, ) .await?; + let generations = if blobs.is_empty() { + Vec::new() + } else { + crate::kv_orphans::register_candidates_hosted(&tx, blobs, attempt_id, spaces).await? + }; + tx.commit().await?; + Ok(generations) + } + + /// Upload staged KV blobs under their registered generations before any + /// sequence lock or KV transaction. + /// + /// A separate function, not inline in the commit future: debug builds give + /// every awaited temporary its own stack slot, and the commit future runs + /// the deepest request path, so the upload loop stays out of its frame. + async fn persist_registered_blobs( + &self, + stages: HashMap<(SpaceId, Path), HashBuffer>, + write_hashes: &HashMap<(SpaceId, Path), Hash>, + blobs: &[(SpaceId, Hash, i64)], + upload_deadline: tokio::time::Instant, + ) -> Result<(), TxStoreError> + where + B: ImmutableWriteStore + ImmutableReadStore + ImmutableDeleteStore, + S: ImmutableStaging, + S::Writable: 'static + Unpin, + { + if tokio::time::Instant::now() >= upload_deadline { + return Err(std::io::Error::new( + std::io::ErrorKind::TimedOut, + "KV upload deadline elapsed before persist", + ) + .into()); + } + tokio::time::timeout_at(upload_deadline, async { + for (key, stage) in stages { + let expected = write_hashes[&key]; + let (space, _) = key; + let index = blobs + .binary_search_by(|(s, v, _)| (s, v).cmp(&(&space, &expected))) + .expect("registered generation for every staged write"); + let hash = self + .storage + .persist(&space, stage, blobs[index].2) + .await + .map_err(TxStoreError::StoreWrite)?; + debug_assert_eq!(hash, expected); + } + Ok::<(), TxStoreError>(()) + }) + .await + .map_err(|_| std::io::Error::new(std::io::ErrorKind::TimedOut, "KV upload timed out"))? + } + /// Perform and record the invocation's side effects inside the KV + /// transaction, after `transact`. Separate from the commit future for the + /// same stack reason as [`Self::persist_registered_blobs`]. + #[allow(clippy::too_many_arguments)] + async fn invocation_side_effects( + &self, + tx: &DatabaseTransaction, + caps: &[crate::util::Capability], + invoker: &str, + options: &KvInvokeOptions, + write_hashes: &HashMap<(SpaceId, Path), Hash>, + deleted_hashes: &HashMap<(SpaceId, Path), Hash>, + caps_read_params: Option<&CapabilitiesReadParams>, + ) -> Result>, TxStoreError> + where + B: ImmutableWriteStore + ImmutableReadStore + ImmutableDeleteStore, + S: ImmutableStaging, + S::Writable: 'static + Unpin, + { let mut results = Vec::new(); - // perform and record side effects for cap in caps.iter().filter_map(|c| { c.resource.tinycloud_resource().and_then(|r| { Some(( @@ -2480,7 +2895,7 @@ where match cap { (space, "kv", "tinycloud.kv/get", path) => { let data = - get_kv(&tx, &self.storage, space, path) + get_kv(tx, &self.storage, space, path) .await .map_err(|e| match e { EitherError::A(e) => TxStoreError::Tx(e.into()), @@ -2500,7 +2915,7 @@ where } (space, "kv", "tinycloud.kv/list", path) => { let (list, truncated) = list_bounded_after( - &tx, + tx, space, path, options.list_limit, @@ -2517,38 +2932,32 @@ where )) } (space, "kv", "tinycloud.kv/put", path) => { - if let Some(stage) = stages.remove(&(space.clone(), path.clone())) { - self.storage - .persist(space, stage) - .await - .map_err(TxStoreError::StoreWrite)?; - let hash = write_hashes - .get(&(space.clone(), path.clone())) - .copied() - .expect("staged KV writes have a content hash"); - results.push(InvocationOutcome::KvWrite(hash)) - } + let hash = write_hashes + .get(&(space.clone(), path.clone())) + .copied() + .expect("staged KV writes have a content hash"); + results.push(InvocationOutcome::KvWrite(hash)); } (space, "kv", "tinycloud.kv/metadata", path) => results.push( - InvocationOutcome::KvMetadata(metadata_with_hash(&tx, space, path).await?), + InvocationOutcome::KvMetadata(metadata_with_hash(tx, space, path).await?), ), (space, "capabilities", "tinycloud.capabilities/read", path) if path.as_str() == "all" => { - match &caps_read_params { + match caps_read_params { None => { // Backward compatible: no params means return all valid delegations results.push(InvocationOutcome::OpenSessions( - get_valid_delegations(&tx, space, self.encryption.as_ref()).await?, + get_valid_delegations(tx, space, self.encryption.as_ref()).await?, )) } Some(CapabilitiesReadParams::List { filters }) => { // List with optional filters results.push(InvocationOutcome::OpenSessions( get_filtered_delegations( - &tx, + tx, space, - &invoker, + invoker, filters.as_ref(), self.encryption.as_ref(), ) @@ -2559,7 +2968,7 @@ where // Get the delegation chain for a specific delegation results.push(InvocationOutcome::DelegationChain( get_delegation_chain( - &tx, + tx, space, delegation_cid, self.encryption.as_ref(), @@ -2572,14 +2981,7 @@ where _ => {} }; } - - // Record the transaction body as successful (post-begin to - // pre-commit) before the commit itself; commit latency is tracked - // separately (EpochPersist on the delegate/revoke path). - tx_body_timer.observe_ok(); - // commit tx if all side effects worked - tx.commit().await?; - Ok((commit, results)) + Ok(results) } /// Execute a successful non-mutating invocation without entering the @@ -4095,7 +4497,7 @@ async fn read_kv_entity( e: current_kv::Model, ) -> Result)>, B::Error> { let content_hash = e.value; - let c = match store.read(space_id, &content_hash).await? { + let c = match store.read(space_id, &content_hash, e.generation).await? { Some(c) => c, None => return Ok(None), }; @@ -5075,6 +5477,31 @@ mod test { .unwrap(); } + #[tokio::test] + async fn sqlite_preupload_registration_reserves_writer_before_authorization_reads() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("candidate.db"); + let conn = sqlite_write_pool(&path, std::time::Duration::from_secs(5), 1).await; + let collector = sqlite_write_pool(&path, std::time::Duration::from_millis(100), 1).await; + let recovering = RecoveringConnection::new(conn); + + let prep = recovering.begin_preparatory().await.unwrap(); + epoch::Entity::find().count(&prep).await.unwrap(); + let error = test_actor("did:key:collector") + .insert(&collector) + .await + .expect_err("collector cannot invalidate the preparatory read snapshot"); + assert_eq!(sqlite_error_code(&error).as_deref(), Some("5"), "{error}"); + prep.execute_unprepared(SQLITE_RESERVE_WRITE_LOCK) + .await + .expect("registration after authorization remains writable"); + prep.commit().await.unwrap(); + test_actor("did:key:collector") + .insert(&collector) + .await + .unwrap(); + } + #[tokio::test] async fn postgres_write_transaction_is_a_plain_begin() { let Some(database_url) = @@ -5230,59 +5657,299 @@ mod test { } #[tokio::test] - async fn legacy_freeze_covers_public_admitted_put_delete_and_deprecated_delete() { + async fn sqlite_prepersist_upload_does_not_invert_delegate_lock_order() { use crate::storage::memory::MemoryStaging; use futures::io::AsyncWriteExt; - let (db, space) = frozen_legacy_fixture().await; - let path: Path = "xyz.tinycloud.tinychat/connectors/fireflies/archive-copy/transcript/old" - .parse() - .unwrap(); - for ability in [ - "tinycloud.kv/put", - "tinycloud.kv/del", - "tinycloud.kv/delete", - ] { - for admitted in [false, true] { - let invocation = legacy_test_invocation(&space, &path, ability); - let mut inputs = HashMap::new(); - if ability == "tinycloud.kv/put" { - let mut stage = HashBuffer::new(Vec::new()); - stage.write_all(b"replacement").await.unwrap(); - inputs.insert( - (space.clone(), path.clone()), - (Metadata(Default::default()), stage), - ); - } - let result = if admitted { - db.invoke_admitted::( - AdmittedInvocation::admit(invocation, 600).await.unwrap(), - inputs, - ) - .await - } else { - db.invoke::(invocation, inputs).await - }; - assert!( - matches!(result, Err(TxStoreError::LegacyMeetingFrozen)), - "{ability} admitted={admitted}" - ); - } - } - } - #[tokio::test] - async fn legacy_freeze_preserves_artifacts_but_allows_cursor_chat_other_space_and_native_snapshot( - ) { - use crate::storage::memory::MemoryStaging; - use futures::io::AsyncWriteExt; - let db = get_db().await.unwrap(); - let space = test_space_id("legacy-scope"); - let other = test_space_id("legacy-other"); - for id in [&space, &other] { - space::ActiveModel { - id: Set(SpaceIdWrap(id.clone())), - } - .insert(&db.conn) + let directory = tempfile::tempdir().unwrap(); + let url = format!( + "sqlite://{}?mode=rwc", + directory.path().join("lock-order.db").display() + ); + let storage = LegacyPausedStore::default(); + let db = SpaceDatabase::new( + Database::connect(url).await.unwrap(), + storage.clone(), + StaticSecret::new(vec![0; 32]).unwrap(), + ) + .await + .unwrap(); + let space = test_space_id("sqlite-lock-order"); + space::ActiveModel { + id: Set(SpaceIdWrap(space.clone())), + } + .insert(&db.conn) + .await + .unwrap(); + let writer_db = db.clone(); + let writer_space = space.clone(); + let writer = tokio::spawn(async move { + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(b"paused upload").await.unwrap(); + writer_db + .invoke_internal_kv_put::( + writer_space, + "lock/key".parse().unwrap(), + Metadata(Default::default()), + stage, + None, + ) + .await + .map_err(|error| error.to_string()) + }); + tokio::time::timeout( + std::time::Duration::from_secs(10), + storage.entered.notified(), + ) + .await + .unwrap(); + let invocation = + legacy_test_invocation(&space, &"lock/other".parse().unwrap(), "tinycloud.kv/del"); + // Keep the generic transaction live after it takes the sequence + // guard. A one-shot poll would leave it suspended indefinitely and + // incorrectly report a deadlock even when both paths have one order. + let writer_gate = tokio::time::timeout( + std::time::Duration::from_secs(3), + db.writer_lock.as_ref().unwrap().lock(), + ) + .await + .expect("ordinary upload must not hold the SQLite writer mutex"); + let delegate_db = db.clone(); + let delegate = tokio::spawn(async move { + delegate_db + .transact(vec![Event::InternalInvocation( + Box::new(invocation), + vec![], + )]) + .await + .map_err(|error| error.to_string()) + }); + tokio::time::timeout(std::time::Duration::from_secs(3), async { + loop { + let acquired = db + .space_sequence_locks + .lock() + .await + .get(&space) + .and_then(Weak::upgrade) + .is_some_and(|lock| lock.try_lock().is_err()); + if acquired { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("generic transaction must acquire the sequence before writer"); + drop(writer_gate); + storage.release.notify_one(); + let (write, delegated) = tokio::time::timeout(std::time::Duration::from_secs(10), async { + tokio::join!(writer, delegate) + }) + .await + .expect("KV upload and generic transaction must not deadlock"); + assert!(write.unwrap().is_ok()); + assert!(delegated.unwrap().is_ok()); + } + + #[tokio::test] + async fn sqlite_ordinary_upload_does_not_hold_writer_or_sequence() { + use crate::storage::memory::MemoryStaging; + use futures::io::AsyncWriteExt; + + let directory = tempfile::tempdir().unwrap(); + let url = format!( + "sqlite://{}?mode=rwc", + directory.path().join("ordinary.db").display() + ); + let storage = LegacyPausedStore::default(); + let db = SpaceDatabase::new( + Database::connect(url).await.unwrap(), + storage.clone(), + StaticSecret::new(vec![0; 32]).unwrap(), + ) + .await + .unwrap(); + let space = test_space_id("sqlite-ordinary-pool-release"); + space::ActiveModel { + id: Set(SpaceIdWrap(space.clone())), + } + .insert(&db.conn) + .await + .unwrap(); + let writer_db = db.clone(); + let writer_space = space.clone(); + let writer = tokio::spawn(async move { + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(b"ordinary upload").await.unwrap(); + writer_db + .invoke_internal_kv_put::( + writer_space, + "notes/paused".parse().unwrap(), + Metadata(Default::default()), + stage, + None, + ) + .await + }); + tokio::time::timeout( + std::time::Duration::from_secs(10), + storage.entered.notified(), + ) + .await + .unwrap(); + // GC can run while an ordinary S3/FS upload is in flight. The + // attempt is newer than its five-minute grace, so it stays durable + // until commit even though the blob has not landed yet. + let attempts_before = crate::models::kv_orphan_attempt::Entity::find() + .all(&db.conn) + .await + .unwrap(); + assert_eq!(attempts_before.len(), 1); + assert_eq!( + crate::kv_orphans::collect_candidates_with_progress( + &db.conn, + &storage, + 256, + crate::kv_orphans::ORPHAN_GRACE, + ) + .await + .unwrap(), + (0, 0) + ); + assert_eq!( + crate::models::kv_orphan_attempt::Entity::find() + .all(&db.conn) + .await + .unwrap(), + attempts_before + ); + let invocation = legacy_test_invocation( + &space, + &"notes/deleted".parse().unwrap(), + "tinycloud.kv/del", + ); + tokio::time::timeout( + std::time::Duration::from_secs(3), + db.transact(vec![Event::InternalInvocation( + Box::new(invocation), + vec![], + )]), + ) + .await + .expect("SQLite writer and sequence must be free during ordinary upload") + .map_err(|error| error.to_string()) + .unwrap(); + storage.release.notify_one(); + tokio::time::timeout(std::time::Duration::from_secs(10), writer) + .await + .unwrap() + .unwrap() + .map_err(|error| error.to_string()) + .unwrap(); + } + + #[tokio::test] + async fn unauthorized_ordinary_put_never_reaches_storage() { + use crate::storage::memory::MemoryStaging; + use futures::io::AsyncWriteExt; + + let directory = tempfile::tempdir().unwrap(); + let url = format!( + "sqlite://{}?mode=rwc", + directory.path().join("unauthorized.db").display() + ); + let storage = LegacyPausedStore::default(); + let db = SpaceDatabase::new( + Database::connect(url).await.unwrap(), + storage.clone(), + StaticSecret::new(vec![0; 32]).unwrap(), + ) + .await + .unwrap(); + let space = test_space_id("unauthorized-ordinary"); + space::ActiveModel { + id: Set(SpaceIdWrap(space.clone())), + } + .insert(&db.conn) + .await + .unwrap(); + let path: Path = "notes/unauthorized".parse().unwrap(); + // The signed invocation presents an unregistered delegation. + let invocation = legacy_test_invocation(&space, &path, "tinycloud.kv/put"); + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(b"unauthorized").await.unwrap(); + let result = tokio::time::timeout( + std::time::Duration::from_secs(3), + db.invoke::( + invocation, + HashMap::from([((space, path), (Metadata(Default::default()), stage))]), + ), + ) + .await + .expect("authorization should fail before persist"); + assert!(result.is_err()); + assert_eq!(storage.uploads.load(std::sync::atomic::Ordering::SeqCst), 0,); + } + + #[tokio::test] + async fn legacy_freeze_covers_public_admitted_put_delete_and_deprecated_delete() { + use crate::storage::memory::MemoryStaging; + use futures::io::AsyncWriteExt; + let (db, space) = frozen_legacy_fixture().await; + let path: Path = "xyz.tinycloud.tinychat/connectors/fireflies/archive-copy/transcript/old" + .parse() + .unwrap(); + for ability in [ + "tinycloud.kv/put", + "tinycloud.kv/del", + "tinycloud.kv/delete", + ] { + for admitted in [false, true] { + let invocation = legacy_test_invocation(&space, &path, ability); + let mut inputs = HashMap::new(); + if ability == "tinycloud.kv/put" { + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(b"replacement").await.unwrap(); + inputs.insert( + (space.clone(), path.clone()), + (Metadata(Default::default()), stage), + ); + } + let result = if admitted { + db.invoke_admitted::( + AdmittedInvocation::admit(invocation, 600).await.unwrap(), + inputs, + ) + .await + } else { + db.invoke::(invocation, inputs).await + }; + assert!( + matches!(&result, Err(TxStoreError::LegacyMeetingFrozen)), + "{ability} admitted={admitted}: {:?}", + result + .as_ref() + .map(|_| ()) + .map_err(|error| error.to_string()) + ); + } + } + } + + #[tokio::test] + async fn legacy_freeze_preserves_artifacts_but_allows_cursor_chat_other_space_and_native_snapshot( + ) { + use crate::storage::memory::MemoryStaging; + use futures::io::AsyncWriteExt; + let db = get_db().await.unwrap(); + let space = test_space_id("legacy-scope"); + let other = test_space_id("legacy-other"); + for id in [&space, &other] { + space::ActiveModel { + id: Set(SpaceIdWrap(id.clone())), + } + .insert(&db.conn) .await .unwrap(); } @@ -5414,6 +6081,11 @@ mod test { inner: MemoryStore, entered: Arc, release: Arc, + fail_after_persist: bool, + uploads: Arc, + pause_remove: bool, + removing: Arc, + remove_release: Arc, } #[async_trait::async_trait] impl StorageSetup for LegacyPausedStore { @@ -5426,23 +6098,30 @@ mod test { impl ImmutableReadStore for LegacyPausedStore { type Error = std::io::Error; type Readable = futures::io::Cursor>; - async fn contains(&self, space: &SpaceId, hash: &Hash) -> Result { - self.inner.contains(space, hash).await + async fn contains( + &self, + space: &SpaceId, + hash: &Hash, + generation: i64, + ) -> Result { + self.inner.contains(space, hash, generation).await } async fn read( &self, space: &SpaceId, hash: &Hash, + generation: i64, ) -> Result>, Self::Error> { - self.inner.read(space, hash).await + self.inner.read(space, hash, generation).await } async fn read_range( &self, space: &SpaceId, hash: &Hash, + generation: i64, range: crate::storage::ByteRangeSpec, ) -> Result>, Self::Error> { - self.inner.read_range(space, hash, range).await + self.inner.read_range(space, hash, generation, range).await } } #[async_trait::async_trait] @@ -5452,10 +6131,42 @@ mod test { &self, space: &SpaceId, stage: HashBuffer>, + generation: i64, ) -> Result { + self.uploads + .fetch_add(1, std::sync::atomic::Ordering::SeqCst); self.entered.notify_one(); self.release.notified().await; - self.inner.persist(space, stage).await + let hash = self.inner.persist(space, stage, generation).await?; + if self.fail_after_persist { + return Err(std::io::Error::other("lost upload response")); + } + Ok(hash) + } + } + #[async_trait::async_trait] + impl ImmutableDeleteStore for LegacyPausedStore { + type Error = std::io::Error; + + async fn remove( + &self, + space: &SpaceId, + hash: &Hash, + generation: i64, + ) -> Result, Self::Error> { + if self.pause_remove { + self.removing.notify_one(); + self.remove_release.notified().await; + } + self.inner.remove(space, hash, generation).await + } + async fn cleanup_stages( + &self, + space: &SpaceId, + hash: &Hash, + generation: i64, + ) -> Result<(), Self::Error> { + self.inner.cleanup_stages(space, hash, generation).await } } #[tokio::test] @@ -5561,6 +6272,516 @@ mod test { )); } + #[tokio::test] + async fn postgres_ordinary_upload_releases_pool_and_sequence_before_persist() { + use crate::storage::memory::MemoryStaging; + use futures::io::AsyncWriteExt; + + let Some(schema) = crate::test_support::PostgresTestSchema::new( + "postgres_ordinary_upload_releases_pool_and_sequence_before_persist", + ) + .await + else { + return; + }; + let storage = LegacyPausedStore::default(); + let db = SpaceDatabase::new( + schema.conn.clone(), + storage.clone(), + StaticSecret::new(vec![0; 32]).unwrap(), + ) + .await + .unwrap(); + let space = test_space_id("pg-ordinary-pool-release"); + space::ActiveModel { + id: Set(SpaceIdWrap(space.clone())), + } + .insert(&db.conn) + .await + .unwrap(); + let pool = schema.conn.get_postgres_connection_pool(); + // The fixture's max_connections is four: reserve three and leave + // precisely one for both preparatory and final transactions. + let mut held = Vec::new(); + for _ in 0..3 { + held.push(pool.acquire().await.unwrap()); + } + let baseline = pool.size() as usize - pool.num_idle(); + let writer_db = db.clone(); + let writer_space = space.clone(); + let writer = tokio::spawn(async move { + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(b"ordinary upload").await.unwrap(); + writer_db + .invoke_internal_kv_put::( + writer_space, + "notes/paused".parse().unwrap(), + Metadata(Default::default()), + stage, + None, + ) + .await + }); + tokio::time::timeout( + std::time::Duration::from_secs(10), + storage.entered.notified(), + ) + .await + .unwrap(); + tokio::time::timeout(std::time::Duration::from_secs(1), async { + loop { + if pool.size() as usize - pool.num_idle() == baseline { + break; + } + tokio::task::yield_now().await; + } + }) + .await + .expect("ordinary upload must release its pooled connection while paused"); + // A different same-space transaction must commit while upload is paused. + let invocation = legacy_test_invocation( + &space, + &"notes/deleted".parse().unwrap(), + "tinycloud.kv/del", + ); + tokio::time::timeout( + std::time::Duration::from_secs(3), + db.transact(vec![Event::InternalInvocation( + Box::new(invocation), + vec![], + )]), + ) + .await + .expect("another same-space write should not wait for an ordinary upload") + .map_err(|error| error.to_string()) + .unwrap(); + storage.release.notify_one(); + tokio::time::timeout(std::time::Duration::from_secs(10), writer) + .await + .unwrap() + .unwrap() + .map_err(|error| error.to_string()) + .unwrap(); + } + + #[tokio::test] + async fn postgres_legacy_freeze_during_slow_upload_waits_for_commit() { + use crate::storage::memory::MemoryStaging; + use futures::io::AsyncWriteExt; + let Some(schema) = crate::test_support::PostgresTestSchema::new( + "postgres_legacy_freeze_during_slow_upload_waits_for_commit", + ) + .await + else { + return; + }; + let storage = LegacyPausedStore::default(); + let first = SpaceDatabase::new( + schema.conn.clone(), + storage.clone(), + StaticSecret::new(vec![0; 32]).unwrap(), + ) + .await + .unwrap(); + let second = SpaceDatabase::new( + schema.conn.clone(), + storage.clone(), + StaticSecret::new(vec![0; 32]).unwrap(), + ) + .await + .unwrap(); + let space = test_space_id("pg-legacy-slow-upload"); + space::ActiveModel { + id: Set(SpaceIdWrap(space.clone())), + } + .insert(&first.conn) + .await + .unwrap(); + let path: Path = "xyz.tinycloud.tinychat/connectors/fireflies/transcript/slow" + .parse() + .unwrap(); + let writer_space = space.clone(); + let writer_path = path.clone(); + let writer = tokio::spawn(async move { + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(b"slow original").await.unwrap(); + first + .invoke_internal_kv_put::( + writer_space, + writer_path, + Metadata(Default::default()), + stage, + None, + ) + .await + .map_err(|error| error.to_string()) + }); + tokio::time::timeout( + std::time::Duration::from_secs(10), + storage.entered.notified(), + ) + .await + .unwrap(); + let freeze_space = space.clone(); + let reader = second.clone(); + let mut freeze = tokio::spawn(async move { + second + .set_legacy_meeting_write_freeze(&freeze_space, true, 0) + .await + }); + let early_ack = + tokio::time::timeout(std::time::Duration::from_millis(100), &mut freeze).await; + storage.release.notify_one(); + let written = tokio::time::timeout(std::time::Duration::from_secs(10), writer) + .await + .unwrap() + .unwrap(); + let acknowledged_while_uploading = early_ack.is_ok(); + let frozen = match early_ack { + Ok(result) => result.unwrap().unwrap().frozen, + Err(_) => { + tokio::time::timeout(std::time::Duration::from_secs(10), freeze) + .await + .unwrap() + .unwrap() + .unwrap() + .frozen + } + }; + assert!( + !acknowledged_while_uploading, + "freeze acknowledged before pre-persisted write committed" + ); + let hash = written.unwrap(); + assert!(frozen); + assert!(storage.inner.contains(&space, &hash, 0).await.unwrap()); + assert_eq!(reader.kv_get(&space, &path).await.unwrap().unwrap().1, hash); + } + + #[tokio::test] + async fn legacy_freeze_during_failed_upload_refunds_and_rejects_later_write_before_upload() { + use crate::storage::{memory::MemoryStaging, StoreSize}; + use futures::io::AsyncWriteExt; + let directory = tempfile::tempdir().unwrap(); + let url = format!( + "sqlite://{}?mode=rwc", + directory.path().join("native.sqlite").display() + ); + let storage = LegacyPausedStore { + fail_after_persist: true, + ..Default::default() + }; + let first = SpaceDatabase::new( + Database::connect(url.clone()).await.unwrap(), + storage.clone(), + StaticSecret::new(vec![0; 32]).unwrap(), + ) + .await + .unwrap(); + let second = SpaceDatabase::new( + Database::connect(url).await.unwrap(), + storage.clone(), + StaticSecret::new(vec![0; 32]).unwrap(), + ) + .await + .unwrap(); + let space = test_space_id("legacy-failed-upload-freeze"); + space::ActiveModel { + id: Set(SpaceIdWrap(space.clone())), + } + .insert(&first.conn) + .await + .unwrap(); + let path: Path = "xyz.tinycloud.tinychat/connectors/fireflies/transcript/failed" + .parse() + .unwrap(); + let writer_space = space.clone(); + let writer_path = path.clone(); + let writer = tokio::spawn(async move { + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(b"failed upload").await.unwrap(); + first + .invoke_internal_kv_put::( + writer_space, + writer_path, + Metadata(Default::default()), + stage, + None, + ) + .await + .map_err(|error| error.to_string()) + }); + tokio::time::timeout( + std::time::Duration::from_secs(10), + storage.entered.notified(), + ) + .await + .unwrap(); + let freeze_space = space.clone(); + let freeze_db = second.clone(); + let mut freeze = tokio::spawn(async move { + freeze_db + .set_legacy_meeting_write_freeze(&freeze_space, true, 0) + .await + }); + let early_ack = + tokio::time::timeout(std::time::Duration::from_millis(100), &mut freeze).await; + storage.release.notify_one(); + let written = tokio::time::timeout(std::time::Duration::from_secs(10), writer) + .await + .unwrap() + .unwrap(); + let acknowledged_while_uploading = early_ack.is_ok(); + let frozen = match early_ack { + Ok(result) => result.unwrap().unwrap().frozen, + Err(_) => { + tokio::time::timeout(std::time::Duration::from_secs(10), freeze) + .await + .unwrap() + .unwrap() + .unwrap() + .frozen + } + }; + assert!( + !acknowledged_while_uploading, + "freeze acknowledged while upload remained in flight" + ); + let error = written.unwrap_err(); + assert!(error.contains("lost upload response"), "{error}"); + assert!(frozen); + let hash = crate::hash::hash(b"failed upload"); + assert!(!storage.inner.contains(&space, &hash, 0).await.unwrap()); + assert_eq!(storage.inner.total_size(&space).await.unwrap(), Some(0)); + assert_eq!( + crate::kv_orphans::collect_candidates(&second.conn, &storage, 1, time::Duration::ZERO) + .await + .unwrap(), + 0, + "immediate refund consumed candidate exactly once" + ); + let mut late_stage = HashBuffer::new(Vec::new()); + late_stage.write_all(b"late").await.unwrap(); + assert!(matches!( + tokio::time::timeout( + std::time::Duration::from_secs(10), + second.invoke_internal_kv_put::( + space.clone(), + path, + Metadata(Default::default()), + late_stage, + None, + ) + ) + .await + .unwrap(), + Err(TxStoreError::LegacyMeetingFrozen) + )); + assert_eq!( + storage.uploads.load(std::sync::atomic::Ordering::SeqCst), + 1, + "a post-freeze write must be refused before storage.persist" + ); + assert_eq!(storage.inner.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn postgres_legacy_freeze_ack_during_orphan_cleanup_preserves_committed_references() { + use crate::storage::{memory::MemoryStaging, StoreSize}; + use futures::io::AsyncWriteExt; + let Some(schema) = crate::test_support::PostgresTestSchema::new( + "postgres_legacy_freeze_ack_during_orphan_cleanup_preserves_committed_references", + ) + .await + else { + return; + }; + let storage = LegacyPausedStore { + fail_after_persist: true, + pause_remove: true, + ..Default::default() + }; + let committed_db = SpaceDatabase::new( + schema.conn.clone(), + storage.inner.clone(), + StaticSecret::new(vec![0; 32]).unwrap(), + ) + .await + .unwrap(); + let writer_db = SpaceDatabase::new( + schema.conn.clone(), + storage.clone(), + StaticSecret::new(vec![0; 32]).unwrap(), + ) + .await + .unwrap(); + let freezer = SpaceDatabase::new( + schema.conn.clone(), + storage.clone(), + StaticSecret::new(vec![0; 32]).unwrap(), + ) + .await + .unwrap(); + let space = test_space_id("pg-freeze-orphan-cleanup"); + space::ActiveModel { + id: Set(SpaceIdWrap(space.clone())), + } + .insert(&schema.conn) + .await + .unwrap(); + let kept_path: Path = "xyz.tinycloud.tinychat/connectors/fireflies/transcript/kept" + .parse() + .unwrap(); + let mut kept_stage = HashBuffer::new(Vec::new()); + kept_stage.write_all(b"committed").await.unwrap(); + let kept_hash = committed_db + .invoke_internal_kv_put::( + space.clone(), + kept_path.clone(), + Metadata(Default::default()), + kept_stage, + None, + ) + .await + .map_err(|error| error.to_string()) + .unwrap(); + let before = get_kv_entity(&schema.conn, &space, &kept_path) + .await + .unwrap() + .unwrap(); + let writes_before = crate::models::kv_write::Entity::find() + .filter(crate::models::kv_write::Column::Space.eq(SpaceIdWrap(space.clone()))) + .count(&schema.conn) + .await + .unwrap(); + let orphan_path: Path = "xyz.tinycloud.tinychat/connectors/fireflies/transcript/orphan" + .parse() + .unwrap(); + let orphan_probe = orphan_path.clone(); + let orphan_hash = crate::hash::hash(b"uncommitted"); + let writer_space = space.clone(); + let mut writer = tokio::spawn(async move { + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(b"uncommitted").await.unwrap(); + writer_db + .invoke_internal_kv_put::( + writer_space, + orphan_path, + Metadata(Default::default()), + stage, + None, + ) + .await + .map_err(|error| error.to_string()) + }); + if tokio::time::timeout( + std::time::Duration::from_secs(10), + storage.entered.notified(), + ) + .await + .is_err() + { + writer.abort(); + panic!("the protected writer did not reach the upload"); + } + let freeze_space = space.clone(); + let mut freeze = tokio::spawn(async move { + freezer + .set_legacy_meeting_write_freeze(&freeze_space, true, 0) + .await + }); + let acknowledged_during_upload = + tokio::time::timeout(std::time::Duration::from_millis(100), &mut freeze) + .await + .is_ok(); + storage.release.notify_one(); + if tokio::time::timeout( + std::time::Duration::from_secs(10), + storage.removing.notified(), + ) + .await + .is_err() + { + storage.remove_release.notify_one(); + writer.abort(); + freeze.abort(); + panic!("the failed upload did not reach orphan cleanup"); + } + let ack_before_cleanup = + tokio::time::timeout(std::time::Duration::from_secs(10), &mut freeze).await; + let orphan_still_present = storage.inner.contains(&space, &orphan_hash, 0).await; + storage.remove_release.notify_one(); + let orphan_still_present = orphan_still_present.unwrap(); + let write_result = + match tokio::time::timeout(std::time::Duration::from_secs(10), &mut writer).await { + Ok(result) => result.unwrap(), + Err(_) => { + writer.abort(); + freeze.abort(); + panic!("the failed write did not finish cleanup"); + } + }; + let acknowledged_before_cleanup = ack_before_cleanup.is_ok(); + let frozen = match ack_before_cleanup { + Ok(result) => result.unwrap().unwrap().frozen, + Err(_) => { + tokio::time::timeout(std::time::Duration::from_secs(10), freeze) + .await + .unwrap() + .unwrap() + .unwrap() + .frozen + } + }; + assert!(!acknowledged_during_upload); + assert!( + acknowledged_before_cleanup, + "the DB abort must release the freeze barrier" + ); + assert!( + orphan_still_present, + "cleanup was still pending at freeze ACK" + ); + assert!(frozen); + assert!(write_result.unwrap_err().contains("lost upload response")); + let after = get_kv_entity(&schema.conn, &space, &kept_path) + .await + .unwrap() + .unwrap(); + assert_eq!((after.seq, after.value), (before.seq, before.value)); + assert_eq!(after.value, kept_hash); + assert!(get_kv_entity(&schema.conn, &space, &orphan_probe) + .await + .unwrap() + .is_none()); + assert_eq!( + crate::models::kv_write::Entity::find() + .filter(crate::models::kv_write::Column::Space.eq(SpaceIdWrap(space.clone()))) + .count(&schema.conn) + .await + .unwrap(), + writes_before, + "post-ACK cleanup must not mutate committed rows" + ); + assert!(storage.inner.contains(&space, &kept_hash, 0).await.unwrap()); + assert!(!storage + .inner + .contains(&space, &orphan_hash, 0) + .await + .unwrap()); + assert_eq!( + storage.inner.total_size(&space).await.unwrap(), + Some(b"committed".len() as u64), + "only the unreferenced candidate may be refunded" + ); + assert_eq!( + crate::kv_orphans::collect_candidates(&schema.conn, &storage, 1, time::Duration::ZERO) + .await + .unwrap(), + 0 + ); + } + #[tokio::test] async fn publication_cleanup_accepts_already_absent_legacy_keys() { use sea_orm::ActiveValue::Set; @@ -5715,6 +6936,7 @@ mod test { epoch: epoch_id, epoch_seq: 0, value: crate::hash::hash(b"shared-value"), + generation: 0, metadata: Metadata(std::collections::BTreeMap::new()), }; kv_write::ActiveModel::from(write.clone()) @@ -6282,6 +7504,7 @@ mod test { epoch: Set(crate::hash::hash(b"query-plan-epoch")), epoch_seq: Set(0), value: Set(crate::hash::hash(format!("value-{seq}").as_bytes())), + generation: Set(0), metadata: Set(Metadata(std::collections::BTreeMap::new())), }); } diff --git a/tinycloud-core/src/db_pool.rs b/tinycloud-core/src/db_pool.rs index cf1dab56..0201f074 100644 --- a/tinycloud-core/src/db_pool.rs +++ b/tinycloud-core/src/db_pool.rs @@ -260,6 +260,22 @@ impl RecoveringConnection { result } + /// Pre-upload authorization and orphan registration share one connection. + /// Reserve SQLite's database writer before authorization reads, just like + /// the other write paths: a separate collector connection must not commit + /// after our snapshot and turn registration into SQLITE_BUSY_SNAPSHOT. + /// This transaction changes only retryable candidate metadata and commits + /// before blob persistence, so its uncertain outcome cannot mean a + /// user-visible KV write committed. + pub(crate) async fn begin_preparatory(&self) -> Result { + let transaction = self + .run("begin_preparatory", true, || { + crate::db::begin_write_transaction(&self.conn, None) + }) + .await?; + self.scope_local_timeout(transaction).await + } + /// The outcome of the one retry. It is never retried again, but a /// connection lost again is still recorded. fn retried(&self, operation: &'static str, result: Result) -> Result { diff --git a/tinycloud-core/src/events/mod.rs b/tinycloud-core/src/events/mod.rs index dab7a329..03b803ff 100644 --- a/tinycloud-core/src/events/mod.rs +++ b/tinycloud-core/src/events/mod.rs @@ -56,6 +56,7 @@ pub(crate) enum Operation { space: SpaceId, key: Path, value: Hash, + generation: i64, metadata: Metadata, }, KvDelete { @@ -72,11 +73,13 @@ impl Operation { space, key, value, + generation, metadata, } => VersionedOperation::KvWrite { space, key, value, + generation, metadata, seq, epoch, @@ -111,6 +114,7 @@ pub(crate) enum VersionedOperation { space: SpaceId, key: Path, value: Hash, + generation: i64, metadata: Metadata, seq: i64, epoch: Hash, @@ -224,6 +228,7 @@ fn hash_inv(inv_hash: &Hash, sp: &SpaceId, ops: &[Operation]) -> Result Some(Op::KvWrite { key, value: value.to_cid(CBOR_CODEC), diff --git a/tinycloud-core/src/kv_orphans.rs b/tinycloud-core/src/kv_orphans.rs new file mode 100644 index 00000000..ae50f5d9 --- /dev/null +++ b/tinycloud-core/src/kv_orphans.rs @@ -0,0 +1,3239 @@ +//! Durable fences for blobs uploaded before a KV transaction's sequence lock. + +use std::{borrow::Cow, error::Error as StdError, future::Future}; + +use sea_orm::{ + sea_query::{Expr, LockBehavior, LockType, OnConflict}, + ColumnTrait, ConnectionTrait, DatabaseTransaction, DbBackend, DbErr, EntityTrait, + FromQueryResult, QueryFilter, QueryOrder, QuerySelect, Statement, TransactionTrait, +}; +use time::{Duration, OffsetDateTime}; +use tinycloud_auth::resource::SpaceId; + +use crate::hash::Hash; +use crate::models::{ + current_kv, kv_blob_generation as ledger, kv_orphan_attempt as attempt, + kv_orphan_candidate as candidate, kv_write, +}; +use crate::storage::{ImmutableDeleteStore, ImmutableReadStore}; +use crate::types::SpaceIdWrap; + +const MAX_BATCH_SIZE: u64 = 256; +/// A failed upload's claim becomes eligible after five minutes of DB time. +/// Ordinary uploads have a 240-second deadline; the server then runs GC every +/// 60 seconds. A genuinely live same-space, same-hash attempt protects an +/// older crash stage and its physical-byte charge until it commits or aborts. +/// The older claim becomes due when the last attempt ends, for the next +/// scheduled pass (within 60 seconds); each upload has at most 240 seconds. +/// A backlog beyond the collector's bounded pages may take further passes. +/// Repeated overlapping attempts can extend this conservative overcharge. +/// A referenced final blob is never deleted or refunded. A single blob key +/// costs at most its physical size; filesystem crash stages are separate +/// physical files whose charges are settled independently. +pub const ORPHAN_GRACE: Duration = Duration::minutes(5); + +const UNAVAILABLE_DELETING: &str = "KV blob is being deleted; retry registration"; +const UNAVAILABLE_MISSING: &str = "KV blob attempt lost its active fence"; + +pub(crate) fn is_candidate_unavailable(error: &DbErr) -> bool { + matches!(error, DbErr::Custom(message) if message == UNAVAILABLE_DELETING || message == UNAVAILABLE_MISSING) +} + +const UNHOSTED_SPACE: &str = "KV blob space is not hosted"; + +/// A registration that [`register_candidates_hosted`] refused because a +/// required space does not exist. Nothing was written. +pub(crate) fn is_space_not_hosted(error: &DbErr) -> bool { + matches!(error, DbErr::Custom(message) if message == UNHOSTED_SPACE) +} + +fn ordered_unique(blobs: &[(SpaceId, Hash)]) -> Cow<'_, [(SpaceId, Hash)]> { + if blobs.windows(2).all(|pair| pair[0] < pair[1]) { + return Cow::Borrowed(blobs); + } + let mut ordered = blobs.to_vec(); + ordered.sort_unstable(); + ordered.dedup(); + Cow::Owned(ordered) +} + +/// Register this invocation's durable claim in a short preparatory transaction. +/// The caller commits before uploading. Replaying the same ID is harmless. +pub async fn register_candidates( + conn: &C, + blobs: &[(SpaceId, Hash)], + attempt_id: &str, +) -> Result, DbErr> { + register_candidates_hosted(conn, blobs, attempt_id, &[]).await +} + +/// PostgreSQL registration: one statement per distinct blob, each run by +/// `query`. The ledger upsert serializes registration against deletion; a +/// deleting fence returns no row, so no caller starts an upload. One +/// statement claims its blob or nothing; several blobs commit together only +/// if `query` runs them inside one transaction. Every row is keyed by the +/// attempt ID, so a second run of a statement writes nothing new. +async fn register_candidates_postgres( + blobs: &[(SpaceId, Hash)], + attempt_id: &str, + hosted_spaces: &[SpaceId], + mut query: F, +) -> Result, DbErr> +where + F: FnMut(Statement) -> Fut, + Fut: Future, DbErr>>, +{ + let hosted_ids: Vec = hosted_spaces.iter().map(ToString::to_string).collect(); + let hosted_count = hosted_ids.len() as i64; + let mut generations = Vec::with_capacity(blobs.len()); + for (space, value) in ordered_unique(blobs).iter() { + let key = SpaceIdWrap(space.clone()); + let row = query(Statement::from_sql_and_values( + DbBackend::Postgres, + "WITH generation AS ( + INSERT INTO kv_blob_generation (space, value, generation) + SELECT $1::text, $2::bytea, 0 + WHERE $5 = 0 + OR (SELECT COUNT(*) FROM space WHERE id = ANY($4::text[])) = $5 + ON CONFLICT (space, value) DO UPDATE + SET generation = kv_blob_generation.generation + RETURNING space, value, generation + ), fence AS ( + INSERT INTO kv_orphan_candidate + (space, value, generation, created_at, deleting_at, deletion_bytes) + SELECT space, value, generation, CURRENT_TIMESTAMP, NULL, NULL + FROM generation WHERE true + ON CONFLICT (space, value) DO UPDATE + SET created_at = EXCLUDED.created_at + WHERE kv_orphan_candidate.deleting_at IS NULL + AND kv_orphan_candidate.generation = EXCLUDED.generation + RETURNING space, value, generation + ), claim AS ( + INSERT INTO kv_orphan_attempt (space, value, attempt_id, generation, created_at) + SELECT space, value, $3, generation, CURRENT_TIMESTAMP FROM fence WHERE true + ON CONFLICT (space, value, attempt_id) DO NOTHING + RETURNING attempt_id + ) + SELECT (SELECT generation FROM fence) AS generation, + (SELECT COUNT(*) FROM claim) AS inserted, + ($5 = 0 + OR (SELECT COUNT(*) FROM space WHERE id = ANY($4::text[])) = $5) AS hosted", + [ + key.into(), + (*value).into(), + attempt_id.to_owned().into(), + hosted_ids.clone().into(), + hosted_count.into(), + ], + )) + .await? + .ok_or_else(|| DbErr::Custom("KV candidate registration returned no row".into()))?; + // The statement wrote nothing when a required space is unhosted. + if !row.try_get::("", "hosted")? { + return Err(DbErr::Custom(UNHOSTED_SPACE.into())); + } + let generation: Option = row.try_get("", "generation")?; + generations.push(( + space.clone(), + *value, + generation.ok_or_else(|| DbErr::Custom(UNAVAILABLE_DELETING.into()))?, + )); + } + Ok(generations) +} + +/// [`register_candidates`] that, on PostgreSQL, also checks hosting (TC-735). +/// With `hosted_spaces` non-empty, every registration statement writes nothing +/// unless each listed space exists, and reports [`is_space_not_hosted`] +/// otherwise, so an unhosted space is refused before any bytes reach the store +/// without a separate COUNT round trip. The caller runs the claims of one +/// invocation in one transaction, so they commit together or not at all. Other +/// backends ignore `hosted_spaces`: their callers count hosted spaces +/// themselves. +pub async fn register_candidates_hosted( + conn: &C, + blobs: &[(SpaceId, Hash)], + attempt_id: &str, + hosted_spaces: &[SpaceId], +) -> Result, DbErr> { + if conn.get_database_backend() == DbBackend::Postgres { + return register_candidates_postgres(blobs, attempt_id, hosted_spaces, |statement| { + conn.query_one(statement) + }) + .await; + } + let mut registered_at = None; + let mut generations = Vec::with_capacity(blobs.len()); + for (space, value) in ordered_unique(blobs).iter() { + let key = SpaceIdWrap(space.clone()); + if conn.get_database_backend() == DbBackend::MySql { + conn.execute(Statement::from_sql_and_values( + DbBackend::MySql, + "INSERT INTO kv_blob_generation (space, value, generation) VALUES (?, ?, 0) + ON DUPLICATE KEY UPDATE generation = generation", + [key.clone().into(), (*value).into()], + )) + .await?; + } else { + let inserted = ledger::Entity::insert(ledger::ActiveModel { + space: sea_orm::ActiveValue::Set(key.clone()), + value: sea_orm::ActiveValue::Set(*value), + generation: sea_orm::ActiveValue::Set(0), + }) + .on_conflict( + OnConflict::columns([ledger::Column::Space, ledger::Column::Value]) + .do_nothing() + .to_owned(), + ) + .exec(conn) + .await; + match inserted { + Ok(_) | Err(DbErr::RecordNotInserted) => {} + Err(error) => return Err(error), + } + } + let generation = if conn.get_database_backend() == DbBackend::MySql { + ledger::Entity::find_by_id((key.clone(), *value)) + .lock_exclusive() + .one(conn) + .await? + } else { + ledger::Entity::find_by_id((key.clone(), *value)) + .one(conn) + .await? + } + .ok_or_else(|| DbErr::Custom("KV generation ledger missing".into()))? + .generation; + if conn.get_database_backend() == DbBackend::MySql { + conn.execute(Statement::from_sql_and_values( + DbBackend::MySql, + "INSERT INTO kv_orphan_candidate + (space, value, generation, created_at, deleting_at, deletion_bytes) + VALUES (?, ?, ?, UTC_TIMESTAMP(6), NULL, NULL) + ON DUPLICATE KEY UPDATE created_at = created_at", + [key.clone().into(), (*value).into(), generation.into()], + )) + .await?; + } else { + let inserted = candidate::Entity::insert(candidate::ActiveModel { + space: sea_orm::ActiveValue::Set(key.clone()), + value: sea_orm::ActiveValue::Set(*value), + generation: sea_orm::ActiveValue::Set(generation), + created_at: sea_orm::ActiveValue::Set(OffsetDateTime::now_utc()), + deleting_at: sea_orm::ActiveValue::Set(None), + deletion_bytes: sea_orm::ActiveValue::Set(None), + }) + .on_conflict( + OnConflict::columns([candidate::Column::Space, candidate::Column::Value]) + .do_nothing() + .to_owned(), + ) + .exec(conn) + .await; + match inserted { + Ok(_) | Err(DbErr::RecordNotInserted) => {} + Err(error) => return Err(error), + } + } + let now = match registered_at { + Some(now) => now, + None => { + let now = database_now(conn).await?; + registered_at = Some(now); + now + } + }; + let active = if conn.get_database_backend() == DbBackend::Sqlite { + candidate::Entity::update_many() + .col_expr(candidate::Column::CreatedAt, Expr::value(now)) + .filter(candidate::Column::Space.eq(key.clone())) + .filter(candidate::Column::Value.eq(*value)) + .filter(candidate::Column::Generation.eq(generation)) + .filter(candidate::Column::DeletingAt.is_null()) + .exec(conn) + .await? + .rows_affected + == 1 + } else { + let row = candidate::Entity::find_by_id((key.clone(), *value)) + .lock_exclusive() + .one(conn) + .await?; + if row.is_some_and(|row| row.deleting_at.is_none() && row.generation == generation) { + candidate::Entity::update_many() + .col_expr(candidate::Column::CreatedAt, Expr::value(now)) + .filter(candidate::Column::Space.eq(key.clone())) + .filter(candidate::Column::Value.eq(*value)) + .exec(conn) + .await?; + true + } else { + false + } + }; + if !active { + return Err(DbErr::Custom(UNAVAILABLE_DELETING.into())); + } + if conn.get_database_backend() == DbBackend::MySql { + conn.execute(Statement::from_sql_and_values( + DbBackend::MySql, + "INSERT INTO kv_orphan_attempt (space, value, attempt_id, generation, created_at) + VALUES (?, ?, ?, ?, ?) + ON DUPLICATE KEY UPDATE attempt_id = attempt_id", + [ + key.into(), + (*value).into(), + attempt_id.to_owned().into(), + generation.into(), + now.into(), + ], + )) + .await?; + } else { + let inserted = attempt::Entity::insert(attempt::ActiveModel { + space: sea_orm::ActiveValue::Set(key), + value: sea_orm::ActiveValue::Set(*value), + attempt_id: sea_orm::ActiveValue::Set(attempt_id.to_owned()), + generation: sea_orm::ActiveValue::Set(generation), + created_at: sea_orm::ActiveValue::Set(now), + retry_after: sea_orm::ActiveValue::Set(None), + }) + .on_conflict( + OnConflict::columns([ + attempt::Column::Space, + attempt::Column::Value, + attempt::Column::AttemptId, + ]) + .do_nothing() + .to_owned(), + ) + .exec(conn) + .await; + match inserted { + Ok(_) | Err(DbErr::RecordNotInserted) => {} + Err(error) => return Err(error), + } + } + generations.push((space.clone(), *value, generation)); + } + Ok(generations) +} + +/// Validate ownership inside the final KV transaction; never create a missing +/// fence here. The caller already finished ordinary storage I/O. +pub async fn lock_candidates( + tx: &DatabaseTransaction, + blobs: &[(SpaceId, Hash, i64)], + attempt_id: &str, +) -> Result<(), DbErr> { + for (space, value, generation) in blobs { + let key = SpaceIdWrap(space.clone()); + if tx.get_database_backend() == DbBackend::Postgres { + // The joined row lock covers both the active fence and this exact + // durable attempt. GC removes attempts only after taking that + // same fence lock, so neither row can disappear before commit. + let owned = tx + .query_one(Statement::from_sql_and_values( + DbBackend::Postgres, + "SELECT 1 AS owned FROM kv_orphan_candidate AS fence + JOIN kv_orphan_attempt AS claim + ON claim.space = fence.space AND claim.value = fence.value + AND claim.generation = fence.generation + WHERE fence.space = $1 AND fence.value = $2 + AND fence.generation = $4 + AND fence.deleting_at IS NULL AND claim.attempt_id = $3 + FOR UPDATE OF fence", + [ + key.into(), + (*value).into(), + attempt_id.to_owned().into(), + (*generation).into(), + ], + )) + .await? + .is_some(); + if !owned { + return Err(DbErr::Custom(UNAVAILABLE_MISSING.into())); + } + continue; + } + let active = if tx.get_database_backend() == DbBackend::Sqlite { + candidate::Entity::update_many() + .col_expr( + candidate::Column::CreatedAt, + Expr::col(candidate::Column::CreatedAt).into(), + ) + .filter(candidate::Column::Space.eq(key.clone())) + .filter(candidate::Column::Value.eq(*value)) + .filter(candidate::Column::Generation.eq(*generation)) + .filter(candidate::Column::DeletingAt.is_null()) + .exec(tx) + .await? + .rows_affected + == 1 + } else { + candidate::Entity::find_by_id((key.clone(), *value)) + .lock_exclusive() + .one(tx) + .await? + .is_some_and(|row| row.deleting_at.is_none() && row.generation == *generation) + }; + if !active { + return Err(DbErr::Custom(UNAVAILABLE_MISSING.into())); + } + let own = attempt::Entity::find_by_id((key, *value, attempt_id.to_owned())) + .filter(attempt::Column::Generation.eq(*generation)); + // MySQL repeatable-read can hold a snapshot, taken during KV authorization, + // from before GC removed attempts. A locking read observes the current rows. + let own = if tx.get_database_backend() == DbBackend::MySql { + own.lock_exclusive() + } else { + own + }; + if own.one(tx).await?.is_none() { + return Err(DbErr::Custom(UNAVAILABLE_MISSING.into())); + } + } + Ok(()) +} + +/// Remove only this invocation's claims in the same transaction as its KV refs. +pub async fn finish_candidates( + tx: &DatabaseTransaction, + blobs: &[(SpaceId, Hash, i64)], + attempt_id: &str, +) -> Result<(), DbErr> { + for (space, value, generation) in blobs { + let key = SpaceIdWrap(space.clone()); + if tx.get_database_backend() == DbBackend::Postgres { + // This is only the *successful* KV commit path: transact already + // wrote the reference in this transaction. Its earlier + // lock_candidates call holds this fence; `locked` also makes the + // dependency explicit for the single statement. PostgreSQL CTEs + // share a snapshot, so exclude our own still-visible attempt + // when deciding whether another writer remains. + let row = tx + .query_one(Statement::from_sql_and_values( + DbBackend::Postgres, + "WITH locked AS MATERIALIZED ( + SELECT space, value FROM kv_orphan_candidate + WHERE space = $1 AND value = $2 AND generation = $4 + AND deleting_at IS NULL + FOR UPDATE + ), refs AS MATERIALIZED ( + SELECT EXISTS( + SELECT 1 FROM kv_write WHERE space = $1 AND value = $2 AND generation = $4 + ) OR EXISTS( + SELECT 1 FROM current_kv WHERE space = $1 AND value = $2 AND generation = $4 + ) AS present + ), removed AS ( + DELETE FROM kv_orphan_attempt + WHERE space = $1 AND value = $2 AND attempt_id = $3 + AND generation = $4 + AND EXISTS(SELECT 1 FROM locked) + AND (SELECT present FROM refs) + RETURNING space, value + ), woken AS ( + UPDATE kv_orphan_attempt AS other + SET retry_after = statement_timestamp() + WHERE other.space = $1 AND other.value = $2 + AND other.generation = $4 AND other.attempt_id <> $3 + AND other.retry_after IS NOT NULL + AND EXISTS(SELECT 1 FROM removed) + RETURNING 1 + ), retired AS ( + DELETE FROM kv_orphan_candidate AS fence + USING removed + WHERE fence.space = removed.space + AND fence.value = removed.value + AND fence.generation = $4 + AND NOT EXISTS( + SELECT 1 FROM kv_orphan_attempt AS other + WHERE other.space = fence.space + AND other.value = fence.value + AND other.generation = $4 + AND other.attempt_id <> $3 + ) + RETURNING 1 + ) + SELECT (SELECT COUNT(*) FROM removed) AS removed, + (SELECT present FROM refs) AS referenced, + (SELECT COUNT(*) FROM woken) AS woken, + (SELECT COUNT(*) FROM retired) AS retired", + [key.into(), (*value).into(), attempt_id.to_owned().into(), (*generation).into()], + )) + .await? + .ok_or_else(|| DbErr::Custom("KV candidate finish returned no row".into()))?; + if !row.try_get::("", "referenced")? { + return Err(DbErr::Custom("KV blob has no committed reference".into())); + } + if row.try_get::("", "removed")? != 1 { + return Err(DbErr::Custom( + "KV blob attempt disappeared before commit".into(), + )); + } + continue; + } + let removed = attempt::Entity::delete_many() + .filter(attempt::Column::Space.eq(key.clone())) + .filter(attempt::Column::Value.eq(*value)) + .filter(attempt::Column::AttemptId.eq(attempt_id)) + .filter(attempt::Column::Generation.eq(*generation)) + .exec(tx) + .await?; + if removed.rows_affected != 1 { + return Err(DbErr::Custom( + "KV blob attempt disappeared before commit".into(), + )); + } + if has_attempts(tx, &key, *value, *generation).await? { + wake_attempts(tx, &key, *value, *generation, attempt_id).await?; + } else if has_references(tx, &key, *value, *generation).await? { + candidate::Entity::delete_many() + .filter(candidate::Column::Space.eq(key)) + .filter(candidate::Column::Value.eq(*value)) + .filter(candidate::Column::Generation.eq(*generation)) + .exec(tx) + .await?; + } + } + Ok(()) +} + +/// A writer leaving makes deferred stage claims eligible immediately +/// without restoring their first-attempt priority. Keep them in the retry +/// queue at the database's current time so an unrelated orphan cannot be +/// starved by repeated departures. Registration age remains immutable. +async fn wake_attempts( + conn: &C, + space: &SpaceIdWrap, + value: Hash, + generation: i64, + departing_attempt: &str, +) -> Result<(), DbErr> { + attempt::Entity::update_many() + .col_expr(attempt::Column::RetryAfter, Expr::cust("CURRENT_TIMESTAMP")) + .filter(attempt::Column::Space.eq(space.clone())) + .filter(attempt::Column::Value.eq(value)) + .filter(attempt::Column::Generation.eq(generation)) + .filter(attempt::Column::AttemptId.ne(departing_attempt)) + .filter(attempt::Column::RetryAfter.is_not_null()) + .exec(conn) + .await?; + Ok(()) +} + +#[derive(Debug, thiserror::Error)] +pub enum CollectError { + #[error("orphan candidate database operation failed: {0}")] + Database(#[from] DbErr), + #[error("orphan blob metadata read failed: {0}")] + Read(String), + #[error("orphan stage cleanup timed out")] + StageTimeout, + #[error("orphan blob deletion failed: {0}")] + Store(E), +} + +async fn has_attempts( + conn: &C, + space: &SpaceIdWrap, + value: Hash, + generation: i64, +) -> Result { + let query = attempt::Entity::find() + .filter(attempt::Column::Space.eq(space.clone())) + .filter(attempt::Column::Value.eq(value)) + .filter(attempt::Column::Generation.eq(generation)); + let query = if conn.get_database_backend() == DbBackend::MySql { + query.lock_exclusive() + } else { + query + }; + Ok(query.one(conn).await?.is_some()) +} + +async fn has_references( + conn: &C, + space: &SpaceIdWrap, + value: Hash, + generation: i64, +) -> Result { + let historical = kv_write::Entity::find() + .filter(kv_write::Column::Space.eq(space.clone())) + .filter(kv_write::Column::Value.eq(value)) + .filter(kv_write::Column::Generation.eq(generation)) + .select_only() + .column(kv_write::Column::Seq); + let current = current_kv::Entity::find() + .filter(current_kv::Column::Space.eq(space.clone())) + .filter(current_kv::Column::Value.eq(value)) + .filter(current_kv::Column::Generation.eq(generation)) + .select_only() + .column(current_kv::Column::Seq); + // MySQL's repeatable-read snapshot could precede the writer whose fence + // we just acquired. Locking reads see its committed references. + let (historical, current) = if conn.get_database_backend() == DbBackend::MySql { + (historical.lock_exclusive(), current.lock_exclusive()) + } else { + (historical, current) + }; + Ok(historical.into_tuple::().one(conn).await?.is_some() + || current.into_tuple::().one(conn).await?.is_some()) +} + +async fn database_now(conn: &C) -> Result { + let sql = match conn.get_database_backend() { + DbBackend::Sqlite => "SELECT CAST(strftime('%s','now') AS INTEGER) AS now_epoch", + DbBackend::Postgres => { + "SELECT CAST(EXTRACT(EPOCH FROM CURRENT_TIMESTAMP) AS BIGINT) AS now_epoch" + } + DbBackend::MySql => "SELECT CAST(UNIX_TIMESTAMP(CURRENT_TIMESTAMP) AS SIGNED) AS now_epoch", + }; + let row = conn + .query_one(Statement::from_string(conn.get_database_backend(), sql)) + .await? + .ok_or_else(|| DbErr::Custom("database clock unavailable".into()))?; + let epoch: i64 = row.try_get("", "now_epoch")?; + OffsetDateTime::from_unix_timestamp(epoch).map_err(|e| DbErr::Custom(e.to_string())) +} + +async fn lock_fence( + tx: &DatabaseTransaction, + space: &SpaceIdWrap, + value: Hash, + generation: i64, + skip_locked: bool, +) -> Result, DbErr> { + if tx.get_database_backend() == DbBackend::Sqlite { + let updated = candidate::Entity::update_many() + .col_expr( + candidate::Column::CreatedAt, + Expr::col(candidate::Column::CreatedAt).into(), + ) + .filter(candidate::Column::Space.eq(space.clone())) + .filter(candidate::Column::Value.eq(value)) + .filter(candidate::Column::Generation.eq(generation)) + .exec(tx) + .await?; + if updated.rows_affected == 0 { + return Ok(None); + } + candidate::Entity::find_by_id((space.clone(), value)) + .filter(candidate::Column::Generation.eq(generation)) + .one(tx) + .await + } else { + let query = candidate::Entity::find_by_id((space.clone(), value)) + .filter(candidate::Column::Generation.eq(generation)); + let query = if skip_locked && tx.get_database_backend() == DbBackend::Postgres { + query.lock_with_behavior(LockType::Update, LockBehavior::SkipLocked) + } else { + query.lock_exclusive() + }; + query.one(tx).await + } +} + +/// Retire a failed attempt or an expired GC group under the hash fence. GC +/// keeps an expired cleanup claim only while a genuinely live sibling exists. +async fn prepare_deletion( + conn: &C, + store: &S, + space: &SpaceId, + value: Hash, + generation: i64, + attempt_id: &str, + gc_cutoff: Option, +) -> Result::Error>> +where + C: ConnectionTrait + TransactionTrait, + S: ImmutableDeleteStore + ImmutableReadStore, +{ + let bytes = store + .read(space, &value, generation) + .await + .map_err(|error| CollectError::Read(error.to_string()))? + .map_or(0, |content| content.len()); + let bytes = + i64::try_from(bytes).map_err(|_| DbErr::Custom("orphan blob size exceeds i64".into()))?; + let now = database_now(conn).await?; + let tx = conn.begin().await?; + let key = SpaceIdWrap(space.clone()); + let Some(fence) = lock_fence(&tx, &key, value, generation, gc_cutoff.is_some()).await? else { + tx.commit().await?; + return Ok(false); + }; + if fence.deleting_at.is_some() { + tx.commit().await?; + return Ok(false); + } + let mut attempts = attempt::Entity::delete_many() + .filter(attempt::Column::Space.eq(key.clone())) + .filter(attempt::Column::Value.eq(value)) + .filter(attempt::Column::Generation.eq(generation)); + if let Some(cutoff) = gc_cutoff { + // The fence excludes new registrations. A live sibling retains this + // expired claim's stage obligation, but its retry eligibility must be + // separate from immutable upload age: other collectors must never + // mistake a deferred crashed attempt for a new live writer. + let live = attempt::Entity::find() + .filter(attempt::Column::Space.eq(key.clone())) + .filter(attempt::Column::Value.eq(value)) + .filter(attempt::Column::Generation.eq(generation)) + .filter(attempt::Column::AttemptId.ne(attempt_id)) + .filter(attempt::Column::CreatedAt.gt(cutoff)) + .one(&tx) + .await? + .is_some(); + if live { + attempt::Entity::update_many() + .col_expr( + attempt::Column::RetryAfter, + Expr::value(now + Duration::seconds(59)), + ) + .filter(attempt::Column::Space.eq(key)) + .filter(attempt::Column::Value.eq(value)) + .filter(attempt::Column::Generation.eq(generation)) + .filter(attempt::Column::AttemptId.eq(attempt_id)) + .filter(attempt::Column::CreatedAt.lte(cutoff)) + .exec(&tx) + .await?; + tx.commit().await?; + return Ok(false); + } + // All siblings are expired. Drain them together under this fence; + // deleting one at a time would make two crashed uploads immortal. + attempts = attempts.filter(attempt::Column::CreatedAt.lte(cutoff)); + } else { + attempts = attempts.filter(attempt::Column::AttemptId.eq(attempt_id)); + } + let removed = attempts.exec(&tx).await?; + if removed.rows_affected == 0 { + tx.commit().await?; + return Ok(false); + } + if gc_cutoff.is_none() && has_attempts(&tx, &key, value, generation).await? { + wake_attempts(&tx, &key, value, generation, attempt_id).await?; + tx.commit().await?; + return Ok(false); + } + if has_references(&tx, &key, value, generation).await? { + // This is a rare local-filesystem cleanup, not a blob upload. Keep + // the fence locked until only incomplete stages are gone, then retire + // the candidate. A timeout or cancellation rolls this transaction + // back, preserving its attempt for the next GC pass. + tokio::time::timeout( + std::time::Duration::from_secs(5), + store.cleanup_stages(space, &value, generation), + ) + .await + .map_err(|_| CollectError::StageTimeout)? + .map_err(CollectError::Store)?; + candidate::Entity::delete_many() + .filter(candidate::Column::Space.eq(key)) + .filter(candidate::Column::Value.eq(value)) + .filter(candidate::Column::Generation.eq(generation)) + .exec(&tx) + .await?; + tx.commit().await?; + return Ok(false); + } + candidate::Entity::update_many() + .col_expr(candidate::Column::DeletingAt, Expr::value(now)) + .col_expr(candidate::Column::DeletionBytes, Expr::value(bytes)) + .filter(candidate::Column::Space.eq(key)) + .filter(candidate::Column::Value.eq(value)) + .filter(candidate::Column::Generation.eq(generation)) + .exec(&tx) + .await?; + tx.commit().await?; + Ok(true) +} + +/// The ledger row of `(space, value)`, locked for the transaction. +async fn lock_ledger( + tx: &DatabaseTransaction, + key: &SpaceIdWrap, + value: Hash, +) -> Result, DbErr> { + if tx.get_database_backend() == DbBackend::Sqlite { + ledger::Entity::update_many() + .col_expr( + ledger::Column::Generation, + Expr::col(ledger::Column::Generation).into(), + ) + .filter(ledger::Column::Space.eq(key.clone())) + .filter(ledger::Column::Value.eq(value)) + .exec(tx) + .await?; + ledger::Entity::find_by_id((key.clone(), value)) + .one(tx) + .await + } else { + ledger::Entity::find_by_id((key.clone(), value)) + .lock_exclusive() + .one(tx) + .await + } +} + +/// Close a deleting fence whose generation something still references, +/// without removing the bytes. A fence left behind while another process +/// committed a reference to the same `(space, content, generation)` must not +/// delete the object that reference reads. The fence is retired, the ledger +/// generation is kept, and a fixed event is logged. Returns whether it did. +async fn retire_if_referenced( + conn: &C, + space: &SpaceId, + value: Hash, + generation: i64, +) -> Result { + let tx = conn.begin().await?; + let key = SpaceIdWrap(space.clone()); + let current = lock_ledger(&tx, &key, value).await?; + let referenced = if current.is_some_and(|row| row.generation == generation) + && lock_fence(&tx, &key, value, generation, false) + .await? + .is_some_and(|fence| fence.deleting_at.is_some()) + { + has_references(&tx, &key, value, generation).await? + } else { + false + }; + if referenced { + candidate::Entity::delete_many() + .filter(candidate::Column::Space.eq(key)) + .filter(candidate::Column::Value.eq(value)) + .filter(candidate::Column::Generation.eq(generation)) + .filter(candidate::Column::DeletingAt.is_not_null()) + .exec(&tx) + .await?; + } + tx.commit().await?; + if referenced { + tracing::warn!( + event = "kv_blob_deletion_skipped_referenced", + "a deleting fence named a blob that a committed write references; it was closed without deleting the blob" + ); + } + Ok(referenced) +} + +/// Complete a committed deleting fence, with no DB transaction or checked-out +/// DB connection during the object-store call. A lost response leaves it for GC. +async fn complete_deletion( + conn: &C, + store: &S, + space: &SpaceId, + value: Hash, + generation: i64, +) -> Result::Error>> +where + C: ConnectionTrait + TransactionTrait, + S: ImmutableDeleteStore, +{ + if retire_if_referenced(conn, space, value, generation).await? { + return Ok(false); + } + let removed = store + .remove(space, &value, generation) + .await + .map_err(CollectError::Store)? + .is_some(); + // Registration locks ledger before fence; use that same order to retire + // exactly the deleting generation and allocate the next storage key. + let tx = conn.begin().await?; + let key = SpaceIdWrap(space.clone()); + let current = lock_ledger(&tx, &key, value).await?; + if current.is_some_and(|row| row.generation == generation) { + let retired = candidate::Entity::delete_many() + .filter(candidate::Column::Space.eq(key.clone())) + .filter(candidate::Column::Value.eq(value)) + .filter(candidate::Column::Generation.eq(generation)) + .filter(candidate::Column::DeletingAt.is_not_null()) + .exec(&tx) + .await?; + if retired.rows_affected == 1 { + ledger::Entity::update_many() + .col_expr( + ledger::Column::Generation, + Expr::value( + generation + .checked_add(1) + .ok_or_else(|| DbErr::Custom("KV blob generation exhausted".into()))?, + ), + ) + .filter(ledger::Column::Space.eq(key)) + .filter(ledger::Column::Value.eq(value)) + .filter(ledger::Column::Generation.eq(generation)) + .exec(&tx) + .await?; + } + } + tx.commit().await?; + Ok(removed) +} + +/// Pending refunds used while reconstructing object-store quota after restart. +/// The storage listing must intersect these keys with objects it actually saw. +pub async fn deleting_fences( + conn: &C, +) -> Result, DbErr> { + Ok(candidate::Entity::find() + .filter(candidate::Column::DeletingAt.is_not_null()) + .order_by_asc(candidate::Column::Space) + .order_by_asc(candidate::Column::Value) + .all(conn) + .await? + .into_iter() + .map(|row| (row.space.0, row.value, row.generation)) + .collect()) +} + +async fn defer_deletion( + conn: &C, + space: &SpaceId, + value: Hash, + generation: i64, +) -> Result<(), DbErr> { + let retry_at = database_now(conn).await? + Duration::seconds(59); + candidate::Entity::update_many() + .col_expr(candidate::Column::DeletingAt, Expr::value(retry_at)) + .filter(candidate::Column::Space.eq(SpaceIdWrap(space.clone()))) + .filter(candidate::Column::Value.eq(value)) + .filter(candidate::Column::Generation.eq(generation)) + .filter(candidate::Column::DeletingAt.is_not_null()) + .exec(conn) + .await?; + Ok(()) +} + +async fn defer_attempt( + conn: &C, + space: &SpaceId, + value: Hash, + generation: i64, + attempt_id: &str, +) -> Result<(), DbErr> { + attempt::Entity::update_many() + .col_expr( + attempt::Column::RetryAfter, + Expr::value(database_now(conn).await? + Duration::seconds(59)), + ) + .filter(attempt::Column::Space.eq(SpaceIdWrap(space.clone()))) + .filter(attempt::Column::Value.eq(value)) + .filter(attempt::Column::Generation.eq(generation)) + .filter(attempt::Column::AttemptId.eq(attempt_id)) + .exec(conn) + .await?; + Ok(()) +} +pub async fn refund_failed( + conn: &C, + store: &S, + blobs: &[(SpaceId, Hash, i64)], + attempt_id: &str, +) -> Result<(), CollectError<::Error>> +where + C: ConnectionTrait + TransactionTrait, + S: ImmutableDeleteStore + ImmutableReadStore, +{ + for (space, value, generation) in blobs { + if prepare_deletion(conn, store, space, *value, *generation, attempt_id, None).await? { + complete_deletion(conn, store, space, *value, *generation).await?; + } + } + Ok(()) +} + +pub async fn collect_candidates( + conn: &C, + store: &S, + max: u64, + min_age: Duration, +) -> Result::Error>> +where + C: ConnectionTrait + TransactionTrait, + S: ImmutableDeleteStore + ImmutableReadStore, +{ + Ok(collect_candidates_with_progress(conn, store, max, min_age) + .await? + .0) +} + +/// Reclaim expired attempts and retry committed deleting fences. Every +/// candidate is handled in a separate short transaction; slow S3 never holds +/// a fence or database connection. +pub async fn collect_candidates_with_progress( + conn: &C, + store: &S, + max: u64, + min_age: Duration, +) -> Result<(usize, usize), CollectError<::Error>> +where + C: ConnectionTrait + TransactionTrait, + S: ImmutableDeleteStore + ImmutableReadStore, +{ + let limit = max.min(MAX_BATCH_SIZE); + if limit == 0 { + return Ok((0, 0)); + } + let now = database_now(conn).await?; + let cutoff = now + .checked_sub(min_age) + .unwrap_or(OffsetDateTime::UNIX_EPOCH); + let mut deleting = candidate::Entity::find() + .filter(candidate::Column::DeletingAt.is_not_null()) + .filter(candidate::Column::DeletingAt.lte(now)) + .order_by_asc(candidate::Column::DeletingAt) + .order_by_asc(candidate::Column::Space) + .order_by_asc(candidate::Column::Value) + .limit(limit) + .all(conn) + .await?; + // A full repeatedly failing DELETE page must not exclude independent + // expired uploads. Give each queue half a page if both have due work. + let available = if deleting.len() as u64 == limit { + limit.div_ceil(2) + } else { + limit - deleting.len() as u64 + }; + let attempts = if available == 0 { + Vec::new() + } else { + // Query the two retry states separately so each bounded page can use + // its age or retry index. Re-reading an oldest failing page every + // minute must not hide an unrelated, never-retried orphan. + // PostgreSQL must exclude held hash fences *before* LIMIT. An ordered + // attempt index scan with a per-row LATERAL fence lock does that while + // keeping the planner off a full hash join/sort of every candidate. + // The retry index also serves NULL retry_after: its leading key is + // constant in that queue. Commit selection before any storage I/O; + // one held hash cannot occupy a page or block an unrelated orphan. + let (mut never_retried, due_retries) = if conn.get_database_backend() == DbBackend::Postgres + { + let tx = conn.begin().await?; + let first = tx + .query_all(Statement::from_sql_and_values( + DbBackend::Postgres, + "SELECT a.* FROM ( + SELECT * FROM kv_orphan_attempt + WHERE created_at <= $1 AND retry_after IS NULL + ORDER BY retry_after, created_at, space, value + ) AS a + CROSS JOIN LATERAL ( + SELECT 1 FROM kv_orphan_candidate AS fence + WHERE fence.space = a.space AND fence.value = a.value + AND fence.generation = a.generation + FOR UPDATE SKIP LOCKED + ) AS available + LIMIT $2", + [cutoff.into(), (available as i64).into()], + )) + .await?; + let second = tx + .query_all(Statement::from_sql_and_values( + DbBackend::Postgres, + "SELECT a.* FROM ( + SELECT * FROM kv_orphan_attempt + WHERE created_at <= $1 AND retry_after <= $2 + ORDER BY retry_after, created_at, space, value + ) AS a + CROSS JOIN LATERAL ( + SELECT 1 FROM kv_orphan_candidate AS fence + WHERE fence.space = a.space AND fence.value = a.value + AND fence.generation = a.generation + FOR UPDATE SKIP LOCKED + ) AS available + LIMIT $3", + [cutoff.into(), now.into(), (available as i64).into()], + )) + .await?; + tx.commit().await?; + ( + first + .iter() + .map(|row| attempt::Model::from_query_result(row, "")) + .collect::, _>>()?, + second + .iter() + .map(|row| attempt::Model::from_query_result(row, "")) + .collect::, _>>()?, + ) + } else { + let never_retried = attempt::Entity::find() + .filter(attempt::Column::CreatedAt.lte(cutoff)) + .filter(attempt::Column::RetryAfter.is_null()) + .order_by_asc(attempt::Column::CreatedAt) + .order_by_asc(attempt::Column::Space) + .order_by_asc(attempt::Column::Value) + .limit(available) + .all(conn) + .await?; + let due_retries = attempt::Entity::find() + .filter(attempt::Column::CreatedAt.lte(cutoff)) + .filter(attempt::Column::RetryAfter.lte(now)) + .order_by_asc(attempt::Column::RetryAfter) + .order_by_asc(attempt::Column::CreatedAt) + .order_by_asc(attempt::Column::Space) + .order_by_asc(attempt::Column::Value) + .limit(available) + .all(conn) + .await?; + (never_retried, due_retries) + }; + never_retried.extend(due_retries); + never_retried.sort_unstable_by(|left, right| { + left.retry_after + .unwrap_or(left.created_at) + .cmp(&right.retry_after.unwrap_or(right.created_at)) + .then_with(|| left.created_at.cmp(&right.created_at)) + .then_with(|| left.space.cmp(&right.space)) + .then_with(|| left.value.cmp(&right.value)) + .then_with(|| left.attempt_id.cmp(&right.attempt_id)) + }); + never_retried.truncate(available as usize); + never_retried + }; + deleting.truncate((limit as usize).saturating_sub(attempts.len())); + let scanned = deleting.len() + attempts.len(); + let mut removed = 0; + let mut first_error = None; + for row in deleting { + match complete_deletion(conn, store, &row.space.0, row.value, row.generation).await { + Ok(did_remove) => removed += usize::from(did_remove), + Err(error) => { + defer_deletion(conn, &row.space.0, row.value, row.generation).await?; + first_error.get_or_insert(error); + } + } + } + for row in attempts { + match prepare_deletion( + conn, + store, + &row.space.0, + row.value, + row.generation, + &row.attempt_id, + Some(cutoff), + ) + .await + { + Ok(true) => { + match complete_deletion(conn, store, &row.space.0, row.value, row.generation).await + { + Ok(did_remove) => removed += usize::from(did_remove), + Err(error) => { + defer_deletion(conn, &row.space.0, row.value, row.generation).await?; + first_error.get_or_insert(error); + } + } + } + Ok(false) => {} + Err(error) => { + defer_attempt( + conn, + &row.space.0, + row.value, + row.generation, + &row.attempt_id, + ) + .await?; + first_error.get_or_insert(error); + } + } + } + if let Some(error) = first_error { + return Err(error); + } + Ok((removed, scanned)) +} + +/// Drain due work for one scheduled collector tick. A full page can contain +/// only temporarily retained claims; never spin indefinitely on it or keep a +/// database connection busy while starving later candidates. +pub async fn drain_candidates_for_interval( + conn: &C, + store: &S, + min_age: Duration, +) -> Result::Error>> +where + C: ConnectionTrait + TransactionTrait, + S: ImmutableDeleteStore + ImmutableReadStore, +{ + const PAGE_SIZE: usize = 256; + const MAX_PAGES_PER_INTERVAL: usize = 4; + let mut removed = 0; + for _ in 0..MAX_PAGES_PER_INTERVAL { + let (deleted, scanned) = + collect_candidates_with_progress(conn, store, PAGE_SIZE as u64, min_age).await?; + removed += deleted; + if scanned < PAGE_SIZE { + break; + } + } + Ok(removed) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::hash::hash; + use crate::models::{ + actor, epoch, invocation, kv_orphan_attempt, kv_orphan_candidate, space as space_model, + }; + use crate::relationships::event_order; + use crate::storage::{ + memory::MemoryStore, HashBuffer, ImmutableReadStore, ImmutableWriteStore, StoreSize, + }; + use futures::io::AsyncWriteExt; + use sea_orm::{ + sea_query::Expr, ConnectOptions, ConnectionTrait, Database, DatabaseConnection, DbBackend, + EntityTrait, QueryFilter, Statement, TransactionTrait, + }; + use sea_orm_migration::MigratorTrait; + use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, + }; + use tinycloud_auth::{resolver::DID_METHODS, ssi::jwk::JWK}; + use tokio::sync::Notify; + + #[derive(Clone)] + struct InterruptedDelete { + inner: Arc, + entered: Arc, + resume: Arc, + pause: Arc, + lose_response: Arc, + fail_read: Arc, + fail_hash: Arc>>, + fail_remove: Arc, + allow_remove: Arc>>, + } + + impl InterruptedDelete { + fn new(inner: Arc) -> Self { + Self { + inner, + entered: Arc::new(Notify::new()), + resume: Arc::new(Notify::new()), + pause: Arc::new(AtomicBool::new(false)), + lose_response: Arc::new(AtomicBool::new(false)), + fail_read: Arc::new(AtomicBool::new(false)), + fail_hash: Arc::new(tokio::sync::Mutex::new(None)), + fail_remove: Arc::new(AtomicBool::new(false)), + allow_remove: Arc::new(tokio::sync::Mutex::new(None)), + } + } + } + + #[async_trait::async_trait] + impl ImmutableReadStore for InterruptedDelete { + type Error = std::io::Error; + type Readable = ::Readable; + + async fn contains( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result { + self.inner.contains(space, id, generation).await + } + + async fn read( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result>, Self::Error> { + if self + .fail_hash + .lock() + .await + .is_some_and(|value| value == *id) + || self.fail_read.swap(false, Ordering::SeqCst) + { + return Err(std::io::Error::new( + std::io::ErrorKind::ConnectionReset, + "lost HEAD response", + )); + } + self.inner.read(space, id, generation).await + } + + async fn read_range( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + range: crate::storage::ByteRangeSpec, + ) -> Result>, Self::Error> { + self.inner.read_range(space, id, generation, range).await + } + } + + #[async_trait::async_trait] + impl ImmutableDeleteStore for InterruptedDelete { + type Error = std::io::Error; + + async fn remove( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result, Self::Error> { + if self.pause.swap(false, Ordering::SeqCst) { + self.entered.notify_one(); + self.resume.notified().await; + } + if self.fail_remove.load(Ordering::SeqCst) + && self.allow_remove.lock().await.as_ref() != Some(id) + { + return Err(std::io::Error::new( + std::io::ErrorKind::ConnectionReset, + "persistent DELETE transport failure", + )); + } + let removed = self.inner.remove(space, id, generation).await?; + if self.lose_response.swap(false, Ordering::SeqCst) { + return Err(std::io::Error::new( + std::io::ErrorKind::ConnectionReset, + "lost DELETE response", + )); + } + Ok(removed) + } + async fn cleanup_stages( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result<(), Self::Error> { + self.inner.cleanup_stages(space, id, generation).await + } + } + + async fn database() -> DatabaseConnection { + let db = Database::connect(ConnectOptions::new("sqlite::memory:".to_owned())) + .await + .unwrap(); + crate::migrations::Migrator::up(&db, None).await.unwrap(); + db.execute(Statement::from_string( + DbBackend::Sqlite, + "PRAGMA foreign_keys = OFF", + )) + .await + .unwrap(); + db + } + + fn space() -> SpaceId { + let jwk = JWK::generate_ed25519().unwrap(); + let did = DID_METHODS.generate(&jwk, "key").unwrap(); + SpaceId::new(did, "files".parse().unwrap()) + } + + async fn upload(store: &MemoryStore, space: &SpaceId, bytes: &[u8]) -> Hash { + upload_generation(store, space, bytes, 0).await + } + + async fn upload_generation( + store: &MemoryStore, + space: &SpaceId, + bytes: &[u8], + generation: i64, + ) -> Hash { + let mut staged = HashBuffer::new(Vec::new()); + staged.write_all(bytes).await.unwrap(); + store.persist(space, staged, generation).await.unwrap() + } + + async fn age(db: &DatabaseConnection, space: &SpaceId, value: Hash) { + kv_orphan_attempt::Entity::update_many() + .col_expr( + kv_orphan_attempt::Column::CreatedAt, + Expr::value(time::OffsetDateTime::now_utc() - time::Duration::hours(2)), + ) + .filter(kv_orphan_attempt::Column::Space.eq(SpaceIdWrap(space.clone()))) + .filter(kv_orphan_attempt::Column::Value.eq(value)) + .exec(db) + .await + .unwrap(); + } + async fn seed_crashed_attempts( + db: &DatabaseConnection, + space: &SpaceId, + value: Hash, + count: usize, + ) { + use sea_orm::ActiveValue::Set; + register(db, &[(space.clone(), value)], "crashed-0").await; + age(db, space, value).await; + let old = time::OffsetDateTime::now_utc() - time::Duration::hours(2); + for start in (1..count).step_by(256) { + kv_orphan_attempt::Entity::insert_many((start..(start + 256).min(count)).map( + |index| kv_orphan_attempt::ActiveModel { + space: Set(SpaceIdWrap(space.clone())), + value: Set(value), + attempt_id: Set(format!("crashed-{index}")), + generation: Set(0), + created_at: Set(old), + retry_after: Set(None), + }, + )) + .exec(db) + .await + .unwrap(); + } + } + + async fn advance_retry_tick(db: &DatabaseConnection, space: &SpaceId, value: Hash) { + kv_orphan_attempt::Entity::update_many() + .col_expr( + kv_orphan_attempt::Column::RetryAfter, + Expr::value(time::OffsetDateTime::now_utc() - Duration::seconds(1)), + ) + .filter(kv_orphan_attempt::Column::Space.eq(SpaceIdWrap(space.clone()))) + .filter(kv_orphan_attempt::Column::Value.eq(value)) + .exec(db) + .await + .unwrap(); + } + + async fn register( + db: &DatabaseConnection, + blobs: &[(SpaceId, Hash)], + id: &str, + ) -> Vec<(SpaceId, Hash, i64)> { + let tx = db.begin().await.unwrap(); + let registered = register_candidates(&tx, blobs, id).await.unwrap(); + tx.commit().await.unwrap(); + registered + } + + fn gen0(blobs: &[(SpaceId, Hash)]) -> Vec<(SpaceId, Hash, i64)> { + blobs + .iter() + .map(|(space, value)| (space.clone(), *value, 0)) + .collect() + } + + async fn reference( + db: &C, + table: &str, + space: &SpaceId, + value: Hash, + generation: i64, + ) { + let sql = match (table, generation) { + ("kv_write", 0) => + "INSERT INTO kv_write (space, key, invocation, seq, epoch, epoch_seq, value, metadata) VALUES (?, 'a', ?, 1, ?, 1, ?, '{}')", + ("kv_write", _) => + "INSERT INTO kv_write (space, key, invocation, seq, epoch, epoch_seq, value, metadata, generation) VALUES (?, 'a', ?, 1, ?, 1, ?, '{}', ?)", + (_, 0) => + "INSERT INTO current_kv (space, key, invocation, seq, epoch, epoch_seq, value, metadata, deleted) VALUES (?, 'a', ?, 1, ?, 1, ?, '{}', 0)", + (_, _) => + "INSERT INTO current_kv (space, key, invocation, seq, epoch, epoch_seq, value, metadata, deleted, generation) VALUES (?, 'a', ?, 1, ?, 1, ?, '{}', 0, ?)", + }; + let mut values: Vec = vec![ + space.to_string().into(), + Vec::::from(hash(table.as_bytes())).into(), + Vec::::from(hash(b"epoch")).into(), + Vec::::from(value).into(), + ]; + if generation != 0 { + values.push(generation.into()); + } + db.execute(Statement::from_sql_and_values( + DbBackend::Sqlite, + sql, + values, + )) + .await + .unwrap(); + } + + #[tokio::test] + async fn historical_and_current_references_survive_collection() { + let db = database().await; + let store = MemoryStore::default(); + let history_space = space(); + let current_space = space(); + let historical = upload(&store, &history_space, b"historical").await; + let current = upload(&store, ¤t_space, b"current").await; + let candidates = [ + (history_space.clone(), historical), + (current_space.clone(), current), + ]; + register(&db, &candidates, "a").await; + for (space, hash) in &candidates { + age(&db, space, *hash).await; + } + reference(&db, "kv_write", &history_space, historical, 0).await; + reference(&db, "current_kv", ¤t_space, current, 0).await; + assert_eq!( + kv_write::Entity::find() + .one(&db) + .await + .unwrap() + .unwrap() + .generation, + 0 + ); + assert_eq!( + current_kv::Entity::find() + .one(&db) + .await + .unwrap() + .unwrap() + .generation, + 0 + ); + + assert_eq!( + collect_candidates(&db, &store, 10, time::Duration::hours(1)) + .await + .unwrap(), + 0 + ); + for (space, hash) in &candidates { + assert!(store.contains(space, hash, 0).await.unwrap()); + assert!(kv_orphan_candidate::Entity::find_by_id(( + crate::types::SpaceIdWrap(space.clone()), + *hash + )) + .one(&db) + .await + .unwrap() + .is_none()); + } + } + + #[tokio::test] + async fn crashed_upload_is_refunded_after_five_minute_grace() { + let db = database().await; + let store = MemoryStore::default(); + let space = space(); + let value = hash(b"failed-precondition"); + register(&db, &[(space.clone(), value)], "a").await; + assert_eq!(upload(&store, &space, b"failed-precondition").await, value); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(b"failed-precondition".len() as u64) + ); + let tx = db.begin().await.unwrap(); + lock_candidates(&tx, &[(space.clone(), value, 0)], "a") + .await + .unwrap(); + tx.rollback().await.unwrap(); + age(&db, &space, value).await; + + assert_eq!( + collect_candidates(&db, &store, 10, time::Duration::minutes(5)) + .await + .unwrap(), + 1 + ); + assert!(!store.contains(&space, &value, 0).await.unwrap()); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + assert!( + kv_orphan_candidate::Entity::find_by_id((crate::types::SpaceIdWrap(space), value)) + .one(&db) + .await + .unwrap() + .is_none() + ); + } + + #[tokio::test] + async fn grace_and_pass_limit_do_not_scan_or_delete_newer_rows() { + let db = database().await; + let store = MemoryStore::default(); + let space = space(); + let mut old = Vec::new(); + for bytes in [b"old-1".as_slice(), b"old-2", b"old-3"] { + let value = upload(&store, &space, bytes).await; + register(&db, &[(space.clone(), value)], "a").await; + age(&db, &space, value).await; + old.push(value); + } + let fresh = upload(&store, &space, b"fresh").await; + register(&db, &[(space.clone(), fresh)], "a").await; + assert_eq!( + collect_candidates(&db, &store, 2, time::Duration::hours(1)) + .await + .unwrap(), + 2 + ); + let mut surviving = 0; + for value in &old { + surviving += usize::from(store.contains(&space, value, 0).await.unwrap()); + } + assert_eq!(surviving, 1); + assert!(store.contains(&space, &fresh, 0).await.unwrap()); + assert_eq!( + collect_candidates(&db, &store, 2, time::Duration::hours(1)) + .await + .unwrap(), + 1 + ); + assert_eq!( + collect_candidates(&db, &store, 2, time::Duration::hours(1)) + .await + .unwrap(), + 0 + ); + assert!(store.contains(&space, &fresh, 0).await.unwrap()); + } + + #[tokio::test] + async fn registering_existing_hash_resets_grace_period() { + let db = database().await; + let store = MemoryStore::default(); + let space = space(); + let value = upload(&store, &space, b"same-hash").await; + register(&db, &[(space.clone(), value)], "a").await; + age(&db, &space, value).await; + register(&db, &[(space.clone(), value)], "b").await; + + assert_eq!( + collect_candidates(&db, &store, 1, time::Duration::hours(1)) + .await + .unwrap(), + 0 + ); + assert!(store.contains(&space, &value, 0).await.unwrap()); + } + + #[tokio::test] + async fn first_failure_preserves_second_attempt_and_its_rollback_cleanup() { + let db = database().await; + let store = MemoryStore::default(); + let space = space(); + let value = hash(b"same candidate race"); + let blobs = [(space.clone(), value)]; + let tx = db.begin().await.unwrap(); + register_candidates(&tx, &blobs, "a").await.unwrap(); + tx.commit().await.unwrap(); + assert_eq!(upload(&store, &space, b"same candidate race").await, value); + let tx = db.begin().await.unwrap(); + register_candidates(&tx, &blobs, "b").await.unwrap(); + tx.commit().await.unwrap(); + + refund_failed(&db, &store, &gen0(&blobs), "a") + .await + .unwrap(); + assert!(store.contains(&space, &value, 0).await.unwrap()); + let b = db.begin().await.unwrap(); + lock_candidates(&b, &gen0(&blobs), "b").await.unwrap(); + b.rollback().await.unwrap(); + refund_failed(&db, &store, &gen0(&blobs), "b") + .await + .unwrap(); + assert!(!store.contains(&space, &value, 0).await.unwrap()); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn successful_same_hash_writer_protects_failed_writers_blob() { + let db = database().await; + let store = MemoryStore::default(); + let space = space(); + let value = upload(&store, &space, b"referenced-dedup").await; + let blobs = [(space.clone(), value)]; + register(&db, &blobs, "a").await; + register(&db, &blobs, "b").await; + let tx = db.begin().await.unwrap(); + lock_candidates(&tx, &gen0(&blobs), "b").await.unwrap(); + reference(&tx, "kv_write", &space, value, 0).await; + finish_candidates(&tx, &gen0(&blobs), "b").await.unwrap(); + tx.commit().await.unwrap(); + refund_failed(&db, &store, &gen0(&blobs), "a") + .await + .unwrap(); + assert!(store.contains(&space, &value, 0).await.unwrap()); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(b"referenced-dedup".len() as u64) + ); + assert_eq!( + collect_candidates(&db, &store, 10, Duration::ZERO) + .await + .unwrap(), + 0 + ); + assert!( + kv_orphan_candidate::Entity::find_by_id((SpaceIdWrap(space), value)) + .one(&db) + .await + .unwrap() + .is_none() + ); + } + #[tokio::test] + async fn expired_same_hash_attempts_drain_as_one_unreferenced_group() { + let db = database().await; + let store = MemoryStore::default(); + let space = space(); + let value = upload(&store, &space, b"two crashed uploads").await; + let blob = [(space.clone(), value)]; + register(&db, &blob, "crashed-a").await; + register(&db, &blob, "crashed-b").await; + age(&db, &space, value).await; + + assert_eq!( + collect_candidates(&db, &store, 256, ORPHAN_GRACE) + .await + .unwrap(), + 1 + ); + assert!(!store.contains(&space, &value, 0).await.unwrap()); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + assert_eq!( + kv_orphan_attempt::Entity::find() + .all(&db) + .await + .unwrap() + .len(), + 0 + ); + assert_eq!( + kv_orphan_candidate::Entity::find() + .all(&db) + .await + .unwrap() + .len(), + 0 + ); + } + + #[tokio::test] + async fn expired_same_hash_attempts_drain_without_removing_committed_reference() { + let db = database().await; + let store = MemoryStore::default(); + let space = space(); + let value = upload(&store, &space, b"referenced retry").await; + let blob = [(space.clone(), value)]; + register(&db, &blob, "crashed-a").await; + register(&db, &blob, "crashed-b").await; + age(&db, &space, value).await; + reference(&db, "kv_write", &space, value, 0).await; + + assert_eq!( + collect_candidates(&db, &store, 256, ORPHAN_GRACE) + .await + .unwrap(), + 0 + ); + assert!(store.contains(&space, &value, 0).await.unwrap()); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(b"referenced retry".len() as u64) + ); + assert_eq!( + kv_orphan_attempt::Entity::find() + .all(&db) + .await + .unwrap() + .len(), + 0 + ); + assert_eq!( + kv_orphan_candidate::Entity::find() + .all(&db) + .await + .unwrap() + .len(), + 0 + ); + } + + #[tokio::test] + async fn staggered_collectors_drain_crashed_claims_after_last_live_writer_exits() { + let directory = tempfile::tempdir().unwrap(); + let url = format!( + "sqlite://{}?mode=rwc", + directory + .path() + .join("staggered-collectors.sqlite") + .display() + ); + let first = Database::connect(ConnectOptions::new(url.clone())) + .await + .unwrap(); + crate::migrations::Migrator::up(&first, None).await.unwrap(); + let second = Database::connect(ConnectOptions::new(url)).await.unwrap(); + for db in [&first, &second] { + db.execute(Statement::from_string( + DbBackend::Sqlite, + "PRAGMA foreign_keys = OFF", + )) + .await + .unwrap(); + } + + let store = MemoryStore::default(); + let space = space(); + let value = upload(&store, &space, b"two stale claims").await; + let blob = [(space.clone(), value)]; + register(&first, &blob, "crashed-a").await; + register(&first, &blob, "crashed-b").await; + age(&first, &space, value).await; + let live = register(&first, &blob, "live-c").await; + + // First collector defers one old claim while C is still uploading. + assert_eq!( + collect_candidates(&first, &store, 1, ORPHAN_GRACE) + .await + .unwrap(), + 0 + ); + refund_failed(&first, &store, &live, "live-c") + .await + .unwrap(); + assert!(store.contains(&space, &value, 0).await.unwrap()); + + // A second collector must not mistake A's retry delay for a live + // writer; with no real uploader left, the group and bytes drain now. + assert_eq!( + collect_candidates(&second, &store, 1, ORPHAN_GRACE) + .await + .unwrap(), + 1 + ); + assert!(!store.contains(&space, &value, 0).await.unwrap()); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + assert!(kv_orphan_attempt::Entity::find() + .all(&second) + .await + .unwrap() + .is_empty()); + } + + #[tokio::test] + async fn failed_gc_read_does_not_make_crashed_sibling_look_live() { + let db = database().await; + let inner = Arc::new(MemoryStore::default()); + let store = InterruptedDelete::new(inner.clone()); + let space = space(); + let value = upload(&inner, &space, b"failed first collector read").await; + let blob = [(space.clone(), value)]; + register(&db, &blob, "crashed-a").await; + register(&db, &blob, "crashed-b").await; + age(&db, &space, value).await; + + store.fail_read.store(true, Ordering::SeqCst); + assert!(collect_candidates(&db, &store, 1, ORPHAN_GRACE) + .await + .is_err()); + assert_eq!( + collect_candidates(&db, &store, 1, ORPHAN_GRACE) + .await + .unwrap(), + 1, + "a failed read must not turn its crashed claim into a live writer" + ); + assert!(!inner.contains(&space, &value, 0).await.unwrap()); + assert_eq!(inner.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn last_failed_writer_wakes_deferred_crash_cleanup() { + let db = database().await; + let store = MemoryStore::default(); + let space = space(); + let value = upload(&store, &space, b"last failed writer").await; + let blob = [(space.clone(), value)]; + register(&db, &blob, "crashed-a").await; + age(&db, &space, value).await; + let live = register(&db, &blob, "live-b").await; + assert_eq!( + collect_candidates(&db, &store, 1, ORPHAN_GRACE) + .await + .unwrap(), + 0 + ); + refund_failed(&db, &store, &live, "live-b").await.unwrap(); + assert_eq!( + collect_candidates(&db, &store, 1, ORPHAN_GRACE) + .await + .unwrap(), + 1, + "the final writer leaving must expose the old claim immediately" + ); + assert!(!store.contains(&space, &value, 0).await.unwrap()); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn full_page_retained_for_live_writer_does_not_hide_later_orphans() { + let db = database().await; + let store = MemoryStore::default(); + let space = space(); + let blocked = upload(&store, &space, b"blocked group").await; + let blocked_blob = [(space.clone(), blocked)]; + for index in 0..256 { + register(&db, &blocked_blob, &format!("crashed-{index}")).await; + } + age(&db, &space, blocked).await; + register(&db, &blocked_blob, "active-writer").await; + + let later = upload(&store, &space, b"independent later orphan").await; + register(&db, &[(space.clone(), later)], "later-crash").await; + age(&db, &space, later).await; + assert_eq!( + tokio::time::timeout( + std::time::Duration::from_secs(3), + drain_candidates_for_interval(&db, &store, ORPHAN_GRACE), + ) + .await + .expect("one collector tick must terminate on a blocked full page") + .unwrap(), + 1, + "the retained page must not starve an independent later orphan" + ); + assert!(!store.contains(&space, &later, 0).await.unwrap()); + assert!(store.contains(&space, &blocked, 0).await.unwrap()); + } + + #[tokio::test] + async fn repeated_failed_oldest_page_cannot_starve_independent_orphan_across_ticks() { + let db = database().await; + let inner = Arc::new(MemoryStore::default()); + let store = InterruptedDelete::new(inner.clone()); + let space = space(); + let blocked = upload(&inner, &space, b"persistently unreadable orphan").await; + seed_crashed_attempts(&db, &space, blocked, 256).await; + *store.fail_hash.lock().await = Some(blocked); + + let healthy = upload(&inner, &space, b"healthy later orphan").await; + register(&db, &[(space.clone(), healthy)], "healthy").await; + kv_orphan_attempt::Entity::update_many() + .col_expr( + kv_orphan_attempt::Column::CreatedAt, + Expr::value(time::OffsetDateTime::now_utc() - Duration::hours(1)), + ) + .filter(kv_orphan_attempt::Column::Value.eq(healthy)) + .exec(&db) + .await + .unwrap(); + + // Logical ticks 0, 60, 120 seconds: make the retry deadline due + // before each subsequent production drain without sleeping a minute. + assert!(drain_candidates_for_interval(&db, &store, ORPHAN_GRACE) + .await + .is_err()); + advance_retry_tick(&db, &space, blocked).await; + assert!(drain_candidates_for_interval(&db, &store, ORPHAN_GRACE) + .await + .is_err()); + assert!( + !inner.contains(&space, &healthy, 0).await.unwrap(), + "a permanently failing prefix must not hide unrelated bytes" + ); + advance_retry_tick(&db, &space, blocked).await; + assert!(drain_candidates_for_interval(&db, &store, ORPHAN_GRACE) + .await + .is_err()); + assert!(inner.contains(&space, &blocked, 0).await.unwrap()); + assert_eq!( + inner.total_size(&space).await.unwrap(), + Some(b"persistently unreadable orphan".len() as u64) + ); + } + + #[tokio::test] + async fn four_live_blocked_pages_cannot_starve_independent_orphan_across_ticks() { + let db = database().await; + let store = MemoryStore::default(); + let space = space(); + let blocked = upload(&store, &space, b"live protected orphan").await; + seed_crashed_attempts(&db, &space, blocked, 1024).await; + let live = register(&db, &[(space.clone(), blocked)], "live-writer").await; + + let healthy = upload(&store, &space, b"healthy after four pages").await; + register(&db, &[(space.clone(), healthy)], "healthy").await; + kv_orphan_attempt::Entity::update_many() + .col_expr( + kv_orphan_attempt::Column::CreatedAt, + Expr::value(time::OffsetDateTime::now_utc() - Duration::hours(1)), + ) + .filter(kv_orphan_attempt::Column::Value.eq(healthy)) + .exec(&db) + .await + .unwrap(); + + assert_eq!( + drain_candidates_for_interval(&db, &store, ORPHAN_GRACE) + .await + .unwrap(), + 0 + ); + assert!(store.contains(&space, &healthy, 0).await.unwrap()); + advance_retry_tick(&db, &space, blocked).await; + assert_eq!( + drain_candidates_for_interval(&db, &store, ORPHAN_GRACE) + .await + .unwrap(), + 1, + "a later healthy hash must progress even across four blocked pages" + ); + assert!(!store.contains(&space, &healthy, 0).await.unwrap()); + advance_retry_tick(&db, &space, blocked).await; + assert_eq!( + drain_candidates_for_interval(&db, &store, ORPHAN_GRACE) + .await + .unwrap(), + 0 + ); + assert!(store.contains(&space, &blocked, 0).await.unwrap()); + refund_failed(&db, &store, &live, "live-writer") + .await + .unwrap(); + assert_eq!( + drain_candidates_for_interval(&db, &store, ORPHAN_GRACE) + .await + .unwrap(), + 1 + ); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn departing_writers_cannot_restore_old_claims_ahead_of_an_unrelated_orphan() { + let db = database().await; + let store = MemoryStore::default(); + let space = space(); + let blocked = upload(&store, &space, b"departing writers").await; + seed_crashed_attempts(&db, &space, blocked, 1024).await; + let mut live = register(&db, &[(space.clone(), blocked)], "live-0").await; + + let healthy = upload(&store, &space, b"unrelated orphan").await; + register(&db, &[(space.clone(), healthy)], "healthy").await; + kv_orphan_attempt::Entity::update_many() + .col_expr( + kv_orphan_attempt::Column::CreatedAt, + Expr::value(time::OffsetDateTime::now_utc() - Duration::hours(1)), + ) + .filter(kv_orphan_attempt::Column::Value.eq(healthy)) + .exec(&db) + .await + .unwrap(); + + assert_eq!( + drain_candidates_for_interval(&db, &store, ORPHAN_GRACE) + .await + .unwrap(), + 0 + ); + for index in 1..=2 { + let next = format!("live-{index}"); + let successor = register(&db, &[(space.clone(), blocked)], &next).await; + refund_failed(&db, &store, &live, &format!("live-{}", index - 1)) + .await + .unwrap(); + live = successor; + // A departure wakes old claims while the successor still protects + // their blob. It must not make them outrank a different orphan. + drain_candidates_for_interval(&db, &store, ORPHAN_GRACE) + .await + .unwrap(); + } + assert!( + !store.contains(&space, &healthy, 0).await.unwrap(), + "repeated live writer departures must not starve an unrelated hash" + ); + assert!(store.contains(&space, &blocked, 0).await.unwrap()); + refund_failed(&db, &store, &live, "live-2").await.unwrap(); + } + + #[tokio::test] + async fn retried_deletes_cannot_hide_healthy_orphans_across_ticks() { + use sea_orm::ActiveValue::Set; + let db = database().await; + let inner = Arc::new(MemoryStore::default()); + let store = InterruptedDelete::new(inner.clone()); + let space = space(); + let old = time::OffsetDateTime::now_utc() - Duration::hours(2); + kv_orphan_candidate::Entity::insert_many((0..256).map(|index| { + kv_orphan_candidate::ActiveModel { + space: Set(SpaceIdWrap(space.clone())), + value: Set(hash(format!("failed-delete-{index}").as_bytes())), + generation: Set(0), + created_at: Set(old), + deleting_at: Set(Some(old)), + deletion_bytes: Set(Some(1)), + } + })) + .exec(&db) + .await + .unwrap(); + store.fail_remove.store(true, Ordering::SeqCst); + let healthy = upload(&inner, &space, b"healthy behind failed deletes").await; + *store.allow_remove.lock().await = Some(healthy); + register(&db, &[(space.clone(), healthy)], "healthy").await; + age(&db, &space, healthy).await; + assert!(drain_candidates_for_interval(&db, &store, ORPHAN_GRACE) + .await + .is_err()); + kv_orphan_candidate::Entity::update_many() + .col_expr( + kv_orphan_candidate::Column::DeletingAt, + Expr::value(time::OffsetDateTime::now_utc() - Duration::seconds(1)), + ) + .filter(kv_orphan_candidate::Column::DeletingAt.is_not_null()) + .exec(&db) + .await + .unwrap(); + assert!(drain_candidates_for_interval(&db, &store, ORPHAN_GRACE) + .await + .is_err()); + assert!( + !inner.contains(&space, &healthy, 0).await.unwrap(), + "retrying deletes must leave capacity for independent healthy attempts" + ); + assert_eq!(inner.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn replayed_registration_requires_own_attempt_without_duplicating_it() { + let db = database().await; + let space = space(); + let value = hash(b"replay"); + let blobs = [(space.clone(), value)]; + register(&db, &blobs, "a").await; + register(&db, &blobs, "a").await; + let attempts = kv_orphan_attempt::Entity::find() + .filter(kv_orphan_attempt::Column::Space.eq(SpaceIdWrap(space.clone()))) + .filter(kv_orphan_attempt::Column::Value.eq(value)) + .all(&db) + .await + .unwrap(); + assert_eq!(attempts.len(), 1); + let tx = db.begin().await.unwrap(); + let error = lock_candidates(&tx, &gen0(&blobs), "b").await.unwrap_err(); + assert!(is_candidate_unavailable(&error)); + tx.rollback().await.unwrap(); + } + + #[tokio::test] + async fn deleting_fence_refuses_new_attempt_until_delete_finishes() { + let db = database().await; + let space = space(); + let value = hash(b"stale delete cannot hit a new write"); + let blobs = [(space.clone(), value)]; + let tx = db.begin().await.unwrap(); + register_candidates(&tx, &blobs, "a").await.unwrap(); + tx.commit().await.unwrap(); + kv_orphan_candidate::Entity::update_many() + .col_expr( + kv_orphan_candidate::Column::DeletingAt, + Expr::value(time::OffsetDateTime::now_utc()), + ) + .filter(kv_orphan_candidate::Column::Space.eq(SpaceIdWrap(space))) + .filter(kv_orphan_candidate::Column::Value.eq(value)) + .exec(&db) + .await + .unwrap(); + let tx = db.begin().await.unwrap(); + assert!(register_candidates(&tx, &blobs, "b").await.is_err()); + tx.rollback().await.unwrap(); + } + + #[tokio::test] + async fn slow_delete_releases_database_and_refuses_new_preflight() { + let db = database().await; + let inner = Arc::new(MemoryStore::default()); + let store = InterruptedDelete::new(inner.clone()); + let space = space(); + let value = upload(&inner, &space, b"slow-delete").await; + let blobs = [(space.clone(), value)]; + let old = register(&db, &blobs, "a").await; + store.pause.store(true, Ordering::SeqCst); + let deleting = tokio::spawn({ + let db = db.clone(); + let store = store.clone(); + let old = old.clone(); + async move { refund_failed(&db, &store, &old, "a").await } + }); + store.entered.notified().await; + let fence = kv_orphan_candidate::Entity::find_by_id((SpaceIdWrap(space.clone()), value)) + .one(&db) + .await + .unwrap() + .unwrap(); + assert_eq!(fence.deletion_bytes, Some(b"slow-delete".len() as i64)); + assert!(fence.deleting_at.is_some()); + let tx = tokio::time::timeout(std::time::Duration::from_secs(1), db.begin()) + .await + .unwrap() + .unwrap(); + let error = register_candidates(&tx, &blobs, "b").await.unwrap_err(); + assert!(is_candidate_unavailable(&error)); + tx.rollback().await.unwrap(); + store.resume.notify_one(); + deleting.await.unwrap().unwrap(); + assert!(!inner.contains(&space, &value, 0).await.unwrap()); + let next = register(&db, &blobs, "b").await; + assert_eq!(next[0].2, 1); + assert_eq!( + upload_generation(&inner, &space, b"slow-delete", 1).await, + value + ); + let tx = db.begin().await.unwrap(); + lock_candidates(&tx, &next, "b").await.unwrap(); + reference(&tx, "kv_write", &space, value, 1).await; + finish_candidates(&tx, &next, "b").await.unwrap(); + tx.commit().await.unwrap(); + assert!(kv_write::Entity::find() + .filter(kv_write::Column::Space.eq(SpaceIdWrap(space.clone()))) + .filter(kv_write::Column::Value.eq(value)) + .one(&db) + .await + .unwrap() + .is_some()); + assert!(inner.contains(&space, &value, 1).await.unwrap()); + assert_eq!( + inner.total_size(&space).await.unwrap(), + Some(b"slow-delete".len() as u64) + ); + } + + #[tokio::test] + async fn lost_delete_response_retries_from_durable_fence_once() { + let db = database().await; + let inner = Arc::new(MemoryStore::default()); + let store = InterruptedDelete::new(inner.clone()); + let space = space(); + let value = upload(&inner, &space, b"lost-delete").await; + let blobs = [(space.clone(), value)]; + let old = register(&db, &blobs, "a").await; + store.lose_response.store(true, Ordering::SeqCst); + assert!(refund_failed(&db, &store, &old, "a").await.is_err()); + let fence = kv_orphan_candidate::Entity::find_by_id((SpaceIdWrap(space.clone()), value)) + .one(&db) + .await + .unwrap() + .unwrap(); + assert!(fence.deleting_at.is_some()); + assert_eq!(fence.deletion_bytes, Some(b"lost-delete".len() as i64)); + assert!(!inner.contains(&space, &value, 0).await.unwrap()); + assert_eq!(deleting_fences(&db).await.unwrap(), old); + assert_eq!( + collect_candidates(&db, &store, 10, Duration::ZERO) + .await + .unwrap(), + 0, + "the retry confirms absence without charging a second refund" + ); + assert!( + kv_orphan_candidate::Entity::find_by_id((SpaceIdWrap(space.clone()), value)) + .one(&db) + .await + .unwrap() + .is_none() + ); + assert!(deleting_fences(&db).await.unwrap().is_empty()); + assert_eq!(inner.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn two_deleters_cannot_remove_a_newly_committed_same_hash() { + let db = database().await; + let inner = Arc::new(MemoryStore::default()); + let store = InterruptedDelete::new(inner.clone()); + let space = space(); + let value = upload(&inner, &space, b"two-deleter-race").await; + let blobs = [(space.clone(), value)]; + let old = register(&db, &blobs, "a").await; + + store.pause.store(true, Ordering::SeqCst); + let first = tokio::spawn({ + let db = db.clone(); + let store = store.clone(); + let old = old.clone(); + async move { refund_failed(&db, &store, &old, "a").await } + }); + tokio::time::timeout(std::time::Duration::from_secs(5), store.entered.notified()) + .await + .unwrap(); + store.pause.store(true, Ordering::SeqCst); + let second_entered = store.entered.notified(); + let second = tokio::spawn({ + let db = db.clone(); + let store = store.clone(); + async move { collect_candidates(&db, &store, 10, Duration::ZERO).await } + }); + tokio::time::timeout(std::time::Duration::from_secs(5), second_entered) + .await + .unwrap(); + store.resume.notify_one(); + tokio::time::timeout(std::time::Duration::from_secs(5), first) + .await + .unwrap() + .unwrap() + .unwrap(); + + let next = register(&db, &blobs, "b").await; + assert_eq!(next[0].2, 1); + let stale = db.begin().await.unwrap(); + assert!(is_candidate_unavailable( + &lock_candidates(&stale, &old, "b").await.unwrap_err() + )); + stale.rollback().await.unwrap(); + assert_eq!( + upload_generation(&inner, &space, b"two-deleter-race", 1).await, + value + ); + let tx = db.begin().await.unwrap(); + lock_candidates(&tx, &next, "b").await.unwrap(); + reference(&tx, "kv_write", &space, value, 1).await; + finish_candidates(&tx, &next, "b").await.unwrap(); + tx.commit().await.unwrap(); + + store.resume.notify_one(); + tokio::time::timeout(std::time::Duration::from_secs(5), second) + .await + .unwrap() + .unwrap() + .unwrap(); + assert!( + inner.contains(&space, &value, 1).await.unwrap(), + "a delayed gen0 DELETE removed newly committed gen1 content" + ); + assert!(!inner.contains(&space, &value, 0).await.unwrap()); + assert_eq!( + kv_write::Entity::find() + .one(&db) + .await + .unwrap() + .unwrap() + .generation, + 1 + ); + assert_eq!( + ledger::Entity::find_by_id((SpaceIdWrap(space), value)) + .one(&db) + .await + .unwrap() + .unwrap() + .generation, + 1 + ); + } + + #[tokio::test] + async fn postgres_upload_fence_skips_locked_candidate() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_upload_fence_skips_locked_candidate", + ) + .await + else { + return; + }; + let store = MemoryStore::default(); + let space = space(); + let value = upload(&store, &space, b"upload-in-progress").await; + let prep = fixture.conn.begin().await.unwrap(); + register_candidates(&prep, &[(space.clone(), value)], "a") + .await + .unwrap(); + prep.commit().await.unwrap(); + kv_orphan_attempt::Entity::update_many() + .col_expr( + kv_orphan_attempt::Column::CreatedAt, + Expr::value(time::OffsetDateTime::now_utc() - time::Duration::hours(2)), + ) + .filter(kv_orphan_attempt::Column::Space.eq(SpaceIdWrap(space.clone()))) + .filter(kv_orphan_attempt::Column::Value.eq(value)) + .exec(&fixture.conn) + .await + .unwrap(); + let tx = fixture.conn.begin().await.unwrap(); + lock_candidates(&tx, &[(space.clone(), value, 0)], "a") + .await + .unwrap(); + let collected = tokio::time::timeout( + std::time::Duration::from_secs(5), + collect_candidates(&fixture.conn, &store, 1, time::Duration::hours(1)), + ) + .await + .expect("collector should skip the locked candidate") + .unwrap(); + assert_eq!(collected, 0); + assert!(store.contains(&space, &value, 0).await.unwrap()); + tx.rollback().await.unwrap(); + assert_eq!( + collect_candidates(&fixture.conn, &store, 1, time::Duration::hours(1)) + .await + .unwrap(), + 1 + ); + } + + #[tokio::test] + async fn postgres_one_statement_registration_refuses_an_unhosted_space_without_writing() { + use sea_orm::{ActiveModelTrait, ActiveValue::Set}; + + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_one_statement_registration_refuses_an_unhosted_space_without_writing", + ) + .await + else { + return; + }; + let store = MemoryStore::default(); + let hosted = space(); + let unhosted = space(); + let value = upload(&store, &hosted, b"hosted-guard").await; + let rows = || async { + ( + ledger::Entity::find() + .all(&fixture.conn) + .await + .unwrap() + .len(), + candidate::Entity::find() + .all(&fixture.conn) + .await + .unwrap() + .len(), + attempt::Entity::find() + .all(&fixture.conn) + .await + .unwrap() + .len(), + ) + }; + let blobs = [(hosted.clone(), value)]; + + // Neither space exists: refused, and not one row is written. + let error = + register_candidates_hosted(&fixture.conn, &blobs, "a", std::slice::from_ref(&hosted)) + .await + .unwrap_err(); + assert!(is_space_not_hosted(&error), "{error}"); + assert_eq!(rows().await, (0, 0, 0)); + + // The blob's own space exists but another space of the invocation does + // not: still refused, still nothing written. + crate::models::space::ActiveModel { + id: Set(SpaceIdWrap(hosted.clone())), + } + .insert(&fixture.conn) + .await + .unwrap(); + let error = register_candidates_hosted( + &fixture.conn, + &blobs, + "a", + &[hosted.clone(), unhosted.clone()], + ) + .await + .unwrap_err(); + assert!(is_space_not_hosted(&error), "{error}"); + assert_eq!(rows().await, (0, 0, 0)); + + // Every listed space exists: one registration, generation zero. + let generations = + register_candidates_hosted(&fixture.conn, &blobs, "a", std::slice::from_ref(&hosted)) + .await + .unwrap(); + assert_eq!(generations, vec![(hosted, value, 0)]); + assert_eq!(rows().await, (1, 1, 1)); + } + + /// A deleting fence for `(space, content, 0)` with no claim, as a stopped + /// node leaves it, while a committed write references the same bytes at + /// generation zero. + async fn stale_fence_with_reference( + db: &DatabaseConnection, + store: &MemoryStore, + ) -> (SpaceId, Hash) { + let space = space(); + let value = upload(store, &space, b"stale-deleting-fence").await; + register(db, &[(space.clone(), value)], "a").await; + kv_orphan_attempt::Entity::delete_many() + .exec(db) + .await + .unwrap(); + kv_orphan_candidate::Entity::update_many() + .col_expr( + kv_orphan_candidate::Column::DeletingAt, + Expr::value(time::OffsetDateTime::now_utc() - time::Duration::hours(1)), + ) + .col_expr( + kv_orphan_candidate::Column::DeletionBytes, + Expr::value(1i64), + ) + .exec(db) + .await + .unwrap(); + if db.get_database_backend() == DbBackend::Postgres { + // Replication role replica skips the foreign keys this fixture does + // not need. + let tx = db.begin().await.unwrap(); + tx.execute_unprepared("SET LOCAL session_replication_role = replica") + .await + .unwrap(); + tx.execute(Statement::from_sql_and_values( + DbBackend::Postgres, + "INSERT INTO kv_write + (space, key, invocation, seq, epoch, epoch_seq, value, metadata, generation) + VALUES ($1, 'a', $2, 1, $3, 1, $4, '{}', 0)", + [ + space.to_string().into(), + Vec::::from(hash(b"stale-invocation")).into(), + Vec::::from(hash(b"epoch")).into(), + Vec::::from(value).into(), + ], + )) + .await + .unwrap(); + tx.commit().await.unwrap(); + } else { + reference(db, "kv_write", &space, value, 0).await; + } + (space, value) + } + + async fn assert_referenced_blob_survives_collection(db: &DatabaseConnection) { + let store = MemoryStore::default(); + let (space, value) = stale_fence_with_reference(db, &store).await; + let removed = collect_candidates(db, &store, 10, Duration::ZERO) + .await + .unwrap(); + assert_eq!( + removed, 0, + "a referenced blob must not be counted as removed" + ); + assert!( + store.contains(&space, &value, 0).await.unwrap(), + "the collector deleted a blob that a committed write references" + ); + assert!(kv_orphan_candidate::Entity::find() + .all(db) + .await + .unwrap() + .is_empty()); + assert_eq!( + ledger::Entity::find_by_id((SpaceIdWrap(space), value)) + .one(db) + .await + .unwrap() + .unwrap() + .generation, + 0 + ); + } + + #[tokio::test] + async fn stale_deleting_fence_does_not_remove_a_referenced_blob() { + assert_referenced_blob_survives_collection(&database().await).await; + } + + #[tokio::test] + async fn postgres_stale_deleting_fence_does_not_remove_a_referenced_blob() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_stale_deleting_fence_does_not_remove_a_referenced_blob", + ) + .await + else { + return; + }; + assert_referenced_blob_survives_collection(&fixture.conn).await; + } + + /// A put of two blobs whose second has a pending deletion claims neither: + /// the first blob's claim, fence and ledger row are rolled back with it. + async fn assert_two_blob_registration_is_all_or_nothing(db: &DatabaseConnection) { + let store = MemoryStore::default(); + let space = space(); + let first = upload(&store, &space, b"all-or-nothing first").await; + let second = upload(&store, &space, b"all-or-nothing second").await; + let mut blobs = [(space.clone(), first), (space.clone(), second)]; + blobs.sort_unstable(); + let blocked = blobs[1].clone(); + register(db, std::slice::from_ref(&blocked), "earlier").await; + kv_orphan_candidate::Entity::update_many() + .col_expr( + kv_orphan_candidate::Column::DeletingAt, + Expr::value(time::OffsetDateTime::now_utc()), + ) + .exec(db) + .await + .unwrap(); + let tx = db.begin().await.unwrap(); + let error = register_candidates(&tx, &blobs, "later").await.unwrap_err(); + assert!(is_candidate_unavailable(&error), "{error}"); + tx.rollback().await.unwrap(); + + let claimed = |value| { + let key = SpaceIdWrap(space.clone()); + async move { + ( + kv_orphan_attempt::Entity::find_by_id((key.clone(), value, "later".to_owned())) + .one(db) + .await + .unwrap() + .is_some(), + kv_orphan_candidate::Entity::find_by_id((key.clone(), value)) + .one(db) + .await + .unwrap() + .is_some(), + ledger::Entity::find_by_id((key, value)) + .one(db) + .await + .unwrap() + .is_some(), + ) + } + }; + assert_eq!( + claimed(blobs[0].1).await, + (false, false, false), + "the first blob kept a claim, fence or ledger row" + ); + assert!(claimed(blocked.1).await.1, "the pending deletion is intact"); + } + + #[tokio::test] + async fn two_blob_registration_is_all_or_nothing() { + assert_two_blob_registration_is_all_or_nothing(&database().await).await; + } + + #[tokio::test] + async fn postgres_two_blob_registration_is_all_or_nothing() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_two_blob_registration_is_all_or_nothing", + ) + .await + else { + return; + }; + assert_two_blob_registration_is_all_or_nothing(&fixture.conn).await; + } + + #[tokio::test] + async fn postgres_held_old_fence_does_not_starve_an_independent_orphan_across_ticks() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_held_old_fence_does_not_starve_an_independent_orphan_across_ticks", + ) + .await + else { + return; + }; + let store = MemoryStore::default(); + let space = space(); + let blocked = upload(&store, &space, b"held oldest fence").await; + seed_crashed_attempts(&fixture.conn, &space, blocked, 256).await; + let healthy = upload(&store, &space, b"independent expired blob").await; + register(&fixture.conn, &[(space.clone(), healthy)], "healthy").await; + kv_orphan_attempt::Entity::update_many() + .col_expr( + kv_orphan_attempt::Column::CreatedAt, + Expr::value(time::OffsetDateTime::now_utc() - Duration::hours(1)), + ) + .filter(kv_orphan_attempt::Column::Value.eq(healthy)) + .exec(&fixture.conn) + .await + .unwrap(); + + let schema: String = fixture + .conn + .query_one(Statement::from_string( + DbBackend::Postgres, + "SELECT current_schema() AS schema_name", + )) + .await + .unwrap() + .unwrap() + .try_get("", "schema_name") + .unwrap(); + let mut options = ConnectOptions::new( + crate::test_support::postgres_test_url( + "postgres_held_old_fence_does_not_starve_an_independent_orphan_across_ticks", + ) + .unwrap(), + ); + options.set_schema_search_path(schema); + let collector = Database::connect(options).await.unwrap(); + let held = fixture.conn.begin().await.unwrap(); + lock_candidates(&held, &[(space.clone(), blocked, 0)], "crashed-0") + .await + .unwrap(); + + // Logical ticks 0/60/120 advance the age cutoff without wall-clock + // sleeps. The other pool holds X throughout; H must make progress. + let mut ticks = Vec::with_capacity(3); + for tick in 0..3 { + let min_age = ORPHAN_GRACE - Duration::seconds(tick * 60); + let snapshot = match tokio::time::timeout( + std::time::Duration::from_secs(20), + drain_candidates_for_interval(&collector, &store, min_age), + ) + .await + { + Ok(Ok(removed)) => ( + removed, + store.contains(&space, &healthy, 0).await.unwrap(), + store.contains(&space, &blocked, 0).await.unwrap(), + ), + _ => (usize::MAX, true, true), + }; + ticks.push(snapshot); + if snapshot.0 == usize::MAX { + break; + } + } + // Always release the independent transaction and pool before checking + // the assertions; otherwise a failing assertion can deadlock schema teardown. + held.rollback().await.unwrap(); + let after_release = tokio::time::timeout( + std::time::Duration::from_secs(20), + drain_candidates_for_interval(&collector, &store, ORPHAN_GRACE), + ) + .await; + let blocked_after_release = store.contains(&space, &blocked, 0).await.unwrap(); + let final_size = store.total_size(&space).await.unwrap(); + collector.close().await.unwrap(); + assert_eq!( + ticks, + vec![(1, false, true), (0, false, true), (0, false, true)], + "the unrelated H must retire while X's fence is still held" + ); + assert_eq!(after_release.unwrap().unwrap(), 1); + assert!(!blocked_after_release); + assert_eq!(final_size, Some(0)); + } + + #[tokio::test] + async fn postgres_expired_same_hash_claims_drain_under_one_fence() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_expired_same_hash_claims_drain_under_one_fence", + ) + .await + else { + return; + }; + let store = MemoryStore::default(); + let space = space(); + let value = upload(&store, &space, b"expired-pg-group").await; + let blob = [(space.clone(), value)]; + register(&fixture.conn, &blob, "crashed-a").await; + register(&fixture.conn, &blob, "crashed-b").await; + age(&fixture.conn, &space, value).await; + assert_eq!( + collect_candidates(&fixture.conn, &store, 256, ORPHAN_GRACE) + .await + .unwrap(), + 1 + ); + assert_eq!( + kv_orphan_attempt::Entity::find() + .all(&fixture.conn) + .await + .unwrap() + .len(), + 0 + ); + assert!(!store.contains(&space, &value, 0).await.unwrap()); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn postgres_committing_writer_does_not_reprioritize_deferred_orphans() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_committing_writer_does_not_reprioritize_deferred_orphans", + ) + .await + else { + return; + }; + let store = MemoryStore::default(); + let space = space(); + let blocked = upload(&store, &space, b"PG protected backlog").await; + seed_crashed_attempts(&fixture.conn, &space, blocked, 256).await; + register(&fixture.conn, &[(space.clone(), blocked)], "committing").await; + assert_eq!( + collect_candidates(&fixture.conn, &store, 256, ORPHAN_GRACE) + .await + .unwrap(), + 0 + ); + + let healthy = upload(&store, &space, b"PG unrelated orphan").await; + register(&fixture.conn, &[(space.clone(), healthy)], "healthy").await; + kv_orphan_attempt::Entity::update_many() + .col_expr( + kv_orphan_attempt::Column::CreatedAt, + Expr::value(time::OffsetDateTime::now_utc() - Duration::hours(1)), + ) + .filter(kv_orphan_attempt::Column::Value.eq(healthy)) + .exec(&fixture.conn) + .await + .unwrap(); + + let actor_id = "did:key:gc-fairness".to_owned(); + actor::Entity::insert(actor::ActiveModel { + id: sea_orm::ActiveValue::Set(actor_id.clone()), + }) + .exec(&fixture.conn) + .await + .unwrap(); + space_model::Entity::insert(space_model::ActiveModel { + id: sea_orm::ActiveValue::Set(SpaceIdWrap(space.clone())), + }) + .exec(&fixture.conn) + .await + .unwrap(); + invocation::Entity::insert(invocation::ActiveModel { + id: sea_orm::ActiveValue::Set(hash(b"invocation")), + invoker: sea_orm::ActiveValue::Set(actor_id), + issued_at: sea_orm::ActiveValue::Set(time::OffsetDateTime::now_utc()), + facts: sea_orm::ActiveValue::Set(None), + serialization: sea_orm::ActiveValue::Set(vec![1]), + }) + .exec(&fixture.conn) + .await + .unwrap(); + epoch::Entity::insert(epoch::ActiveModel { + seq: sea_orm::ActiveValue::Set(1), + id: sea_orm::ActiveValue::Set(hash(b"epoch")), + space: sea_orm::ActiveValue::Set(SpaceIdWrap(space.clone())), + }) + .exec(&fixture.conn) + .await + .unwrap(); + event_order::Entity::insert(event_order::ActiveModel { + seq: sea_orm::ActiveValue::Set(1), + epoch: sea_orm::ActiveValue::Set(hash(b"epoch")), + epoch_seq: sea_orm::ActiveValue::Set(1), + event: sea_orm::ActiveValue::Set(hash(b"invocation")), + space: sea_orm::ActiveValue::Set(SpaceIdWrap(space.clone())), + }) + .exec(&fixture.conn) + .await + .unwrap(); + + let tx = fixture.conn.begin().await.unwrap(); + lock_candidates(&tx, &[(space.clone(), blocked, 0)], "committing") + .await + .unwrap(); + tx.execute(Statement::from_sql_and_values( + DbBackend::Postgres, + "INSERT INTO kv_write (space, key, invocation, seq, epoch, epoch_seq, value, metadata) + VALUES ($1, 'a', $2, 1, $3, 1, $4, '{}')", + [ + space.to_string().into(), + Vec::::from(hash(b"invocation")).into(), + Vec::::from(hash(b"epoch")).into(), + Vec::::from(blocked).into(), + ], + )) + .await + .unwrap(); + finish_candidates(&tx, &[(space.clone(), blocked, 0)], "committing") + .await + .unwrap(); + tx.commit().await.unwrap(); + + assert_eq!( + collect_candidates(&fixture.conn, &store, 1, ORPHAN_GRACE) + .await + .unwrap(), + 1, + "a writer's commit cannot restore old claims ahead of an unrelated orphan" + ); + assert!(!store.contains(&space, &healthy, 0).await.unwrap()); + assert!(store.contains(&space, &blocked, 0).await.unwrap()); + } + + #[tokio::test] + async fn postgres_failed_attempt_cannot_erase_another_durable_registration() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_failed_attempt_cannot_erase_another_durable_registration", + ) + .await + else { + return; + }; + let store = MemoryStore::default(); + let space = space(); + let value = upload(&store, &space, b"same-hash overlap").await; + let blobs = [(space.clone(), value)]; + register(&fixture.conn, &blobs, "a").await; + register(&fixture.conn, &blobs, "b").await; + refund_failed(&fixture.conn, &store, &gen0(&blobs), "a") + .await + .unwrap(); + assert!(store.contains(&space, &value, 0).await.unwrap()); + let tx = fixture.conn.begin().await.unwrap(); + lock_candidates(&tx, &gen0(&blobs), "b").await.unwrap(); + tx.rollback().await.unwrap(); + refund_failed(&fixture.conn, &store, &gen0(&blobs), "b") + .await + .unwrap(); + assert!(!store.contains(&space, &value, 0).await.unwrap()); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + assert!(deleting_fences(&fixture.conn).await.unwrap().is_empty()); + } + + #[tokio::test] + async fn postgres_delete_in_flight_blocks_dedup_and_retries_lost_response() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_delete_in_flight_blocks_dedup_and_retries_lost_response", + ) + .await + else { + return; + }; + let inner = Arc::new(MemoryStore::default()); + let store = InterruptedDelete::new(inner.clone()); + let space = space(); + let value = upload(&inner, &space, b"delete-in-flight").await; + let blobs = [(space.clone(), value)]; + let old = register(&fixture.conn, &blobs, "a").await; + store.pause.store(true, Ordering::SeqCst); + let deleting = tokio::spawn({ + let conn = fixture.conn.clone(); + let store = store.clone(); + let old = old.clone(); + async move { refund_failed(&conn, &store, &old, "a").await } + }); + tokio::time::timeout(std::time::Duration::from_secs(5), store.entered.notified()) + .await + .unwrap(); + assert_eq!(deleting_fences(&fixture.conn).await.unwrap(), old); + let tx = fixture.conn.begin().await.unwrap(); + let unavailable = register_candidates(&tx, &blobs, "b").await.unwrap_err(); + assert!(is_candidate_unavailable(&unavailable)); + tx.rollback().await.unwrap(); + store.resume.notify_one(); + deleting.await.unwrap().unwrap(); + assert!(!inner.contains(&space, &value, 0).await.unwrap()); + let second = register(&fixture.conn, &blobs, "b").await; + assert_eq!(second[0].2, 1); + assert_eq!( + upload_generation(&inner, &space, b"delete-in-flight", 1).await, + value + ); + let tx = fixture.conn.begin().await.unwrap(); + lock_candidates(&tx, &second, "b").await.unwrap(); + tx.rollback().await.unwrap(); + assert!(inner.contains(&space, &value, 1).await.unwrap()); + refund_failed(&fixture.conn, &store, &second, "b") + .await + .unwrap(); + assert!(!inner.contains(&space, &value, 1).await.unwrap()); + + // A successful physical DELETE with a lost reply keeps only the + // deleting fence, which GC consumes without a second refund. + let third = register(&fixture.conn, &blobs, "c").await; + assert_eq!(third[0].2, 2); + assert_eq!( + upload_generation(&inner, &space, b"delete-in-flight", 2).await, + value + ); + store.lose_response.store(true, Ordering::SeqCst); + assert!(refund_failed(&fixture.conn, &store, &third, "c") + .await + .is_err()); + assert_eq!(deleting_fences(&fixture.conn).await.unwrap(), third); + assert!(!inner.contains(&space, &value, 2).await.unwrap()); + assert_eq!( + collect_candidates(&fixture.conn, &store, 10, Duration::ZERO) + .await + .unwrap(), + 0 + ); + assert!(deleting_fences(&fixture.conn).await.unwrap().is_empty()); + assert_eq!(inner.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn postgres_generation_survives_independent_pool_and_delayed_delete() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_generation_survives_independent_pool_and_delayed_delete", + ) + .await + else { + return; + }; + let inner = Arc::new(MemoryStore::default()); + let store = InterruptedDelete::new(inner.clone()); + let space = space(); + let value = upload(&inner, &space, b"independent-pool").await; + let blobs = [(space.clone(), value)]; + let first = register(&fixture.conn, &blobs, "first").await; + assert_eq!(first[0].2, 0); + store.pause.store(true, Ordering::SeqCst); + let deletion = tokio::spawn({ + let conn = fixture.conn.clone(); + let store = store.clone(); + let first = first.clone(); + async move { refund_failed(&conn, &store, &first, "first").await } + }); + tokio::time::timeout(std::time::Duration::from_secs(5), store.entered.notified()) + .await + .unwrap(); + store.pause.store(true, Ordering::SeqCst); + let second_entered = store.entered.notified(); + let delayed = tokio::spawn({ + let conn = fixture.conn.clone(); + let store = store.clone(); + async move { collect_candidates(&conn, &store, 1, Duration::ZERO).await } + }); + tokio::time::timeout(std::time::Duration::from_secs(5), second_entered) + .await + .unwrap(); + store.resume.notify_one(); + deletion.await.unwrap().unwrap(); + let schema: String = fixture + .conn + .query_one(Statement::from_string( + DbBackend::Postgres, + "SELECT current_schema() AS schema_name", + )) + .await + .unwrap() + .unwrap() + .try_get("", "schema_name") + .unwrap(); + let url = crate::test_support::postgres_test_url( + "postgres_generation_survives_independent_pool_and_delayed_delete", + ) + .unwrap(); + let mut options = ConnectOptions::new(url); + options.set_schema_search_path(schema); + let restarted = Database::connect(options).await.unwrap(); + let second = register(&restarted, &blobs, "second").await; + assert_eq!(second[0].2, 1); + assert_eq!( + upload_generation(&inner, &space, b"independent-pool", 1).await, + value + ); + let tx = restarted.begin().await.unwrap(); + lock_candidates(&tx, &second, "second").await.unwrap(); + tx.rollback().await.unwrap(); + store.resume.notify_one(); + delayed.await.unwrap().unwrap(); + assert!(inner.contains(&space, &value, 1).await.unwrap()); + assert_eq!( + ledger::Entity::find_by_id((SpaceIdWrap(space.clone()), value)) + .one(&fixture.conn) + .await + .unwrap() + .unwrap() + .generation, + 1 + ); + let tx = restarted.begin().await.unwrap(); + lock_candidates(&tx, &second, "second").await.unwrap(); + tx.rollback().await.unwrap(); + restarted.close().await.unwrap(); + } + + #[tokio::test] + async fn mysql_attempt_fence_preserves_other_writer_and_recovers_lost_delete() { + let Ok(url) = std::env::var("TINYCLOUD_TEST_MYSQL_URL") else { + eprintln!("skipping MySQL orphan fence (TINYCLOUD_TEST_MYSQL_URL unset)"); + return; + }; + let (prefix, _) = url.rsplit_once('/').expect("MySQL URL needs a database"); + let name = format!( + "tc735_orphan_{}_{}", + std::process::id(), + OffsetDateTime::now_utc().unix_timestamp_nanos() + ); + let admin = Database::connect(url.as_str()).await.unwrap(); + admin + .execute(Statement::from_string( + DbBackend::MySql, + format!("CREATE DATABASE {name}"), + )) + .await + .unwrap(); + let db = Database::connect(format!("{prefix}/{name}")).await.unwrap(); + // The older full MySQL migrator fails on unrelated indexed-column + // widths. Exercise the production model columns and transaction + // semantics here without altering unrelated migrations. + for ddl in [ + "CREATE TABLE kv_blob_generation ( + space VARCHAR(191) NOT NULL, value VARBINARY(34) NOT NULL, + generation BIGINT NOT NULL, PRIMARY KEY (space, value))", + "CREATE TABLE kv_orphan_candidate ( + space VARCHAR(191) NOT NULL, value VARBINARY(34) NOT NULL, + generation BIGINT NOT NULL, created_at DATETIME(6) NOT NULL, + deleting_at DATETIME(6) NULL, deletion_bytes BIGINT NULL, + PRIMARY KEY (space, value))", + "CREATE TABLE kv_orphan_attempt ( + space VARCHAR(191) NOT NULL, value VARBINARY(34) NOT NULL, + attempt_id VARCHAR(32) NOT NULL, generation BIGINT NOT NULL, + created_at DATETIME(6) NOT NULL, retry_after DATETIME(6) NULL, + PRIMARY KEY (space, value, attempt_id))", + "CREATE TABLE kv_write (space VARCHAR(191), value VARBINARY(34), + generation BIGINT NOT NULL DEFAULT 0, seq BIGINT NOT NULL, INDEX (space, value))", + "CREATE TABLE current_kv (space VARCHAR(191), value VARBINARY(34), + generation BIGINT NOT NULL DEFAULT 0, seq BIGINT NOT NULL, INDEX (space, value))", + ] { + db.execute(Statement::from_string(DbBackend::MySql, ddl)) + .await + .unwrap(); + } + let inner = Arc::new(MemoryStore::default()); + let store = InterruptedDelete::new(inner.clone()); + let space = space(); + let value = upload(&inner, &space, b"mysql-overlap").await; + let blobs = [(space.clone(), value)]; + register(&db, &blobs, "a").await; + register(&db, &blobs, "b").await; + refund_failed(&db, &store, &gen0(&blobs), "a") + .await + .unwrap(); + assert!(inner.contains(&space, &value, 0).await.unwrap()); + let tx = db.begin().await.unwrap(); + lock_candidates(&tx, &gen0(&blobs), "b").await.unwrap(); + tx.rollback().await.unwrap(); + store.lose_response.store(true, Ordering::SeqCst); + assert!(refund_failed(&db, &store, &gen0(&blobs), "b") + .await + .is_err()); + assert_eq!(deleting_fences(&db).await.unwrap(), gen0(&blobs)); + let tx = db.begin().await.unwrap(); + assert!(is_candidate_unavailable( + ®ister_candidates(&tx, &blobs, "c").await.unwrap_err(), + )); + tx.rollback().await.unwrap(); + assert_eq!( + collect_candidates(&db, &store, 10, Duration::ZERO) + .await + .unwrap(), + 0 + ); + let third = register(&db, &blobs, "c").await; + assert_eq!(third[0].2, 1); + assert_eq!( + upload_generation(&inner, &space, b"mysql-overlap", 1).await, + value + ); + refund_failed(&db, &store, &third, "c").await.unwrap(); + assert_eq!(inner.total_size(&space).await.unwrap(), Some(0)); + let expired = upload(&inner, &space, b"mysql-two-expired").await; + let expired_blob = [(space.clone(), expired)]; + register(&db, &expired_blob, "expired-a").await; + register(&db, &expired_blob, "expired-b").await; + age(&db, &space, expired).await; + assert_eq!( + collect_candidates(&db, &store, 256, ORPHAN_GRACE) + .await + .unwrap(), + 1 + ); + assert!(!inner.contains(&space, &expired, 0).await.unwrap()); + assert_eq!(inner.total_size(&space).await.unwrap(), Some(0)); + + let blocked = upload(&inner, &space, b"mysql-departing-writers").await; + seed_crashed_attempts(&db, &space, blocked, 256).await; + let live = register(&db, &[(space.clone(), blocked)], "mysql-live").await; + assert_eq!( + collect_candidates(&db, &store, 256, ORPHAN_GRACE) + .await + .unwrap(), + 0 + ); + let healthy = upload(&inner, &space, b"mysql-independent-orphan").await; + register(&db, &[(space.clone(), healthy)], "mysql-healthy").await; + kv_orphan_attempt::Entity::update_many() + .col_expr( + kv_orphan_attempt::Column::CreatedAt, + Expr::value(time::OffsetDateTime::now_utc() - Duration::hours(1)), + ) + .filter(kv_orphan_attempt::Column::Value.eq(healthy)) + .exec(&db) + .await + .unwrap(); + let successor = register(&db, &[(space.clone(), blocked)], "mysql-successor").await; + refund_failed(&db, &store, &live, "mysql-live") + .await + .unwrap(); + assert_eq!( + collect_candidates(&db, &store, 1, ORPHAN_GRACE) + .await + .unwrap(), + 1, + "a departing MySQL writer cannot starve a different hash" + ); + assert!(!inner.contains(&space, &healthy, 0).await.unwrap()); + assert!(inner.contains(&space, &blocked, 0).await.unwrap()); + refund_failed(&db, &store, &successor, "mysql-successor") + .await + .unwrap(); + db.close().await.unwrap(); + admin + .execute(Statement::from_string( + DbBackend::MySql, + format!("DROP DATABASE {name}"), + )) + .await + .unwrap(); + admin.close().await.unwrap(); + } + + #[tokio::test] + async fn successful_commit_removes_candidate_without_collecting_blob() { + let db = database().await; + let store = MemoryStore::default(); + let space = space(); + let value = hash(b"committed"); + register(&db, &[(space.clone(), value)], "a").await; + let tx = db.begin().await.unwrap(); + lock_candidates(&tx, &[(space.clone(), value, 0)], "a") + .await + .unwrap(); + assert_eq!(upload(&store, &space, b"committed").await, value); + reference(&tx, "kv_write", &space, value, 0).await; + finish_candidates(&tx, &[(space.clone(), value, 0)], "a") + .await + .unwrap(); + tx.commit().await.unwrap(); + assert_eq!( + collect_candidates(&db, &store, 10, time::Duration::ZERO) + .await + .unwrap(), + 0 + ); + assert!(store.contains(&space, &value, 0).await.unwrap()); + } + + #[tokio::test] + async fn failed_old_metadata_read_does_not_starve_next_attempt() { + let db = database().await; + let inner = Arc::new(MemoryStore::default()); + let store = InterruptedDelete::new(inner.clone()); + let space = space(); + let old = upload(&inner, &space, b"old failed HEAD").await; + let next = upload(&inner, &space, b"next after HEAD").await; + register(&db, &[(space.clone(), old)], "a").await; + register(&db, &[(space.clone(), next)], "b").await; + age(&db, &space, old).await; + age(&db, &space, next).await; + store.fail_read.store(true, Ordering::SeqCst); + assert!(collect_candidates(&db, &store, 1, Duration::hours(1)) + .await + .is_err()); + assert!(inner.contains(&space, &old, 0).await.unwrap()); + assert_eq!( + collect_candidates(&db, &store, 1, Duration::hours(1)) + .await + .unwrap(), + 1 + ); + assert!(!inner.contains(&space, &next, 0).await.unwrap()); + } +} diff --git a/tinycloud-core/src/kv_sync.rs b/tinycloud-core/src/kv_sync.rs index 17183048..168d5494 100644 --- a/tinycloud-core/src/kv_sync.rs +++ b/tinycloud-core/src/kv_sync.rs @@ -896,7 +896,7 @@ pub(crate) mod tests { use crate::storage::memory::{MemoryStaging, MemoryStore}; use crate::storage::HashBuffer; use crate::InvocationOutcome; - use futures::io::AsyncWriteExt; + use futures::io::{AsyncReadExt, AsyncWriteExt}; use sea_orm::{ ActiveModelTrait, ActiveValue::Set, ConnectOptions, Database, DatabaseConnection, TransactionTrait, @@ -1694,6 +1694,445 @@ pub(crate) mod tests { ); } + #[tokio::test] + async fn unauthorized_put_never_uploads_a_blob() { + use crate::storage::ImmutableReadStore; + let store = MemoryStore::default(); + let db = SpaceDatabase::new( + Database::connect(ConnectOptions::new("sqlite::memory:".to_string())) + .await + .unwrap(), + store.clone(), + StaticSecret::new([0u8; 32].to_vec()).unwrap(), + ) + .await + .unwrap(); + let signer = Owner::generate(); + let other = Owner::generate(); + host(&db, &other).await; + let op = Op::Put("notes/forbidden", b"unauthorized"); + let result = invoke_in( + &db, + &signer, + &[(&other.space, &op)], + KvInvokeOptions::default(), + ) + .await; + assert!( + result.is_err(), + "the signer cannot write another owner's space" + ); + assert!( + !store + .contains(&other.space, &crate::hash::hash(b"unauthorized"), 0) + .await + .unwrap(), + "authorization failure must not create a blob" + ); + } + + #[tokio::test] + async fn gen_zero_legacy_read_and_reuploaded_generation_get_and_range() { + use crate::storage::{ByteRangeSpec, ImmutableReadStore, ImmutableWriteStore, RangeRead}; + let store = MemoryStore::default(); + let db = SpaceDatabase::new( + Database::connect(ConnectOptions::new("sqlite::memory:".to_string())) + .await + .unwrap(), + store.clone(), + StaticSecret::new([0u8; 32].to_vec()).unwrap(), + ) + .await + .unwrap(); + let owner = Owner::generate(); + host(&db, &owner).await; + + let original = run(&db, &owner, &[Op::Put("notes/legacy", b"bare")]).await[0]; + let legacy_key: Path = "notes/legacy".parse().unwrap(); + let legacy = current_kv::Entity::find_by_id(( + SpaceIdWrap(owner.space.clone()), + legacy_key.clone().into(), + )) + .one(db.connection()) + .await + .unwrap() + .unwrap(); + assert_eq!(legacy.generation, 0); + assert_eq!(legacy.value, original); + assert_eq!( + store.read_to_vec(&owner.space, &original, 0).await.unwrap(), + Some(b"bare".to_vec()), + "the older bare-key reader can still read a first-generation row" + ); + + let bytes = b"regenerated"; + let hash = crate::hash::hash(bytes); + let tx = db.connection().begin().await.unwrap(); + let old = crate::kv_orphans::register_candidates( + &tx, + &[(owner.space.clone(), hash)], + "orphan-before-resave", + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + assert_eq!(old[0].2, 0); + let mut staged = HashBuffer::new(Vec::new()); + staged.write_all(bytes).await.unwrap(); + store.persist(&owner.space, staged, 0).await.unwrap(); + crate::kv_orphans::refund_failed(db.connection(), &store, &old, "orphan-before-resave") + .await + .unwrap(); + + assert_eq!( + run(&db, &owner, &[Op::Put("notes/resaved", bytes)]).await[0], + hash + ); + let new_key: Path = "notes/resaved".parse().unwrap(); + let new_row = current_kv::Entity::find_by_id(( + SpaceIdWrap(owner.space.clone()), + new_key.clone().into(), + )) + .one(db.connection()) + .await + .unwrap() + .unwrap(); + assert_eq!(new_row.generation, 1); + assert!(!store.contains(&owner.space, &hash, 0).await.unwrap()); + let (_, _, content) = db.kv_get(&owner.space, &new_key).await.unwrap().unwrap(); + let (_, mut reader) = content.into_inner(); + let mut read = Vec::new(); + reader.read_to_end(&mut read).await.unwrap(); + assert_eq!(read, bytes); + let (_, _, range) = db + .kv_get_range( + &owner.space, + &new_key, + ByteRangeSpec::Inclusive { start: 2, end: 5 }, + ) + .await + .unwrap() + .unwrap(); + let RangeRead::Content { content, .. } = range else { + panic!("expected generated range"); + }; + let (_, mut reader) = content.into_inner(); + let mut read = Vec::new(); + reader.read_to_end(&mut read).await.unwrap(); + assert_eq!(read, &bytes[2..=5]); + } + + #[tokio::test] + async fn rejected_put_refunds_quota_without_deleting_committed_hash() { + use crate::storage::{ImmutableReadStore, StoreSize}; + let store = MemoryStore::default(); + let db = SpaceDatabase::new( + Database::connect(ConnectOptions::new("sqlite::memory:".to_string())) + .await + .unwrap(), + store.clone(), + StaticSecret::new([0u8; 32].to_vec()).unwrap(), + ) + .await + .unwrap(); + let owner = Owner::generate(); + host(&db, &owner).await; + let committed = run(&db, &owner, &[Op::Put("notes/a", b"kept")]).await[0]; + let mut options = KvInvokeOptions::default(); + options.preconditions.insert( + (owner.space.clone(), "notes/a".parse().unwrap()), + KvPrecondition::DoesNotExist, + ); + let rejected = invoke(&db, &owner, &[Op::Put("notes/a", b"rejected")], options) + .await + .unwrap_err(); + assert!(rejected.contains("precondition"), "{rejected}"); + let orphan = crate::hash::hash(b"rejected"); + assert!( + !store.contains(&owner.space, &orphan, 0).await.unwrap(), + "a known precondition failure must remove its unreferenced upload" + ); + assert!(store.contains(&owner.space, &committed, 0).await.unwrap()); + assert_eq!(store.total_size(&owner.space).await.unwrap(), Some(4)); + let duplicate = invoke(&db, &owner, &[Op::Put("notes/a", b"kept")], { + let mut options = KvInvokeOptions::default(); + options.preconditions.insert( + (owner.space.clone(), "notes/a".parse().unwrap()), + KvPrecondition::DoesNotExist, + ); + options + }) + .await + .unwrap_err(); + assert!(duplicate.contains("precondition"), "{duplicate}"); + assert!(store.contains(&owner.space, &committed, 0).await.unwrap()); + assert_eq!(store.total_size(&owner.space).await.unwrap(), Some(4)); + } + + #[tokio::test] + async fn failed_database_write_refunds_uploaded_blob_immediately() { + use crate::storage::{ImmutableReadStore, StoreSize}; + use sea_orm::{ConnectionTrait, DbBackend, Statement}; + let store = MemoryStore::default(); + let db = SpaceDatabase::new( + Database::connect(ConnectOptions::new("sqlite::memory:".to_string())) + .await + .unwrap(), + store.clone(), + StaticSecret::new([0u8; 32].to_vec()).unwrap(), + ) + .await + .unwrap(); + let owner = Owner::generate(); + host(&db, &owner).await; + db.connection() + .execute(Statement::from_string( + DbBackend::Sqlite, + "CREATE TRIGGER fail_kv_write BEFORE INSERT ON kv_write BEGIN SELECT RAISE(FAIL, 'blocked'); END", + )) + .await + .unwrap(); + let error = invoke( + &db, + &owner, + &[Op::Put("notes/failure", b"uploaded")], + KvInvokeOptions::default(), + ) + .await + .unwrap_err(); + assert!(error.contains("blocked"), "{error}"); + assert!(!store + .contains(&owner.space, &crate::hash::hash(b"uploaded"), 0) + .await + .unwrap()); + assert_eq!(store.total_size(&owner.space).await.unwrap(), Some(0)); + } + + /// A put of two blobs whose later-registered blob has a pending deletion is + /// refused, and nothing of it is claimed: the earlier blob keeps no claim, + /// fence or ledger row. + #[tokio::test] + async fn two_blob_put_with_a_pending_deletion_claims_nothing() { + use crate::models::{kv_blob_generation, kv_orphan_attempt, kv_orphan_candidate}; + use crate::storage::ImmutableReadStore; + use crate::types::SpaceIdWrap; + use sea_orm::{ActiveModelTrait, ActiveValue::Set, EntityTrait}; + + let store = MemoryStore::default(); + let db = SpaceDatabase::new( + Database::connect(ConnectOptions::new("sqlite::memory:".to_string())) + .await + .unwrap(), + store.clone(), + StaticSecret::new([0u8; 32].to_vec()).unwrap(), + ) + .await + .unwrap(); + let owner = Owner::generate(); + host(&db, &owner).await; + let (one, two): (&[u8], &[u8]) = (b"two-blob first", b"two-blob second"); + let (early, late) = if crate::hash::hash(one) < crate::hash::hash(two) { + (one, two) + } else { + (two, one) + }; + let space = SpaceIdWrap(owner.space.clone()); + let late_hash = crate::hash::hash(late); + let now = time::OffsetDateTime::now_utc(); + kv_blob_generation::ActiveModel { + space: Set(space.clone()), + value: Set(late_hash), + generation: Set(0), + } + .insert(db.connection()) + .await + .unwrap(); + kv_orphan_candidate::ActiveModel { + space: Set(space.clone()), + value: Set(late_hash), + generation: Set(0), + created_at: Set(now), + deleting_at: Set(Some(now)), + deletion_bytes: Set(Some(1)), + } + .insert(db.connection()) + .await + .unwrap(); + + invoke( + &db, + &owner, + &[Op::Put("notes/early", early), Op::Put("notes/late", late)], + KvInvokeOptions::default(), + ) + .await + .unwrap_err(); + + let early_hash = crate::hash::hash(early); + let claims = kv_orphan_attempt::Entity::find() + .all(db.connection()) + .await + .unwrap(); + let fences = kv_orphan_candidate::Entity::find() + .all(db.connection()) + .await + .unwrap(); + let ledger = kv_blob_generation::Entity::find() + .all(db.connection()) + .await + .unwrap(); + assert!(claims.iter().all(|row| row.value != early_hash)); + assert!(fences.iter().all(|row| row.value != early_hash)); + assert!(ledger.iter().all(|row| row.value != early_hash)); + assert!( + fences.iter().any(|row| row.value == late_hash), + "the pending deletion is intact" + ); + assert!(!store.contains(&owner.space, &early_hash, 0).await.unwrap()); + } + + /// Simulate an object store that accepted the PUT but lost its success + /// response. The caller must still delete and refund the possible object. + #[derive(Clone, Default)] + struct AmbiguousUploadStore(MemoryStore); + + #[async_trait::async_trait] + impl crate::storage::StorageSetup for AmbiguousUploadStore { + type Error = std::io::Error; + async fn create(&self, space: &SpaceId) -> Result<(), Self::Error> { + self.0.create(space).await + } + } + + #[async_trait::async_trait] + impl crate::storage::ImmutableReadStore for AmbiguousUploadStore { + type Error = std::io::Error; + type Readable = futures::io::Cursor>; + async fn contains( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result { + self.0.contains(space, id, generation).await + } + async fn read( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result>, Self::Error> { + self.0.read(space, id, generation).await + } + async fn read_range( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + range: crate::storage::ByteRangeSpec, + ) -> Result>, Self::Error> { + self.0.read_range(space, id, generation, range).await + } + } + + #[async_trait::async_trait] + impl crate::storage::ImmutableWriteStore for AmbiguousUploadStore { + type Error = std::io::Error; + async fn persist( + &self, + space: &SpaceId, + staged: HashBuffer>, + generation: i64, + ) -> Result { + self.0.persist(space, staged, generation).await?; + Err(std::io::Error::other("lost PUT response")) + } + } + + #[async_trait::async_trait] + impl crate::storage::ImmutableDeleteStore for AmbiguousUploadStore { + type Error = std::io::Error; + async fn remove( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result, Self::Error> { + self.0.remove(space, id, generation).await + } + async fn cleanup_stages( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result<(), Self::Error> { + self.0.cleanup_stages(space, id, generation).await + } + } + + #[tokio::test] + async fn failed_upload_refunds_physical_bytes_immediately() { + use crate::storage::{ImmutableReadStore, StoreSize}; + let store = AmbiguousUploadStore::default(); + let db = SpaceDatabase::new( + Database::connect(ConnectOptions::new("sqlite::memory:".to_string())) + .await + .unwrap(), + store.clone(), + StaticSecret::new([0u8; 32].to_vec()).unwrap(), + ) + .await + .unwrap(); + let owner = Owner::generate(); + space::ActiveModel { + id: Set(SpaceIdWrap(owner.space.clone())), + } + .insert(db.connection()) + .await + .unwrap(); + let op = Op::Put("notes/failure", b"uploaded"); + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(b"uploaded").await.unwrap(); + let mut inputs = std::collections::HashMap::new(); + inputs.insert( + ( + owner.space.clone(), + "notes/failure".parse::().unwrap(), + ), + (Metadata(Default::default()), stage), + ); + let error = match db + .invoke::(owner.invocation_in(&[(&owner.space, &op)]), inputs) + .await + { + Ok(_) => panic!("the upload response was lost"), + Err(error) => error.to_string(), + }; + assert!(error.contains("lost PUT response"), "{error}"); + assert!(!store + .contains(&owner.space, &crate::hash::hash(b"uploaded"), 0) + .await + .unwrap()); + assert_eq!(store.0.total_size(&owner.space).await.unwrap(), Some(0)); + assert_eq!( + crate::kv_orphans::collect_candidates( + db.connection(), + &store, + 1, + time::Duration::ZERO, + ) + .await + .unwrap(), + 0, + "the immediate failure path consumed the candidate" + ); + assert_eq!( + store.0.total_size(&owner.space).await.unwrap(), + Some(0), + "a later GC pass must not refund the same object twice" + ); + } + // ── PostgreSQL (run in CI's PG16 job; `TINYCLOUD_TEST_POSTGRES_URL`) ── /// Drops its schema when it goes out of scope, including while a failed @@ -1741,6 +2180,7 @@ pub(crate) mod tests { /// ("sorry, too many clients already" at `CREATE SCHEMA`). struct PostgresFixture { dbs: Vec, + store: MemoryStore, _schema: SchemaGuard, // Last, so the schema is dropped before the next fixture starts. _serial: tokio::sync::MutexGuard<'static, ()>, @@ -1772,6 +2212,7 @@ pub(crate) mod tests { schema: schema.clone(), }; let mut dbs = Vec::with_capacity(instances); + let store = MemoryStore::default(); for _ in 0..instances { let mut options = ConnectOptions::new(url.clone()); options @@ -1783,7 +2224,7 @@ pub(crate) mod tests { Database::connect(options) .await .expect("connect to isolated TC-732 schema"), - MemoryStore::default(), + store.clone(), StaticSecret::new([0u8; 32].to_vec()).unwrap(), ) .await @@ -1792,12 +2233,58 @@ pub(crate) mod tests { } Some(Self { dbs, + store, _schema: guard, _serial: serial, }) } } + /// PG's per-space lock may be held by another process, but independent + /// content-addressed uploads must still finish before acquiring it. + #[tokio::test] + async fn postgres_blob_persists_before_space_sequence_lock() { + use crate::storage::ImmutableReadStore; + let Some(fixture) = + PostgresFixture::new("postgres_blob_persists_before_space_sequence_lock", 2).await + else { + return; + }; + let owner = std::sync::Arc::new(Owner::generate()); + host(&fixture.dbs[0], &owner).await; + let holder = fixture.dbs[0].connection().begin().await.unwrap(); + lock_space_sequences(&holder, [&owner.space]).await.unwrap(); + let db = fixture.dbs[1].clone(); + let writer_owner = owner.clone(); + let writer = tokio::spawn(async move { + run( + &db, + &writer_owner, + &[Op::Put("notes/held", b"persist-before-lock")], + ) + .await + }); + let hash = crate::hash::hash(b"persist-before-lock"); + let visible = tokio::time::timeout(std::time::Duration::from_secs(3), async { + loop { + if fixture + .store + .contains(&owner.space, &hash, 0) + .await + .unwrap() + { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + }) + .await + .is_ok(); + holder.commit().await.unwrap(); + writer.await.unwrap(); + assert!(visible, "blob upload must not wait for the sequence lock"); + } + /// `(seq, epoch bytes, epoch_seq)`: the order the feed promises. fn order_key(pos: &KvSyncAnchor) -> (i64, Vec, i64) { (pos.seq, Vec::::from(pos.epoch), pos.epoch_seq) @@ -2192,6 +2679,7 @@ pub(crate) mod tests { #[tokio::test] async fn postgres_conditional_put_is_atomic_under_concurrency() { use crate::models::kv_write; + use crate::storage::StoreSize; const INSTANCES: usize = 4; const CONTENDERS: usize = 32; let Some(fixture) = PostgresFixture::new( @@ -2268,6 +2756,7 @@ pub(crate) mod tests { who(&created) ); let mut committed = vec![created[0].1]; + let mut committed_bytes = format!("create-{}", created[0].0).len() as u64; // Before the first replace no etag has been replaced yet: present // one that was never written. let mut stale = crate::hash::hash(b"never written"); @@ -2290,6 +2779,7 @@ pub(crate) mod tests { ); stale = current; committed.push(replaced[0].1); + committed_bytes += format!("replace{round}-{}", replaced[0].0).len() as u64; } run(&fixture.dbs[1], &owner, &[Op::Del("notes/x")]).await; let recreated = race("recreate".into(), create()).await; @@ -2300,6 +2790,7 @@ pub(crate) mod tests { who(&recreated) ); committed.push(recreated[0].1); + committed_bytes += format!("recreate-{}", recreated[0].0).len() as u64; let over_live = race("create-over-live".into(), create()).await; assert!( over_live.is_empty(), @@ -2329,6 +2820,11 @@ pub(crate) mod tests { writes.iter().map(|write| write.value).collect::>(), committed ); + assert_eq!( + fixture.store.total_size(&owner.space).await.unwrap(), + Some(committed_bytes), + "failed contenders must refund their physical bytes without touching committed blobs" + ); assert_eq!( assert_linear_history(conn, &owner.space, "conditional").await, committed.len() + 1 diff --git a/tinycloud-core/src/lib.rs b/tinycloud-core/src/lib.rs index 4f240064..2443cdb9 100644 --- a/tinycloud-core/src/lib.rs +++ b/tinycloud-core/src/lib.rs @@ -17,6 +17,7 @@ pub mod events; pub mod failure_diagnostic; pub mod hash; pub mod keys; +pub mod kv_orphans; pub mod kv_sync; pub mod manifest; pub mod meeting_legacy_guard; diff --git a/tinycloud-core/src/migrations/m20260724_010000_current_kv.rs b/tinycloud-core/src/migrations/m20260724_010000_current_kv.rs index 3387b81a..b8a5d2ab 100644 --- a/tinycloud-core/src/migrations/m20260724_010000_current_kv.rs +++ b/tinycloud-core/src/migrations/m20260724_010000_current_kv.rs @@ -119,6 +119,7 @@ fn latest_writes() -> SelectStatement { (write.clone(), kv_write::Column::Metadata), ]) .expr_as(Expr::exists(exact_tombstone), current_kv::Column::Deleted) + .expr_as(Expr::val(0_i64), current_kv::Column::Generation) .from_as(kv_write::Entity, write) .cond_where(Condition::all().not().add(Expr::exists(newer_write))); query.to_owned() @@ -141,6 +142,7 @@ async fn backfill_and_validate(manager: &SchemaManager<'_>) -> Result<(), DbErr> current_kv::Column::Value, current_kv::Column::Metadata, current_kv::Column::Deleted, + current_kv::Column::Generation, ]) .select_from(expected_select.clone()) .map_err(|error| DbErr::Migration(error.to_string()))?; @@ -175,7 +177,10 @@ mod tests { use crate::models::{actor, epoch, invocation, space}; use crate::relationships::event_order; use crate::types::{Metadata, SpaceIdWrap}; - use sea_orm::{ActiveModelTrait, ActiveValue::Set, ConnectOptions, Database, PaginatorTrait}; + use sea_orm::{ + ActiveModelTrait, ActiveValue::Set, ConnectOptions, ConnectionTrait, Database, DbBackend, + PaginatorTrait, Statement, + }; use sea_orm_migration::MigratorTrait; use std::collections::BTreeMap; use time::OffsetDateTime; @@ -233,12 +238,13 @@ mod tests { epoch: epoch_id, epoch_seq: 0, value: hash(format!("value-{label}").as_bytes()), + generation: 0, metadata: Metadata(BTreeMap::from([("label".to_string(), label.to_string())])), }; - kv_write::ActiveModel::from(write.clone()) - .insert(db) - .await - .unwrap(); + let mut legacy = kv_write::ActiveModel::from(write.clone()); + // The column is omitted here; a fresh table defaults it to zero. + legacy.generation = sea_orm::ActiveValue::NotSet; + legacy.insert(db).await.unwrap(); write } @@ -340,4 +346,65 @@ mod tests { "backfill must preserve immutable delete history" ); } + #[tokio::test] + async fn postgres_populated_pre_current_kv_backfill_decodes_generation() { + let Some(url) = crate::test_support::postgres_test_url( + "postgres_populated_pre_current_kv_backfill_decodes_generation", + ) else { + return; + }; + let admin = Database::connect(url.clone()).await.unwrap(); + let schema = format!( + "tc735_old_kv_{}_{}", + std::process::id(), + OffsetDateTime::now_utc().unix_timestamp_nanos() + ); + admin + .execute(Statement::from_string( + DbBackend::Postgres, + format!("CREATE SCHEMA {schema}"), + )) + .await + .unwrap(); + let mut options = ConnectOptions::new(url); + options.set_schema_search_path(schema.clone()); + let db = Database::connect(options).await.unwrap(); + Migrator::up(&db, Some(13)).await.unwrap(); + let did: DIDBuf = DID_METHODS + .generate(&JWK::generate_ed25519().unwrap(), "key") + .unwrap(); + let space_id = SpaceId::new(did, "migration".parse().unwrap()); + actor::ActiveModel { + id: Set("did:key:migration".to_string()), + } + .insert(&db) + .await + .unwrap(); + space::ActiveModel { + id: Set(SpaceIdWrap(space_id.clone())), + } + .insert(&db) + .await + .unwrap(); + let write = insert_write(&db, &space_id, "older-key", "older-value", 1).await; + let backfill = Migration.up(&SchemaManager::new(&db)).await; + let current = if backfill.is_ok() { + current_kv::Entity::find().one(&db).await.unwrap() + } else { + None + }; + db.close().await.unwrap(); + admin + .execute(Statement::from_string( + DbBackend::Postgres, + format!("DROP SCHEMA {schema} CASCADE"), + )) + .await + .unwrap(); + backfill.unwrap(); + let current = current.unwrap(); + assert_eq!(current.invocation, write.invocation); + assert_eq!(current.value, write.value); + assert_eq!(current.generation, 0); + } } diff --git a/tinycloud-core/src/migrations/m20260725_000000_request_path_indexes.rs b/tinycloud-core/src/migrations/m20260725_000000_request_path_indexes.rs index 8d9ea32f..77c9fb06 100644 --- a/tinycloud-core/src/migrations/m20260725_000000_request_path_indexes.rs +++ b/tinycloud-core/src/migrations/m20260725_000000_request_path_indexes.rs @@ -543,6 +543,8 @@ mod tests { epoch: Set(epoch_id), epoch_seq: Set(i), value: Set(hash(format!("tc282-value-{i}").as_bytes())), + // The column is omitted; it defaults to zero. + generation: sea_orm::ActiveValue::NotSet, metadata: Set(Metadata(BTreeMap::new())), } .insert(db) diff --git a/tinycloud-core/src/migrations/m20261023_010000_kv_orphan_candidates.rs b/tinycloud-core/src/migrations/m20261023_010000_kv_orphan_candidates.rs new file mode 100644 index 00000000..77a2fb07 --- /dev/null +++ b/tinycloud-core/src/migrations/m20261023_010000_kv_orphan_candidates.rs @@ -0,0 +1,245 @@ +use sea_orm::Schema; +use sea_orm_migration::prelude::*; + +use crate::models::{kv_blob_generation, kv_orphan_attempt, kv_orphan_candidate}; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + /// Re-runnable on PostgreSQL and SQLite: every table and index is created + /// if absent and the `generation` columns are added only where missing, so + /// applying the migration again over existing tables (and their rows) + /// changes nothing. MySQL index creation has no such guard. + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + let schema = Schema::new(manager.get_database_backend()); + manager + .create_table( + schema + .create_table_from_entity(kv_blob_generation::Entity) + .if_not_exists() + .to_owned(), + ) + .await?; + for table in ["kv_write", "current_kv"] { + // A fresh install builds the initial table from today's entity, + // while an upgraded schema still has the pre-generation column set. + if !manager.has_column(table, "generation").await? { + manager + .alter_table( + Table::alter() + .table(Alias::new(table)) + .add_column( + ColumnDef::new(Alias::new("generation")) + .big_integer() + .not_null() + .default(0), + ) + .to_owned(), + ) + .await?; + } + } + manager + .create_table( + schema + .create_table_from_entity(kv_orphan_candidate::Entity) + .if_not_exists() + .to_owned(), + ) + .await?; + manager + .create_table( + schema + .create_table_from_entity(kv_orphan_attempt::Entity) + .if_not_exists() + .to_owned(), + ) + .await?; + manager + .create_index( + Index::create() + .if_not_exists() + .name("idx_kv_orphan_candidate_deleting") + .table(kv_orphan_candidate::Entity) + .col(kv_orphan_candidate::Column::DeletingAt) + .col(kv_orphan_candidate::Column::Space) + .col(kv_orphan_candidate::Column::Value) + .to_owned(), + ) + .await?; + manager + .create_index( + Index::create() + .if_not_exists() + .name("idx_kv_orphan_attempt_age") + .table(kv_orphan_attempt::Entity) + .col(kv_orphan_attempt::Column::CreatedAt) + .col(kv_orphan_attempt::Column::Space) + .col(kv_orphan_attempt::Column::Value) + .to_owned(), + ) + .await?; + manager + .create_index( + Index::create() + .if_not_exists() + .name("idx_kv_orphan_attempt_retry") + .table(kv_orphan_attempt::Entity) + .col(kv_orphan_attempt::Column::RetryAfter) + .col(kv_orphan_attempt::Column::CreatedAt) + .col(kv_orphan_attempt::Column::Space) + .col(kv_orphan_attempt::Column::Value) + .to_owned(), + ) + .await?; + Ok(()) + } + + async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { + for table in ["current_kv", "kv_write"] { + manager + .alter_table( + Table::alter() + .table(Alias::new(table)) + .drop_column(Alias::new("generation")) + .to_owned(), + ) + .await?; + } + manager + .drop_table(Table::drop().table(kv_orphan_attempt::Entity).to_owned()) + .await?; + manager + .drop_table(Table::drop().table(kv_orphan_candidate::Entity).to_owned()) + .await?; + manager + .drop_table(Table::drop().table(kv_blob_generation::Entity).to_owned()) + .await + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::hash::hash; + use crate::migrations::Migrator; + use crate::types::SpaceIdWrap; + use sea_orm::{ + ActiveModelTrait, ActiveValue::Set, ConnectOptions, ConnectionTrait, Database, + DatabaseConnection, EntityTrait, Statement, + }; + use sea_orm_migration::MigratorTrait; + use time::OffsetDateTime; + use tinycloud_auth::{resolver::DID_METHODS, resource::SpaceId, ssi::jwk::JWK}; + + const VERSION: &str = "m20261023_010000_kv_orphan_candidates"; + + async fn seed(db: &DatabaseConnection) { + let did = DID_METHODS + .generate(&JWK::generate_ed25519().unwrap(), "key") + .unwrap(); + let space = SpaceIdWrap(SpaceId::new(did, "files".parse().unwrap())); + let value = hash(b"rerun"); + let now = OffsetDateTime::now_utc(); + kv_blob_generation::ActiveModel { + space: Set(space.clone()), + value: Set(value), + generation: Set(3), + } + .insert(db) + .await + .unwrap(); + kv_orphan_candidate::ActiveModel { + space: Set(space.clone()), + value: Set(value), + generation: Set(3), + created_at: Set(now), + deleting_at: Set(None), + deletion_bytes: Set(None), + } + .insert(db) + .await + .unwrap(); + kv_orphan_attempt::ActiveModel { + space: Set(space), + value: Set(value), + attempt_id: Set("attempt".to_owned()), + generation: Set(3), + created_at: Set(now), + retry_after: Set(None), + } + .insert(db) + .await + .unwrap(); + } + + async fn counts(db: &DatabaseConnection) -> (usize, usize, usize) { + ( + kv_blob_generation::Entity::find() + .all(db) + .await + .unwrap() + .len(), + kv_orphan_candidate::Entity::find() + .all(db) + .await + .unwrap() + .len(), + kv_orphan_attempt::Entity::find() + .all(db) + .await + .unwrap() + .len(), + ) + } + + /// Applying the migration again over its own tables, with rows in them, + /// changes nothing, whether directly or through the migrator after its + /// history row is deleted. + async fn assert_rerunnable(db: &DatabaseConnection) { + seed(db).await; + let backend = db.get_database_backend(); + Migration.up(&SchemaManager::new(db)).await.unwrap(); + assert_eq!(counts(db).await, (1, 1, 1)); + + db.execute(Statement::from_string( + backend, + format!("DELETE FROM seaql_migrations WHERE version = '{VERSION}'"), + )) + .await + .unwrap(); + Migrator::up(db, None).await.unwrap(); + assert_eq!(counts(db).await, (1, 1, 1)); + let history = db + .query_one(Statement::from_string( + backend, + format!("SELECT version FROM seaql_migrations WHERE version = '{VERSION}'"), + )) + .await + .unwrap(); + assert!(history.is_some(), "the history row is recorded again"); + } + + #[tokio::test] + async fn sqlite_migration_reruns_over_populated_tables() { + let db = Database::connect(ConnectOptions::new("sqlite::memory:".to_owned())) + .await + .unwrap(); + Migrator::up(&db, None).await.unwrap(); + assert_rerunnable(&db).await; + } + + #[tokio::test] + async fn postgres_migration_reruns_over_populated_tables() { + let Some(fixture) = crate::test_support::PostgresTestSchema::new( + "postgres_migration_reruns_over_populated_tables", + ) + .await + else { + return; + }; + assert_rerunnable(&fixture.conn).await; + } +} diff --git a/tinycloud-core/src/migrations/mod.rs b/tinycloud-core/src/migrations/mod.rs index 15a82db0..4cd92e83 100644 --- a/tinycloud-core/src/migrations/mod.rs +++ b/tinycloud-core/src/migrations/mod.rs @@ -31,6 +31,7 @@ pub mod m20261018_000000_epoch_head; pub mod m20261020_000000_current_kv_key_c_index; pub mod m20261021_000000_sql_store_manifest; pub mod m20261022_000000_encryption_network_key; +pub mod m20261023_010000_kv_orphan_candidates; pub struct Migrator; @@ -70,6 +71,7 @@ impl MigratorTrait for Migrator { Box::new(m20261020_000000_current_kv_key_c_index::Migration), Box::new(m20261021_000000_sql_store_manifest::Migration), Box::new(m20261022_000000_encryption_network_key::Migration), + Box::new(m20261023_010000_kv_orphan_candidates::Migration), ] } } diff --git a/tinycloud-core/src/models/current_kv.rs b/tinycloud-core/src/models/current_kv.rs index 2c5ce082..a033744a 100644 --- a/tinycloud-core/src/models/current_kv.rs +++ b/tinycloud-core/src/models/current_kv.rs @@ -14,6 +14,8 @@ pub struct Model { pub epoch: Hash, pub epoch_seq: i64, pub value: Hash, + #[sea_orm(default_value = 0)] + pub generation: i64, pub metadata: Metadata, pub deleted: bool, } diff --git a/tinycloud-core/src/models/invocation.rs b/tinycloud-core/src/models/invocation.rs index 99b51569..3f0ca680 100644 --- a/tinycloud-core/src/models/invocation.rs +++ b/tinycloud-core/src/models/invocation.rs @@ -794,6 +794,7 @@ async fn save( VersionedOperation::KvWrite { key, value, + generation, metadata, space, seq, @@ -804,6 +805,7 @@ async fn save( invocation: hash, key: key.clone().into(), value: *value, + generation: *generation, space: space.clone().into(), metadata: metadata.clone(), seq: *seq, @@ -945,6 +947,7 @@ pub(crate) async fn upsert_current_kv( epoch: write.epoch, epoch_seq: write.epoch_seq, value: write.value, + generation: write.generation, metadata: write.metadata, deleted: false, }; @@ -958,6 +961,7 @@ pub(crate) async fn upsert_current_kv( const NEWER: &str = "VALUES(`seq`) > `seq` OR (VALUES(`seq`) = `seq` AND VALUES(`epoch`) > `epoch`) OR (VALUES(`seq`) = `seq` AND VALUES(`epoch`) = `epoch` AND VALUES(`epoch_seq`) > `epoch_seq`) OR (VALUES(`seq`) = `seq` AND VALUES(`epoch`) = `epoch` AND VALUES(`epoch_seq`) = `epoch_seq` AND `deleted`)"; for (column, incoming) in [ (current_kv::Column::Invocation, "invocation"), + (current_kv::Column::Generation, "generation"), (current_kv::Column::Value, "value"), (current_kv::Column::Metadata, "metadata"), (current_kv::Column::Deleted, "deleted"), @@ -980,6 +984,7 @@ pub(crate) async fn upsert_current_kv( current_kv::Column::Epoch, current_kv::Column::EpochSeq, current_kv::Column::Value, + current_kv::Column::Generation, current_kv::Column::Metadata, current_kv::Column::Deleted, ]) @@ -1235,6 +1240,7 @@ mod tests { epoch: crate::hash::hash(format!("epoch-{label}").as_bytes()), epoch_seq: 0, value: crate::hash::hash(format!("value-{label}").as_bytes()), + generation: 0, metadata: crate::types::Metadata(BTreeMap::from([( "label".to_string(), label.to_string(), diff --git a/tinycloud-core/src/models/kv_blob_generation.rs b/tinycloud-core/src/models/kv_blob_generation.rs new file mode 100644 index 00000000..b421e7db --- /dev/null +++ b/tinycloud-core/src/models/kv_blob_generation.rs @@ -0,0 +1,18 @@ +use crate::hash::Hash; +use crate::types::SpaceIdWrap; +use sea_orm::entity::prelude::*; + +#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)] +#[sea_orm(table_name = "kv_blob_generation")] +pub struct Model { + #[sea_orm(primary_key, auto_increment = false)] + pub space: SpaceIdWrap, + #[sea_orm(primary_key, auto_increment = false)] + pub value: Hash, + pub generation: i64, +} + +#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)] +pub enum Relation {} + +impl ActiveModelBehavior for ActiveModel {} diff --git a/tinycloud-core/src/models/kv_orphan_attempt.rs b/tinycloud-core/src/models/kv_orphan_attempt.rs new file mode 100644 index 00000000..46912aff --- /dev/null +++ b/tinycloud-core/src/models/kv_orphan_attempt.rs @@ -0,0 +1,24 @@ +use crate::hash::Hash; +use crate::types::SpaceIdWrap; +use sea_orm::entity::prelude::*; +use time::OffsetDateTime; + +#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)] +#[sea_orm(table_name = "kv_orphan_attempt")] +pub struct Model { + #[sea_orm(primary_key, auto_increment = false)] + pub space: SpaceIdWrap, + #[sea_orm(primary_key, auto_increment = false)] + pub value: Hash, + #[sea_orm(primary_key, auto_increment = false)] + pub attempt_id: String, + pub generation: i64, + pub created_at: OffsetDateTime, + /// Collector backoff only; `created_at` remains the immutable upload age. + pub retry_after: Option, +} + +#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)] +pub enum Relation {} + +impl ActiveModelBehavior for ActiveModel {} diff --git a/tinycloud-core/src/models/kv_orphan_candidate.rs b/tinycloud-core/src/models/kv_orphan_candidate.rs new file mode 100644 index 00000000..92db3626 --- /dev/null +++ b/tinycloud-core/src/models/kv_orphan_candidate.rs @@ -0,0 +1,22 @@ +use crate::hash::Hash; +use crate::types::SpaceIdWrap; +use sea_orm::entity::prelude::*; +use time::OffsetDateTime; + +#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel)] +#[sea_orm(table_name = "kv_orphan_candidate")] +pub struct Model { + #[sea_orm(primary_key, auto_increment = false)] + pub space: SpaceIdWrap, + #[sea_orm(primary_key, auto_increment = false)] + pub value: Hash, + pub generation: i64, + pub created_at: OffsetDateTime, + pub deleting_at: Option, + pub deletion_bytes: Option, +} + +#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)] +pub enum Relation {} + +impl ActiveModelBehavior for ActiveModel {} diff --git a/tinycloud-core/src/models/kv_write.rs b/tinycloud-core/src/models/kv_write.rs index c6c1bd2e..93a7ddf7 100644 --- a/tinycloud-core/src/models/kv_write.rs +++ b/tinycloud-core/src/models/kv_write.rs @@ -16,6 +16,8 @@ pub struct Model { pub epoch: Hash, pub epoch_seq: i64, pub value: Hash, + #[sea_orm(default_value = 0)] + pub generation: i64, pub metadata: Metadata, } diff --git a/tinycloud-core/src/models/mod.rs b/tinycloud-core/src/models/mod.rs index 3c058d5f..07b49772 100644 --- a/tinycloud-core/src/models/mod.rs +++ b/tinycloud-core/src/models/mod.rs @@ -20,7 +20,10 @@ pub mod hook_delivery; pub mod hook_subscription; pub mod invocation; pub mod invocation_replay; +pub mod kv_blob_generation; pub mod kv_delete; +pub mod kv_orphan_attempt; +pub mod kv_orphan_candidate; pub mod kv_write; pub mod meeting_legacy_write_guard; pub mod owner_share_policy; diff --git a/tinycloud-core/src/storage/either.rs b/tinycloud-core/src/storage/either.rs index d36b9197..fbe5e910 100644 --- a/tinycloud-core/src/storage/either.rs +++ b/tinycloud-core/src/storage/either.rs @@ -25,20 +25,32 @@ where { type Readable = AsyncEither; type Error = EitherError; - async fn contains(&self, space: &SpaceId, id: &Hash) -> Result { + async fn contains( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result { match self { - Self::A(l) => l.contains(space, id).await.map_err(Self::Error::A), - Self::B(r) => r.contains(space, id).await.map_err(Self::Error::B), + Self::A(l) => l + .contains(space, id, generation) + .await + .map_err(Self::Error::A), + Self::B(r) => r + .contains(space, id, generation) + .await + .map_err(Self::Error::B), } } async fn read( &self, space: &SpaceId, id: &Hash, + generation: i64, ) -> Result>, Self::Error> { match self { Self::A(l) => l - .read(space, id) + .read(space, id, generation) .await .map(|o| { o.map(|c| { @@ -48,7 +60,7 @@ where }) .map_err(Self::Error::A), Self::B(r) => r - .read(space, id) + .read(space, id, generation) .await .map(|o| { o.map(|c| { @@ -64,11 +76,12 @@ where &self, space: &SpaceId, id: &Hash, + generation: i64, range: ByteRangeSpec, ) -> Result>, Self::Error> { match self { Self::A(store) => store - .read_range(space, id, range) + .read_range(space, id, generation, range) .await .map(|read| { read.map(|read| match read { @@ -91,7 +104,7 @@ where }) .map_err(Self::Error::A), Self::B(store) => store - .read_range(space, id, range) + .read_range(space, id, generation, range) .await .map(|read| { read.map(|read| match read { @@ -154,10 +167,17 @@ where &self, space: &SpaceId, staged: HashBuffer, + generation: i64, ) -> Result { match self { - Self::A(a) => a.persist(space, staged).await.map_err(Self::Error::A), - Self::B(b) => b.persist(space, staged).await.map_err(Self::Error::B), + Self::A(a) => a + .persist(space, staged, generation) + .await + .map_err(Self::Error::A), + Self::B(b) => b + .persist(space, staged, generation) + .await + .map_err(Self::Error::B), } } } @@ -199,10 +219,38 @@ where B: ImmutableDeleteStore, { type Error = EitherError; - async fn remove(&self, space: &SpaceId, id: &Hash) -> Result, Self::Error> { + async fn remove( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result, Self::Error> { + match self { + Self::A(l) => l + .remove(space, id, generation) + .await + .map_err(Self::Error::A), + Self::B(r) => r + .remove(space, id, generation) + .await + .map_err(Self::Error::B), + } + } + async fn cleanup_stages( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result<(), Self::Error> { match self { - Self::A(l) => l.remove(space, id).await.map_err(Self::Error::A), - Self::B(r) => r.remove(space, id).await.map_err(Self::Error::B), + Self::A(store) => store + .cleanup_stages(space, id, generation) + .await + .map_err(Self::Error::A), + Self::B(store) => store + .cleanup_stages(space, id, generation) + .await + .map_err(Self::Error::B), } } } @@ -227,3 +275,49 @@ where } } } + +#[cfg(test)] +mod tests { + use super::*; + use futures::io::{AsyncReadExt, AsyncWriteExt}; + + #[tokio::test] + async fn either_dispatches_each_generation() { + let space: SpaceId = "tinycloud:key:test:either-generations".parse().unwrap(); + for store in [ + Either::A(MemoryStore::default()), + Either::B(MemoryStore::default()), + ] { + let mut old = HashBuffer::new(Vec::new()); + old.write_all(b"same bytes").await.unwrap(); + let hash = old.hash(); + store.persist(&space, old, 0).await.unwrap(); + let mut next = HashBuffer::new(Vec::new()); + next.write_all(b"same bytes").await.unwrap(); + store.persist(&space, next, 1).await.unwrap(); + assert_eq!(store.remove(&space, &hash, 0).await.unwrap(), Some(())); + assert!(!store.contains(&space, &hash, 0).await.unwrap()); + assert!(store.contains(&space, &hash, 1).await.unwrap()); + assert_eq!( + store.read_to_vec(&space, &hash, 1).await.unwrap(), + Some(b"same bytes".to_vec()) + ); + let RangeRead::Content { mut content, .. } = store + .read_range( + &space, + &hash, + 1, + ByteRangeSpec::Inclusive { start: 5, end: 9 }, + ) + .await + .unwrap() + .unwrap() + else { + panic!("expected range"); + }; + let mut bytes = Vec::new(); + content.read_to_end(&mut bytes).await.unwrap(); + assert_eq!(bytes, b"bytes"); + } + } +} diff --git a/tinycloud-core/src/storage/memory.rs b/tinycloud-core/src/storage/memory.rs index 9e4bb569..fb3d6e69 100644 --- a/tinycloud-core/src/storage/memory.rs +++ b/tinycloud-core/src/storage/memory.rs @@ -16,7 +16,7 @@ pub struct MemoryStore { spaces: Arc>>, } -type Blocks = DashMap>; +type Blocks = DashMap<(Hash, i64), Vec>; #[derive(Default, Debug, Clone, Hash, PartialEq, Eq)] pub struct MemoryStaging; @@ -65,11 +65,16 @@ impl ImmutableReadStore for MemoryStore { type Error = io::Error; type Readable = Cursor>; - async fn contains(&self, space: &SpaceId, id: &Hash) -> Result { + async fn contains( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result { Ok(self .spaces .get(space) - .map(|o| o.contains_key(id)) + .map(|o| o.contains_key(&(*id, generation))) .unwrap_or(false)) } @@ -77,10 +82,11 @@ impl ImmutableReadStore for MemoryStore { &self, space: &SpaceId, id: &Hash, + generation: i64, ) -> Result>, Self::Error> { let start = Instant::now(); let result = match self.spaces.get(space) { - Some(o) => match o.get(id) { + Some(o) => match o.get(&(*id, generation)) { Some(data) => { let len = data.len() as u64; let reader = Cursor::new(data.clone()); @@ -102,12 +108,13 @@ impl ImmutableReadStore for MemoryStore { &self, space: &SpaceId, id: &Hash, + generation: i64, range: ByteRangeSpec, ) -> Result>, Self::Error> { let Some(space) = self.spaces.get(space) else { return Ok(None); }; - let Some(data) = space.get(id) else { + let Some(data) = space.get(&(*id, generation)) else { return Ok(None); }; let total_size = data.len() as u64; @@ -126,9 +133,10 @@ impl ImmutableReadStore for MemoryStore { &self, space: &SpaceId, id: &Hash, + generation: i64, ) -> Result>, VecReadError> { match self.spaces.get(space) { - Some(o) => Ok(o.get(id).map(|data| data.clone())), + Some(o) => Ok(o.get(&(*id, generation)).map(|data| data.clone())), None => Ok(None), } } @@ -142,6 +150,7 @@ impl ImmutableWriteStore for MemoryStore { &self, space: &SpaceId, mut staged: HashBuffer>, + generation: i64, ) -> Result { let start = Instant::now(); let hash = staged.hash(); @@ -154,7 +163,7 @@ impl ImmutableWriteStore for MemoryStore { .or_insert_with(|| Arc::new(DashMap::new())) .clone(); - space_storage.insert(hash, data); + space_storage.insert((hash, generation), data); crate::telemetry::observe_stage( crate::telemetry::InvocationStage::BlockWrite, crate::telemetry::StageOutcome::Ok, @@ -168,6 +177,7 @@ impl ImmutableWriteStore for MemoryStore { space: &SpaceId, mut staged: HashBuffer>, hash: &Hash, + generation: i64, ) -> Result<(), KeyedWriteError> { let start = Instant::now(); if hash != &staged.hash() { @@ -187,7 +197,7 @@ impl ImmutableWriteStore for MemoryStore { .or_insert_with(|| Arc::new(DashMap::new())) .clone(); - space_storage.insert(*hash, data); + space_storage.insert((*hash, generation), data); crate::telemetry::observe_stage( crate::telemetry::InvocationStage::BlockWrite, crate::telemetry::StageOutcome::Ok, @@ -201,13 +211,27 @@ impl ImmutableWriteStore for MemoryStore { impl ImmutableDeleteStore for MemoryStore { type Error = io::Error; - async fn remove(&self, space: &SpaceId, id: &Hash) -> Result, Self::Error> { + async fn remove( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result, Self::Error> { Ok(self .spaces .get(space) - .and_then(|o| o.remove(id)) + .and_then(|o| o.remove(&(*id, generation))) .map(|_| ())) } + async fn cleanup_stages( + &self, + _space: &SpaceId, + _id: &Hash, + _generation: i64, + ) -> Result<(), Self::Error> { + // Memory staging has no filesystem artifact after the writer exits. + Ok(()) + } } #[async_trait] @@ -234,3 +258,48 @@ impl StoreSize for MemoryStore { .sum::()) } } + +#[cfg(test)] +mod tests { + use super::*; + use futures::io::{AsyncReadExt, AsyncWriteExt}; + + #[tokio::test] + async fn same_hash_generations_are_independent() { + let store = MemoryStore::default(); + let space: SpaceId = "tinycloud:key:test:memory-generations".parse().unwrap(); + let mut old = HashBuffer::new(Vec::new()); + old.write_all(b"same bytes").await.unwrap(); + let hash = old.hash(); + store.persist(&space, old, 0).await.unwrap(); + let mut next = HashBuffer::new(Vec::new()); + next.write_all(b"same bytes").await.unwrap(); + store.persist(&space, next, 1).await.unwrap(); + assert_eq!(store.total_size(&space).await.unwrap(), Some(20)); + assert_eq!(store.remove(&space, &hash, 0).await.unwrap(), Some(())); + assert!(!store.contains(&space, &hash, 0).await.unwrap()); + assert!(store.contains(&space, &hash, 1).await.unwrap()); + assert_eq!( + store.read_to_vec(&space, &hash, 1).await.unwrap(), + Some(b"same bytes".to_vec()) + ); + let RangeRead::Content { mut content, .. } = store + .read_range( + &space, + &hash, + 1, + ByteRangeSpec::Inclusive { start: 5, end: 9 }, + ) + .await + .unwrap() + .unwrap() + else { + panic!("expected range"); + }; + let mut bytes = Vec::new(); + content.read_to_end(&mut bytes).await.unwrap(); + assert_eq!(bytes, b"bytes"); + assert_eq!(store.remove(&space, &hash, 0).await.unwrap(), None); + assert_eq!(store.total_size(&space).await.unwrap(), Some(10)); + } +} diff --git a/tinycloud-core/src/storage/mod.rs b/tinycloud-core/src/storage/mod.rs index 4dc263a8..657d6bd0 100644 --- a/tinycloud-core/src/storage/mod.rs +++ b/tinycloud-core/src/storage/mod.rs @@ -124,28 +124,36 @@ pub enum KeyedWriteError { pub trait ImmutableReadStore: Send + Sync { type Error: StdError + Send + Sync; type Readable: futures::io::AsyncRead + Send + Sync; - async fn contains(&self, space: &SpaceId, id: &Hash) -> Result; + async fn contains( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result; async fn read( &self, space: &SpaceId, id: &Hash, + generation: i64, ) -> Result>, Self::Error>; async fn read_range( &self, space: &SpaceId, id: &Hash, + generation: i64, range: ByteRangeSpec, ) -> Result>, Self::Error>; async fn read_to_vec( &self, space: &SpaceId, id: &Hash, + generation: i64, ) -> Result>, VecReadError> where Self::Readable: Send, { use futures::io::AsyncReadExt; - let (l, r) = match self.read(space, id).await? { + let (l, r) = match self.read(space, id, generation).await? { None => return Ok(None), Some(r) => r.into_inner(), }; @@ -179,17 +187,19 @@ where &self, space: &SpaceId, staged: HashBuffer, + generation: i64, ) -> Result; async fn persist_keyed( &self, space: &SpaceId, mut staged: HashBuffer, hash: &Hash, + generation: i64, ) -> Result<(), KeyedWriteError> { if hash != &staged.hash() { return Err(KeyedWriteError::IncorrectHash); }; - self.persist(space, staged).await?; + self.persist(space, staged, generation).await?; Ok(()) } } @@ -197,7 +207,22 @@ where #[async_trait] pub trait ImmutableDeleteStore: Send + Sync { type Error: StdError + Send + Sync; - async fn remove(&self, space: &SpaceId, id: &Hash) -> Result, Self::Error>; + async fn remove( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result, Self::Error>; + + /// Reclaim incomplete, unreferenced staging files without removing the + /// published blob. Called only after the orphan fence found no active + /// attempts; storage backends without persistent stages have nothing to do. + async fn cleanup_stages( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result<(), Self::Error>; } #[async_trait] @@ -218,33 +243,41 @@ where { type Error = S::Error; type Readable = S::Readable; - async fn contains(&self, space: &SpaceId, id: &Hash) -> Result { - self.contains(space, id).await + async fn contains( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result { + (**self).contains(space, id, generation).await } async fn read( &self, space: &SpaceId, id: &Hash, + generation: i64, ) -> Result>, Self::Error> { - self.read(space, id).await + (**self).read(space, id, generation).await } async fn read_range( &self, space: &SpaceId, id: &Hash, + generation: i64, range: ByteRangeSpec, ) -> Result>, Self::Error> { - self.read_range(space, id, range).await + (**self).read_range(space, id, generation, range).await } async fn read_to_vec( &self, space: &SpaceId, id: &Hash, + generation: i64, ) -> Result>, VecReadError> where Self::Readable: Send, { - self.read_to_vec(space, id).await + (**self).read_to_vec(space, id, generation).await } } @@ -272,16 +305,20 @@ where &self, space: &SpaceId, staged: HashBuffer, + generation: i64, ) -> Result { - self.persist(space, staged).await + (**self).persist(space, staged, generation).await } async fn persist_keyed( &self, space: &SpaceId, staged: HashBuffer, hash: &Hash, + generation: i64, ) -> Result<(), KeyedWriteError> { - self.persist_keyed(space, staged, hash).await + (**self) + .persist_keyed(space, staged, hash, generation) + .await } } @@ -291,8 +328,21 @@ where S: ImmutableDeleteStore, { type Error = S::Error; - async fn remove(&self, space: &SpaceId, id: &Hash) -> Result, Self::Error> { - self.remove(space, id).await + async fn remove( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result, Self::Error> { + (**self).remove(space, id, generation).await + } + async fn cleanup_stages( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result<(), Self::Error> { + (**self).cleanup_stages(space, id, generation).await } } diff --git a/tinycloud-node-server/src/lib.rs b/tinycloud-node-server/src/lib.rs index c8927dda..b0c62748 100644 --- a/tinycloud-node-server/src/lib.rs +++ b/tinycloud-node-server/src/lib.rs @@ -817,9 +817,23 @@ pub async fn app_with_control( encryption_backend, ); + // Migrate before opening S3: its one bucket listing both reconstructs + // SpaceSizes and records which in-flight DELETE fences were counted. + // Object sizes come from that listing; a separate HEAD afterwards could + // observe a DELETE in progress. The boot migration is bounded by + // `lock_timeout`; the second pass in `new_with_encryption` finds the + // history current, then runs the CID-alias backfill with the column key + // before any request is served. + tinycloud_core::db::migrate(&database_connection).await?; + let deleting_fences = tinycloud_core::kv_orphans::deleting_fences(&database_connection).await?; + let block_store: BlockStores = match &tinycloud_config.storage.blocks { + Either::A(s3) => Either::A(s3.open_with_deleting(&deleting_fences).await?), + Either::B(local) => Either::B(local.open().await?), + }; + let orphan_store = block_store.clone(); let tinycloud = TinyCloud::new_with_encryption( database_connection, - tinycloud_config.storage.blocks.open().await?, + block_store, key_setup.setup(()).await?, Some(webhook_encryption.clone()), ) @@ -918,6 +932,8 @@ pub async fn app_with_control( )?; spawn_webhook_dispatcher(webhook_dispatcher); + let orphan_conn = seed_conn.clone(); + // TC-326: capture handles for the telemetry DB sampler before `tinycloud` // is moved into Rocket-managed state below. let telemetry_enabled = tinycloud_config.telemetry.enabled; @@ -975,6 +991,33 @@ pub async fn app_with_control( }) }, )) + .attach(AdHoc::on_liftoff("kv-orphan-cleanup", move |rocket| { + let conn = orphan_conn.clone(); + let store = orphan_store.clone(); + let shutdown = rocket.shutdown(); + Box::pin(async move { + tokio::spawn(async move { + let mut interval = tokio::time::interval(std::time::Duration::from_secs(60)); + tokio::pin!(shutdown); + loop { + tokio::select! { + _ = interval.tick() => { + let result = tokio::select! { + result = tinycloud_core::kv_orphans::drain_candidates_for_interval( + &conn, &store, tinycloud_core::kv_orphans::ORPHAN_GRACE, + ) => result, + _ = &mut shutdown => break, + }; + if let Err(error) = result { + ::tracing::warn!(%error, "KV orphan collection failed"); + } + } + _ = &mut shutdown => break, + } + } + }); + }) + })) .manage(tinycloud) .manage(node_public_keys) .manage(sql_service); diff --git a/tinycloud-node-server/src/routes/mod.rs b/tinycloud-node-server/src/routes/mod.rs index da081fa4..dcc413cb 100644 --- a/tinycloud-node-server/src/routes/mod.rs +++ b/tinycloud-node-server/src/routes/mod.rs @@ -2363,6 +2363,16 @@ async fn invoke_impl( if let TxStoreError::KvTooManyMutations { .. } = &e { return Err(too_many_mutations().into()); } + if let TxStoreError::KvCandidateUnavailable = &e { + // The attempted blob cannot be committed: its durable + // registration expired or a prior DELETE is in flight. + // No KV write landed; retry with a fresh invocation. + return Err(( + Status::ServiceUnavailable, + format!("{UNAVAILABLE_RETRYABLE}: {e}"), + ) + .into()); + } if let TxStoreError::Tx(TxError::ReadAudit(error)) = &e { return Err(read_audit_failure(Service::Kv, error).into()); } @@ -13066,6 +13076,7 @@ mod tests { epoch: Set(tinycloud_core::hash::hash(b"tc908-foreign-epoch")), epoch_seq: Set(1), value: Set(tinycloud_core::hash::hash(b"foreign secret")), + generation: Set(0), metadata: Set(Metadata(Default::default())), deleted: Set(false), } diff --git a/tinycloud-node-server/src/storage/file_system.rs b/tinycloud-node-server/src/storage/file_system.rs index 40b5692f..501b0939 100644 --- a/tinycloud-node-server/src/storage/file_system.rs +++ b/tinycloud-node-server/src/storage/file_system.rs @@ -1,25 +1,34 @@ -use super::size::SpaceSizes; +#[cfg(test)] +use super::TestPause; +use super::{size::SpaceSizes, BlobGenerationKey, BlobOperationLocks}; use core::pin::Pin; use futures::{ - future::{Either as AsyncEither, TryFutureExt}, + future::Either as AsyncEither, io::{AsyncWrite, AsyncWriteExt}, stream::TryStreamExt, task::{Context, Poll}, }; use pin_project::pin_project; use serde::{Deserialize, Serialize}; +#[cfg(test)] +use std::sync::atomic::{AtomicBool, Ordering}; use std::{ - collections::HashMap, + collections::{hash_map::Entry, HashMap}, + fmt::Write as _, io::{Error as IoError, ErrorKind, SeekFrom}, path::{Path, PathBuf}, + sync::Arc, time::Instant, }; use tempfile::{NamedTempFile, PathPersistError}; use tinycloud_auth::{resource::SpaceId, ssi::dids::DIDBuf}; use tinycloud_core::{hash::Hash, storage::*}; +#[cfg(test)] +use tokio::sync::Notify; use tokio::{ fs::{create_dir_all, metadata, remove_file, File}, io::{AsyncReadExt, AsyncSeekExt}, + sync::Mutex, }; use tokio_stream::wrappers::ReadDirStream; @@ -29,20 +38,62 @@ use tokio_util::compat::{Compat, TokioAsyncReadCompatExt}; pub struct FileSystemStore { path: PathBuf, sizes: SpaceSizes, + // Reserve the full size before writing a temporary file. A cancellation + // or failed publish leaves a conservative charge until cleanup confirms + // that neither the stage nor the final object remains. + pending: Arc>>, + // Paths and charges from the same boot listing as `sizes`. A later upload + // reservation for the same hash is a separate charge, not a replacement. + boot_stages: Arc>>, + operations: BlobOperationLocks, + #[cfg(test)] + before_memory_write: TestPause, + #[cfg(test)] + before_pending_refund: TestPause, + #[cfg(test)] + before_reserve_charge: TestPause, + #[cfg(test)] + after_memory_contains: TestPause, + #[cfg(test)] + fail_stage_cleanup: Arc, + #[cfg(test)] + after_stage_unlink: TestPause, } impl FileSystemStore { async fn new(path: PathBuf) -> Result { - // get the size of the directory - let sizes = store_sizes(&path).await?.into(); - Ok(Self { path, sizes }) + // Count each stage and its enclosing space from one listing snapshot. + let (sizes, boot_stages) = store_sizes(&path).await?; + Ok(Self { + path, + sizes: sizes.into(), + pending: Arc::new(Mutex::new(HashMap::new())), + boot_stages: Arc::new(Mutex::new(boot_stages)), + operations: BlobOperationLocks::default(), + #[cfg(test)] + before_memory_write: Arc::new(Mutex::new(None)), + #[cfg(test)] + before_pending_refund: Arc::new(Mutex::new(None)), + #[cfg(test)] + after_memory_contains: Arc::new(Mutex::new(None)), + #[cfg(test)] + before_reserve_charge: Arc::new(Mutex::new(None)), + #[cfg(test)] + fail_stage_cleanup: Arc::new(AtomicBool::new(false)), + #[cfg(test)] + after_stage_unlink: Arc::new(Mutex::new(None)), + }) } - fn get_path(&self, space: &SpaceId, mh: &Hash) -> PathBuf { + fn get_path(&self, space: &SpaceId, mh: &Hash, generation: i64) -> PathBuf { + let mut name = base64::encode_config(mh.as_ref(), base64::URL_SAFE); + if generation != 0 { + write!(&mut name, "@{generation}").expect("writing to String cannot fail"); + } self.path .join(space.suffix()) .join(space.name().as_str()) - .join(base64::encode_config(mh.as_ref(), base64::URL_SAFE)) + .join(name) } async fn increment_size(&self, space: &SpaceId, size: u64) { @@ -51,6 +102,236 @@ impl FileSystemStore { async fn decrement_size(&self, space: &SpaceId, size: u64) { self.sizes.decrement_size(space, size).await; } + + async fn reserve(&self, space: &SpaceId, hash: Hash, generation: i64, size: u64) { + let key = (space.clone(), hash, generation); + let mut pending = self.pending.lock().await; + if let Entry::Vacant(slot) = pending.entry(key) { + #[cfg(test)] + if let Some((entered, release)) = self.before_reserve_charge.lock().await.take() { + entered.notify_one(); + release.notified().await; + } + // Charge before publishing the reservation. Cancellation while + // waiting on SpaceSizes leaves neither a charge nor a claim. + self.increment_size(space, size).await; + slot.insert(size); + } + } + + async fn settle(&self, space: &SpaceId, hash: Hash, generation: i64) { + self.pending + .lock() + .await + .remove(&(space.clone(), hash, generation)); + } + + async fn refund_pending(&self, space: &SpaceId, hash: Hash, generation: i64) { + let key = (space.clone(), hash, generation); + let mut pending = self.pending.lock().await; + if let Some(&reserved) = pending.get(&key) { + #[cfg(test)] + if let Some((entered, release)) = self.before_pending_refund.lock().await.take() { + entered.notify_one(); + release.notified().await; + } + // Retain the refund obligation until the size change completes. + // No await may separate decrement from clearing this claim. + if reserved != 0 { + self.decrement_size(space, reserved).await; + } + pending.remove(&key); + } + } + + async fn discard_failed_stage( + &self, + space: &SpaceId, + hash: Hash, + generation: i64, + path: tempfile::TempPath, + ) -> Result<(), IoError> { + // Do not refund a reservation until its unpublished bytes are gone. + path.close()?; + self.refund_pending(space, hash, generation).await; + Ok(()) + } + + fn stage_prefix(path: &Path) -> String { + format!( + ".tc-stage-{}-", + path.file_name() + .expect("blob path has a name") + .to_string_lossy() + ) + } + + /// A process crash can leave a named temporary file in a space directory. + /// Boot quota reconstruction counts its actual bytes; candidate GC removes + /// files for that hash and refunds them along with any published object. + async fn remove_stages( + &self, + space: &SpaceId, + path: &Path, + refund_each: bool, + ) -> Result<(u64, u64), IoError> { + let Some(parent) = path.parent() else { + return Ok((0, 0)); + }; + let mut entries = match tokio::fs::read_dir(parent).await { + Ok(entries) => entries, + Err(error) if error.kind() == ErrorKind::NotFound => return Ok((0, 0)), + Err(error) => return Err(error), + }; + let prefix = Self::stage_prefix(path); + let mut removed = 0u64; + let mut boot_refunded = 0u64; + while let Some(entry) = entries.next_entry().await? { + if !entry.file_name().to_string_lossy().starts_with(&prefix) + || !entry.file_type().await?.is_file() + { + continue; + } + let size = entry.metadata().await?.len(); + let stage_path = entry.path(); + match remove_file(&stage_path).await { + Ok(()) => { + removed = removed + .checked_add(size) + .ok_or_else(|| IoError::other("staged blob byte count overflow"))?; + #[cfg(test)] + if let Some((entered, release)) = self.after_stage_unlink.lock().await.take() { + entered.notify_one(); + release.notified().await; + } + let counted_at_boot = self + .boot_stages + .lock() + .await + .remove(&stage_path) + .unwrap_or(0); + boot_refunded = boot_refunded + .checked_add(counted_at_boot) + .ok_or_else(|| IoError::other("boot stage byte count overflow"))?; + if counted_at_boot != 0 || refund_each { + self.decrement_size( + space, + if counted_at_boot != 0 { + counted_at_boot + } else { + size + }, + ) + .await; + } + } + Err(error) if error.kind() == ErrorKind::NotFound => {} + Err(error) => return Err(error), + } + } + Ok((removed, boot_refunded)) + } + + async fn persist_temp( + &self, + space: &SpaceId, + hash: Hash, + generation: i64, + file: TempFileStage, + ) -> Result<(), FileSystemStoreError> { + let _operation = self + .operations + .lock((space.clone(), hash, generation)) + .await; + if self.contains(space, &hash, generation).await? { + self.settle(space, hash, generation).await; + return Ok(()); + } + let size = file.size().await?; + self.reserve(space, hash, generation, size).await; + let (_, path) = file.into_inner(); + match path.persist(self.get_path(space, &hash, generation)) { + Ok(()) => { + self.settle(space, hash, generation).await; + Ok(()) + } + Err(PathPersistError { error, path }) => { + self.discard_failed_stage(space, hash, generation, path) + .await?; + Err(error.into()) + } + } + } + + async fn persist_memory( + &self, + space: &SpaceId, + hash: Hash, + generation: i64, + bytes: Vec, + ) -> Result<(), FileSystemStoreError> { + let _operation = self + .operations + .lock((space.clone(), hash, generation)) + .await; + if self.contains(space, &hash, generation).await? { + self.settle(space, hash, generation).await; + return Ok(()); + } + #[cfg(test)] + let pause = self.after_memory_contains.lock().await.take(); + #[cfg(test)] + if let Some((entered, release)) = pause { + entered.notify_one(); + release.notified().await; + } + self.reserve(space, hash, generation, bytes.len() as u64) + .await; + let target = self.get_path(space, &hash, generation); + // Stage on the target filesystem so the final rename is atomic and + // never depends on a cross-device copy. + let staged = match tempfile::Builder::new() + .prefix(&Self::stage_prefix(&target)) + .tempfile_in(target.parent().expect("blob path has a parent")) + { + Ok(staged) => staged, + Err(error) => { + self.refund_pending(space, hash, generation).await; + return Err(error.into()); + } + }; + let (file, path) = staged.into_parts(); + #[cfg(test)] + let pause = self.before_memory_write.lock().await.take(); + #[cfg(test)] + if let Some((entered, release)) = pause { + entered.notify_one(); + release.notified().await; + } + let mut writer = File::from_std(file).compat(); + let written = async { + writer.write_all(&bytes).await?; + writer.flush().await + } + .await; + drop(writer); + if let Err(error) = written { + self.discard_failed_stage(space, hash, generation, path) + .await?; + return Err(error.into()); + } + match path.persist(target) { + Ok(()) => { + self.settle(space, hash, generation).await; + Ok(()) + } + Err(PathPersistError { error, path }) => { + self.discard_failed_stage(space, hash, generation, path) + .await?; + Err(error.into()) + } + } + } } #[derive(Serialize, Deserialize, Debug, Clone, Hash, PartialEq, Eq)] @@ -113,16 +394,22 @@ pub enum FileSystemStoreError { impl ImmutableReadStore for FileSystemStore { type Error = FileSystemStoreError; type Readable = Compat>; - async fn contains(&self, space: &SpaceId, id: &Hash) -> Result { - Ok(self.get_path(space, id).exists()) + async fn contains( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result { + Ok(self.get_path(space, id, generation).exists()) } async fn read( &self, space: &SpaceId, id: &Hash, + generation: i64, ) -> Result>, Self::Error> { let start = Instant::now(); - let result = match File::open(self.get_path(space, id)).await { + let result = match File::open(self.get_path(space, id, generation)).await { Ok(file) => { let size = file.metadata().await?.len(); Ok(Some(Content::new(size, file.take(size).compat()))) @@ -142,9 +429,10 @@ impl ImmutableReadStore for FileSystemStore { &self, space: &SpaceId, id: &Hash, + generation: i64, range: ByteRangeSpec, ) -> Result>, Self::Error> { - match File::open(self.get_path(space, id)).await { + match File::open(self.get_path(space, id, generation)).await { Ok(mut file) => { let total_size = file.metadata().await?.len(); let Some(range) = range.resolve(total_size) else { @@ -176,49 +464,72 @@ impl StoreSize for FileSystemStore { } // get the sum size of all files in this directory (recurse into subdirectories with space ID names) -async fn store_sizes>(path: &P) -> Result, IoError> { +async fn store_sizes>( + path: &P, +) -> Result<(HashMap, HashMap), IoError> { ReadDirStream::new(tokio::fs::read_dir(path).await?) // for every entry in the store dir - .try_fold(HashMap::new(), |mut acc, entry| async move { - // if its a directory and the suffix is a valid string - if let (true, Ok(ref suffix)) = ( - entry.metadata().await?.is_dir(), - entry.file_name().into_string(), - ) { - let mut ds = ReadDirStream::new(tokio::fs::read_dir(entry.path()).await?); - let did: DIDBuf = ["did:", suffix.as_str()].concat().parse().map_err(|_| { - IoError::new(ErrorKind::InvalidData, format!("Invalid DID: {suffix}")) - })?; - // go through each suffix directory - while let Some(entry) = ds.try_next().await? { - // for each entry in the suffix directory - // if its a directory and the name is a valid string - if let (true, Ok(name)) = ( - entry.metadata().await?.is_dir(), - entry.file_name().into_string(), - ) { - // get the space ID from suffix and name - let space = - SpaceId::new(did.clone(), name.try_into().map_err(IoError::other)?); - let size = space_size(&entry.path()).await?; - acc.insert(space, size); + .try_fold( + (HashMap::new(), HashMap::new()), + |(mut acc, mut stages), entry| async move { + // if its a directory and the suffix is a valid string + if let (true, Ok(ref suffix)) = ( + entry.metadata().await?.is_dir(), + entry.file_name().into_string(), + ) { + let mut ds = ReadDirStream::new(tokio::fs::read_dir(entry.path()).await?); + let did: DIDBuf = ["did:", suffix.as_str()].concat().parse().map_err(|_| { + IoError::new(ErrorKind::InvalidData, format!("Invalid DID: {suffix}")) + })?; + // go through each suffix directory + while let Some(entry) = ds.try_next().await? { + // for each entry in the suffix directory + // if its a directory and the name is a valid string + if let (true, Ok(name)) = ( + entry.metadata().await?.is_dir(), + entry.file_name().into_string(), + ) { + // get the space ID from suffix and name + let space = + SpaceId::new(did.clone(), name.try_into().map_err(IoError::other)?); + let (size, space_stages) = space_size(&entry.path()).await?; + acc.insert(space, size); + stages.extend(space_stages); + } } - } - }; - Ok(acc) - }) + }; + Ok((acc, stages)) + }, + ) .await } -async fn space_size>(path: &P) -> Result { - // get the sum size of all files in this directory (do not recurse into subdirectories) +async fn space_size>(path: &P) -> Result<(u64, HashMap), IoError> { + // This one directory listing establishes both the charged total and the + // exact incomplete stages which contributed to that total at boot. ReadDirStream::new(tokio::fs::read_dir(path).await?) - .try_fold(0, |acc, entry| async move { - entry - .metadata() - .map_ok(|m| if m.is_dir() { acc } else { acc + m.len() }) - .await - }) + .try_fold( + (0u64, HashMap::new()), + |(acc, mut stages), entry| async move { + let metadata = entry.metadata().await?; + if metadata.is_dir() { + return Ok((acc, stages)); + } + let size = metadata.len(); + if metadata.is_file() + && entry + .file_name() + .to_str() + .is_some_and(|name| name.starts_with(".tc-stage-")) + { + stages.insert(entry.path(), size); + } + let total = acc + .checked_add(size) + .ok_or_else(|| IoError::other("space byte count overflow"))?; + Ok((total, stages)) + }, + ) .await } @@ -275,17 +586,13 @@ impl ImmutableWriteStore for FileSystemStore { &self, space: &SpaceId, staged: HashBuffer<::Writable>, + generation: i64, ) -> Result { let start = Instant::now(); let result = async move { let (mut h, f) = staged.into_inner(); let hash = h.finalize(); - if !self.contains(space, &hash).await? { - let size = f.size().await?; - let (_, path) = f.into_inner(); - path.persist(self.get_path(space, &hash))?; - self.increment_size(space, size).await; - } + self.persist_temp(space, hash, generation, f).await?; Ok(hash) } .await; @@ -305,19 +612,13 @@ impl ImmutableWriteStore for FileSystemStore { &self, space: &SpaceId, staged: HashBuffer<::Writable>, + generation: i64, ) -> Result { let start = Instant::now(); let result = async move { let (mut h, v) = staged.into_inner(); let hash = h.finalize(); - if !self.contains(space, &hash).await? { - let file = File::create(self.get_path(space, &hash)).await?; - let size = v.len() as u64; - let mut writer = futures::io::BufWriter::new(file.compat()); - writer.write_all(&v).await?; - writer.flush().await?; - self.increment_size(space, size).await; - } + self.persist_memory(space, hash, generation, v).await?; Ok(hash) } .await; @@ -339,30 +640,19 @@ impl ImmutableWriteStore as ImmutableStaging>::Writable>, + generation: i64, ) -> Result { let start = Instant::now(); let result = async move { let (mut h, f) = staged.into_inner(); let hash = h.finalize(); - if !self.contains(space, &hash).await? { - match f { - AsyncEither::Left(t_file) => { - let size = t_file.size().await?; - let (_, path) = t_file.into_inner(); - path.persist(self.get_path(space, &hash))?; - self.increment_size(space, size).await; - } - AsyncEither::Right(v) => { - let file = File::create(self.get_path(space, &hash)).await?; - let size = v.len() as u64; - let mut writer = futures::io::BufWriter::new(file.compat()); - writer.write_all(&v).await?; - writer.flush().await?; - self.increment_size(space, size).await; - } + match f { + AsyncEither::Left(file) => self.persist_temp(space, hash, generation, file).await?, + AsyncEither::Right(bytes) => { + self.persist_memory(space, hash, generation, bytes).await? } - }; + } Ok(hash) } .await; @@ -378,21 +668,90 @@ impl ImmutableWriteStore Result, Self::Error> { - let path = self.get_path(space, id); - let size = match metadata(&path).await { - Ok(m) => m.len(), - Err(e) if e.kind() == ErrorKind::NotFound => return Ok(None), - Err(e) => return Err(e.into()), - }; - match remove_file(path).await { - Ok(()) => { - self.decrement_size(space, size).await; - Ok(Some(())) + async fn remove( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result, Self::Error> { + let store = self.clone(); + let space = space.clone(); + let id = *id; + // Finish unlink and quota settlement under the same physical lock + // even if the request or collector is cancelled after unlink. + tokio::spawn(async move { + let key = (space.clone(), id, generation); + let _operation = store.operations.lock(key.clone()).await; + let path = store.get_path(&space, &id, generation); + let pending = store.pending.lock().await.contains_key(&key); + // Boot stages and a new reservation are independently charged. + let (staged, boot_refunded) = store.remove_stages(&space, &path, !pending).await?; + let final_size = match metadata(&path).await { + Ok(meta) => Some(meta.len()), + Err(error) if error.kind() == ErrorKind::NotFound => None, + Err(error) => return Err(error.into()), + }; + let removed_final = if let Some(size) = final_size { + match remove_file(&path).await { + Ok(()) => Some(size), + Err(error) if error.kind() == ErrorKind::NotFound => None, + Err(error) => return Err(error.into()), + } + } else { + None + }; + if pending { + let reserved = store.pending.lock().await.remove(&key).unwrap_or(0); + store + .decrement_size( + &space, + reserved.max( + staged + .saturating_sub(boot_refunded) + .saturating_add(removed_final.unwrap_or(0)), + ), + ) + .await; + } else if let Some(size) = removed_final { + store.decrement_size(&space, size).await; } - Err(e) if e.kind() == ErrorKind::NotFound => Ok(None), - Err(e) => Err(e.into()), - } + Ok::<_, FileSystemStoreError>((removed_final.is_some() || staged != 0).then_some(())) + }) + .await + .map_err(IoError::other)? + } + async fn cleanup_stages( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result<(), Self::Error> { + let store = self.clone(); + let space = space.clone(); + let id = *id; + // The DB fence may time out or its caller may be cancelled. Finish + // each unlink and its quota refund together before releasing the + // physical lock; the retained DB attempt makes a retry idempotent. + tokio::spawn(async move { + let key = (space.clone(), id, generation); + let _operation = store.operations.lock(key.clone()).await; + #[cfg(test)] + if store.fail_stage_cleanup.load(Ordering::SeqCst) { + return Err(IoError::other("injected stage cleanup failure").into()); + } + let pending = store.pending.lock().await.contains_key(&key); + // This path is for a referenced final blob. Only incomplete + // stages are removed; never unlink a committed row's object. + store + .remove_stages(&space, &store.get_path(&space, &id, generation), !pending) + .await?; + if pending { + store.refund_pending(&space, id, generation).await; + } + Ok::<(), FileSystemStoreError>(()) + }) + .await + .map_err(IoError::other)? } } @@ -407,7 +766,817 @@ impl StorageConfig for TempFileSystemStage { #[cfg(test)] mod test { use super::*; - use futures::io::AsyncReadExt; + use futures::io::{AsyncReadExt, AsyncWriteExt}; + + #[tokio::test] + async fn memory_publish_is_atomic_for_concurrent_same_hash_paths() { + let dir = tempfile::tempdir().unwrap(); + let store = FileSystemConfig::new(dir.path()).open().await.unwrap(); + let space: SpaceId = "tinycloud:key:test:atomic".parse().unwrap(); + store.create(&space).await.unwrap(); + let data = b"same-content two distinct KV paths".to_vec(); + let entered = Arc::new(Notify::new()); + let release = Arc::new(Notify::new()); + *store.before_memory_write.lock().await = Some((entered.clone(), release)); + let mut first_stage = HashBuffer::new(Vec::new()); + first_stage.write_all(&data).await.unwrap(); + let hash = first_stage.hash(); + let first_store = store.clone(); + let first_space = space.clone(); + let first = tokio::spawn(async move { + ImmutableWriteStore::::persist( + &first_store, + &first_space, + first_stage, + 0, + ) + .await + }); + tokio::time::timeout(std::time::Duration::from_secs(2), entered.notified()) + .await + .unwrap(); + let prematurely_visible = store.contains(&space, &hash, 0).await.unwrap(); + let mut second_stage = HashBuffer::new(Vec::new()); + second_stage.write_all(&data).await.unwrap(); + let second_store = store.clone(); + let second_space = space.clone(); + let mut second = tokio::spawn(async move { + ImmutableWriteStore::::persist( + &second_store, + &second_space, + second_stage, + 0, + ) + .await + }); + let early_second = + tokio::time::timeout(std::time::Duration::from_millis(100), &mut second).await; + first.abort(); + assert!(first.await.unwrap_err().is_cancelled()); + let second_hash = match early_second { + Ok(result) => result.unwrap().unwrap(), + Err(_) => tokio::time::timeout(std::time::Duration::from_secs(2), second) + .await + .unwrap() + .unwrap() + .unwrap(), + }; + assert_eq!(second_hash, hash); + let stored = store.read_to_vec(&space, &hash, 0).await.unwrap().unwrap(); + assert!( + !prematurely_visible, + "a complete candidate must be published before same-hash dedup" + ); + assert_eq!(stored, data, "a committed alias must read all staged bytes"); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(data.len() as u64) + ); + } + + #[tokio::test] + async fn same_generation_concurrent_memory_put_charges_once() { + let dir = tempfile::tempdir().unwrap(); + let store = FileSystemConfig::new(dir.path()).open().await.unwrap(); + let space: SpaceId = "tinycloud:key:test:one-charge".parse().unwrap(); + store.create(&space).await.unwrap(); + let data = b"same content for two KV records".to_vec(); + let entered = Arc::new(Notify::new()); + let release = Arc::new(Notify::new()); + *store.after_memory_contains.lock().await = Some((entered.clone(), release.clone())); + let mut first_stage = HashBuffer::new(Vec::new()); + first_stage.write_all(&data).await.unwrap(); + let hash = first_stage.hash(); + let first_store = store.clone(); + let first_space = space.clone(); + let first = tokio::spawn(async move { + ImmutableWriteStore::::persist( + &first_store, + &first_space, + first_stage, + 0, + ) + .await + }); + tokio::time::timeout(std::time::Duration::from_secs(2), entered.notified()) + .await + .unwrap(); + let mut second_stage = HashBuffer::new(Vec::new()); + second_stage.write_all(&data).await.unwrap(); + let second_store = store.clone(); + let second_space = space.clone(); + let mut second = tokio::spawn(async move { + ImmutableWriteStore::::persist( + &second_store, + &second_space, + second_stage, + 0, + ) + .await + }); + let early_second = + tokio::time::timeout(std::time::Duration::from_millis(100), &mut second).await; + release.notify_one(); + assert_eq!(first.await.unwrap().unwrap(), hash); + let second_hash = match early_second { + Ok(result) => result.unwrap().unwrap(), + Err(_) => tokio::time::timeout(std::time::Duration::from_secs(2), second) + .await + .unwrap() + .unwrap() + .unwrap(), + }; + assert_eq!(second_hash, hash); + assert_eq!( + store.read_to_vec(&space, &hash, 0).await.unwrap(), + Some(data.clone()) + ); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(data.len() as u64) + ); + assert_eq!(store.remove(&space, &hash, 0).await.unwrap(), Some(())); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn restart_counts_then_collects_crash_leftover_memory_stage() { + let dir = tempfile::tempdir().unwrap(); + let store = FileSystemConfig::new(dir.path()).open().await.unwrap(); + let space: SpaceId = "tinycloud:key:test:stage-restart".parse().unwrap(); + store.create(&space).await.unwrap(); + let hash = tinycloud_core::hash::hash(b"unfinished memory stage"); + let path = store.get_path(&space, &hash, 0); + let prefix = format!(".tc-stage-{}-", path.file_name().unwrap().to_str().unwrap()); + let mut stage = tempfile::Builder::new() + .prefix(&prefix) + .tempfile_in(path.parent().unwrap()) + .unwrap(); + std::io::Write::write_all(&mut stage, b"partial").unwrap(); + let stage_path = stage.path().to_path_buf(); + stage.into_temp_path().keep().unwrap(); + drop(store); + + let reopened = FileSystemConfig::new(dir.path()).open().await.unwrap(); + assert_eq!(reopened.total_size(&space).await.unwrap(), Some(7)); + reopened.remove(&space, &hash, 0).await.unwrap(); + assert_eq!(reopened.total_size(&space).await.unwrap(), Some(0)); + assert!(!stage_path.exists()); + } + + #[tokio::test] + async fn cancelled_delete_finishes_boot_stage_refund_before_retry() { + let dir = tempfile::tempdir().unwrap(); + let store = FileSystemConfig::new(dir.path()).open().await.unwrap(); + let space: SpaceId = "tinycloud:key:test:cancel-stage".parse().unwrap(); + store.create(&space).await.unwrap(); + let value = tinycloud_core::hash::hash(b"cancelled stage delete"); + let target = store.get_path(&space, &value, 0); + let mut stage = tempfile::Builder::new() + .prefix(&FileSystemStore::stage_prefix(&target)) + .tempfile_in(target.parent().unwrap()) + .unwrap(); + std::io::Write::write_all(&mut stage, b"partial").unwrap(); + let stage_path = stage.path().to_path_buf(); + stage.into_temp_path().keep().unwrap(); + drop(store); + let store = FileSystemConfig::new(dir.path()).open().await.unwrap(); + assert_eq!(store.total_size(&space).await.unwrap(), Some(7)); + let entered = Arc::new(Notify::new()); + let release = Arc::new(Notify::new()); + *store.after_stage_unlink.lock().await = Some((entered.clone(), release.clone())); + let deleting_store = store.clone(); + let deleting_space = space.clone(); + let deletion = + tokio::spawn(async move { deleting_store.remove(&deleting_space, &value, 0).await }); + tokio::time::timeout(std::time::Duration::from_secs(2), entered.notified()) + .await + .unwrap(); + assert!(!stage_path.exists()); + deletion.abort(); + assert!(deletion.await.unwrap_err().is_cancelled()); + release.notify_one(); + assert_eq!( + tokio::time::timeout( + std::time::Duration::from_secs(2), + store.remove(&space, &value, 0) + ) + .await + .unwrap() + .unwrap(), + None + ); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn boot_counted_stage_and_failed_retry_refund_both_charges_once() { + use tinycloud_core::sea_orm::{ + ConnectionTrait, Database, DbBackend, Statement, TransactionTrait, + }; + use tinycloud_core::sea_orm_migration::MigratorTrait; + use tinycloud_core::{hash::hash, kv_orphans, migrations::Migrator}; + + let db = Database::connect("sqlite::memory:").await.unwrap(); + Migrator::up(&db, None).await.unwrap(); + db.execute_unprepared("PRAGMA foreign_keys = OFF") + .await + .unwrap(); + let dir = tempfile::tempdir().unwrap(); + let store = FileSystemConfig::new(dir.path()).open().await.unwrap(); + let space: SpaceId = "tinycloud:key:test:stage-and-reservation".parse().unwrap(); + store.create(&space).await.unwrap(); + let bytes = b"retrying the same hash after a crash"; + let value = hash(bytes); + let blob = [(space.clone(), value)]; + let tx = db.begin().await.unwrap(); + kv_orphans::register_candidates(&tx, &blob, "crashed-a") + .await + .unwrap(); + tx.commit().await.unwrap(); + + let target = store.get_path(&space, &value, 0); + let mut stage = tempfile::Builder::new() + .prefix(&FileSystemStore::stage_prefix(&target)) + .tempfile_in(target.parent().unwrap()) + .unwrap(); + std::io::Write::write_all(&mut stage, b"partial").unwrap(); + let boot_stage = stage.path().to_path_buf(); + stage.into_temp_path().keep().unwrap(); + drop(store); + let store = FileSystemConfig::new(dir.path()).open().await.unwrap(); + assert_eq!(store.total_size(&space).await.unwrap(), Some(7)); + + let tx = db.begin().await.unwrap(); + let failed = kv_orphans::register_candidates(&tx, &blob, "failed-b") + .await + .unwrap(); + tx.commit().await.unwrap(); + let entered = Arc::new(Notify::new()); + let release = Arc::new(Notify::new()); + *store.before_memory_write.lock().await = Some((entered.clone(), release)); + let mut retry = HashBuffer::new(Vec::new()); + retry.write_all(bytes).await.unwrap(); + let uploading_store = store.clone(); + let uploading_space = space.clone(); + let uploading = tokio::spawn(async move { + ImmutableWriteStore::::persist( + &uploading_store, + &uploading_space, + retry, + 0, + ) + .await + }); + tokio::time::timeout(std::time::Duration::from_secs(2), entered.notified()) + .await + .unwrap(); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(7 + bytes.len() as u64) + ); + uploading.abort(); + assert!(uploading.await.unwrap_err().is_cancelled()); + kv_orphans::refund_failed(&db, &store, &failed, "failed-b") + .await + .unwrap(); + + let unrelated = b"other"; + let mut other_stage = HashBuffer::new(Vec::new()); + other_stage.write_all(unrelated).await.unwrap(); + let other = + ImmutableWriteStore::::persist(&store, &space, other_stage, 0) + .await + .unwrap(); + db.execute(Statement::from_sql_and_values( + DbBackend::Sqlite, + "UPDATE kv_orphan_attempt SET created_at = ? WHERE attempt_id = ?", + [ + (time::OffsetDateTime::now_utc() - time::Duration::minutes(6)).into(), + "crashed-a".into(), + ], + )) + .await + .unwrap(); + assert_eq!( + kv_orphans::collect_candidates(&db, &store, 256, kv_orphans::ORPHAN_GRACE) + .await + .unwrap(), + 1 + ); + assert!(!boot_stage.exists()); + assert!(!store.contains(&space, &value, 0).await.unwrap()); + assert_eq!( + store.read_to_vec(&space, &other, 0).await.unwrap(), + Some(unrelated.to_vec()) + ); + assert_eq!(store.total_size(&space).await.unwrap(), Some(5)); + assert_eq!( + kv_orphans::collect_candidates(&db, &store, 256, kv_orphans::ORPHAN_GRACE) + .await + .unwrap(), + 0 + ); + assert_eq!(store.total_size(&space).await.unwrap(), Some(5)); + } + + #[tokio::test] + async fn referenced_retry_collects_crashed_stage_without_deleting_committed_blob() { + referenced_retry_with_crashed_stage(false, false, false).await; + } + + #[tokio::test] + async fn referenced_retry_retains_stage_recovery_if_crash_attempt_expires_first() { + referenced_retry_with_crashed_stage(true, false, false).await; + } + + #[tokio::test] + async fn referenced_retry_retains_candidate_when_stage_cleanup_fails() { + referenced_retry_with_crashed_stage(false, true, false).await; + } + + #[tokio::test] + async fn referenced_retry_cancelled_cleanup_completes_stage_refund() { + referenced_retry_with_crashed_stage(false, false, true).await; + } + + async fn referenced_retry_with_crashed_stage( + collect_before_commit: bool, + fail_cleanup: bool, + cancel_cleanup: bool, + ) { + use tinycloud_core::sea_orm::{ + ConnectionTrait, Database, DbBackend, EntityTrait, PaginatorTrait, Statement, + TransactionTrait, + }; + use tinycloud_core::sea_orm_migration::MigratorTrait; + use tinycloud_core::{ + hash::hash, + kv_orphans, + migrations::Migrator, + models::{kv_orphan_attempt, kv_orphan_candidate}, + }; + + let db = Database::connect("sqlite::memory:").await.unwrap(); + Migrator::up(&db, None).await.unwrap(); + db.execute_unprepared("PRAGMA foreign_keys = OFF") + .await + .unwrap(); + let dir = tempfile::tempdir().unwrap(); + let cfg = FileSystemConfig::new(dir.path()); + let store = cfg.open().await.unwrap(); + let space: SpaceId = "tinycloud:key:test:referenced-stage".parse().unwrap(); + store.create(&space).await.unwrap(); + let data = b"complete same-hash retry after crash"; + let value = hash(data); + let blobs = [(space.clone(), value)]; + let tx = db.begin().await.unwrap(); + kv_orphans::register_candidates(&tx, &blobs, "crashed-A") + .await + .unwrap(); + tx.commit().await.unwrap(); + + let path = store.get_path(&space, &value, 0); + let staged = tempfile::Builder::new() + .prefix(&FileSystemStore::stage_prefix(&path)) + .tempfile_in(path.parent().unwrap()) + .unwrap(); + let stage_path = staged.path().to_path_buf(); + std::fs::write(&stage_path, b"partial").unwrap(); + staged.into_temp_path().keep().unwrap(); + drop(store); + let store = cfg.open().await.unwrap(); + assert_eq!(store.total_size(&space).await.unwrap(), Some(7)); + + let tx = db.begin().await.unwrap(); + let registered = kv_orphans::register_candidates(&tx, &blobs, "retry-B") + .await + .unwrap(); + tx.commit().await.unwrap(); + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(data).await.unwrap(); + ImmutableWriteStore::::persist( + &store, + &space, + stage, + registered[0].2, + ) + .await + .unwrap(); + if collect_before_commit { + // A's DB-time claim is older than the five-minute grace; B's + // fresh claim still protects its in-flight upload. + db.execute(Statement::from_sql_and_values( + DbBackend::Sqlite, + "UPDATE kv_orphan_attempt SET created_at = ? WHERE attempt_id = ?", + [ + (time::OffsetDateTime::now_utc() - time::Duration::minutes(6)).into(), + "crashed-A".into(), + ], + )) + .await + .unwrap(); + kv_orphans::collect_candidates(&db, &store, 256, kv_orphans::ORPHAN_GRACE) + .await + .unwrap(); + assert!(stage_path.exists()); + } + let tx = db.begin().await.unwrap(); + kv_orphans::lock_candidates(&tx, ®istered, "retry-B") + .await + .unwrap(); + tx.execute(Statement::from_sql_and_values( + DbBackend::Sqlite, + "INSERT INTO kv_write (space,key,invocation,seq,epoch,epoch_seq,value,metadata,generation) + VALUES (?,'retry',?,1,?,1,?,'{}',?)", + [ + space.to_string().into(), + Vec::::from(hash(b"invocation")).into(), + Vec::::from(hash(b"epoch")).into(), + Vec::::from(value).into(), + registered[0].2.into(), + ], + )) + .await + .unwrap(); + kv_orphans::finish_candidates(&tx, ®istered, "retry-B") + .await + .unwrap(); + tx.commit().await.unwrap(); + + if fail_cleanup { + store.fail_stage_cleanup.store(true, Ordering::SeqCst); + assert!( + kv_orphans::collect_candidates(&db, &store, 256, time::Duration::ZERO) + .await + .is_err() + ); + assert!(stage_path.exists()); + assert_eq!( + kv_orphan_candidate::Entity::find() + .count(&db) + .await + .unwrap(), + 1 + ); + assert_eq!( + kv_orphan_attempt::Entity::find().count(&db).await.unwrap(), + 1 + ); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(7 + data.len() as u64) + ); + store.fail_stage_cleanup.store(false, Ordering::SeqCst); + kv_orphans::refund_failed(&db, &store, ®istered, "crashed-A") + .await + .unwrap(); + } else if cancel_cleanup { + let entered = Arc::new(Notify::new()); + let release = Arc::new(Notify::new()); + *store.after_stage_unlink.lock().await = Some((entered.clone(), release.clone())); + let collecting_db = db.clone(); + let collecting_store = store.clone(); + let collector = tokio::spawn(async move { + kv_orphans::collect_candidates( + &collecting_db, + &collecting_store, + 256, + time::Duration::ZERO, + ) + .await + }); + tokio::time::timeout(std::time::Duration::from_secs(2), entered.notified()) + .await + .unwrap(); + assert!(!stage_path.exists()); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(7 + data.len() as u64) + ); + collector.abort(); + assert!(collector.await.unwrap_err().is_cancelled()); + release.notify_one(); + tokio::time::timeout( + std::time::Duration::from_secs(2), + kv_orphans::refund_failed(&db, &store, ®istered, "crashed-A"), + ) + .await + .unwrap() + .unwrap(); + } else { + kv_orphans::collect_candidates(&db, &store, 256, time::Duration::ZERO) + .await + .unwrap(); + } + assert_eq!( + store.read_to_vec(&space, &value, 0).await.unwrap(), + Some(data.to_vec()), + "GC must preserve the committed retry" + ); + assert!(!stage_path.exists(), "the crashed stage is unreferenced"); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(data.len() as u64) + ); + assert_eq!( + kv_orphan_candidate::Entity::find() + .count(&db) + .await + .unwrap(), + 0 + ); + assert_eq!( + kv_orphan_attempt::Entity::find().count(&db).await.unwrap(), + 0 + ); + } + + #[tokio::test] + async fn expired_crash_stage_is_refunded_after_overlapping_retry_commits_or_aborts() { + use tinycloud_core::sea_orm::{ + ConnectionTrait, Database, DbBackend, EntityTrait, PaginatorTrait, Statement, + TransactionTrait, + }; + use tinycloud_core::sea_orm_migration::MigratorTrait; + use tinycloud_core::{ + hash::hash, + kv_orphans, + migrations::Migrator, + models::{kv_orphan_attempt, kv_orphan_candidate, kv_write}, + }; + + for commits in [true, false] { + let db = Database::connect("sqlite::memory:").await.unwrap(); + Migrator::up(&db, None).await.unwrap(); + db.execute_unprepared("PRAGMA foreign_keys = OFF") + .await + .unwrap(); + let dir = tempfile::tempdir().unwrap(); + let cfg = FileSystemConfig::new(dir.path()); + let store = cfg.open().await.unwrap(); + let space: SpaceId = "tinycloud:key:test:overlap-bound".parse().unwrap(); + store.create(&space).await.unwrap(); + let data = b"complete retry after crashed stage"; + let value = hash(data); + let blobs = [(space.clone(), value)]; + let tx = db.begin().await.unwrap(); + kv_orphans::register_candidates(&tx, &blobs, "crashed-A") + .await + .unwrap(); + tx.commit().await.unwrap(); + let target = store.get_path(&space, &value, 0); + let mut stage = tempfile::Builder::new() + .prefix(&FileSystemStore::stage_prefix(&target)) + .tempfile_in(target.parent().unwrap()) + .unwrap(); + std::io::Write::write_all(&mut stage, b"partial").unwrap(); + let stage_path = stage.path().to_path_buf(); + stage.into_temp_path().keep().unwrap(); + drop(store); + let store = cfg.open().await.unwrap(); + assert_eq!(store.total_size(&space).await.unwrap(), Some(7)); + db.execute(Statement::from_sql_and_values( + DbBackend::Sqlite, + "UPDATE kv_orphan_attempt SET created_at = ? WHERE attempt_id = ?", + [ + (time::OffsetDateTime::now_utc() - time::Duration::minutes(6)).into(), + "crashed-A".into(), + ], + )) + .await + .unwrap(); + + let tx = db.begin().await.unwrap(); + let retry = kv_orphans::register_candidates(&tx, &blobs, "live-B") + .await + .unwrap(); + tx.commit().await.unwrap(); + let mut upload = HashBuffer::new(Vec::new()); + upload.write_all(data).await.unwrap(); + assert_eq!( + ImmutableWriteStore::::persist( + &store, &space, upload, retry[0].2 + ) + .await + .unwrap(), + value + ); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(7 + data.len() as u64) + ); + assert_eq!( + kv_orphans::collect_candidates(&db, &store, 256, kv_orphans::ORPHAN_GRACE) + .await + .unwrap(), + 0, + "A's expired stage must remain charged while B is live" + ); + assert!(stage_path.exists()); + + if commits { + let tx = db.begin().await.unwrap(); + kv_orphans::lock_candidates(&tx, &retry, "live-B") + .await + .unwrap(); + tx.execute(Statement::from_sql_and_values( + DbBackend::Sqlite, + "INSERT INTO kv_write (space,key,invocation,seq,epoch,epoch_seq,value,metadata,generation) + VALUES (?,'retry',?,1,?,1,?,'{}',?)", + [ + space.to_string().into(), + Vec::::from(hash(b"invocation")).into(), + Vec::::from(hash(b"epoch")).into(), + Vec::::from(value).into(), + retry[0].2.into(), + ], + )) + .await + .unwrap(); + kv_orphans::finish_candidates(&tx, &retry, "live-B") + .await + .unwrap(); + tx.commit().await.unwrap(); + } else { + kv_orphans::refund_failed(&db, &store, &retry, "live-B") + .await + .unwrap(); + } + + assert_eq!( + kv_orphans::collect_candidates(&db, &store, 256, kv_orphans::ORPHAN_GRACE) + .await + .unwrap(), + usize::from(!commits), + "A's crash charge must settle on the next pass after B leaves" + ); + assert!(!stage_path.exists()); + assert_eq!( + store.read_to_vec(&space, &value, 0).await.unwrap(), + commits.then(|| data.to_vec()), + "a committed retry must remain readable; an aborted one must be absent" + ); + assert_eq!( + kv_write::Entity::find().count(&db).await.unwrap(), + u64::from(commits) + ); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(if commits { data.len() as u64 } else { 0 }) + ); + assert_eq!( + kv_orphans::collect_candidates(&db, &store, 256, kv_orphans::ORPHAN_GRACE) + .await + .unwrap(), + 0, + "settled crash attempts cannot refund twice" + ); + assert_eq!( + kv_orphan_candidate::Entity::find() + .count(&db) + .await + .unwrap(), + 0 + ); + assert_eq!( + kv_orphan_attempt::Entity::find().count(&db).await.unwrap(), + 0 + ); + } + } + + #[tokio::test] + async fn failed_file_creation_refunds_reserved_quota() { + let dir = tempfile::tempdir().unwrap(); + let store = FileSystemConfig::new(dir.path()).open().await.unwrap(); + let space: SpaceId = "tinycloud:key:test:failed".parse().unwrap(); + store.create(&space).await.unwrap(); + tokio::fs::remove_dir(store.path.join(space.suffix()).join(space.name().as_str())) + .await + .unwrap(); + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(b"failed-write").await.unwrap(); + let hash = stage.hash(); + assert!( + ImmutableWriteStore::::persist(&store, &space, stage, 0) + .await + .is_err() + ); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + assert_eq!(store.remove(&space, &hash, 0).await.unwrap(), None); + assert!(!store.contains(&space, &hash, 0).await.unwrap()); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + + let mut staged = HashBuffer::new( + TempFileSystemStage + .get_staging_buffer(&space) + .await + .unwrap(), + ); + staged.write_all(b"failed-stage").await.unwrap(); + let staged_hash = staged.hash(); + assert!( + ImmutableWriteStore::::persist(&store, &space, staged, 0) + .await + .is_err() + ); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + assert_eq!(store.remove(&space, &staged_hash, 0).await.unwrap(), None); + } + + #[tokio::test] + async fn cancelled_failed_upload_retains_refund_until_quota_lock_is_available() { + let dir = tempfile::tempdir().unwrap(); + let store = FileSystemConfig::new(dir.path()).open().await.unwrap(); + let space: SpaceId = "tinycloud:key:test:cancel-refund".parse().unwrap(); + store.create(&space).await.unwrap(); + let bytes = b"failed upload quota"; + let hash = tinycloud_core::hash::hash(bytes); + let target = store.get_path(&space, &hash, 0); + + let writing = Arc::new(Notify::new()); + let resume_write = Arc::new(Notify::new()); + *store.before_memory_write.lock().await = Some((writing.clone(), resume_write.clone())); + let refunding = Arc::new(Notify::new()); + let resume_refund = Arc::new(Notify::new()); + *store.before_pending_refund.lock().await = + Some((refunding.clone(), resume_refund.clone())); + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(bytes).await.unwrap(); + let uploading_store = store.clone(); + let uploading_space = space.clone(); + let upload = tokio::spawn(async move { + ImmutableWriteStore::::persist( + &uploading_store, + &uploading_space, + stage, + 0, + ) + .await + }); + tokio::time::timeout(std::time::Duration::from_secs(2), writing.notified()) + .await + .unwrap(); + tokio::fs::create_dir(&target).await.unwrap(); + let quota_lock = store.sizes.lock_updates_for_test().await; + resume_write.notify_one(); + tokio::time::timeout(std::time::Duration::from_secs(2), refunding.notified()) + .await + .unwrap(); + resume_refund.notify_one(); + tokio::task::yield_now().await; + upload.abort(); + assert!(upload.await.unwrap_err().is_cancelled()); + drop(quota_lock); + tokio::fs::remove_dir(&target).await.unwrap(); + assert_eq!(store.remove(&space, &hash, 0).await.unwrap(), None); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn cancelled_reservation_cannot_refund_bytes_never_charged() { + let dir = tempfile::tempdir().unwrap(); + let store = FileSystemConfig::new(dir.path()).open().await.unwrap(); + let space: SpaceId = "tinycloud:key:test:cancel-reserve".parse().unwrap(); + store.create(&space).await.unwrap(); + let bytes = b"unacquired charge"; + let hash = tinycloud_core::hash::hash(bytes); + let checked = Arc::new(Notify::new()); + let resume_check = Arc::new(Notify::new()); + *store.after_memory_contains.lock().await = Some((checked.clone(), resume_check.clone())); + let charging = Arc::new(Notify::new()); + let resume_charge = Arc::new(Notify::new()); + *store.before_reserve_charge.lock().await = Some((charging.clone(), resume_charge.clone())); + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(bytes).await.unwrap(); + let uploading_store = store.clone(); + let uploading_space = space.clone(); + let upload = tokio::spawn(async move { + ImmutableWriteStore::::persist( + &uploading_store, + &uploading_space, + stage, + 0, + ) + .await + }); + tokio::time::timeout(std::time::Duration::from_secs(2), checked.notified()) + .await + .unwrap(); + let quota_lock = store.sizes.lock_updates_for_test().await; + resume_check.notify_one(); + tokio::time::timeout(std::time::Duration::from_secs(2), charging.notified()) + .await + .unwrap(); + resume_charge.notify_one(); + tokio::task::yield_now().await; + upload.abort(); + assert!(upload.await.unwrap_err().is_cancelled()); + drop(quota_lock); + assert_eq!(store.remove(&space, &hash, 0).await.unwrap(), None); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + } #[tokio::test] async fn test_file_system_store() { @@ -423,11 +1592,11 @@ mod test { let mut stage = tfs.stage(&space_id).await.unwrap(); futures::io::copy(&mut &data[..], &mut stage).await.unwrap(); - let hash = ImmutableWriteStore::::persist(&store, &space_id, stage) + let hash = ImmutableWriteStore::::persist(&store, &space_id, stage, 0) .await .unwrap(); - assert!(store.contains(&space_id, &hash).await.unwrap()); + assert!(store.contains(&space_id, &hash, 0).await.unwrap()); assert_eq!( store.total_size(&space_id).await.unwrap(), Some(data.len() as u64) @@ -435,7 +1604,7 @@ mod test { let mut buf = Vec::new(); store - .read(&space_id, &hash) + .read(&space_id, &hash, 0) .await .unwrap() .unwrap() @@ -445,7 +1614,11 @@ mod test { assert_eq!(buf, data); assert_eq!( - store.read_to_vec(&space_id, &hash).await.unwrap().unwrap(), + store + .read_to_vec(&space_id, &hash, 0) + .await + .unwrap() + .unwrap(), data ); @@ -453,6 +1626,7 @@ mod test { .read_range( &space_id, &hash, + 0, ByteRangeSpec::Inclusive { start: 6, end: 10 }, ) .await @@ -479,20 +1653,98 @@ mod test { assert!(matches!( store - .read_range(&space_id, &hash, ByteRangeSpec::From { start: 99 }) + .read_range(&space_id, &hash, 0, ByteRangeSpec::From { start: 99 }) .await .unwrap() .unwrap(), RangeRead::Unsatisfiable { total_size: 11 } )); - assert_eq!(store.remove(&space_id, &hash).await.unwrap(), Some(())); - assert_eq!(store.remove(&space_id, &hash).await.unwrap(), None); - assert!(!store.contains(&space_id, &hash).await.unwrap()); + assert_eq!(store.remove(&space_id, &hash, 0).await.unwrap(), Some(())); + assert_eq!(store.remove(&space_id, &hash, 0).await.unwrap(), None); + assert!(!store.contains(&space_id, &hash, 0).await.unwrap()); assert_eq!(store.total_size(&space_id).await.unwrap(), Some(0)); assert_eq!( - store.read(&space_id, &hash).await.unwrap().map(|_| ()), + store.read(&space_id, &hash, 0).await.unwrap().map(|_| ()), None ); } + #[tokio::test] + async fn reads_preexisting_bare_hash_file_as_generation_zero() { + let dir = tempfile::tempdir().unwrap(); + let space: SpaceId = "tinycloud:key:test:old-file".parse().unwrap(); + let initial = FileSystemConfig::new(dir.path()).open().await.unwrap(); + initial.create(&space).await.unwrap(); + let mut data = HashBuffer::new(Vec::new()); + data.write_all(b"legacy").await.unwrap(); + let hash = data.hash(); + let bare = initial.get_path(&space, &hash, 0); + tokio::fs::write(&bare, b"legacy").await.unwrap(); + let loaded = FileSystemConfig::new(dir.path()).open().await.unwrap(); + assert_eq!( + loaded.read_to_vec(&space, &hash, 0).await.unwrap(), + Some(b"legacy".to_vec()) + ); + assert_eq!(loaded.read_to_vec(&space, &hash, 1).await.unwrap(), None); + assert_eq!(loaded.total_size(&space).await.unwrap(), Some(6)); + } + + #[tokio::test] + async fn generations_keep_legacy_key_and_account_for_both_files_on_restart() { + let dir = tempfile::tempdir().unwrap(); + let space: SpaceId = "tinycloud:key:test:generations".parse().unwrap(); + let store = FileSystemConfig::new(dir.path()).open().await.unwrap(); + store.create(&space).await.unwrap(); + let mut old = HashBuffer::new(Vec::new()); + old.write_all(b"same bytes").await.unwrap(); + let hash = old.hash(); + let legacy_path = store.get_path(&space, &hash, 0); + assert_eq!( + legacy_path.file_name().unwrap(), + base64::encode_config(hash.as_ref(), base64::URL_SAFE).as_str() + ); + ImmutableWriteStore::::persist(&store, &space, old, 0) + .await + .unwrap(); + assert_eq!( + store.read_to_vec(&space, &hash, 0).await.unwrap().unwrap(), + b"same bytes" + ); + let mut next = HashBuffer::new(Vec::new()); + next.write_all(b"same bytes").await.unwrap(); + ImmutableWriteStore::::persist(&store, &space, next, 1) + .await + .unwrap(); + assert_ne!(legacy_path, store.get_path(&space, &hash, 1)); + assert_eq!(store.total_size(&space).await.unwrap(), Some(20)); + let restarted = FileSystemConfig::new(dir.path()).open().await.unwrap(); + assert_eq!(restarted.total_size(&space).await.unwrap(), Some(20)); + assert_eq!(store.remove(&space, &hash, 0).await.unwrap(), Some(())); + assert!(store.contains(&space, &hash, 1).await.unwrap()); + assert_eq!( + store.read_to_vec(&space, &hash, 1).await.unwrap().unwrap(), + b"same bytes" + ); + let RangeRead::Content { mut content, .. } = store + .read_range( + &space, + &hash, + 1, + ByteRangeSpec::Inclusive { start: 5, end: 9 }, + ) + .await + .unwrap() + .unwrap() + else { + panic!("expected generation-one range"); + }; + let mut range_bytes = Vec::new(); + content.read_to_end(&mut range_bytes).await.unwrap(); + assert_eq!(range_bytes, b"bytes"); + assert_eq!(store.total_size(&space).await.unwrap(), Some(10)); + assert_eq!(store.remove(&space, &hash, 0).await.unwrap(), None); + assert_eq!(store.total_size(&space).await.unwrap(), Some(10)); + assert_eq!(store.remove(&space, &hash, 1).await.unwrap(), Some(())); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + } } diff --git a/tinycloud-node-server/src/storage/mod.rs b/tinycloud-node-server/src/storage/mod.rs index df299470..802dc7e4 100644 --- a/tinycloud-node-server/src/storage/mod.rs +++ b/tinycloud-node-server/src/storage/mod.rs @@ -1,3 +1,42 @@ pub mod file_system; pub mod s3; pub mod size; + +use tinycloud_auth::resource::SpaceId; +use tinycloud_core::hash::Hash; + +use std::{ + collections::HashMap, + sync::{Arc, Weak}, +}; +use tokio::sync::{Mutex, OwnedMutexGuard}; + +/// One physical blob lifetime; a later generation must never share its +/// quota reservation or deletion target with an earlier generation. +type BlobGenerationKey = (SpaceId, Hash, i64); +#[cfg(test)] +type TestPause = Arc, Arc)>>>; + +/// Serialize physical I/O for one generation, not the entire store. The weak +/// registry drops inactive keys, so a long-running node does not retain one +/// lock per content hash it has ever written or collected. No DB connection +/// is held while this guard waits or while an object-store request runs. +#[derive(Debug, Clone, Default)] +struct BlobOperationLocks(Arc>>>>); + +impl BlobOperationLocks { + async fn lock(&self, key: BlobGenerationKey) -> OwnedMutexGuard<()> { + let lock = { + let mut registry = self.0.lock().await; + registry.retain(|_, weak| weak.strong_count() > 0); + if let Some(lock) = registry.get(&key).and_then(Weak::upgrade) { + lock + } else { + let lock = Arc::new(Mutex::new(())); + registry.insert(key, Arc::downgrade(&lock)); + lock + } + }; + lock.lock_owned().await + } +} diff --git a/tinycloud-node-server/src/storage/s3.rs b/tinycloud-node-server/src/storage/s3.rs index 33aaf0ff..0fc0b57b 100644 --- a/tinycloud-node-server/src/storage/s3.rs +++ b/tinycloud-node-server/src/storage/s3.rs @@ -16,12 +16,26 @@ use futures::{ use rocket::{async_trait, http::hyper::Uri}; use serde::{Deserialize, Serialize}; use serde_with::{serde_as, DisplayFromStr}; -use std::{collections::HashMap, io::Error as IoError, ops::AddAssign, pin::Pin, time::Instant}; +use std::{ + collections::{hash_map::Entry, HashMap}, + fmt::Write as _, + future::Future, + io::Error as IoError, + ops::AddAssign, + pin::Pin, + sync::Arc, + time::Instant, +}; use tinycloud_auth::resource::SpaceId; use tinycloud_core::{hash::Hash, storage::*}; use tokio_util::bytes::Bytes; -use super::{file_system, size::SpaceSizes}; +#[cfg(test)] +use super::TestPause; +use super::{file_system, size::SpaceSizes, BlobGenerationKey, BlobOperationLocks}; +use tokio::sync::Mutex; +#[cfg(test)] +use tokio::sync::Notify; /// Pinned so an SDK bump cannot change client defaults unnoticed. pub fn behavior_version() -> BehaviorVersion { @@ -49,6 +63,19 @@ pub struct S3BlockStore { pub client: Client, pub bucket: String, sizes: SpaceSizes, + // A PUT can succeed on S3 even if its response fails. Reserve its size + // before sending; this shared map refunds a missing object on known + // failure without subtracting another writer's committed bytes. + pending: Arc>>, + // A DELETE can land on S3 while its response is lost. Keep the pre-DELETE + // size until absence is confirmed, so retry refunds exactly once. On + // restart, the initial bucket listing reconstructs only physical bytes. + deleting: Arc>>, + operations: BlobOperationLocks, + #[cfg(test)] + after_delete_head: TestPause, + #[cfg(test)] + after_put_head: TestPause, } #[serde_as] @@ -64,7 +91,18 @@ pub struct S3BlockConfig { impl StorageConfig for S3BlockConfig { type Error = S3Error; async fn open(&self) -> Result { - S3BlockStore::new_(self).await + S3BlockStore::new_(self, &[]).await + } +} + +impl S3BlockConfig { + /// Reconstruct quota and the charge for in-progress DELETEs from one + /// bucket listing, before the node starts its orphan collector. + pub async fn open_with_deleting( + &self, + deleting: &[(SpaceId, Hash, i64)], + ) -> Result { + S3BlockStore::new_(self, deleting).await } } @@ -98,29 +136,55 @@ async fn new_client(config: &S3BlockConfig) -> Client { Client::from_conf(sdk_config) } +async fn file_body( + file: file_system::TempFileStage, +) -> Result<(u64, ByteStream, tempfile::TempPath), S3StoreError> { + let size = file.size().await?; + let (_file, path) = file.into_inner(); + let body = ByteStream::from_path(&path).await?; + Ok((size, body, path)) +} + impl S3BlockStore { - async fn new_(config: &S3BlockConfig) -> Result { + async fn new_( + config: &S3BlockConfig, + deleting: &[(SpaceId, Hash, i64)], + ) -> Result { let client = new_client(config).await; + let deleting_keys: HashMap<_, _> = deleting + .iter() + .map(|(space, hash, generation)| { + ( + Self::key_for(space, hash, *generation), + (space.clone(), *hash, *generation), + ) + }) + .collect(); let mut pages = client .list_objects_v2() .bucket(&config.bucket) .into_paginator() .send(); - // get the sum of all objects per space across all pages + // get the sum of all objects per space across all pages, and the + // physical size of every object a pending DELETE still names let mut acc: HashMap = HashMap::new(); + let mut counted_deletes = HashMap::new(); while let Some(page) = pages.try_next().await.map_err(S3Error::from)? { for content in page.contents() { - let Some((space, size)) = content.key().and_then(|key| { - let (o, _) = key.rsplit_once('/')?; - let space: SpaceId = o.parse().ok()?; - match content.size() { - Some(size) if size > 0 => Some((space, size as u64)), - _ => None, - } - }) else { + let Some(key) = content.key() else { continue }; + let Some((prefix, _)) = key.rsplit_once('/') else { + continue; + }; + let Ok(space) = prefix.parse::() else { continue; }; - acc.entry(space).or_insert(0).add_assign(size); + let size = content.size().unwrap_or_default(); + if size > 0 { + acc.entry(space).or_insert(0).add_assign(size as u64); + } + if let Some((space, hash, generation)) = deleting_keys.get(key) { + counted_deletes.insert((space.clone(), *hash, *generation), size.max(0) as u64); + } } } let sizes = acc.into(); @@ -128,15 +192,31 @@ impl S3BlockStore { client, bucket: config.bucket.clone(), sizes, + pending: Arc::new(Mutex::new(HashMap::new())), + deleting: Arc::new(Mutex::new(counted_deletes)), + operations: BlobOperationLocks::default(), + #[cfg(test)] + after_delete_head: Arc::new(Mutex::new(None)), + #[cfg(test)] + after_put_head: Arc::new(Mutex::new(None)), }) } - fn key(&self, space: &SpaceId, id: &Hash) -> String { - format!( - "{}/{}", - space, - base64::encode_config(id.as_ref(), base64::URL_SAFE) - ) + fn key_for(space: &SpaceId, id: &Hash, generation: i64) -> String { + let mut key = format!("{space}/"); + // Hash digests fit in 64 bytes; encode on the stack so generated + // reads allocate only their storage-key String. + let mut encoded = [0u8; 88]; + let len = base64::encode_config_slice(id.as_ref(), base64::URL_SAFE, &mut encoded); + key.push_str(std::str::from_utf8(&encoded[..len]).expect("base64 is ASCII")); + if generation != 0 { + write!(&mut key, "@{generation}").expect("writing to String cannot fail"); + } + key + } + + fn key(&self, space: &SpaceId, id: &Hash, generation: i64) -> String { + Self::key_for(space, id, generation) } async fn increment_size(&self, space: &SpaceId, size: u64) { @@ -146,35 +226,78 @@ impl S3BlockStore { self.sizes.decrement_size(space, size).await; } - /// The stored size of an object, or `None` if absent. HeadObject, not - /// GetObjectAttributes: that call needs an explicit attribute list, and - /// S3 (and MinIO) reject it without one. - async fn object_size(&self, key: &str) -> Result, S3StoreError> { - match self + async fn put_if_absent( + &self, + space: &SpaceId, + hash: Hash, + generation: i64, + body: F, + ) -> Result<(), S3StoreError> + where + F: FnOnce() -> Fut, + Fut: Future>, + { + let key = (space.clone(), hash, generation); + // Dedup, reservation, PUT and settlement are one physical generation + // operation. A second KV path cannot charge the same bytes again. + let _operation = self.operations.lock(key.clone()).await; + if self.contains(space, &hash, generation).await? { + // A retry of an ambiguous PUT can find the object it uploaded. + self.pending.lock().await.remove(&key); + return Ok(()); + } + #[cfg(test)] + let pause = self.after_put_head.lock().await.take(); + #[cfg(test)] + if let Some((entered, release)) = pause { + entered.notify_one(); + release.notified().await; + } + let (size, body, guard) = body().await?; + let mut pending = self.pending.lock().await; + if let Entry::Vacant(slot) = pending.entry(key.clone()) { + slot.insert(size); + self.increment_size(space, size).await; + } + drop(pending); + let upload = self .client - .head_object() + .put_object() .bucket(&self.bucket) - .key(key) + .key(self.key(space, &hash, generation)) + .body(body) .send() - .await - { - Ok(o) => Ok(Some(o.content_length().unwrap_or_default().try_into()?)), - Err(SdkError::ServiceError(e)) if matches!(e.err(), HeadObjectError::NotFound(_)) => { - Ok(None) - } - Err(e) => Err(S3Error::from(e).into()), + .await; + drop(guard); + upload.map_err(S3Error::from)?; + self.pending.lock().await.remove(&key); + Ok(()) + } + + async fn refund_missing_pending(&self, space: &SpaceId, id: &Hash, generation: i64) { + let key = (space.clone(), *id, generation); + let mut pending = self.pending.lock().await; + if let Some(size) = pending.get(&key).copied() { + self.decrement_size(space, size).await; + pending.remove(&key); } } - async fn delete(&self, key: &str) -> Result<(), S3StoreError> { - self.client - .delete_object() - .bucket(&self.bucket) - .key(key) - .send() - .await - .map_err(S3Error::from)?; - Ok(()) + async fn refund_confirmed_absence(&self, space: &SpaceId, id: &Hash, generation: i64) { + let key = (space.clone(), *id, generation); + let mut deleting = self.deleting.lock().await; + if let Some(size) = deleting.get(&key).copied() { + let mut pending = self.pending.lock().await; + // Hold both reservations until the quota update finishes. Once + // it runs, both removals are synchronous and cancellation cannot + // cause a second refund on the next HEAD 404. + self.decrement_size(space, size).await; + deleting.remove(&key); + pending.remove(&key); + } else { + drop(deleting); + self.refund_missing_pending(space, id, generation).await; + } } } @@ -216,12 +339,17 @@ impl From for S3StoreError { impl ImmutableReadStore for S3BlockStore { type Error = S3StoreError; type Readable = IntoAsyncRead; - async fn contains(&self, space: &SpaceId, id: &Hash) -> Result { + async fn contains( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result { match self .client .head_object() .bucket(&self.bucket) - .key(self.key(space, id)) + .key(self.key(space, id, generation)) .send() .await { @@ -237,13 +365,14 @@ impl ImmutableReadStore for S3BlockStore { &self, space: &SpaceId, id: &Hash, + generation: i64, ) -> Result>, Self::Error> { let start = Instant::now(); let res = self .client .get_object() .bucket(&self.bucket) - .key(self.key(space, id)) + .key(self.key(space, id, generation)) .send() .await; let result = match res { @@ -268,9 +397,10 @@ impl ImmutableReadStore for S3BlockStore { &self, space: &SpaceId, id: &Hash, + generation: i64, range: ByteRangeSpec, ) -> Result>, Self::Error> { - let key = self.key(space, id); + let key = self.key(space, id, generation); let output = self .client .get_object() @@ -300,7 +430,7 @@ impl ImmutableReadStore for S3BlockStore { .client .head_object() .bucket(&self.bucket) - .key(self.key(space, id)) + .key(self.key(space, id, generation)) .send() .await; match head { @@ -353,24 +483,17 @@ impl ImmutableWriteStore for S3BlockStore { &self, space: &SpaceId, staged: HashBuffer<::Writable>, + generation: i64, ) -> Result { let start = Instant::now(); let result = async move { let (mut h, f) = staged.into_inner(); let hash = h.finalize(); - if !self.contains(space, &hash).await? { - let size = f.len() as u64; - self.client - .put_object() - .bucket(&self.bucket) - .key(self.key(space, &hash)) - .body(ByteStream::from(f)) - .send() - .await - .map_err(S3Error::from)?; - self.increment_size(space, size).await; - } + self.put_if_absent(space, hash, generation, || async move { + Ok((f.len() as u64, ByteStream::from(f), ())) + }) + .await?; Ok(hash) } .await; @@ -390,26 +513,15 @@ impl ImmutableWriteStore for S3BlockStore { &self, space: &SpaceId, staged: HashBuffer<::Writable>, + generation: i64, ) -> Result { let start = Instant::now(); let result = async move { let (mut h, f) = staged.into_inner(); let hash = h.finalize(); - if !self.contains(space, &hash).await? { - let size = f.size().await?; - let (_file, path) = f.into_inner(); - - self.client - .put_object() - .bucket(&self.bucket) - .key(self.key(space, &hash)) - .body(ByteStream::from_path(&path).await?) - .send() - .await - .map_err(S3Error::from)?; - self.increment_size(space, size).await; - } + self.put_if_absent(space, hash, generation, || file_body(f)) + .await?; Ok(hash) } .await; @@ -431,41 +543,25 @@ impl ImmutableWriteStore as ImmutableStaging>::Writable>, + generation: i64, ) -> Result { let start = Instant::now(); let result = async move { let (mut h, f) = staged.into_inner(); let hash = h.finalize(); - if !self.contains(space, &hash).await? { - match f { - AsyncEither::Left(t_file) => { - let size = t_file.size().await?; - let (_file, path) = t_file.into_inner(); - self.client - .put_object() - .bucket(&self.bucket) - .key(self.key(space, &hash)) - .body(ByteStream::from_path(&path).await?) - .send() - .await - .map_err(S3Error::from)?; - self.increment_size(space, size).await; - } - AsyncEither::Right(b) => { - let size = b.len() as u64; - self.client - .put_object() - .bucket(&self.bucket) - .key(self.key(space, &hash)) - .body(ByteStream::from(b)) - .send() - .await - .map_err(S3Error::from)?; - self.increment_size(space, size).await; - } + match f { + AsyncEither::Left(t_file) => { + self.put_if_absent(space, hash, generation, || file_body(t_file)) + .await?; + } + AsyncEither::Right(b) => { + self.put_if_absent(space, hash, generation, || async move { + Ok((b.len() as u64, ByteStream::from(b), ())) + }) + .await?; } - }; + } Ok(hash) } .await; @@ -481,14 +577,65 @@ impl ImmutableWriteStore Result, Self::Error> { - let key = self.key(space, id); - let Some(size) = self.object_size(&key).await? else { - return Ok(None); + async fn remove( + &self, + space: &SpaceId, + id: &Hash, + generation: i64, + ) -> Result, Self::Error> { + // A stale positive HEAD must not recreate a charge after another + // deleter has already confirmed absence and refunded it. + let _operation = self.operations.lock((space.clone(), *id, generation)).await; + let size: u64 = match self + .client + .head_object() + .bucket(&self.bucket) + .key(self.key(space, id, generation)) + .send() + .await + { + Ok(object) => object.content_length().unwrap_or_default().try_into()?, + Err(SdkError::ServiceError(e)) if matches!(e.err(), HeadObjectError::NotFound(_)) => { + self.refund_confirmed_absence(space, id, generation).await; + return Ok(None); + } + Err(error) => return Err(S3Error::from(error).into()), }; - self.delete(&key).await?; - self.decrement_size(space, size).await; - Ok(Some(())) + #[cfg(test)] + let pause = self.after_delete_head.lock().await.take(); + #[cfg(test)] + if let Some((entered, release)) = pause { + entered.notify_one(); + release.notified().await; + } + self.deleting + .lock() + .await + .entry((space.clone(), *id, generation)) + .or_insert(size); + match self + .client + .delete_object() + .bucket(&self.bucket) + .key(self.key(space, id, generation)) + .send() + .await + { + Ok(_) => { + self.refund_confirmed_absence(space, id, generation).await; + Ok(Some(())) + } + Err(e) => Err(S3Error::from(e).into()), + } + } + async fn cleanup_stages( + &self, + _space: &SpaceId, + _id: &Hash, + _generation: i64, + ) -> Result<(), Self::Error> { + // Staging is not in the bucket; an incomplete S3 PUT has no visible key. + Ok(()) } } @@ -507,6 +654,564 @@ impl StoreSize for S3BlockStore { mod tests { use super::*; + #[tokio::test] + async fn filesystem_body_keeps_staged_file_until_deferred_read() { + use futures::io::AsyncWriteExt; + + let data = b"nonempty deferred S3 upload"; + let mut file = file_system::TempFileStage::new(tempfile::NamedTempFile::new().unwrap()); + file.write_all(data).await.unwrap(); + file.flush().await.unwrap(); + + let (size, body, guard) = file_body(file).await.unwrap(); + assert_eq!(size, data.len() as u64); + assert_eq!(body.collect().await.unwrap().into_bytes().as_ref(), data); + drop(guard); + } + + #[tokio::test] + async fn minio_persists_each_staging_arm_and_accepts_restage_retry() { + use futures::io::AsyncWriteExt; + + let Ok(endpoint) = std::env::var("TC735_TEST_MINIO_ENDPOINT") else { + return; + }; + let bucket = std::env::var("TC735_TEST_MINIO_BUCKET") + .expect("TC735_TEST_MINIO_BUCKET is required with TC735_TEST_MINIO_ENDPOINT"); + let store = S3BlockConfig { + bucket: bucket.clone(), + endpoint: Some(endpoint.parse().unwrap()), + } + .open() + .await + .unwrap(); + let nonce = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos(); + let space: SpaceId = format!("tinycloud:key:test:tc735-stage-{nonce}") + .parse() + .unwrap(); + store.create(&space).await.unwrap(); + let mut hashes = Vec::new(); + + for mode in 0..3 { + let data = format!( + "TC-735 S3 staged upload {nonce} {} {mode}", + std::process::id() + ) + .into_bytes(); + let mut initial_hash = None; + for _ in 0..2 { + // A retry re-stages the same bytes; it never rewinds an SDK body. + let hash = if mode == 0 { + let mut stage = file_system::TempFileSystemStage + .stage(&space) + .await + .unwrap(); + stage.write_all(&data).await.unwrap(); + ImmutableWriteStore::::persist( + &store, &space, stage, 0, + ) + .await + .unwrap() + } else { + let staging = if mode == 1 { + either::Either::A(file_system::TempFileSystemStage) + } else { + either::Either::B(memory::MemoryStaging) + }; + let mut stage = staging.stage(&space).await.unwrap(); + stage.write_all(&data).await.unwrap(); + ImmutableWriteStore::< + either::Either, + >::persist(&store, &space, stage, 0) + .await + .unwrap() + }; + assert_eq!(*initial_hash.get_or_insert(hash), hash); + if !hashes.contains(&hash) { + hashes.push(hash); + } + let object = store + .client + .get_object() + .bucket(&bucket) + .key(store.key(&space, &hash, 0)) + .send() + .await + .unwrap(); + assert_eq!( + object.body.collect().await.unwrap().into_bytes().as_ref(), + data.as_slice() + ); + } + } + for hash in hashes { + store.remove(&space, &hash, 0).await.unwrap(); + } + } + + #[tokio::test] + async fn minio_same_generation_concurrent_put_charges_once() { + use futures::io::AsyncWriteExt; + let Ok(endpoint) = std::env::var("TC735_TEST_MINIO_ENDPOINT") else { + return; + }; + let bucket = std::env::var("TC735_TEST_MINIO_BUCKET") + .expect("TC735_TEST_MINIO_BUCKET is required with TC735_TEST_MINIO_ENDPOINT"); + let store = S3BlockConfig { + bucket, + endpoint: Some(endpoint.parse().unwrap()), + } + .open() + .await + .unwrap(); + let nonce = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos(); + let space: SpaceId = format!("tinycloud:key:test:tc735-same-charge-{nonce}") + .parse() + .unwrap(); + store.create(&space).await.unwrap(); + let data = b"same bytes under two KV paths"; + let entered = Arc::new(Notify::new()); + let release = Arc::new(Notify::new()); + *store.after_put_head.lock().await = Some((entered.clone(), release.clone())); + let mut first_stage = memory::MemoryStaging.stage(&space).await.unwrap(); + first_stage.write_all(data).await.unwrap(); + let hash = first_stage.hash(); + let first_store = store.clone(); + let first_space = space.clone(); + let first = tokio::spawn(async move { + ImmutableWriteStore::::persist( + &first_store, + &first_space, + first_stage, + 0, + ) + .await + }); + tokio::time::timeout(std::time::Duration::from_secs(3), entered.notified()) + .await + .unwrap(); + let mut second_stage = memory::MemoryStaging.stage(&space).await.unwrap(); + second_stage.write_all(data).await.unwrap(); + let second_store = store.clone(); + let second_space = space.clone(); + let mut second = tokio::spawn(async move { + ImmutableWriteStore::::persist( + &second_store, + &second_space, + second_stage, + 0, + ) + .await + }); + let early_second = + tokio::time::timeout(std::time::Duration::from_secs(2), &mut second).await; + release.notify_one(); + assert_eq!(first.await.unwrap().unwrap(), hash); + let second_hash = match early_second { + Ok(result) => result.unwrap().unwrap(), + Err(_) => second.await.unwrap().unwrap(), + }; + assert_eq!(second_hash, hash); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(data.len() as u64) + ); + assert_eq!( + store.read_to_vec(&space, &hash, 0).await.unwrap(), + Some(data.to_vec()) + ); + assert_eq!(store.remove(&space, &hash, 0).await.unwrap(), Some(())); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + } + + #[tokio::test] + async fn minio_stale_head_cannot_recreate_refunded_deletion_charge() { + use futures::io::AsyncWriteExt; + let Ok(endpoint) = std::env::var("TC735_TEST_MINIO_ENDPOINT") else { + return; + }; + let bucket = std::env::var("TC735_TEST_MINIO_BUCKET") + .expect("TC735_TEST_MINIO_BUCKET is required with TC735_TEST_MINIO_ENDPOINT"); + let store = S3BlockConfig { + bucket, + endpoint: Some(endpoint.parse().unwrap()), + } + .open() + .await + .unwrap(); + let nonce = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos(); + let space: SpaceId = format!("tinycloud:key:test:tc735-stale-head-{nonce}") + .parse() + .unwrap(); + store.create(&space).await.unwrap(); + let data = b"same generation charge"; + let mut stage = memory::MemoryStaging.stage(&space).await.unwrap(); + stage.write_all(data).await.unwrap(); + let hash = ImmutableWriteStore::::persist(&store, &space, stage, 0) + .await + .unwrap(); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(data.len() as u64) + ); + + let entered = Arc::new(Notify::new()); + let release = Arc::new(Notify::new()); + *store.after_delete_head.lock().await = Some((entered.clone(), release.clone())); + let first_store = store.clone(); + let first_space = space.clone(); + let first = tokio::spawn(async move { first_store.remove(&first_space, &hash, 0).await }); + tokio::time::timeout(std::time::Duration::from_secs(3), entered.notified()) + .await + .unwrap(); + let second_store = store.clone(); + let second_space = space.clone(); + let mut second = + tokio::spawn(async move { second_store.remove(&second_space, &hash, 0).await }); + // Without per-generation serialization, the second DELETE completes + // while the first retains a stale positive HEAD. With it, the second + // waits and then observes the confirmed absence. + let early_second = + tokio::time::timeout(std::time::Duration::from_secs(2), &mut second).await; + release.notify_one(); + let first_result = first.await.unwrap().unwrap(); + let second_result = match early_second { + Ok(result) => result.unwrap().unwrap(), + Err(_) => second.await.unwrap().unwrap(), + }; + assert!(first_result.is_some() || second_result.is_some()); + assert!(!store.contains(&space, &hash, 0).await.unwrap()); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(0), + "the same generation can refund physical bytes only once" + ); + } + + #[tokio::test] + async fn minio_lost_delete_response_refunds_once_and_restart_reconstructs() { + use futures::io::AsyncWriteExt; + + let Ok(endpoint) = std::env::var("TC735_TEST_MINIO_ENDPOINT") else { + return; + }; + let bucket = std::env::var("TC735_TEST_MINIO_BUCKET") + .expect("TC735_TEST_MINIO_BUCKET is required with TC735_TEST_MINIO_ENDPOINT"); + let config = S3BlockConfig { + bucket: bucket.clone(), + endpoint: Some(endpoint.parse().unwrap()), + }; + let store = config.open().await.unwrap(); + let nonce = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos(); + let space: SpaceId = format!("tinycloud:key:test:tc735-delete-{nonce}") + .parse() + .unwrap(); + store.create(&space).await.unwrap(); + let data = format!("TC-735 ambiguous DELETE {} {nonce}", std::process::id()).into_bytes(); + let mut stage = memory::MemoryStaging.stage(&space).await.unwrap(); + stage.write_all(&data).await.unwrap(); + let hash = ImmutableWriteStore::::persist(&store, &space, stage, 0) + .await + .unwrap(); + assert_eq!( + store.total_size(&space).await.unwrap(), + Some(data.len() as u64) + ); + + // A replacement process lists the object *before* the old DELETE + // lands. Its deletion reservation must come from that same listing. + let counted_boot = config + .open_with_deleting(&[(space.clone(), hash, 0)]) + .await + .unwrap(); + let counted_before = counted_boot.total_size(&space).await.unwrap().unwrap(); + assert_eq!( + counted_boot + .deleting + .lock() + .await + .get(&(space.clone(), hash, 0)), + Some(&(data.len() as u64)) + ); + // DELETE lands on S3, but its response is lost. The original process + // and the boot-counted process each reconcile their own physical count. + store + .deleting + .lock() + .await + .insert((space.clone(), hash, 0), data.len() as u64); + store + .client + .delete_object() + .bucket(&bucket) + .key(store.key(&space, &hash, 0)) + .send() + .await + .unwrap(); + assert_eq!(store.remove(&space, &hash, 0).await.unwrap(), None); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + assert_eq!(store.remove(&space, &hash, 0).await.unwrap(), None); + assert_eq!(store.total_size(&space).await.unwrap(), Some(0)); + assert_eq!(counted_boot.remove(&space, &hash, 0).await.unwrap(), None); + assert_eq!( + counted_boot.total_size(&space).await.unwrap(), + Some(counted_before - data.len() as u64) + ); + assert_eq!(counted_boot.remove(&space, &hash, 0).await.unwrap(), None); + assert_eq!( + counted_boot.total_size(&space).await.unwrap(), + Some(counted_before - data.len() as u64) + ); + + // DELETE before startup, or a crash after the first refund but before + // the DB fence was removed: neither boot listing counted this object. + let after_boot = config + .open_with_deleting(&[(space.clone(), hash, 0)]) + .await + .unwrap(); + let rebuilt = after_boot.total_size(&space).await.unwrap(); + assert!(!after_boot + .deleting + .lock() + .await + .contains_key(&(space.clone(), hash, 0))); + assert_eq!(after_boot.remove(&space, &hash, 0).await.unwrap(), None); + assert_eq!(after_boot.total_size(&space).await.unwrap(), rebuilt); + } + + #[tokio::test] + async fn minio_postgres_boot_reconciles_deleting_fence_from_same_listing() { + use futures::io::AsyncWriteExt; + use tinycloud_core::{ + db::migrate, + db_pool::RecoveringConnection, + kv_orphans::{collect_candidates, deleting_fences, register_candidates}, + sea_orm::{ + ConnectOptions, ConnectionTrait, Database, DbBackend, Statement, TransactionTrait, + }, + }; + + let (Ok(endpoint), Ok(pg_url)) = ( + std::env::var("TC735_TEST_MINIO_ENDPOINT"), + std::env::var("TINYCLOUD_TEST_POSTGRES_URL"), + ) else { + return; + }; + let bucket = std::env::var("TC735_TEST_MINIO_BUCKET") + .expect("TC735_TEST_MINIO_BUCKET is required with TC735_TEST_MINIO_ENDPOINT"); + let nonce = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos(); + let schema = format!("tc735_s3_boot_{}_{}", std::process::id(), nonce); + let admin = Database::connect(pg_url.as_str()).await.unwrap(); + admin + .execute(Statement::from_string( + DbBackend::Postgres, + format!("CREATE SCHEMA {schema}"), + )) + .await + .unwrap(); + let mut options = ConnectOptions::new(pg_url); + options.set_schema_search_path(schema.clone()); + let conn = Database::connect(options).await.unwrap(); + migrate(&RecoveringConnection::new(conn.clone())) + .await + .unwrap(); + + let config = S3BlockConfig { + bucket: bucket.clone(), + endpoint: Some(endpoint.parse().unwrap()), + }; + let uploader = config.open().await.unwrap(); + let space: SpaceId = format!("tinycloud:key:test:tc735-boot-{nonce}") + .parse() + .unwrap(); + uploader.create(&space).await.unwrap(); + for delete_before_boot in [true, false] { + let data = format!("TC-735 PG boot {nonce} {delete_before_boot}").into_bytes(); + let mut stage = memory::MemoryStaging.stage(&space).await.unwrap(); + stage.write_all(&data).await.unwrap(); + let hash = + ImmutableWriteStore::::persist(&uploader, &space, stage, 0) + .await + .unwrap(); + let tx = conn.begin().await.unwrap(); + let registered = register_candidates(&tx, &[(space.clone(), hash)], "aborted") + .await + .unwrap(); + assert_eq!(registered, vec![(space.clone(), hash, 0)]); + tx.execute(Statement::from_sql_and_values( + DbBackend::Postgres, + "DELETE FROM kv_orphan_attempt WHERE space = $1 AND value = $2", + [space.to_string().into(), hash.into()], + )) + .await + .unwrap(); + tx.execute(Statement::from_sql_and_values( + DbBackend::Postgres, + "UPDATE kv_orphan_candidate SET deleting_at = CURRENT_TIMESTAMP - INTERVAL '1 second', + deletion_bytes = $3 WHERE space = $1 AND value = $2", + [ + space.to_string().into(), + hash.into(), + (data.len() as i64).into(), + ], + )) + .await + .unwrap(); + tx.commit().await.unwrap(); + let deleting = deleting_fences(&conn).await.unwrap(); + assert_eq!(deleting, vec![(space.clone(), hash, 0)]); + if delete_before_boot { + uploader + .client + .delete_object() + .bucket(&bucket) + .key(uploader.key(&space, &hash, 0)) + .send() + .await + .unwrap(); + } + let boot = config.open_with_deleting(&deleting).await.unwrap(); + let counted = boot.total_size(&space).await.unwrap().unwrap_or(0); + assert_eq!( + boot.deleting + .lock() + .await + .contains_key(&(space.clone(), hash, 0)), + !delete_before_boot, + ); + if !delete_before_boot { + assert_eq!(counted, data.len() as u64); + uploader + .client + .delete_object() + .bucket(&bucket) + .key(uploader.key(&space, &hash, 0)) + .send() + .await + .unwrap(); + } + assert_eq!( + collect_candidates(&conn, &boot, 10, time::Duration::ZERO) + .await + .unwrap(), + 0 + ); + assert!(deleting_fences(&conn).await.unwrap().is_empty()); + assert_eq!( + boot.total_size(&space).await.unwrap().unwrap_or(0), + if delete_before_boot { + counted + } else { + counted - data.len() as u64 + }, + ); + } + conn.close().await.unwrap(); + admin + .execute(Statement::from_string( + DbBackend::Postgres, + format!("DROP SCHEMA {schema} CASCADE"), + )) + .await + .unwrap(); + admin.close().await.unwrap(); + } + + #[tokio::test] + async fn minio_generations_survive_old_delete_and_rebuild_quota_together() { + use futures::io::{AsyncReadExt, AsyncWriteExt}; + let Ok(endpoint) = std::env::var("TC735_TEST_MINIO_ENDPOINT") else { + return; + }; + let bucket = std::env::var("TC735_TEST_MINIO_BUCKET") + .expect("TC735_TEST_MINIO_BUCKET is required with TC735_TEST_MINIO_ENDPOINT"); + let config = S3BlockConfig { + bucket, + endpoint: Some(endpoint.parse().unwrap()), + }; + let store = config.open().await.unwrap(); + let nonce = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos(); + let space: SpaceId = format!("tinycloud:key:test:tc735-generation-{nonce}") + .parse() + .unwrap(); + store.create(&space).await.unwrap(); + let mut old = memory::MemoryStaging.stage(&space).await.unwrap(); + old.write_all(b"same bytes").await.unwrap(); + let hash = old.hash(); + let legacy_key = format!( + "{}/{}", + space, + base64::encode_config(hash.as_ref(), base64::URL_SAFE) + ); + assert_eq!(store.key(&space, &hash, 0), legacy_key); + ImmutableWriteStore::::persist(&store, &space, old, 0) + .await + .unwrap(); + let mut next = memory::MemoryStaging.stage(&space).await.unwrap(); + next.write_all(b"same bytes").await.unwrap(); + ImmutableWriteStore::::persist(&store, &space, next, 1) + .await + .unwrap(); + assert_eq!(store.key(&space, &hash, 1), format!("{legacy_key}@1")); + let boot = config + .open_with_deleting(&[(space.clone(), hash, 0)]) + .await + .unwrap(); + assert_eq!(boot.total_size(&space).await.unwrap(), Some(20)); + assert_eq!( + boot.deleting.lock().await.get(&(space.clone(), hash, 0)), + Some(&10) + ); + assert_eq!( + boot.read_to_vec(&space, &hash, 0).await.unwrap(), + Some(b"same bytes".to_vec()) + ); + let RangeRead::Content { mut content, .. } = boot + .read_range( + &space, + &hash, + 1, + ByteRangeSpec::Inclusive { start: 5, end: 9 }, + ) + .await + .unwrap() + .unwrap() + else { + panic!("expected generation-one range"); + }; + let mut range_bytes = Vec::new(); + content.read_to_end(&mut range_bytes).await.unwrap(); + assert_eq!(range_bytes, b"bytes"); + assert_eq!(boot.remove(&space, &hash, 0).await.unwrap(), Some(())); + assert_eq!(boot.remove(&space, &hash, 0).await.unwrap(), None); + assert!(boot.contains(&space, &hash, 1).await.unwrap()); + assert_eq!( + boot.read_to_vec(&space, &hash, 1).await.unwrap(), + Some(b"same bytes".to_vec()) + ); + assert_eq!(boot.total_size(&space).await.unwrap(), Some(10)); + boot.remove(&space, &hash, 1).await.unwrap(); + assert_eq!(boot.total_size(&space).await.unwrap(), Some(0)); + } + #[test] fn parses_s3_content_range() { let (range, total_size) = parse_content_range("bytes 1048576-2097151/67108864").unwrap(); diff --git a/tinycloud-node-server/src/storage/size.rs b/tinycloud-node-server/src/storage/size.rs index 4119daee..56077bec 100644 --- a/tinycloud-node-server/src/storage/size.rs +++ b/tinycloud-node-server/src/storage/size.rs @@ -26,6 +26,12 @@ impl SpaceSizes { *s = s.saturating_sub(size); } } + #[cfg(test)] + pub(crate) async fn lock_updates_for_test( + &self, + ) -> tokio::sync::RwLockWriteGuard<'_, HashMap> { + self.0.write().await + } pub async fn get_size(&self, space: &SpaceId) -> Option { self.0.read().await.get(space).copied() } diff --git a/tinycloud-node-server/tests/postgres_connection_recovery.rs b/tinycloud-node-server/tests/postgres_connection_recovery.rs index 14bb70ff..1a67361e 100644 --- a/tinycloud-node-server/tests/postgres_connection_recovery.rs +++ b/tinycloud-node-server/tests/postgres_connection_recovery.rs @@ -800,6 +800,118 @@ async fn postgres_requests_survive_one_stale_pooled_connection() { fixture.drop_schema().await; } +/// Ordinary KV writes begin with an idempotent preparatory transaction. +/// A stale idle connection must not produce an orphan or prevent the write. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn postgres_ordinary_put_recovers_stale_connection_without_orphan() { + let Some(url) = postgres_url("postgres_ordinary_put_recovers_stale_connection_without_orphan") + else { + return; + }; + let mut fixture = Fixture::new("candidate", &url, None).await; + fixture.fill_pool().await; + let victim = fixture.order_idle_queue(0).await; + fixture.terminate(&BTreeSet::from([victim])).await; + let before = fixture.stats(); + fixture + .put("after/preflight/stale") + .await + .expect("ordinary write survives stale pooled connection"); + let recovered = delta(before, fixture.stats()); + assert_eq!( + (recovered.losses, recovered.retries), + (1, 1), + "{recovered:?}" + ); + assert_eq!(fixture.row_count("kv_orphan_candidate").await, 0); + assert_eq!(fixture.row_count("kv_orphan_attempt").await, 0); + fixture.get("after/preflight/stale").await.unwrap(); + fixture.drop_schema().await; +} + +/// A protected write crosses the durable freeze row before storage.persist. +/// Losing an idle pooled connection must not bypass that fence or strand its +/// candidate; the resulting KV remains readable after the recovered commit. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn postgres_protected_preupload_fence_survives_stale_connection() { + let Some(url) = postgres_url("postgres_protected_preupload_fence_survives_stale_connection") + else { + return; + }; + let mut fixture = Fixture::new("protected", &url, None).await; + fixture.fill_pool().await; + let victim = fixture.order_idle_queue(0).await; + fixture.terminate(&BTreeSet::from([victim])).await; + let before = fixture.stats(); + let key = "xyz.tinycloud.tinychat/connectors/fireflies/transcript/stale-pool"; + fixture + .put(key) + .await + .expect("protected put survives reconnect"); + fixture.get(key).await.expect("committed blob is readable"); + let recovered = delta(before, fixture.stats()); + assert_eq!( + (recovered.losses, recovered.retries), + (1, 1), + "{recovered:?}" + ); + assert_eq!(fixture.row_count("kv_orphan_candidate").await, 0); + fixture.drop_schema().await; +} + +/// A dead backend during the protected row lock rolls back that transaction. +/// Because storage has not run yet, the invocation can acquire a fresh +/// transaction, retake the fence, then upload and commit exactly once. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn postgres_preupload_freeze_lock_recovers_a_lost_transaction() { + let Some(url) = postgres_url("postgres_preupload_freeze_lock_recovers_a_lost_transaction") + else { + return; + }; + let mut fixture = Fixture::new("guardloss", &url, None).await; + fixture + .admin + .execute_unprepared(&format!( + "CREATE SEQUENCE {schema}.kill_once; \ + CREATE FUNCTION {schema}.kill_guard_once() RETURNS trigger LANGUAGE plpgsql AS $$ \ + BEGIN \ + IF nextval('{schema}.kill_once') = 1 THEN \ + PERFORM pg_terminate_backend(pg_backend_pid()); \ + END IF; \ + RETURN NEW; \ + END $$; \ + CREATE TRIGGER kill_guard_once BEFORE INSERT ON {schema}.meeting_legacy_write_guard \ + FOR EACH ROW EXECUTE FUNCTION {schema}.kill_guard_once()", + schema = fixture.schema + )) + .await + .unwrap(); + let key = "xyz.tinycloud.tinychat/connectors/fireflies/transcript/guard-loss"; + fixture + .put(key) + .await + .expect("retry preupload transaction after dead guard connection"); + fixture + .get(key) + .await + .expect("only committed value survives"); + let killed = fixture + .admin + .query_one(Statement::from_string( + DbBackend::Postgres, + format!("SELECT last_value AS n FROM {}.kill_once", fixture.schema), + )) + .await + .unwrap() + .unwrap() + .try_get::("", "n") + .unwrap(); + assert_eq!(killed, 2, "row lock was retried exactly once"); + assert_eq!(fixture.row_count("kv_write").await, 2); // seed plus this put + assert_eq!(fixture.row_count("kv_orphan_candidate").await, 0); + fixture.drop_schema().await; +} + /// A network cut: a proxy resets every client connection while idle (EOF/RST, /// no FATAL message). Requests still succeed. #[tokio::test(flavor = "multi_thread", worker_threads = 4)] diff --git a/tinycloud-node-server/tests/postgres_kv_write_throughput.rs b/tinycloud-node-server/tests/postgres_kv_write_throughput.rs index b0a9985d..58857cc8 100644 --- a/tinycloud-node-server/tests/postgres_kv_write_throughput.rs +++ b/tinycloud-node-server/tests/postgres_kv_write_throughput.rs @@ -54,7 +54,8 @@ use tinycloud_core::sea_orm::{ }; use tinycloud_core::storage::memory::{MemoryStaging, MemoryStore}; use tinycloud_core::storage::{ - HashBuffer, ImmutableReadStore, ImmutableWriteStore, StorageConfig, StorageSetup, + HashBuffer, ImmutableDeleteStore, ImmutableReadStore, ImmutableWriteStore, StorageConfig, + StorageSetup, StoreSize, }; use tinycloud_core::types::{Metadata, SpaceIdWrap}; use tinycloud_core::{SpaceDatabase, TxError, TxStoreError}; @@ -73,6 +74,7 @@ type Db = SpaceDatabase; trait BenchStore: ImmutableWriteStore + ImmutableReadStore + + ImmutableDeleteStore + StorageSetup + Clone + Send @@ -84,6 +86,7 @@ trait BenchStore: impl BenchStore for B where B: ImmutableWriteStore + ImmutableReadStore + + ImmutableDeleteStore + StorageSetup + Clone + Send @@ -187,6 +190,8 @@ struct CellReport { attempted: usize, ok: usize, wall: Duration, + started_at: time::OffsetDateTime, + ended_at: time::OffsetDateTime, p50: Duration, p95: Duration, errors: BTreeMap, @@ -327,6 +332,7 @@ async fn run_cell( batches.push(batch); } + let started_at = time::OffsetDateTime::now_utc(); let start = Instant::now(); let handles = batches .into_iter() @@ -366,6 +372,7 @@ async fn run_cell( } } let wall = start.elapsed(); + let ended_at = time::OffsetDateTime::now_utc(); latencies.sort_unstable(); let spaces = owners.iter().map(|o| o.space.clone()).collect::>(); @@ -377,6 +384,8 @@ async fn run_cell( attempted: latencies.len(), ok, wall, + started_at, + ended_at, p50: percentile(&latencies, 50), p95: percentile(&latencies, 95), errors, @@ -403,9 +412,9 @@ fn print_table(store: &str, reports: &[CellReport]) { println!(); println!( "| layout | concurrency | puts | ok | errors | ok ops/s | p50 ms | p95 ms | \ - event_order rows | dup (space,seq) groups | error kinds |" + event_order rows | dup (space,seq) groups | UTC start | UTC end | error kinds |" ); - println!("|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|"); + println!("|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|---|---|"); for r in reports { let error_count = r.attempted - r.ok; let kinds = if r.errors.is_empty() { @@ -418,7 +427,7 @@ fn print_table(store: &str, reports: &[CellReport]) { .join("; ") }; println!( - "| {} | {} | {} | {} | {} | {:.1} | {:.2} | {:.2} | {} | {} | {} |", + "| {} | {} | {} | {} | {} | {:.1} | {:.2} | {:.2} | {} | {} | {} | {} | {} |", r.layout.label(), r.concurrency, r.attempted, @@ -429,6 +438,8 @@ fn print_table(store: &str, reports: &[CellReport]) { ms(r.p95), r.event_rows, r.duplicate_seq_groups, + r.started_at, + r.ended_at, kinds, ); } @@ -514,6 +525,127 @@ async fn postgres_kv_write_throughput() { print_table(&store, &reports); } +/// Exercise the real S3 delete/accounting path for a rolled-back upload. +/// The regular benchmark intentionally has no failed writes. +#[tokio::test] +#[ignore = "manual PG16 + S3 integration test"] +async fn postgres_s3_orphan_cleanup_reclaims_space_usage() { + let database_url = std::env::var("TINYCLOUD_TEST_POSTGRES_URL").unwrap(); + let config: S3BlockConfig = serde_json::from_value(serde_json::json!({ + "bucket": std::env::var("TC732_BENCH_S3_BUCKET").unwrap(), + "endpoint": std::env::var("TC732_BENCH_S3_ENDPOINT").unwrap(), + })) + .unwrap(); + let store: S3BlockStore = config.open().await.unwrap(); + let admin = Database::connect(ConnectOptions::new(database_url.clone())) + .await + .unwrap(); + let schema = format!( + "tc735gc_{}_{}", + std::process::id(), + time::OffsetDateTime::now_utc().unix_timestamp_nanos() + ); + admin + .execute(Statement::from_string( + DbBackend::Postgres, + format!("CREATE SCHEMA {schema}"), + )) + .await + .unwrap(); + let mut options = ConnectOptions::new(database_url); + options.set_schema_search_path(schema.clone()); + let conn = Database::connect(options).await.unwrap(); + let _db = SpaceDatabase::new( + conn.clone(), + store.clone(), + StaticSecret::new(vec![0; 32]).unwrap(), + ) + .await + .unwrap(); + let owner = Owner::generate().unwrap(); + store.create(&owner.space).await.unwrap(); + let mut stage = HashBuffer::new(Vec::new()); + stage.write_all(b"tc735-orphan-cleanup").await.unwrap(); + let hash = stage.hash(); + let first = tinycloud_core::kv_orphans::register_candidates( + &conn, + &[(owner.space.clone(), hash)], + "crashed", + ) + .await + .unwrap(); + assert_eq!(first[0].2, 0, "the first upload uses the bare key"); + assert_eq!( + ImmutableWriteStore::::persist(&store, &owner.space, stage, first[0].2) + .await + .unwrap(), + hash + ); + assert_eq!(store.total_size(&owner.space).await.unwrap(), Some(20)); + assert_eq!( + tinycloud_core::kv_orphans::collect_candidates(&conn, &store, 1, time::Duration::ZERO,) + .await + .unwrap(), + 1 + ); + assert!(!store + .contains(&owner.space, &hash, first[0].2) + .await + .unwrap()); + assert_eq!(store.total_size(&owner.space).await.unwrap(), Some(0)); + // Simulate a known failure whose object removal succeeded but whose + // candidate-row deletion did not commit. A later GC pass must not debit + // the physical-byte counter a second time. + let mut failed = HashBuffer::new(Vec::new()); + failed.write_all(b"tc735-known-failure").await.unwrap(); + let failed_hash = failed.hash(); + let candidate = [(owner.space.clone(), failed_hash)]; + let failed_attempt = + tinycloud_core::kv_orphans::register_candidates(&conn, &candidate, "known-failure") + .await + .unwrap(); + ImmutableWriteStore::::persist( + &store, + &owner.space, + failed, + failed_attempt[0].2, + ) + .await + .unwrap(); + tinycloud_core::kv_orphans::refund_failed(&conn, &store, &failed_attempt, "known-failure") + .await + .unwrap(); + assert_eq!(store.total_size(&owner.space).await.unwrap(), Some(0)); + let retry = + tinycloud_core::kv_orphans::register_candidates(&conn, &candidate, "retry-after-delete") + .await + .unwrap(); + assert_eq!( + retry[0].2, 1, + "confirmed deletion advances storage generation" + ); + assert_eq!( + tinycloud_core::kv_orphans::collect_candidates(&conn, &store, 1, time::Duration::ZERO,) + .await + .unwrap(), + 0, + "the stale candidate has no physical object left to remove" + ); + assert!(!store + .contains(&owner.space, &failed_hash, retry[0].2) + .await + .unwrap()); + assert_eq!(store.total_size(&owner.space).await.unwrap(), Some(0)); + conn.close().await.unwrap(); + admin + .execute(Statement::from_string( + DbBackend::Postgres, + format!("DROP SCHEMA {schema} CASCADE"), + )) + .await + .unwrap(); +} + async fn run_matrix( conn: DatabaseConnection, store: B, diff --git a/tinycloud-node-server/tests/postgres_round_trip_budget.rs b/tinycloud-node-server/tests/postgres_round_trip_budget.rs index 3a7c343a..4c99b67b 100644 --- a/tinycloud-node-server/tests/postgres_round_trip_budget.rs +++ b/tinycloud-node-server/tests/postgres_round_trip_budget.rs @@ -129,36 +129,49 @@ use tokio::{ }; /// Round trips allowed per flow; see the module docs. Each invocation's -/// transaction-scoped authorization snapshot now fetches edges, delegations, -/// abilities and revocations in one tagged query (TC-735); capability checks -/// and connection-release pings are unchanged. Measured on PostgreSQL: +/// transaction-scoped authorization snapshot fetches edges, delegations, +/// abilities and revocations in one tagged query (TC-735, #302); capability +/// checks and connection-release pings are part of each flow's count. Measured +/// on PostgreSQL before the pre-persist write path: /// -/// - sign-in 71 = peer key 0 + host `/delegate` 18 + session `/delegate` 19 -/// + account-space SQL write 34; -/// - kv put 35; -/// - kv get 16, including the read-audit commit. -/// -/// TC-1274: each epoch-writing commit (both `/delegate`s and the kv put) -/// gained one round trip, the `epoch_head` cache write, which replaces the -/// head anti-join over the space's whole history (measured on main: 75, 37, -/// 22; with the cache: 77, 38, 22). +/// - sign-in 71 = peer key 0 plus host `/delegate` 18 plus session +/// `/delegate` 19 plus account-space SQL write 34, including TC-1274's +/// `epoch_head` cache write on each epoch-writing commit; +/// - kv get 16, including its durable read-audit commit. /// /// TC-1274: a batch of 100 `kv/get` or `kv/metadata` capabilities is one -/// `current_kv` lookup per space, so it costs what a single kv get costs -/// (measured before the batching: 216 round trips and 98 `current_kv` -/// statements for the 98 distinct keys the fixture asks for). The flows also -/// check the responses against the fixture, and the test fails if a batch -/// reads `current_kv` in more than one statement. +/// `current_kv` lookup per space, so it costs what a single kv get costs. The +/// flows also check the responses against the fixture, and the test fails if a +/// batch reads `current_kv` in more than one statement. +/// +/// KV PUT is 42: 35 round trips for the put itself (31 statements and pings, +/// with #302 and the `epoch_head` cache write) plus seven around the upload: +/// the preparatory transaction's `BEGIN`, its authorization graph read, the +/// candidate-and-attempt claim CTE (which also checks that every invocation +/// space is hosted) and its `COMMIT`; the preparatory connection's release +/// ping; the joined active-fence/own-attempt lock that excludes GC after +/// upload; and the successful-commit CTE that retires the attempt and an +/// already referenced last fence. A failed last attempt persists a deleting +/// fence until physical absence is confirmed. SQLite and MySQL use the +/// equivalent guarded multi-statement paths rather than the PostgreSQL-only +/// CTEs. +/// +/// Multi-write puts (a multipart `kv/put` of several keys, one invocation): +/// the two-key rows use the same request shape, so they differ only in how +/// many distinct blobs the keys hold. Each distinct blob adds its registration +/// statement, its fence lock and its finish; every key adds its own +/// precondition read and write rows. All claims of one put share one +/// preparatory transaction. /// -/// TC-735: the one-statement authorization snapshot (#302) removes six trips -/// from sign-in and from every read invocation (kv get and both batches: -/// main alone is 77 and 22) and three from kv put (main alone: 38). -const BUDGETS: [(&str, u64); 5] = [ +/// Each value is the count measured on PostgreSQL in CI, used as a ceiling. +const BUDGETS: [(&str, u64); 7] = [ ("sign-in", 71), - ("kv put", 35), + ("kv put", 42), ("kv get", 16), ("kv batch get", 16), ("kv batch head", 16), + ("kv put 2 blobs", 48), + ("kv put 1 blob 2 keys", 45), ]; /// Full sign-in plus put plus get rounds run before the measured one. @@ -431,24 +444,28 @@ impl FrontendParser { /// route tests): the server receives the last one, the node never sees a /// reply to it, and the connection is reset. Armed for `times` connections. #[derive(Clone, Default)] -struct Cut(Arc>); +struct Cut(Arc>); /// The markers still to match, in order, and how many connections to cut. type Armed = Option<(Vec>, usize)>; impl Cut { fn arm(&self, markers: &[&[u8]], times: usize) { - *self.0.lock().unwrap() = Some((markers.iter().map(|m| m.to_vec()).collect(), times)); + *self.0.lock() = Some((markers.iter().map(|m| m.to_vec()).collect(), times)); } fn armed(&self) -> bool { - self.0.lock().unwrap().is_some() + self.0.lock().is_some() + } + + fn disarm(&self) { + *self.0.lock() = None; } /// Advance a connection's `progress` through the armed markers over /// `bytes`; whether the connection must now be cut. fn observe(&self, progress: &mut usize, bytes: &[u8]) -> bool { - let mut armed = self.0.lock().unwrap(); + let mut armed = self.0.lock(); let Some((markers, times)) = armed.as_mut() else { *progress = 0; return false; @@ -956,10 +973,88 @@ async fn round(node: &Node, account: &Account, out: &mut Vec) -> Resul read.as_bytes() == KV_VALUE, "kv get returned {read:?}, not the value just put" ); + multi_write_puts(node, &session, out).await?; batch_reads(node, &session, out).await?; Ok(()) } +/// TC-735: one invocation writing two keys is a multipart `kv/put` batch. Its +/// pre-upload registration, fence lock and finish run once per DISTINCT blob +/// (three round trips each), whatever the number of keys. Two keys holding +/// different bytes are two blobs; two keys holding the same bytes are one. +fn multipart_put(keys: &[String], values: &[&[u8]]) -> (ContentType, Vec) { + let boundary = "tc735-multi-write-boundary"; + let mut body = Vec::new(); + for (key, value) in keys.iter().zip(values) { + body.extend_from_slice( + format!( + "--{boundary}\r\nContent-Disposition: form-data; name=\"{key}\"\r\n\ + Content-Type: application/octet-stream\r\n\r\n" + ) + .as_bytes(), + ); + body.extend_from_slice(value); + body.extend_from_slice(b"\r\n"); + } + body.extend_from_slice(format!("--{boundary}--\r\n").as_bytes()); + ( + ContentType::new("multipart", "form-data").with_params(("boundary", boundary)), + body, + ) +} + +async fn multi_write_puts(node: &Node, session: &Session, out: &mut Vec) -> Result<()> { + let distinct = vec!["multi/distinct-a".to_owned(), "multi/distinct-b".to_owned()]; + let (content_type, body) = multipart_put( + &distinct, + &[b"tc735 multi-write blob a", b"tc735 multi-write blob b"], + ); + expect_ok( + "kv put 2 blobs", + node.measure( + "kv put 2 blobs", + "POST /invoke kv/put x2 (two blobs)", + node.client + .post("/invoke") + .header(Header::new( + "Authorization", + batch_invocation(session, "tinycloud.kv/put", &distinct)?, + )) + .header(content_type) + .body(body), + out, + ) + .await?, + )?; + + let same = vec!["multi/same-a".to_owned(), "multi/same-b".to_owned()]; + let (content_type, body) = multipart_put( + &same, + &[ + b"tc735 multi-write shared blob", + b"tc735 multi-write shared blob", + ], + ); + expect_ok( + "kv put 1 blob 2 keys", + node.measure( + "kv put 1 blob 2 keys", + "POST /invoke kv/put x2 (one blob)", + node.client + .post("/invoke") + .header(Header::new( + "Authorization", + batch_invocation(session, "tinycloud.kv/put", &same)?, + )) + .header(content_type) + .body(body), + out, + ) + .await?, + )?; + Ok(()) +} + /// Keys the batch flows read: `batch/000`..`batch/099` holding `v`, of /// which `DELETED_KEY` is then deleted, plus two oversized values. const BATCH_ITEMS: usize = 100; @@ -1667,11 +1762,13 @@ async fn postgres_write_routes_tell_retryable_failures_from_unknown_outcomes() { let (status, body, _) = kv_put(&node, &session, "tc1054/warm").await?; ensure!(status == Status::Ok, "warm-up put: {status} {body}"); - // KV put: no connection survives BEGIN, so nothing was written. + // KV put's preparatory BEGIN retries once in the recovering pool and + // once around the idempotent preparation (two attempts each). Exhaust + // both layers before asserting the public retryable response. node.settle().await?; - cut.arm(&[b"BEGIN"], 2); + cut.arm(&[b"BEGIN"], 4); let (status, body, retry_after) = kv_put(&node, &session, "tc1054/retryable").await?; - ensure!(!cut.armed(), "both BEGIN attempts were cut"); + ensure!(!cut.armed(), "all preparatory BEGIN attempts were cut"); ensure!( status == Status::ServiceUnavailable && body.starts_with("unavailable_retryable:"), "kv put before any write: {status} {body}" @@ -1784,6 +1881,196 @@ async fn postgres_write_routes_tell_retryable_failures_from_unknown_outcomes() { result.expect("route outcomes"); } +/// TC-735: a KV put's preparatory `BEGIN` is retried a bounded number of times +/// (two in the recovering pool, around two passes of the preparatory block), +/// and only while the `BEGIN` itself fails. Once a statement of the commit +/// transaction has run, a lost connection is an unknown outcome and nothing is +/// retried: a put that retried would answer 200. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn postgres_kv_put_begin_retry_is_bounded_and_never_follows_a_statement() { + let Some(url) = + postgres_test_url("postgres_kv_put_begin_retry_is_bounded_and_never_follows_a_statement") + else { + return; + }; + let admin = Database::connect(url.as_str()) + .await + .expect("connect to PostgreSQL"); + let schema = format!( + "tc735_begin_{}_{}", + std::process::id(), + time::OffsetDateTime::now_utc().unix_timestamp_nanos() + ); + admin + .execute(Statement::from_string( + DbBackend::Postgres, + format!("CREATE SCHEMA {schema}"), + )) + .await + .expect("create the isolated schema"); + let cut = Cut::default(); + let (node, _datadir) = start_node(&url, &schema, cut.clone(), None) + .await + .expect("start the node"); + + let result = async { + let account = Account::new(43)?; + let session = sign_in(&node, &account).await?; + let (status, body, _) = kv_put(&node, &session, "tc735/warm").await?; + ensure!(status == Status::Ok, "warm-up put: {status} {body}"); + + // Five BEGIN cuts are armed, but a put makes four attempts at most, so + // one cut is still armed after it answers. + node.settle().await?; + cut.arm(&[b"BEGIN"], 5); + let (status, body, _) = kv_put(&node, &session, "tc735/bounded").await?; + ensure!( + cut.armed(), + "a fifth BEGIN was attempted: the retry is not bounded" + ); + ensure!( + status == Status::ServiceUnavailable && body.starts_with("unavailable_retryable:"), + "kv put whose BEGIN never succeeded: {status} {body}" + ); + cut.disarm(); + + // Cut a statement inside the commit transaction (the fence lock that + // follows the upload): the put is not retried. + node.settle().await?; + cut.arm(&[b"FOR UPDATE OF fence"], 1); + let (status, body, retry_after) = kv_put(&node, &session, "tc735/after-statement").await?; + ensure!(!cut.armed(), "the fence lock statement was cut"); + ensure!( + status == Status::InternalServerError && body.starts_with("outcome_unknown:"), + "kv put after a statement of its transaction ran: {status} {body}" + ); + ensure!( + retry_after.is_none(), + "no Retry-After on outcome_unknown: {retry_after:?}" + ); + Ok(()) + } + .await; + node.pool().close().await; + admin + .execute(Statement::from_string( + DbBackend::Postgres, + format!("DROP SCHEMA {schema} CASCADE"), + )) + .await + .expect("drop the isolated schema"); + result.expect("begin retry bounds"); +} + +/// TC-735: a two-blob put whose later-registered blob has a pending deletion is +/// refused with 503 and claims nothing. The earlier blob keeps no claim, fence +/// or ledger row, so the claims share the preparatory transaction. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn postgres_two_blob_put_with_a_pending_deletion_claims_nothing() { + let Some(url) = + postgres_test_url("postgres_two_blob_put_with_a_pending_deletion_claims_nothing") + else { + return; + }; + let admin = Database::connect(url.as_str()) + .await + .expect("connect to PostgreSQL"); + let schema = format!( + "tc735_claims_{}_{}", + std::process::id(), + time::OffsetDateTime::now_utc().unix_timestamp_nanos() + ); + admin + .execute(Statement::from_string( + DbBackend::Postgres, + format!("CREATE SCHEMA {schema}"), + )) + .await + .expect("create the isolated schema"); + let (node, _datadir) = start_node(&url, &schema, Cut::default(), None) + .await + .expect("start the node"); + + let result = async { + let account = Account::new(44)?; + let session = sign_in(&node, &account).await?; + let (status, body, _) = kv_put(&node, &session, "tc735/warm").await?; + ensure!(status == Status::Ok, "warm-up put: {status} {body}"); + + let (one, two): (&[u8], &[u8]) = (b"route two-blob first", b"route two-blob second"); + let (early, late) = if tinycloud_core::hash::hash(one) < tinycloud_core::hash::hash(two) { + (one, two) + } else { + (two, one) + }; + let space = account.space.to_string(); + let early_hash = Vec::::from(tinycloud_core::hash::hash(early)); + let late_hash = Vec::::from(tinycloud_core::hash::hash(late)); + let pool = node.pool(); + tinycloud_core::sea_orm::sqlx::query( + "INSERT INTO kv_blob_generation (space, value, generation) VALUES ($1, $2, 0)", + ) + .bind(&space) + .bind(&late_hash) + .execute(pool) + .await?; + tinycloud_core::sea_orm::sqlx::query( + "INSERT INTO kv_orphan_candidate + (space, value, generation, created_at, deleting_at, deletion_bytes) + VALUES ($1, $2, 0, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, 1)", + ) + .bind(&space) + .bind(&late_hash) + .execute(pool) + .await?; + + let keys = vec![ + "tc735/pending-early".to_owned(), + "tc735/pending-late".to_owned(), + ]; + let (content_type, multipart) = multipart_put(&keys, &[early, late]); + let (status, body, _) = dispatch( + node.client + .post("/invoke") + .header(Header::new( + "Authorization", + batch_invocation(&session, "tinycloud.kv/put", &keys)?, + )) + .header(content_type) + .body(multipart), + ) + .await; + ensure!( + status == Status::ServiceUnavailable && body.starts_with("unavailable_retryable:"), + "two-blob put with a pending deletion: {status} {body}" + ); + for table in [ + "kv_orphan_attempt", + "kv_orphan_candidate", + "kv_blob_generation", + ] { + let rows: i64 = tinycloud_core::sea_orm::sqlx::query_scalar(&format!( + "SELECT count(*) FROM {table} WHERE value = $1" + )) + .bind(&early_hash) + .fetch_one(pool) + .await?; + ensure!(rows == 0, "{table} kept {rows} row(s) for the earlier blob"); + } + Ok(()) + } + .await; + node.pool().close().await; + admin + .execute(Statement::from_string( + DbBackend::Postgres, + format!("DROP SCHEMA {schema} CASCADE"), + )) + .await + .expect("drop the isolated schema"); + result.expect("two-blob put with a pending deletion"); +} + /// The `postgres` object of `/info`: exactly the two counters. async fn postgres_info(node: &Node) -> Result<(u64, u64)> { let (status, body, _) = dispatch(node.client.get("/info")).await; diff --git a/tinycloud-node-server/tests/s3_block_store.rs b/tinycloud-node-server/tests/s3_block_store.rs index 7860b194..7243b9b0 100644 --- a/tinycloud-node-server/tests/s3_block_store.rs +++ b/tinycloud-node-server/tests/s3_block_store.rs @@ -50,7 +50,7 @@ fn value(n: usize) -> Vec { async fn put(store: &S3BlockStore, space: &SpaceId, data: &[u8]) -> Result { let mut staged = HashBuffer::new(Vec::new()); staged.write_all(data).await.unwrap(); - ImmutableWriteStore::::persist(store, space, staged).await + ImmutableWriteStore::::persist(store, space, staged, 0).await } #[tokio::test] @@ -65,14 +65,17 @@ async fn remove_deletes_the_block_and_releases_its_size() -> Result<()> { ); ensure!( - store.remove(&space, &removed).await? == Some(()), + store.remove(&space, &removed, 0).await? == Some(()), "remove of a stored block" ); ensure!( - !store.contains(&space, &removed).await?, + !store.contains(&space, &removed, 0).await?, "object still exists" ); - ensure!(store.contains(&space, &kept).await?, "other block removed"); + ensure!( + store.contains(&space, &kept, 0).await?, + "other block removed" + ); ensure!( store.total_size(&space).await? == Some(17), "size drops by the removed block's length" @@ -80,13 +83,13 @@ async fn remove_deletes_the_block_and_releases_its_size() -> Result<()> { // Removing again, or removing a block never stored, is a no-op. ensure!( - store.remove(&space, &removed).await?.is_none(), + store.remove(&space, &removed, 0).await?.is_none(), "second remove" ); let mut unstored = HashBuffer::new(Vec::new()); unstored.write_all(b"never stored").await?; ensure!( - store.remove(&space, &unstored.hash()).await?.is_none(), + store.remove(&space, &unstored.hash(), 0).await?.is_none(), "absent block" ); ensure!(