Repository navigation
Conversation
Replace DatabaseArtifactRepository::load/load_sized with load_meta (metadata columns only) and read_chunks (4 MiB substr slices of payload then delta_payload, guarded by revision + checkpoint hash, hashes verified while streaming) into an ArtifactSink. SQL and DuckDB hydration stream into staged cache files (published only after the read verified); export and the Forbid-growth size check read metadata only. Drops the bulk-read scope. Adds docs/runbooks/tc-822-artifact-payload-storage.md (Postgres STORAGE EXTERNAL operator step).
The slice statement was not on the read-only call allowlist, so a lost connection mid-slice was not retried (blackhole_mid_hydration CI failure).
… at three detections
…ice per part (TC-822)
- CacheFileSink::commit removes every staged temp when a rename fails - sweep stale <file>.tmp.<pid>.<n> staging files before a SQL/DuckDB hydration (live in-process staging files are registered and never swept) - tc1274 lock-wait / hydration-bound tests no longer time a cold hydration - drop the dead SqlSizes seeded flag - read_chunks doc: SQLite memory bound is one blob, CPU O(n^2/chunk) - slice-size gate also checks the part's own size; read_artifact resets the sink on Ok(None); MemorySink reserves the part size; exhausted ArtifactChanged maps to SqlError::Unavailable (DuckDbError has no such variant, stays Internal)
|
APPROVE: see TC-822 (reviewed exact head 83e3beb…; scope 5ccc305..83e3beb) Note: the literal range 5ccc305..83e3beb also contains the merged main (#370/#374/#375/#380); the follow-up's own change is the single commit 83e3beb (5 files, +266/-45), which is what I reviewed. No data-loss or corruption path found. (a) New tests fail without their fixes (built once,
(b) Sweep vs live hydration: safe in-process.
(c) Non-blocking nits
CI green; reviewed statically plus the targeted test runs above. |
…n (TC-822) Details: TC-1491
…822) Details: TC-1491
Stacked on #376 (base is
tc1491/p-bounded-reads, not main, because #376 has not merged yet; retarget to main once it does). No migrations, no schema change.Fixes from the #376 review:
CacheFileSink::commitremoves all staged temps when a rename fails (test: non-empty directory at either destination).<file>.tmp.<pid>.<n>staging files beside the SQL and DuckDB cache files are swept before a hydration begins. Only that exact name pattern; files this process still has in flight are tracked and never swept (also covers DuckDBimport_db). Tests: orphans removed; live, finished-but-uncommitted, malformed and other-database names kept.tc1274_a_held_operation_lock...warms the databases before the 200 ms bounds apply, so it asserts the lock wait, not disk speed;tc1274_a_hydration_past_its_own_bound_is_retiredrestores default bounds for the fresh hydration.SqlSizesseeded flag.read_chunksdoc corrected for SQLite (one blob transiently; CPU O(n^2/chunk)).Ok(None);MemorySinkreserves the part size via a newsize_hint; exhaustedArtifactChangedisSqlError::Unavailable(nothing written, retry safe).DuckDbErrorhas no retryable variant and adding one touches every route mapping, so DuckDB staysInternal(commented).Also removed two unused imports in
duckdb_diagnostics_policy.rs(clippy with the duckdb feature).Tests: full
tinycloud-coresuite with--features duckdband node-server lib pass; sql::service / database_artifacts / duckdb lib tests run 3x; clippy -D warnings clean with and without duckdb.