[kriszyp]

⊙ Problem

Main's Integration Tests shard 6 is red since chore(deps): Update rocksdb-js to 2.12.0 (#3193): QA-822 fails at invariants 1 and 4.

Not a defect. rocksdb-js 2.12.0 ships Let transaction-log retention purge the flushed-position and current segments (rocksdb-js#911): a purge may now delete every fully flushed file. QA-822 required the old #799 floor. The derived-index runtime also reads fileCount === 0 as "never written", which an emptied log now reports too, so an index could skip commits.

🤔 Is the requested fix the right one?

The task asked to keep the referenced file and not weaken the test; under #911 that file is gone by design. The owner ruled on 2026-10-18 that Harper pins #911. The invariants now assert the witness itself and fail on 2.11.0, where the floor file survived.

💡 Solution

QA-822 asserts the txn.state witness: never the sentinel, writer above it when no files remain, never moves back, exact bytes on disk while down, every fully flushed file gone, files holding newer rows kept.

The derived-index cursor gains floors: { <log>: <sequence> } for a log with no completed transaction at rebuild. Such a log is read from its beginning only while every file from its floor on is retained, checked before every anchor and catch-up.

🤔 Why not the one-line predicate #911 suggested?

Measured here: with fileCount === 0 ? currentSequenceNumber <= 1 : oldestSequenceNumber === 1 alone, an index over a database with one emptied peer log goes permanently unavailable. The rebuild rejected the emptied log, and nothing recorded whether it was emptied before or after the boundary. The floor is that record; the predicate is its floor-1 case.

🤔 One rebuild after upgrade for indexes over an emptied log

A pre-existing cursor has no floor, so its first resume rejects it and rebuilds once; later resumes are quiet. Correct for integrity, but a large HNSW rebuild is long. Release-note item, with #911's downgrade warning (remove txn.state before downgrading below 2.12.0 once a store's last file was purged).

🤔 Anchored idle logs still rebuild once per purge cycle

Floors cover logs with no committed transaction at the boundary. A log anchored by timestamp whose last transaction's file is later emptied still fails exactStart on the next acquisition and rebuilds once; the new cursor then floors it. Fixing that needs a per-anchor seal (writer position at a caught-up point, accepted when equal to lastFlushedPosition). Follow-up.

⚖️ Alternatives

Rejected options

• Revert harper#3193 to 2.11.0: green with no code, but drops 2.12.0's backup verifyTables and prebuild fixes, blocks the harper-pro consumers Extract Fastify routes into a standalone component #911 serves, and the test needs the same rewrite next bump.
• Keep the old predicate, file a bug: leaves the silent-staleness defect the dependency owner flagged.
• Header-only witness segment in rocksdb-js: declined in Extract Fastify routes into a standalone component #911 and does not answer the resume question.
• Store the floor in cursor.coverage: optional, published after catch-up, dropped by the runtime's clone.
• format: 2: an older full-text decoder then fails closed instead of rebuilding once; an older runtime ignores floors either way.

🔧 Changes

• integrationTests/database/qa822-purge-log-invariants.test.ts: invariant 1 and invariant 4 restated; invariant 4 samples txn.state (LE uint32 offset, uint32 sequence) and the file list in the shutdown callback, while the process is down, and requires the reopened writer to sit where the purge retired it; arm 1 purges twice through one helper, keeping rows newer than the first cutoff (the keep half) and then emptying the store, asserted, before the same-process rows.
• resources/derivedIndexRuntime.ts: DerivedIndexCursor.floors; #captureBoundary records a no-tail log's oldest retained file (or the writer's sequence with no files) when past 1; #retainsFrom(log, floor) is fileCount === 0 ? currentSequenceNumber <= floor : oldestSequenceNumber <= floor; #validateLogSet covers floor-only logs and judges unanchored ones by their floor; #anchor re-proves the floor before the first exact anchor; #checkUnanchoredLogs runs inside #reconcileDurableCursor, the one path every catch-up publication takes, against the newest accepted vector; isValidCursor, cloneCursor and sameCursor treat floors as identity. Look hardest at the three proof sites: a missed one is a silently stale index.
• resources/indexes/fullTextDerivedIndex.ts: the payload codec validates, sorts and accepts floors; cloneCursor and sameCursor keep and compare them, with no extra work for floors when neither cursor has any.
• Cost: unanchored logs (emptied or never-written) are the only ones probed with getStats(), once per caught-up pass on the background runner; anchored logs pay nothing new.
• resources/indexes/DESIGN.md: cursor shape; retention rule, proof sites, rollback behaviour.

✅ Verification

Planning review Framing-Verdict: chosen-approach-sound.

• QA-822 integration test: 9/9 on this build (2/9 before, CI's fingerprint); its data arm purges twice, keeping rows newer than the cutoff and then emptying the store.
• npm run test:unit:resources on the final head: 4304 passing, 0 failing, 54 pending.
• Fails on base: the first-commit native-backend cases all fail against a separately built origin/main dist.
• npm run lint:required and Prettier clean.

New tests

• derivedIndexRuntimeNativeBackend.test.js (its fake store reports per-log retention stats): legacy cursor over an emptied log condemns once and anchors by floor; intact floor resumes without rebuild (three layouts); crossed floor condemns (two layouts); floor-only log gone condemns; floor re-proven before the first exact anchor, and before catch-up under sustained local traffic with no remote transaction to anchor; first commit retires the floor; a floor the accepted batch already replaced is not re-proven against the older durable cursor; uncommitted entries floor at the oldest file; tampered floors read as unoffered.
• derivedIndexRuntimeRocks.test.js: the lifecycle on a real RocksDB store with a peer log emptied by purgeLogs under 2.12.0, including a pre-floors cursor.
• fullTextDerivedIndex.test.js: floors survive deliver, flush and coverage publication; sorted round-trip; 0, 1.5, 2^32, an array and an empty name rejected.

Refs #3193

_{— Claude Fable 5.1}

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ub3YgzNx9s7KVfKJT4Kx7t

Release target: main — adopts the rocksdb-js 2.12.0 retention contract that only main pins (#3193); the release lines stay on 2.11.x
Related PRs: #2743 independent, #2338 independent
Complexity: complicated

Deferred findings

• b39ba5bde0c9a4a2 major reliability — resources/derivedIndexRuntime.ts — anchored log emptied by 2.12 retention fails exactStart on reacquire and forces a full rebuild — NOT FILED
• f70e81c290b13b80 minor correctness — resources/derivedIndexRuntime.ts — installCursor ignores a synchronous rebuild started by validateLogSet and publishes ready over rebuilding — NOT FILED

Origin — the dispatch brief this PR was written from

Investigate transaction-log purge retention-floor failures after rocksdb-js 2.12.0

Investigate and fix the Harper main Integration Tests regression associated with harper#3193. At integrationTests/database/qa822-purge-log-invariants.test.ts:573 and :736, transaction-log purge deletes every log file while lastFlushedPosition still names seq 2 / offset 7213886, violating the retained-floor and clean-reopen invariants. Reproduce the failure, trace the behavior across the Harper purge and rocksdb-js dependency boundary, then implement the smallest correct fix. If the fix is not obvious, recommend reverting harper#3193 rather than making a speculative change.

Acceptance

The retention floor's referenced log remains available after purge and after a clean write-free reopen; the QA-822 invariants at :573 and :736 pass on affected Node.js v22/v24/v26 Integration legs. Add focused regression coverage if needed. Report the established cause and why the chosen fix is correct. Keep the change scoped to Harper/rocksdb-js transaction-log retention; do not alter the test to weaken the invariant.

_{Dispatch: task main-red-harper-3193-retention-floor-20261013 · queued by ci-health-monitor · ran by claude/fable/high · worker kzyp-xps-1}

_{Review-Coverage: authored=claude; ran=gemini,muse,codex; adjudicated=domain; declined=cursor-grok,cursor-composer,cursor-kimi,cursor-muse; rounds=10; full=6 @ 9d84a36}

_{Review-Attention: deep ~30m (critical: derivedIndexRuntime.ts, fullTextDerivedIndex.ts; decisions: floors-in-cursor, floor-proof-sites, floors-as-identity, scope-unanchored-only, txn-state-byte-layout, do-less-alternative) @ 9d84a36}

[gemini-code-assist]

Code Review

This pull request updates the transaction log purging and indexing mechanism to align with rocksdb-js#911 (v2.12.0), where fully flushed logs can be completely emptied of files. It introduces a 'floors' property to the DerivedIndexCursor to anchor logs that do not have a completed transaction when a rebuild boundary is captured, ensuring log sequences are tracked correctly even when no files remain on disk. The changes span the runtime implementation, full-text index serialization/deserialization, design documentation, and comprehensive integration and unit tests. There are no review comments provided, so I have no feedback to evaluate.