fix(cloud): incrementally replay imported histories

[Neonforge98]

Problem

Cloud replay network writes were already cursor/segment based, but each dirty background pass for an imported history still loaded, normalized, and hashed the complete local transcript before slicing the network delta. Long-running Claude Code, Codex App, and Cursor IDE sessions therefore paid O(total history) CPU, RAM, and I/O for every new turn, which made background upload appear stalled on large histories.

Solution

Add provider-native, bounded replay checkpoints for Claude Code, Codex App, and Cursor IDE. The first upload remains a complete authoritative anchor. Later uploads validate ordered provider turn IDs plus a compact Merkle frontier, then reread only the previously mutable final turn and newly appended turns while preserving the existing epoch/segment protocol.

Incremental preparation is capped at 50 turns and 16 segments. Invalid checkpoints, reordered or duplicate turn IDs, shrinks, oversized windows, unsupported providers, reader errors, and OCC conflicts all fall back to a complete authoritative reread; a bounded suffix is never used as a rewrite body. The checkpoint is optional and local-only, so existing cursors keep their current behavior. No server schema migration or public cloud wire migration is included.

Potential risks

A historical provider body mutation that preserves every provider-native turn ID outside the reread overlap cannot be detected solely from the compact checkpoint until another invariant fails or a future full re-anchor occurs. Codex catalogs beyond the bounded 4,096-turn cache intentionally lose incremental eligibility as their retained ID prefix shifts and use the full path. Large deltas beyond the turn or segment caps also trade performance for the existing authoritative path.

Rollback is a client revert. Existing persisted cursors remain readable because importedReplay is optional; reverting only removes the bounded preparation optimization. The server continues to store the same epochs, segments, tails, and total counts.

Architecture audit

All 10 architecture-audit layers were covered. Compilation and clippy are green; the production chain was traced from the Cloud sync pass through the source registry, Tauri handlers, and provider readers; no dead parallel entry point was added. Provider-native turnId is explicitly distinguished from normalized event IDs. Unsupported/default cases use the full authoritative loader. Provider parsing remains inside orgtrack-core, the two IPC commands are registered through the canonical handler list, and all three capable providers expose the same ordered-ID plus bounded-window pair. Real serialized segment payloads were exercised against managed Cloud; there is no new initialization path or asymmetric multi-field resolver.

Performance audit

Area	Verdict	Evidence	Change or reason kept	Verification
Background work	keep	Existing dirty-pass scheduler remains the sole owner; no timer, worker, subscription, or retry cadence was added	Replace complete imported-history preparation with bounded provider reads	Visible, hidden, and repeated station-switch measurements
Memory	fix	50-turn IPC cap, 16-segment append cap, Merkle frontier capped at 54 entries, Codex catalogs capped at 4,096 turns across 8 sessions	Retain commitments and offsets, not complete normalized transcripts	Bound/fallback unit tests plus true-device append
Scope/isolation	keep	Existing cursor key remains orgId + sessionId; source readers use the isolated instance data home	No new global identity cache	Primary and Instance 2 ran concurrently on separate data homes and ports
Rendering/hot path	keep	No TSX, React subscription, or rendered-state change	Work stays in the existing background pass and blocking readers stay behind spawn_blocking	No UI regression surface; five My Station / Agent's Station cycles

Performance verdict: pass

Verification

• pnpm typecheck — passed.
• Pre-commit lint-staged (ESLint fix + Prettier), staged TypeScript check, and scoped cargo clippy for org2 + orgtrack_core — passed.
• pnpm vitest run src/features/Org2Cloud/org2CloudSessionSync.shrink.test.ts src/features/Org2Cloud/org2CloudSyncEngine.sessions.test.ts src/api/tauri/externalHistory/imported/tests/sources.test.ts — 64/64 passed.
• TZ=UTC pnpm test — 931/931 test files and 7,852/7,852 tests passed. The command remains non-zero because develop already leaves one unhandled undici WebSocket Event-type exception originating from useCloudOrgSyncStatus.test.ts.
• cargo test -p orgtrack_core --lib --no-fail-fast — 537 passed, 8 ignored, 0 failed.
• cargo clippy -p orgtrack_core --all-targets -- -D warnings — passed.
• cargo check -p org2 — passed; only the existing future-incompatibility notice for block 0.1.6 was emitted.
• pnpm tauri:build:fast -- --instance 2 — built the signed-in isolated macOS test app successfully.
• True-device managed-Cloud test: established full anchors, appended one new turn to synthetic Claude Code, Codex App, and Cursor IDE histories, and verified the server kept epoch 1 while advancing to two segments. Final server counts were Claude Code 6 events, Cursor IDE 6 events, and Codex App 12 events. No target-session full fallback, rewrite, OCC conflict, or push failure was logged.
• Lifecycle measurements with the primary app and Instance 2 open: visible idle CPU stayed at 0.0–0.1%; after five station-switch cycles RSS stabilized at 242,448–242,496 KiB; hidden idle CPU stayed at 0.0–0.2% and RSS at 246,096–246,208 KiB. One 2.8% sample coincided with the configured 60-second source scan and returned to idle.
• git diff --check — passed.
• cargo fmt --all --check was not used as a gate because current develop has unrelated formatting drift in agent-core. Every changed Rust file was formatted directly with rustfmt edition 2021.

No screenshot is attached because this PR has no user-visible UI change. Synthetic Cloud rows were soft-deleted and the isolated CC, Codex, and Cursor fixtures and caches were removed after verification.

[Neonforge98]

Review + follow-up fixes landed in 2b9a608 (fix(cloud): harden incremental imported replay checkpoints).

Findings fixed:

1. Infinite synchronous loop on a zero-frozen frontier (critical). trimMerkleFrontier([]) spun forever ([].at(-1) == null holds and pop() is a no-op on an empty array), freezing the renderer the first time a merkle plan carried zero frozen events (e.g. a session whose only turn is still mutable). Caught by the new property test's total=0 case; fixed with a length guard.
2. Flat→merkle migration forced an O(total) epoch rewrite. Every pre-checkpoint flat-v1 cursor on a capable provider hit the cursorHashMode === frozenHashMode gate and re-uploaded its entire intact history — contradicting "existing cursors keep their current behavior", and heaviest on exactly the large histories this PR targets. Intactness now accepts a commitment match in either hash mode (both commit to the same per-event hash vector), so legacy cursors migrate through the ordinary delta append and adopt the checkpoint there.
3. Transient probe failure escalated to full re-uploads. A read hiccup in the turn-id probe downgraded the plan to flat-v1, mismatch-rewrote once, then rewrote again when the probe recovered. With the dual-mode check both directions ride the append path; an unchanged session keeps its still-valid checkpoint on a failed probe.
4. hashStringList used a \n join over provider-native turn ids — element-boundary collisions possible; now length-delimited via stableStringify.
5. The shrink two-pass dance forced plan() (full transcript hashing) before its skip-return; the observation count is now computed as baseEventCount + events.length.
6. Codex reader returned Ok(empty) for an unparseable cloud turn id (indistinguishable from a legitimately empty turn); it now errors like the Claude reader, and the frontend maps reader errors to the authoritative full path.

Merkle helpers are extracted to org2CloudMerkleFrontier.ts with direct unit tests (build/append equivalence across split points, JSON-persistence round-trip stability of the frontier's hole/null forms, structural validation) plus engine tests for flat-cursor migration, transient probe failure, and a checkpoint persistence round trip.

Verification: Org2Cloud suite 113 files / 1085 tests green; typecheck green; cargo test green for the touched modules (the 8 copilot/kimi failures and one clippy unused-import are pre-existing Windows-environment issues, present on the unmodified branch). Dual-instance real-machine run (Windows, managed Cloud, synthetic Claude history in org 0aefaa1f): legacy build anchored a flat cursor (epoch 1, no checkpoint); after swapping to this branch, an appended turn rode a delta append — epoch stayed 1, frozenSeq 1→2, checkpoint materialized with a structurally correct frontier; a further live append took the bounded path (zero fallback warnings, reloadTurnId advanced); an injected local shrink was refused by the two-strike guard with the cloud rows untouched; the checkpoint survived two cold restarts; and the receiving account rendered the incrementally-pushed final round verbatim from the Team Sessions row.

[Neonforge98]

Follow-up in 88aadae: an escape-class sweep (boundary / cross-version / degraded / delayed-manifestation lenses) over this PR's surface found one more real hazard. createZodJsonStorage answers a failed whole-store parse with the initial value, so ONE malformed push-cursor entry — disk corruption, or a future checkpoint version: 2 rolled back onto this build (the zod literal(1) rejects it) — reset EVERY push cursor at load, and a full reset re-anchors every pushed session through an epoch rewrite on its next pass: an epoch-churn storm in the #608 shape whose individual rewrites all look legitimate. The checkpoint made this failure surface strictly larger, and the PR's rollback claim ("existing persisted cursors remain readable") did not hold in the future-to-this-build direction. The cursors record now parses per entry and sheds only invalid ones (one cursor lost = one session re-anchors, the designed recovery), with unit coverage for the rollback, oversized-frontier, and garbage-entry shapes. The same whole-store-reset shape exists pre-PR in the accessSettings store (worse blast radius: silent effective-off retracts); filed separately rather than scope-creeping this PR.