Skip to content

[G2] Identity and fingerprint compatibility, cursor migration, and state rebuild from the relayhistory store #561

Description

@willwashburn

Part of #553 (group 2 — depends on burn #555 identity vectors and burn #557 adapter; parallel with #558/#559).

Problem

Switching sources must not double-count or lose history in an existing burn.sqlite. Three mechanisms are in play:

  1. Row identity — ledger/fingerprint.rs: turn_id_fingerprint = sha256("source|sessionId|messageId")[..16] (:39), compaction = source|sessionId|ts (:50), relationship = source|sessionId|relType|relatedSessionId|agentId|parentToolUseId (:60), tool_result_event = source|sessionId|toolUseId|eventIndex (:73), user_turn = source|sessionId|userUuid (:84). If relayhistory's message_id, event_index, or synthesized user_uuid differ from burn's, existing rows are duplicated.
  2. Content identity — turn_content_fingerprint (:97-115) over ts|model|input+output|cacheRead|cache5m+cache1h|args_hash[..4]: catches the same turn under a different message id — but only if ts (ms precision, source formatting) and args_hash (stable-stringify of tool_use.input) are byte-identical.
  3. Progress state — archive_state.upstream_cursors_json (ingest/cursors.rs, per-file {inode, offsetBytes, mtimeMs, …} tagged union; the Codex cursor serializes the whole parser state machine) and source_fingerprint. The relayhistory backend uses a single named consumer watermark (relayhistory Add cross-harness ghost user-installed surface detector (#166) #179) instead.

source values differ too: burn claude-code|codex|opencode vs relayhistory claude|codex|opencode|cursor|grok|relay.

Scope

  1. Source mapping table (one place, ingest/backend/relayhistory/adapter.rs): claude → claude-code, others identity; document that burn's SourceKind strings are the ledger's and never change.
  2. Identity parity tests against the [G1] Characterization lock: ledger golden snapshots for the full fixture corpus and a burn state parity ledger differ #555 vectors: for each corpus turn, the adapter's (message_id, ts, args_hash) and therefore both fingerprints equal the builtin values. Where relayhistory's fact differs by design (e.g. it stores ts as ts_ms integer while burn kept the ISO string), the adapter converts back to burn's canonical form; add a vector test for ISO ↔ ms round-trip with sub-ms truncation behaviour.
  3. user_uuid / event_index reproduction: Codex/OpenCode synthesized user_uuid (codex.rs:746-769, opencode.rs:965-973) and Claude event_index must be regenerated identically from relayhistory's ordered evidence (event_index comes from relayhistory Make burn compare take models as a required positional (#159) #171; assert equality on user-turn-blocks for all three harnesses).
  4. Cursor migration: on first open with the relayhistory backend, if upstream_cursors_json is present and the burn consumer cursor is absent: (a) run a one-time reconciliation pass that pulls the full store and relies on upsert-by-fingerprint (no duplicates by construction if 2–3 hold), (b) write the consumer watermark = head_revision, (c) keep the old cursors blob under archive_state.legacy_cursors_json for rollback, (d) record archive_state.source_backend = "relayhistory". Switching back to builtin restores the legacy blob. Add burn state status output for the active backend and watermark.
  5. burn state rebuild (query_verbs/state.rs, ledger.rs:381-452 rebuild_derivable): under the relayhistory backend, rebuild = drop derivable tables, replay stamp relationships, reset the consumer watermark to 0, re-pull. Document that the harness logs are no longer needed for a rebuild — the relayhistory DB is.
  6. Schema version: bump SCHEMA_VERSION (ledger/schema.rs:71, currently 7) for the new archive_state columns with the documented migration.
  7. Real-ledger test: take a builtin-built ledger from the corpus, switch backend, run one pass, burn state parity --against the pre-switch copy → zero differences; switch back → zero differences.

Acceptance

  • Identity vectors match for 100% of corpus turns, tool-result events, user turns, compactions, relationships.
  • Switching backends on a populated ledger produces no new turns rows (assert row count and fingerprint set equality).
  • burn state rebuild under relayhistory reproduces the ledger snapshot for every corpus case.
  • Rollback to builtin restores cursors and continues incrementally (no full re-parse — assert cursor offsets preserved).

Out of scope

Deleting the builtin backend (#562).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions