Skip to content

F6: fix two replay bugs — genesis-index users FK, created_tx rewrite on edit - #63

Merged
lucca65 merged 3 commits into
masterfrom
fix/replay-bugs
Aug 8, 2026
Merged

F6: fix two replay bugs — genesis-index users FK, created_tx rewrite on edit#63
lucca65 merged 3 commits into
masterfrom
fix/replay-bugs

Conversation

@lucca65

@lucca65 lucca65 commented Aug 5, 2026

Copy link
Copy Markdown
Member

F6 — Two pre-existing replay bugs: both confirmed, both fixed

From the follow-up handoff — two second-hand reports from the escrow indexer work. Both reproduce; evidence and fixes below. Two commits: e982794 (bug 1), b120dd4 (bug 2). yarn format clean.

Bug 1 — community creator missing from users on a fresh genesis index: CONFIRMED

Reproduction (before): fresh schema-only DB, index from block 1. Local chain's first real action is cambiatus.cm::create at block 104 (creator alice):

Cambiatus >>> Create Community 104
Something went wrong while creating community
  ... code: '23503', constraint: 'network_account_id_fkey',
  detail: 'Key (account_id)=(alice) is not present in table "users".'   <- swallowed by .catch(logError)
Cambiatus >>> New Netlink 104
An error has occured. error is: ... "network_community_id_fkey"
  detail: 'Key (community_id)=(0,TST) is not present in table "communities".'
Process will exit now.                                                   <- unhandledRejection -> logExit

Result: 0 users, 0 communities, 0 network rows — the whole createCommunity transaction rolled back (FK failure swallowed), and the very next netlink crashed the process (crash-loop under pm2). Prod never sees this because backend sign-ups pre-populate users; a genesis index of a fresh DB has no such seeding.

Fix: createCommunity now upserts the creator into users (INSERT ... ON CONFLICT DO NOTHING, same pattern as netlink) inside the same transaction, before the network/network_roles inserts.

Acceptance (after): fresh DB from block 1 — 0 exit events; 6 users, 5 communities, 19 network, 20 network_roles; alice present with created_block=104; all 5 communities indexed.

Bug 2 — upsertObjective/upsertAction rewrite created_tx on edit: CONFIRMED

Reproduction (before): created objective id=2 (tx f0332fb6…, block 32590) and action id=3 (tx bbebf474…, block 32850), then edited both (block 32950). After indexing, both rows carried the EDIT's created_block/created_tx. Then a safe rewind (UPDATE _index_state SET block_number=32589 … WHERE id=1 + targeted _processed_actions delete, per scripts/reindex-runbook.md) and replay:

Cambiatus >>> Upsert Objective 32590
Could not resolve chain objective id, falling back to serial
  Error: resolveCreatedObjectiveId: all 2 chain objectives for community 0,TST are already in the DB
Something went wrong while creating objective
  error: duplicate key value violates unique constraint "community_objectives_pkey"
An error has occured. error is: error: current transaction is aborted, commands ignored
Process will exit now.

Exactly the reported chain: edit destroyed created_tx → create guard count({created_tx})=0 on replay → create re-executes → chain-id resolution finds nothing unknown → serial fallback → pkey violation → aborted block tx → process exit → crash-loop.

On the migration comment (a decision, not silent compliance): backend migration 20260702120000_add_idempotency_backstops.exs says "the event-source update path rewrites created_tx on every edit…" — but only to justify not adding a unique index; it simultaneously relies on "the created_tx create-path guards", which the rewrite defeats. It documents current (buggy) behavior, not a design goal. The fix is therefore to stop the rewrite; the no-unique-index decision still stands on its own (two creates in one tx would legitimately share created_tx). The backend comment itself may want a follow-up touch — flagged, not changed here.

Fix: in upsertObjective and upsertAction, created_block/created_tx/created_at/created_eos_account moved out of the shared data into the create path only; update paths write just mutable fields.

Acceptance (after):

  1. Provenance restored on the two vandalized rows (mirrors the prod remediation note below).
  2. New create + on-chain edit under fixed code: mutable fields updated, created_tx/created_block unchanged.
  3. Full rewind + replay: 0 crashes, 0 serial fallbacks, 0 duplicate keys — objectives 3 rows, actions 4 rows (no duplicates), all created_tx values intact, 8 ledger rows re-claimed.

Same-class observations (report only, not fixed)

  • Prod rows already edited pre-fix keep a destroyed created_tx — their create guard is still broken. The _processed_actions ledger protects them in normal operation, but any ledger-cleared replay over their create blocks re-triggers the pkey crash-loop. Consider a one-time remediation restoring objectives/actions.created_tx (+block/at) from chain history.
  • upsertRole also rewrites created_tx on every edit — same class, but benign: its guard keys on the (name, community_id) natural key.
  • Most updaters still swallow errors (.catch(logError)) — the known "claimed ledger row without applied writes" window from the runbook. Pre-existing, documented, untouched.
  • Local-chain-only cosmetic: rewinding with block_hash='' skips the handler's seek so the reader re-walks from block 1 (ledger makes it no-ops), and the non-monotonic cambiatus.tk history can leave _index_state at a stale low block after catch-up.

lucca65 and others added 3 commits August 5, 2026 14:31
The creator's network row FK-references users(account), but createCommunity
never created that row — prod only worked because backend sign-ups had
pre-populated users. On a from-genesis index of a fresh DB the network
insert violated network_account_id_fkey, the swallowed rollback dropped
the whole community, and the next netlink crashed the process
(network_community_id_fkey -> unhandledRejection -> exit -> crash-loop).

Mirror netlink's idempotent INSERT ... ON CONFLICT DO NOTHING inside the
same transaction, before the network/role inserts.
…n edit

The upsertObjective/upsertAction update paths carried created_block /
created_tx / created_at / created_eos_account into the save() UPDATE, so
every edit rewrote the row's create provenance with the edit's values.
That destroyed the create-path replay guard (count({created_tx})): after
any create+edit, replaying the range no longer recognized the create and
re-executed it — resolveCreated*Id found no unknown chain id, fell back
to the serial, and the insert hit community_objectives_pkey, aborting the
block transaction and crash-looping the indexer.

Keep the created_* fields create-only; update paths now touch only
mutable fields. The backend 20260702120000 migration comment cited the
rewrite only as the reason no unique index on created_tx was added — it
documents the old behavior, it is not a rationale for keeping it.

Verified end-to-end on the local chain: create+edit leaves created_tx /
created_block untouched, and a ledger-cleared replay of the range skips
the creates via the guard (row counts and created_tx unchanged).
Resolves the conflict with F2/F3: the created_* provenance stays on the
create path only (this PR's point — the replay guard keys on created_tx, so an
edit must not rewrite it), and every timestamp this PR newly writes now goes
through toUTC, matching the convention master just adopted. That covers the
creator's users row in createCommunity and both create-path provenance blocks.

Also drops src/config/fresh.js, a local reproduction config whose own header
says it is not meant to be committed; it was swept into master by a careless
`git add -A` while landing F0. Added to .gitignore so it stays local.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lucca65
lucca65 merged commit 91aee8d into master Aug 8, 2026
2 checks passed
@lucca65
lucca65 deleted the fix/replay-bugs branch August 8, 2026 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant