Skip to content

F0: stop claim-id serial drift — resolve ids by paging the primary index - #61

Merged
lucca65 merged 3 commits into
masterfrom
fix/claim-id-drift
Aug 8, 2026
Merged

F0: stop claim-id serial drift — resolve ids by paging the primary index#61
lucca65 merged 3 commits into
masterfrom
fix/claim-id-drift

Conversation

@lucca65

@lucca65 lucca65 commented Aug 5, 2026

Copy link
Copy Markdown
Member

Claim rows were landing on DB serials instead of their chain ids, so the Elm app signed verifyclaim(db_id) against a claim that either did not exist or belonged to someone else.

Root cause

claimaction does not carry the generated id, so resolveClaimId read it back off chain. That read used the byaction secondary index — and on prod nodeos (v2.0.7) the get_table_rows walk is time-budgeted, so it stops early and sets more: true. The old code threw on more, and the caller caught the throw and fell back to a serial.

Measured against prod on 2026-08-07:

  • the identical bounded byaction query returned 31, 35, 38, 51, 60, 67, 79 rows on seven consecutive calls, always more: true — the count tracks page-cache warmth, so it is not even deterministic. Once warm the same query returned all 115 rows with more: false.
  • an unbounded read with limit: 5000 returned 27 rows, more: true.

So more is an honest truncation signal. An earlier revision of this PR read it as spurious and replaced the guard with "did we hit limit" — that would have accepted a truncated set, and since the target claim is the newest one for its pair (exactly what truncation drops first), resolution would return undefined and the claim would be skipped and never inserted. That trades silent wrong ids for silently dropped claims.

Fix

Resolve against the primary index, walking forward from a watermark (the highest claim id already recorded) and paging on next_key until more is false. Claim ids come from one global counter (get_available_id("claims")) and blocks are processed in order, so the claim a claimaction created is the first chain claim for its (action, claimer) above that watermark.

Truncation then costs an extra request instead of corrupting the answer, and a claim skipped earlier sits below the watermark once anything later is recorded, so its id can never be handed out twice. A secondary index cannot be used this way at all: its next_key comes back as the secondary key (389 for a query bounded to action 389), so a truncated secondary read cannot be resumed.

The serial fallback is gone. A resolve failure raises ResolveError; ledgered un-claims the action's global_seq and pages Sentry. Note this is real data loss until someone reindexes the range — demux's cursor advances regardless — but a missing claim can be reindexed and a wrong primary key cannot be undone.

The more-based guard is restored for the objective/action create-path resolvers, whose sets fit in one walk budget (objective 93 → 2 rows, more: false).

Verification

scripts/verify-claim-resolver.js against the prod chain: 636/636 probes returned the contract-truthful id, including probes whose match sits hundreds of ids above the watermark and so must cross several truncated pages. An exhausted pair throws instead of inventing an id.

Remediation

The checked-in mapping is replaced by a generator (scripts/build-claims-id-remediation.py), because a static map goes stale within hours while drift is still growing.

The mapping is derived from claim content (action_id, claimer, proof_photo, proof_code), not sequence position: on prod, 223 of 282 rows in the affected range are already correct and the bad ones are isolated displacements, not a uniform shift, so a position-based alignment yields a wrong and non-monotonic map. The previous static SQL mapped 19907 → 19915 where the proof photos give 19907 → 19922 (19915 is already correct), and one of its targets is now held by a live row, so it would abort on the primary key.

It also repoints notification_history.payload, which embeds the claim id a second time as double-encoded json ({"record":{"id":N}}) with no FK — renumbering only the column left every notification for a moved claim pointing at whichever claim took its old id.

Rehearsed end to end against a local copy of the real prod rows (282 claims, 550 checks, 452 notifications): commits with parked_left 0, orphan_checks 0, payload_mismatches 0, and afterwards all 284 claims match the chain exactly on (id, action_id, claimer, proof_photo), every claim keeping its checks and notifications.

Current output: 59 renumbers + 2 backfills (chain claims 19871 welovecircus and 19904 bananadacult, which never got a DB row — two claimactions for the same pair in one transaction, collapsed by the (created_tx, action_id, claimer_id) dedup guard).

lucca65 and others added 3 commits August 5, 2026 11:36
…aging guard

The byaction get_table_rows query filtered correctly all along, but on
nodeos v2.0.7 (prod) `more` is true whenever any row with a higher key
exists past the upper bound — it is not a truncation signal. The guard in
claimsForAction treated it as one and threw for virtually every claim, and
claimAction's catch then inserted with the DB serial ('falling back to
serial'), drifting DB claim ids off the chain ids (db_id+2 as of
2026-08-05; verifyclaim(db_id) names a different claim on chain).

- chain.js: replace the `res.more` truncation check with a real one
  (rows.length >= limit) in claimsForAction / actionsForObjective /
  objectivesForCommunity; add ResolveError type.
- community.js claimAction: no serial fallback. A resolve failure throws
  ResolveError, so it can never reach the INSERT with a serial id.
- updaters.js ledgered: catch ResolveError, un-claim the action's
  global_seq in _processed_actions (a later pass picks it up), page via
  Sentry, and let the block commit so the indexer keeps running. Other
  error types still propagate (crash-loop behavior unchanged).
- scripts/claims-id-drift-remediation-2026-08-05.sql: renumber the 59
  drifted claims (with checks/notification_history following), backfill
  the 2 chain claims missing from the DB, realign the serial. For Lucca
  to review and run — pre-flight + verification queries included.
The previous commit on this branch diagnosed the drift as nodeos reporting a
spurious `more` on bounded secondary-index reads, and replaced the truncation
guard with a "did we hit `limit`" check. Measured against prod (v2.0.7) on
2026-08-07, that is not what happens:

  * The identical bounded `byaction` claim query returned 31, 35, 38, 51, 60,
    67 then 79 rows on seven consecutive calls, always with `more: true`. The
    row count tracks page-cache warmth; once warm the same query returned all
    115 rows with `more: false`.
  * An unbounded read with `limit: 5000` returned 27 rows, `more: true`.

So the table walk is time-budgeted: `more` is an honest truncation signal, and a
row count below `limit` proves nothing. The replacement guard would have accepted
a truncated set, and because the target claim is the newest one for its pair --
the row truncation drops first -- resolution would return undefined and the claim
would be skipped entirely rather than inserted. That trades silent wrong ids for
silently dropped claims.

Resolve against the primary index instead, walking forward from a watermark (the
highest claim id already recorded) and paging on `next_key` until `more` is
false. Claim ids come from one global counter, and blocks are processed in order,
so the claim a `claimaction` created is the first chain claim for its
(action, claimer) above that watermark. Truncation then costs an extra request
instead of corrupting the answer, and a claim skipped earlier sits below the
watermark once anything later is recorded, so its id can never be handed out
twice. A secondary index cannot be used this way at all: its `next_key` comes
back as the secondary key (389 for a query bounded to action 389), so a truncated
secondary read cannot be resumed.

Verified with scripts/verify-claim-resolver.js against the prod chain: 636/636
probes returned the contract-truthful id, including probes whose match sits
hundreds of ids above the watermark and so must cross several truncated pages;
an exhausted pair throws instead of inventing an id.

The `more`-based guard is restored for the objective/action create-path
resolvers, whose sets are small enough to walk in one budget (objective 93 ->
2 rows, `more: false`); it errs toward throwing.

Replace the checked-in remediation SQL with a generator. The mapping is derived
from claim CONTENT (action, claimer, proof_photo, proof_code) rather than
sequence position: on prod, 223 of 282 rows in the affected range are already
correct and the bad ones are isolated displacements, not a uniform shift, so a
position-based alignment produces a wrong and non-monotonic map. The static SQL
also mapped 19907 -> 19915 where the proof photos show 19907 -> 19922 (19915 is
already correct), and predates claims created since, so one of its targets is now
held by a live row and it would abort on the primary key.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
notification_history carries the claim id twice: the claim_id column (a real FK,
already renumbered) and again inside payload as double-encoded json,
{"record":{"id":N}}, with nothing enforcing it. Renumbering only the column left
every notification for a moved claim pointing at whichever claim now holds its
old id.

Verified on prod 2026-08-08: all 452 notification rows in the affected range have
exactly that shape, record.id always equals claim_id, and re-encoding the payload
round-trips byte-identically, so the rewrite is mechanical. Added a pre-COMMIT
check that no renumbered claim's payload disagrees with its claim_id, scoped to
the renumbered ids because other notification types carry a different shape.

Rehearsed end to end against a local copy of the real prod rows (282 claims, 550
checks, 452 notifications): the transaction commits with parked_left 0,
orphan_checks 0, payload_mismatches 0, and afterwards all 284 claims match the
chain exactly on (id, action_id, claimer, proof_photo) with every claim keeping
its checks and notifications.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lucca65 lucca65 changed the title F0: stop claim-id serial drift — more is not truncation on nodeos v2.0; fail loud, never invent ids F0: stop claim-id serial drift — resolve ids by paging the primary index Aug 8, 2026
@lucca65
lucca65 merged commit 8f37a07 into master Aug 8, 2026
2 checks passed
@lucca65
lucca65 deleted the fix/claim-id-drift branch August 8, 2026 11:51
lucca65 added a commit that referenced this pull request Aug 8, 2026
#61 removed the serial fallback for claims. The same fallback was still in the
objective and action create paths, and it is the same defect: when the chain
read failed, the resolver substituted a DB serial that silently names a
DIFFERENT row. For an action that is worse than for a claim — claimaction
carries action_id, so later claims attach to the wrong action, and the action
becomes un-editable. That is the shape of the phantom action ids 399-406 the
audit found.

Removing the fallback alone would not have been safe, because the reads
underneath it were still truncatable:

  * actionsForObjective used the `byobjective` SECONDARY index. Secondary reads
    cannot be resumed — nodeos returns the secondary key as next_key — so a
    short read just looks like "this objective has fewer actions than it does",
    and the caller then hands the create an id that is already taken. It now
    pages the whole `action` table through the primary index and filters
    client-side: 400 rows / 5 calls on prod, on the create path only.
  * objectivesForCommunity took a single call at limit 2000 and trusted it. It
    now pages its scope until more=false.

Both go through one pager, and assertComplete is gone with them: a lone call is
never proof of a complete set at any table size, so there is nothing left for a
throw-on-`more` guard to protect.

Chain reads are now retried (3 attempts, backing off) before giving up. Without
a fallback a failed read costs a skipped create until someone reindexes, and
about 2 in 100 calls came back without a rows array while paging the action
table on prod — the acceptance check below failed 2 of 105 calls before the
retry and passes cleanly after. The node's own message is carried into the
error, since Sentry is not running and the log line is the only diagnostic.
The claim reader shares the retry for the same reason.

Create-path inserts log AND rethrow instead of swallowing. A swallowed insert
drops the row while ledgered still records the action as processed, so no
reindex revisits it — the likeliest explanation for the two claims found
missing on prod.

Verified with scripts/verify-create-resolvers.js against the prod chain: replay
each parent's creation history (first k ids known, ask for the next) and require
the resolver to name id k+1 every time. 14 communities / 95 objective steps and
77 action steps across the busiest objectives all pass, and an exhausted parent
throws instead of inventing an id. verify-claim-resolver.js still 636/636.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant