F0: stop claim-id serial drift — resolve ids by paging the primary index - #61
Merged
Conversation
…aging guard
The byaction get_table_rows query filtered correctly all along, but on
nodeos v2.0.7 (prod) `more` is true whenever any row with a higher key
exists past the upper bound — it is not a truncation signal. The guard in
claimsForAction treated it as one and threw for virtually every claim, and
claimAction's catch then inserted with the DB serial ('falling back to
serial'), drifting DB claim ids off the chain ids (db_id+2 as of
2026-08-05; verifyclaim(db_id) names a different claim on chain).
- chain.js: replace the `res.more` truncation check with a real one
(rows.length >= limit) in claimsForAction / actionsForObjective /
objectivesForCommunity; add ResolveError type.
- community.js claimAction: no serial fallback. A resolve failure throws
ResolveError, so it can never reach the INSERT with a serial id.
- updaters.js ledgered: catch ResolveError, un-claim the action's
global_seq in _processed_actions (a later pass picks it up), page via
Sentry, and let the block commit so the indexer keeps running. Other
error types still propagate (crash-loop behavior unchanged).
- scripts/claims-id-drift-remediation-2026-08-05.sql: renumber the 59
drifted claims (with checks/notification_history following), backfill
the 2 chain claims missing from the DB, realign the serial. For Lucca
to review and run — pre-flight + verification queries included.
The previous commit on this branch diagnosed the drift as nodeos reporting a
spurious `more` on bounded secondary-index reads, and replaced the truncation
guard with a "did we hit `limit`" check. Measured against prod (v2.0.7) on
2026-08-07, that is not what happens:
* The identical bounded `byaction` claim query returned 31, 35, 38, 51, 60,
67 then 79 rows on seven consecutive calls, always with `more: true`. The
row count tracks page-cache warmth; once warm the same query returned all
115 rows with `more: false`.
* An unbounded read with `limit: 5000` returned 27 rows, `more: true`.
So the table walk is time-budgeted: `more` is an honest truncation signal, and a
row count below `limit` proves nothing. The replacement guard would have accepted
a truncated set, and because the target claim is the newest one for its pair --
the row truncation drops first -- resolution would return undefined and the claim
would be skipped entirely rather than inserted. That trades silent wrong ids for
silently dropped claims.
Resolve against the primary index instead, walking forward from a watermark (the
highest claim id already recorded) and paging on `next_key` until `more` is
false. Claim ids come from one global counter, and blocks are processed in order,
so the claim a `claimaction` created is the first chain claim for its
(action, claimer) above that watermark. Truncation then costs an extra request
instead of corrupting the answer, and a claim skipped earlier sits below the
watermark once anything later is recorded, so its id can never be handed out
twice. A secondary index cannot be used this way at all: its `next_key` comes
back as the secondary key (389 for a query bounded to action 389), so a truncated
secondary read cannot be resumed.
Verified with scripts/verify-claim-resolver.js against the prod chain: 636/636
probes returned the contract-truthful id, including probes whose match sits
hundreds of ids above the watermark and so must cross several truncated pages;
an exhausted pair throws instead of inventing an id.
The `more`-based guard is restored for the objective/action create-path
resolvers, whose sets are small enough to walk in one budget (objective 93 ->
2 rows, `more: false`); it errs toward throwing.
Replace the checked-in remediation SQL with a generator. The mapping is derived
from claim CONTENT (action, claimer, proof_photo, proof_code) rather than
sequence position: on prod, 223 of 282 rows in the affected range are already
correct and the bad ones are isolated displacements, not a uniform shift, so a
position-based alignment produces a wrong and non-monotonic map. The static SQL
also mapped 19907 -> 19915 where the proof photos show 19907 -> 19922 (19915 is
already correct), and predates claims created since, so one of its targets is now
held by a live row and it would abort on the primary key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
notification_history carries the claim id twice: the claim_id column (a real FK,
already renumbered) and again inside payload as double-encoded json,
{"record":{"id":N}}, with nothing enforcing it. Renumbering only the column left
every notification for a moved claim pointing at whichever claim now holds its
old id.
Verified on prod 2026-08-08: all 452 notification rows in the affected range have
exactly that shape, record.id always equals claim_id, and re-encoding the payload
round-trips byte-identically, so the rewrite is mechanical. Added a pre-COMMIT
check that no renumbered claim's payload disagrees with its claim_id, scoped to
the renumbered ids because other notification types carry a different shape.
Rehearsed end to end against a local copy of the real prod rows (282 claims, 550
checks, 452 notifications): the transaction commits with parked_left 0,
orphan_checks 0, payload_mismatches 0, and afterwards all 284 claims match the
chain exactly on (id, action_id, claimer, proof_photo) with every claim keeping
its checks and notifications.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 8, 2026
lucca65
added a commit
that referenced
this pull request
Aug 8, 2026
#61 removed the serial fallback for claims. The same fallback was still in the objective and action create paths, and it is the same defect: when the chain read failed, the resolver substituted a DB serial that silently names a DIFFERENT row. For an action that is worse than for a claim — claimaction carries action_id, so later claims attach to the wrong action, and the action becomes un-editable. That is the shape of the phantom action ids 399-406 the audit found. Removing the fallback alone would not have been safe, because the reads underneath it were still truncatable: * actionsForObjective used the `byobjective` SECONDARY index. Secondary reads cannot be resumed — nodeos returns the secondary key as next_key — so a short read just looks like "this objective has fewer actions than it does", and the caller then hands the create an id that is already taken. It now pages the whole `action` table through the primary index and filters client-side: 400 rows / 5 calls on prod, on the create path only. * objectivesForCommunity took a single call at limit 2000 and trusted it. It now pages its scope until more=false. Both go through one pager, and assertComplete is gone with them: a lone call is never proof of a complete set at any table size, so there is nothing left for a throw-on-`more` guard to protect. Chain reads are now retried (3 attempts, backing off) before giving up. Without a fallback a failed read costs a skipped create until someone reindexes, and about 2 in 100 calls came back without a rows array while paging the action table on prod — the acceptance check below failed 2 of 105 calls before the retry and passes cleanly after. The node's own message is carried into the error, since Sentry is not running and the log line is the only diagnostic. The claim reader shares the retry for the same reason. Create-path inserts log AND rethrow instead of swallowing. A swallowed insert drops the row while ledgered still records the action as processed, so no reindex revisits it — the likeliest explanation for the two claims found missing on prod. Verified with scripts/verify-create-resolvers.js against the prod chain: replay each parent's creation history (first k ids known, ask for the next) and require the resolver to name id k+1 every time. 14 communities / 95 objective steps and 77 action steps across the busiest objectives all pass, and an exhausted parent throws instead of inventing an id. verify-claim-resolver.js still 636/636. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claim rows were landing on DB serials instead of their chain ids, so the Elm app signed
verifyclaim(db_id)against a claim that either did not exist or belonged to someone else.Root cause
claimactiondoes not carry the generated id, soresolveClaimIdread it back off chain. That read used thebyactionsecondary index — and on prod nodeos (v2.0.7) theget_table_rowswalk is time-budgeted, so it stops early and setsmore: true. The old code threw onmore, and the caller caught the throw and fell back to a serial.Measured against prod on 2026-08-07:
byactionquery returned 31, 35, 38, 51, 60, 67, 79 rows on seven consecutive calls, alwaysmore: true— the count tracks page-cache warmth, so it is not even deterministic. Once warm the same query returned all 115 rows withmore: false.limit: 5000returned 27 rows,more: true.So
moreis an honest truncation signal. An earlier revision of this PR read it as spurious and replaced the guard with "did we hitlimit" — that would have accepted a truncated set, and since the target claim is the newest one for its pair (exactly what truncation drops first), resolution would returnundefinedand the claim would be skipped and never inserted. That trades silent wrong ids for silently dropped claims.Fix
Resolve against the primary index, walking forward from a watermark (the highest claim id already recorded) and paging on
next_keyuntilmoreis false. Claim ids come from one global counter (get_available_id("claims")) and blocks are processed in order, so the claim aclaimactioncreated is the first chain claim for its (action, claimer) above that watermark.Truncation then costs an extra request instead of corrupting the answer, and a claim skipped earlier sits below the watermark once anything later is recorded, so its id can never be handed out twice. A secondary index cannot be used this way at all: its
next_keycomes back as the secondary key (389for a query bounded to action 389), so a truncated secondary read cannot be resumed.The serial fallback is gone. A resolve failure raises
ResolveError;ledgeredun-claims the action'sglobal_seqand pages Sentry. Note this is real data loss until someone reindexes the range — demux's cursor advances regardless — but a missing claim can be reindexed and a wrong primary key cannot be undone.The
more-based guard is restored for the objective/action create-path resolvers, whose sets fit in one walk budget (objective 93 → 2 rows,more: false).Verification
scripts/verify-claim-resolver.jsagainst the prod chain: 636/636 probes returned the contract-truthful id, including probes whose match sits hundreds of ids above the watermark and so must cross several truncated pages. An exhausted pair throws instead of inventing an id.Remediation
The checked-in mapping is replaced by a generator (
scripts/build-claims-id-remediation.py), because a static map goes stale within hours while drift is still growing.The mapping is derived from claim content (
action_id, claimer, proof_photo, proof_code), not sequence position: on prod, 223 of 282 rows in the affected range are already correct and the bad ones are isolated displacements, not a uniform shift, so a position-based alignment yields a wrong and non-monotonic map. The previous static SQL mapped19907 → 19915where the proof photos give19907 → 19922(19915is already correct), and one of its targets is now held by a live row, so it would abort on the primary key.It also repoints
notification_history.payload, which embeds the claim id a second time as double-encoded json ({"record":{"id":N}}) with no FK — renumbering only the column left every notification for a moved claim pointing at whichever claim took its old id.Rehearsed end to end against a local copy of the real prod rows (282 claims, 550 checks, 452 notifications): commits with
parked_left 0,orphan_checks 0,payload_mismatches 0, and afterwards all 284 claims match the chain exactly on (id, action_id, claimer, proof_photo), every claim keeping its checks and notifications.Current output: 59 renumbers + 2 backfills (chain claims
19871welovecircus and19904bananadacult, which never got a DB row — twoclaimactions for the same pair in one transaction, collapsed by the(created_tx, action_id, claimer_id)dedup guard).