Skip to content

Skip stream entries whose content does not match their trusty URI (fixes #208) - #209

Merged
tkuhn merged 1 commit into
mainfrom
fix-poison-nanopub-wedge
Aug 30, 2026
Merged

Skip stream entries whose content does not match their trusty URI (fixes #208)#209
tkuhn merged 1 commit into
mainfrom
fix-poison-nanopub-wedge

Conversation

@tkuhn

@tkuhn tkuhn commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Root cause (corrects the diagnosis in #208)

The poison entry (RAAAAA…, counter 88951) has a valid signature but an invalid trusty hash — its content does not hash to its 43-char artifact code (knowledgepixels/nanopub-registry#164). The Jelly layer decodes it as a success item; nothing in the load path verifies the trusty hash, so it passed every constructor check and died in the repo writers:

java.lang.ArrayIndexOutOfBoundsException: Index 30 out of bounds for length 30
	at org.nanopub.NanopubUtils.updateXorChecksum(NanopubUtils.java:419)
	at com.knowledgepixels.query.NanopubLoader.loadNanopubToRepo(NanopubLoader.java:770)

The generic retry loops (8 attempts, backoff up to ~90 s, per repo task, per poll) treated this deterministic error as transient — so each poll cycle ground through ~10–20 minutes of retries, threw, and started over, forever. It was not actually silent (Failed to load nanopub <…> to repo 'full': Index 30 out of bounds lines were present but sparse), and the liveness age stamp already behaves correctly during failing batches — asks 2 and 3 of the issue were based on wrong incident-time observations.

Fix

  • NanopubLoader: TrustyNanopubUtils.isValidTrustyNanopub(np) is now the first constructor check (~6 ms/np). On mismatch: ERROR log, note in the admin repo, aborted = true — riding the existing abort flow, so the counter advances past the entry. One bad stream entry can no longer stall ingestion; a future resync sails past counter 88951 with a note instead of wedging. Also guards the latent NPE when getArtifactCode returns null.
  • JellyNanopubLoader.loadUpdates: a "loader wedged" ERROR with operator guidance once 3 consecutive batch failures pass without counter movement. Log-only by design — a store outage produces the same signature, and count-based skipping there would silently drop real nanopubs.
  • Decode-level (!isSuccess) failures still throw and retry on purpose: truncated stream reads are transient, and skipping would leave holes.

Test

PoisonNanopubSkipTest uses the real captured registry stream segment (?afterCounter=88950 from registry.petapico.org, 2026-08-30) as a fixture: asserts the poison entry is aborted with the note and returns normally from load(), and that the valid sibling entry (88952) is not rejected. The abort path runs entirely on the calling thread, which is what makes it testable against InMemoryTripleStore. All 517 tests pass.

Fixes #208.

🤖 Generated with Claude Code

An entry whose signature is valid but whose content does not hash to its
trusty URI artifact code (e.g. the RAAAAA... nanopub at counter 88951,
2026-08-29 — knowledgepixels/nanopub-registry#164) used to pass every
loader check and reach the repo writers, where
NanopubUtils.updateXorChecksum threw ArrayIndexOutOfBoundsException on
the short artifact code. The retry loops treated that deterministic
error as transient, so every poll re-fetched the same entry and
re-failed for minutes at a time, wedging ingestion fleet-wide on one
counter (#208).

- NanopubLoader: verify TrustyNanopubUtils.isValidTrustyNanopub as the
  first constructor check; on mismatch, log an ERROR and route the entry
  through the existing aborted/notes flow so the counter advances past
  it. Also guards the latent NPE when getArtifactCode returns null.
- JellyNanopubLoader.loadUpdates: log a "loader wedged" ERROR with
  operator guidance once 3 consecutive batch failures pass without the
  counter moving. Log-only by design: a store outage produces the same
  signature, and skipping there would silently drop real nanopubs.
- Decode-level (!isSuccess) failures still throw and retry on purpose:
  a truncated stream read is transient, and skipping there would leave
  holes.
- Regression test with the real captured registry segment (poison entry
  88951 plus a valid sibling) as fixture.

Fixes #208.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tkuhn
tkuhn merged commit 5f46746 into main Aug 30, 2026
8 checks passed
@tkuhn
tkuhn deleted the fix-poison-nanopub-wedge branch August 30, 2026 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An invalid entry in the registry stream stalls the loader silently — no error, no skip, no liveness alarm

1 participant