fix(indexer): bound NFT metadata lookups with a worker pool and per-cycle budget

[ezedike-evan]

closes #205

Summary

pollOnce awaited getNftMetadata (and the metadata fetch) one token at a time, so N new tokens cost N serial round trips and a hung metadata source could stall the cursor.

Changes

• New src/indexer/nft-metadata.ts with enrichNftMetadata: a worker pool (default 8, NFT_METADATA_CONCURRENCY) under a per-cycle time budget (default 10000 ms, NFT_METADATA_BUDGET_MS). No lookup starts after the budget is spent and an in-flight lookup that outlives it is abandoned, so the loop returns in roughly the budget regardless of source speed.
• Failures are per token (cache read, fetch, upsert). Skipped or timed-out tokens are not written, so they stay uncached and are retried next cycle.
• src/indexer.ts now calls it in place of the serial loop. Logs never include the raw error object (may carry DB errors or provider URLs); no metric labels added.
• .env.example documents the two env vars.

Before / after

Unit test: 20 tokens with 50 ms lookups. Serial: at least 1000 ms. Pool of 5: about 200 ms (asserted under 700 ms, peak concurrency asserted at most 5).

Verification

• npx jest src/__tests__/nftMetadata.test.ts: 10 passed (slow source, failing source, mixed batch, dedupe/cache, budget, env parsing). The module did not exist before, so these tests cannot pass on the original code.
• npx jest src/__tests__/multiNetworkIndexer.test.ts src/__tests__/nft.test.ts: 35 passed.
• npm run typecheck: clean.
• Full suite not run (shared, heavily loaded machine).

Caveats

Timing tests use real timers with generous margins. Abandoned in-flight fetches are not cancelled (the RPC client has no abort here); they finish in the background and their result is discarded.

[drips-wave]

@ezedike-evan Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Miracle656]

Merged. Verified locally: typecheck clean and the full jest suite green at 552 passed / 46 suites.

enrichNftMetadata delivers all three things #205 asked for, and the tests actually demonstrate each rather than asserting the function runs: a worker pool with a measured peak in-flight count, a budget that a never-settling fetch cannot outlive, and per-token isolation shown by the exact set of tokens that still got stored around a thrower. The detail I'd call out is expect(errors).toEqual(["NFT metadata lookup failed"]) on an error carrying https://secret.provider/url — that pins the repo rule that a provider URL never reaches the log, not just that an error was counted. Deferring rather than writing an empty row on timeout is also the right choice: those tokens stay uncached and get retried instead of being poisoned.

I pushed a merge commit to your branch with two changes:

1. Rebased onto fix(indexer): share one per-batch pipeline between the single and parallel ingest paths #219, which landed first. fix(indexer): share one per-batch pipeline between the single and parallel ingest paths #219 moved the whole per-batch pipeline out of src/indexer.ts and into src/indexer/batch.ts so the single and sharded ingest paths share one body, so your serial loop was no longer where you left it. I resolved src/indexer.ts to main's version and ported your enrichNftMetadata call into batch.ts. This is strictly better than the original placement: the bound now applies to the parallel path too, which previously did no metadata work at all. fix(indexer): share one per-batch pipeline between the single and parallel ingest paths #219's ingestParity.test.ts passes against the result, which is real evidence the move preserved behaviour — NftMetadata rows are part of the snapshot it compares across both paths. One consequence worth knowing: the budget is per batch, so a sharded run with N workers can spend up to N budgets concurrently. Fine at the defaults, but it means NFT_METADATA_CONCURRENCY is per worker, not global.
2. src/__tests__/nftMetadata.test.ts hardcoded CBC42KFZO33TYVFDOUXFRWXYYXHFGH7W5GM4IJQSXKGFINKL2XPP4XTE as CONTRACT. It is 56 characters but fails StrKey.isValidContract — the checksum is wrong. Nothing broke, because the deps are mocked and no decode happens, but a fabricated address in a fixture has bitten this repo before, including one that threw at import and ran zero tests while the PR reported green. Replaced with StrKey.encodeContract(Buffer.alloc(32, 11)), valid by construction.

One pre-existing behaviour I left alone: fetch: ... .catch(() => ({})) turns a failed lookup into an empty metadata object which is then upserted, so a transient RPC failure caches an empty record permanently and it is never retried. That came from the original code, and your deferred path correctly avoids introducing a second instance of it, but it is worth its own issue.

https://claude.ai/code/session_01USgemLt4Rnz4SGB1Srf3GB