[Backend] Indexer Dead-Letter Event Replay, Inspection & Triage Management API

[blurbeast]

Overview & Background

The database model IndexerDeadLetterEvent captures Soroban events that failed processing (e.g. unexpected payload format, transient database lock, RPC timeout during processing). This ensures a single malformed event does not freeze the indexer worker cursor. However, currently there are no administrative endpoints to query, inspect, retry, or dismiss these dead-letter events.

---

Detailed Problem Statement

1. Zero Visibility for Operators:
   • When events fail, they reside silently in the database without any alerting or administration UI.
2. Manual Intervention Required:
   • Resolving dropped events currently requires executing raw SQL queries and manually reconstructing state mutations.
3. No Controlled Replay Mechanism:
   • Once an indexer bug is patched and deployed, operators cannot trigger a clean replay of quarantined events.

---

Technical Specification & Architecture

1. Administrative Dead-Letter Endpoints

Implement in backend/src/routes/v1/admin.routes.ts:

• GET /api/v1/admin/indexer/dead-letter:
  • Query parameters: page, limit, ledgerSequence, startDate, endDate.
  • Returns paginated list of dead-letter records with error logs and raw payloads.
• POST /api/v1/admin/indexer/dead-letter/:id/replay:
  • Re-injects the dead-letter event payload into the indexer pipeline (indexer.service.ts).
  • If processing succeeds: deletes the dead-letter entry and creates corresponding StreamEvent and Stream records.
  • If processing fails: increments attempts counter and updates lastAttemptAt and errorMessage.
• POST /api/v1/admin/indexer/dead-letter/replay-all:
  • Iterates and replays all pending dead-letter records sequentially.
• DELETE /api/v1/admin/indexer/dead-letter/:id:
  • Permanently discards an unrecoverable event with audit log notation.

2. Authorization

• Protected by requireAdminAuth JWT middleware.

---

Target Files

• backend/src/routes/v1/admin.routes.ts
• backend/src/controllers/admin.controller.ts
• backend/src/services/indexer.service.ts
• backend/tests/indexer-service.test.ts

---

Acceptance Criteria

• Admin can retrieve paginated list of failed events with complete error diagnostics.
• Single and batch replay successfully executes events through the pipeline and cleans up dead-letter table.
• Protected by admin authentication; unauthorized requests receive 401/403.
• Unit and integration tests verify error logging, replay execution, and deletion.

[pimaster900]

@pimaster900 has applied to work on this issue as part of the Stellar Wave Program's 9th wave.

i would like to work on this

ℹ️ Repo Maintainers: To accept this application, review their application or assign @pimaster900 to this issue.

[Jewelly13]

@Jewelly13 has applied to work on this issue as part of the Stellar Wave Program's 9th wave.

Kindly assign this issue to me

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Jewelly13 to this issue.

[divinemike019]

@divinemike019 has applied to work on this issue as part of the Stellar Wave Program's 9th wave.

Hi maintainer, I like to work on this issue
I’ll focus on [specific issue requirement], reviewing the existing [relevant component/service/module] and implementing the required changes while following the project’s current patterns. I’ll also add or update relevant tests to cover the change and verify that existing functionality remains unaffected.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @divinemike019 to this issue.

[Mikey-222]

@Mikey-222 has applied to work on this issue as part of the Stellar Wave Program's 9th wave.

can i take this

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Mikey-222 to this issue.

[drips-wave]

Congratulations, @Mikey-222! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on September 30, 2026.

🧑‍💻 @Mikey-222: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊