DLQ Workbench

DLQ Workbench is a self-hosted operations console for inspecting, grouping and safely replaying RabbitMQ dead-letter messages.

Important

This is a maintenance-only reference implementation, not a supported production product. Review the known limitations and run it against a disposable broker before evaluating it with real data.

It is deliberately narrower than a general RabbitMQ management console. The core workflow is:

inspect/search -> select -> immutable replay plan -> dry run
               -> explicit approval -> execution -> audit

Where it fits

RabbitMQ's Management plugin and Shovel solve adjacent problems. Management is useful for broker administration and small-scale message inspection; Shovel is useful for continuously forwarding messages between endpoints. DLQ Workbench explores a narrower incident-response workflow: inspect a bounded sample, select failure groups, preview an immutable replay plan, approve it, and retain an audit record.

	Management plugin	Shovel	DLQ Workbench
Bounded message inspection	basic	no	yes
Sensitive-field masking	external	n/a	yes
Grouping by failure fingerprint	external	no	yes
Selective replay plan	manual	routing/filter rules	yes
Explicit dry run and approval	no	no	yes
Replay progress and cancellation	no	operational controls	yes
Payload-free operator audit	external	external	yes
Application-level RBAC	no	no	yes

If the goal is simply to forward a queue, Shovel is the smaller and better supported choice. This repository is useful when studying the safety mechanics around selective replay and operator accountability.

What is implemented

• RabbitMQ connection storage with encrypted passwords and DLQ/DLX discovery.
• Bounded, non-destructive inspection of at most 100 messages.
• Recursive masking, failure fingerprints and combined masked search filters.
• Confirmed single replay and immutable multi-group batch replay.
• Publisher confirms, mandatory publishing, rate limits, route locks, cancellation, progress and partial-failure reporting.
• Replay-loop detection, idempotency warnings, circuit breaking and crash recovery with an exclusive worker lease.
• Local session authentication and hierarchical viewer, operator, admin RBAC.
• Payload-free append-only audit, Prometheus metrics, health probes and optional privacy-bounded OpenTelemetry traces.
• React UI, reproducible Docker Compose demo and real-broker integration/chaos tests.

The workflow, screen by screen

Discovery reads queue arguments and policies to find which DLQ belongs to which source queue, and how confident that mapping is:

Inspection reads a bounded set of messages, masks sensitive fields recursively and returns every message to the queue. Failures are grouped by fingerprint, so six messages collapse into three actionable causes:

The operator selects the groups worth retrying — not the whole queue:

The plan freezes the exact selected messages by fingerprint and payload hash, validates the target route and requires the count to be typed out by hand:

Execution is rate limited, cancellable and reports progress per message:

Every step lands in a payload-free audit trail that PostgreSQL will not let the application rewrite:

Quick start

Requirements: Docker Desktop or Docker Engine with Compose v2.

Copy-Item .env.example .env
docker compose up --build -d
docker compose --profile demo run --rm demo-seed

Open http://localhost:5173 and sign in with the development bootstrap account:

admin / workbench-local

Add the bundled RabbitMQ using the container-network addresses:

Management URL: http://rabbitmq:15672
AMQP endpoint:  amqp://rabbitmq:5672
Username:       workbench
Password:       workbench
Virtual host:   /

The example password and encryption key are for the local demo only. See the full quick start before changing or resetting the stack.

Documentation

Document	Purpose
Quick start	Start the stack and complete the first safe inspection
Demo and recording guide	Reproduce the GIF and the complete safe batch replay flow
Release process	Quality gates, artifacts, SBOMs and Git tags
Changelog	Versioned product changes
Security policy	Supported status and vulnerability reporting
Contributing	Defect reports, scope and development checks
Operator guide	Roles, inspection, single/batch replay and recovery
Configuration	Environment variables, secrets and deployment boundaries
Architecture	Components, data flow and sensitive-data boundaries
Observability	Health, metrics, logs and OpenTelemetry
API reference	Authentication, RBAC and endpoint inventory
Troubleshooting	Common startup, RabbitMQ, auth and replay failures
Architecture decisions	Decision records and their consequences

Safety invariants

• Inspection and search requeue every message and never persist raw payloads.
• Search runs only over the bounded, masked in-memory representation; search values are not written to PostgreSQL or audit.
• A batch plan stores exact fingerprint/payload-hash snapshot items, not the search that produced them. Execution never repeats the original search.
• A DLQ message is acknowledged only after the target publish is confirmed.
• Replayed messages preserve their properties and correlation ID and gain an x-dlq-workbench-replay-id marker.
• Only one active replay can own a source/target route.
• A worker that loses its lease stops publishing. Reconciliation closes an abandoned running plan as terminal interrupted; it never resumes it automatically.
• Audit, metrics and Workbench-authored traces exclude raw payloads, credentials and search values.
• Arbitrary payload transformation code is not supported.

The detailed threat and failure model is distributed across the operator guide, architecture overview and ADRs.

Known limitations

These include scope boundaries and known operational defects in 0.1.0. Read them before deciding whether the tool fits your situation.

• 100-message horizon. Inspection, search, grouping and every replay plan are bounded to the first 100 messages of a queue. A DLQ holding thousands of messages is visible only through that window, and the UI does not yet show the true queue depth. The bound is what makes inspection provably safe; it is also what makes this a triage tool rather than a bulk-drain tool.
• RabbitMQ only. Kafka, SQS and Redis Streams are out of scope. The domain layer is broker-agnostic, but the infrastructure layer is not.
• JSON payloads only. Non-JSON bodies (protobuf, Avro, MessagePack, plain text) are reported as unparseable with size and headers, but not rendered.
• Masking is key-based. Sensitive values are redacted by matching the field name. A secret stored under an innocuous key is not detected.
• Connections are create-only. There is no endpoint to edit or delete a saved RabbitMQ connection once it exists.
• Discovery-driven queue selection. Only queues returned by discovery can be inspected from the UI; there is no free-text queue entry.
• One job at a time per worker. Route locking and worker leases are built for several worker processes, but a single worker processes batches serially.
• No retention policy. The append-only audit trigger blocks DELETE entirely, so pruning history requires dropping the trigger first.
• Approved-plan lifecycle. An approved plan that is never started keeps its source/target route reserved until it is cancelled. There is no active-plan list in the UI, so retain the plan URL/ID until the operation is terminal.

Development checks

Backend (Python 3.12):

python -m venv .venv
python -m pip install -e ".[dev]"
ruff check .
mypy src
pytest -m "not integration"
pytest -m integration

The integration selection needs a running Docker daemon and starts temporary PostgreSQL, Redis and RabbitMQ containers.

Frontend (Node 24 and pnpm 11 in CI):

cd frontend
pnpm install --frozen-lockfile
pnpm run typecheck
pnpm run build
pnpm audit --audit-level moderate

Supply-chain checks and CycloneDX SBOM generation run in CI.

Project status

Version 0.1.0 is a finished reference implementation in maintenance mode. There is no feature roadmap or production support commitment. Issues that document reproducible defects are welcome; forks are encouraged for continued development.

Licensed under Apache-2.0.