You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9c1155a
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: src/content/articles/dead-letter-queue-design-replay-debugging/index.mdx
+17-12Lines changed: 17 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,42 +22,47 @@ The problem isn't the DLQ concept - it's three specific design failures that mak
22
22
23
23
## Classify Failures at Write Time
24
24
25
-
Messages fail for different reasons, and those reasons determine how you should handle them. Most DLQs treat all failures the same - a message failed, it goes in the queue, someone will look at it eventually. That "eventually" often becomes "never" because manual triage doesn't scale.
25
+
Messages fail for different reasons, and those reasons determine how you should handle them. <Highlighter>Most DLQs treat all failures the same - a message failed, it goes in the queue, someone will look at it eventually.</Highlighter> That "eventually" often becomes "never" because manual triage doesn't scale.
26
26
27
27
The failure taxonomy matters because different failure types require fundamentally different responses:
28
28
29
29
<List
30
-
variant="check-icons-list"
30
+
variant="zebra-list"
31
31
items={[
32
32
{
33
-
text: "Transient failures are temporary issues that resolve themselves: database connection timeouts, downstream service unavailability, rate limit errors, lock contention. These should succeed on retry. The right strategy is automatic replay after a cooldown period - no human intervention required if your retry policy is sound.",
33
+
lead: "Transient failures",
34
+
text: "These are temporary issues that resolve themselves: database connection timeouts, downstream service unavailability, rate limit errors, lock contention. These should succeed on retry. The right strategy is automatic replay after a cooldown period - no human intervention required if your retry policy is sound.",
34
35
},
35
36
{
36
-
text: "Permanent failures won't resolve without intervention: invalid message format, referenced entity doesn't exist, business rule violation, schema version mismatch. These will never succeed without a fix. Someone needs to look at the payload, understand what's wrong, and either fix the data or discard the message.",
37
+
lead: "Permanent failures",
38
+
text: "These won't resolve without intervention: invalid message format, referenced entity doesn't exist, business rule violation, schema version mismatch. These will never succeed without a fix. Someone needs to look at the payload, understand what's wrong, and either fix the data or discard the message.",
37
39
},
38
40
{
39
-
text: "Poison messages crash the consumer: null pointer in a required field, infinite loop triggered by specific data, memory exhaustion from an oversized payload. The consumer dies on every attempt. These are the most dangerous because they can take down your processing capacity. The strategy is to isolate immediately, fix the consumer code, then replay.",
41
+
lead: "Poison messages",
42
+
text: "These crash the consumer: null pointer in a required field, infinite loop triggered by specific data, memory exhaustion from an oversized payload. The consumer dies on every attempt. These are the most dangerous because they can take down your processing capacity. The strategy is to isolate immediately, fix the consumer code, then replay.",
40
43
},
41
44
{
42
-
text: "Ordering failures fail due to sequence issues: an update arriving before the create, a delete for a non-existent record. These may succeed if replayed in correct order. The strategy is to hold for reordering, then replay - but this often requires understanding the business context to determine the correct sequence.",
45
+
lead: "Poison messages",
46
+
text: "These fail due to sequence issues: an update arriving before the create, a delete for a non-existent record. These may succeed if replayed in correct order. The strategy is to hold for reordering, then replay - but this often requires understanding the business context to determine the correct sequence.",
43
47
},
44
48
]}
45
49
/>
46
50
47
51
When you classify failures at write time - the moment the message moves to the DLQ - you enable automation. Transient failures route to an auto-replay queue with a delay. Permanent failures route to a team-specific triage queue. Poison messages trigger an immediate alert. Without classification, every message requires a human to open it, read the error, and decide what to do. That's why DLQs accumulate for months.
48
52
49
-
The classification should be explicit in your DLQ schema, not inferred later. Your consumer already knows why it failed - a `ValidationError` is permanent, a `ConnectionError` is transient, an unhandled exception might be poison. Map exception types to failure classifications programmatically, and use that classification to drive routing and alerting.
50
-
51
53
<Callouttype="info">
52
54
If your DLQ treats all failures the same, you're forcing humans to do what software should do. Classification is the foundation that makes everything else - auto-replay, targeted alerts, efficient triage - possible.
53
55
</Callout>
54
56
57
+
The classification should be explicit in your DLQ schema, not inferred later. Your consumer already knows why it failed - a `ValidationError` is permanent, a `ConnectionError` is transient, an unhandled exception might be poison. Map exception types to failure classifications programmatically, and use that classification to drive routing and alerting.
58
+
55
59
## Capture Context Before It's Gone
56
60
57
61
Traditional DLQs fail at their job because they don't capture enough context. Most give you the original message payload, a timestamp when the message was DLQ'd, and maybe the number of retries. That's it. When you're debugging a failure three weeks later, you're staring at a payload with no idea what went wrong.
58
62
59
63
<Table
60
-
variant="vertical-column-delineation-table"
64
+
variant="grid-and-accent-header-table"
65
+
fullWidth={false}
61
66
content={{
62
67
figure: "Context required for a debuggable DLQ.",
63
68
thead: { th: ["What You Need for Debugging", "Why It Matters"] },
@@ -122,14 +127,14 @@ For teams using Kafka, ecosystem tooling bridges some of this gap. Kpow and Cond
122
127
123
128
Database-backed storage trades native queue features for query capability - and for most teams, it's a worthwhile trade. With DLQ messages in PostgreSQL or MongoDB, debugging becomes straightforward. Find all messages that failed with a specific error type. Filter by source queue, time range, or failure classification. Join against your application tables to enrich the view. Build operational dashboards that show failure trends over time.
124
129
125
-
Several libraries make database-backed DLQ storage easy to implement. _pg-boss_ is a Node.js job queue backed by PostgreSQL with built-in retry policies and dead-lettering - failed jobs stay in Postgres tables where you can query them directly. _pgmq_ is a lightweight message queue native to Postgres supporting DLQ management. _River_ provides Go applications with Postgres-backed message persistence and sophisticated redrive policies. _Wolverine_ gives .NET teams PostgreSQL-backed messaging with integrated DLQ handling.
126
-
127
-
The decision tree is relatively simple. If you're running Kafka at scale, evaluate the ecosystem tools - they'll save weeks of custom development. If you're using Postgres and a supported language, the database-backed libraries give you queryable DLQ storage essentially for free. If your needs include custom enrichment, schema validation before replay, or conditional routing based on failure type, you'll end up building a dedicated DLQ service regardless of storage choice.
130
+
Several libraries make database-backed DLQ storage easy to implement. `pg-boss` is a Node.js job queue backed by PostgreSQL with built-in retry policies and dead-lettering - failed jobs stay in Postgres tables where you can query them directly. `pgmq` is a lightweight message queue native to Postgres supporting DLQ management. `River` provides Go applications with Postgres-backed message persistence and sophisticated redrive policies. `Wolverine` gives .NET teams PostgreSQL-backed messaging with integrated DLQ handling.
128
131
129
132
<Callouttype="warning">
130
133
Queue-based DLQ storage optimizes for simplicity at the cost of visibility. Database-backed storage optimizes for understanding at the cost of complexity. For anything beyond low-volume, simple workflows, the visibility is worth it.
131
134
</Callout>
132
135
136
+
The decision tree is relatively simple. If you're running Kafka at scale, evaluate the ecosystem tools - they'll save weeks of custom development. If you're using Postgres and a supported language, the database-backed libraries give you queryable DLQ storage essentially for free. If your needs include custom enrichment, schema validation before replay, or conditional routing based on failure type, you'll end up building a dedicated DLQ service regardless of storage choice.
137
+
133
138
## The Payoff
134
139
135
140
With these three fixes - classification, context capture, queryable storage - the opening scenario plays out differently. The team queries the DLQ and finds that 80% of the 50,000 messages are `transient_infrastructure` failures from a two-week period when the downstream payment processor had intermittent issues. Those can be bulk-replayed now that the processor is healthy.
Copy file name to clipboardExpand all lines: src/content/articles/dead-letter-queue-design-replay-debugging/pdf.mdx
+44-19Lines changed: 44 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -39,7 +39,10 @@ A DLQ that you can't debug and can't replay isn't a safety mechanism — it's a
39
39
Messages fail for different reasons, and those reasons determine how you should handle them. Understanding the failure taxonomy is the first step toward building a DLQ you can actually drain.
40
40
41
41
<List
42
-
variant="check-icons-list"
42
+
variant="chevron-list"
43
+
classes={{
44
+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
45
+
}}
43
46
items={[
44
47
{
45
48
lead: "Transient failures are temporary issues that resolve themselves:",
@@ -62,6 +65,7 @@ Messages fail for different reasons, and those reasons determine how you should
62
65
63
66
<Diagram
64
67
src={messageProcessingDiagram}
68
+
size="full"
65
69
title="Message processing flow with DLQ routing by failure type."
66
70
description="Flowchart showing how message processing routes successes and failures. A message starts at Message Received and moves to a Processing Attempt decision. Success leads to Acknowledge. Failure goes to a Retry Policy decision. If retries remain, the message is sent to Requeue with Backoff and then returns to Processing Attempt. If maximum retries are exceeded, the message moves to DLQ. From there, a Failure Type decision routes the message by classification: Transient goes to Auto-Replay Queue, Permanent goes to Manual Review Queue, and Poison goes to Quarantine Queue. The diagram shows that the DLQ is not a single sink for every failure, but a routing point where failure type determines the next operational action."
67
71
/>
@@ -139,7 +143,7 @@ The schema you choose for your DLQ messages determines what you can do with them
@@ -311,7 +318,7 @@ Using a database for DLQ storage trades simplicity for capability. You lose nati
311
318
The schema only matters if you populate it correctly, and the window for capturing context is narrow. Once a message moves to the DLQ, the processing context — trace spans, in-memory state, the specific consumer instance — is gone. You can't reconstruct it later. This means the DLQ write path needs to be a core part of your consumer implementation, not an afterthought.
312
319
313
320
<List
314
-
variant="check-icons-list"
321
+
variant="numbered-with-background-list"
315
322
items={[
316
323
{
317
324
lead: "Enrich synchronously, not asynchronously.",
@@ -343,7 +350,10 @@ The tooling landscape for DLQ management splits into three categories: broker-na
343
350
If you're keeping dead letters in the broker itself (SQS DLQs, RabbitMQ dead letter exchanges, Kafka error topics), several tools provide inspection and redrive capabilities without additional infrastructure.
344
351
345
352
<List
346
-
variant="check-icons-list"
353
+
variant="timeline-list"
354
+
classes={{
355
+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
356
+
}}
347
357
items={[
348
358
{
349
359
lead: "Kafka ecosystems",
@@ -367,7 +377,10 @@ The limitation of broker-native tools is query capability. You can't easily answ
367
377
Moving failed messages to a database unlocks SQL queries, which changes the operational model entirely. Several libraries make this straightforward.
368
378
369
379
<List
370
-
variant="check-icons-list"
380
+
variant="timeline-list"
381
+
classes={{
382
+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
383
+
}}
371
384
items={[
372
385
{
373
386
lead: "pg-boss",
@@ -397,7 +410,10 @@ When your messages use structured schemas (Protobuf, Avro, JSON Schema), inspect
397
410
The challenge is that most DLQ tooling treats messages as opaque blobs. You can see the bytes, but understanding what they mean requires external knowledge of your schema versions. For teams with strict schema evolution policies, this becomes a significant operational burden — you might have messages in the DLQ that were written against three different schema versions, and each requires different handling.
398
411
399
412
<List
400
-
variant="check-icons-list"
413
+
variant="timeline-list"
414
+
classes={{
415
+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
416
+
}}
401
417
items={[
402
418
{
403
419
lead: "Conduktor and Kpow_(mentioned earlier for Kafka)",
@@ -419,7 +435,10 @@ For complex needs — schema validation, enrichment from external services, cond
419
435
This is where the Python worker from the schema design section fits in. That script runs as a Lambda function triggered by new DLQ entries, or as a containerized worker polling your database. The deployment model depends on volume and latency requirements:
420
436
421
437
<List
422
-
variant="check-icons-list"
438
+
variant="timeline-list"
439
+
classes={{
440
+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
441
+
}}
423
442
items={[
424
443
{
425
444
lead: "Serverless (AWS Lambda, Azure Functions)",
@@ -457,7 +476,7 @@ The triage process starts with classification. If your consumer classified the f
457
476
For messages that weren't classified — either because the consumer didn't capture enough context or because the failure mode was unexpected — you need a manual triage step. Someone looks at the error message, checks the payload, and decides: is this fixable, and if so, how?
458
477
459
478
<Table
460
-
variant="vertical-column-delineation-table"
479
+
variant="numbered-review-table"
461
480
content={{
462
481
figure: "DLQ triage routing by failure classification.",
@@ -509,7 +528,10 @@ The fix itself falls into three categories. _Data fixes_ mean correcting the pay
509
528
When you get paged for a DLQ alert, resist the urge to immediately start replaying messages. The first few minutes should be assessment, not action.
510
529
511
530
<List
512
-
variant="check-icons-list"
531
+
variant="zebra-list"
532
+
classes={{
533
+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
534
+
}}
513
535
items={[
514
536
{
515
537
lead: "Check the trend, not just the number.",
@@ -535,11 +557,11 @@ The escalation triggers should be automatic, not judgment calls. Page the respon
535
557
The metrics you track should answer operational questions, not just populate dashboards. Every metric should have an associated action: if this number goes up (or down), what do I do?
536
558
537
559
<List
538
-
variant="check-icons-list"
560
+
variant="chevron-list"
539
561
items={[
540
562
{
541
563
lead: "DLQ depth by classification",
542
-
text: "Is your primary operational metric. Track it per queue and per failure type, because \"500 messages in the DLQ\" means different things depending on composition. 500 transient failures after a brief outage will auto-resolve. 500 validation failures means something is generating bad data upstream. Alert when depth exceeds thresholds, but set different thresholds for different classifications — transient failures can accumulate briefly without concern, but poison messages should alert immediately.",
564
+
text: "Your primary operational metric. Track it per queue and per failure type, because \"500 messages in the DLQ\" means different things depending on composition. 500 transient failures after a brief outage will auto-resolve. 500 validation failures means something is generating bad data upstream. Alert when depth exceeds thresholds, but set different thresholds for different classifications — transient failures can accumulate briefly without concern, but poison messages should alert immediately.",
543
565
},
544
566
{
545
567
lead: "Inflow rate",
@@ -557,7 +579,10 @@ The metrics you track should answer operational questions, not just populate das
@@ -624,8 +649,8 @@ The investment required isn't enormous. Enrich messages at failure time with the
624
649
625
650
Most of this work is one-time infrastructure that pays dividends every time something goes wrong. The alternative — deleting 50,000 messages because nobody knows what they are or whether they're safe to replay — is a data loss event dressed up as operational hygiene.
626
651
627
-
Start with the basics: capture full context at write time, classify failures so transient issues auto-resolve, and give your team tools to investigate permanent failures. You can add sophistication later — multi-tier DLQs, automated enrichment, schema validation on replay — but the fundamentals matter more than the advanced features.
628
-
629
652
<Callouttype="success">
630
653
The measure of a good DLQ system isn't how few messages end up there — it's how quickly you can understand why they failed and how confidently you can replay them. Design for debugging and recovery, not just storage.
631
654
</Callout>
655
+
656
+
Start with the basics: capture full context at write time, classify failures so transient issues auto-resolve, and give your team tools to investigate permanent failures. You can add sophistication later — multi-tier DLQs, automated enrichment, schema validation on replay — but the fundamentals matter more than the advanced features.
0 commit comments