Skip to content

Commit 9c1155a

Browse files
committed
List and Table updates to dead-letter-queue-design-replay-debugging article
1 parent 0f74dbb commit 9c1155a

3 files changed

Lines changed: 64 additions & 31 deletions

File tree

‎_TODO.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -325,4 +325,7 @@ src/content/articles/internal-developer-portal-platform-self-service-actions/pdf
325325
classes={{
326326
figure: "sm:ml-8 sm:mr-12 mb-6",
327327
}}
328+
classes={{
329+
tbody: '!text-content',
330+
}}
328331
/>

‎src/content/articles/dead-letter-queue-design-replay-debugging/index.mdx‎

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -22,42 +22,47 @@ The problem isn't the DLQ concept - it's three specific design failures that mak
2222

2323
## Classify Failures at Write Time
2424

25-
Messages fail for different reasons, and those reasons determine how you should handle them. Most DLQs treat all failures the same - a message failed, it goes in the queue, someone will look at it eventually. That "eventually" often becomes "never" because manual triage doesn't scale.
25+
Messages fail for different reasons, and those reasons determine how you should handle them. <Highlighter>Most DLQs treat all failures the same - a message failed, it goes in the queue, someone will look at it eventually.</Highlighter> That "eventually" often becomes "never" because manual triage doesn't scale.
2626

2727
The failure taxonomy matters because different failure types require fundamentally different responses:
2828

2929
<List
30-
variant="check-icons-list"
30+
variant="zebra-list"
3131
items={[
3232
{
33-
text: "Transient failures are temporary issues that resolve themselves: database connection timeouts, downstream service unavailability, rate limit errors, lock contention. These should succeed on retry. The right strategy is automatic replay after a cooldown period - no human intervention required if your retry policy is sound.",
33+
lead: "Transient failures",
34+
text: "These are temporary issues that resolve themselves: database connection timeouts, downstream service unavailability, rate limit errors, lock contention. These should succeed on retry. The right strategy is automatic replay after a cooldown period - no human intervention required if your retry policy is sound.",
3435
},
3536
{
36-
text: "Permanent failures won't resolve without intervention: invalid message format, referenced entity doesn't exist, business rule violation, schema version mismatch. These will never succeed without a fix. Someone needs to look at the payload, understand what's wrong, and either fix the data or discard the message.",
37+
lead: "Permanent failures",
38+
text: "These won't resolve without intervention: invalid message format, referenced entity doesn't exist, business rule violation, schema version mismatch. These will never succeed without a fix. Someone needs to look at the payload, understand what's wrong, and either fix the data or discard the message.",
3739
},
3840
{
39-
text: "Poison messages crash the consumer: null pointer in a required field, infinite loop triggered by specific data, memory exhaustion from an oversized payload. The consumer dies on every attempt. These are the most dangerous because they can take down your processing capacity. The strategy is to isolate immediately, fix the consumer code, then replay.",
41+
lead: "Poison messages",
42+
text: "These crash the consumer: null pointer in a required field, infinite loop triggered by specific data, memory exhaustion from an oversized payload. The consumer dies on every attempt. These are the most dangerous because they can take down your processing capacity. The strategy is to isolate immediately, fix the consumer code, then replay.",
4043
},
4144
{
42-
text: "Ordering failures fail due to sequence issues: an update arriving before the create, a delete for a non-existent record. These may succeed if replayed in correct order. The strategy is to hold for reordering, then replay - but this often requires understanding the business context to determine the correct sequence.",
45+
lead: "Poison messages",
46+
text: "These fail due to sequence issues: an update arriving before the create, a delete for a non-existent record. These may succeed if replayed in correct order. The strategy is to hold for reordering, then replay - but this often requires understanding the business context to determine the correct sequence.",
4347
},
4448
]}
4549
/>
4650

4751
When you classify failures at write time - the moment the message moves to the DLQ - you enable automation. Transient failures route to an auto-replay queue with a delay. Permanent failures route to a team-specific triage queue. Poison messages trigger an immediate alert. Without classification, every message requires a human to open it, read the error, and decide what to do. That's why DLQs accumulate for months.
4852

49-
The classification should be explicit in your DLQ schema, not inferred later. Your consumer already knows why it failed - a `ValidationError` is permanent, a `ConnectionError` is transient, an unhandled exception might be poison. Map exception types to failure classifications programmatically, and use that classification to drive routing and alerting.
50-
5153
<Callout type="info">
5254
If your DLQ treats all failures the same, you're forcing humans to do what software should do. Classification is the foundation that makes everything else - auto-replay, targeted alerts, efficient triage - possible.
5355
</Callout>
5456

57+
The classification should be explicit in your DLQ schema, not inferred later. Your consumer already knows why it failed - a `ValidationError` is permanent, a `ConnectionError` is transient, an unhandled exception might be poison. Map exception types to failure classifications programmatically, and use that classification to drive routing and alerting.
58+
5559
## Capture Context Before It's Gone
5660

5761
Traditional DLQs fail at their job because they don't capture enough context. Most give you the original message payload, a timestamp when the message was DLQ'd, and maybe the number of retries. That's it. When you're debugging a failure three weeks later, you're staring at a payload with no idea what went wrong.
5862

5963
<Table
60-
variant="vertical-column-delineation-table"
64+
variant="grid-and-accent-header-table"
65+
fullWidth={false}
6166
content={{
6267
figure: "Context required for a debuggable DLQ.",
6368
thead: { th: ["What You Need for Debugging", "Why It Matters"] },
@@ -122,14 +127,14 @@ For teams using Kafka, ecosystem tooling bridges some of this gap. Kpow and Cond
122127

123128
Database-backed storage trades native queue features for query capability - and for most teams, it's a worthwhile trade. With DLQ messages in PostgreSQL or MongoDB, debugging becomes straightforward. Find all messages that failed with a specific error type. Filter by source queue, time range, or failure classification. Join against your application tables to enrich the view. Build operational dashboards that show failure trends over time.
124129

125-
Several libraries make database-backed DLQ storage easy to implement. _pg-boss_ is a Node.js job queue backed by PostgreSQL with built-in retry policies and dead-lettering - failed jobs stay in Postgres tables where you can query them directly. _pgmq_ is a lightweight message queue native to Postgres supporting DLQ management. _River_ provides Go applications with Postgres-backed message persistence and sophisticated redrive policies. _Wolverine_ gives .NET teams PostgreSQL-backed messaging with integrated DLQ handling.
126-
127-
The decision tree is relatively simple. If you're running Kafka at scale, evaluate the ecosystem tools - they'll save weeks of custom development. If you're using Postgres and a supported language, the database-backed libraries give you queryable DLQ storage essentially for free. If your needs include custom enrichment, schema validation before replay, or conditional routing based on failure type, you'll end up building a dedicated DLQ service regardless of storage choice.
130+
Several libraries make database-backed DLQ storage easy to implement. `pg-boss` is a Node.js job queue backed by PostgreSQL with built-in retry policies and dead-lettering - failed jobs stay in Postgres tables where you can query them directly. `pgmq` is a lightweight message queue native to Postgres supporting DLQ management. `River` provides Go applications with Postgres-backed message persistence and sophisticated redrive policies. `Wolverine` gives .NET teams PostgreSQL-backed messaging with integrated DLQ handling.
128131

129132
<Callout type="warning">
130133
Queue-based DLQ storage optimizes for simplicity at the cost of visibility. Database-backed storage optimizes for understanding at the cost of complexity. For anything beyond low-volume, simple workflows, the visibility is worth it.
131134
</Callout>
132135

136+
The decision tree is relatively simple. If you're running Kafka at scale, evaluate the ecosystem tools - they'll save weeks of custom development. If you're using Postgres and a supported language, the database-backed libraries give you queryable DLQ storage essentially for free. If your needs include custom enrichment, schema validation before replay, or conditional routing based on failure type, you'll end up building a dedicated DLQ service regardless of storage choice.
137+
133138
## The Payoff
134139

135140
With these three fixes - classification, context capture, queryable storage - the opening scenario plays out differently. The team queries the DLQ and finds that 80% of the 50,000 messages are `transient_infrastructure` failures from a two-week period when the downstream payment processor had intermittent issues. Those can be bulk-replayed now that the processor is healthy.

‎src/content/articles/dead-letter-queue-design-replay-debugging/pdf.mdx‎

Lines changed: 44 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,10 @@ A DLQ that you can't debug and can't replay isn't a safety mechanism — it's a
3939
Messages fail for different reasons, and those reasons determine how you should handle them. Understanding the failure taxonomy is the first step toward building a DLQ you can actually drain.
4040

4141
<List
42-
variant="check-icons-list"
42+
variant="chevron-list"
43+
classes={{
44+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
45+
}}
4346
items={[
4447
{
4548
lead: "Transient failures are temporary issues that resolve themselves:",
@@ -62,6 +65,7 @@ Messages fail for different reasons, and those reasons determine how you should
6265

6366
<Diagram
6467
src={messageProcessingDiagram}
68+
size="full"
6569
title="Message processing flow with DLQ routing by failure type."
6670
description="Flowchart showing how message processing routes successes and failures. A message starts at Message Received and moves to a Processing Attempt decision. Success leads to Acknowledge. Failure goes to a Retry Policy decision. If retries remain, the message is sent to Requeue with Backoff and then returns to Processing Attempt. If maximum retries are exceeded, the message moves to DLQ. From there, a Failure Type decision routes the message by classification: Transient goes to Auto-Replay Queue, Permanent goes to Manual Review Queue, and Poison goes to Quarantine Queue. The diagram shows that the DLQ is not a single sink for every failure, but a routing point where failure type determines the next operational action."
6771
/>
@@ -139,7 +143,7 @@ The schema you choose for your DLQ messages determines what you can do with them
139143
### Design Principles
140144

141145
<List
142-
variant="check-icons-list"
146+
variant="zebra-list"
143147
items={[
144148
{
145149
lead: "Preserve the original message exactly.",
@@ -257,38 +261,41 @@ Code: Background worker that writes DLQ envelopes.
257261
Where you store your DLQ messages affects what operations you can perform on them:
258262

259263
<Table
260-
variant="vertical-column-delineation-table"
264+
variant="timeline-labels-table"
265+
classes={{
266+
tbody: '!text-content',
267+
}}
261268
content={{
262269
figure: "DLQ storage options comparison.",
263270
thead: { th: ["Storage Type", "Pros", "Cons", "Best For"] },
264271
tbody: {
265272
tr: [
266273
{
267-
th: "Same queue system (SQS, RabbitMQ)",
274+
th: { label: "Same queue system (SQS, RabbitMQ)", color: "secondary" },
268275
td: [
269276
"Simple setup, native tooling",
270277
"Limited querying, no analytics",
271278
"Small volume, simple workflows",
272279
],
273280
},
274281
{
275-
th: "Database (PostgreSQL)",
282+
th: { label: "Database (PostgreSQL)", color: "success" },
276283
td: [
277284
"Full querying, transactions",
278285
"Need to build tooling",
279286
"Complex workflows, audit requirements",
280287
],
281288
},
282289
{
283-
th: "Document store (MongoDB)",
290+
th: { label: "Document store (MongoDB)", color: "info" },
284291
td: [
285292
"Flexible schema, good querying",
286293
"Another system to manage",
287294
"High volume, varied message types",
288295
],
289296
},
290297
{
291-
th: "Time-series (Elasticsearch)",
298+
th: { label: "Time-series (Elasticsearch)", color: "warning" },
292299
td: [
293300
"Great analytics, dashboards",
294301
"Overkill for small volumes",
@@ -311,7 +318,7 @@ Using a database for DLQ storage trades simplicity for capability. You lose nati
311318
The schema only matters if you populate it correctly, and the window for capturing context is narrow. Once a message moves to the DLQ, the processing context — trace spans, in-memory state, the specific consumer instance — is gone. You can't reconstruct it later. This means the DLQ write path needs to be a core part of your consumer implementation, not an afterthought.
312319

313320
<List
314-
variant="check-icons-list"
321+
variant="numbered-with-background-list"
315322
items={[
316323
{
317324
lead: "Enrich synchronously, not asynchronously.",
@@ -343,7 +350,10 @@ The tooling landscape for DLQ management splits into three categories: broker-na
343350
If you're keeping dead letters in the broker itself (SQS DLQs, RabbitMQ dead letter exchanges, Kafka error topics), several tools provide inspection and redrive capabilities without additional infrastructure.
344351

345352
<List
346-
variant="check-icons-list"
353+
variant="timeline-list"
354+
classes={{
355+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
356+
}}
347357
items={[
348358
{
349359
lead: "Kafka ecosystems",
@@ -367,7 +377,10 @@ The limitation of broker-native tools is query capability. You can't easily answ
367377
Moving failed messages to a database unlocks SQL queries, which changes the operational model entirely. Several libraries make this straightforward.
368378

369379
<List
370-
variant="check-icons-list"
380+
variant="timeline-list"
381+
classes={{
382+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
383+
}}
371384
items={[
372385
{
373386
lead: "pg-boss",
@@ -397,7 +410,10 @@ When your messages use structured schemas (Protobuf, Avro, JSON Schema), inspect
397410
The challenge is that most DLQ tooling treats messages as opaque blobs. You can see the bytes, but understanding what they mean requires external knowledge of your schema versions. For teams with strict schema evolution policies, this becomes a significant operational burden — you might have messages in the DLQ that were written against three different schema versions, and each requires different handling.
398411

399412
<List
400-
variant="check-icons-list"
413+
variant="timeline-list"
414+
classes={{
415+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
416+
}}
401417
items={[
402418
{
403419
lead: "Conduktor and Kpow_(mentioned earlier for Kafka)",
@@ -419,7 +435,10 @@ For complex needs — schema validation, enrichment from external services, cond
419435
This is where the Python worker from the schema design section fits in. That script runs as a Lambda function triggered by new DLQ entries, or as a containerized worker polling your database. The deployment model depends on volume and latency requirements:
420436

421437
<List
422-
variant="check-icons-list"
438+
variant="timeline-list"
439+
classes={{
440+
wrapper: "sm:ml-6 sm:mr-18 mb-6",
441+
}}
423442
items={[
424443
{
425444
lead: "Serverless (AWS Lambda, Azure Functions)",
@@ -457,7 +476,7 @@ The triage process starts with classification. If your consumer classified the f
457476
For messages that weren't classified — either because the consumer didn't capture enough context or because the failure mode was unexpected — you need a manual triage step. Someone looks at the error message, checks the payload, and decides: is this fixable, and if so, how?
458477

459478
<Table
460-
variant="vertical-column-delineation-table"
479+
variant="numbered-review-table"
461480
content={{
462481
figure: "DLQ triage routing by failure classification.",
463482
thead: { th: ["Classification", "Auto-Handling", "Human Action Required"] },
@@ -509,7 +528,10 @@ The fix itself falls into three categories. _Data fixes_ mean correcting the pay
509528
When you get paged for a DLQ alert, resist the urge to immediately start replaying messages. The first few minutes should be assessment, not action.
510529

511530
<List
512-
variant="check-icons-list"
531+
variant="zebra-list"
532+
classes={{
533+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
534+
}}
513535
items={[
514536
{
515537
lead: "Check the trend, not just the number.",
@@ -535,11 +557,11 @@ The escalation triggers should be automatic, not judgment calls. Page the respon
535557
The metrics you track should answer operational questions, not just populate dashboards. Every metric should have an associated action: if this number goes up (or down), what do I do?
536558

537559
<List
538-
variant="check-icons-list"
560+
variant="chevron-list"
539561
items={[
540562
{
541563
lead: "DLQ depth by classification",
542-
text: "Is your primary operational metric. Track it per queue and per failure type, because \"500 messages in the DLQ\" means different things depending on composition. 500 transient failures after a brief outage will auto-resolve. 500 validation failures means something is generating bad data upstream. Alert when depth exceeds thresholds, but set different thresholds for different classifications — transient failures can accumulate briefly without concern, but poison messages should alert immediately.",
564+
text: "Your primary operational metric. Track it per queue and per failure type, because \"500 messages in the DLQ\" means different things depending on composition. 500 transient failures after a brief outage will auto-resolve. 500 validation failures means something is generating bad data upstream. Alert when depth exceeds thresholds, but set different thresholds for different classifications — transient failures can accumulate briefly without concern, but poison messages should alert immediately.",
543565
},
544566
{
545567
lead: "Inflow rate",
@@ -557,7 +579,10 @@ The metrics you track should answer operational questions, not just populate das
557579
/>
558580

559581
<Table
560-
variant="vertical-column-delineation-table"
582+
variant="grid-and-accent-header-table"
583+
classes={{
584+
tbody: '!text-content',
585+
}}
561586
content={{
562587
figure: "DLQ alerting thresholds and responses.",
563588
thead: { th: ["Metric", "Warning Threshold", "Critical Threshold", "Response"] },
@@ -624,8 +649,8 @@ The investment required isn't enormous. Enrich messages at failure time with the
624649

625650
Most of this work is one-time infrastructure that pays dividends every time something goes wrong. The alternative — deleting 50,000 messages because nobody knows what they are or whether they're safe to replay — is a data loss event dressed up as operational hygiene.
626651

627-
Start with the basics: capture full context at write time, classify failures so transient issues auto-resolve, and give your team tools to investigate permanent failures. You can add sophistication later — multi-tier DLQs, automated enrichment, schema validation on replay — but the fundamentals matter more than the advanced features.
628-
629652
<Callout type="success">
630653
The measure of a good DLQ system isn't how few messages end up there — it's how quickly you can understand why they failed and how confidently you can replay them. Design for debugging and recovery, not just storage.
631654
</Callout>
655+
656+
Start with the basics: capture full context at write time, classify failures so transient issues auto-resolve, and give your team tools to investigate permanent failures. You can add sophistication later — multi-tier DLQs, automated enrichment, schema validation on replay — but the fundamentals matter more than the advanced features.

0 commit comments

Comments
 (0)