Skip to content

Commit 3f474c0

Browse files
committed
Final pass idempotent-message-handlers-deduplication-retries article
1 parent 30bb187 commit 3f474c0

2 files changed

Lines changed: 65 additions & 47 deletions

File tree

  • src/content/articles/idempotent-message-handlers-deduplication-retries

‎src/content/articles/idempotent-message-handlers-deduplication-retries/index.mdx‎

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -12,16 +12,16 @@ featured: true
1212
import idempotencyFlowDiagram from "./diagrams/idempotency-flow-second-delivery-skips-processing-entirely.jpg"
1313

1414
*[ACK]: Acknowledgment
15-
*[UUID]: Universally Unique Identifier
16-
*[ULID]: Universally Unique Lexicographically Sortable Identifier
17-
*[TTL]: Time To Live
1815
*[DLQ]: Dead Letter Queue
16+
*[TTL]: Time To Live
17+
*[ULID]: Universally Unique Lexicographically Sortable Identifier
18+
*[UUID]: Universally Unique Identifier
1919

2020
Your payment service receives a charge message, successfully charges the customer's card, and then the process crashes before it can acknowledge the message back to the queue. The queue sees no acknowledgment, assumes failure, and redelivers. Without idempotency, that's a double charge. With a properly designed idempotent handler, the second delivery is recognized as a duplicate and returns success without touching the payment gateway again.
2121

2222
This isn't a hypothetical edge case. I've seen it happen in production - payment services that charged customers twice, inventory systems where stock counts drifted negative from duplicate decrements, and welcome emails that arrived in triplicate. The common thread? Handlers that assumed each message would arrive exactly once.
2323

24-
Every production queue operates on at-least-once delivery - "probably more than once." The question isn't __if__ duplicates arrive, it's when.
24+
Every production queue operates on at-least-once delivery - "_probably more than once_." The question isn't _if_ duplicates arrive, it's when.
2525

2626
This article covers the two foundational patterns for building idempotent handlers: designing idempotency keys that reliably identify duplicates, and choosing the right storage strategy for your deduplication state.
2727

@@ -32,23 +32,26 @@ The idempotency key is the identifier your handler uses to recognize duplicate m
3232
There are four common strategies, each with distinct tradeoffs:
3333

3434
<List
35-
variant="check-icons-list"
35+
variant="chevron-list"
36+
classes={{
37+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
38+
}}
3639
items={[
3740
{
3841
lead: "Producer-supplied message ID",
39-
text: "is the cleanest approach. The producer generates a UUID or ULID when creating the message and includes it in the payload. This key survives redeliveries because it's part of the message content, not queue metadata. The downside is that it requires producer discipline - every producer must generate and include a unique ID.",
42+
text: "The cleanest approach. The producer generates a UUID or ULID when creating the message and includes it in the payload. This key survives redeliveries because it's part of the message content, not queue metadata. The downside is that it requires producer discipline - every producer must generate and include a unique ID.",
4043
},
4144
{
4245
lead: "Content hash",
43-
text: "works when you can't control producers. Hash the message payload with SHA-256 and use that as the key. Identical content produces identical keys automatically. The risk is that sometimes identical content should be processed multiple times (two separate orders for the same product), and a content hash would incorrectly dedupe them.",
46+
text: "Works when you can't control producers. Hash the message payload with SHA-256 and use that as the key. Identical content produces identical keys automatically. The risk is that sometimes identical content should be processed multiple times (two separate orders for the same product), and a content hash would incorrectly dedupe them.",
4447
},
4548
{
4649
lead: "Business key composite",
47-
text: "combines entity identifiers with operation context: `order:12345:payment:v3`. This approach has semantic meaning, making debugging easier, but requires careful thought about what constitutes a unique operation.",
50+
text: 'Combines entity identifiers with operation context: <span class="font-mono">order:12345:payment:v3</span>. This approach has semantic meaning, making debugging easier, but requires careful thought about what constitutes a unique operation.',
4851
},
4952
{
5053
lead: "Queue message ID",
51-
text: "- avoid this approach. SQS assigns a new MessageId on each redelivery. RabbitMQ's delivery tag changes. This approach only deduplicates within a single delivery attempt, which defeats the purpose entirely.",
54+
text: 'Avoid this approach. SQS assigns a new <span class="font-mono">MessageId</span> on each redelivery. RabbitMQ\'s delivery tag changes. This approach only deduplicates within a single delivery attempt, which defeats the purpose entirely.',
5255
},
5356
]}
5457
/>
@@ -74,10 +77,10 @@ Once you have a stable idempotency key, you need somewhere to store the record o
7477

7578
A dedicated idempotency store sits between your handler and your business logic. Before processing, the handler checks the store. If the key exists and shows "completed," return the cached result. If the key exists and shows "processing," another instance is handling it - either wait or fail fast. If the key doesn't exist, acquire a lock and proceed.
7679

77-
Redis works well for high-throughput scenarios where you need sub-millisecond latency. The critical detail is the `NX` flag on the SET command - it only sets the key if it doesn't already exist, making the check-and-lock operation atomic.
78-
7980
<Newsletter />
8081

82+
Redis works well for high-throughput scenarios where you need sub-millisecond latency. The critical detail is the `NX` flag on the SET command - it only sets the key if it doesn't already exist, making the check-and-lock operation atomic.
83+
8184
### Database-Level Deduplication
8285

8386
For financial operations or anywhere you need transactional consistency, database-level deduplication is often better than a separate store. The idempotency key becomes a unique constraint on your business table.
@@ -112,14 +115,16 @@ The sequence diagram below shows the full flow. Notice that on the second delive
112115

113116
<Diagram
114117
src={idempotencyFlowDiagram}
118+
size="full"
115119
title="Idempotency flow - second delivery skips processing entirely."
116120
description="Sequence diagram showing how an idempotent message handler processes a first delivery and short-circuits a duplicate. A queue delivers a message to the handler, which checks the idempotency store for key order:123:pay. The store reports the key is not found and grants the lock. The handler processes the payment through the database, receives success, marks the idempotency key as completed with a cached result, and acknowledges the message back to the queue. Later, after a note indicating the consumer crashed and the message was redelivered, the queue sends the same message again. The handler checks the same idempotency key, the store reports it is already completed with a cached result, and the handler immediately acknowledges the message without processing the payment again."
117121
/>
118122

119123
Which storage should you use? It depends on your consistency requirements and latency budget:
120124

121125
<Table
122-
variant="vertical-column-delineation-table"
126+
variant="criteria-highlight-table"
127+
fullWidth={false}
123128
content={{
124129
figure: "Idempotency store options comparison.",
125130
thead: {

0 commit comments

Comments
 (0)