You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3f474c0
Browse filesBrowse the repository at this point in the historyBrowse files
Your payment service receives a charge message, successfully charges the customer's card, and then the process crashes before it can acknowledge the message back to the queue. The queue sees no acknowledgment, assumes failure, and redelivers. Without idempotency, that's a double charge. With a properly designed idempotent handler, the second delivery is recognized as a duplicate and returns success without touching the payment gateway again.
21
21
22
22
This isn't a hypothetical edge case. I've seen it happen in production - payment services that charged customers twice, inventory systems where stock counts drifted negative from duplicate decrements, and welcome emails that arrived in triplicate. The common thread? Handlers that assumed each message would arrive exactly once.
23
23
24
-
Every production queue operates on at-least-once delivery - "probably more than once." The question isn't __if__ duplicates arrive, it's when.
24
+
Every production queue operates on at-least-once delivery - "_probably more than once_." The question isn't _if_ duplicates arrive, it's when.
25
25
26
26
This article covers the two foundational patterns for building idempotent handlers: designing idempotency keys that reliably identify duplicates, and choosing the right storage strategy for your deduplication state.
27
27
@@ -32,23 +32,26 @@ The idempotency key is the identifier your handler uses to recognize duplicate m
32
32
There are four common strategies, each with distinct tradeoffs:
33
33
34
34
<List
35
-
variant="check-icons-list"
35
+
variant="chevron-list"
36
+
classes={{
37
+
wrapper: "sm:ml-6 sm:mr-12 mb-6",
38
+
}}
36
39
items={[
37
40
{
38
41
lead: "Producer-supplied message ID",
39
-
text: "is the cleanest approach. The producer generates a UUID or ULID when creating the message and includes it in the payload. This key survives redeliveries because it's part of the message content, not queue metadata. The downside is that it requires producer discipline - every producer must generate and include a unique ID.",
42
+
text: "The cleanest approach. The producer generates a UUID or ULID when creating the message and includes it in the payload. This key survives redeliveries because it's part of the message content, not queue metadata. The downside is that it requires producer discipline - every producer must generate and include a unique ID.",
40
43
},
41
44
{
42
45
lead: "Content hash",
43
-
text: "works when you can't control producers. Hash the message payload with SHA-256 and use that as the key. Identical content produces identical keys automatically. The risk is that sometimes identical content should be processed multiple times (two separate orders for the same product), and a content hash would incorrectly dedupe them.",
46
+
text: "Works when you can't control producers. Hash the message payload with SHA-256 and use that as the key. Identical content produces identical keys automatically. The risk is that sometimes identical content should be processed multiple times (two separate orders for the same product), and a content hash would incorrectly dedupe them.",
44
47
},
45
48
{
46
49
lead: "Business key composite",
47
-
text: "combines entity identifiers with operation context: `order:12345:payment:v3`. This approach has semantic meaning, making debugging easier, but requires careful thought about what constitutes a unique operation.",
50
+
text: 'Combines entity identifiers with operation context: <span class="font-mono">order:12345:payment:v3</span>. This approach has semantic meaning, making debugging easier, but requires careful thought about what constitutes a unique operation.',
48
51
},
49
52
{
50
53
lead: "Queue message ID",
51
-
text: "- avoid this approach. SQS assigns a new MessageId on each redelivery. RabbitMQ's delivery tag changes. This approach only deduplicates within a single delivery attempt, which defeats the purpose entirely.",
54
+
text: 'Avoid this approach. SQS assigns a new <span class="font-mono">MessageId</span> on each redelivery. RabbitMQ\'s delivery tag changes. This approach only deduplicates within a single delivery attempt, which defeats the purpose entirely.',
52
55
},
53
56
]}
54
57
/>
@@ -74,10 +77,10 @@ Once you have a stable idempotency key, you need somewhere to store the record o
74
77
75
78
A dedicated idempotency store sits between your handler and your business logic. Before processing, the handler checks the store. If the key exists and shows "completed," return the cached result. If the key exists and shows "processing," another instance is handling it - either wait or fail fast. If the key doesn't exist, acquire a lock and proceed.
76
79
77
-
Redis works well for high-throughput scenarios where you need sub-millisecond latency. The critical detail is the `NX` flag on the SET command - it only sets the key if it doesn't already exist, making the check-and-lock operation atomic.
78
-
79
80
<Newsletter />
80
81
82
+
Redis works well for high-throughput scenarios where you need sub-millisecond latency. The critical detail is the `NX` flag on the SET command - it only sets the key if it doesn't already exist, making the check-and-lock operation atomic.
83
+
81
84
### Database-Level Deduplication
82
85
83
86
For financial operations or anywhere you need transactional consistency, database-level deduplication is often better than a separate store. The idempotency key becomes a unique constraint on your business table.
@@ -112,14 +115,16 @@ The sequence diagram below shows the full flow. Notice that on the second delive
112
115
113
116
<Diagram
114
117
src={idempotencyFlowDiagram}
118
+
size="full"
115
119
title="Idempotency flow - second delivery skips processing entirely."
116
120
description="Sequence diagram showing how an idempotent message handler processes a first delivery and short-circuits a duplicate. A queue delivers a message to the handler, which checks the idempotency store for key order:123:pay. The store reports the key is not found and grants the lock. The handler processes the payment through the database, receives success, marks the idempotency key as completed with a cached result, and acknowledges the message back to the queue. Later, after a note indicating the consumer crashed and the message was redelivered, the queue sends the same message again. The handler checks the same idempotency key, the store reports it is already completed with a cached result, and the handler immediately acknowledges the message without processing the payment again."
117
121
/>
118
122
119
123
Which storage should you use? It depends on your consistency requirements and latency budget:
0 commit comments