[Bug] ACP checkout can be retried after an ambiguous merchant response

[SergeevDmitry]

Context & Rationale

If the merchant creates an order and the response then times out or fails schema validation, the client retries and we may ask the merchant to create it again. ACP has durable idempotency, but dispatch() releases the claim on a 5xx, in-flight rows are deleted once expires_at passes regardless of state, and BackendRequest carries no idempotency key for the HTTP executor to forward. The merchant has no way to recognise the retry as the same operation.

Version: 1.4.0, commit a05a08f.

Steps to reproduce (planned): run the ACP adapter against a merchant fixture whose complete endpoint records the order, sleeps past our timeout, then answers 200; send complete, let it time out, send it again with the same body; count orders on the fixture.

Expected: one order; the second call replays the first result or is refused as "outcome unknown".
Actual (from source): two orders.

Proposed Changes

• Add an optional idempotency key to BackendRequest and forward it to the merchant through a configured header or body field.
• Derive one operation key per ACP side-effecting operation that survives client retries, restart and later recovery, rather than being tied to the per-call requestId.
• Keep an ambiguous operation unresolved instead of releasing its claim on 5xx, timeout or malformed response.
• Document what the merchant must implement (idempotent handling of the key plus a status lookup) and which guarantees weaken when they do not.

Non-goals / Invariants Preserved

• A 5xx does not prove the merchant did nothing.
• Client-supplied headers are not trusted as idempotency data.
• Checkout state and payment stay with the merchant; no gateway-side checkout database.

Impact & Value

Done when the merchant recognises a repeated side-effecting operation as the same one across client retry, network retry, gateway restart, and a timeout that arrives after the merchant has already acted, so no duplicate order can result. ACP clients keep their existing retry behaviour.

Open questions

• Header or body field for the key? The ACP spec reserves neither as far as I can tell. Config option, or pick one and document it?
• Should retention expiry of an in-flight row be allowed at all, or should unresolved rows clear only through operator reconciliation (the transaction-recovery tracking issue)?

Test plan

Merchant success then timeout / connection loss / malformed body → single order, safe retry or explicit unresolved. Concurrent duplicates → one dispatch. Restart between attempt and retry → key survives. Bearer-token rotation → key scope unchanged. Existing completed-response replay and body-conflict behaviour unchanged.

[Revinand]

Header or body field for the key? The ACP spec reserves neither as far as I can tell. Config option, or pick one and document it?

Header, fixed, not configurable: Idempotency-Key, the de-facto standard and the one ACP already mandates inbound. The value is a digest of (deployment, endpoint, caller key) rather than the caller's raw key, because ACP scopes that key per endpoint - one client may legitimately send the same key to create and to complete. A statically configured header of that name is overridden, since a fixed key makes every request look like a retry of the first.

Should retention expiry of an in-flight row be allowed at all, or should unresolved rows clear only through operator reconciliation (the transaction-recovery tracking issue)?

No, only completed rows do. Sweeping an unresolved row hands the next retry a clean key, which is the bug. The one exception is the v1 to v2 migration, which discards rows it cannot rekey and logs a count of them.