Deliver generic signed transport callbacks and reconcile unmatched provider events

[alexeygrigorev]

External parent epic: DataTalksClub/website#21

Lifecycle prerequisite: #2
Template/version prerequisite: #1

Audited baseline: c0fdfac632f057f1e6a467111613928c4899b13b

Outcome

Replace the CMP-specific Bearer callback assumption with a generic tenant-scoped, redacted, versioned HMAC callback outbox and make late/unmatched provider events converge safely.

Every Relay transport transition needed by a client projection must create durable callback work after the state commit. Callback loss is recovered through bounded retry and the authoritative message-status API. Valid provider events that cannot yet be correlated are deduplicated, retained redacted for seven days, retried for correlation, and made operator-visible if unresolved.

Baseline gaps

At the audited commit:

• callbacks are represented by CmpCallback and configured through CMP-specific URL/token fields;
• callback delivery uses a static Bearer token rather than timestamped HMAC signing;
• payloads include full normalized email, contact identifiers, and merged message metadata;
• the event contract is not a generic versioned tenant transport contract;
• callback rows/delivery are tied to CMP naming and behavior;
• provider-event correlation does not provide the required seven-day unmatched-event recovery/operator visibility contract; and
• no website-safe callback/OpenAPI consumer fixture exists.

Scope

Generic callback endpoint configuration and secret boundary

• Introduce tenant/client-scoped callback configuration independent of CMP product naming.
• Use a separate callback signing secret from every API credential.
• Obtain the retrievable signing secret only through an approved server-side secret boundary/reference; never return it after initial authorized provisioning and never place it in URLs, ordinary database metadata, logs, errors, audit rows, docs examples, screenshots, or test artifacts.
• Support explicit callback enable/disable, endpoint URL, contract version, secret rotation with a bounded verification overlap, and immediate revocation/disablement.
• Validate the callback destination against an explicit client configuration boundary. Do not permit arbitrary per-message callback URLs or redirects to unapproved origins.
• Real secret provisioning and external endpoint configuration remain human/infrastructure work outside this issue; deterministic tests use non-secret fixtures.

Versioned redacted transport event

Define a canonical JSON event with stable serialization and explicit contract_version. It contains only:

• stable Relay event ID;
• tenant/client-safe association;
• event type and authoritative transport status from Guarantee idempotent transactional delivery and ambiguity-safe provider state #2;
• occurrence time and monotonic transition/version information;
• Relay message ID;
• client correlation/idempotency key;
• immutable template key/version;
• safe reason code where applicable; and
• optional redacted attempt/reconciliation summary required for convergence.

It must never contain:

• full recipient/email;
• subject, plain/HTML body, context, headers, attachments/message parts;
• API credential or callback secret;
• raw provider payload, provider diagnostic, or unrestricted metadata; or
• unrelated tenant/contact/campaign identifiers.

Persist the exact canonical body/hash used for retry so every attempt signs and sends the same logical event.

HMAC/timestamp delivery

• Sign <timestamp>.<raw-body> with HMAC-SHA-256 using the client callback secret.
• Send X-Relay-Timestamp, X-Relay-Signature, content type, event/contract identifiers, and no Bearer credential.
• Store callback work in the same database transaction as, or through a durable transactionally coupled outbox created by, the authoritative state transition. A committed transition cannot be silently missing callback work.
• Dispatch only after commit.
• Retry timeout, connection, and retryable HTTP failures with bounded exponential backoff and jitter policy that remains deterministic under tests.
• Treat duplicate successful acknowledgement as success and never create a second callback row for one client/event/contract.
• Record attempts, response status class, next attempt, terminal failure, and safe error code without response bodies or protected headers.
• Callback failure never rolls back or regresses authoritative Relay transport state.

Provider-event correlation and recovery

• Deduplicate provider events by tenant/provider/event ID before applying state.
• Validate and normalize provider ingress first; retain no unnecessary raw payload in the unmatched record.
• When a valid event cannot be correlated to a Relay message:
  • store a redacted unmatched-event record with stable identity, safe event class, provider correlation hash/reference, occurrence/receipt time, retry schedule, and expiration;
  • retry correlation with bounded backoff for seven days;
  • apply the event exactly once if correlation later succeeds;
  • preserve guarded/reorder-tolerant transitions from Guarantee idempotent transactional delivery and ambiguity-safe provider state #2 and never regress delivered/bounced/complained state; and
  • expose unresolved count/age/safe identifiers to operators and metrics without recipient/provider payload leakage.
• At seven days, move unresolved events to an operator-visible terminal state; do not silently delete or invent a message association.
• Duplicate/out-of-order/late provider events and duplicate callback attempts must converge without duplicate transition/callback work.

Consumer and reconciliation contract

• Extend Relay OpenAPI/API docs with the generic callback schema, signature algorithm, header contract, versioning rules, safe reason codes, and deterministic verification fixture.
• Keep GET /api/transactional/messages/{message_id} from Guarantee idempotent transactional delivery and ambiguity-safe provider state #2 authoritative so a client can reconcile when callbacks are delayed/lost.
• Provide safe metrics/status for callback backlog/oldest age/failures and unmatched-event count/oldest age/expiration.
• Preserve a migration path for existing CMP callbacks only when explicitly bounded and tested; no new client may depend on the old Bearer/full-email contract.

Expected files

Implementation should remain within Relay-owned integration/event surfaces such as:

• mailing/models.py and new forward-only migrations;
• a generic callback service module under mailing/services/, replacing or narrowly adapting mailing/services/cmp_callbacks.py;
• mailing/services/ses_webhooks.py and provider-event correlation helpers;
• mailing/services/transactional.py / lifecycle transition event hooks from Guarantee idempotent transactional delivery and ambiguity-safe provider state #2;
• callback dispatcher management command/task and scheduling configuration;
• mailing/views.py, mailing/services/api_docs.py, docs/api.md, and OpenAPI fixtures;
• mailing/tests/test_cmp_callbacks.py only for bounded legacy behavior;
• new generic callback tests plus mailing/tests/test_ses_webhooks.py and status/operations tests.

Do not redesign website callback handling, campaigns, tracking analytics, inbound mail, Taskdeck, or infrastructure.

Acceptance criteria

• Generic callback configuration is tenant-scoped and uses a separate rotatable/revocable signing secret through a safe server-side secret boundary.
• Every client-visible transport transition from Guarantee idempotent transactional delivery and ambiguity-safe provider state #2 creates exactly one durable generic callback event after the authoritative state commit.
• The versioned payload contains only stable IDs/status/time/correlation/template/safe reason fields and excludes recipient, bodies, context, credentials, raw provider payload, and unrestricted metadata.
• HMAC covers the exact timestamp.raw-body bytes and deterministic fixtures verify valid, invalid, rotated, replay-window, and tampered signatures.
• Delivery retries retryable failures with bounded backoff, records only safe attempt data, and never regresses transport state.
• Callback row/event deduplication prevents duplicate work under repeated transition processing or dispatcher concurrency.
• Provider event IDs deduplicate before transition application.
• Valid unmatched events are stored redacted, retried for correlation for seven days, applied once when matched, and become operator-visible terminal exceptions when unresolved.
• Duplicate, out-of-order, and late delivered/bounce/complaint events obey Guarantee idempotent transactional delivery and ambiguity-safe provider state #2's guarded monotonic lifecycle and do not emit contradictory duplicate callbacks.
• Authoritative status polling supports client reconciliation through callback loss or delay.
• Metrics/operational status cover callback backlog/age/failure and unmatched-event backlog/age/expiration without PII.
• Existing CMP callback compatibility, if retained, is explicitly bounded; website/new clients cannot receive the Bearer/full-email payload.
• Unit, migration, concurrency, provider-ingress, dispatcher, redaction, and OpenAPI/contract tests pass without external network/AWS credentials.

Mandatory failure tests

1. Transition commit/rollback, crash before dispatcher wakeup, response loss after callback receiver commit, repeated dispatcher execution, and concurrent callback claims.
2. Correct, incorrect, missing, expired, future, replayed, rotated-overlap, and revoked HMAC signatures using the published canonical fixture.
3. Endpoint timeout, DNS/connection failure, redirect to unapproved origin, 2xx, retryable 429/5xx, permanent 4xx, retry exhaustion, and secret/endpoint disablement.
4. Payload redaction canaries for email, subject/body/context, headers/message parts, credentials, provider payload, and arbitrary metadata.
5. Duplicate provider event ID, provider event before message correlation, late correlation, duplicate correlation workers, out-of-order accepted/delivered/bounced/complained, and unresolved seven-day expiry.
6. Cross-tenant event/message/config access and callback routing.
7. Lost callbacks followed by status reconciliation proving authoritative convergence without re-sending email.

Validation

Use uv, local receiver fakes, deterministic clocks, and provider fixtures only:

git diff --check
uv run python manage.py makemigrations --check --dry-run
uv run pytest mailing/tests/test_cmp_callbacks.py mailing/tests/test_ses_webhooks.py mailing/tests/test_transactional_api.py
make lint
make test

No live callback endpoint, secret provisioning, Relay sandbox request, AWS credential, SES request, infrastructure change, deployment, or email send is permitted.

Browser and screenshots

This is a backend callback/provider-event/OpenAPI contract. No visible operator-page change is required, so browser testing and screenshots are not applicable unless UI scope is separately approved.

Dependencies

• External ownership and consumer requirements: website #21.
• Depends on Harden scoped credentials and versioned transactional template rendering #1 for tenant credential/template-version identity and Guarantee idempotent transactional delivery and ambiguity-safe provider state #2 for authoritative guarded states, attempts, message IDs, and transition events.
• Website callback consumption/reconciliation is separate website work; this issue supplies the Relay half and deterministic contract fixtures.
• Website real-send remains disabled until Harden scoped credentials and versioned transactional template rendering #1, Guarantee idempotent transactional delivery and ambiguity-safe provider state #2, and this issue are deployed at exact pinned commit/OpenAPI versions and pass website consumer contract tests.

Non-goals

• No website callback endpoint, EmailDelivery projection, scheduler, Studio/admin UI, or Datamailer migration.
• No request-hash idempotency, lease/state, or ambiguity implementation beyond consuming Guarantee idempotent transactional delivery and ambiguity-safe provider state #2.
• No template lifecycle/rendering beyond consuming Harden scoped credentials and versioned transactional template rendering #1 identifiers.
• No engagement/open/click analytics expansion, marketing/newsletter behavior, or tracking pixels.
• No real callback secret provision/rotation, live endpoint registration, infrastructure mutation, deployment, production rollout, broad recipient access, or email send.
• No raw provider-payload retention for convenience.

Process handoff

Follow docs/PROCESS.md in an isolated Relay issue worktree. The engineer implements and tests without committing; a separate tester verifies exact callback, redaction, retry, and reconciliation criteria; a product manager accepts; only then is the issue committed with Closes #N and moved through the repository lifecycle without a pull request.

[alexeygrigorev]

Groomed on creation

Owned Relay slice for the generic redacted HMAC callback outbox, retry contract, provider-event deduplication, seven-day unmatched-event reconciliation, and consumer fixtures required by website #21.

Pinned to audited baseline c0fdfac632f057f1e6a467111613928c4899b13b and dependent on #1/#2's identity, template-version, state, and transition contracts. No live secret, callback endpoint, deployment, provider call, or email send is authorized.