Remote destinations silently lose the majority of a session's events - no self-healing replay

[colombod]

The problem

Against a remote destination (Azure/APIM + Entra), the telemetry hook silently fails to
deliver the large majority of a session's events — while reporting itself healthy — and the
only recovery path is a human noticing a console warning and running
context-intelligence-upload by hand.

This is not a drain-window problem (see #111 — the close_drain_timeout 10→20 bump addresses only
short tails, and says so explicitly). It is a throughput problem with no self-healing.

Measured evidence

Aggregated from the durable shutdown_undelivered records in
~/.amplifier/context-intelligence-logs/forwarding-*.jsonl on one workstation —
215 shutdowns over 15 days, 211 of them against the Azure/APIM destination:

          sum      median    max    nonzero
total    53160      163      2188    215/215
queued   31673      157       256    211/215
inflight   213        1         1    213/215
dropped  21274        0      1935     77/215

• 53,160 events never reached the server. 21,274 of those were dropped on queue overflow.
• The median session ends with 163 undelivered events and a 157-deep queue.
• queued hit the dispatch_queue_capacity ceiling (256) in 4 records; 77 records shed events.
• breaker_open=False on all 215. degraded_seconds=0 on 202.

That last line is the important one: the destination is healthy. Nothing is down, nothing is
rejecting, the circuit breaker never opened. The client is simply producing faster than it
delivers.

Root cause

_DestinationDispatcher runs one worker with one in-flight event at a time — the worker loop
pulls a single event, awaits _post, then pulls the next. Its delivery ceiling is therefore
roughly one event per round-trip.

A localhost POST is sub-millisecond, so this is invisible locally. A remote POST pays
connect + TLS + APIM + per-request Entra token + graph write — commonly several hundred ms. That
puts the ceiling at a few events/sec, while a busy Amplifier session emits far more. The queue
grows monotonically for the whole session, hits the 256 cap, and starts shedding.

The drain window at close() cannot help: no bounded window empties a queue that is already
150+ deep, and by then some events were dropped hours earlier.

Why this is worth fixing properly

Everything needed for silent, safe recovery already exists:

1. Durable local log — every event is written to events.jsonl before any forward is
   attempted. Nothing is ever actually lost.
2. Idempotent delivery — build_payload stamps a deterministic idempotency_key
   (SHA-256 over {event, workspace, data}) and the server dedups on it. Replaying an
   already-delivered event is a no-op, so a sweep cannot create duplicates.

The missing third piece is small: a per-destination delivery watermark and a backlog sweep.
Grep the module for offset / resume / backlog and the only hits are log strings telling the
human to run the upload CLI. There is no automatic catch-up of any kind.

Suggested direction (to be designed, not prescribed here)

Sketches, in rough order of value:

• Replay on start / on reconnect. Persist a per-destination watermark; on session start (and
  after the breaker closes) sweep undelivered events from events.jsonl, bounded by count and
  age so a launch never re-POSTs a week of history.
• Continuous background catch-up. Same mechanism, running during the session rather than only
  at boundaries — this is what would actually keep a busy session's queue from growing all day.
• Amortize the per-request cost. Batch N events per POST (needs a server-side batch endpoint)
  and/or allow a small number of concurrent in-flight POSTs. Ordering and the breaker's
  accounting both need thought here.
• Make the warning proportional. A 2-event tail and a 2,188-event backlog currently print the
  same shape of line. If the bundle recovers by itself, most of these stop needing a console
  warning at all — the durable forwarding-*.jsonl record already preserves the trail.

Gate

This crosses a seam (client↔server boundary, networking, hook config wiring), so per AGENTS.md
"done" requires a real DTU run plus captured evidence — not unit tests alone.

Acceptance

• Given a remote destination slower than the producer, when a session ends with undelivered
  events, then a subsequent session delivers them without user action and without a console
  warning.
• Given events already delivered, when a sweep replays them, then the server records no
  duplicates (verified against a real server, not a mock).
• Given a genuinely broken destination, when a sweep runs, then the failure is still surfaced
  loudly — self-healing must not mask a real outage.

Generated with Amplifier