J3: optional off-box audit event sink

[henleda]

Closes the J3 roadmap item. audit.log is written by the same process that makes the change, to a local file — so the box that made an unauthorised change is the box that can edit the only record of it. This ships each entry to a collector as it is written, so the local file stops being the only copy.

VPCOPILOT_AUDIT_SINK=https://collector.example.com/ingest   # POST the entry as the body
VPCOPILOT_AUDIT_SINK=syslog://10.0.0.9:514                  # RFC 3164 datagram over UDP
VPCOPILOT_AUDIT_SINK=syslog:///var/run/syslog               # …or a local unix datagram socket
VPCOPILOT_AUDIT_SINK=stdout                                 # a JSON line for a log-scraping runtime
VPCOPILOT_AUDIT_SINK=off                                    # deliberately disabled
vpcopilot audit-sink --send                                 # prove it lands

Hooked inside audit.record — the one chokepoint — so all 30 call sites and all three surfaces (CLI, console, MCP) inherit it with no call-site change and no new record() parameter (**detail swallows unknown kwargs and json.dumps has no default=, so a stray log= would be serialized into the entry or raise after the LB was already mutated). No new dependency: httpx is already a base dep, syslog is a raw socket.

Acceptance, as met

criterion	how
a dead collector never fails an apply, logs one warning	verified live — a real audit-backfill against a black-holed sink exited 0, wrote its sidecar and its audit entry, and emitted exactly one warning
configured in .env and the ⚙ Setup page	verified by POSTing /api/config in a running console and watching it take effect with no restart
the local audit.log stays authoritative	the line is serialized once and handed to both destinations, so the shipped copy is byte-identical to the line on disk (verified live); a failed local write delivers nothing

"Logs one warning" is a design constraint, not a log line: a hung collector at a 5 s timeout times the several entries one apply writes is a stall the criterion plainly forbids, so a failure starts a 60 s cooldown — one timeout and one warning per outage, not per entry. The warning goes to stderr, never stdout, so it cannot corrupt the MCP protocol stream even where K1's stdout swap is not in force.

Refusing to guess, applied to a transport

A run succeeds whether or not the sink works, so a misconfigured sink would otherwise be invisible. Every "not delivering" state is distinguishable from "delivering":

• an unusable value reports its reason on every surface, and is not silently equivalent to having no sink;
• a 3xx is a failure — the body was never re-sent. Refused rather than followed on purpose: the request carries the bearer token, and chasing a Location to an unconfigured host is how a credential travels;
• UDP reports sent, unconfirmed, because a datagram to a port with nothing bound succeeds at the send call. Same three-state honesty as export --verify's present-unverified.

Run for real

• HTTP: real POST to a live collector — correct content-type, bearer token, and a body byte-identical to the on-disk line. Driven end to end by a real vpcopilot audit-backfill.
• Syslog wire format: validated against a strict RFC 3164 parser — <134> (local0.info), space-padded day, hostname, vpcopilot[pid], JSON payload intact.
• Oversize, against a real kernel: a realistic drift_detected with a 60-field diff measured 5,481 bytes — 2.7× the macOS /var/run/syslog SO_SNDBUF of 2048, which refuses more with EMSGSIZE. Nothing is hardcoded; the kernel sets the limit and only a refusal triggers a reduced envelope. A 16,139-byte entry was refused on UDP and a 363-byte valid-JSON envelope arrived naming the action, finding_id, true byte count and where the full record lives, while the on-disk entry kept all 120 changes.
• Console: the Setup card, the status readout and the Send test event button exercised in a browser.
• Could not be confirmed, and the roadmap says so rather than claiming it: that an entry reaches a real syslog daemon's store. macOS discards local0.info by default, and the control proves it is the box and not the frame — /usr/bin/logger -p local0.info also produces zero hits.

Adversarial review

35 raised across six failure dimensions; 4 confirmed after independent skeptics tried to refute each. Every one was the same shape — not delivering, rendering as delivering:

• The one input that produced silence was the one this module exists to make loud. urlsplit raises on a dropped IPv6 bracket (https://[2001:db8::1/x) — the typo this item's own IPv6 support invites. Unguarded that became a CLI traceback and a console 500, while emit's catch-all swallowed it with no warning and no last_error.
• A 3xx counted as delivered. Reproduced against a real 302 server: sent, delivered 1, exit 0, redirect target never contacted.
• sent over UDP meant "handed to the kernel" — audit-sink --send against a dead collector printed sent and exited 0.
• The test suite shipped fabricated audit records to a real collector. With the variable exported, one full run sent 267 datagrams of invented apply_waf / retire / rollback_failed entries, indistinguishable at the collector from real changes. Now cleared structurally in conftest.py; verified with a counter (1 deliberate record registers, the full suite adds zero).

Five more were found by self-review before the reviewers reported, including a password in the sink URL being printed on every surface (urlsplit puts userinfo in netloc) and an IPv6 collector being unreachable (hardcoded AF_INET).

Pre-existing fixes, called out rather than slipped in

None is caused by this change; the first is a raise inside audit.record, which no audit-integrity item should ship past.

• runmeta._save named its temp file by PID only, so two threads writing a fresh out dir shared it and the loser of the os.replace got FileNotFoundError out of audit.record — a change already made to a load balancer that could not be recorded. Reproduced 12 times out of 12 with 8 threads and no sink configured at all. The console starts every apply on its own daemon thread with no job lock.
• runmeta.run_id minting was an unguarded read-modify-write, so those same concurrent first writes produced eight different run_ids — seven audit entries carrying a join key run.json does not contain. Now thread-atomic; single-threaded behaviour byte-identical, and the remaining cross-process case is stated rather than pretended away.
• The console's .bad CSS class was used and never defined, so the H2 dependency preview's "could not check" and manifest-parse errors rendered as ordinary body text — a warning styled like content, in the preview built to prevent exactly that. Found while validating the Setup card; the fix repairs three pre-existing call sites.

No regressions

audit.record's output is unchanged — no new key, no reordering, no envelope — pinned by a test comparing an entry written with and without a sink. 854 tests (was 785), coverage 79% → 80%, ruff clean, suite run the way CI runs it.

🤖 Generated with Claude Code