feat(platform-api): map hexgate.messages spans into LlmMessageEvent

[victorludvig]

Design: LLM message logging design · implementation spec.

What is changing

LlmMessageEvent(AuditEnvelope) in schemas.py, mirroring the llm_message columns of #183 including run_id; capped_input_messages() / capped_output_messages() / capped_system_instructions() in jobs/enricher/enforcement.py (key-redact like arguments, head+tail cap via the PR 1 helpers, truncate never reject, each returning the cut flag); _message_fields() and a map_span branch in jobs/enricher/mapping.py that ORs the enricher's cuts into the SDK's truncated; as_json_value() in coerce.py for the array-shaped attributes (the two bool flags reach the model raw — Pydantic coerces them). SCOPE_MESSAGES is not added to KNOWN_SCOPES here — the scope stays a loud unknown_scope DLQ reject until PR 4.

jobs/enricher/dlq.py now parses and redacts the three gen_ai.* message arrays before dead-lettering, the way it already does for arguments / hint / attributes — the DLQ is this PR's intended destination for every message span, and a tool-call message carries the same caller arguments a decision does.

Why is this change necessary

consumer.py buckets events by type and commits the offset after the inserts, so a scope accepted before it has a bucket would be validated and then silently dropped with no DLQ record. Keeping registration for PR 4 means this PR can be reverted alone and main stays non-lossy. The DLQ redaction closes the gap that routing message spans there would otherwise open on a 30-day, no-ACL topic.

Tests

platform/api/tests/jobs/enricher/test_mapping.py (calls _message_fields() directly: happy path, absent optional fields, SDK truncated kept, over-cap stored short, required attributes, run_id in/out); test_enforcement.py (cap boundary at and one byte over, head+tail marker, tool-call secret redacted, 8 KiB output/system caps); test_coerce.py (as_json_value); test_dlq.py (secret inside a message array redacted, bare-string field dropped).

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 91.66667% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
platform/api/hexgate_api/jobs/enricher/mapping.py	70.00%	2 Missing and 1 partial ⚠️
hexgate/audit.py	81.81%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[guillaume-hexamind]

Nested JSON-string tool-call arguments escape key redaction entirely — severity 72/100

Where: platform/api/hexgate_api/jobs/enricher/enforcement.py:77 (_capped_content) and, live today, platform/api/hexgate_api/jobs/enricher/dlq.py:70 (_redacted_attributes).

What. redact (hexgate/audit.py:109) matches dict keys; a string value is a leaf and comes back unchanged. The decision path works because ARGUMENTS is a top-level JSON string that as_json_dict parses first, so its keys become real keys. Inside a message array the equivalent payload sits one level deeper, and the raw OpenAI wire shape serializes tool_call.arguments as a JSON string, not an object. The string is never parsed, so the substring key match never reaches inside it.

Reproduced against this branch:

capped_input_messages([{"role": "assistant", "parts": [
    {"type": "tool_call", "name": "login",
     "arguments": "{\"user\":\"bob\",\"password\":\"hunter2\"}"}]}])
# → [... "arguments": "{\"user\":\"bob\",\"password\":\"hunter2\"}"]   secret intact

capped_input_messages([{"role": "assistant", "parts": [
    {"type": "tool_call", "name": "login",
     "arguments": {"user": "bob", "password": "hunter2"}}]}])
# → [... "arguments": {"user": "bob", "password": "[REDACTED]"}]      redacted

Same on the DLQ path: a hexgate.messages span carrying the string-shaped payload produces an envelope with hunter2 in plain text. That is the invariant dlq.py:107 states — "this topic has 30-day retention and no ACLs, so unredacted arguments must never reach it." This path is reachable today, since SCOPE_MESSAGES is deliberately outside KNOWN_SCOPES and every message span is therefore DLQ'd as unknown_scope.

Test gap. Both new tests use the dict shape only — test_dlq.py::test_when_a_message_span_is_rejected_then_dlq_redacts_inside_the_messages and test_enforcement.py:120. The string variant is the one that fails.

Two things that bound the severity, for fairness:

• Nothing emits hexgate.messages yet (no producer anywhere outside mapping.py and tests), so this is latent, not leaking now.
• It is not new here: capped_arguments({"body": '{"password": "hunter2"}'}) already returns the secret unredacted on the live decision path. This PR extends the existing gap to a new field rather than opening it.

It still scores high because the OTel GenAI semconv normalizes tool_call.arguments to an object while the provider wire format uses a string — so whether this bites depends entirely on whether the not-yet-written emitter normalizes, and this module's documented posture is that it explicitly does not trust emitters. Fixing it before the emitter ships is much cheaper than after, once the DLQ's 30-day retention has started accumulating.

Suggested fix. In _redact (or a wrapper at both call sites), attempt json.loads on string leaves whose key looks like arguments/input, and redact the parsed result when it yields a dict. A narrower option is to apply it only under tool_call-shaped parts. Either way, worth a test with the string-shaped arguments alongside the existing dict-shaped ones.

_{🤖 Generated with Claude Code}

[guillaume-hexamind]

One blocking issue above; I approve of the logic otherwise

[victorludvig]

Nested JSON-string tool-call arguments escape key redaction entirely — severity 72/100

Where: platform/api/hexgate_api/jobs/enricher/enforcement.py:77 (_capped_content) and, live today, platform/api/hexgate_api/jobs/enricher/dlq.py:70 (_redacted_attributes).

What. redact (hexgate/audit.py:109) matches dict keys; a string value is a leaf and comes back unchanged. The decision path works because ARGUMENTS is a top-level JSON string that as_json_dict parses first, so its keys become real keys. Inside a message array the equivalent payload sits one level deeper, and the raw OpenAI wire shape serializes tool_call.arguments as a JSON string, not an object. The string is never parsed, so the substring key match never reaches inside it.

Reproduced against this branch:

capped_input_messages([{"role": "assistant", "parts": [
    {"type": "tool_call", "name": "login",
     "arguments": "{\"user\":\"bob\",\"password\":\"hunter2\"}"}]}])
# → [... "arguments": "{\"user\":\"bob\",\"password\":\"hunter2\"}"]   secret intact

capped_input_messages([{"role": "assistant", "parts": [
    {"type": "tool_call", "name": "login",
     "arguments": {"user": "bob", "password": "hunter2"}}]}])
# → [... "arguments": {"user": "bob", "password": "[REDACTED]"}]      redacted

Same on the DLQ path: a hexgate.messages span carrying the string-shaped payload produces an envelope with hunter2 in plain text. That is the invariant dlq.py:107 states — "this topic has 30-day retention and no ACLs, so unredacted arguments must never reach it." This path is reachable today, since SCOPE_MESSAGES is deliberately outside KNOWN_SCOPES and every message span is therefore DLQ'd as unknown_scope.

Test gap. Both new tests use the dict shape only — test_dlq.py::test_when_a_message_span_is_rejected_then_dlq_redacts_inside_the_messages and test_enforcement.py:120. The string variant is the one that fails.

Two things that bound the severity, for fairness:

• Nothing emits hexgate.messages yet (no producer anywhere outside mapping.py and tests), so this is latent, not leaking now.
• It is not new here: capped_arguments({"body": '{"password": "hunter2"}'}) already returns the secret unredacted on the live decision path. This PR extends the existing gap to a new field rather than opening it.

It still scores high because the OTel GenAI semconv normalizes tool_call.arguments to an object while the provider wire format uses a string — so whether this bites depends entirely on whether the not-yet-written emitter normalizes, and this module's documented posture is that it explicitly does not trust emitters. Fixing it before the emitter ships is much cheaper than after, once the DLQ's 30-day retention has started accumulating.

Suggested fix. In _redact (or a wrapper at both call sites), attempt json.loads on string leaves whose key looks like arguments/input, and redact the parsed result when it yields a dict. A narrower option is to apply it only under tool_call-shaped parts. Either way, worth a test with the string-shaped arguments alongside the existing dict-shaped ones.

🤖 Generated with Claude Code

Fixed in 594aae8: redact now parses string values under arguments, redacts inside, and serialises back, at both the enforcement and DLQ call sites, with string-shaped tests beside the object-shaped ones; the pre-existing decision-path gap under arbitrary keys stays out of scope here.