feat(clickhouse): add llm_message table

[victorludvig]

What is changing

• platform/clickhouse/init/schema.sql — new table hexgate_audit.llm_message: the shared eight-column envelope, then model, turn_key, message_seq, resynced, truncated, and input_messages / output_messages / system_instructions as String CODEC(ZSTD(3)) holding the official gen_ai.* JSON shapes. ReplacingMergeTree(received_at), PARTITION BY toYYYYMM(received_at), ORDER BY (project_id, session_id, turn_key, message_seq, event_id), 180-day TTL.
• platform/clickhouse/migrations/0003_add_llm_message.sql — the same CREATE TABLE, byte-identical once comments are stripped, for hand application to volumes that already exist. Header follows 0001: apply before the enricher that writes to the table is deployed, and what breaks if it is skipped.
• docs/internals/audit-pipeline.md §5.1 — the table, its sort key, and why it is a separate table.

Nothing writes to or reads from the table yet; that is PR 4 and PR 6 of the stack.

Design: LLM message logging design · implementation spec.

Why is this change necessary

Message content is large, opt-in (nothing emits hexgate.messages yet) and read by session, so it does not belong as columns on llm_invocation, which is aggregated by user and model. The sort key is session-first with (turn_key, message_seq) next because the one read pattern is "reconstruct this session's transcript in order": a list's rows land adjacent and ordered, and event_id last keeps ReplacingMergeTree dedup to SDK retries of the same event. init/ runs once on an empty volume, so the committed migration is how staging and production get the table.

Tests

• Both files applied in a throwaway clickhouse local (image clickhouse/clickhouse-server:24.10): init/schema.sql followed by the migration, confirming the statements parse and the migration is idempotent against a volume that already ran init. system.columns shows the 16 columns in the intended order; system.tables shows the sort key project_id, session_id, turn_key, message_seq, event_id and partition key toYYYYMM(received_at).
• Comment-stripped diff of the two CREATE TABLE statements is empty.
• No Python tests: the insert path and verify_schema that exercise the table arrive with PR 4.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[guillaume-hexamind]

llm_message sort key omits occurred_at — score 78/100

platform/clickhouse/init/schema.sql:160 and platform/clickhouse/migrations/0003_add_llm_message.sql:49

Issue. The sort key is (project_id, session_id, turn_key, message_seq, event_id) — no time column. All three sibling tables end theirs …, occurred_at, event_id (schema.sql:60, :87, :116). But the read this table is designed for needs occurred_at: the doc bullet added in this PR (docs/internals/audit-pipeline.md:525) says "A session with a sub-agent or handoff has several turn_keys; readers order across them by occurred_at." That ordering has no index support, and it can never get any.

Why it can't be deferred. MODIFY ORDER BY cannot add occurred_at to this key at all — not mid-key, not appended. Verified on clickhouse/clickhouse-server:24.10 against this exact DDL:

-- mid-key
ALTER TABLE m MODIFY ORDER BY (project_id, session_id, turn_key, message_seq, occurred_at, event_id);
Code: 36. Primary key must be a prefix of the sorting key, but the column
in the position 4 is occurred_at, not event_id. (BAD_ARGUMENTS)

-- appended
ALTER TABLE m MODIFY ORDER BY (project_id, session_id, turn_key, message_seq, event_id, occurred_at);
Code: 36. Existing column occurred_at is used in the expression that was added
to the sorting key. You can add expressions that use only the newly added
columns. (BAD_ARGUMENTS)

The only later route is dropping and recreating the table. Unlike 0001/0002's additive ADD COLUMN, this is a one-way door.

Why now is free. Nothing emits hexgate.messages yet — hexgate/tracing/semconv.py:108-111 defines TURN_KEY/MESSAGE_SEQ/RESYNCED but there are no producers outside tests/tracing/test_semconv.py, no LlmMessageEvent, and no insert path in platform/api. The table is empty on every deployment, so today the fix costs two edited lines and nothing else.

Suggested fix — in both files:

ORDER BY (project_id, session_id, turn_key, message_seq, occurred_at, event_id)

This keeps event_id last (dedup unchanged), keeps a list's rows adjacent and ordered, matches the sibling convention, and gives the documented cross-turn_key read an indexed tiebreak. Worth updating the sort-key comment at schema.sql:157-159 and the doc bullet at :525 to match.

_{Partial mitigation if this ships as-is: ALTER TABLE … ADD INDEX occurred_at TYPE minmax + MATERIALIZE INDEX does apply later (verified), which recovers time pruning but not sort-key ordering.}

[guillaume-hexamind]

One blocking issue, approve of the logic otherwise

[victorludvig]

Agreed the column belongs in the sort key — but not in the position suggested.

After message_seq, occurred_at only separates retries of one event. Third is the position that pays: it matches the order the PR 6 endpoint returns rows in (occurred_at, message_seq), so the read is in-order rather than sorted — and when session_id is empty, where the prefix stops being selective, it's the only thing left pruning. It has to be first after session_id because anything grouping the rows ahead of it wins; a sub-agent's message_seq restarts at 0, so time is all that orders the lists against each other.

ORDER BY (project_id, session_id, occurred_at, message_seq, event_id)

turn_key leaves the key — with occurred_at ahead of it, it would only break ties event_id already resolves. It stays a column for grouping and gap detection.

Also added run_id, mirroring llm_invocation, so a transcript is attributable to its run when session_id is empty. A column, not a key entry: before occurred_at it would split the session read into per-run blocks, and uuid4 wouldn't order runs chronologically anyway.

Applied both files in a throwaway clickhouse local (image clickhouse/clickhouse-server:24.10): system.tables reports the sorting key above, and system.columns lists all 17 columns in the intended order.