Skip to content

TC-959: ship typed TC-1274 SQL contention diagnostics through the log sidecar - #342

Open
samgbafa wants to merge 4 commits into
mainfrom
skgbafa/tc-959-f6-log-allowlist
Open

samgbafa wants to merge 4 commits into
mainfrom
skgbafa/tc-959-f6-log-allowlist

Conversation

@samgbafa

@samgbafa samgbafa commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

TC-959 / TC-1274. Log-shipper only: no Rust, node, image, or node per-key limiter change. #309's reviewed sidecar is unchanged except for the regenerated embedded Vector config.

What changes

The N0 Vector sidecar ships the seven TC-1274 (F6) SQL contention diagnostics as typed, per-occurrence records. It no longer folds them into the unclassified count.

  • Admission. diagnostics.json typed declares each code's exact target, level and field list. The compiler generates straight-line VRL that checks each field against its closed type in fields.json: bounded integers, fixed enums, and space_hash/db_hash matching ^[0-9a-f]{64}$.
    • A wrong level, a missing field, or any out-of-type, out-of-range, out-of-enum or bad-hash value rejects the whole record into the invalid_fields gate count. No partial record ships.
    • Undeclared fields are never copied: raw db names, the pool operation label, error text, nested values.
    • The same message from any other target stays unclassified.
  • Routing. event_route.typed_diagnostic sends typed records to the sink without the diagnostic reduce. Untyped codes keep their count aggregation.
  • Cap (Sam's decision tc-959-f6-log-cap).
    • At most 10 records per code per 60 s window, each of the seven codes counted separately.
    • Enforced by a Lua v2 counter keyed only by the fixed codes. Vector's throttle would drop silently.
    • Excess records are dropped. Only their number leaves, as the new typed_throttled gate reason in logship_gate_summary_total.
  • Budget. budget.mjs reserves the cap at each code's widest valid record (474 B): 70 records/min, 1,990,800 B per storm hour, 1,433,376,000 B per 30 sustained days. The runbook states the trade-off and the totals.
  • Compose. The only change is the regenerated VECTOR_CONFIG_JSON line.

Known gates before merge

  1. F6 emitter contract. The F6 node PR is not published yet. Targets, field sets, levels and enum values follow the F6 design and the local F6 tree, and must be confirmed against the emitted PR:
    • operation is deliberately not admitted;
    • outcome includes incomplete.
  2. Canonical schema pin. log-shipper/fields.json is canonical tinycloud-dev main da29faa4 plus the F6 fields, typed_throttled, and the seven event codes. canonical.json is not re-pinned, so CI's tc959-check schema-pin assertion fails until the canonical F6 schema PR lands with these exact bytes and the pin is updated in this branch.
  3. Release path. This rides the main release via ci: promote a released main image with the reviewed N0 compose template #315's single rebind. ci: promote a released main image with the reviewed N0 compose template #315's template hash, fixture and docs are untouched here; its owner recomputes them on the final bytes.

Update

Merged main 65d0e3c9 (#346). The embedded Vector config is regenerated with the byte-exact generator, and the N0 pin in docker.yml, docs/releasing.md, the runbook and the CI fixture now names this head's compose, superseding #346's 6563d51b…. db_hash stays F6-only, and the cap stays at 10 per code per minute.

Evidence

Tests were written first and observed RED before the implementation.

  • Before the cap-to-10 change: native Vector 0.59.0 (pinned tarball checksum) passed errors_slow 415/415 and all_requests 415/415.
  • At 10/min, before a host disk hold paused the suite:
    • node --test 34/34;
    • native-check passed: the F6 record is exact, no canary reached the console;
    • vector validate passed;
    • native localhost smoke: 16 lock_wait → 10 shipped plus typed_throttled:6; 6 invalid shapes → invalid_fields:6; no canary on wire or console; no db/operation/error/sql keys.
  • Still to run after the host go: the full native Vector suite at 10/min.

No production, secret or Docker access.

Admit the seven F6 messages only from their exact target and level, with
only their declared fields: bounded integers, fixed enums and keyed
space_hash/db_hash of exactly 64 lowercase hex. Any invalid field rejects
the whole record into the invalid_fields gate count; raw db names, the pool
operation label, error text and nested values are never copied. Typed
records bypass the diagnostic reduce. Regenerates the embedded Vector
config; fields.json still needs the canonical schema re-pin.
Sam's decision (tc-959-f6-log-cap): the shipper admits at most 10 records
per typed code per fixed 60 s window, each of the seven codes counted
separately, as defence in depth over the node's per-key limiter. A Lua
counter keyed only by the fixed codes enforces it. Excess records are
dropped and only their count leaves, as the new typed_throttled gate
reason. budget.mjs reserves the cap at each code's widest valid record:
70 records/min, 1,433,376,000 bytes per 30 sustained days.
…-allowlist

# Conflicts:
#	docker-compose.dstack-postgres.yaml
#	scripts/tc959-docker.test.mjs

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant