TC-959: ship typed TC-1274 SQL contention diagnostics through the log sidecar

.github/workflows/docker.yml

@@ -658,7 +658,7 @@ jobs:
               raise SystemExit('::error::Deployment destination is not the fixed reviewed path')
           native = 'ghcr.io/tinycloudlabs/tinycloud-node'
           reviewed = native + '@sha256:4bdae4fc232d886bb56bfbce4ec62ed4b8fb4e1369a01f6afb73d06dd68c188f'
-          n0_hash = '6563d51bffe4b537d49b0c087a50e86753aaaf5ba533f1e52c7c2214f7f5f04e'
+          n0_hash = '1e62ceb326679c8862c8ae822288b48c54997983a4b011b84d516e943e4f1859'
           pre_n0_hash = 'df239749ca0a9c09893a855ae2a8abb94090b8556515ad002105c58ac91f4c91'
           pre_n0_template_hash = '28302d6761d841b7cc5c11314aed2d377735e6bbdcaf85972a752e03bc9fdc93'
           rollback = os.environ.get('ROLLBACK_PRE_N0') == 'true'

CHANGELOG.md

@@ -20,6 +20,7 @@
 - On PostgreSQL, the boot's schema migrations wait at most 30 s for any lock (`SET LOCAL lock_timeout`, in the same transaction that lifts `statement_timeout`) (TC-1176). If another session holds a conflicting lock on a table a migration changes, such as an open write on `delegation` during the first boot after 1.20.6, the node now fails to start with an error naming the lock wait instead of hanging. All pending migrations roll back together, so nothing is applied or recorded, and the next start applies them. The limit ends with that transaction and does not reach pooled sessions. SQLite and MySQL are unchanged.
 - `/attestation` quotes the SHA-256 of `key_binding` as REPORTDATA and returns it as `key_binding_sha256`; the quote no longer carries the raw binding. Before answering 200 the node checks that the returned TDX v4 quote's REPORTDATA field holds that digest (a structural check; it does not verify the quote). When the TEE can't produce a bound quote, the endpoint returns 503 instead of 200 `mode: "classic"`, and the dstack client reports a guest-agent error status instead of a parse error. `nonce` must be 1-64 characters of `A-Z a-z 0-9 - _` (400 otherwise), and the endpoint now shares the public routes' per-IP rate limit. Verifiers recompute the digest from their own nonce and expected DID and must still verify the quote; see `docs/attestation.md` (TC-963).
 - Prepare a compose-only GELF-to-Vector sidecar for the unchanged node 1.20.6 and ingress image digests. The default unsampled `errors_slow` profile emits valid requests with status >=400 or duration strictly over one second; explicit `all_requests` remains available, and all accepted typed WARN/ERROR diagnostics remain eligible for bounded count aggregation. Document eligible-request volume accounting, accepted BEST-EFFORT outage loss, the failed local burst evidence and required zero-observed-loss healthy hosted burst, plus the config-only deployment/rollback privacy packet. No Rust or application image change and no deployment authorization; see `docs/runbooks/tc-959-n0-log-shipping.md` (TC-959 N0).
+- The N0 log sidecar ships the seven TC-1274 SQL contention diagnostics (`sql_operation_lock_wait`, `sql_operation_lock_still_waiting`, `sql_operation_lock_held`, `sql_operation_still_running`, `sql_actor_message_slow`, `db_pool_acquire_timeout`, `db_connect_failed`) one record per occurrence, matched by exact target, message and level. Each record carries only its declared typed fields: bounded integer timings and counts, fixed `op`/`phase`/`kind`/`outcome`/`class` values, and keyed `space_hash`/`db_hash` of exactly 64 lowercase hex. Any invalid field rejects the whole record into the content-free gate count; raw database names, the pool operation label, error text and nested values are never copied. Per Sam's decision, the shipper caps them at 10 records per code per 60-second window, each code counted separately, and reports drops as the `typed_throttled` gate count; `budget.mjs` reserves that cap. Other diagnostics keep their count aggregation (TC-959, TC-1274).
 
 ## [1.20.7] - 2026-10-14
 

docs/releasing.md

@@ -294,10 +294,10 @@ gh workflow run docker.yml -R TinyCloudLabs/tinycloud-node --ref main \
 ```
 
 The served/OCI identity remains **1.20.6-dstack / e18578ff9162986ddd813259c60f73866e3e60d1**.
-The N0 raw lock **changed in #346 (TC-959)**, which passes the validated N2 correlation fields through the log sidecar:
-`6563d51bffe4b537d49b0c087a50e86753aaaf5ba533f1e52c7c2214f7f5f04e`,
-from #346 final reviewed head.
-It supersedes the #309 lock `ca110b0bae6d5a06a1002fb80df3f71cffec11659c9f9dbfa5d72624735546f6`,
+The N0 raw lock **changed in the TC-1274 F6 log-shipper PR (#342)**, which ships the typed SQL contention diagnostics with a per-code cap, on top of #346's N2 correlation-field pass-through (previous lock `6563d51bffe4b537d49b0c087a50e86753aaaf5ba533f1e52c7c2214f7f5f04e`):
+`1e62ceb326679c8862c8ae822288b48c54997983a4b011b84d516e943e4f1859`,
+from #342 final reviewed head.
+It supersedes the #346 and #309 locks, the latter `ca110b0bae6d5a06a1002fb80df3f71cffec11659c9f9dbfa5d72624735546f6`,
 from #309 reviewed head `44eeb20a1461ef66b5af492e3be4c42be2728b67`,
 [approval](https://github.com/TinyCloudLabs/tinycloud-node/pull/309#issuecomment-6224650050).
 Current pre-N0 tag/pinned equivalents are separately hash-checked; arbitrary

docs/runbooks/tc-959-n0-log-shipping.md

@@ -31,6 +31,7 @@ Parsing failures become fixed counters; no log-level suppression is used.
 | `node_request`, `svc=api` | `method`, canonical `route` template, integer `status`, nonnegative integer `latency_us`; plus, only when present and exactly well-formed: `request_id` (lowercase UUID, version 1-8), `request_id_replaced` (boolean), `account_hash` and `space_hash` (64 lowercase hex, keyed pseudonyms). A malformed value is dropped and the request still ships. | Individually when valid and eligible under the selected request profile; no sampling |
 | `ingress_request`, `svc=ingress` | Same method/route/status, distinct numeric `request_time_seconds` | Individually when valid and eligible under the selected request profile; no sampling |
 | `node_warning` / `node_error`, `svc=api` | Fixed `event_code`, integer `event_count` | All accepted typed application WARN/ERROR records remain eligible under either profile; grouped counts approximately every 60 seconds |
+| `node_warning` / `node_error`, `svc=api`, TC-1274 SQL contention `event_code` | `event_count=1` plus only that code's declared fields (`log-shipper/diagnostics.json` `typed`): integer `*_ms`, `ahead_approx`, `suppressed_count`, `pool_size`/`pool_idle`/`pool_max`; fixed `op`/`phase`/`kind`/`outcome`/`class`; keyed `space_hash`/`db_hash`, each exactly 64 lowercase hex | Individually, one record per occurrence, under either profile; never reduced; at most 10 per code per 60-second window, excess counted as `typed_throttled` |
 | `logship_gate_summary_total`, `svc=logship` | Fixed `reason`, integer `count` | Count every rejection that reaches the boundary; emit approximately every 60 seconds per fixed reason |
 
 **`request_id` provenance:** `request_id` is the caller's `X-Request-ID` when that header is a single valid UUID (version 1-8, RFC 4122 variant), and a node-generated UUID otherwise (`request_id_replaced=true`). The value therefore carries caller-chosen bits, and a version-1 UUID can embed the client's MAC address and a timestamp, per the #140 contract. The sidecar validates only the shape; it cannot tell a caller-chosen id from a random one. `account_hash` and `space_hash` are the node's keyed pseudonyms, never raw identifiers.
@@ -43,13 +44,19 @@ Sam approved **`errors_slow` as the deployment default**. Compose exposes `TINYC
 - Exactly one second does **not** qualify as slow. A request at that equality still qualifies when its status is at least 400.
 - `all_requests`: explicitly forward every valid node/ingress request, including 1xx, 2xx and 3xx. It is an operator-selected alternative, not an automatic fallback.
 
-Both profiles are unsampled: there is no sampler, random selection, throttle or byte-budget drop. All accepted typed WARN/ERROR diagnostics bypass request-profile filtering and retain their existing fixed-code count aggregation. Invalid records still go to content-free gate counts, not to request output. Profile selection does not make transport durable or imply a volume allowance.
+Both request profiles are unsampled: there is no sampler, random selection, throttle or byte-budget drop for requests. All accepted typed WARN/ERROR diagnostics bypass request-profile filtering. Untyped codes retain their fixed-code count aggregation; the seven TC-1274 SQL contention codes are shipped per occurrence instead, under the explicit per-code cap Sam chose (next section). Invalid records still go to content-free gate counts, not to request output. Profile selection does not make transport durable or imply a volume allowance.
+
+### TC-1274 SQL contention diagnostics
+
+`sql_operation_lock_wait`, `sql_operation_lock_still_waiting`, `sql_operation_lock_held` and `sql_operation_still_running` from `tinycloud_core::sql::service`, `sql_actor_message_slow` from `tinycloud_core::sql::database`, and `db_pool_acquire_timeout` and `db_connect_failed` from `tinycloud_core::db_pool` are matched by exact (target, message) and fixed level. Each record carries only the fields `diagnostics.json` declares for its code, each checked against its closed schema type. A missing field, a wrong type, an out-of-range integer, a value outside its enum, a hash that is not exactly 64 lowercase hex, or a wrong level rejects the whole record into the `invalid_fields` gate count; no partial record is shipped. Undeclared fields, including a raw `db` name, the pool `operation` label, error text and nested values, are never copied. The same message from any other target stays `unclassified`. The field sets and enums must match the emitted F6 PR before promotion.
+
+The shipper caps these records as defence in depth over the node's own per-(space, database, event) limiter, which cannot bound the total when many databases contend. Sam chose the value (decision `tc-959-f6-log-cap`): `diagnostics.json` `typed_limit` allows at most `records` (10) per code, counted separately for each of the seven codes, in each fixed `window_secs` (60 s) window of the Vector process clock. A window boundary can admit up to twice that within one minute, never more per window. This is a deliberate trade-off: during a storm each code yields 10 per-occurrence samples per minute, enough to tell one long actor message from a sustained queue, and the exact `typed_throttled` count shows the true volume. Excess records are dropped, and only their number leaves, as the `typed_throttled` gate reason, so throttling is visible and never silent; any non-zero `typed_throttled` should alert once the Better Stack alerting slice lands. The counter is a Lua transform keyed only by the seven fixed codes. It restarts empty with the process.
 
 Rocket logs template query parts such as `/attestation?<nonce>` and `/public/<space_id>/kv?<prefix>`. Only explicit literal aliases map to canonical paths; arbitrary input is never split at `?` or normalized into a supposedly safe URI. `<unmatched>` is a fixed sentinel. TRACE/CONNECT map to `<other>`; ingress's existing `<other>` route maps to `<unmatched>`. All other unexpected typed fields are rejected and counted, never relayed.
 
 Every node-labelled JSON WARN/ERROR record with a valid formatter timestamp is eligible, including dependency targets, missing targets and missing message fields. Only the actual `tinycloud` / `tinycloud::*` and `tinycloud_core` / `tinycloud_core::*` targets may map known messages to internal fixed codes. Other targets always become `unclassified`; a dependency cannot borrow an internal code by matching its message. No target/message/error value is copied. Plaintext, malformed JSON/timestamps and oversized records remain content-free gate rejections, not a promise to classify unstructured stderr.
 
-**No fatal-panic claim:** 1.20.6's panic hook emits plain text, not a typed JSON panic event. Non-JSON panic lines contribute only to a content-free rejection count if received. N0 neither emits nor declares `node_fatal_panic`, and cannot distinguish a panic from other non-JSON output. Request IDs, replacement flags, account/space HMACs, DB timings and trace propagation require N2. N0 ignores and omits them, including forged input values. Shared future contract: canonical `ops/logging/request-id.md` and `request-id-vectors.json`, version 1, at the revision in `log-shipper/canonical.json`.
+**No fatal-panic claim:** 1.20.6's panic hook emits plain text, not a typed JSON panic event. Non-JSON panic lines contribute only to a content-free rejection count if received. N0 neither emits nor declares `node_fatal_panic`, and cannot distinguish a panic from other non-JSON output. Request IDs, replacement flags, account HMACs, per-request DB timings and trace propagation require N2. N0 ignores and omits them, including forged input values; the only hashes it ships are the SQL contention diagnostics' `space_hash`/`db_hash`. Shared future contract: canonical `ops/logging/request-id.md` and `request-id-vectors.json`, version 1, at the revision in `log-shipper/canonical.json`.
 
 ## Volume: explicit limit, never hidden sampling
 
@@ -67,7 +74,7 @@ Under `errors_slow`, node and ingress eligibility are evaluated independently: a
 
 The representative requests-only rate is **26.39 eligible pairs/minute**, or **26 whole eligible pairs/minute**. A singleton JSON-array framing assumption gives 423 bytes/pair and 26.27/minute; production explicitly uses NDJSON. This is arithmetic, not a claim that either profile fits a budget at actual traffic.
 
-Diagnostics are **additional**, not secretly subtracted, sampled or dropped. The continuous storm model covers all 52 codes at both severities (104 groups/minute) plus nine rejection groups/minute: **811,339,200 bytes/30 days** with maximum-width count records, separate from the request allocation. Real idle/no-diagnostic traffic is lower; grouped reduce flush timing is not a strict calendar-minute emission ceiling. At 60 **eligible pairs/minute** plus that continuous storm, the model totals 1,902,571,200 bytes. [Public Better Stack pricing](https://betterstack.com/pricing), read 2026-10-16, lists 3 GB/month retained for three days. That allowance is shared with other sources; it is not a service allocation, spend approval or assurance that an account cannot exceed it. Measure actual eligible traffic and billing before an authorized promotion; profile approval alone does not establish budget fit.
+Diagnostics are **additional**, not secretly subtracted from the request allocation; only the SQL contention records are sampled, at the explicit cap above. The continuous storm model covers all 59 codes at both severities (118 groups/minute) plus ten rejection groups/minute: **919,123,200 bytes/30 days** with maximum-width count records, separate from the request allocation. Real idle/no-diagnostic traffic is lower; grouped reduce flush timing is not a strict calendar-minute emission ceiling. The SQL contention records are reserved separately at the shipper cap, 7 codes × 10 = 70 records/minute of at most 474 bytes each with the newline: **1,990,800 bytes per storm hour** and **1,433,376,000 bytes/30 days** if sustained for the whole month. At 60 **eligible pairs/minute** plus both storms sustained, the model totals 3,443,731,200 bytes. [Public Better Stack pricing](https://betterstack.com/pricing), read 2026-10-16, lists 3 GB/month retained for three days. That allowance is shared with other sources; it is not a service allocation, spend approval or assurance that an account cannot exceed it. Measure actual eligible traffic and billing before an authorized promotion; profile approval alone does not establish budget fit.
 
 ## Known transport and process risks
 
@@ -146,7 +153,7 @@ Keep these identities distinct:
 | --- | --- |
 | Before: verifier `docker_compose_file` / hash | Raw preserved pre-N0 compose: `df239749ca0a9c09893a855ae2a8abb94090b8556515ad002105c58ac91f4c91` |
 | Before: application `compose_hash` | `bfe9028a58b23238188aa7d70b82282a7cd6414cd83dcc47a3b295c6f64fe2f8`; a distinct reported identity, not the raw-file hash |
-| After: current profile-enabled raw compose | `6563d51bffe4b537d49b0c087a50e86753aaaf5ba533f1e52c7c2214f7f5f04e`; bind this artifact to the final reviewed PR head before any promotion |
+| After: current profile-enabled raw compose | `1e62ceb326679c8862c8ae822288b48c54997983a4b011b84d516e943e4f1859`; bind this artifact to the final reviewed PR head before any promotion |
 
 The raw after hash identifies the profile-enabled compose, not deployment approval. Record its final reviewed PR head and required CI evidence in the officer handoff. Do not infer a raw-file hash from the distinct application-reported compose hash.