Skip to content

TC-959: pass validated N2 correlation fields through the log sidecar - #346

Merged
samgbafa merged 3 commits into
mainfrom
skgbafa/tc-959-correlation-passthrough
Oct 17, 2026
Merged

samgbafa merged 3 commits into
mainfrom
skgbafa/tc-959-correlation-passthrough

Conversation

@samgbafa

Copy link
Copy Markdown
Contributor

Request rows reach Better Stack without request_id, account_hash or space_hash because the sidecar sanitizer builds node_request from fixed fields only. N2 (#305, merged) emits them, but they never leave the CVM, so node rows can't be joined to each other or to F6 events (TC-1274 evidence).

Change (sidecar only; no node code):

  • log-shipper/sanitize.vrl: on node_request, copy request_id (lowercase UUID, version 1-8, variant 8-b), request_id_replaced (boolean), account_hash and space_hash (64 lowercase hex) only when they match exactly. A malformed or raw value is dropped and the request still ships. fields.json already reserves all four with the same patterns, so the canonical pin is unchanged.
  • Ingress rows, WARN/ERROR diagnostics and db_hash are untouched. db_hash stays F6-only (TC-959: ship typed TC-1274 SQL contention diagnostics through the log sidecar #342). The nginx ingress log carries no request id yet, so node/ingress joins are not enabled by this PR.
  • cases.mjs: the old "valid correlation fields are discarded" case is replaced. New cases: valid fields pass exactly, absent hashes stay absent, malformed ids/hashes dropped, raw DID/space identifiers in a hash field dropped, uppercase / wrong-variant / version-9 ids dropped, diagnostics and ingress cannot carry them.
  • Embedded VECTOR_CONFIG_JSON regenerated in the compose and the CI fixture (the generator reproduced the previous line byte for byte before the change).
  • Byte budget counts the widest valid node request with the fields: node 439 B, pair 421 to 662 B. Whole requests/min inside the 0.48 GB requests-only allocation falls from 26 to 16. Observed errors/slow volume is about 31 node rows/min today (about 7.7 MB so far in the 3 GB shared plan), so this fits the shared plan but exceeds the requests-only allocation if it grows.

Evidence: vector test 69/69 on Vector 0.59.0 (errors_slow); the new valid-passthrough, absent-hash and raw-identifier cases fail against the old sanitizer (RED), pass now. node --test log-shipper scripts 32/32, native-check passed, tc959-check (embedded parity) passed. Docker compose proof not run here (host hold).

Privacy: widens what leaves the CVM by four request-level fields: a random request UUID and two keyed HMAC-SHA256 pseudonyms (secret-derived, per N2). No raw identifiers. Needs a privacy review before merge. Targets 1.21.0 with N2. Whichever of this and #342 lands second rebases (both touch log-shipper/ and the compose line).

…ode_request

request_id (UUID v1-8), request_id_replaced (bool), account_hash and space_hash
(64 lowercase hex) are copied only when exactly well-formed; anything else is
dropped and the request still ships. Ingress rows, diagnostics and db_hash are
unchanged. Regenerates the embedded Vector config and counts the widest valid
node request in the byte budget.
@samgbafa

Copy link
Copy Markdown
Contributor Author

REQUEST_CHANGES: see TC-959 (privacy/security review of head 75fd628)

… assertCapture, anchor tests, request_id provenance note
@samgbafa

Copy link
Copy Markdown
Contributor Author

REQUEST_CHANGES: see TC-959 (privacy/security re-review of head f0e360e; one-line provenance fix in docs/releasing.md:297-300)

@samgbafa

Copy link
Copy Markdown
Contributor Author

APPROVE: see TC-959 (privacy/security review of head 94592b7; pending a green rerun of test (workspace, PostgreSQL-backed), attempt 1 failed only the known TC-1071 flake)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant