Repository navigation
TC-959: pass validated N2 correlation fields through the log sidecar - #346
Merged
Merged
Conversation
…ode_request request_id (UUID v1-8), request_id_replaced (bool), account_hash and space_hash (64 lowercase hex) are copied only when exactly well-formed; anything else is dropped and the request still ships. Ingress rows, diagnostics and db_hash are unchanged. Regenerates the embedded Vector config and counts the widest valid node request in the byte budget.
Contributor
Author
|
REQUEST_CHANGES: see TC-959 (privacy/security review of head 75fd628) |
… assertCapture, anchor tests, request_id provenance note
Contributor
Author
|
REQUEST_CHANGES: see TC-959 (privacy/security re-review of head f0e360e; one-line provenance fix in docs/releasing.md:297-300) |
Contributor
Author
|
APPROVE: see TC-959 (privacy/security review of head 94592b7; pending a green rerun of test (workspace, PostgreSQL-backed), attempt 1 failed only the known TC-1071 flake) |
samgbafa
marked this pull request as ready for review
October 17, 2026 22:22
This was referenced Oct 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Request rows reach Better Stack without
request_id,account_hashorspace_hashbecause the sidecar sanitizer buildsnode_requestfrom fixed fields only. N2 (#305, merged) emits them, but they never leave the CVM, so node rows can't be joined to each other or to F6 events (TC-1274 evidence).Change (sidecar only; no node code):
log-shipper/sanitize.vrl: onnode_request, copyrequest_id(lowercase UUID, version 1-8, variant 8-b),request_id_replaced(boolean),account_hashandspace_hash(64 lowercase hex) only when they match exactly. A malformed or raw value is dropped and the request still ships.fields.jsonalready reserves all four with the same patterns, so the canonical pin is unchanged.db_hashare untouched.db_hashstays F6-only (TC-959: ship typed TC-1274 SQL contention diagnostics through the log sidecar #342). The nginx ingress log carries no request id yet, so node/ingress joins are not enabled by this PR.cases.mjs: the old "valid correlation fields are discarded" case is replaced. New cases: valid fields pass exactly, absent hashes stay absent, malformed ids/hashes dropped, raw DID/space identifiers in a hash field dropped, uppercase / wrong-variant / version-9 ids dropped, diagnostics and ingress cannot carry them.VECTOR_CONFIG_JSONregenerated in the compose and the CI fixture (the generator reproduced the previous line byte for byte before the change).Evidence:
vector test69/69 on Vector 0.59.0 (errors_slow); the new valid-passthrough, absent-hash and raw-identifier cases fail against the old sanitizer (RED), pass now.node --test log-shipper scripts32/32,native-checkpassed,tc959-check(embedded parity) passed. Docker compose proof not run here (host hold).Privacy: widens what leaves the CVM by four request-level fields: a random request UUID and two keyed HMAC-SHA256 pseudonyms (secret-derived, per N2). No raw identifiers. Needs a privacy review before merge. Targets 1.21.0 with N2. Whichever of this and #342 lands second rebases (both touch
log-shipper/and the compose line).