adaptive_export: restore dark-vector tracepoint table export onto main (8 dx_* tables) - #89
Conversation
|
@ConstanzeTU (build-agent) — TTL image request 🏗️ Could you build a TTL adaptive_export image from this branch head (
Branch builds clean locally ( |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughEight dark-vector tracepoint tables and three export presets are added. PxL enrichment now supports pid-keyed dark-vector data, while ClickHouse registration, plugin initialization, passthrough query timeouts, environment normalization, and mypy cache behavior are updated. ChangesDark-vector export support
Adaptive export runtime configuration
Tooling configuration
Estimated code review effort: 3 (Moderate) | ~30 minutes Sequence Diagram(s)sequenceDiagram
participant AdaptiveExport
participant QueryFor
participant PodEnrichPxL
participant process_stats
participant ClickHouse
AdaptiveExport->>QueryFor: request dark-vector table data
QueryFor->>PodEnrichPxL: generate table-specific PxL
PodEnrichPxL->>process_stats: merge process metadata by pid
process_stats-->>PodEnrichPxL: return pod and namespace
PodEnrichPxL-->>QueryFor: return enriched PxL and pod filter
QueryFor->>ClickHouse: write rows to registered table
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/vizier/services/adaptive_export/internal/passthrough/passthrough.go`:
- Around line 73-80: Wire the new passthrough.Config.QueryTimeout field through
the production configuration path in cmd/main.go, including its environment or
configuration source and parsing alongside the existing passthrough settings.
Ensure the constructed Config passes the configured duration to the passthrough
service instead of relying on the hardcoded default.
- Around line 73-79: Align the QueryTimeout documentation with the value
assigned by New: either change New’s default from 150 seconds to the documented
180-second OrderQuery budget, or update the QueryTimeout comment to accurately
describe 150 seconds. Keep the comment and runtime default consistent.
In `@src/vizier/services/adaptive_export/internal/pxl/compile.go`:
- Around line 80-101: Update PodEnrichPxL to accept the query’s relStart value
and use it for the process_stats DataFrame instead of the hardcoded “-5m”
window. Pass the computed relStart from CompilePassthrough and QueryFor, and
adjust affected tests to cover historical query ranges.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 86fd4c39-1c08-4a70-b694-8ec5b64b1ac6
📒 Files selected for processing (9)
src/vizier/services/adaptive_export/internal/clickhouse/apply.gosrc/vizier/services/adaptive_export/internal/clickhouse/ddl.gosrc/vizier/services/adaptive_export/internal/clickhouse/schema.sqlsrc/vizier/services/adaptive_export/internal/passthrough/passthrough.gosrc/vizier/services/adaptive_export/internal/pxl/compile.gosrc/vizier/services/adaptive_export/internal/pxl/compile_test.gosrc/vizier/services/adaptive_export/internal/pxl/queryfor.gosrc/vizier/services/adaptive_export/internal/pxl/tables.gosrc/vizier/services/adaptive_export/internal/pxl/tables_test.go
| // QueryTimeout bounds a single table's pixie query (entlein/dx#7). The | ||
| // firehose pull used to bound query+write by Refresh, which is far too tight | ||
| // for a heavy protocol: pgsql_events carries full SQL text and its | ||
| // socket_tracer parse is expensive, so the ExecuteScript deadline-exceeded and | ||
| // pgsql_events landed 0 rows in forensic_db. Decoupled from Refresh and | ||
| // defaulted generous (matches the OrderQuery path's 180s budget). | ||
| QueryTimeout time.Duration |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Correct the timeout documentation mismatch.
The comment says the default matches an OrderQuery 180-second budget, but New sets QueryTimeout to 150 seconds. Update either the default or the documentation so operators are not misled.
Also applies to: 116-118
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/vizier/services/adaptive_export/internal/passthrough/passthrough.go`
around lines 73 - 79, Align the QueryTimeout documentation with the value
assigned by New: either change New’s default from 150 seconds to the documented
180-second OrderQuery budget, or update the QueryTimeout comment to accurately
describe 150 seconds. Keep the comment and runtime default consistent.
| // QueryTimeout bounds a single table's pixie query (entlein/dx#7). The | ||
| // firehose pull used to bound query+write by Refresh, which is far too tight | ||
| // for a heavy protocol: pgsql_events carries full SQL text and its | ||
| // socket_tracer parse is expensive, so the ExecuteScript deadline-exceeded and | ||
| // pgsql_events landed 0 rows in forensic_db. Decoupled from Refresh and | ||
| // defaulted generous (matches the OrderQuery path's 180s budget). | ||
| QueryTimeout time.Duration | ||
| Tables []string |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Wire QueryTimeout into production configuration.
cmd/main.go constructs passthrough.Config without setting this field, so production always receives the hardcoded 150-second default. If this setting is intended to be configurable, add the corresponding environment/configuration wiring; otherwise, document that it is fixed.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/vizier/services/adaptive_export/internal/passthrough/passthrough.go`
around lines 73 - 80, Wire the new passthrough.Config.QueryTimeout field through
the production configuration path in cmd/main.go, including its environment or
configuration source and parsing alongside the existing passthrough settings.
Ensure the constructed Config passes the configured duration to the passthrough
service instead of relying on the hardcoded default.
|
TTL adaptive_export image built ✅ — run 29839326430, built from PR head Pull either tag (same digest Only the AE image was rebuilt — kelvin/pem/metadata untouched. Also fixed the failing |
|
@ConstanzeTU (build-agent) — re-request TTL at new head Folded dc_snoop into the dark-table export and removed execve: Could you rebuild the TTL adaptive_export image from |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/vizier/services/adaptive_export/internal/pxl/compile.go (1)
64-66: 🎯 Functional Correctness | 🔴 Critical | 🏗️ Heavy liftPass the query window into dark-vector enrichment.
PodEnrichPxL(table)still cannot use the caller’srelStart, soprocess_statsenrichment remains limited to the fixed five-minute window. Queries older than five minutes can therefore lose pod and namespace attribution. Accept and pass the computed query window, then add a regression test for a historical query.Also applies to: 80-102
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/vizier/services/adaptive_export/internal/pxl/compile.go` around lines 64 - 66, Update the compile flow around PodEnrichPxL to accept and forward the computed relStart and relEnd query window, and use those bounds when generating process_stats enrichment instead of the fixed five-minute range. Propagate the new parameters through the relevant enrichment helper while preserving existing output, and add a regression test covering a query older than five minutes with pod and namespace attribution.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/vizier/services/adaptive_export/internal/pxl/compile.go`:
- Around line 64-66: Update the compile flow around PodEnrichPxL to accept and
forward the computed relStart and relEnd query window, and use those bounds when
generating process_stats enrichment instead of the fixed five-minute range.
Propagate the new parameters through the relevant enrichment helper while
preserving existing output, and add a regression test covering a query older
than five minutes with pod and namespace attribution.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 1337e953-a2b1-414f-852d-d79224cc5313
📒 Files selected for processing (7)
src/vizier/services/adaptive_export/internal/clickhouse/apply.gosrc/vizier/services/adaptive_export/internal/clickhouse/ddl.gosrc/vizier/services/adaptive_export/internal/clickhouse/schema.sqlsrc/vizier/services/adaptive_export/internal/pxl/compile.gosrc/vizier/services/adaptive_export/internal/pxl/compile_test.gosrc/vizier/services/adaptive_export/internal/pxl/tables.gosrc/vizier/services/adaptive_export/internal/pxl/tables_test.go
|
TTL adaptive_export image rebuilt ✅ (dc_snoop fold, execve dropped) — run 29848515911, built from new head Pull either tag (same digest AE-only rebuild, |
|
@ConstanzeTU one more TTL at |
|
TTL adaptive_export image rebuilt ✅ (dark-vector pod filter fix: Pull either tag (same digest AE-only, |
|
@ConstanzeTU TTL at |
|
TTL adaptive_export image built ✅ (3 boot-registered retention/export scripts: ch-dc_snoop, ch-stack_trace, ch-creds_change) — run 29860622585, from head Pull either tag (same digest AE-only, |
|
@ConstanzeTU — ready for linter fix + release �my end is validated 🎯 Root-caused and fixed the retention-script registration failure: the pixie-api-key had a trailing newline (from a Two asks:
Then I'll deploy aeprod35 on a fresh rig and confirm end-to-end (scripts register + tables export). Thanks! |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/vizier/services/adaptive_export/internal/config/config.go`:
- Around line 166-174: Update the debug logging in the configuration-loading
flow to stop emitting credential-bearing values, especially pixieAPIKey and
clickhouseDSN. Log only presence/status for the API key and a sanitized
connection description for the DSN, while preserving non-sensitive configuration
logging and behavior in the surrounding config loader.
- Line 174: Update the PIXIE_ENDPOINT handling around getEnvWithDefault and
pixieHost so whitespace-only values receive defPixieHostname after trimming,
allowing the existing Kubernetes fallback check to run. Apply trimming before
the defaulting decision or within getEnvWithDefault, and add a regression test
covering a whitespace-only PIXIE_ENDPOINT.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: f7b7804e-f8aa-4510-9b8c-46478995a9ae
📒 Files selected for processing (2)
src/vizier/services/adaptive_export/internal/config/config.gosrc/vizier/services/adaptive_export/internal/script/presets/stack_trace.pxl
| clickhouseDSN := strings.TrimSpace(os.Getenv(envClickHouseDSN)) | ||
| pixieClusterID := strings.TrimSpace(os.Getenv(envPixieClusterID)) | ||
| // TrimSpace: a secret sourced via `kubectl --from-file` keeps the file's | ||
| // trailing newline. In the pixie-api-key gRPC metadata header that newline is | ||
| // an HTTP/2 protocol violation → the cloud PluginService replies RST_STREAM | ||
| // PROTOCOL_ERROR (looks like an auth failure but isn't). Trim it defensively. | ||
| pixieAPIKey := strings.TrimSpace(os.Getenv(envPixieAPIKey)) | ||
| clusterName := strings.TrimSpace(os.Getenv(envClusterName)) | ||
| pixieHost := strings.TrimSpace(getEnvWithDefault(envPixieEndpoint, defPixieHostname)) |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Do not log the API key or credential-bearing DSN.
These values are emitted verbatim by the debug logs below, including pixieAPIKey and clickhouseDSN. Redact credentials and log only presence/status or sanitized connection details.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/vizier/services/adaptive_export/internal/config/config.go` around lines
166 - 174, Update the debug logging in the configuration-loading flow to stop
emitting credential-bearing values, especially pixieAPIKey and clickhouseDSN.
Log only presence/status for the API key and a sanitized connection description
for the DSN, while preserving non-sensitive configuration logging and behavior
in the surrounding config loader.
| // PROTOCOL_ERROR (looks like an auth failure but isn't). Trim it defensively. | ||
| pixieAPIKey := strings.TrimSpace(os.Getenv(envPixieAPIKey)) | ||
| clusterName := strings.TrimSpace(os.Getenv(envClusterName)) | ||
| pixieHost := strings.TrimSpace(getEnvWithDefault(envPixieEndpoint, defPixieHostname)) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n -A8 -B4 'func getEnvWithDefault|pixieHost :=|envPixieEndpoint' \
src/vizier/services/adaptive_export/internal/config/config.goRepository: k8sstormcenter/pixie
Length of output: 1780
🏁 Script executed:
#!/bin/bash
sed -n '160,240p' src/vizier/services/adaptive_export/internal/config/config.go
printf '\n---\n'
sed -n '330,350p' src/vizier/services/adaptive_export/internal/config/config.go
printf '\n---\n'
rg -n 'defPixieHostname|pixieHost|Kubernetes|fallback|work.pixie' src/vizier/services/adaptive_export/internal/config/config.goRepository: k8sstormcenter/pixie
Length of output: 4941
Apply the default after trimming
PIXIE_ENDPOINT values that are only whitespace become "" after TrimSpace, so the later pixieHost == defPixieHostname check skips the Kubernetes fallback. Trim before defaulting, or trim inside getEnvWithDefault, and add a regression test.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/vizier/services/adaptive_export/internal/config/config.go` at line 174,
Update the PIXIE_ENDPOINT handling around getEnvWithDefault and pixieHost so
whitespace-only values receive defPixieHostname after trimming, allowing the
existing Kubernetes fallback check to run. Apply trimming before the defaulting
decision or within getEnvWithDefault, and add a regression test covering a
whitespace-only PIXIE_ENDPOINT.
|
CI linter green + release cut ✅ Linter — the red
All 8 PR checks green at Release — cut
|
|
Release
Cut from |
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/vizier/services/adaptive_export/cmd/main.go`:
- Around line 267-274: Update the retry handling around
EnsureClickHousePluginEnabled so the warning log and six-second sleep execute
only when attempt < 5; preserve the existing retry behavior and final error
handling after the fifth failed attempt.
- Around line 261-268: Update the success logging in the retry flow around
EnsureClickHousePluginEnabled so it never emits the credential-bearing exportURL
returned from NativeDSN(). Log only plugin enabled/status information, or use a
properly redacted DSN if the endpoint must be included; preserve the existing
retry and enablement behavior.
In `@src/vizier/services/adaptive_export/internal/config/config.go`:
- Line 473: Format the clickhouse Host method and the surrounding config.go file
using the repository’s standard Go formatter or lint target, ensuring the
formatting error reported at Host is resolved without changing behavior.
- Around line 470-471: Update clickhouse.NativeDSN to construct the URL with
url.UserPassword for escaping credentials and net.JoinHostPort for correctly
formatting the host and port, while preserving the database component. Add a
regression test covering user and password values containing reserved characters
such as @, :, /, and %.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 85fe279c-2286-4218-a1e0-8d4ce4d3bcac
📒 Files selected for processing (6)
mypy.inisrc/vizier/services/adaptive_export/cmd/main.gosrc/vizier/services/adaptive_export/internal/config/config.gosrc/vizier/services/adaptive_export/internal/script/presets/creds_change.pxlsrc/vizier/services/adaptive_export/internal/script/presets/dc_snoop.pxlsrc/vizier/services/adaptive_export/internal/script/presets/stack_trace.pxl
|
TTL adaptive_export image built ✅ (AE deploys dark-vector bpftraces at boot; schema/event_time + native-DSN fixes) — run 29940139432, from head Pull either tag (same digest AE-only, |
|
TTL adaptive_export image built ✅ (confirm tracepoint deploy by table, not the mutation stream) — run 29943812599, from head Pull either tag (same digest AE-only, |
|
TTL adaptive_export image built ✅ (pid→pod/namespace enrichment for dc_snoop + creds_change) — run 29954929754, from head Pull either tag (same digest AE-only, |
|
TTL adaptive_export image built ✅ (full k8s metadata enrichment: namespace/pod/container/node) — run 29958451404, from head Pull either tag (same digest AE-only, |
|
TTL adaptive_export image built ✅ (AE auto-creates dc_snoop/creds_change/stack_trace canonical dark tables) — run 29988774397, from head Pull either tag (same digest AE-only, |
|
TTL adaptive_export image built ✅ (dark tables carry consistent full k8s metadata + nanosecond-ts/single-auth invariant tests) — run 29991926285, from head Pull either tag (same digest AE-only, |
|
TTL adaptive_export image built ✅ (ASCII-only .pxl comments lint fix) — run 30037696068, from head Pull either tag (same digest AE-only, |
|
TTL adaptive_export image built ✅ — run 30039555017, from head This head = your Pull either tag (same digest AE-only, |
Live ClickHouse evidence — AE aeprod44, rig 6a632fbf (pixie-only)2026-07-24T17:36:52Z · fresh deploy: AE auto-creates all tables (ingest_writer/apply.go), self-deploys the bpftraces, exports via the retention plugin. dx not deployed — pure AE producer output. Single-pod cluster-setup leader election (aeprod44) → no duplicate registrations. Dedup fix — 14 cron scripts (not 28), 0% duplication
(Before the leader-election fix: 28 cron scripts = 2x the 14 presets → 35% duplicate rows. Now 14 scripts → 0%.) dc_snoop — full k8s attribution (namespace / pod / container / node)
creds_change — privilege-escalation calibration, captured ONCE, fully attributed
Nanosecond timestamps (event_time = time_, DateTime64(9))
Calibration (TestCredsChangeCalibration) PASS: trace fires → AE exports → forensic_db.creds_change carries pid+comm+namespace/pod/container/node, escalation stored exactly once. All AE invariant unit tests green (nanosecond timestamps, single-auth, full-metadata, leader election). |
|
@build-agent please cut a clean AE release for the new tip Context: I cut Ask: re-cut the release from |
|
Re-cutting as annotated Confirmed your diagnosis: |
|
Release
Same commit |
…iltin golangci flagged main.go: (1) gci/gofumpt formatting — the new env-var consts and controller.Config fields broke alignment; (2) predeclared — leaderNode's local var `min` shadows the Go 1.21 builtin. Reformatted with gci+gofumpt (repo sections standard/default/prefix(px.dev)) and renamed the var to `smallest`. Builds clean; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com>
Two CI lint failures from the config-driven-presets work: - internal/script/BUILD.bazel: presets_test.go was added without the gazelle-generated pl_go_test target -> 'Gazelle was not run'. Added the script_test target (matches gazelle diff + cmd/BUILD.bazel style). - dc_snoop.pxl: the runtime sentinel '#__DC_SNOOP_EXCLUSION__' tripped flake8 E265 (block comment needs '# '). Renamed to '# __DC_SNOOP_EXCLUSION__' and updated the matching strings.Replace target in presets.go so the env-injected exclusion still substitutes byte-for-byte. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com>
…nce lands The dark tables (dc_snoop/creds_change/dx_*) came back EMPTY: the AE filtered them by pod, but an incident's transient malignant pids (whoami/cat/getent children) are too short-lived to enter process_stats, so their ns/pod resolves blank and the filter dropped exactly the evidence. The AE is node-local (pem-direct → the node's own PEM), so QueryFor now keeps every dark row in the window and only drops the infra/self comms (DC_SNOOP_EXCLUDE_COMMS, env-tunable) — the workload's dark activity is captured node-scoped, no relevance filtering. Signed-off-by: entlein <einentlein@gmail.com>
pem-direct is the robust query path for the node-scoped AE: node-local (matches its scope), desync-immune (bypasses the kelvin/broker aggregation that the recurring PEM desync silently breaks) and fast. Default to HOST_IP:50305 when the deploy provides HOST_IP (downward API) + PL_JWT_SIGNING_KEY; explicit ADAPTIVE_VIZIER_DIRECT_ADDR still wins; cloud passthrough only as fallback. Signed-off-by: entlein <einentlein@gmail.com>
The stray adaptive_export ELF binary got committed at the repo root again (a repo-root `go build` output). Removed it and added `/cmd` to .gitignore so it stops recurring. No source/BUILD change; the real binary target is //src/vizier/services/adaptive_export/cmd. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com>
Follow-up to fd9108a, which removed the binary but did not land the .gitignore rule. Adds `/cmd` so a repo-root `go build` output stops getting re-committed. No source/BUILD change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com>
…insert=0 writes Two fresh-rig blockers for the full evidence set: 1. pem-direct (:50305) serves fast node-local queries but REFUSES mutations (direct-query: mutations out of scope #29), so the bpftrace deploy failed and the dark tables (dc_snoop/creds_change/stack_trace) stayed empty. Deploy the tracepoints via the in-cluster broker (:50300) with the same JWT when the query adapter is pem-direct; keep querying via pem-direct. 2. Fresh ClickHouse defaults async_insert=1 → the AE's INSERT buffered and returned written_rows=0 (evidence trickled in minutes later / looked lost). Write with SETTINGS async_insert=0 so evidence lands + is counted immediately. Signed-off-by: entlein <einentlein@gmail.com>
…ess_stats merge) The dc_snoop node-scope query timed out / silently dropped: it merged the node's ENTIRE dark stream (Formatter/vector/runc/... thousands of rows/window) against process_stats before filtering. Reorder: drop the infra/self comms FIRST, then merge — the pid-join now runs on the handful of workload rows so the dark capture completes reliably. Also expand the default comm-exclusion (runc:[2:INIT], CgrpMemUsgObsr, Formatter, iptables-save, vector-worker, metrics-server, ...) in CODE, since the env value with '[' ':' breaks the PxL filter. Signed-off-by: entlein <einentlein@gmail.com>
…ne unit) The control API (dx -> AE: /export/start t_end, /query window) carries unix timestamps, and the evidence pipeline's ONE unit is nanoseconds (event_time, dx referral windows). The handlers read them with time.Unix(v, 0) — i.e. as SECONDS — so dx's nanosecond t_end (~1.78e18) became a year-56-billion instant. Effect: every dx-steered OrderExportAll captured [t_end-600s, t_end] over that garbage instant, a window that overlaps NO data, so the full-evidence set (all dark tables: dc_snoop/conn_stats/creds_change/stack_trace, and every protocol table) silently returned zero rows on the dx-steered path. Only the self-steer path (its own now-based window) ever wrote anything, which is why redis_events populated but the dx-ordered dark capture never did. The same garbage window also flowed into ae_reconcile.win_start/win_end, overflowing the DateTime formatter and making CH reject the reconcile insert (HTTP 400) — which hid the whole failure from the reconcile instrument. Fix: read the control timestamps as nanoseconds, time.Unix(0, ns), matching the pipeline. No unit autodetection — one unit, everywhere. Test + struct doc-comments updated to nanoseconds. Signed-off-by: entlein <einentlein@gmail.com>
The dark-vector pod-resolution merge scanned process_stats over a 5-minute window. On a busy node that table samples every live pid every ~10-30s, so the scan is large and is the dominant cost of the dark query — heavy enough that, sharing the fan-out's query-slot budget with the fast native-table queries, the dark capture either starved (too few slots) or, once the steer windows were real, saturated the node-local PEM (too many). A 2-minute window still resolves the pods that matter — long-lived workload pids (redis-server) are sampled continuously — while transient attack pids never enter process_stats and resolve blank either way. Cuts the merge scan ~2.5x so the dark tables complete within the fan-out budget alongside the native queries. Signed-off-by: entlein <einentlein@gmail.com>
…ral event keys The evidence tables were plain MergeTree, so the dx-steered OrderExportAll — which re-pulls a rolling 600s window every ExportAllFloor — re-inserted the SAME kernel/ protocol events as fresh rows on every overlapping capture. dc_snoop showed 4,323 rows for one redis incident where the true unique count is a fraction of that. Switch the seven evidence tables to ReplacingMergeTree keyed on each event's NATURAL identity so re-pulls of the same event collapse: dc_snoop (time_, pid, comm, t, file, pod) creds_change (time_, pid, comm, old_uid, new_uid, pod) stack_trace (time_, upid, stack_trace_id, pod) redis_events (hostname, event_time, time_, upid, trace_role, remote_port, local_port, latency, req_cmd) dns_events (... , req_body) http_events (... , req_method, req_path) conn_stats (hostname, event_time, time_, upid, remote_addr, remote_port, trace_role) Keys are deliberately conservative — nanosecond time_ + pid/upid + connection tuple + a payload discriminator — so two DISTINCT events never share a key (false-collapse would drop real evidence, worse than a dup). Dedup is applied on background merge; exact counts read with FINAL / count(DISTINCT key). Bookkeeping tables already used ReplacingMergeTree; this brings the evidence tables in line. Signed-off-by: entlein <einentlein@gmail.com>
The per-anomaly fan-out (QueryFor / OrderExportAll) queried px.DataFrame(table= 'stack_trace') — the ClickHouse table name, which is NOT a Pixie table — so the stack_trace evidence table never populated on the steered path. The native continuous profiler is 'stack_traces.beta' (upid-keyed, always-on, no tracepoint); only the retention preset used it, and that path is disabled. Fix: pixieSourceFor() maps stack_trace -> 'stack_traces.beta' for the DataFrame source (dotted-name DataFrames compile fine in a direct query — verified live), and a dedicated QueryFor branch resolves namespace/pod/container/hostname via ctx + upid_to_node_name and stamps event_time=time_, exactly like the export preset, then scopes to the alert's pod with the namespaced '<ns>/<pod>' key (ctx['pod'] is namespaced — verified live: pl/vizier-metadata-0 -> 112 samples). stack_trace is upid-native, NOT a dark-vector pid-merge table, so it takes the native resolution path, not the process_stats merge. Verified on a live cluster: the profiler emits across all pods, and the exact generated query returns real samples for a filtered pod. Signed-off-by: entlein <einentlein@gmail.com>
e75fb36 to
517e159
Compare
|
@build-agent — could you cut a TTL adaptive_export image (ttl.sh, ~24h) from branch
Need it for re-verify on rig 6a7613ce (expect |
|
TTL adaptive_export image built ✅ (ppid capture fix + ppid ancestry filter) — run 31211366899, from Pull either tag (same digest (GHCR tag, same throwaway image as always.) AE-only, |
|
@build-agent — one more TTL adaptive_export image please, from |
|
TTL adaptive_export image built ✅ (refresh tracepoints on boot: delete-then-upsert) — run 31212204755, from Pull either tag (same digest (GHCR tag, same throwaway image.) AE-only, |
|
@build-agent — TTL adaptive_export image please, from |
|
TTL adaptive_export image built ✅ (drop 9th printf arg — fixes bpftrace budget overflow / zero capture) — run 31215728948, from Pull either tag (same digest (GHCR tag, same throwaway image.) AE-only, |
|
@build-agent — TTL adaptive_export image from |
|
TTL adaptive_export image built ✅ (restore capturing tracepoint form + %llu ppid fix) — run 31217304334, from Pull either tag (same digest (GHCR tag, same throwaway image.) AE-only, |
|
@build-agent — TTL adaptive_export image from |
|
TTL adaptive_export image built ✅ (final form: revert delete-then-upsert, plain UpsertTracepoint) — run 31218540088, from Pull either tag (same digest (GHCR tag, same throwaway image.) AE-only, |
|
@build-agent — TTL adaptive_export image from |
|
TTL adaptive_export image built ✅ (delete hardcoded comm blocklist → ppid ancestry cut) — run 31246012852, from Pull either tag (same digest (GHCR tag, same throwaway image.) AE-only, |
|
@build-agent — please rebuild a TTL adaptive_export image from |
|
TTL adaptive_export image built ✅ (#94-on-#92 stack: dx-steered pem-direct + dc_snoop ppid/ancestry) — run 31397944896, from Pull either tag (same digest (GHCR tag, same throwaway image.) AE-only, |
|
@build-agent — please build the AE (adaptive_export) image from branch Context: #89 restored dark-vector export but only in the retention builder; the streaming TableScanner ( Reply with |
2 participants
Summary: Restore the adaptive_export dark-vector tracepoint table export onto main. The dark-table export was built on
feat/ae-dark-tables-pgsql-steering(ec8382588+ pid→pod enrichmentddf6d18e3) and never merged, so main and everyaeprod29→aeprod34/v0.14.20build carry only the 13 protocol tables. This cherry-picks the two dark-table commits cleanly onto main: 8dx_*tables (dx_execve,dx_vfs_events,dx_unlink,dx_dlookup,dx_mprotect,dx_creds,dx_bpf,dx_ptrace) with their ClickHouse DDL, plusIsDarkVector+PodEnrichPxLpid-keyed enrichment (dark tables have noupid, so pod is resolved via aprocess_statspid-merge instead ofupid_to_pod_name), and the pgsql passthrough query timeout from the same commit. Net export set becomes 8 dark + 13 protocol = 21 tables; the 6 control/evidence tables are unchanged.Test Plan:
go build ./src/vizier/services/adaptive_export/...passes;go test ./src/vizier/services/adaptive_export/...is green across 14 packages with 0 failures; theschema.sqlembed is already covered by the existingembedsrcson main so no BUILD.bazel change is needed. Live validation against deployed tracepoints on a v0.14.20 rig follows on the TTL adaptive_export image before the release build is cut.Type of change: /kind feature
Changelog Message: adaptive_export now exports the 8 dark-vector tracepoint tables (
dx_*) alongside the 13 protocol tables, with pid-keyed pod enrichment for the dark tables.Full context (Why / What / Validation)
Why
The AE dark-vector tracepoint table export was built on
feat/ae-dark-tables-pgsql-steering(commitec8382588+ pid→pod enrichmentddf6d18e3) and never merged to main.merge-base --is-ancestorconfirms it: main'sinternal/pxl/tables.gohas 0dx_*tables. Theaeprod29→aeprod34line (and the currentv0.14.20build) were all cut from the main line and carry only the 13 protocol tables — the dark tables were only ever in the one-offaeprod28-rc1image.The dx generic-precorrelation loop needs AE to export the full evidence slice — all dark tables + all protocols per kubescape anomaly — so this restores the dark-table export onto the current (fixed) main AE rather than reviving the stale
aeprod28-rc1.What
Cherry-picked the two dark-table commits cleanly onto
main(no conflicts):ec8382588— 8 dark-vector tables inbuiltinTables(dx_execve,dx_vfs_events,dx_unlink,dx_dlookup,dx_mprotect,dx_creds,dx_bpf,dx_ptrace) + their CH DDL (apply.goOperatorOwnedTables,ddl.go,schema.sql).ddf6d18e3—IsDarkVector+PodEnrichPxLinQueryFor: the dark tables are pid-keyed (noupid), so pod is resolved via aprocess_statspid-merge (bare pod name) rather thanupid_to_pod_name. (Also carries the pgsql firehose timeout from the same commit.)Net export set after this PR is 8 dark + 13 protocol = 21 tables; the 6 control/evidence tables (
adaptive_attribution,ae_reconcile,trigger_watermark,dx_evidence_graph{,_malignant},dx_evidence_manifest) are unchanged.Validation
go build ./src/vizier/services/adaptive_export/...— OKgo test ./src/vizier/services/adaptive_export/...— 14 packages green, 0 failschema.sqlembed already covered byembedsrcson main (dark DDL appends to the existing schema) — no BUILD.bazel change needed.Live validation of the exported dark tables (against deployed tracepoints on a v0.14.20 rig) to follow on the TTL image.