This PR tracks the changes of the fork vs upstream - #61
Draft
entlein wants to merge 40 commits into
Draft
Conversation
… -> ContainerProfile ConvertUserProfilesToContainerProfile produces the single user-defined ContainerProfile equivalent to a legacy user-authored ApplicationProfile + NetworkNeighborhood pair, reusing the existing projectUserProfiles merge onto an empty base. The differential oracle pins the migration contract at the enforcement level: for representative user-defined shapes (opens/exec argv wildcards, HTTP endpoints, egress/ingress + LabelSelector) the ProjectedContainerProfile from the legacy AP+NN overlay path equals the one from using the converted ContainerProfile as the base with no overlay. Behaviour-preserving by construction. Signed-off-by: entlein <einentlein@gmail.com>
…e authoritative base When the user-defined-profile pod label names a ContainerProfile carrying managed-by: User, the cache now uses it directly as the container's base profile (the migrated "new way"), instead of overlaying a legacy ApplicationProfile + NetworkNeighborhood pair. It falls back to the legacy AP+NN pair when no such CP exists, which still fires the existing deprecation signal. - add-time (tryPopulateEntry) and refresh (reconciler) both prefer the user CP, gated on the managed-by: User annotation so a learned CP at the same name is never mistaken for a user-defined one - UserCPRef/UserCPRV bookkeeping mirrors the legacy UserAPRV/UserNNRV RV tracking so the reconciler re-fetches and rebuilds only when the user CP changes Signed-off-by: entlein <einentlein@gmail.com>
Test_28 now creates one ContainerProfile (managed-by: User) carrying the merged exec/syscall + egress/selector surfaces, instead of a separate user-authored ApplicationProfile + NetworkNeighborhood pair — exercising the migrated read-path end to end. Assertions unchanged. Signed-off-by: entlein <einentlein@gmail.com>
…verlay The new path (converted CP as base, no overlay) is ~35% faster and allocates ~27% less than the legacy AP+NN overlay per projection, since it skips the two-object merge. Signed-off-by: entlein <einentlein@gmail.com>
Test_27 (opens R0002, both the regex and the curl-wildcard profile sites), Test_32 (R0040 argv wildcards), and Test_33 (opens wildcard anchoring — the previously-fixed one) now each create a single user-defined ContainerProfile (managed-by: User) carrying the merged exec/open/syscall (+ egress/selector for 32) surfaces, replacing the legacy ApplicationProfile + NetworkNeighborhood pair. Assertions unchanged. Test_28 was ported in an earlier commit. Signed-off-by: entlein <einentlein@gmail.com>
containsDynamicSegment recognised only the one-segment DynamicIdentifier
('⋯'), so a path-surface opens entry bearing the zero-or-more
WildcardIdentifier ('*') — e.g. '/etc/ssl/*' — was routed to Values as if it
were a literal. was_path_opened tolerates this (Values and Patterns are both
matched via CompareDynamic), but it is wrong for any consumer that treats
Values as exact membership, and it drops '*'-only entries a rule needs when
spec.All is false and no prefix/suffix matcher retains them. Recognise both
wildcard markers. Regression test pins '/etc/ssl/*' -> Patterns.
Pre-existing in upstream (identical containsDynamicSegment).
Signed-off-by: Entlein <eineintlein@gmail.com>
Signed-off-by: entlein <einentlein@gmail.com>
Test_33's anchoring subtests assert R0002 alerts, but the rule's file-access monitoring is opt-in (monitored prefixes incl. /etc/). Test_33 never applied r0002-files-access-enabled.yaml (Test_27 does via enableR0002ForTest), so R0002 never evaluated the opens and every 'expect alert' case silently passed as a no-alert — invisible because Test_33 had never run in CI. Enable it like Test_27. Verified on a live cluster: /etc/ssl/* correctly alerts on the bare parent /etc/ssl and stays silent on the child /etc/ssl/openssl.cnf. Signed-off-by: entlein <einentlein@gmail.com>
…tations A user-authored profile is authoritative and complete by definition — it should not carry the learning-lifecycle status/completion markers, nor a managed-by annotation. The pod's user-defined-profile label is what declares it user-authored (a signature, added by the signing tooling, is the integrity marker). - read-path/reconciler no longer gate on managed-by: the label-referenced CP is used as authoritative, and the entry State is forced to Completed+Full so the rule engine enforces it despite the absent completion annotation - the converter emits clean CPs (name + namespace + spec only) - fake client models an absent overlay-name fetch (drives the legacy fallback) Signed-off-by: entlein <einentlein@gmail.com>
…mple - Test_28 now loads its user-defined ContainerProfile from a real yaml (resources/containerprofile-user-defined-network.yaml) via a loader helper, so the fixture doubles as the copy-pasteable "how to author a user-defined profile" example. - The example — and all the test CPs — carry only name + spec: the nonsensical learning-lifecycle annotations (status/completion) and the managed-by marker are dropped (node-agent now treats a label-referenced CP as authoritative and forces the enforce-state itself). Signed-off-by: entlein <einentlein@gmail.com>
Migrate the network-wildcards NetworkNeighborhood fixtures into their user-authored ContainerProfile form so users can copy-paste them to author user-defined-profile allow-lists: - per-container spec.egress/ingress (NN's spec.containers[] collapses, one CP document per container; fixture 20 splits into two) - no lifecycle annotations — name only (namespace injected by tooling), matching the clean user-managed CP contract - teaching comments preserved verbatim from the NN fixtures Adds 00-fusioncore-homoglyph-attack.yaml: a pinned single-vendor allow-list and the look-alike (homoglyph) domains its exact-match dnsNames compare rejects (each fires R0005). All 21 documents strict-parse against v1beta1 ContainerProfile and carry zero annotations (verified). Signed-off-by: entlein <einentlein@gmail.com>
The component tests build their user-defined profiles inline as ContainerProfiles (Test_27/32/33) or load a CP yaml (Test_28); the legacy ApplicationProfile / NetworkNeighborhood fixtures they were derived from are no longer referenced by any Go test. Migrate the two CT-relevant ones to their user-authored CP form and delete the orphan: - exec-arg-wildcards-profile.yaml (AP curl-32-overlay) -> containerprofile-exec-arg-wildcards.yaml. CP form mirrors Test_32's inline CP exactly (same execs incl. busybox-symlink + literal-* entries, same syscalls, matchLabels app: curl-32). Demonstrates the exec-arg wildcard grammar for authoring. - known-network-neighborhood.yaml (NN fusioncore-network) -> containerprofile-fusioncore-network.yaml. Clean user-managed CP: name only, no managed-by / status / completion annotations. - user-profile.yaml: deleted. Zero references anywhere; its nginx/server exec surface matches no current test or deployment. Both new documents strict-parse against v1beta1 ContainerProfile and carry zero annotations (verified). CT compiles unchanged (go vet -tags component); no Go test referenced the deleted files. Signed-off-by: entlein <einentlein@gmail.com>
Signed-off-by: entlein <einentlein@gmail.com>
Signed-off-by: entlein <einentlein@gmail.com>
… decommission them -step 1 of many -- I understand that this has a long tail of decommissions Signed-off-by: entlein <einentlein@gmail.com>
…storage etc Signed-off-by: entlein <einentlein@gmail.com>
…icontainer labels, tests not reviewed yet Signed-off-by: entlein <einentlein@gmail.com>
Signed-off-by: entlein <einentlein@gmail.com>
Co-authored-by: Matthias Bertschy <matthias.bertschy@gmail.com> Signed-off-by: Duck <70207455+entlein@users.noreply.github.com>
Signed-off-by: entlein <einentlein@gmail.com>
Apply had assertion-style unit tests but no frozen contract for its full projected output. Add TestApply_Golden: a corpus (execs incl. literal-* and ellipsis args, opens incl. trailing-* and dynamic segments, syscalls, capabilities, endpoints, ingress/egress with CIDR/*-sentinel/DNS, call stacks) projected via the real CompileSpec, frozen to goldens (regenerate with UPDATE_GOLDEN=1). Plus idempotency and spec-hash-stability invariants. This is a characterization/regression freeze, not a two-path differential -- the legacy AP/NN projection was removed, so there is no second implementation to diff. Signed-off-by: entlein <einentlein@gmail.com>
The projection golden corpus set the deprecated DNS/IPAddress singulars (to distinct values) alongside the modern lists. Drop them: the corpus now uses DNSNames/IPAddresses only (still covering DNS name, literal IP, CIDR and the * sentinel). Goldens regenerated from the CP-native corpus. Signed-off-by: entlein <einentlein@gmail.com>
…ace) The rulemanager CEL libraries already read ContainerProfile data but still carried AP/NN naming. Rename applicationprofile -> containerprofile and networkneighborhood -> containerprofilenetwork (dirs via git mv, packages, types, constructors, LibraryName), and unify the CEL function namespace: ap.* and nn.* -> cp.* (cp.was_executed, cp.is_domain_in_egress, ...). Update cel.go registration, the bundled default-rules.yaml, and the network-wildcards test rules to the cp.* namespace so rules compile against the renamed engine. Rule content mirrors the rulelibrary migrate/sbob rename. Signed-off-by: entlein <einentlein@gmail.com>
The CP-form fixtures lived in a parallel network-wildcards-cp/ dir alongside the now-dead network-wildcards/. 'cp' is an implementation detail, not part of what the fixtures are. Remove the dead dir, move the CP fixtures to network-wildcards/ (update the nnlint/fixtures test references), and drop the explanatory README. Signed-off-by: entlein <einentlein@gmail.com>
Signed-off-by: entlein <einentlein@gmail.com>
Signed-off-by: entlein <einentlein@gmail.com>
Resolve the Test_35 collision: upstream added Test_35_ExecTTYFieldTest at the same location as our Test_35_MultiContainerPerContainerBinding. Keep both; renumber ours to Test_36_MultiContainerPerContainerBinding.
Add cluster-free unit coverage for the migration's rule-side surface: - cp.* HTTP evaluators (http.go, was 0%/no file): endpoint/host match, method and prefix/suffix variants, empty-CP, nil-cache error. - isExecInPodSpec + the wasExecuted/wasExecutedWithArgs podspec-exempt fall-through. - containerprofilenetwork matchers: no-match / wildcard-domain / empty-neighborhood. - Declarations smoke+drift guard: compile every cp.* overload, assert the decl set. - profilehelper GetProjectedContainerProfile/GetPodSpec/GetContainerName. - HasFinalApplicationProfile enforce-vs-learn gate (Completed+Full -> enforce). containerprofile 37->69%, containerprofilenetwork 75->80%, profilehelper 95%, HasFinalApplicationProfile 100%. Test-only; no production changes. Signed-off-by: entlein <einentlein@gmail.com>
… racy learning) Test_20/21 hung to the 20-minute global panic in WaitForContainerProfileCompletion*: Test_20's blacklist wait can never succeed (the old completed profile stays matched and blacklisted after the deployment restart), and Test_21's multi-container profile never reached 'completed'. Both raced the natural learn -> daemonset-restart -> deployment-restart -> re-learn cycle. Rewrite to enforce an AUTHORED user-defined ContainerProfile, updated in place: an action not in the profile alerts; the same action, once added, does not. Determinism comes from a positive reload gate -- the update that adds the subject also removes a canary, so the canary starts alerting the moment node-agent reloads the revision (alert-appears signal), confirming the reload before the negative assertion. Single-container workloads; bounded pollers that dump ContainerProfile status on timeout (never a 20m panic); Eventually-style polling on the real alert condition instead of fixed sleeps. Also fixes a latent bug: the old 'no new alert' check filtered on a process_name label that alerts never carry, so it always matched zero and passed regardless; the rewrite keys on comm (process) and rule_id (network). Verified: both pass twice on a live rig (rc5l), 0 'database is locked'. This now tests profile ENFORCEMENT (authored partial->full), not natural learning. Signed-off-by: entlein <einentlein@gmail.com>
… sleep Test_16 slept a hard-coded 175s after the restart-inducing 'service nginx stop' before exec'ing the ls violation (the WaitForContainerProfileCompletion guard was commented out), then asserted the 'Unexpected process launched' alert. On a quiet system that had ~110s of slack; under full-CT concurrency the storage single-writer SQLite contention (database is locked) defers the merged profile's completion-status write past the fixed window, so ls ran against a non-enforcing profile and no alert fired -> the assertion failed (observed as Test_16 red in a full CT with 12 lock events). Replace the fixed sleep with a bounded poll (5m, fail-fast, dumps ContainerProfiles on timeout) for the MERGED ContainerProfile reaching 'completed' after the restart -- keyed on the merged profile only (name has no -<32 hex> suffix), not WaitForContainerProfileCompletion which requires ALL matching profiles completed and would hang on the transient per-instance profiles the restart spawns. Then exec ls and poll (Eventually) for the alert. Validated 3/3 on a live rig; runtime ~241s -> ~90s since it no longer over-sleeps. Signed-off-by: entlein <einentlein@gmail.com>
Signed-off-by: entlein <einentlein@gmail.com>
Signed-off-by: entlein <einentlein@gmail.com>
Signed-off-by: entlein <einentlein@gmail.com>
… us to find a way of catching it. its still not clear what caused the issue Signed-off-by: entlein <einentlein@gmail.com>
Signed-off-by: entlein <einentlein@gmail.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
* cel/containerprofilenetwork: add cp.was_selector_in_{ingress,egress}
Runtime label-based network matching, complementing was_address_in_*: resolve a
peer IP to its pod via the k8s object cache and match it against the profile
ingress/egress podSelector + namespaceSelector. Matches the peer by IDENTITY, so
it survives pod IP churn (learned IPs go stale on reschedule; selectors do not).
- projection carries each neighbor selector (IngressPeers/EgressPeers) instead of
dropping it at extract time
- wasSelectorIn shared impl + was_selector_in_ingress / was_selector_in_egress
direction wrappers, registered under the cp network library with cost estimates
- unit test for the podSelector + namespaceSelector match logic
* cel/containerprofilenetwork: match peer selectors on gadget-enriched labels
was_selector_in_{ingress,egress} resolved the peer IP to a pod through the
node-local K8s object cache (its pod watch is filtered by spec.nodeName), so
it could only match peers scheduled on the same node as the workload. A
matching peer on any other node did not resolve, was treated as unknown, and
alerted — ingress detection worked intra-node but not inter-node.
Match instead on the peer identity the network gadget's IP resolver already
attaches to the event (namespace + pod labels), resolved against a
cluster-wide pod inventory before the event reaches CEL. The IP-to-pod lookup
is dropped entirely; matching on peer identity is stable across pod IP churn
and holds across nodes.
- utils/cel: expose event.dstNamespace and event.dstPodLabels
- was_selector_in_{ingress,egress}: signature is now
(containerId, namespace, podLabels); the map argument bypasses the scalar
function cache
- add a compile guard that type-checks the selector rule expressions against
the real event object type
---------
Co-authored-by: ConstanzeTU <74674840+ConstanzeTU@users.noreply.github.com>
… chain The storage build dispatches this repo's build.yaml on the matching branch to rebuild node-agent with the freshly built storage (go mod replace via STORAGE_REF). This branch previously carried only the upstream .github, so the dispatch failed with 'Workflow does not have workflow_dispatch'. Add the dispatchable build workflow so the same-branch rebuild chain resolves.
merge opens gadget fix
Signed-off-by: entlein <einentlein@gmail.com>
…oc/test gaps (#77) * docs(containerprofilecache): update doc comments to the ContainerProfile-only design; restore network-wildcards fixture README The AP/NN decommission left several doc comments describing the removed overlay architecture: refreshAllEntries claimed to fast-skip on UserAPRV/UserNNRV (fields that no longer exist), addContainer and tryPopulateEntry said they fetch user-authored AP/NN CRDs, the workloadName naming note still described the AP/NN aggregation target, and namespacedName was documented as a legacy-CRD identifier. All now describe the authored ContainerProfile flow that actually runs. Also drop a review-metadata reference from a projection_apply comment. tests/resources/network-wildcards/README.md was deleted with the legacy fixtures, but the directory's ContainerProfile fixtures are still consumed by containerprofilenetwork/fixtures_test.go and were left undocumented. Restore the README updated to the ContainerProfile fixture shape and current test names, including fixture 00 which postdates the deleted version. No functional change; go build and the package tests are unchanged. * test(projection): pin '*' exec-path classification as Pattern The wildcard-classification fix routes any '*'- or dynamic-segment-bearing entry on a path surface into Patterns instead of Values, but only the Opens surface had a pinning test. Execs paths flow through the same projection, so add the mirror case: a '*'-bearing exec path and a dynamic-segment exec path are Patterns, a literal exec path stays a Value. * feat(containerprofilecache): make authored-profile adoption visible (log + metric) Adopting a user-authored ContainerProfile as the authoritative base for a container silently switches what the rule engine enforces; the legacy overlay path emitted metrics at the equivalent point, and the unresolved case already has container_profile_user_defined_unresolved_total. Close the asymmetry: - Info log naming the namespace/profile when an authored CP is adopted on the add path. - New counter (prometheus container_profile_user_defined_adopted_total, OTEL node_agent.container_profile.user_defined_adopted.total), implemented across the prometheus/OTEL/noop/mock managers. gofmt applied to the touched metrics files (they were unformatted; no CI job runs gofmt). * feat(containerprofilecache): recover container subtypes via grouped authored documents The AP/NN decommission dropped the container subtype groups (containers/ initContainers/ephemeralContainers) the legacy specs carried: the flat ContainerProfile could not describe a pod's init or ephemeral containers, and the multi-container component test only exercised two REGULAR containers - so the subtype contract was silently broken. Storage now restores the subtype groups on ContainerProfileSpec (fork storage module, pinned via replace); this consumes them: - resolveAuthoredContainerSection maps an authored document to the per- container view: flat documents pass through (single-container convention unchanged); grouped documents select this container's section by name across all three subtype groups, inheriting pod-level architectures and the workload selector. A grouped document that does not cover the container resolves to nil - never enforce a sibling's section - with a Warning on the add path. - Wired into both the add path (tryPopulateEntry) and the refresh path (refreshOneEntry), before the learned-annotation validation. - Unit tests: section selection across all three groups, flat pass-through, uncovered-container nil (TestResolveAuthoredContainerSection), and an add-path test proving a regular+init+ephemeral trio each adopt only their own section from one shared document (TestUserDefinedCP_GroupedDocumentPerSubtype). Component test: Test_37_MultiSubtypeGroupedProfileDocument binds one grouped document to a pod with a regular container, an init container whose startup command runs a binary its section forbids (the init phase itself must alert), and a runtime-attached ephemeral container (new TestWorkload.AddEphemeralContainer helper using the ephemeralcontainers subresource). Per-section allow/forbid is asserted for all three subtypes. Test_36 and Test_37 are added to the component-tests matrix - Test_36 was never listed, so it never ran in CI. * test: renumber the multi-subtype component test to Test_48 Test_37 is already claimed by the signed-bundle overlay test in the fork CI harness matrix; renumber to the next free slot so both suites can coexist. * chore: bump storage pin to the subtype-group deflation fix * test(component): assert the grouped document round-trips before driving traffic If a storage-side write path strips the subtype groups, Test_48's per-section assertions fail later with misleading R0001 noise. Assert the served document still carries all three groups (and the app section's execs) right after apply, so a storage regression fails fast with the actual cause. * chore: bump storage pin to the protobuf-marshaller regeneration * test(component): size Test_48 waits to the observed init/ephemeral adoption latency The per-section enforcement now works end to end, but the init and ephemeral containers ran their forbidden binary 30s after start while authored-profile adoption for those containers has been observed to take 60-70s on a loaded runner - the exec raced adoption and produced no alert. Lengthen the in-pod sleeps to 75s (fixture command, matching profile args) and the test's waits accordingly. * fix(containerwatcher): populate shared data for init containers during the init phase getSharedWatchedContainerData refused to proceed while the pod phase was Pending - but a pod executing its init containers is Pending by definition, so shared data (and with it authored-profile adoption and rule enforcement) could only arrive after the init phase completed. An init container could never be enforced during its own execution; observed as an authored init section whose forbidden binary produced no alert because adoption landed seconds after the init container had already exited. The phase gate existed because ImageID is empty in containerStatuses while the pod is pending. Check that directly: when the pod is Pending, proceed as soon as THIS container's status entry (containers, initContainers, or ephemeralContainers) carries a non-empty ImageID, and keep retrying otherwise. * test(component): downgrade Test_48 ephemeral assertions to a tracked known limitation The ephemeral debug container is adopted (section selected, monitor started, tracers attached) but no events from it ever reach the rule engine - verified in CI and interactively on a fresh cluster (zero exec/syscall/capability events while the container demonstrably ran to completion). The ephemeralContainers profile-selection contract stays covered by the cache unit tests; event delivery for ephemeral containers is a container-watcher/ tracer scope issue independent of profile projection, tracked as follow-up. Log the counts and self-signal when tracing starts working so the assertions can be promoted back. * test(component): widen Test_48 init margin to 100s for CI runner variance Init-phase enforcement is proven (interactive validation: adoption 3s after deploy, init R0001 fires), but CI runners intermittently take longer than 75s to complete the first adoption; give the init container a 100s runway. * test(containerwatcher,profilecache): pin event delivery across container end-of-life Failing tests for the teardown race behind issue #79 (init/ephemeral exec loss): events emitted during a container's life are dropped when processed after the container's removal. - TestProcessEvent_DeliversEventForJustRemovedContainer[_NoPriorEvent]: EventHandlerFactory.ProcessEvent silently drops events whose container has left the live collection (evidence: run 31846699597, Test_48, init container terminal exec at 22:44:26, remove processed 22:44:26, zero R0001; ladder run1 total loss). - TestProjectedProfile_SurvivesContainerRemovalGrace: the projected profile is deleted immediately on the remove callback, so in-flight events lose profile resolution and ProfileDependency=Required rules suppress as profile_incomplete. Both tests fail on current code by design; the fix must provide a removal grace window covering the event pipeline delay. * fix(containerwatcher,profilecache): grace period for events in flight at container end-of-life An event emitted during a container's life can be processed after the container's removal: the ordered event queue (50ms collection tick + batching) and the worker pool delay evaluation past teardown. For a container whose final process performs the exec and exits immediately (init container with a terminal exec, ephemeral debug container), the alert-carrying exec event loses this race and is dropped. Observed failure: run 31846699597, Test_48_MultiSubtypeGroupedProfileDocument, init container setup (sh -c "sleep 75; /usr/bin/id"): remove processed 22:44:26, terminal exec evaluated afterwards, zero R0001 while 98 R0003 fired during the container's life (assertion: 'id is not in the setup section (initContainers)', component_test.go:3488). Same-shaped loss for the ephemeral leg (R0001(debug,id)=0, remove 22:46:15). Root cause, two drop points on the remove path: 1. EventHandlerFactory.ProcessEvent resolved container info only from the live container collection plus a lazily-populated cache, silently dropping events for just-removed containers that never had a prior event processed. 2. ContainerProfileCache deleted the projected-profile entry immediately (async) on the remove callback, so rules with ProfileDependency=Required suppressed in-flight events as profile_incomplete. Fix: keep container info and the projected profile resolvable for a 10s grace after removal, then evict: - the factory now receives container lifecycle callbacks, warms its lookup cache on add, and defers eviction by the grace period; - the profile cache defers deleteContainer by the grace period and the reconciler's terminated-eviction honors the same grace (mark on first Terminated observation, evict on a later tick), so a reconciler tick landing inside the window cannot reintroduce the race. Tests: TestProcessEvent_DeliversEventForJustRemovedContainer{,_NoPriorEvent}, TestProcessEvent_RemovedContainerEvictedAfterGrace, TestProjectedProfile_{SurvivesContainerRemovalGrace,EvictedAfterRemovalGrace}, TestReconciler_HonorsRemovalGraceForTerminatedContainer (all red on the pre-fix code); TestReconcilerEvictsTerminatedContainer, TestInitContainerEvictionViaRemoveEvent, TestMissedRemoveEventEvictedByReconciler updated to the graced contract. Regression: go test ./pkg/containerwatcher/... ./pkg/objectcache/... ./pkg/rulemanager/... passes (tracers field tests skipped locally: they require the tracers.tar gadget bundle, unavailable off-CI); -race clean on both touched packages. * test(e2e): add issue-79 end-of-life exec-delivery ladder script Deterministic rig-side measurement for acceptance tests T4/T5: N repeated init runs (terminal forbidden exec after a configurable runway) and N ephemeral-container runs (terminal whoami+id), each asserting R0001 delivery via node-agent logs. Exits non-zero unless both legs are N/N. * test(profilecache): pin reconciler classification of status-lagged containers Failing tests for the ephemeral total-loss leg of issue #79: the reconciler classifies a container that is absent from all published status lists as reaped, but a just-attached ephemeral container is exactly that (kubelet publishes ephemeralContainerStatuses seconds after the attach), and an init container whose entry carries an empty PodUID hits the same branch while its status has no ContainerID yet. The entry is evicted, nothing re-adds it, and every ProfileDependency=Required rule is suppressed for the container's entire life. Live-cluster evidence: ephemeral container adopted at +1s, reconciler tick 3s later (entries_before=2 entries_after=1), zero alerts of any class over its 75s life while the same pod alerted for other containers. Contract pinned: a container still declared in the pod SPEC without a published status is not reaped; absent from both spec and status is; a termination mark resets when the container is observed alive again. * fix(profilecache): do not classify status-lagged containers as reaped The reconciler evicted any cache entry whose container was absent from all published status lists once any statuses existed. A just-attached ephemeral container is exactly that: the pod spec already declares it while kubelet publishes its ephemeralContainerStatuses entry seconds later. The freshly-adopted profile entry was evicted on the next tick, nothing re-added it, and every ProfileDependency=Required rule was suppressed for the container's entire life — zero alerts of any class (issue #79 T5, ephemeral 0/N). Init-container entries created before the pod reached the k8s cache (empty PodUID, status ContainerID not yet published) hit the same branch, contributing to the init intermittency. Observed: ephemeral container adopted +1s after attach; reconciler tick 3s later logged entries_before=2 entries_after=1; zero alerts over the container's 75s life while the same pod alerted for its other containers. Fix: absence from published statuses only counts as reaped when the container is also absent from the pod SPEC (containers, initContainers, ephemeralContainers). Additionally, the termination mark introduced with the removal grace now resets when a marked container is observed alive again, so a later genuine termination gets a full grace window. Tests (red pre-fix): TestReconciler_KeepsEphemeralContainerAwaitingStatus, TestReconciler_KeepsInitContainerAwaitingStatusWithEmptyPodUID, TestReconciler_TerminationMarkResetsWhenContainerReappears; negative contract TestReconciler_EvictsContainerRemovedFromSpecAndStatus. Regression: go test -race ./pkg/objectcache/... ./pkg/containerwatcher/v2/ ./pkg/rulemanager/... passes. * fix(objectcache): do not evict status-lagged containers from the profile cache The ContainerProfileCache reconciler classified any cache entry whose container was absent from the pod's published status lists as reaped and evicted it (reconciler.go isContainerTerminated). But kubelet publishes the status groups incrementally: a just-attached ephemeral container has no ephemeralContainerStatuses entry for several seconds while it is already running and traced, and an entry added before the pod reached the k8s cache carries an empty PodUID, which made the (Name, PodUID) pre-running fallback unreachable for init containers. Eviction is permanent (no re-add path exists), so every ProfileDependency=Required rule (R0001/R0003/R0004) was silently suppressed for the container's whole life: total alert loss for ephemeral containers, intermittent exec-alert loss for init containers (issue #79, CI run 31846699597). Evidence (live rig, issue #79): ephemeral container adopted at +1s, evicted at the next reconciler tick +3s (entries 2->1), exec events verifiably reached ReportEnrichedEvent at +75s and were dropped by the Required-profile gate; the exec gadget's mntns filter map contained the container's mntns the whole time (kernel/tracer exonerated). Fix: - treat absence from the status lists as reaped only when the pod SPEC does not name the container either; a status entry with the same name under a different non-empty ContainerID still evicts (replaced instance) - allow the pre-running (Name, PodUID) fallback to match when the stored PodUID is empty - backfill PodUID from the container runtime metadata when the pod is not yet in the k8s cache at entry-build time New tests fail on the pre-fix code and pass with the fix: TestReconcilerKeepsJustAttachedEphemeralContainer, TestReconcilerKeepsInitContainerWithEmptyStoredPodUID. Regression guards (both-ways green): eviction after published termination, gone from spec+status, replaced instance. Full objectcache, rulemanager and containerprofilemanager suites pass unchanged. * test(e2e): fix issue-79 ladder measurement - read all node-agent pods, match containerName exactly, race-free readiness wait The ladder under-counted to 0/N while the node-agent logs showed 5/5 R0001 for both the init and the ephemeral container: it read only one DaemonSet pod's logs (the workload can land on any node), its grep could not match the alert JSON's containerName field, and its readiness wait raced pod creation. * fix(utils): restore IsResolvedFullPath dropped by the trace_open rework The gadget rework removed IsResolvedFullPath from path.go while datasource_event.go still guards the fname fallback with it, leaving the tree uncompilable. Restore the helper unchanged; whether the guard is still needed under kernel-side full-path resolution can be decided separately. * fix(utils): align the trace_open pick with upstream kubescape#889 Supersedes the IsResolvedFullPath restoration one commit back: upstream kubescape#889 retires the helper entirely and replaces the call-site guard with a plain empty-fpath fallback (the patched gadget resolves full paths kernel-side, so the fragment hazard no longer exists). Take upstream's GetFullPath body, drop the helper again together with its fork-only test, leaving path.go and normalize_path_test.go byte-identical to upstream main. * fix(gadgets): root paths from detached procfs mounts in trace_open walk The backward dentry walk in get_path_str terminates at the mount-tree root of the file's vfsmount. runc >= 1.2 accesses procfs during container init and exec through a private detached mount created via fsopen(2)/fsmount(2); such a mount has no mountpoint (mnt_parent == mnt), so the walk correctly reaches its top with no /proc dentry to prepend and emits prefix-stripped paths such as /1/task/1/fd. The kernel's own d_path reports the same rootless string, so no userspace consumer can recover the prefix afterwards. Detect this termination case by checking the superblock magic of the final dentry: if it is PROC_SUPER_MAGIC the walk ended inside a procfs instance that is the top of its own mount chain, which cannot be the real global root, and the path is canonicalized by prepending proc/. Attached procfs mounts are unaffected because their walk continues through mnt_mountpoint before reaching this branch, and non-procfs detached mounts keep their previous behavior. Verified on kernel 6.1.167 (x86_64): fsopen/fsmount reproducer emits /proc/1/task/1/fd (was /1/task/1/fd), detached procfs root open emits /proc, detached tmpfs root open remains /, regular file paths unchanged, program accepted by the verifier. Refs #81 --------- Co-authored-by: entlein <einentlein@gmail.com> Co-authored-by: k8sstormcenter-bot <k8sstormcenter@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| needs: build | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| actions: write |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Informational only