feat(network): symmetric R0011/R0012 with NetworkPolicy-style internal allowlisting and port-aware alerting - #923
feat(network): symmetric R0011/R0012 with NetworkPolicy-style internal allowlisting and port-aware alerting#923entlein wants to merge 23 commits into
Conversation
…c allowlisting Signed-off-by: entlein <einentlein@gmail.com>
Signed-off-by: entlein <einentlein@gmail.com>
Signed-off-by: entlein <einentlein@gmail.com>
…rt if delcared and violated Signed-off-by: entlein <einentlein@gmail.com>
Let a ContainerProfile allowlist cluster-infrastructure egress/ingress by Service name, Service label selector, or host entity instead of a broad ipAddresses serviceCIDR that blinds R0011/R0012 to lateral movement. Each serviceRef/serviceSelector/entity neighbor resolves at projection time to the concrete ClusterIP + backing-endpoint (or node/gateway) IPs it stands for, carrying its own ports, and is appended as an ordinary selector-free ipAddresses neighbor. The existing port-sensitive address matcher enforces it unchanged; unresolved selectors contribute nothing (never a match-all). - pkg/networkpeer: Resolve/Matches/ResolveIPs + Lister over Service, EndpointSlice and Node informers, with a generation counter so a profile projected before the informers synced re-projects once the view changes. - objectcache/reconciler: mark profiles that use service resolution and re-project them when the lister generation advances; plain profiles keep the identical old fast-skip path. - cmd/main.go: cluster-wide Service/EndpointSlice informers + a node-scoped Node informer, started non-blocking (no WaitForCacheSync on the hot path). - fail closed on ServiceSelector MatchExpressions / empty matchLabels and on any namespaceSelector other than kubernetes.io/metadata.name=<ns>. - Test_50 component test (serviceRef egress allowed, external egress still fires R0011) + resolve/expand/lister unit tests + fixture-lint R-NN-12 extended to accept the new target fields. Depends on the storage schema fields ServiceRefNamespace/ServiceRefName/ ServiceSelector/Entity; go.mod pins the fork's storage until the companion upstream storage PR lands. Signed-off-by: tanzee <einentlein@gmail.com>
…rviceRef Component test Test_50 now generates its traffic from a real Flux source-controller reconciling HelmRepository CRs instead of exec'ing curl, and its ContainerProfile is network-only (no syscalls/execs, which only add false-positive surface to a network test). The profile names every peer as a Kubernetes object: serviceRef default/kubernetes for the apiserver, serviceRef kube-system/kube-dns for resolution, and a serviceSelector role=helm-repo fanning across the two repo Services. The negative is the lateral move a serviceCIDR entry hides: the HelmRepository URL is repointed at a sibling Service on the same port that the selector does not cover, and the controller fetches it itself. Verified on kind: 0 alerts for the named peers, R0011 within 15s for the sibling. Fixes found while validating end to end: - ClusterRole was missing discovery.k8s.io/endpointslices, so the informer was forbidden and Service endpoint IPs never resolved — the feature silently degraded to ClusterIP-only. - Service/EndpointSlice informers are now gated behind networkServiceResolutionEnabled and strip managedFields/annotations (and per-endpoint fields beyond Addresses) via SetTransform, so agents that do not use the feature pay no cluster-wide list+watch and the cache stays small on those that do. - serviceRef/serviceSelector now also imply the Service cluster FQDN as a dnsName, so a client dialling the Service by name is allowlisted without a parallel dnsNames entry. - specFromNeighbor no longer allocates a discarded port slice for every plain ipAddresses neighbor. - R0011 no longer excludes private destinations: in-cluster lateral movement is exactly what this feature exists to expose. Signed-off-by: tanzee <einentlein@gmail.com>
Relaxing the shipped R0011 to fire on private destinations made kube-dns egress alert for every workload that does not name it: Test_21 gained a spurious R0011 and Test_28 lost allowed_fusioncore_no_alert and mitm_coredns_poisoning. Restore the stock expression and express the internal-egress predicate as a test-only rule (R9911) bound by podSelector to this suite's pods, so nothing outside it changes. Verified on kind: Test_50 passes both phases against the stock ruleset, and Test_21 + all six Test_28 subtests are green again. Signed-off-by: tanzee <einentlein@gmail.com>
Hardcoding the flag in the ConfigMap made it impossible to measure the feature's cost against itself. Expose it as nodeAgent.config.networkServiceResolution (on in the test chart, so Test_50 still exercises it) so an A/B can toggle resolution without rebuilding the image. Signed-off-by: tanzee <einentlein@gmail.com>
The CEL result cache keys on SpecHash + SyncChecksum. Re-projecting a serviceRef/serviceSelector/entity profile against a moved cluster view changes neither: SpecHash tracks the rule projection spec, and SyncChecksum comes from a learned CP annotation an authored profile does not carry at all. So a result computed before the Service/EndpointSlice informers filled — 'this ClusterIP is not in egress' — was served from the LRU indefinitely, and the re-projection the lister generation correctly triggered had no observable effect. Egress to an allowlisted Service kept alerting. Carry the resolution generation on the projected profile and include it in the key, so the cache moves whenever the resolved addresses can have moved. Signed-off-by: tanzee <einentlein@gmail.com>
R0011 keeps its external-only scope (!is_private_ip); internal traffic gets its own rule instead of widening R0011 — rewriting R0011's scope broke Test_21/28 (kube-dns FPs) when tried in the fork CT. R0012 alerts on OUTGOING to private addresses (loopback excluded — is_private_ip counts 127.0.0.1/::1 as private) not allowlisted by the profile's egress addresses, which includes serviceRef/serviceSelector-resolved entries. Uses the port-aware matcher; behaves address-only until the port projection lands, then becomes port-sensitive with no rules change. A selector clause (was_selector_in_egress) is added one-line when the peer-selector fields merge. Same defaults as R0011; uniqueId keyed on addr_port_proto; bound in the default binding (new rule names are inert until bound). Signed-off-by: tanzee <einentlein@gmail.com>
Per design review: R0011 (egress) and R0012 (ingress, new) are symmetric twins. Neither uses is_private_ip — internal and external peers are treated alike, so lateral movement to unlisted internal peers alerts; only loopback is excluded. Allowlisting internal traffic is the profile's job (addresses, resolved serviceRef/serviceSelector entries), not the rule's. Both use the port-aware matcher (address-only until port projection lands). On HOST (incoming) events the gadget's dstAddr/dstPort carry the remote peer and local port. R0011's scope widens to internal egress: component tests whose profiles do not list kube-dns et al. will alert until their profiles do — that pressure is the feature. Signed-off-by: tanzee <einentlein@gmail.com>
…t/network-v2 # Conflicts: # pkg/objectcache/projection_types.go
…into feat/network-v2
# Conflicts: # tests/chart/templates/node-agent/default-rules.yaml
portalerts carried its own copy of the celnetworkselector peer-selector functions; the merge kept both and the package no longer compiled. One copy remains. Signed-off-by: tanzee <einentlein@gmail.com>
…xes) Signed-off-by: tanzee <einentlein@gmail.com>
… twin The scoped R9911 rule and its binding are gone — the widened R0011 covers internal egress, so the decoy pivot asserts the shipped rule. Test_51 mirrors it for ingress: nginx serves a serviceRef-listed client (flux source-controller, resolution covers its ClusterIP and pod endpoint IPs) with zero R0012, then an unlisted k6 client joins and R0012 must fire. Both use only real controller/loadgen traffic. Signed-off-by: tanzee <einentlein@gmail.com>
…eset Deployable over any kubescape install to replace the stock rules; namespace templated. A drift test pins the chart copy to the CI-validated test-chart copy so the shipped semantics are always the tested ones. Signed-off-by: tanzee <einentlein@gmail.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The selector clause was deferred while the engine lived on a separate branch, then forgotten when that branch merged: selectors resolved and matched but no rule consulted them. Both rules now also allowlist via was_selector_in_egress/ingress, matching the form already deployed downstream. Signed-off-by: tanzee <einentlein@gmail.com>
| @@ -0,0 +1,44 @@ | |||
| apiVersion: kubescape.io/v1 | |||
There was a problem hiding this comment.
chicken, please remove those here, thats the PR to rulelibrary, but add them to the tests/** folder
| ruleExpression: | ||
| - eventType: "network" | ||
| expression: "event.pktType == 'OUTGOING' && !net.is_private_ip(event.dstAddr) && !cp.was_address_in_egress(event.containerId, event.dstAddr)" | ||
| expression: "event.pktType == 'OUTGOING' && !event.dstAddr.startsWith('127.') && event.dstAddr != '::1' && !cp.was_address_port_protocol_in_egress(event.containerId, event.dstAddr, event.dstPort, event.proto) && !cp.was_selector_in_egress(event.containerId, event.dstNamespace, event.dstPodLabels)" |
There was a problem hiding this comment.
Will make the rules broad and OFF by default. I.e. the private ip and localhost filters will be removed
| load.js: | | ||
| import http from 'k6/http'; | ||
| import { sleep } from 'k6'; | ||
| export const options = { vus: 2, duration: '30m' }; |
matthyx
left a comment
matthyx
left a comment
There was a problem hiding this comment.
Reviewed as requested. This lines up with the author's own "NOT READY FOR REVIEW" flag — the networkpeer resolution/fail-closed design (empty selectors → nil, MatchExpressions rejected, ClusterIPNone excluded, gateway IP verified via ipNet.Contains) is sound, but there are concrete blockers before this can merge:
Critical
go.mod/go.sum:replace github.com/kubescape/storage => github.com/k8sstormcenter/storage v0.0.240-...pins the module to a fork, with the pre-replace requirement left at the placeholderv0.0.0-00010101000000-000000000000. This is unbuildable for any consumer until kubescape/storage#364 merges and a real pseudo-version replaces it.- Unrelated dependency downgrades pulled in alongside the fork pin:
kubescape/backend v0.0.39 → v0.0.31andgotest.tools/v3 v3.5.2 → v3.5.0. These look like tidying against the stale fork rather than an intentional change — please restore both unless there's a reason to downgrade.
High
pkg/objectcache/containerprofilecache/reconciler.go(~L499-510): the cache-invalidation fix readslisterGen()afterApply(...)runs, in two separate calls. If aBump()happens during resolution, the new generation gets stamped onto IPs resolved against the old view, sorefreshOneEntry's staleness check never fires and the stale result is served from the LRU indefinitely — the exact bug this fix was meant to close.containerprofilecache.go(~L618-620) does this correctly by reading the generation once, before resolution — please match that pattern here.- The widened R0011 (drops
net.is_private_ip) and new R0012 shipenabled: trueindefault-rules.yaml/binding, butEnableNetworkServiceResolutiondefaults tofalseinpkg/config/config.go— only the test chart'svalues.yamlturns it on. On upgrade, existing profiles start alerting on all internal peers with no serviceRef/serviceSelector allowlisting available to quiet them (resolution is off by default). Either default resolution on alongside the widened rules, or ship R0012/the widened R0011 disabled until resolution is on by default. pkg/networkpeer/expand.go(~L127): nilnamespaceSelectoron aserviceSelectorresolves cluster-wide ("cluster-wide by design" per the comment), which contradicts both the PR description ("nil namespaceSelector = same namespace, as in NetworkPolicy") and the same-namespace default used elsewhere (network.go'snamespaceSelectorMatches).resolveServicesdoesn't have the profile's namespace in scope to fix this today —serviceSelector: {app: foo}currently allowlists that label across every namespace in the cluster, which is a much broader allowlist than the feature intends.- RBAC (
discovery.k8s.io/endpointslices) only lands intests/chart's ClusterRole. Per the PR description, the shipped chart lives in kubescape/helm-charts and "needs to be moved" — without a companion PR there, real deployments enabling this feature get informer 403s and silently degrade to ClusterIP-only resolution (the exact failure mode already found and fixed once during this PR's own validation).
Medium
expand.go'shasServiceFieldsaccepts aServiceRefNamespace-only neighbor, butspecFromNeighborrejects it — causes permanent re-projection churn for that shape.pkg/utils/cel.go'sdstPodLabelsreturns a rawmap[string]stringwhile sibling CEL accessors returnceltypes-wrapped values — worth double-checking this doesn't break type coercion in CEL expressions that consume it.
No fork images found in shipped config (only a ghcr.io/fluxcd reference in a test fixture, which is fine). Given the go.mod fork dependency alone, this isn't mergeable yet — requesting changes rather than approving.
|
thanks for catching the regression during the merge/rebase **AI, ** Clusterrole RBAC: Thanks for bringing it up -> AI, check how much surface this introduces to the deaomnset overall and align with how e.g. cilium minimizes that role
fork dependencies: for the CTs to run, AI, please patch storage-tag.sh with our fork ref, else it builds against upstream which doesnt have the change. That defeats the test Human readers, I m also adding an alert on Reason this is in Draft:
Namespaces: That is correct: namespaces are not read - meaning, if there is a collision in names across namespaces they act as allowlists -> I think, that should be how it works. Exec summary : 🙏 thanks for catching the technical bugs again (this is embarassing) while I sort out my brain |
- reconciler: read lister generation once BEFORE service resolution and stamp that same gen as ResolvedGen/ListerGen, matching addContainer's ordering; a Bump() during resolution now invalidates the projection instead of masking stale IPs behind the fast-skip. - config: networkServiceResolutionEnabled now defaults true (rules ship enabled, resolution must match); test chart configmap falls back to true via hasKey so an explicit false still renders false. - deps: restore accidental downgrades kubescape/backend v0.0.31->v0.0.39 and gotest.tools/v3 v3.5.0->v3.5.2; k8sstormcenter/storage replace pin unchanged (tidy normalized the require placeholder to v0.0.258, the replace still governs). - networkpeer: hasServiceFields no longer counts a ServiceRefNamespace-only neighbor that specFromNeighbor rejects, ending permanent re-projection churn on such profiles; test pins the agreement. - rbac: drop unnecessary get verb on endpointslices (cache-backed informer needs only list+watch). - ct: storage-tag.sh emits the fork storage image tag (net-v2-rc1) while go.mod replaces storage with k8sstormcenter/storage, and the test chart pulls ghcr.io/k8sstormcenter/storage, so CTs run a server that has the serviceRef/dnsNames schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A8UV2B7b6dpDJQci31aC6c Signed-off-by: tanzee <einentlein@gmail.com>
|
Here s the benchmakr from this run BTW @matthyx : could have sworn that benchmark-thingy posted to the PR for the last round, now the widget is getting 403. Possibly intentional, possibly not. EDIT: entlein does not believe that her nonsense eBPF Dedup Benchmark Results Node-Agent Resource UsageMetric BEFORE AFTER DeltaAvg CPU (cores) 0.218 0.213 -2.0% Dedup Effectiveness: no data available |
Peer matching is on pod labels; a nil namespaceSelector no longer requires the same namespace — namespace is consulted only when the selector is explicitly set (collision disambiguation). This aligns peer selectors with the serviceSelector nil-namespace semantics. An empty podSelector now matches NOTHING (fail closed → the peer alerts), the opposite of NetworkPolicy's match-all: an allowlist entry must name what it permits. An unresolved peer still never matches. Full truth tables added: wasSelectorInPeers and namespaceSelectorMatches (pod/namespace edge cases), and serviceRef (via the always-present default/kubernetes API server) and entity:host resolution + matching in networkpeer. Signed-off-by: tanzee <einentlein@gmail.com>
|
@entlein I think the benchmark is accurate on |
|
NICE! |
excludeNamespaces (Config.SkipNamespace) filters which workloads node-agent profiles; the selector resolver queries a cluster-wide Service/Node view that takes no namespace-exclusion input. The resulting source/peer asymmetry — a workload in an excluded namespace is never profiled, yet any monitored profile may still allowlist a Service in that excluded namespace via serviceRef or an unscoped serviceSelector — is easy to overlook. These pin it down: serviceRef into an excluded ns resolves; an unscoped serviceSelector fans across the exclusion boundary; authored NamespaceLabels is the only mechanism that scopes fanout (and can deliberately target an excluded ns); the host entity is orthogonal. Asserted under both the exclude-denylist and include-allowlist forms of SkipNamespace. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A8UV2B7b6dpDJQci31aC6c
2 participants
NOT READY FOR REVIEW
One coherent change replacing the partial PRs #902, #905, #915 (kept open for reference until this lands; storage companion: kubescape/storage#364).
Still carries residue from local testing that wont be upstreamed. The design is getting somewhere now.
Feature: Ports alert, Internal and External Network Slices can be selected for allowlisting
Rules — symmetric egress/ingress, no IP-class gate
127.*,::1) excluded. Lateral movement to an unlisted internal peer alerts; a serviceCIDR-wide allowlist entry no longer blinds detection.was_address_port_protocol_in_*: an allowed address on a violated port alerts; an entry with no ports (orPort: 0) means any port, computed per-(port,protocol).Allowlisting internal traffic, NetworkPolicy-style
podSelectoron profile neighbors, matched at event time (was_selector_in_*)serviceRef{Name},serviceSelector,entity: hostresolve at projection time to ClusterIP + endpoint IPs + the Service FQDN, feeding the ordinary address/DNS surfaces. Unresolvable selectors contribute nothing — never a match-all. Informers are gated behindnetworkServiceResolutionEnabled(default ON) and strip managedFields/annotations;measured cost MUST BE REMEASURED
Chart: needs to be moved
Validation
ghcr.io/k8sstormcenter/{node-agent,storage}:net-v2-rc1.Dependency
go.modtemporarily replaceskubescape/storagewith the fork branch of kubescape/storage#364 (schema fields + generated code + loss-guards in collapse/deflate/NetworkPolicy generation). After #364 merges, the replace drops for a pseudo-version pin — no storage release required.🤖 Generated with Claude Code