feat(kubescape): add vulnerability drill-down tools and bound CVE output - #77
Open
slashben wants to merge 2 commits into
Open
feat(kubescape): add vulnerability drill-down tools and bound CVE output#77slashben wants to merge 2 commits into
slashben wants to merge 2 commits into
Conversation
An agent could not answer "what are the worst vulnerabilities in my cluster?"
without either being lied to or blowing its context window. The only route to
any severity information was list_vulnerabilities, which returned 200,793 B
(~50k tokens) for a single image, and nothing aggregated across manifests at
all, so cluster-scope questions had no answer at any price.
The Kubescape storage API already exposes a full aggregation ladder that this
provider never used: cluster -> namespace -> workload -> manifest -> CVE, where
every level except the last is a handful of integers plus a reference.
Two new tools walk it:
kubescape_vulnerability_overview
Severity totals per namespace from the server-side aggregates, worst
first. One call, ~1 KB, from vulnerabilitysummaries (cluster-scoped, one
object per namespace).
kubescape_list_vulnerable_workloads
Workloads ranked by severity, each row carrying the manifest_name that
kubescape_list_vulnerabilities takes, read from the summary's
spec.vulnerabilitiesRef. One call, ~1.9 KB per workload.
Both take ListOptions.ResourceVersion = "fullSpec".
That sentinel is load-bearing and not obvious. The storage server strips spec
from EVERY list response by default, so severity counters come back zero and
vulnerabilitiesRef comes back empty -- not an error, and indistinguishable from
a clean cluster. Measured on storage v0.0.298, nginx:1.14.0 has 76 critical CVEs
and a default LIST of its summary reports 0. This is the same failure shape as
the vulnerability_count bug, one level up, so listing these resources without
the sentinel would report a vulnerable cluster as clean.
Verified against a live cluster that the typed client transmits it: a wire
capture shows resourceVersion=fullSpec on the request, and the returned counts
match the values an individual GET reports.
Because that correctness rests on a vendor sentinel, the overview also refuses
to present an all-zero cluster as good news: it says the result could not be
confirmed and points at a per-manifest check. A wrong "no vulnerabilities" is
the most expensive answer this provider can give.
kubescape_list_vulnerabilities is reshaped to fit the ladder:
- severity_summary always describes the WHOLE manifest, even when the array
is filtered or truncated, so the aggregate survives a bounded response.
- The array is capped by limit (default 20), ordered worst severity first with
id as a tiebreak for stable output, and reports total_count, returned_count
and truncated so the agent knows it holds partial data.
- Records carry only id, severity and fix_state. Measured field shares of the
old payload: description 56.1%, data_source 19.5%, fix_versions 7.9%. The
description was truncated at 200 chars mid-word, so it was lossy prose that
invited reasoning from a fragment; the full text, data source and fix
versions are all in get_vulnerability_details.
- New severity and fixable_only filters for drill-down.
- Grype emits one match per affected package, so the same CVE recurs. With the
package fields trimmed away those rows are byte-identical, and duplicates
would consume the limit budget: the first 20 records for nginx:1.14.0 held
CVE-2017-12424 and CVE-2017-15670 twice each. Matches are now collapsed by
CVE id with an affected_artifacts count, and a CVE left unfixed in any
package is never reported as fixed. 466 matches become 293 distinct CVEs.
Measured against the same image on a live cluster: 3,013 B, against 200,793 B
before -- 67x smaller, with a complete and correct severity summary.
Also fixes a miscount in that summary: severityCounts had no Negligible bucket,
so the 102 Negligible CVEs in nginx:1.14.0 were reported as "Unknown": 102.
Severity handling is now driven by one ordered list used for both bucketing and
ranking, so a severity cannot be dropped into the wrong bucket again.
Relevancy is surfaced where the API provides it -- "relevant" counts CVEs whose
code node-agent observed loaded at runtime, e.g. 76 critical of which 30 are
reachable. It is emitted only when non-zero: the field is `json:"relevant,omitempty"`
upstream, so a zero is indistinguishable from "relevancy was never computed",
and node-agent needs a learning period before it reports anything. Rendering
absent as 0 would claim nothing is reachable when nobody has looked yet.
One value from the summaries cannot be trusted. vulnerabilitiesRef reports the
WORKLOAD's namespace, but the manifests live in the Kubescape namespace, so
following it verbatim yields NotFound -- confirmed on a live cluster, where a GET
of docker.io-library-nginx-1.14.0-e34030 in the referenced namespace fails while
the same GET in `kubescape` succeeds. Only the name is taken from the ref; the
namespace comes from the operator namespace, overridable with kubescape_namespace.
Handing the agent a pointer that 404s would defeat the point of the chain.
Tests are fake-client based and need no cluster. Note that client-go's fake
drops ResourceVersion from recorded actions, so a reactor cannot observe the
sentinel; the tests wrap the typed client to record the ListOptions each handler
actually passes, which pins the invariant at the real call site.
Signed-off-by: Ben Hirschberg <ben@armosec.io>
Docs-exempt: no existing doc describes the vulnerability tool surface
Signed-off-by: Ben <ben@armosec.io>
…cription The description asserted that a missing 'relevant' count means relevancy was not computed. That overstates what can be known: the upstream field is `json:"relevant,omitempty"`, so an absent key is equally a genuine zero, and the two are indistinguishable in the source data. The design proposed deriving a relevancy availability verdict from the summary's kubescape.io/status annotation. Measurement rules that out -- workloads with status=ready were observed with 'relevant' absent -- so no such field is emitted. The description now states the ambiguity in both directions and tells the caller not to read an absent value as a measured zero. Signed-off-by: Ben Hirschberg <ben@armosec.io> Docs-exempt: tool description wording, no behavioral change Signed-off-by: Ben <ben@armosec.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An agent cannot currently answer "what are the worst vulnerabilities in my cluster?" without either being lied to or blowing its context window:
list_vulnerabilitiesreturns 200,793 B (~50k tokens) for a single image.get_vulnerability_detailsneeds acve_id, obtainable only from that 200 KB call.The Kubescape storage API already exposes a full aggregation ladder this provider never used —
cluster → namespace → workload → manifest → CVE, where every level except the last is a handful of integers plus a reference. This PR walks it.Every number below was measured on a live standalone cluster (kind, chart 1.40.3,
storage:v0.0.298,kubevuln:v0.3.159,keepLocal=true, no ARMO backend), driving the built binary over MCP stdio.Measured results
vulnerability_overview(cluster)vulnerability_overview(namespace)list_vulnerable_workloads(all ns)list_vulnerable_workloads(ns, limit 2)list_vulnerabilities— nginx:1.14.0, defaultlist_vulnerabilities—limit=0(all 293)list_vulnerabilities—severity=Criticalget_vulnerability_detailsA complete drill-down — cluster posture → worst workloads → that image's CVEs → one CVE in full — now costs ~11 KB total, and answers questions that previously had no answer.
The two new tools
kubescape_vulnerability_overview— severity totals per namespace from the server-side aggregates, worst first. One call.kubescape_list_vulnerable_workloads— workloads ranked by severity, each row carrying themanifest_namethatlist_vulnerabilitiestakes. One call.Live output, verified against
kubectl:{"cluster_totals": {"Critical": {"all": 99, "relevant": 53}, "High": {"all": 242, "relevant": 140}, ...}, "namespaces": [{"namespace": "verify-targets", "workload_count": 4, ...}], "next_step": "call kubescape_list_vulnerable_workloads with a namespace to rank its workloads"}{"workload": "deployment/vuln-nginx", "container": "nginx", "image": "docker.io/library/nginx:1.14.0", "severities": {"Critical": {"all": 76, "relevant": 30}, ...}, "manifest_name": "docker.io-library-nginx-1.14.0-e34030", "manifest_namespace": "kubescape", "workload_manifest_name": "replicaset-vuln-nginx-646cbbfc97-nginx-6e99-e15f"}resourceVersion=fullSpecis load-bearing — please read this bitThe storage server strips
specfrom every LIST response by default. Severity counters come back0andvulnerabilitiesRefcomes back empty. That is not an error, and it is indistinguishable from a healthy cluster with no vulnerabilities.Measured on the same objects, same moment:
?resourceVersion=fullSpecvulnerabilitysummariesvulnerabilitymanifestsummariesvulnerabilitymanifestsmatches=nilSo
nginx:1.14.0has 76 critical CVEs and a default LIST of its summary reports 0. This is the same failure shape as thevulnerability_countbug in #76, one level up — building these tools the obvious way would report a vulnerable cluster as clean. The sentinel isResourceVersionFullSpecinstorage/pkg/apis/softwarecomposition/register.go;GetListbranches on it.Verified with a Go probe that the typed client transmits it, capturing the wire request via
rest.Config.WrapTransport:No stripping, rewriting, rejection or warning; returned counts match what an individual GET reports.
RESTClient().Param("resourceVersion","fullSpec")also works as a fallback. Caveat: this covers plainList()only —Watch()has differentResourceVersionsemantics and was not tested.Because that correctness rests on a vendor sentinel,
overviewrefuses to present an all-zero cluster as good news — it reports that the result could not be confirmed and points at a per-manifest check. "No vulnerabilities" is the most expensive wrong answer this provider can give, so it should never be produced by a silent mechanism failure. Happy to drop this if you find it noisy.list_vulnerabilitiesreshapedseverity_summaryalways describes the whole manifest, even when the array is filtered or truncated — the aggregate survives a bounded response.limit(default 20), worst severity first,idas tiebreak for stable output, withtotal_count/returned_count/truncatedso the agent knows it holds partial data.id,severity,fix_state,affected_artifacts. Measured field shares of the old payload:description56.1%,data_source19.5%,fix_versions7.9%. The description was truncated at 200 chars mid-word ("...to prevent command inje..."), so it was lossy prose inviting reasoning from a fragment; full text, data source and fix versions are all inget_vulnerability_details.severityandfixable_onlyfilters.Also fixes a miscount:
severityCountshad noNegligiblebucket, so the 102 Negligible CVEs innginx:1.14.0were reported as"Unknown": 102— visible in the live response before this change. Severity handling is now driven by one ordered list used for both bucketing and ranking.Two defects the live run surfaced
Duplicate CVE rows. Grype emits one match per affected package, so the same CVE recurs. Once the package fields were trimmed those rows became byte-identical, and duplicates consumed the
limitbudget — the first 20 records heldCVE-2017-12424andCVE-2017-15670twice each. Matches are now collapsed by CVE id with anaffected_artifactscount, and a CVE left unfixed in any package is never reported as fixed. 466 matches → 293 distinct CVEs.vulnerabilitiesRefcarries an unusable namespace. It reports the workload's namespace, but manifests live in the Kubescape namespace:Only the name is taken from the ref; the namespace comes from the operator namespace, overridable via
kubescape_namespace. Passing the ref's namespace through would hand the agent a pointer that 404s. This looks like an upstream bug inkubescape/storageworth reporting separately.Relevancy
VulnerabilityCountersis{All, Relevant}, where relevant means node-agent observed the vulnerable code loaded at runtime — e.g. 76 critical, 30 reachable. Nothing surfaced this before; it's arguably the strongest triage signal in the product.It is emitted only when non-zero. The field is
json:"relevant,omitempty"upstream, so a zero is indistinguishable from "relevancy was never computed" — and node-agent needs a learning period, loggingcontainer profile is partial (workload restart required)until then. Rendering absent as0would claim nothing is reachable when nobody has looked yet, so the tool description states the ambiguity explicitly in both directions.There is deliberately no derived
relevancy: available|unavailablefield. The obvious signal for one does not work: workloads carryingkubescape.io/status=readywere measured withrelevantabsent, so the annotation says nothing about whether relevancy was computed. A confident availability verdict from it would be a guess dressed as a fact.Testing
go test ./pkg/kubescape/...— 68 pass, no cluster required.go build,go vet,golangci-lint, andgo test -tags=test ./pkg/... ./internal/...(747 pass) all clean. The pre-existingtest/e2esuite needs a live kind cluster and was not run.All tests written before the code and observed failing. One note: client-go's fake drops
ResourceVersionfrom recorded actions, so a reactor cannot observe the sentinel — the tests wrap the typed client to record theListOptionseach handler actually passes, pinning the invariant at the real call site rather than in the fake's bookkeeping.Open questions for reviewers
list_vulnerability_manifestsstill worth keeping? Oncelist_vulnerable_workloadsexists it is strictly weaker — same index, no counts, no pointers. Left untouched here for compatibility.overview— deliberate caution or noise?overviewandlist_vulnerable_workloadscould merge behind ascopeargument, at some cost in clarity for the calling model.Relationship to other PRs
Independent of #75 (health checks) and #76 (
vulnerability_count), all three branched frommain. #76 and this one both touchhandleListVulnerabilityManifests' neighbourhood, so whichever lands later needs a trivial rebase.Not included: the permanent
initErrorwith no re-init path, and theinternal/errorsfallback that marks unrecoverable errorsRetryable: Yes— the latter is visible in this PR's own validation error (invalid severity "Nope"reportsRetryable: Yes), which is worth fixing separately.Ticket
None.