Skip to content

feat(kubescape): add vulnerability drill-down tools and bound CVE output - #77

Open
slashben wants to merge 2 commits into
kagent-dev:mainfrom
slashben:feat/kubescape-vulnerability-drilldown
Open

feat(kubescape): add vulnerability drill-down tools and bound CVE output#77
slashben wants to merge 2 commits into
kagent-dev:mainfrom
slashben:feat/kubescape-vulnerability-drilldown

Conversation

@slashben

@slashben slashben commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

An agent cannot currently answer "what are the worst vulnerabilities in my cluster?" without either being lied to or blowing its context window:

  • list_vulnerabilities returns 200,793 B (~50k tokens) for a single image.
  • get_vulnerability_details needs a cve_id, obtainable only from that 200 KB call.
  • Nothing aggregates across manifests, so cluster-scope questions have no answer at any price.

The Kubescape storage API already exposes a full aggregation ladder this provider never used — cluster → namespace → workload → manifest → CVE, where every level except the last is a handful of integers plus a reference. This PR walks it.

Every number below was measured on a live standalone cluster (kind, chart 1.40.3, storage:v0.0.298, kubevuln:v0.3.159, keepLocal=true, no ARMO backend), driving the built binary over MCP stdio.

Measured results

Call Before After
vulnerability_overview (cluster) impossible 1,114 B
vulnerability_overview (namespace) impossible 1,116 B
list_vulnerable_workloads (all ns) impossible 3,807 B
list_vulnerable_workloads (ns, limit 2) impossible 2,057 B
list_vulnerabilities — nginx:1.14.0, default 200,793 B 3,013 B (67× smaller)
list_vulnerabilitieslimit=0 (all 293) 200,793 B 38,707 B
list_vulnerabilitiesseverity=Critical n/a 3,020 B
get_vulnerability_details 5,104 B 5,104 B (unchanged)

A complete drill-down — cluster posture → worst workloads → that image's CVEs → one CVE in full — now costs ~11 KB total, and answers questions that previously had no answer.

The two new tools

kubescape_vulnerability_overview — severity totals per namespace from the server-side aggregates, worst first. One call.

kubescape_list_vulnerable_workloads — workloads ranked by severity, each row carrying the manifest_name that list_vulnerabilities takes. One call.

Live output, verified against kubectl:

{"cluster_totals": {"Critical": {"all": 99, "relevant": 53},
                    "High": {"all": 242, "relevant": 140}, ...},
 "namespaces": [{"namespace": "verify-targets", "workload_count": 4, ...}],
 "next_step": "call kubescape_list_vulnerable_workloads with a namespace to rank its workloads"}
{"workload": "deployment/vuln-nginx", "container": "nginx",
 "image": "docker.io/library/nginx:1.14.0",
 "severities": {"Critical": {"all": 76, "relevant": 30}, ...},
 "manifest_name": "docker.io-library-nginx-1.14.0-e34030",
 "manifest_namespace": "kubescape",
 "workload_manifest_name": "replicaset-vuln-nginx-646cbbfc97-nginx-6e99-e15f"}

resourceVersion=fullSpec is load-bearing — please read this bit

The storage server strips spec from every LIST response by default. Severity counters come back 0 and vulnerabilitiesRef comes back empty. That is not an error, and it is indistinguishable from a healthy cluster with no vulnerabilities.

Measured on the same objects, same moment:

Resource default LIST ?resourceVersion=fullSpec
vulnerabilitysummaries 903 B — all zeros 996 B — crit=99 high=242 med=230
vulnerabilitymanifestsummaries 12,908 B — all zeros, refs empty 7,648 B — correct counts + refs
vulnerabilitymanifests 14,054 B — matches=nil 3,635,500 B ⚠️ never

So nginx:1.14.0 has 76 critical CVEs and a default LIST of its summary reports 0. This is the same failure shape as the vulnerability_count bug in #76, one level up — building these tools the obvious way would report a vulnerable cluster as clean. The sentinel is ResourceVersionFullSpec in storage/pkg/apis/softwarecomposition/register.go; GetList branches on it.

Verified with a Go probe that the typed client transmits it, capturing the wire request via rest.Config.WrapTransport:

List(ctx, ListOptions{})                            → GET .../vulnerabilitymanifestsummaries?
List(ctx, ListOptions{ResourceVersion:"fullSpec"})  → GET .../vulnerabilitymanifestsummaries?resourceVersion=fullSpec

No stripping, rewriting, rejection or warning; returned counts match what an individual GET reports. RESTClient().Param("resourceVersion","fullSpec") also works as a fallback. Caveat: this covers plain List() only — Watch() has different ResourceVersion semantics and was not tested.

Because that correctness rests on a vendor sentinel, overview refuses to present an all-zero cluster as good news — it reports that the result could not be confirmed and points at a per-manifest check. "No vulnerabilities" is the most expensive wrong answer this provider can give, so it should never be produced by a silent mechanism failure. Happy to drop this if you find it noisy.

list_vulnerabilities reshaped

  • severity_summary always describes the whole manifest, even when the array is filtered or truncated — the aggregate survives a bounded response.
  • Array capped by limit (default 20), worst severity first, id as tiebreak for stable output, with total_count / returned_count / truncated so the agent knows it holds partial data.
  • Records carry id, severity, fix_state, affected_artifacts. Measured field shares of the old payload: description 56.1%, data_source 19.5%, fix_versions 7.9%. The description was truncated at 200 chars mid-word ("...to prevent command inje..."), so it was lossy prose inviting reasoning from a fragment; full text, data source and fix versions are all in get_vulnerability_details.
  • New severity and fixable_only filters.

Also fixes a miscount: severityCounts had no Negligible bucket, so the 102 Negligible CVEs in nginx:1.14.0 were reported as "Unknown": 102 — visible in the live response before this change. Severity handling is now driven by one ordered list used for both bucketing and ranking.

Two defects the live run surfaced

Duplicate CVE rows. Grype emits one match per affected package, so the same CVE recurs. Once the package fields were trimmed those rows became byte-identical, and duplicates consumed the limit budget — the first 20 records held CVE-2017-12424 and CVE-2017-15670 twice each. Matches are now collapsed by CVE id with an affected_artifacts count, and a CVE left unfixed in any package is never reported as fixed. 466 matches → 293 distinct CVEs.

vulnerabilitiesRef carries an unusable namespace. It reports the workload's namespace, but manifests live in the Kubescape namespace:

$ kubectl get vulnerabilitymanifestsummary -n verify-targets deployment-vuln-nginx-nginx -o jsonpath='{.spec.vulnerabilitiesRef.all}'
{"kind":"vulnerabilitymanifests","name":"docker.io-library-nginx-1.14.0-e34030","namespace":"verify-targets"}

$ kubectl get vulnerabilitymanifest -n verify-targets docker.io-library-nginx-1.14.0-e34030
Error from server (NotFound): ... not found
$ kubectl get vulnerabilitymanifest -n kubescape docker.io-library-nginx-1.14.0-e34030
docker.io-library-nginx-1.14.0-e34030   2026-09-02T07:51:06Z

Only the name is taken from the ref; the namespace comes from the operator namespace, overridable via kubescape_namespace. Passing the ref's namespace through would hand the agent a pointer that 404s. This looks like an upstream bug in kubescape/storage worth reporting separately.

Relevancy

VulnerabilityCounters is {All, Relevant}, where relevant means node-agent observed the vulnerable code loaded at runtime — e.g. 76 critical, 30 reachable. Nothing surfaced this before; it's arguably the strongest triage signal in the product.

It is emitted only when non-zero. The field is json:"relevant,omitempty" upstream, so a zero is indistinguishable from "relevancy was never computed" — and node-agent needs a learning period, logging container profile is partial (workload restart required) until then. Rendering absent as 0 would claim nothing is reachable when nobody has looked yet, so the tool description states the ambiguity explicitly in both directions.

There is deliberately no derived relevancy: available|unavailable field. The obvious signal for one does not work: workloads carrying kubescape.io/status=ready were measured with relevant absent, so the annotation says nothing about whether relevancy was computed. A confident availability verdict from it would be a guess dressed as a fact.

Testing

go test ./pkg/kubescape/...68 pass, no cluster required. go build, go vet, golangci-lint, and go test -tags=test ./pkg/... ./internal/... (747 pass) all clean. The pre-existing test/e2e suite needs a live kind cluster and was not run.

All tests written before the code and observed failing. One note: client-go's fake drops ResourceVersion from recorded actions, so a reactor cannot observe the sentinel — the tests wrap the typed client to record the ListOptions each handler actually passes, pinning the invariant at the real call site rather than in the fake's bookkeeping.

Open questions for reviewers

  1. Is list_vulnerability_manifests still worth keeping? Once list_vulnerable_workloads exists it is strictly weaker — same index, no counts, no pointers. Left untouched here for compatibility.
  2. The all-zero warning in overview — deliberate caution or noise?
  3. Provider goes 10 → 12 tools. If that is too much surface, overview and list_vulnerable_workloads could merge behind a scope argument, at some cost in clarity for the calling model.

Relationship to other PRs

Independent of #75 (health checks) and #76 (vulnerability_count), all three branched from main. #76 and this one both touch handleListVulnerabilityManifests' neighbourhood, so whichever lands later needs a trivial rebase.

Not included: the permanent initError with no re-init path, and the internal/errors fallback that marks unrecoverable errors Retryable: Yes — the latter is visible in this PR's own validation error (invalid severity "Nope" reports Retryable: Yes), which is worth fixing separately.

Ticket

None.

An agent could not answer "what are the worst vulnerabilities in my cluster?"
without either being lied to or blowing its context window. The only route to
any severity information was list_vulnerabilities, which returned 200,793 B
(~50k tokens) for a single image, and nothing aggregated across manifests at
all, so cluster-scope questions had no answer at any price.

The Kubescape storage API already exposes a full aggregation ladder that this
provider never used: cluster -> namespace -> workload -> manifest -> CVE, where
every level except the last is a handful of integers plus a reference.

Two new tools walk it:

  kubescape_vulnerability_overview
      Severity totals per namespace from the server-side aggregates, worst
      first. One call, ~1 KB, from vulnerabilitysummaries (cluster-scoped, one
      object per namespace).

  kubescape_list_vulnerable_workloads
      Workloads ranked by severity, each row carrying the manifest_name that
      kubescape_list_vulnerabilities takes, read from the summary's
      spec.vulnerabilitiesRef. One call, ~1.9 KB per workload.

Both take ListOptions.ResourceVersion = "fullSpec".

That sentinel is load-bearing and not obvious. The storage server strips spec
from EVERY list response by default, so severity counters come back zero and
vulnerabilitiesRef comes back empty -- not an error, and indistinguishable from
a clean cluster. Measured on storage v0.0.298, nginx:1.14.0 has 76 critical CVEs
and a default LIST of its summary reports 0. This is the same failure shape as
the vulnerability_count bug, one level up, so listing these resources without
the sentinel would report a vulnerable cluster as clean.

Verified against a live cluster that the typed client transmits it: a wire
capture shows resourceVersion=fullSpec on the request, and the returned counts
match the values an individual GET reports.

Because that correctness rests on a vendor sentinel, the overview also refuses
to present an all-zero cluster as good news: it says the result could not be
confirmed and points at a per-manifest check. A wrong "no vulnerabilities" is
the most expensive answer this provider can give.

kubescape_list_vulnerabilities is reshaped to fit the ladder:

  - severity_summary always describes the WHOLE manifest, even when the array
    is filtered or truncated, so the aggregate survives a bounded response.
  - The array is capped by limit (default 20), ordered worst severity first with
    id as a tiebreak for stable output, and reports total_count, returned_count
    and truncated so the agent knows it holds partial data.
  - Records carry only id, severity and fix_state. Measured field shares of the
    old payload: description 56.1%, data_source 19.5%, fix_versions 7.9%. The
    description was truncated at 200 chars mid-word, so it was lossy prose that
    invited reasoning from a fragment; the full text, data source and fix
    versions are all in get_vulnerability_details.
  - New severity and fixable_only filters for drill-down.

  - Grype emits one match per affected package, so the same CVE recurs. With the
    package fields trimmed away those rows are byte-identical, and duplicates
    would consume the limit budget: the first 20 records for nginx:1.14.0 held
    CVE-2017-12424 and CVE-2017-15670 twice each. Matches are now collapsed by
    CVE id with an affected_artifacts count, and a CVE left unfixed in any
    package is never reported as fixed. 466 matches become 293 distinct CVEs.

  Measured against the same image on a live cluster: 3,013 B, against 200,793 B
  before -- 67x smaller, with a complete and correct severity summary.

Also fixes a miscount in that summary: severityCounts had no Negligible bucket,
so the 102 Negligible CVEs in nginx:1.14.0 were reported as "Unknown": 102.
Severity handling is now driven by one ordered list used for both bucketing and
ranking, so a severity cannot be dropped into the wrong bucket again.

Relevancy is surfaced where the API provides it -- "relevant" counts CVEs whose
code node-agent observed loaded at runtime, e.g. 76 critical of which 30 are
reachable. It is emitted only when non-zero: the field is `json:"relevant,omitempty"`
upstream, so a zero is indistinguishable from "relevancy was never computed",
and node-agent needs a learning period before it reports anything. Rendering
absent as 0 would claim nothing is reachable when nobody has looked yet.

One value from the summaries cannot be trusted. vulnerabilitiesRef reports the
WORKLOAD's namespace, but the manifests live in the Kubescape namespace, so
following it verbatim yields NotFound -- confirmed on a live cluster, where a GET
of docker.io-library-nginx-1.14.0-e34030 in the referenced namespace fails while
the same GET in `kubescape` succeeds. Only the name is taken from the ref; the
namespace comes from the operator namespace, overridable with kubescape_namespace.
Handing the agent a pointer that 404s would defeat the point of the chain.

Tests are fake-client based and need no cluster. Note that client-go's fake
drops ResourceVersion from recorded actions, so a reactor cannot observe the
sentinel; the tests wrap the typed client to record the ListOptions each handler
actually passes, which pins the invariant at the real call site.

Signed-off-by: Ben Hirschberg <ben@armosec.io>

Docs-exempt: no existing doc describes the vulnerability tool surface
Signed-off-by: Ben <ben@armosec.io>
…cription

The description asserted that a missing 'relevant' count means relevancy was
not computed. That overstates what can be known: the upstream field is
`json:"relevant,omitempty"`, so an absent key is equally a genuine zero, and
the two are indistinguishable in the source data.

The design proposed deriving a relevancy availability verdict from the summary's
kubescape.io/status annotation. Measurement rules that out -- workloads with
status=ready were observed with 'relevant' absent -- so no such field is
emitted. The description now states the ambiguity in both directions and tells
the caller not to read an absent value as a measured zero.

Signed-off-by: Ben Hirschberg <ben@armosec.io>

Docs-exempt: tool description wording, no behavioral change
Signed-off-by: Ben <ben@armosec.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant