fix(kubescape): report health correctly and make the level filter work

[slashben]

Summary

kubescape_check_health reports healthy: false on a fully working Kubescape install, and the advertised level filter on kubescape_list_vulnerability_manifests silently does nothing. Both were verified on a live cluster running the kubescape-operator chart 1.40.4 (scanner v4.0.12), with all 7 pods Running and every data tool returning correct results.

Three fixes, all in pkg/kubescape. No new dependencies; one client removed.

1. check_health probed for CRDs that never exist

The spdx.softwarecomposition.kubescape.io resources are served by an aggregated API server — the storage pod, wired via APIService v1beta1.spdx.softwarecomposition.kubescape.io. They are not CRDs, and no CRD by those names exists in any configuration.

handleCheckHealth did four CustomResourceDefinitions().Get(...) lookups, which therefore return IsNotFound on every install, forever:

"healthy": false,
"summary": "Kubescape has issues that need attention",
"vulnerability_crd": {"status":"error","message":"VulnerabilityManifests CRD not installed - vulnerability scanning may not be enabled"},
"configuration_crd": {"status":"error","message":"WorkloadConfigurationScans CRD not installed - configuration scanning may not be enabled"}

…emitted while kubectl get vulnerabilitymanifests -A returned data. Confirmed independently:

$ kubectl get crd | grep spdx.softwarecomposition          # -> nothing
$ kubectl get apiservice v1beta1.spdx.softwarecomposition.kubescape.io \
    -o jsonpath='{.spec.service}{" "}{.status.conditions[*].type}={.status.conditions[*].status}'
{"name":"storage","namespace":"kubescape","port":443} Available=True
$ kubectl api-resources --api-group=spdx.softwarecomposition.kubescape.io | wc -l   # -> 15

check_health is the tool an agent calls first, and answering "is this working?" is its whole job — so an agent would tell the user Kubescape is broken and recommend reinstalling something already installed correctly.

Fix. Availability is now probed by listing each resource through the spdxClient the data tools already use. That proves the exact path those tools depend on rather than a proxy for it — discovery could report a resource as registered while the storage pod is dead and every tool still fails. One list per resource answers both "is the API reachable?" and "is there data?", so this makes four fewer API calls than before. The apiextensions client is no longer needed and is removed.

Check keys are unchanged (vulnerability_crd, configuration_crd, …) so existing consumers of the output are unaffected; only the lookup and the message wording change. "CRD not installed" becomes "VulnerabilityManifests API not available - the Kubescape storage service may be unavailable, or vulnerability scanning may not be enabled".

This also fixes a latent gating bug: application_profiles_data and network_neighborhoods_data were nested inside the always-failing CRD lookup, so those two checks never ran at all.

2. Health pod selectors matched the wrong pods

app.kubernetes.io/name=kubescape-operator is the chart-wide label carried by every pod the chart creates. Per-component identity is the plain app label:

$ kubectl get pods -n kubescape -o custom-columns='NAME:.metadata.name,APP_NAME:.metadata.labels.app\.kubernetes\.io/name,APP:.metadata.labels.app'
kubescape-64c56c75c6-hjnfc   kubescape-operator   kubescape
kubevuln-c6bb59f9c-v5xlq     kubescape-operator   kubevuln
node-agent-7vfq2             kubescape-operator   node-agent
operator-5d854fdc8f-j75sb    kubescape-operator   operator
storage-5ff6f76c7f-ngb2d     kubescape-operator   storage

$ kubectl get pods -n kubescape -l app.kubernetes.io/name=kubescape-operator | wc -l   # -> 7
$ kubectl get pods -n kubescape -l app.kubernetes.io/name=storage            | wc -l   # -> 0

So "operator_pods": "7/7 pods running" was really an all-kubescape-pods count — if node-agent died you got 6/7 with no way to tell which component failed — and "storage_pods": "No storage pods found" was emitted while storage-5ff6f76c7f-ngb2d was 1/1 Running.

Fix. Selectors are now app=operator and app=storage.

A missing storage pod is now an error that fails the health check, not a warning: every data tool in this provider reads through the storage service, so the one check that would catch the failure mode breaking all nine of them previously matched nothing and would not have failed the check if it had.

Worth a reviewer's eye: app=operator is measured correct on chart 1.40.4. If an older chart labels differently this swaps a false pass for a false fail. I think that trade is right — a false fail is visible and fixable, whereas today's false fail is unconditional — but I'm happy to add a fallback to the chart-wide selector if you'd prefer.

3. The level filter silently did nothing

list_vulnerability_manifests advertises level: image, workload, or both. The provider's logic was correct and the labels really exist on the objects, but all three calls returned byte-identical output:

call	bytes	sha256[:12]
no level	1898	6f1e27dfb8fb
level=image	1898	6f1e27dfb8fb
level=workload	1898	6f1e27dfb8fb

The root cause is the storage server version the chart deploys. Storage servers before v0.0.305 ignore labelSelector on list — a nonsense selector returns every row — across all six resources:

resource                        all  bogus-selector  honored?
vulnerabilitymanifests           5              5    IGNORED
workloadconfigurationscans       8              8    IGNORED
applicationprofiles             10             10    IGNORED
networkneighborhoods            10             10    IGNORED
sbomsyfts                        4              4    IGNORED
configurationscansummaries       2              2    IGNORED
control: pods                   29              0    honored

kubescape/storage#362 added selector support and shipped in v0.0.305 (2026-08-20), so this is already fixed upstream. But kubescape-operator still pins storage: v0.0.298 (2026-07-25, seven releases earlier), which is what the measurements above were taken against, and a client of this API cannot know which server version it is talking to. kubescape/storage#363 tracks the remaining generated/aggregated resources.

Fix. Filtering now happens client-side on isImageLevel — the same predicate already computed to populate the image_level/workload_level fields in the response — so the filter and the output cannot disagree, and it returns the same answer against a pre- or post-v0.0.305 storage server. An unrecognised level is now a validation error instead of silently meaning "both".

Once the chart ships a storage ≥ v0.0.305, pushing this back to a server-side selector would save transferring the unfiltered list. That would need a version floor or a capability probe to stay correct on older installs, so I have left it client-side here; happy to revisit if you would rather gate on a minimum storage version.

Testing

go test ./pkg/kubescape/... — 60 pass, no cluster required. Full make fmt vet lint test clean (./pkg/... ./internal/..., 739 pass). The pre-existing test/e2e suite needs a live kind cluster and was not run.

New tests were written first and observed failing against the current code. Two points worth noting:

• Health tests seed pods carrying both labels — the chart-wide one and app=<component> — i.e. the real shape. That is what makes the old selector fail and the new one pass.
• The level-filter test installs a reactor that reproduces a pre-v0.0.305 storage server ignoring labelSelector, returning every object regardless. Without it the fake clientset honours selectors, and a server-side filter passes the test while returning unfiltered results against the storage version the chart actually deploys — which is exactly how this bug survived.

Two existing tests whose premise was "no CRDs installed" are rewritten to exercise real API-unavailability via a ServiceUnavailable reactor, including a partial-degradation case (one resource down, another still reporting ok).

Out of scope

Deliberately not in this PR, to keep it reviewable: unbounded tool output (list_vulnerabilities returned 200 KB / ~50k tokens for a single image on a 4-workload cluster), the permanent initError with no re-init path, and the internal/errors fallback that marks unrecoverable errors Retryable: Yes. Happy to open issues for these if useful.

Ticket

None.