Kubernetes: image pull failures (sandbox and supervisor image) give no diagnostic feedback in sandbox status, sandbox create or logs #5028

User Story

I run OpenShell on Kubernetes in managed workspace mode. When a sandbox image couldn't be pulled,
the CLI's output and logs made no mention of an invalid image. That sent me down the wrong path of
investigation, when the problem was simply an image that was invalid or couldn't be pulled. The
status also showed PodScheduled, which made me think the image had been pulled successfully. Only
the Kubernetes events showed that the pull had failed.

OpenShell doesn't provide any logs or information about the pod events, which would help diagnose
the issue. The same goes for the supervisor: the CLI never returns information about problems with
the supervisor pod. In short, the OpenShell CLI doesn't give proper diagnostic feedback on
image-related failures.

Problem Statement

In Kubernetes mode, a sandbox whose workload image or supervisor image can't be pulled stays
Provisioning with Ready=False ConfigurationPending ("Waiting for effective configuration
validation before workload activation") for the whole wait. After the image-preparation deadline
(30 min by default) it moves to Error with a generic ImagePreparationTimedOut. Neither the
status nor the event stream ever names the image or the pull error. Two gaps cause this.

1. Status: the reason is overwritten. The reason is available:

• the workload pod has waiting.reason: ImagePullBackOff with the kubelet's message;
• Agent Sandbox v1.0.8 puts it into the Sandbox CR's Ready condition:
  Pod exists with phase: Pending; init container "workspace-init" is waiting: ImagePullBackOff;
• the supervisor pod os-supervisor-<sandbox id> has ErrImagePull / ImagePullBackOff.

It is lost twice:

• the Kubernetes driver replaces Ready with DependenciesNotReady while the sandbox runtime
  bootstraps;
• the gateway then replaces Ready with ConfigurationPending until configuration is accepted.

The supervisor pod's state isn't read at all. In the supervisor case the workload pod even shows
Running 1/1 while OpenShell reports Provisioning.

2. Events: not forwarded in managed and operator modes. Only the shared-namespace watch watches
Kubernetes Events. In managed and operator workspace modes the driver uses the cluster-wide
watch, which watches Sandbox CRs only. So sandbox create prints only "Requesting compute..."
with no progress steps, and openshell logs shows none of the pod's events (Scheduled, Pulling,
Failed, BackOff). The Helm chart already grants cluster-wide events list/watch, so this is a
driver gap, not a permissions one. Supervisor pod events aren't matched to their sandbox even in
shared mode, because the pod's name is neither the Sandbox CR's name nor the indexed pod name.

Impact / Why This Matters

Image problems are among the most common first-run failures: a typo in a tag, a private registry
without pull secrets, a mirror that doesn't have the image. Today the only way to diagnose them is
kubectl access to the workspace namespace, which CLI users often don't have. That workaround
needs cluster permissions and knowledge of OpenShell's internal pod names, including the
supervisor pod's.

Acceptance Criteria

• When the workload image can't be pulled, sandbox get shows a condition with the pull reason
  (for example ImagePullBackOff) and the image reference, and sandbox create prints it when it
  gives up waiting.
• The same holds for the supervisor image, naming it as the supervisor image.
• The reason survives the driver's and the gateway's Ready rewrites. For example, it's kept as its
  own condition, or carried into the DependenciesNotReady / ConfigurationPending message.
• In managed and operator modes, events for a sandbox's pods (workload and supervisor) reach
  WatchSandbox / follow_events as they do in shared mode. They show up in sandbox create
  progress and in openshell logs.
• Optional: openshell status (or gateway health) warns when the configured supervisor image
  can't be pulled, so a bad supervisor tag is visible before the first sandbox.

Reproduction Steps

Versions: OpenShell 0.1.6 (Helm chart oci://ghcr.io/nvidia/openshell/helm-chart 0.1.6 and the
CLI 0.1.6), Agent Sandbox v1.0.8 (sandbox-with-extensions.yaml), kind v0.33.0, Kubernetes v1.37.0.
Helm values:

server:
  telemetryEnabled: false
  defaultRuntimeClassName: runc
  drivers:
    kubernetes:
      workspaceMode: managed

Workload image:

openshell --workspace repro sandbox create --name img-missing \
  --from docker.io/library/busybox:0.0.0-does-not-exist --no-auto-providers --no-tty -- true
# Requesting compute...
# sandbox provisioning timed out ... Last reported status: ConfigurationPending:
#   Waiting for effective configuration validation before workload activation
openshell --workspace repro sandbox get img-missing
# Phase Provisioning; PodScheduled=True; Bootstrapping=True SandboxRuntimeGenerationStarting;
# Ready=False ConfigurationPending. Nothing about the image.
openshell --workspace repro logs img-missing
# only "Creating sandbox in Kubernetes", "Fetching sandbox from Kubernetes",
# "Starting sandbox supervision"

# What Kubernetes knows:
kubectl -n openshell-openshell-repro get pods
# img-missing   0/1   Init:ImagePullBackOff
kubectl -n openshell-openshell-repro get sandboxes.agents.x-k8s.io img-missing -o jsonpath='{.status.conditions}'
# ... "Pod exists with phase: Pending; init container \"workspace-init\" is waiting: ImagePullBackOff"
kubectl -n openshell-openshell-repro get events --field-selector involvedObject.name=img-missing
# Scheduled, Pulling, Failed ("... not found"), ErrImagePull, BackOff, ImagePullBackOff

Supervisor image:

helm upgrade openshell oci://ghcr.io/nvidia/openshell/helm-chart --version 0.1.6 -n openshell \
  -f values.yaml --set supervisor.image.tag=0.0.0-does-not-exist --wait
openshell --workspace repro sandbox create --name sup-bad --no-auto-providers --no-tty -- true
# same ConfigurationPending timeout, nothing about the supervisor
kubectl -n openshell-openshell-repro get pods
# os-supervisor-<sandbox id>   0/1   ImagePullBackOff
# sup-bad                      1/1   Running

Suggested UX (if applicable)

No response

Environment

• OpenShell: 0.1.6 (openshell --version: openshell 0.1.6), gateway from Helm chart oci://ghcr.io/nvidia/openshell/helm-chart 0.1.6
• OS: macOS 27.0.1 (arm64), Docker Engine 29.8.2 (OrbStack)
• Deployment: Kubernetes driver, workspaceMode: managed, kind v0.33.0 / Kubernetes v1.37.0 (single node), RuntimeClass runc
• Agent Sandbox: v1.0.8 (sandbox-with-extensions.yaml)
• Helm values: server.telemetryEnabled=false, server.defaultRuntimeClassName=runc, server.drivers.kubernetes.workspaceMode=managed

Logs

openshell --workspace repro sandbox get img-missing          # or add -o json        ☁  ☸ kind-koinon-osrepro

Sandbox:

  Id: 2ff21e26-1a7b-49a2-a640-14ae153f93ea
  Name: img-missing
  Phase: Provisioning
  Configuration: ConfigurationPending: Waiting for effective configuration validation before workload activation
  Resource version: 32
  Restart policy: never
  Main process instance: -
  Last exit code: -
  Restart count: 0
  Next restart: -
  Annotations:
    internal.openshell.ai/auth-epoch: 1
    internal.openshell.ai/compute-driver: kubernetes
    internal.openshell.ai/compute-runtime-identity: kubernetes://openshell-openshell-repro/876414ec-8379-42b9-8deb-a047f4214891/d9710c1d-2d01-4940-95b4-71b07f1976ff
    internal.openshell.ai/gateway-token-id: c3926377-34f0-4836-bc46-56202ef63793
    internal.openshell.ai/isolation-backend: openshell-sandbox
    internal.openshell.ai/runtime-generation: 1be2c98e-78c1-4d3a-8823-66c79c05271d
  Conditions:
    - PodScheduled: True (PodScheduled)
      Last transition: 2026-10-29T10:10:36Z
    - Bootstrapping: True (SandboxRuntimeGenerationStarting) - replacement sandbox-runtime generation is starting
    - ConfigurationReady: False (ConfigurationPending) - Waiting for effective configuration validation before workload activation
    - Ready: False (ConfigurationPending) - Waiting for effective configuration validation before workload activation
  Policy source: sandbox

kc -n $N get pods                                   # img-missing  0/1  Init:ImagePullBackOff
kc -n $N get pod img-missing -o jsonpath='{.status.initContainerStatuses[*].state.waiting}{"\n"}'
kc -n $N get sandboxes.agents.x-k8s.io img-missing -o jsonpath='{.status.conditions}{"\n"}'

NAME                                                 READY   STATUS                  RESTARTS   AGE
img-missing                                          0/1     Init:ImagePullBackOff   0          4m52s
os-supervisor-2ff21e26-1a7b-49a2-a640-14ae153f93ea   0/1     Running                 0          4m52s
{"message":"Back-off pulling image \"docker.io/library/busybox:0.0.0-does-not-exist\": ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image \"docker.io/library/busybox:0.0.0-does-not-exist\": failed to resolve reference \"docker.io/library/busybox:0.0.0-does-not-exist\": docker.io/library/busybox:0.0.0-does-not-exist: not found","reason":"ImagePullBackOff"}
[{"lastTransitionTime":"2026-10-29T10:10:32Z","message":"Sandbox is not suspended","observedGeneration":2,"reason":"NotSuspended","status":"False","type":"Suspended"},{"lastTransitionTime":"2026-10-29T10:10:32Z","message":"Pod exists with phase: Pending; init container \"workspace-init\" is waiting: ImagePullBackOff","observedGeneration":2,"reason":"DependenciesNotReady","status":"False","type":"Ready"},{"lastTransitionTime":"2026-10-29T10:10:36Z","message":"","observedGeneration":2,"reason":"PodScheduled","status":"True","type":"PodScheduled"}]