Skip to content

fix(e2e): make shared supervisor harness work with gateway TLS - #4984

Merged
drew merged 1 commit into
NVIDIA:mainfrom
FrostGod:fix/e2e-shared-supervisor-harness/FrostGod
Oct 28, 2026
Merged

drew merged 1 commit into
NVIDIA:mainfrom
FrostGod:fix/e2e-shared-supervisor-harness/FrostGod

Conversation

@FrostGod

Copy link
Copy Markdown
Collaborator

Summary

The shared-supervisor e2e tasks (e2e:kubernetes:shared-supervisor, -ha, -scaling) fail during harness setup, before any test runs. This fixes the gateway port-forward readiness probe for TLS mode and waits for supervisor pool members to attach before the suites start.

Related Issue

No issue yet. Failures fixed:

  • start_gateway_portforward probes with plaintext curl http://.... With OPENSHELL_E2E_KUBE_TLS=1 the gateway's TLS listener resets the connection (curl exit 56), so the probe never passes and setup fails.
  • Pool members pass their TCP readiness probe 50–77 s before they attach to the gateway. Conformance starts immediately, and sandbox create fails with "the shared supervisor has no capacity for another sandbox instance".

Changes

  • start_gateway_portforward: in TLS mode, check that the forward accepts TCP instead of sending plaintext HTTP. Non-TLS mode is unchanged.
  • New wait_for_shared_supervisors, called after the health port-forward: a no-op unless shared pool Pods (app.kubernetes.io/component=supervisor) exist. Otherwise waits up to 180 s for every member to log gateway accepted the supervisor host (the signal the shared-supervisor tests use), then 2 s for the membership record.

Testing

  • bash -n e2e/with-kube-gateway.sh
  • Equivalent logic ran locally on a two-node k3d cluster: e2e:kubernetes:shared-supervisor, -ha, and -scaling all passed (conformance 8/8 plus each task's test); without it all three fail at setup
  • shellcheck (not available locally)
  • Re-run of the three tasks with this exact commit

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

The TLS port-forward readiness probe sent plaintext HTTP, which the gateway's
TLS listener resets, so the shared-supervisor e2e tasks failed before any
test ran. In TLS mode, check only that the forward accepts TCP.

Shared supervisor pool members are TCP-ready before they attach to the
gateway, and the gateway refuses sandbox creates until one attaches. Wait
for every pool member to attach before handing the gateway to the suites.

Signed-off-by: divesh <dgude@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@FrostGod
FrostGod requested a review from drew October 27, 2026 22:48
@drew

drew commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

/ok to test 937abc2

@drew drew added the test:e2e Requires end-to-end coverage label Oct 28, 2026
@drew
drew enabled auto-merge October 28, 2026 03:49
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 937abc2. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@FrostGod

Copy link
Copy Markdown
Collaborator Author

/ok to test 937abc2

@FrostGod

Copy link
Copy Markdown
Collaborator Author

/ok to test 937abc2

@drew
drew added this pull request to the merge queue Oct 28, 2026
Merged via the queue into NVIDIA:main with commit 21be349 Oct 28, 2026
201 of 203 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants