From 83f9de6dd99310dfd1f843fa4088b6d8b313df7b Mon Sep 17 00:00:00 2001 From: Tyler Potts <49161327+tylerpotts@users.noreply.github.com> Date: Mon, 24 Aug 2026 14:58:41 -0500 Subject: [PATCH 1/4] docs: design spec for GPU profile image auto-derivation (#230) --- .../2026-08-24-gpu-profile-image-design.md | 79 +++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-24-gpu-profile-image-design.md diff --git a/docs/superpowers/specs/2026-08-24-gpu-profile-image-design.md b/docs/superpowers/specs/2026-08-24-gpu-profile-image-design.md new file mode 100644 index 0000000..bab0e6c --- /dev/null +++ b/docs/superpowers/specs/2026-08-24-gpu-profile-image-design.md @@ -0,0 +1,79 @@ +# GPU profile image auto-derivation + +Issue: https://github.com/nebari-dev/data-science-pack/issues/230 + +## Problem + +Deployers who add GPU JupyterLab profiles must hardcode +`quay.io/nebari/nebari-data-science-pack-jupyterlab-gpu:sha-` in their +overlay. CPU profiles inherit the chart's image tag on every pack update; GPU +profiles silently fall behind until someone remembers to bump the SHA. + +Both `nebari-data-science-pack-jupyterlab` and +`nebari-data-science-pack-jupyterlab-gpu` are built by the same +`build-images.yaml` workflow from the same commit, so they always share the +same `sha-` tag. The chart therefore already knows the correct GPU +image ref: `-gpu:`. + +## Design + +A new per-profile boolean `gpu: true` in `jupyterhub.custom.profiles`: + +```yaml +- slug: gpu + display_name: "GPU Access" + gpu: true + access: yaml + groups: [gpu-access] + kubespawner_override: + # no image needed — injected automatically + node_selector: {node.kubernetes.io/instance-type: g4dn.xlarge} + extra_resource_limits: {nvidia.com/gpu: 1} +``` + +### Components + +1. **Helm helper** `nebari-data-science-pack.gpuJupyterlabImage` + (`templates/_helpers.tpl`): renders + `<.Values.jupyterhub.singleuser.image.name>-gpu:`; empty when + name/tag unset. + +2. **Chart-derived config** (`templates/hub-config.yaml`): add + `"gpu-image"` to `_CHART_DERIVED`, following the existing `nebi-image` + pattern. Deployers can override via `jupyterhub.custom.gpu-image` + (documented as `""` placeholder in `values.yaml`). + +3. **Spawner config** (`config/jupyterhub/01-spawner.py`): at load, + `_resolve_gpu_profiles(profiles, gpu_image)` walks `custom.profiles`: + * `gpu: true` and no `kubespawner_override.image` → inject `gpu_image` + * `gpu: true` with explicit image → leave the image alone (explicit wins) + * the `gpu` key is always stripped so KubeSpawner never sees it + * `gpu_image` empty → strip key, inject nothing (falls back to the + z2jh singleuser default image, today's behavior) + + Resolution happens once at module load, before `_render_profile_list` + filtering, so jhub-apps' server-types endpoint sees the same image. + +### Automatic tag currency + +`scripts/bump_image_tags.py` already bumps `jupyterhub.singleuser.image.tag` +every release; the derived GPU ref follows with zero script changes. + +### Out of scope (YAGNI) + +* Auto-injecting `extra_resource_limits` / tolerations — cluster-specific, + already documented. +* `profile_options.image.choices` for GPU profiles — deployer-defined + profiles rarely carry them; explicit choices keep winning if present. + +## Testing + +* Unit (`tests/unit/test_spawner_profiles.py`): injection, explicit-image + precedence, key stripping, empty-gpu-image fallback, load-time wiring. +* Chart (`tests/unit/test_chart_derived.py`): rendered `_CHART_DERIVED` + contains the derived `gpu-image` ref from the default values. + +## Docs + +* `docs/src/content/docs/server-profiles.md` GPU section: document `gpu: true`. +* `values.yaml` comments for `gpu-image` + profile example. From 03cc910a5bf75a450490fef47c99cf7e78c73991 Mon Sep 17 00:00:00 2001 From: Tyler Potts <49161327+tylerpotts@users.noreply.github.com> Date: Mon, 24 Aug 2026 15:03:04 -0500 Subject: [PATCH 2/4] feat(profiles): auto-inject GPU jupyterlab image for gpu: true profiles Profiles marked gpu: true in jupyterhub.custom.profiles get kubespawner_override.image set to the chart-derived GPU image (-gpu:) unless an explicit image is present. Both jupyterlab images are built from the same commit with the same sha tag, so GPU profiles now track pack updates automatically instead of pinning a -gpu SHA in the deployer overlay. Deployers can override the injected ref chart-wide via jupyterhub.custom.gpu-image. The gpu key is stripped before profiles reach KubeSpawner. Closes https://github.com/nebari-dev/data-science-pack/issues/230 --- config/jupyterhub/01-spawner.py | 29 +++++++- docs/src/content/docs/server-profiles.md | 31 ++++++++ docs/src/content/docs/values-reference.md | 1 + templates/_helpers.tpl | 13 ++++ templates/hub-config.yaml | 4 + tests/unit/test_chart_derived.py | 22 ++++++ tests/unit/test_spawner_profiles.py | 91 +++++++++++++++++++++++ values.yaml | 11 +++ 8 files changed, 201 insertions(+), 1 deletion(-) diff --git a/config/jupyterhub/01-spawner.py b/config/jupyterhub/01-spawner.py index 917ee4b..dd71a4d 100644 --- a/config/jupyterhub/01-spawner.py +++ b/config/jupyterhub/01-spawner.py @@ -442,7 +442,34 @@ async def _render_profile_list(spawner): return visible -_profiles = get_config("custom.profiles", []) +def _resolve_gpu_profiles(profiles, gpu_image): + """Inject the chart-derived GPU image into ``gpu: true`` profiles. + + Both jupyterlab images are built from the same commit with the same + ``sha-`` tag, so the chart derives the GPU ref from + ``singleuser.image`` (override via ``custom.gpu-image``). Marking a + profile ``gpu: true`` keeps its image current across pack updates + without hardcoding a SHA in the deployer overlay (issue #230). + + An explicit ``kubespawner_override.image`` always wins. The ``gpu`` key + is stripped either way — KubeSpawner must never see it. Returns new + dicts; the input list is left untouched. + """ + resolved = [] + for profile in profiles: + if profile.get("gpu"): + profile = {k: v for k, v in profile.items() if k != "gpu"} + override = dict(profile.get("kubespawner_override") or {}) + if gpu_image and not override.get("image"): + override["image"] = gpu_image + profile["kubespawner_override"] = override + resolved.append(profile) + return resolved + + +_profiles = _resolve_gpu_profiles( + get_config("custom.profiles", []), get_chart_config("gpu-image") +) if _profiles: c.KubeSpawner.profile_list = _render_profile_list log.info( diff --git a/docs/src/content/docs/server-profiles.md b/docs/src/content/docs/server-profiles.md index 7693945..5fc960c 100644 --- a/docs/src/content/docs/server-profiles.md +++ b/docs/src/content/docs/server-profiles.md @@ -150,6 +150,37 @@ kubectl -n data-science logs deploy/hub | grep -i "profiles:\|groups" ## GPU profiles +### The GPU image, without hardcoding a SHA + +Mark a profile `gpu: true` and the chart injects the matching GPU JupyterLab image +(`nebari-data-science-pack-jupyterlab-gpu`) at the chart's current tag: + +```yaml + - slug: gpu-instance + display_name: "G4 GPU Instance" + gpu: true + access: yaml + groups: + - gpu-access + kubespawner_override: + # no image needed — the -gpu image is injected automatically + node_selector: + node.kubernetes.io/instance-type: g4dn.xlarge + extra_resource_limits: + nvidia.com/gpu: 1 +``` + +Both JupyterLab images are built from the same commit and share the same `sha-` tag, so +the derived ref (`-gpu:`) is always valid and +GPU profiles now track pack updates exactly like CPU profiles — no more stale SHAs in the +overlay ([issue #230](https://github.com/nebari-dev/data-science-pack/issues/230)). + +An explicit `kubespawner_override.image` always wins over the injection, and +`jupyterhub.custom.gpu-image` overrides which image gets injected chart-wide. Like the +gating keys, `gpu` is stripped before the profile reaches KubeSpawner. + +### Scheduling onto GPU nodes + A GPU profile requests the resource through `extra_resource_limits`, but scheduling onto a tainted GPU node group also needs a toleration — whether you must add it yourself depends on whether the cluster runs the `ExtendedResourceToleration` admission controller (EKS and GKE diff --git a/docs/src/content/docs/values-reference.md b/docs/src/content/docs/values-reference.md index 0e1b61d..d695d35 100644 --- a/docs/src/content/docs/values-reference.md +++ b/docs/src/content/docs/values-reference.md @@ -190,6 +190,7 @@ always win. | `external-url` | `""` *(derived)* | Hub bind hostname. | | `nebi-image` | `""` *(derived)* | `repository:tag` copied into user pods. | | `nebi-image-pull-policy` | `IfNotPresent` | — | +| `gpu-image` | `""` *(derived)* | GPU JupyterLab image injected into `gpu: true` profiles. Derived as `-gpu:` — see [Server profiles](/server-profiles/#gpu-profiles). | | `jhub-app-proxy-version` | `v0.2.3` | Installed at app-spawn time. Must be ≥ v0.2.3 for apps to run inside a Nebi (pixi) environment; older versions only activate conda and fall back to the base env. | | `nebi-remote-url` | `""` *(derived)* | Browser-facing Nebi URL. | | `nebi-internal-url` | `""` *(derived)* | In-cluster Nebi URL. | diff --git a/templates/_helpers.tpl b/templates/_helpers.tpl index 877556d..c7e37db 100644 --- a/templates/_helpers.tpl +++ b/templates/_helpers.tpl @@ -162,6 +162,19 @@ In-cluster Nebi URL. Order of precedence: Nebi image reference (repository:tag). Empty when nebi.image.tag is not pinned — Python init-container code path stays a no-op. */}} +{{/* +GPU JupyterLab image (full ref). Derived from jupyterhub.singleuser.image — +the -gpu variant is built from the same commit as the CPU image, so it always +shares the same sha tag. Empty when name/tag unset (plain kind deploys). +Deployer override lives in jupyterhub.custom.gpu-image. +*/}} +{{- define "nebari-data-science-pack.gpuJupyterlabImage" -}} +{{- $img := ((.Values.jupyterhub).singleuser).image | default dict -}} +{{- if and $img.name $img.tag -}} +{{- printf "%s-gpu:%s" $img.name $img.tag -}} +{{- end -}} +{{- end -}} + {{- define "nebari-data-science-pack.nebiImage" -}} {{- $repo := .Values.nebi.image.repository | default "quay.io/nebari/nebi" -}} {{- $tag := .Values.nebi.image.tag | default "" -}} diff --git a/templates/hub-config.yaml b/templates/hub-config.yaml index 7a3ebc2..028b54c 100644 --- a/templates/hub-config.yaml +++ b/templates/hub-config.yaml @@ -41,6 +41,10 @@ data: _CHART_DERIVED = { "external-url": {{ $hubHost | quote }}, "nebi-image": {{ include "nebari-data-science-pack.nebiImage" . | quote }}, + # GPU jupyterlab image injected into ``gpu: true`` profiles by + # 01-spawner.py. Tracks singleuser.image so GPU profiles stay on + # the chart's tag across pack updates. + "gpu-image": {{ include "nebari-data-science-pack.gpuJupyterlabImage" . | quote }}, "nebi-remote-url": {{ include "nebari-data-science-pack.nebiRemoteURL" . | quote }}, "nebi-internal-url": {{ include "nebari-data-science-pack.nebiInternalURL" . | quote }}, "keycloak-token-url": {{ include "nebari-data-science-pack.keycloakTokenURL" . | quote }}, diff --git a/tests/unit/test_chart_derived.py b/tests/unit/test_chart_derived.py index fe94183..b47f291 100644 --- a/tests/unit/test_chart_derived.py +++ b/tests/unit/test_chart_derived.py @@ -148,3 +148,25 @@ def test_get_chart_config_explicit_override_wins(rendered_chart_derived): ) got = ns["get_chart_config"]("external-url") assert got == "explicit.example.com" + + +def test_gpu_image_derived_from_singleuser_image(rendered_chart_derived): + """The chart derives the GPU jupyterlab image from singleuser.image + (same sha tag, `-gpu` repo suffix) so `gpu: true` profiles track pack + updates without hardcoded SHAs (issue #230).""" + ns = _exec_chart_derived(rendered_chart_derived, z2jh_values={}) + got = ns["get_chart_config"]("gpu-image") + assert re.fullmatch( + r"quay\.io/nebari/nebari-data-science-pack-jupyterlab-gpu:sha-[0-9a-f]{7}", got + ), ( + f"get_chart_config('gpu-image') returned {got!r}; expected the " + "singleuser image name with a -gpu suffix and the same tag." + ) + + +def test_gpu_image_explicit_override_wins(rendered_chart_derived): + ns = _exec_chart_derived( + rendered_chart_derived, + z2jh_values={"custom.gpu-image": "example.com/lab-gpu:v1"}, + ) + assert ns["get_chart_config"]("gpu-image") == "example.com/lab-gpu:v1" diff --git a/tests/unit/test_spawner_profiles.py b/tests/unit/test_spawner_profiles.py index 4d1a48f..c4e9119 100644 --- a/tests/unit/test_spawner_profiles.py +++ b/tests/unit/test_spawner_profiles.py @@ -268,6 +268,97 @@ def test_render_profile_list_hides_restricted_profile_from_outsider(): assert [p["slug"] for p in visible] == ["small"] +GPU_IMAGE = "quay.io/nebari/nebari-data-science-pack-jupyterlab-gpu:sha-5dfee5e" + + +def test_gpu_profile_gets_derived_image_injected(): + """A ``gpu: true`` profile with no explicit image gets the chart-derived + GPU image, so deployers stop hardcoding SHAs (issue #230).""" + mod, _ = _load() + + profiles = [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}] + resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + + assert resolved[0]["kubespawner_override"]["image"] == GPU_IMAGE + assert resolved[0]["kubespawner_override"]["cpu_limit"] == 4 + + +def test_gpu_profile_explicit_image_wins(): + mod, _ = _load() + + profiles = [{"slug": "gpu", "gpu": True, "kubespawner_override": {"image": "custom:1"}}] + resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + + assert resolved[0]["kubespawner_override"]["image"] == "custom:1" + + +def test_gpu_key_is_stripped_before_kubespawner(): + mod, _ = _load() + + profiles = [ + {"slug": "gpu", "gpu": True}, + {"slug": "gpu2", "gpu": True, "kubespawner_override": {"image": "custom:1"}}, + ] + resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + + assert all("gpu" not in p for p in resolved) + # A profile without kubespawner_override still gets the image injected. + assert resolved[0]["kubespawner_override"]["image"] == GPU_IMAGE + + +def test_non_gpu_profile_is_untouched(): + mod, _ = _load() + + profiles = [{"slug": "small", "kubespawner_override": {"cpu_limit": 1}}] + resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + + assert resolved == profiles + + +def test_gpu_profile_without_derived_image_falls_back_to_default(): + """When the chart cannot derive a GPU image (singleuser.image unset), + the gpu key is still stripped and no image is injected.""" + mod, _ = _load() + + profiles = [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}] + resolved = mod._resolve_gpu_profiles(profiles, "") + + assert "gpu" not in resolved[0] + assert "image" not in resolved[0]["kubespawner_override"] + + +def test_gpu_resolution_does_not_mutate_input_profiles(): + mod, _ = _load() + + profiles = [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}] + mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + + assert profiles == [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}] + + +def test_gpu_image_injected_at_load_time(): + """Module load resolves gpu profiles from custom.profiles + custom.gpu-image, + so both the spawner and jhub-apps see the injected image.""" + z2jh = sys.modules["z2jh"] + prior = z2jh.get_config + + def fake_get_config(key, default=None): + if key == "custom.profiles": + return [{"slug": "gpu", "gpu": True}] + if key == "custom.gpu-image": + return GPU_IMAGE + return default + + z2jh.get_config = fake_get_config + try: + mod, _ = _load() + assert mod._profiles == [ + {"slug": "gpu", "kubespawner_override": {"image": GPU_IMAGE}} + ] + finally: + z2jh.get_config = prior + + def test_profile_list_is_the_filtering_callable_when_profiles_configured(): """When profiles exist, KubeSpawner.profile_list is wired to the per-user callable, not the raw static list.""" diff --git a/values.yaml b/values.yaml index de2f7ce..41ec257 100644 --- a/values.yaml +++ b/values.yaml @@ -390,6 +390,12 @@ jupyterhub: # If empty, derived as `:`. nebi-image: "" nebi-image-pull-policy: "IfNotPresent" + # GPU JupyterLab image (full ref) injected into profiles marked ``gpu: true`` + # (see the profiles docs below). If empty, derived as + # ``-gpu:`` — the GPU variant is + # built from the same commit as the CPU image, so the tags always match and + # GPU profiles track pack updates automatically. + gpu-image: "" # jhub-app-proxy version installed at app-spawn time. Must be >= v0.2.3 for # apps to run inside a Nebi (pixi) environment; older versions only support # conda activation and fall back to the base env (missing packages). @@ -486,9 +492,14 @@ jupyterhub: # access: yaml # groups: # - gpu-access + # gpu: true # inject the -gpu jupyterlab image automatically # kubespawner_override: # extra_resource_limits: # nvidia.com/gpu: 1 + # Profiles marked ``gpu: true`` get ``kubespawner_override.image`` set to + # the chart's GPU jupyterlab image (see ``gpu-image`` above) unless an + # explicit image is given, so deployers never hardcode a -gpu SHA. The + # ``gpu`` key is stripped before the profile reaches KubeSpawner. # NOTE: when bumping singleuser.image.tag below, also bump the # ``image: ...`` lines inside each profile_options.image.choices.default # entry so the profile selector shows the right tag. (z2jh values.yaml From da477a777684f5bd1af3a654dfea88ef5ef5270a Mon Sep 17 00:00:00 2001 From: Tyler Potts <49161327+tylerpotts@users.noreply.github.com> Date: Wed, 26 Aug 2026 13:11:33 -0500 Subject: [PATCH 3/4] fix(profiles): address review on GPU image injection - warn when a gpu: true profile declares profile_options.image, since the selected choice's image replaces the injected one at spawn time - strip the gpu key whatever its value (gpu: false was leaking through) - warn when no GPU image can be derived and custom.gpu-image is unset - log the injected ref per profile and the chart-wide gpu-image at load - document the gpu strip mechanism next to _PROFILE_GATING_KEYS - _helpers.tpl: move gpuJupyterlabImage below nebiImage so nebiImage keeps its doc comment; add the numbered precedence list - docs/values: drop 'always valid'; mirrored-registry + amd64-only notes; explicit pins are never bumped by bump_image_tags.py; profile_options caveat - spec: add Why this approach / Rejected alternatives - tests: docstrings + assert messages, cover the new warnings and logs --- config/jupyterhub/01-spawner.py | 63 ++++++-- docs/src/content/docs/server-profiles.md | 40 ++++- .../2026-08-24-gpu-profile-image-design.md | 39 ++++- templates/_helpers.tpl | 31 ++-- tests/unit/test_spawner_profiles.py | 144 +++++++++++++++--- values.yaml | 16 +- 6 files changed, 279 insertions(+), 54 deletions(-) diff --git a/config/jupyterhub/01-spawner.py b/config/jupyterhub/01-spawner.py index dd71a4d..913413d 100644 --- a/config/jupyterhub/01-spawner.py +++ b/config/jupyterhub/01-spawner.py @@ -330,7 +330,12 @@ def _setup_trust_bundle(spawner): # ``kubespawner_override`` accepts any valid KubeSpawner trait so deployers # can add node_selector, image, extra_resource_limits (GPU), etc. without # code changes. Empty list = no profile selector (single-instance mode). -# Keys used only for group gating; KubeSpawner must never see them. +# Keys used only for group gating; KubeSpawner must never see them. They are +# stripped per user in _filter_profiles because gating is a per-user decision. +# The ``gpu`` marker is the other chart-only key — it is stripped once at load +# in _resolve_gpu_profiles because its effect (image injection) is the same for +# every user and jhub-apps reads the resolved list too. A future custom key +# belongs in whichever of the two matches how it is evaluated. _PROFILE_GATING_KEYS = ("access", "groups", "users") @@ -452,30 +457,64 @@ def _resolve_gpu_profiles(profiles, gpu_image): without hardcoding a SHA in the deployer overlay (issue #230). An explicit ``kubespawner_override.image`` always wins. The ``gpu`` key - is stripped either way — KubeSpawner must never see it. Returns new - dicts; the input list is left untouched. + is stripped whatever its value — KubeSpawner must never see it. Returns + new dicts; the input list is left untouched. + + Two cases only warn, because raising here would break hub startup (and + therefore login) for every user: + * ``gpu_image`` is empty — the chart could not derive a ref + (``singleuser.image.name``/``tag`` empty) and ``custom.gpu-image`` + is unset. The profile falls back to the CPU default image. + * the profile declares ``profile_options.image`` — KubeSpawner applies + the selected choice's ``kubespawner_override`` AFTER the profile-level + one and replaces rather than merges, so the choice's image silently + wins over the injected one at spawn time (while jhub-apps still + displays the injected one). """ resolved = [] for profile in profiles: - if profile.get("gpu"): - profile = {k: v for k, v in profile.items() if k != "gpu"} - override = dict(profile.get("kubespawner_override") or {}) - if gpu_image and not override.get("image"): + if "gpu" not in profile: + resolved.append(profile) + continue + name = profile.get("slug") or profile.get("display_name") + is_gpu = bool(profile["gpu"]) + profile = {k: v for k, v in profile.items() if k != "gpu"} + if not is_gpu: + resolved.append(profile) + continue + override = dict(profile.get("kubespawner_override") or {}) + if not override.get("image"): + if gpu_image: override["image"] = gpu_image - profile["kubespawner_override"] = override + log.info("profiles: %r is gpu: true — injected image %s", name, gpu_image) + else: + log.warning( + "profiles: %r is marked gpu: true but no GPU image could be " + "derived (singleuser.image.name/tag empty?) and custom.gpu-image " + "is unset — the profile will spawn the CPU default image", + name, + ) + if "image" in (profile.get("profile_options") or {}): + log.warning( + "profiles: %r is marked gpu: true but declares profile_options.image; " + "the selected choice's image overrides the injected GPU image at spawn " + "time — drop the option or point its choices at the GPU image", + name, + ) + profile["kubespawner_override"] = override resolved.append(profile) return resolved -_profiles = _resolve_gpu_profiles( - get_config("custom.profiles", []), get_chart_config("gpu-image") -) +_gpu_image = get_chart_config("gpu-image") +_profiles = _resolve_gpu_profiles(get_config("custom.profiles", []), _gpu_image) if _profiles: c.KubeSpawner.profile_list = _render_profile_list log.info( - "profiles: loaded %d profile(s): %s", + "profiles: loaded %d profile(s): %s (gpu-image=%s)", len(_profiles), [p.get("slug") or p.get("display_name") for p in _profiles], + _gpu_image or "", ) else: log.info("profiles: none configured — single-instance mode") diff --git a/docs/src/content/docs/server-profiles.md b/docs/src/content/docs/server-profiles.md index 5fc960c..27fbea0 100644 --- a/docs/src/content/docs/server-profiles.md +++ b/docs/src/content/docs/server-profiles.md @@ -171,13 +171,41 @@ Mark a profile `gpu: true` and the chart injects the matching GPU JupyterLab ima ``` Both JupyterLab images are built from the same commit and share the same `sha-` tag, so -the derived ref (`-gpu:`) is always valid and -GPU profiles now track pack updates exactly like CPU profiles — no more stale SHAs in the -overlay ([issue #230](https://github.com/nebari-dev/data-science-pack/issues/230)). +the derived ref (`-gpu:`) exists on +`quay.io/nebari` for every release and GPU profiles track pack updates exactly like CPU +profiles — no more stale SHAs in the overlay +([issue #230](https://github.com/nebari-dev/data-science-pack/issues/230)). -An explicit `kubespawner_override.image` always wins over the injection, and -`jupyterhub.custom.gpu-image` overrides which image gets injected chart-wide. Like the -gating keys, `gpu` is stripped before the profile reaches KubeSpawner. +Like the gating keys, `gpu` is stripped (whatever its value) before the profile reaches +KubeSpawner. The hub logs the injected ref at startup: + +```bash +kubectl -n data-science logs deploy/hub | grep "profiles:.*gpu" +``` + +:::caution[Mirrored registries] +The derivation only rewrites the image *name*. If you point `jupyterhub.singleuser.image.name` +at a mirror (ECR, an airgapped registry), the chart derives `-gpu:`, which does +not exist unless you mirrored it too. Nothing validates the ref at `helm upgrade` time — the +first GPU spawn fails with `ImagePullBackOff`. Either mirror the `-gpu` image under that name +or set `jupyterhub.custom.gpu-image` to wherever it lives. + +The `-gpu` image is built for `linux/amd64` only; the CPU image is multi-arch. +::: + +Two things override the injection: + +- **An explicit `kubespawner_override.image`** always wins. Note that + `scripts/bump_image_tags.py` only rewrites the CPU image ref, so a hand-pinned `-gpu` + image stays frozen across releases — the exact problem `gpu: true` exists to solve. Prefer + the flag over pinning. +- **`profile_options.image`.** KubeSpawner applies the selected choice's + `kubespawner_override` *after* the profile-level one and replaces rather than merges, so + an image choice silently puts the CPU image on the GPU node (while jhub-apps' Create App + still displays the injected GPU image). The shipped CPU profiles carry such an option — + do not copy it onto a `gpu: true` profile. The hub logs a warning if you do. + +`jupyterhub.custom.gpu-image` changes which image gets injected chart-wide. ### Scheduling onto GPU nodes diff --git a/docs/superpowers/specs/2026-08-24-gpu-profile-image-design.md b/docs/superpowers/specs/2026-08-24-gpu-profile-image-design.md index bab0e6c..f46e718 100644 --- a/docs/superpowers/specs/2026-08-24-gpu-profile-image-design.md +++ b/docs/superpowers/specs/2026-08-24-gpu-profile-image-design.md @@ -54,6 +54,37 @@ A new per-profile boolean `gpu: true` in `jupyterhub.custom.profiles`: Resolution happens once at module load, before `_render_profile_list` filtering, so jhub-apps' server-types endpoint sees the same image. +### Why a boolean `gpu: true` rather than `image-variant: gpu` + +* It matches how deployers already think about the profile (the issue asks + to "specify a JupyterLab profile as a GPU node"), and reads naturally + next to `extra_resource_limits: {nvidia.com/gpu: 1}`. +* `-gpu` is the only published runtime variant. `-base` is a build stage, + not something a profile could select, so today there is exactly one axis. +* It is not a one-way door. If a `-rocm` or ARM-specific runtime image + ships later, `image-variant: ` can be added with the same + load-time mechanism and `gpu: true` becomes sugar for + `image-variant: gpu` (one line in `_resolve_gpu_profiles`, no + deprecation cycle, both keys keep working). + +### Rejected alternatives + +* **`image-variant: gpu` string key now.** Same code today, but generalises + a namespace (`-:` plus a per-variant override map) + for variants that do not exist. YAGNI; see above for the upgrade path. +* **Injecting into `profile_options.image.choices` too.** Choices always + carry an explicit image (that is their purpose), so injecting there + would overwrite explicit deployer values and contradict "explicit wins". + Instead the hub warns when a `gpu: true` profile declares + `profile_options.image`, and the docs say not to combine them. +* **Templating the profile list in Helm.** Profiles are deployer-authored + values consumed by the z2jh subchart via `custom.profiles`; this chart's + templates never see the merged list, and z2jh values cannot reference + each other. +* **Raising on an empty derived image.** Would break hub startup, and + therefore login, for every user. A warning plus fallback to the CPU + default image is the right level. + ### Automatic tag currency `scripts/bump_image_tags.py` already bumps `jupyterhub.singleuser.image.tag` @@ -63,8 +94,12 @@ every release; the derived GPU ref follows with zero script changes. * Auto-injecting `extra_resource_limits` / tolerations — cluster-specific, already documented. -* `profile_options.image.choices` for GPU profiles — deployer-defined - profiles rarely carry them; explicit choices keep winning if present. +* Rewriting `profile_options.image.choices` for GPU profiles — explicit + choices keep winning at spawn time; the hub warns when a `gpu: true` + profile declares that option (see Rejected alternatives). +* Teaching `scripts/bump_image_tags.py` to bump explicitly pinned `-gpu` + refs, and recording the build invariant (both jupyterlab images from one + `build-images.yaml` run) — follow-ups. ## Testing diff --git a/templates/_helpers.tpl b/templates/_helpers.tpl index c7e37db..c52bb30 100644 --- a/templates/_helpers.tpl +++ b/templates/_helpers.tpl @@ -162,11 +162,26 @@ In-cluster Nebi URL. Order of precedence: Nebi image reference (repository:tag). Empty when nebi.image.tag is not pinned — Python init-container code path stays a no-op. */}} +{{- define "nebari-data-science-pack.nebiImage" -}} +{{- $repo := .Values.nebi.image.repository | default "quay.io/nebari/nebi" -}} +{{- $tag := .Values.nebi.image.tag | default "" -}} +{{- if $tag -}} +{{- printf "%s:%s" $repo $tag -}} +{{- end -}} +{{- end -}} + {{/* -GPU JupyterLab image (full ref). Derived from jupyterhub.singleuser.image — -the -gpu variant is built from the same commit as the CPU image, so it always -shares the same sha tag. Empty when name/tag unset (plain kind deploys). -Deployer override lives in jupyterhub.custom.gpu-image. +GPU JupyterLab image (full ref), injected by 01-spawner.py into profiles +marked ``gpu: true``. Order of precedence at spawn time: + 1. profile's own kubespawner_override.image (explicit, never touched) + 2. .Values.jupyterhub.custom.gpu-image (deployer override, read at runtime + by get_chart_config — NOT consulted here) + 3. -gpu: + (this helper; the -gpu variant is built from the same commit as the CPU + image, so it always shares the same sha tag) + 4. "" — name or tag empty (schema-valid in z2jh, e.g. a plain kind deploy); + the spawner then warns and leaves the profile on the CPU default image. + The guard matters: without it an empty name renders "-gpu:". */}} {{- define "nebari-data-science-pack.gpuJupyterlabImage" -}} {{- $img := ((.Values.jupyterhub).singleuser).image | default dict -}} @@ -175,14 +190,6 @@ Deployer override lives in jupyterhub.custom.gpu-image. {{- end -}} {{- end -}} -{{- define "nebari-data-science-pack.nebiImage" -}} -{{- $repo := .Values.nebi.image.repository | default "quay.io/nebari/nebi" -}} -{{- $tag := .Values.nebi.image.tag | default "" -}} -{{- if $tag -}} -{{- printf "%s:%s" $repo $tag -}} -{{- end -}} -{{- end -}} - {{/* Keycloak token endpoint. Order of precedence: 1. .Values.jupyterhub.custom.keycloak-token-url (explicit) diff --git a/tests/unit/test_spawner_profiles.py b/tests/unit/test_spawner_profiles.py index c4e9119..dcde929 100644 --- a/tests/unit/test_spawner_profiles.py +++ b/tests/unit/test_spawner_profiles.py @@ -279,20 +279,28 @@ def test_gpu_profile_gets_derived_image_injected(): profiles = [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}] resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) - assert resolved[0]["kubespawner_override"]["image"] == GPU_IMAGE - assert resolved[0]["kubespawner_override"]["cpu_limit"] == 4 + override = resolved[0]["kubespawner_override"] + assert override["image"] == GPU_IMAGE, ( + f"expected the derived GPU image to be injected, got {override!r}" + ) + assert override["cpu_limit"] == 4, "other kubespawner_override keys must survive" def test_gpu_profile_explicit_image_wins(): + """An explicit ``kubespawner_override.image`` is never replaced — the + deployer opted out of derivation for that profile.""" mod, _ = _load() profiles = [{"slug": "gpu", "gpu": True, "kubespawner_override": {"image": "custom:1"}}] resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) - assert resolved[0]["kubespawner_override"]["image"] == "custom:1" + got = resolved[0]["kubespawner_override"]["image"] + assert got == "custom:1", f"explicit image was overwritten with {got!r}" def test_gpu_key_is_stripped_before_kubespawner(): + """The ``gpu`` marker is chart-only: KubeSpawner must never see it, and a + profile with no ``kubespawner_override`` at all still gets the image.""" mod, _ = _load() profiles = [ @@ -301,44 +309,119 @@ def test_gpu_key_is_stripped_before_kubespawner(): ] resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) - assert all("gpu" not in p for p in resolved) - # A profile without kubespawner_override still gets the image injected. - assert resolved[0]["kubespawner_override"]["image"] == GPU_IMAGE + assert all("gpu" not in p for p in resolved), f"gpu key leaked: {resolved!r}" + assert resolved[0]["kubespawner_override"]["image"] == GPU_IMAGE, ( + "a gpu profile without kubespawner_override should still get the image" + ) + + +def test_gpu_false_is_also_stripped(): + """``gpu: false`` is a valid way to write "not a GPU profile"; the key is + stripped regardless of value, as the docs promise, and nothing is injected.""" + mod, _ = _load() + + profiles = [{"slug": "cpu", "gpu": False, "kubespawner_override": {"cpu_limit": 1}}] + resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + + assert resolved == [{"slug": "cpu", "kubespawner_override": {"cpu_limit": 1}}], ( + f"gpu: false must be stripped without injecting an image, got {resolved!r}" + ) def test_non_gpu_profile_is_untouched(): + """Profiles without the ``gpu`` key pass through byte-for-byte.""" mod, _ = _load() profiles = [{"slug": "small", "kubespawner_override": {"cpu_limit": 1}}] resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) - assert resolved == profiles + assert resolved == profiles, f"non-gpu profile was modified: {resolved!r}" -def test_gpu_profile_without_derived_image_falls_back_to_default(): - """When the chart cannot derive a GPU image (singleuser.image unset), - the gpu key is still stripped and no image is injected.""" +def test_gpu_profile_without_derived_image_falls_back_to_default(caplog): + """When the chart cannot derive a GPU image (singleuser.image.name unset + or empty — schema-valid in z2jh), the gpu key is still stripped, no image + is injected, and the hub warns: this silently lands the CPU image on a GPU + node, but raising would break hub startup and therefore login.""" mod, _ = _load() profiles = [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}] - resolved = mod._resolve_gpu_profiles(profiles, "") + with caplog.at_level("WARNING"): + resolved = mod._resolve_gpu_profiles(profiles, "") - assert "gpu" not in resolved[0] - assert "image" not in resolved[0]["kubespawner_override"] + assert "gpu" not in resolved[0], "gpu key must be stripped even without an image" + assert "image" not in resolved[0]["kubespawner_override"], ( + "no image should be injected when the derived ref is empty" + ) + warnings = [r.getMessage() for r in caplog.records if r.levelname == "WARNING"] + assert any("gpu" in w and "gpu-image" in w for w in warnings), ( + f"expected a warning naming the profile and custom.gpu-image, got {warnings!r}" + ) + + +def test_gpu_profile_with_image_choices_warns(caplog): + """KubeSpawner applies ``profile_options.image.choices.*.kubespawner_override`` + AFTER the profile-level override and replaces rather than merges, so an + image choice silently defeats the injection. The hub must say so.""" + mod, _ = _load() + + profiles = [ + { + "slug": "gpu", + "gpu": True, + "profile_options": { + "image": { + "display_name": "Image", + "choices": { + "default": { + "display_name": "cpu-lab:sha-1", + "default": True, + "kubespawner_override": {"image": "cpu-lab:sha-1"}, + } + }, + } + }, + } + ] + with caplog.at_level("WARNING"): + mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + + warnings = [r.getMessage() for r in caplog.records if r.levelname == "WARNING"] + assert any("profile_options" in w and "gpu" in w for w in warnings), ( + f"expected a warning that profile_options.image overrides the GPU image, got {warnings!r}" + ) + + +def test_gpu_profile_without_image_choices_does_not_warn(caplog): + """The choices warning is specific: a plain gpu profile (or one with + non-image profile_options) stays quiet.""" + mod, _ = _load() + + profiles = [ + {"slug": "gpu", "gpu": True}, + {"slug": "gpu2", "gpu": True, "profile_options": {"size": {"choices": {}}}}, + ] + with caplog.at_level("WARNING"): + mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + + warnings = [r.getMessage() for r in caplog.records if r.levelname == "WARNING"] + assert warnings == [], f"unexpected warnings: {warnings!r}" def test_gpu_resolution_does_not_mutate_input_profiles(): + """Resolution returns new dicts; the z2jh-provided list is left intact.""" mod, _ = _load() profiles = [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}] mod._resolve_gpu_profiles(profiles, GPU_IMAGE) - assert profiles == [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}] + assert profiles == [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}], ( + f"input profiles were mutated: {profiles!r}" + ) -def test_gpu_image_injected_at_load_time(): - """Module load resolves gpu profiles from custom.profiles + custom.gpu-image, - so both the spawner and jhub-apps see the injected image.""" +def _load_with_gpu_profile(): + """Load 01-spawner.py with one ``gpu: true`` profile and a derived image.""" z2jh = sys.modules["z2jh"] prior = z2jh.get_config @@ -351,14 +434,33 @@ def fake_get_config(key, default=None): z2jh.get_config = fake_get_config try: - mod, _ = _load() - assert mod._profiles == [ - {"slug": "gpu", "kubespawner_override": {"image": GPU_IMAGE}} - ] + return _load() finally: z2jh.get_config = prior +def test_gpu_image_injected_at_load_time(): + """Module load resolves gpu profiles from custom.profiles + custom.gpu-image, + so both the spawner and jhub-apps see the injected image.""" + mod, _ = _load_with_gpu_profile() + + assert mod._profiles == [{"slug": "gpu", "kubespawner_override": {"image": GPU_IMAGE}}], ( + f"load-time resolution did not inject the GPU image: {mod._profiles!r}" + ) + + +def test_load_log_names_the_injected_gpu_image(caplog): + """``kubectl logs deploy/hub`` must be able to answer which image a GPU + profile got — the load-time info line carries the derived ref.""" + with caplog.at_level("INFO"): + _load_with_gpu_profile() + + infos = [r.getMessage() for r in caplog.records if r.levelname == "INFO"] + assert any(GPU_IMAGE in m and "gpu" in m for m in infos), ( + f"expected an info line naming the injected GPU image, got {infos!r}" + ) + + def test_profile_list_is_the_filtering_callable_when_profiles_configured(): """When profiles exist, KubeSpawner.profile_list is wired to the per-user callable, not the raw static list.""" diff --git a/values.yaml b/values.yaml index 41ec257..948a571 100644 --- a/values.yaml +++ b/values.yaml @@ -395,6 +395,11 @@ jupyterhub: # ``-gpu:`` — the GPU variant is # built from the same commit as the CPU image, so the tags always match and # GPU profiles track pack updates automatically. + # If you mirror ``singleuser.image.name`` into a private/airgapped registry, + # mirror the ``-gpu`` image too or set this explicitly — the derived ref is + # not validated at install time, so a missing mirror only shows up as + # ImagePullBackOff on the first GPU spawn. The ``-gpu`` image is linux/amd64 + # only (the CPU image is multi-arch). gpu-image: "" # jhub-app-proxy version installed at app-spawn time. Must be >= v0.2.3 for # apps to run inside a Nebi (pixi) environment; older versions only support @@ -499,7 +504,16 @@ jupyterhub: # Profiles marked ``gpu: true`` get ``kubespawner_override.image`` set to # the chart's GPU jupyterlab image (see ``gpu-image`` above) unless an # explicit image is given, so deployers never hardcode a -gpu SHA. The - # ``gpu`` key is stripped before the profile reaches KubeSpawner. + # ``gpu`` key is stripped (whatever its value) before the profile reaches + # KubeSpawner. Two caveats: + # * an explicit ``kubespawner_override.image`` is never touched — and + # scripts/bump_image_tags.py only rewrites the CPU image, so a pinned + # -gpu ref stays frozen across releases (the exact problem gpu: true + # exists to avoid). Prefer the flag over pinning. + # * do not combine ``gpu: true`` with ``profile_options.image`` — the + # selected choice's image replaces the injected one at spawn time + # (KubeSpawner applies choice overrides after profile overrides). The + # hub logs a warning if you do. # NOTE: when bumping singleuser.image.tag below, also bump the # ``image: ...`` lines inside each profile_options.image.choices.default # entry so the profile selector shows the right tag. (z2jh values.yaml From c6f2201e45ccbae67a7469dfad9507b100f5364a Mon Sep 17 00:00:00 2001 From: Tyler Potts <49161327+tylerpotts@users.noreply.github.com> Date: Wed, 26 Aug 2026 13:19:21 -0500 Subject: [PATCH 4/4] refactor(profiles): image-variant: instead of boolean gpu: true Per review: a string key generalises to future -rocm/arm64 variants without carrying a boolean alongside it. The derivation moves from Helm into 01-spawner.py (-:) because only the profile knows the variant name; custom.gpu-image becomes the custom.image-variants map of per-variant full-ref overrides. Drops the gpuJupyterlabImage helper and the _CHART_DERIVED entry. --- config/jupyterhub/01-spawner.py | 92 +++++---- docs/src/content/docs/server-profiles.md | 25 ++- docs/src/content/docs/values-reference.md | 2 +- .../2026-08-24-gpu-profile-image-design.md | 94 ++++----- templates/_helpers.tpl | 20 -- templates/hub-config.yaml | 4 - tests/unit/test_chart_derived.py | 21 -- tests/unit/test_spawner_profiles.py | 188 +++++++++++------- values.yaml | 41 ++-- 9 files changed, 250 insertions(+), 237 deletions(-) diff --git a/config/jupyterhub/01-spawner.py b/config/jupyterhub/01-spawner.py index 913413d..95560bd 100644 --- a/config/jupyterhub/01-spawner.py +++ b/config/jupyterhub/01-spawner.py @@ -332,9 +332,9 @@ def _setup_trust_bundle(spawner): # code changes. Empty list = no profile selector (single-instance mode). # Keys used only for group gating; KubeSpawner must never see them. They are # stripped per user in _filter_profiles because gating is a per-user decision. -# The ``gpu`` marker is the other chart-only key — it is stripped once at load -# in _resolve_gpu_profiles because its effect (image injection) is the same for -# every user and jhub-apps reads the resolved list too. A future custom key +# ``image-variant`` is the other chart-only key — it is stripped once at load +# in _resolve_image_variants because its effect (image injection) is the same +# for every user and jhub-apps reads the resolved list too. A future custom key # belongs in whichever of the two matches how it is evaluated. _PROFILE_GATING_KEYS = ("access", "groups", "users") @@ -447,74 +447,84 @@ async def _render_profile_list(spawner): return visible -def _resolve_gpu_profiles(profiles, gpu_image): - """Inject the chart-derived GPU image into ``gpu: true`` profiles. +def _resolve_image_variants(profiles, base_name, base_tag, overrides): + """Inject a variant of the singleuser image into ``image-variant`` profiles. - Both jupyterlab images are built from the same commit with the same - ``sha-`` tag, so the chart derives the GPU ref from - ``singleuser.image`` (override via ``custom.gpu-image``). Marking a - profile ``gpu: true`` keeps its image current across pack updates - without hardcoding a SHA in the deployer overlay (issue #230). + Every jupyterlab image variant (today only ``gpu``) is built from the same + commit as the CPU image and shares its ``sha-`` tag, so a profile + marked ``image-variant: gpu`` resolves to + ``-gpu:`` and stays current + across pack updates without a hardcoded SHA in the overlay (issue #230). - An explicit ``kubespawner_override.image`` always wins. The ``gpu`` key - is stripped whatever its value — KubeSpawner must never see it. Returns - new dicts; the input list is left untouched. + Order of precedence for the injected image: + 1. the profile's own ``kubespawner_override.image`` — never touched + 2. ``custom.image-variants.`` — deployer override, full ref + (mirrored registries, a variant published elsewhere) + 3. ``-:`` — derived + 4. nothing — ``base_name``/``base_tag`` empty (schema-valid in z2jh); + the profile falls back to the CPU default image and the hub warns. + + The ``image-variant`` key is stripped whatever its value — KubeSpawner + must never see it. Returns new dicts; the input list is left untouched. Two cases only warn, because raising here would break hub startup (and - therefore login) for every user: - * ``gpu_image`` is empty — the chart could not derive a ref - (``singleuser.image.name``/``tag`` empty) and ``custom.gpu-image`` - is unset. The profile falls back to the CPU default image. - * the profile declares ``profile_options.image`` — KubeSpawner applies - the selected choice's ``kubespawner_override`` AFTER the profile-level - one and replaces rather than merges, so the choice's image silently - wins over the injected one at spawn time (while jhub-apps still - displays the injected one). + therefore login) for every user: the empty-ref fallback above, and a + profile that also declares ``profile_options.image`` — KubeSpawner applies + the selected choice's ``kubespawner_override`` AFTER the profile-level one + and replaces rather than merges, so the choice's image silently wins over + the injected one at spawn time (while jhub-apps still displays the + injected one). """ resolved = [] for profile in profiles: - if "gpu" not in profile: + if "image-variant" not in profile: resolved.append(profile) continue name = profile.get("slug") or profile.get("display_name") - is_gpu = bool(profile["gpu"]) - profile = {k: v for k, v in profile.items() if k != "gpu"} - if not is_gpu: + variant = profile["image-variant"] + profile = {k: v for k, v in profile.items() if k != "image-variant"} + if not variant: resolved.append(profile) continue override = dict(profile.get("kubespawner_override") or {}) if not override.get("image"): - if gpu_image: - override["image"] = gpu_image - log.info("profiles: %r is gpu: true — injected image %s", name, gpu_image) + image = (overrides or {}).get(variant) + if not image and base_name and base_tag: + image = f"{base_name}-{variant}:{base_tag}" + if image: + override["image"] = image + log.info("profiles: %r image-variant %r — injected image %s", name, variant, image) else: log.warning( - "profiles: %r is marked gpu: true but no GPU image could be " - "derived (singleuser.image.name/tag empty?) and custom.gpu-image " - "is unset — the profile will spawn the CPU default image", - name, + "profiles: %r has image-variant %r but no image could be derived " + "(singleuser.image.name/tag empty?) and custom.image-variants.%s is " + "unset — the profile will spawn the CPU default image", + name, variant, variant, ) if "image" in (profile.get("profile_options") or {}): log.warning( - "profiles: %r is marked gpu: true but declares profile_options.image; " - "the selected choice's image overrides the injected GPU image at spawn " - "time — drop the option or point its choices at the GPU image", - name, + "profiles: %r has image-variant %r but declares profile_options.image; " + "the selected choice's image overrides the injected one at spawn time — " + "drop the option or point its choices at the variant image", + name, variant, ) profile["kubespawner_override"] = override resolved.append(profile) return resolved -_gpu_image = get_chart_config("gpu-image") -_profiles = _resolve_gpu_profiles(get_config("custom.profiles", []), _gpu_image) +_profiles = _resolve_image_variants( + get_config("custom.profiles", []), + get_config("singleuser.image.name", ""), + get_config("singleuser.image.tag", ""), + get_config("custom.image-variants", {}) or {}, +) if _profiles: c.KubeSpawner.profile_list = _render_profile_list log.info( - "profiles: loaded %d profile(s): %s (gpu-image=%s)", + "profiles: loaded %d profile(s): %s", len(_profiles), [p.get("slug") or p.get("display_name") for p in _profiles], - _gpu_image or "", ) else: log.info("profiles: none configured — single-instance mode") diff --git a/docs/src/content/docs/server-profiles.md b/docs/src/content/docs/server-profiles.md index 27fbea0..f3024bc 100644 --- a/docs/src/content/docs/server-profiles.md +++ b/docs/src/content/docs/server-profiles.md @@ -152,13 +152,13 @@ kubectl -n data-science logs deploy/hub | grep -i "profiles:\|groups" ### The GPU image, without hardcoding a SHA -Mark a profile `gpu: true` and the chart injects the matching GPU JupyterLab image +Mark a profile `image-variant: gpu` and the chart injects the matching GPU JupyterLab image (`nebari-data-science-pack-jupyterlab-gpu`) at the chart's current tag: ```yaml - slug: gpu-instance display_name: "G4 GPU Instance" - gpu: true + image-variant: gpu access: yaml groups: - gpu-access @@ -176,8 +176,10 @@ the derived ref (`-gpu:`) exists on profiles — no more stale SHAs in the overlay ([issue #230](https://github.com/nebari-dev/data-science-pack/issues/230)). -Like the gating keys, `gpu` is stripped (whatever its value) before the profile reaches -KubeSpawner. The hub logs the injected ref at startup: +The derivation is generic — `image-variant: ` resolves to +`-:` — but `gpu` is the only variant +published today. Like the gating keys, `image-variant` is stripped (whatever its value) before +the profile reaches KubeSpawner. The hub logs the injected ref at startup: ```bash kubectl -n data-science logs deploy/hub | grep "profiles:.*gpu" @@ -188,7 +190,12 @@ The derivation only rewrites the image *name*. If you point `jupyterhub.singleus at a mirror (ECR, an airgapped registry), the chart derives `-gpu:`, which does not exist unless you mirrored it too. Nothing validates the ref at `helm upgrade` time — the first GPU spawn fails with `ImagePullBackOff`. Either mirror the `-gpu` image under that name -or set `jupyterhub.custom.gpu-image` to wherever it lives. +or map it under `jupyterhub.custom.image-variants`: + +```yaml + image-variants: + gpu: 123456789012.dkr.ecr.us-east-1.amazonaws.com/lab-gpu:sha-abc1234 +``` The `-gpu` image is built for `linux/amd64` only; the CPU image is multi-arch. ::: @@ -197,15 +204,15 @@ Two things override the injection: - **An explicit `kubespawner_override.image`** always wins. Note that `scripts/bump_image_tags.py` only rewrites the CPU image ref, so a hand-pinned `-gpu` - image stays frozen across releases — the exact problem `gpu: true` exists to solve. Prefer - the flag over pinning. + image stays frozen across releases — the exact problem `image-variant` exists to solve. Prefer + the key over pinning. - **`profile_options.image`.** KubeSpawner applies the selected choice's `kubespawner_override` *after* the profile-level one and replaces rather than merges, so an image choice silently puts the CPU image on the GPU node (while jhub-apps' Create App still displays the injected GPU image). The shipped CPU profiles carry such an option — - do not copy it onto a `gpu: true` profile. The hub logs a warning if you do. + do not copy it onto an `image-variant` profile. The hub logs a warning if you do. -`jupyterhub.custom.gpu-image` changes which image gets injected chart-wide. +`jupyterhub.custom.image-variants.` changes which image gets injected for a variant chart-wide. ### Scheduling onto GPU nodes diff --git a/docs/src/content/docs/values-reference.md b/docs/src/content/docs/values-reference.md index d695d35..ee143d5 100644 --- a/docs/src/content/docs/values-reference.md +++ b/docs/src/content/docs/values-reference.md @@ -190,7 +190,7 @@ always win. | `external-url` | `""` *(derived)* | Hub bind hostname. | | `nebi-image` | `""` *(derived)* | `repository:tag` copied into user pods. | | `nebi-image-pull-policy` | `IfNotPresent` | — | -| `gpu-image` | `""` *(derived)* | GPU JupyterLab image injected into `gpu: true` profiles. Derived as `-gpu:` — see [Server profiles](/server-profiles/#gpu-profiles). | +| `image-variants` | `{}` | Per-variant image overrides for `image-variant: ` profiles. Default derivation is `-:` — see [Server profiles](/server-profiles/#gpu-profiles). | | `jhub-app-proxy-version` | `v0.2.3` | Installed at app-spawn time. Must be ≥ v0.2.3 for apps to run inside a Nebi (pixi) environment; older versions only activate conda and fall back to the base env. | | `nebi-remote-url` | `""` *(derived)* | Browser-facing Nebi URL. | | `nebi-internal-url` | `""` *(derived)* | In-cluster Nebi URL. | diff --git a/docs/superpowers/specs/2026-08-24-gpu-profile-image-design.md b/docs/superpowers/specs/2026-08-24-gpu-profile-image-design.md index f46e718..85d6c7d 100644 --- a/docs/superpowers/specs/2026-08-24-gpu-profile-image-design.md +++ b/docs/superpowers/specs/2026-08-24-gpu-profile-image-design.md @@ -17,12 +17,12 @@ image ref: `-gpu:`. ## Design -A new per-profile boolean `gpu: true` in `jupyterhub.custom.profiles`: +A new per-profile key `image-variant: ` in `jupyterhub.custom.profiles`: ```yaml - slug: gpu display_name: "GPU Access" - gpu: true + image-variant: gpu access: yaml groups: [gpu-access] kubespawner_override: @@ -33,54 +33,54 @@ A new per-profile boolean `gpu: true` in `jupyterhub.custom.profiles`: ### Components -1. **Helm helper** `nebari-data-science-pack.gpuJupyterlabImage` - (`templates/_helpers.tpl`): renders - `<.Values.jupyterhub.singleuser.image.name>-gpu:`; empty when - name/tag unset. - -2. **Chart-derived config** (`templates/hub-config.yaml`): add - `"gpu-image"` to `_CHART_DERIVED`, following the existing `nebi-image` - pattern. Deployers can override via `jupyterhub.custom.gpu-image` - (documented as `""` placeholder in `values.yaml`). - -3. **Spawner config** (`config/jupyterhub/01-spawner.py`): at load, - `_resolve_gpu_profiles(profiles, gpu_image)` walks `custom.profiles`: - * `gpu: true` and no `kubespawner_override.image` → inject `gpu_image` - * `gpu: true` with explicit image → leave the image alone (explicit wins) - * the `gpu` key is always stripped so KubeSpawner never sees it - * `gpu_image` empty → strip key, inject nothing (falls back to the - z2jh singleuser default image, today's behavior) +1. **Spawner config** (`config/jupyterhub/01-spawner.py`): at load, + `_resolve_image_variants(profiles, base_name, base_tag, overrides)` + walks `custom.profiles`, reading `singleuser.image.name`/`.tag` and + `custom.image-variants` from z2jh. For each `image-variant: `: + * no `kubespawner_override.image` → inject, in order of precedence, + `custom.image-variants.` if set, else + `-:`; if neither can be produced + (`singleuser.image.name`/`tag` empty, schema-valid in z2jh) warn and + fall back to the CPU default image + * explicit image → leave it alone (explicit wins) + * the key is always stripped so KubeSpawner never sees it + * `profile_options.image` present → warn (see Rejected alternatives) Resolution happens once at module load, before `_render_profile_list` filtering, so jhub-apps' server-types endpoint sees the same image. -### Why a boolean `gpu: true` rather than `image-variant: gpu` +2. **Override map** `jupyterhub.custom.image-variants: {}` in `values.yaml` + — full refs per variant, for mirrored registries or a variant published + elsewhere. Read directly via z2jh `get_config` (it is a mapping, so the + `get_chart_config` empty-string convention does not apply). + +No Helm-side change: the derivation needs the variant name, which only the +profile knows, so it lives in Python where both inputs are available. + +### Why `image-variant: ` rather than a boolean `gpu: true` -* It matches how deployers already think about the profile (the issue asks - to "specify a JupyterLab profile as a GPU node"), and reads naturally - next to `extra_resource_limits: {nvidia.com/gpu: 1}`. -* `-gpu` is the only published runtime variant. `-base` is a build stage, - not something a profile could select, so today there is exactly one axis. -* It is not a one-way door. If a `-rocm` or ARM-specific runtime image - ships later, `image-variant: ` can be added with the same - load-time mechanism and `gpu: true` becomes sugar for - `image-variant: gpu` (one line in `_resolve_gpu_profiles`, no - deprecation cycle, both keys keep working). +* The image naming already has a variant axis (`-gpu`), and a `-rocm` or + arm64 runtime image is plausible. A boolean would have to coexist with a + string key forever once it shipped in `values.yaml`. +* Same amount of code today; the derivation string is the only place the + variant name appears. +* Cost: `image-variant: gpu` reads slightly less naturally than `gpu: true` + in an overlay. Mitigated by the example in `values.yaml` and the docs. ### Rejected alternatives -* **`image-variant: gpu` string key now.** Same code today, but generalises - a namespace (`-:` plus a per-variant override map) - for variants that do not exist. YAGNI; see above for the upgrade path. +* **Boolean `gpu: true`.** The first draft of this PR. Rejected in review + for the reason above: it is an API surface we cannot drop without a + deprecation cycle, and the string key costs nothing extra. +* **Deriving in Helm (`_CHART_DERIVED["gpu-image"]`).** Also the first + draft. Works for one hardcoded variant but cannot be generic — Helm does + not see the profile list the z2jh subchart consumes, so it cannot know + which variant names are in use. * **Injecting into `profile_options.image.choices` too.** Choices always carry an explicit image (that is their purpose), so injecting there would overwrite explicit deployer values and contradict "explicit wins". - Instead the hub warns when a `gpu: true` profile declares + Instead the hub warns when a variant profile declares `profile_options.image`, and the docs say not to combine them. -* **Templating the profile list in Helm.** Profiles are deployer-authored - values consumed by the z2jh subchart via `custom.profiles`; this chart's - templates never see the merged list, and z2jh values cannot reference - each other. * **Raising on an empty derived image.** Would break hub startup, and therefore login, for every user. A warning plus fallback to the CPU default image is the right level. @@ -95,20 +95,22 @@ every release; the derived GPU ref follows with zero script changes. * Auto-injecting `extra_resource_limits` / tolerations — cluster-specific, already documented. * Rewriting `profile_options.image.choices` for GPU profiles — explicit - choices keep winning at spawn time; the hub warns when a `gpu: true` - profile declares that option (see Rejected alternatives). + choices keep winning at spawn time; the hub warns when a variant profile + declares that option (see Rejected alternatives). * Teaching `scripts/bump_image_tags.py` to bump explicitly pinned `-gpu` refs, and recording the build invariant (both jupyterlab images from one `build-images.yaml` run) — follow-ups. ## Testing -* Unit (`tests/unit/test_spawner_profiles.py`): injection, explicit-image - precedence, key stripping, empty-gpu-image fallback, load-time wiring. -* Chart (`tests/unit/test_chart_derived.py`): rendered `_CHART_DERIVED` - contains the derived `gpu-image` ref from the default values. +* Unit (`tests/unit/test_spawner_profiles.py`): derivation, generic + variant names, override map, explicit-image precedence, key stripping + (including empty variant), empty-base fallback + warning, + `profile_options.image` warning (+ negative), input non-mutation, + load-time wiring from z2jh keys, load-time log naming the injected ref. ## Docs -* `docs/src/content/docs/server-profiles.md` GPU section: document `gpu: true`. -* `values.yaml` comments for `gpu-image` + profile example. +* `docs/src/content/docs/server-profiles.md` GPU section: document `image-variant`. +* `docs/src/content/docs/values-reference.md`: `image-variants` row. +* `values.yaml` comments for `image-variants` + profile example. diff --git a/templates/_helpers.tpl b/templates/_helpers.tpl index c52bb30..877556d 100644 --- a/templates/_helpers.tpl +++ b/templates/_helpers.tpl @@ -170,26 +170,6 @@ pinned — Python init-container code path stays a no-op. {{- end -}} {{- end -}} -{{/* -GPU JupyterLab image (full ref), injected by 01-spawner.py into profiles -marked ``gpu: true``. Order of precedence at spawn time: - 1. profile's own kubespawner_override.image (explicit, never touched) - 2. .Values.jupyterhub.custom.gpu-image (deployer override, read at runtime - by get_chart_config — NOT consulted here) - 3. -gpu: - (this helper; the -gpu variant is built from the same commit as the CPU - image, so it always shares the same sha tag) - 4. "" — name or tag empty (schema-valid in z2jh, e.g. a plain kind deploy); - the spawner then warns and leaves the profile on the CPU default image. - The guard matters: without it an empty name renders "-gpu:". -*/}} -{{- define "nebari-data-science-pack.gpuJupyterlabImage" -}} -{{- $img := ((.Values.jupyterhub).singleuser).image | default dict -}} -{{- if and $img.name $img.tag -}} -{{- printf "%s-gpu:%s" $img.name $img.tag -}} -{{- end -}} -{{- end -}} - {{/* Keycloak token endpoint. Order of precedence: 1. .Values.jupyterhub.custom.keycloak-token-url (explicit) diff --git a/templates/hub-config.yaml b/templates/hub-config.yaml index 028b54c..7a3ebc2 100644 --- a/templates/hub-config.yaml +++ b/templates/hub-config.yaml @@ -41,10 +41,6 @@ data: _CHART_DERIVED = { "external-url": {{ $hubHost | quote }}, "nebi-image": {{ include "nebari-data-science-pack.nebiImage" . | quote }}, - # GPU jupyterlab image injected into ``gpu: true`` profiles by - # 01-spawner.py. Tracks singleuser.image so GPU profiles stay on - # the chart's tag across pack updates. - "gpu-image": {{ include "nebari-data-science-pack.gpuJupyterlabImage" . | quote }}, "nebi-remote-url": {{ include "nebari-data-science-pack.nebiRemoteURL" . | quote }}, "nebi-internal-url": {{ include "nebari-data-science-pack.nebiInternalURL" . | quote }}, "keycloak-token-url": {{ include "nebari-data-science-pack.keycloakTokenURL" . | quote }}, diff --git a/tests/unit/test_chart_derived.py b/tests/unit/test_chart_derived.py index b47f291..ffc2ca9 100644 --- a/tests/unit/test_chart_derived.py +++ b/tests/unit/test_chart_derived.py @@ -149,24 +149,3 @@ def test_get_chart_config_explicit_override_wins(rendered_chart_derived): got = ns["get_chart_config"]("external-url") assert got == "explicit.example.com" - -def test_gpu_image_derived_from_singleuser_image(rendered_chart_derived): - """The chart derives the GPU jupyterlab image from singleuser.image - (same sha tag, `-gpu` repo suffix) so `gpu: true` profiles track pack - updates without hardcoded SHAs (issue #230).""" - ns = _exec_chart_derived(rendered_chart_derived, z2jh_values={}) - got = ns["get_chart_config"]("gpu-image") - assert re.fullmatch( - r"quay\.io/nebari/nebari-data-science-pack-jupyterlab-gpu:sha-[0-9a-f]{7}", got - ), ( - f"get_chart_config('gpu-image') returned {got!r}; expected the " - "singleuser image name with a -gpu suffix and the same tag." - ) - - -def test_gpu_image_explicit_override_wins(rendered_chart_derived): - ns = _exec_chart_derived( - rendered_chart_derived, - z2jh_values={"custom.gpu-image": "example.com/lab-gpu:v1"}, - ) - assert ns["get_chart_config"]("gpu-image") == "example.com/lab-gpu:v1" diff --git a/tests/unit/test_spawner_profiles.py b/tests/unit/test_spawner_profiles.py index dcde929..ee4ad53 100644 --- a/tests/unit/test_spawner_profiles.py +++ b/tests/unit/test_spawner_profiles.py @@ -268,98 +268,122 @@ def test_render_profile_list_hides_restricted_profile_from_outsider(): assert [p["slug"] for p in visible] == ["small"] -GPU_IMAGE = "quay.io/nebari/nebari-data-science-pack-jupyterlab-gpu:sha-5dfee5e" +BASE_NAME = "quay.io/nebari/nebari-data-science-pack-jupyterlab" +BASE_TAG = "sha-5dfee5e" +GPU_IMAGE = f"{BASE_NAME}-gpu:{BASE_TAG}" -def test_gpu_profile_gets_derived_image_injected(): - """A ``gpu: true`` profile with no explicit image gets the chart-derived - GPU image, so deployers stop hardcoding SHAs (issue #230).""" +def _resolve(mod, profiles, base_name=BASE_NAME, base_tag=BASE_TAG, overrides=None): + return mod._resolve_image_variants(profiles, base_name, base_tag, overrides or {}) + + +def test_variant_profile_gets_derived_image_injected(): + """An ``image-variant: gpu`` profile with no explicit image gets + ``-gpu:``, so deployers stop + hardcoding SHAs (issue #230).""" mod, _ = _load() - profiles = [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}] - resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + profiles = [{"slug": "gpu", "image-variant": "gpu", "kubespawner_override": {"cpu_limit": 4}}] + resolved = _resolve(mod, profiles) override = resolved[0]["kubespawner_override"] assert override["image"] == GPU_IMAGE, ( - f"expected the derived GPU image to be injected, got {override!r}" + f"expected the derived variant image to be injected, got {override!r}" ) assert override["cpu_limit"] == 4, "other kubespawner_override keys must survive" -def test_gpu_profile_explicit_image_wins(): +def test_variant_name_is_generic(): + """The derivation is ``-:`` for any variant string — + a future -rocm or arm64 build needs no code change.""" + mod, _ = _load() + + resolved = _resolve(mod, [{"slug": "amd", "image-variant": "rocm"}]) + + got = resolved[0]["kubespawner_override"]["image"] + assert got == f"{BASE_NAME}-rocm:{BASE_TAG}", f"unexpected derived ref {got!r}" + + +def test_variant_override_map_wins_over_derivation(): + """``custom.image-variants.`` replaces the derived ref chart-wide — + the escape hatch for mirrored registries.""" + mod, _ = _load() + + resolved = _resolve( + mod, + [{"slug": "gpu", "image-variant": "gpu"}], + overrides={"gpu": "mirror.example.com/lab-gpu:v1"}, + ) + + got = resolved[0]["kubespawner_override"]["image"] + assert got == "mirror.example.com/lab-gpu:v1", f"override map ignored, got {got!r}" + + +def test_variant_profile_explicit_image_wins(): """An explicit ``kubespawner_override.image`` is never replaced — the deployer opted out of derivation for that profile.""" mod, _ = _load() - profiles = [{"slug": "gpu", "gpu": True, "kubespawner_override": {"image": "custom:1"}}] - resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + profiles = [{"slug": "gpu", "image-variant": "gpu", "kubespawner_override": {"image": "custom:1"}}] + resolved = _resolve(mod, profiles, overrides={"gpu": "override:1"}) got = resolved[0]["kubespawner_override"]["image"] assert got == "custom:1", f"explicit image was overwritten with {got!r}" -def test_gpu_key_is_stripped_before_kubespawner(): - """The ``gpu`` marker is chart-only: KubeSpawner must never see it, and a - profile with no ``kubespawner_override`` at all still gets the image.""" +def test_variant_key_is_stripped_before_kubespawner(): + """The ``image-variant`` marker is chart-only: KubeSpawner must never see + it, and a profile with no ``kubespawner_override`` still gets the image.""" mod, _ = _load() profiles = [ - {"slug": "gpu", "gpu": True}, - {"slug": "gpu2", "gpu": True, "kubespawner_override": {"image": "custom:1"}}, + {"slug": "gpu", "image-variant": "gpu"}, + {"slug": "gpu2", "image-variant": "gpu", "kubespawner_override": {"image": "custom:1"}}, + {"slug": "cpu", "image-variant": ""}, ] - resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + resolved = _resolve(mod, profiles) - assert all("gpu" not in p for p in resolved), f"gpu key leaked: {resolved!r}" + assert all("image-variant" not in p for p in resolved), f"key leaked: {resolved!r}" assert resolved[0]["kubespawner_override"]["image"] == GPU_IMAGE, ( - "a gpu profile without kubespawner_override should still get the image" + "a variant profile without kubespawner_override should still get the image" ) - - -def test_gpu_false_is_also_stripped(): - """``gpu: false`` is a valid way to write "not a GPU profile"; the key is - stripped regardless of value, as the docs promise, and nothing is injected.""" - mod, _ = _load() - - profiles = [{"slug": "cpu", "gpu": False, "kubespawner_override": {"cpu_limit": 1}}] - resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) - - assert resolved == [{"slug": "cpu", "kubespawner_override": {"cpu_limit": 1}}], ( - f"gpu: false must be stripped without injecting an image, got {resolved!r}" + assert resolved[2] == {"slug": "cpu"}, ( + f"an empty variant must be stripped without injecting, got {resolved[2]!r}" ) -def test_non_gpu_profile_is_untouched(): - """Profiles without the ``gpu`` key pass through byte-for-byte.""" +def test_non_variant_profile_is_untouched(): + """Profiles without the ``image-variant`` key pass through byte-for-byte.""" mod, _ = _load() profiles = [{"slug": "small", "kubespawner_override": {"cpu_limit": 1}}] - resolved = mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + resolved = _resolve(mod, profiles) - assert resolved == profiles, f"non-gpu profile was modified: {resolved!r}" + assert resolved == profiles, f"plain profile was modified: {resolved!r}" -def test_gpu_profile_without_derived_image_falls_back_to_default(caplog): - """When the chart cannot derive a GPU image (singleuser.image.name unset - or empty — schema-valid in z2jh), the gpu key is still stripped, no image - is injected, and the hub warns: this silently lands the CPU image on a GPU - node, but raising would break hub startup and therefore login.""" +def test_variant_without_base_image_falls_back_to_default(caplog): + """When ``singleuser.image.name``/``tag`` is empty (schema-valid in z2jh) + and no override is set, the key is still stripped, no image is injected, + and the hub warns: this silently lands the CPU image on a GPU node, but + raising would break hub startup and therefore login.""" mod, _ = _load() - profiles = [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}] + profiles = [{"slug": "gpu", "image-variant": "gpu", "kubespawner_override": {"cpu_limit": 4}}] with caplog.at_level("WARNING"): - resolved = mod._resolve_gpu_profiles(profiles, "") + resolved = _resolve(mod, profiles, base_name="", base_tag=BASE_TAG) - assert "gpu" not in resolved[0], "gpu key must be stripped even without an image" + assert "image-variant" not in resolved[0], "key must be stripped even without an image" assert "image" not in resolved[0]["kubespawner_override"], ( - "no image should be injected when the derived ref is empty" + "no image should be injected when the ref cannot be derived" ) warnings = [r.getMessage() for r in caplog.records if r.levelname == "WARNING"] - assert any("gpu" in w and "gpu-image" in w for w in warnings), ( - f"expected a warning naming the profile and custom.gpu-image, got {warnings!r}" + assert any("gpu" in w and "image-variants" in w for w in warnings), ( + f"expected a warning naming the profile and custom.image-variants, got {warnings!r}" ) -def test_gpu_profile_with_image_choices_warns(caplog): +def test_variant_profile_with_image_choices_warns(caplog): """KubeSpawner applies ``profile_options.image.choices.*.kubespawner_override`` AFTER the profile-level override and replaces rather than merges, so an image choice silently defeats the injection. The hub must say so.""" @@ -368,7 +392,7 @@ def test_gpu_profile_with_image_choices_warns(caplog): profiles = [ { "slug": "gpu", - "gpu": True, + "image-variant": "gpu", "profile_options": { "image": { "display_name": "Image", @@ -384,53 +408,55 @@ def test_gpu_profile_with_image_choices_warns(caplog): } ] with caplog.at_level("WARNING"): - mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + _resolve(mod, profiles) warnings = [r.getMessage() for r in caplog.records if r.levelname == "WARNING"] assert any("profile_options" in w and "gpu" in w for w in warnings), ( - f"expected a warning that profile_options.image overrides the GPU image, got {warnings!r}" + f"expected a warning that profile_options.image overrides the variant image, got {warnings!r}" ) -def test_gpu_profile_without_image_choices_does_not_warn(caplog): - """The choices warning is specific: a plain gpu profile (or one with +def test_variant_profile_without_image_choices_does_not_warn(caplog): + """The choices warning is specific: a plain variant profile (or one with non-image profile_options) stays quiet.""" mod, _ = _load() profiles = [ - {"slug": "gpu", "gpu": True}, - {"slug": "gpu2", "gpu": True, "profile_options": {"size": {"choices": {}}}}, + {"slug": "gpu", "image-variant": "gpu"}, + {"slug": "gpu2", "image-variant": "gpu", "profile_options": {"size": {"choices": {}}}}, ] with caplog.at_level("WARNING"): - mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + _resolve(mod, profiles) warnings = [r.getMessage() for r in caplog.records if r.levelname == "WARNING"] assert warnings == [], f"unexpected warnings: {warnings!r}" -def test_gpu_resolution_does_not_mutate_input_profiles(): +def test_variant_resolution_does_not_mutate_input_profiles(): """Resolution returns new dicts; the z2jh-provided list is left intact.""" mod, _ = _load() - profiles = [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}] - mod._resolve_gpu_profiles(profiles, GPU_IMAGE) + profiles = [{"slug": "gpu", "image-variant": "gpu", "kubespawner_override": {"cpu_limit": 4}}] + _resolve(mod, profiles) - assert profiles == [{"slug": "gpu", "gpu": True, "kubespawner_override": {"cpu_limit": 4}}], ( - f"input profiles were mutated: {profiles!r}" - ) + assert profiles == [ + {"slug": "gpu", "image-variant": "gpu", "kubespawner_override": {"cpu_limit": 4}} + ], f"input profiles were mutated: {profiles!r}" -def _load_with_gpu_profile(): - """Load 01-spawner.py with one ``gpu: true`` profile and a derived image.""" +def _load_with_variant_profile(overrides=None): + """Load 01-spawner.py with one ``image-variant: gpu`` profile and the + z2jh ``singleuser.image`` values the hub reads in production.""" z2jh = sys.modules["z2jh"] prior = z2jh.get_config def fake_get_config(key, default=None): - if key == "custom.profiles": - return [{"slug": "gpu", "gpu": True}] - if key == "custom.gpu-image": - return GPU_IMAGE - return default + return { + "custom.profiles": [{"slug": "gpu", "image-variant": "gpu"}], + "custom.image-variants": overrides or {}, + "singleuser.image.name": BASE_NAME, + "singleuser.image.tag": BASE_TAG, + }.get(key, default) z2jh.get_config = fake_get_config try: @@ -439,25 +465,35 @@ def fake_get_config(key, default=None): z2jh.get_config = prior -def test_gpu_image_injected_at_load_time(): - """Module load resolves gpu profiles from custom.profiles + custom.gpu-image, - so both the spawner and jhub-apps see the injected image.""" - mod, _ = _load_with_gpu_profile() +def test_variant_image_injected_at_load_time(): + """Module load resolves variant profiles from custom.profiles + + singleuser.image + custom.image-variants, so both the spawner and + jhub-apps see the injected image.""" + mod, _ = _load_with_variant_profile() assert mod._profiles == [{"slug": "gpu", "kubespawner_override": {"image": GPU_IMAGE}}], ( - f"load-time resolution did not inject the GPU image: {mod._profiles!r}" + f"load-time resolution did not inject the variant image: {mod._profiles!r}" ) -def test_load_log_names_the_injected_gpu_image(caplog): - """``kubectl logs deploy/hub`` must be able to answer which image a GPU +def test_variant_override_map_applied_at_load_time(): + """``custom.image-variants`` is read from z2jh at load and wins over + the derivation.""" + mod, _ = _load_with_variant_profile(overrides={"gpu": "mirror.example.com/lab-gpu:v1"}) + + got = mod._profiles[0]["kubespawner_override"]["image"] + assert got == "mirror.example.com/lab-gpu:v1", f"override map ignored at load, got {got!r}" + + +def test_load_log_names_the_injected_variant_image(caplog): + """``kubectl logs deploy/hub`` must be able to answer which image a variant profile got — the load-time info line carries the derived ref.""" with caplog.at_level("INFO"): - _load_with_gpu_profile() + _load_with_variant_profile() infos = [r.getMessage() for r in caplog.records if r.levelname == "INFO"] assert any(GPU_IMAGE in m and "gpu" in m for m in infos), ( - f"expected an info line naming the injected GPU image, got {infos!r}" + f"expected an info line naming the injected image, got {infos!r}" ) diff --git a/values.yaml b/values.yaml index 948a571..4ea31c0 100644 --- a/values.yaml +++ b/values.yaml @@ -390,17 +390,20 @@ jupyterhub: # If empty, derived as `:`. nebi-image: "" nebi-image-pull-policy: "IfNotPresent" - # GPU JupyterLab image (full ref) injected into profiles marked ``gpu: true`` - # (see the profiles docs below). If empty, derived as - # ``-gpu:`` — the GPU variant is - # built from the same commit as the CPU image, so the tags always match and - # GPU profiles track pack updates automatically. + # Per-variant image overrides (full refs) for profiles marked + # ``image-variant: `` (see the profiles docs below). By default a + # variant resolves to ``-:`` + # — every variant is built from the same commit as the CPU image, so the tags + # always match and variant profiles track pack updates automatically. ``gpu`` + # is the only variant published today (linux/amd64 only; the CPU image is + # multi-arch). # If you mirror ``singleuser.image.name`` into a private/airgapped registry, - # mirror the ``-gpu`` image too or set this explicitly — the derived ref is - # not validated at install time, so a missing mirror only shows up as - # ImagePullBackOff on the first GPU spawn. The ``-gpu`` image is linux/amd64 - # only (the CPU image is multi-arch). - gpu-image: "" + # mirror the ``-gpu`` image too or map it here — the derived ref is not + # validated at install time, so a missing mirror only shows up as + # ImagePullBackOff on the first GPU spawn. + # image-variants: + # gpu: 123456789012.dkr.ecr.us-east-1.amazonaws.com/lab-gpu:sha-abc1234 + image-variants: {} # jhub-app-proxy version installed at app-spawn time. Must be >= v0.2.3 for # apps to run inside a Nebi (pixi) environment; older versions only support # conda activation and fall back to the base env (missing packages). @@ -497,20 +500,20 @@ jupyterhub: # access: yaml # groups: # - gpu-access - # gpu: true # inject the -gpu jupyterlab image automatically + # image-variant: gpu # inject the -gpu jupyterlab image automatically # kubespawner_override: # extra_resource_limits: # nvidia.com/gpu: 1 - # Profiles marked ``gpu: true`` get ``kubespawner_override.image`` set to - # the chart's GPU jupyterlab image (see ``gpu-image`` above) unless an - # explicit image is given, so deployers never hardcode a -gpu SHA. The - # ``gpu`` key is stripped (whatever its value) before the profile reaches - # KubeSpawner. Two caveats: + # Profiles marked ``image-variant: `` get ``kubespawner_override.image`` + # set to ``-:`` (or the + # ``image-variants`` override above) unless an explicit image is given, so + # deployers never hardcode a -gpu SHA. The ``image-variant`` key is stripped + # (whatever its value) before the profile reaches KubeSpawner. Two caveats: # * an explicit ``kubespawner_override.image`` is never touched — and # scripts/bump_image_tags.py only rewrites the CPU image, so a pinned - # -gpu ref stays frozen across releases (the exact problem gpu: true - # exists to avoid). Prefer the flag over pinning. - # * do not combine ``gpu: true`` with ``profile_options.image`` — the + # -gpu ref stays frozen across releases (the exact problem image-variant + # exists to avoid). Prefer the key over pinning. + # * do not combine ``image-variant`` with ``profile_options.image`` — the # selected choice's image replaces the injected one at spawn time # (KubeSpawner applies choice overrides after profile overrides). The # hub logs a warning if you do.