From c02823f0a07dfb6a0eefc2cc6e7496dc9b7de358 Mon Sep 17 00:00:00 2001 From: Johnny Bouder Date: Thu, 20 Aug 2026 13:16:25 -0400 Subject: [PATCH] docs: document pack setup for administrators Closes #89. Adds five administrator pages to the existing Astro + Starlight site, covering the four questions the issue asks: Administration admin-setup cluster prerequisites, the single required value and everything derived from it, what the chart creates, the Keycloak bootstrap Job, and a verification checklist server-profiles profile_list sizing, profile_options image choices, access: all/yaml/keycloak gating, idle culling, and the two per-user PVCs nebi-integration what nebi-pack must already provide, the derived OIDC client IDs, the 3-step token exchange and its five inputs, both NetworkPolicies, workspace storage, and admin-provisioned registries mlflow-integration tracking URI, the egress rule, the client library, verification, and what the integration does not give you Reference values-reference field-by-field for keycloak, subdomains, nebariapp, singleuser, singleuserCuller, sharedStorage, nebi, rbac.bootstrap, jupyterhub.custom, and the upstream passthrough Content is derived from values.yaml, the templates, _helpers.tpl, and config/jupyterhub/*.py rather than paraphrased from the README, so the derivation rules and defaults match what the chart renders. Three things that are easy to get wrong get explicit callouts: - jupyterhub.hub.extraVolumes / extraVolumeMounts are lists, so overriding either drops the custom-config and oauth-client mounts and the hub silently falls back to dummy auth with an empty jupyterhub_config.d. - The base domain is keycloak.hostname minus its first label, so a single-label value derives an empty hub hostname, Nebi URL, and token URL - the chart renders and nothing routes. - z2jh defaults singleuser egress to deny private IPs, and a NetworkPolicy rule must name the pod port (5000 for MLflow), not the Service port, because the rule is evaluated after kube-proxy has already translated the ClusterIP. Existing pages are touched as little as possible so the other open docs PRs stay mergeable: configuration.md gains a "Detailed guides" list appended below its section table, and index.md gains an Administration section. Neither restructures what is there. Also adds a Documentation section to the README, which had no link to the published site. --- README.md | 33 +++ docs/astro.config.mjs | 10 + docs/src/content/docs/admin-setup.md | 172 +++++++++++ docs/src/content/docs/configuration.md | 16 +- docs/src/content/docs/index.md | 8 + docs/src/content/docs/mlflow-integration.md | 172 +++++++++++ docs/src/content/docs/nebi-integration.md | 225 +++++++++++++++ docs/src/content/docs/server-profiles.md | 201 +++++++++++++ docs/src/content/docs/values-reference.md | 303 ++++++++++++++++++++ 9 files changed, 1139 insertions(+), 1 deletion(-) create mode 100644 docs/src/content/docs/admin-setup.md create mode 100644 docs/src/content/docs/mlflow-integration.md create mode 100644 docs/src/content/docs/nebi-integration.md create mode 100644 docs/src/content/docs/server-profiles.md create mode 100644 docs/src/content/docs/values-reference.md diff --git a/README.md b/README.md index 55def79..54be6e9 100644 --- a/README.md +++ b/README.md @@ -160,6 +160,39 @@ To release a new version: **Note:** Enable GitHub Pages on the `gh-pages` branch in repo settings after the first release. +## Documentation + +The docs site lives in [`docs/`](docs/) and is built with [Astro](https://astro.build) + +[Starlight](https://starlight.astro.build) using the shared `@nebari/starlight` theme. It +deploys to [packs.nebari.dev/data-science-pack/](https://packs.nebari.dev/data-science-pack/) +on every merge to `main`; pull requests that touch `docs/` get a preview URL posted as a +comment. + +Administrator guides: + +- [Admin setup](https://packs.nebari.dev/data-science-pack/admin-setup/) - cluster + prerequisites, the one required value, and what the chart creates. +- [Values reference](https://packs.nebari.dev/data-science-pack/values-reference/) - + field-by-field detail for every value. +- [Server profiles](https://packs.nebari.dev/data-science-pack/server-profiles/) - sizing + JupyterLab servers and gating profiles by group. +- [Nebi integration](https://packs.nebari.dev/data-science-pack/nebi-integration/) - images, + OIDC clients, token exchange, registries. +- [MLflow integration](https://packs.nebari.dev/data-science-pack/mlflow-integration/) - + letting notebooks log experiments to MLflow. + +```bash +cd docs +npm ci +npm run dev # dev server with hot reload at http://localhost:4321 +npm run build # static build into docs/dist/ +npm test # unit tests +``` + +Pages live in `docs/src/content/docs/` - each `.md` or `.mdx` file becomes a page, and the +sidebar is configured in `docs/astro.config.mjs`. See [`docs/README.md`](docs/README.md) for +details. + ## License Apache License 2.0 - see [LICENSE](LICENSE) for details. diff --git a/docs/astro.config.mjs b/docs/astro.config.mjs index 0c54baf..c33bd12 100644 --- a/docs/astro.config.mjs +++ b/docs/astro.config.mjs @@ -33,10 +33,20 @@ export default defineConfig({ { label: 'Shared Storage', slug: 'shared-storage' }, ], }, + { + label: 'Administration', + items: [ + { label: 'Admin setup', slug: 'admin-setup' }, + { label: 'Server profiles', slug: 'server-profiles' }, + { label: 'Nebi integration', slug: 'nebi-integration' }, + { label: 'MLflow integration', slug: 'mlflow-integration' }, + ], + }, { label: 'Reference', items: [ { label: 'Configuration', slug: 'configuration' }, + { label: 'Values reference', slug: 'values-reference' }, { label: 'NebariApp Integration', slug: 'nebariapp-integration' }, ], }, diff --git a/docs/src/content/docs/admin-setup.md b/docs/src/content/docs/admin-setup.md new file mode 100644 index 0000000..78b3c38 --- /dev/null +++ b/docs/src/content/docs/admin-setup.md @@ -0,0 +1,172 @@ +--- +title: Admin setup +description: Deploying and configuring the Data Science Pack as a cluster administrator. +--- + +This is the administrator's entry point: what the chart needs from the cluster, what it +derives on its own, and where each knob lives. For a five-minute install, start with +[Quick Start](/quick-start/) instead. + +## What the cluster must provide + +| Requirement | Why | Optional? | +|---|---|---| +| [nebari-operator](https://github.com/nebari-dev/nebari-operator) | Reconciles the `NebariApp` into routing, TLS, and a Keycloak OIDC client | Yes — set `nebariapp.enabled: false` | +| Envoy Gateway | The `NebariApp`'s HTTPRoute attaches to it | With the operator | +| cert-manager | Issues the TLS certificate for the hub hostname | With the operator | +| Keycloak (`bitnami/keycloakx`) | Identity provider; the operator provisions the hub client in it | With the operator | +| A ReadWriteMany StorageClass | Per-group shared directories | Yes — see [Shared Storage](/shared-storage/) | +| A default (RWO) StorageClass | Per-user home PVCs and Nebi workspace PVCs | No | +| Namespace label `nebari.dev/managed=true` | The operator ignores `NebariApp`s in unlabeled namespaces | No, when the operator is used | + +Without the operator the chart still installs — dummy authenticator, no routing, no shared +Keycloak. That is the local-development path, not a deployment mode. + +## One required field + +The chart is built around a single input. Everything else is derived by subdomain +convention and can be overridden individually: + +```yaml +keycloak: + hostname: keycloak.example.com +``` + +From that one value: + +| Derived | Rule | Example | +|---|---|---| +| Base domain | `keycloak.hostname` minus its first label | `example.com` | +| Hub hostname | `.` | `hub.example.com` | +| Nebi external URL | `https://.` | `https://nebi.example.com` | +| Keycloak token URL | `https:///realms//…/token` | — | +| Hub OIDC client ID | `jupyterhub--` | `jupyterhub-data-science-pack-nebari-data-science-pack` | +| Nebi OIDC client ID | `nebi--nebari-nebi-pack` | `nebi-nebi-pack-nebari-nebi-pack` | + +:::caution[Derivation needs a dotted hostname] +The base domain is `keycloak.hostname` with its first label stripped. A single-label value +like `keycloak` yields an empty base domain, and the hub hostname, Nebi URL, and token URL +all come out empty — the chart renders, and nothing routes. Set `nebariapp.hostname` and +`nebi.remoteURL` explicitly in that case. +::: + +## Install + +```bash +helm repo add nebari https://raw.githubusercontent.com/nebari-dev/helm-repository/gh-pages/ +helm repo update + +kubectl create namespace data-science +kubectl label namespace data-science nebari.dev/managed=true + +helm install data-science-pack nebari/nebari-data-science-pack \ + --namespace data-science \ + --set keycloak.hostname=keycloak.example.com +``` + +Also available as an OCI artifact: + +```bash +helm install data-science-pack \ + oci://quay.io/nebari/charts/nebari-data-science-pack --version +``` + +:::note[The release name is load-bearing] +Two things are derived from it: the hub OIDC client ID, and the Secret name the hub mounts +at `/etc/oauth` +(`{Release.Name}-{Chart.Name}-oidc-client`, hardcoded in +`jupyterhub.hub.extraVolumes`). Installing under a non-default release name means updating +that `secretName` — and the matching `secretKeyRef` under `jupyterhub.hub.extraEnv` — or the +hub silently falls back to dummy auth. +::: + +## Where each knob lives + +Configuration splits across three layers. Knowing which one you are in explains most +"my value did nothing" reports. + +| Layer | Path | What it is | +|---|---|---| +| Chart values | `keycloak`, `subdomains`, `nebariapp`, `singleuser`, `singleuserCuller`, `sharedStorage`, `nebi`, `rbac` | This chart's own values | +| Chart-derived hub config | `jupyterhub.custom.*` | Read by the Python files in `jupyterhub_config.d/` via `get_chart_config()` | +| Upstream passthrough | everything else under `jupyterhub.*` | Handed verbatim to [Zero to JupyterHub](https://z2jh.jupyter.org/) | + +Field-by-field detail for all three is in the [Values reference](/values-reference/). + +:::caution[Two z2jh lists replace rather than merge] +`jupyterhub.hub.extraVolumes` and `extraVolumeMounts` are lists, so overriding either +**replaces** the chart's entries. Both carry required mounts — `custom-config` +(the `jupyterhub_config.d` ConfigMap) and `oauth-client` (the OIDC Secret). Drop them and +the hub comes up with an empty config directory and dummy auth. + +Re-include the chart's entries in any override. The same applies to +`jupyterhub.hub.initContainers`, which carries the CA-bundle merge step. +::: + +## What the chart creates + +Beyond the z2jh subchart's own objects: + +| Object | Template | Purpose | +|---|---|---| +| `NebariApp` | `nebariapp.yaml` | Routing, TLS, Keycloak client, landing-page card | +| Hub config ConfigMap | `hub-config.yaml` | The four `jupyterhub_config.d/` Python files | +| Singleuser config ConfigMap | `singleuser-config.yaml` | Per-pod config mounted by the spawner | +| Nebi config ConfigMap | `singleuser-nebi-config.yaml` | Admin-provisioned Nebi registries — only when customized | +| Shared PVC (+ NFS server) | `shared-pvc.yaml`, `nfs-server.yaml` | Per-group shared storage | +| NFS client installer | `nfs-client-installer.yaml` | DaemonSet installing `nfs-common`, opt-in | +| Keycloak RBAC bootstrap Job | `keycloak-rbac-bootstrap-job.yaml` | post-install/upgrade hook; groups mapper + shared-mount role | +| Two NetworkPolicies | `singleuser-gateway-egress.yaml`, `hub-nebi-networkpolicy.yaml` | Egress the subchart's policy does not cover | + +## The Keycloak bootstrap job + +`rbac.bootstrap.enabled` defaults to `true`. It runs as a post-install/post-upgrade hook in +the `keycloak` namespace, authenticates with the admin credentials Secret, and is +idempotent — it skips cleanly when `kcAdminCredentialSecret` is unset, so the chart still +installs on clusters that have not surfaced one. + +It does four things: + +1. Adds the `oidc-group-membership-mapper` to the `groups` client scope. Without it the + `groups` claim is empty, and both shared storage and `access: yaml` profile gating + silently fall back to "no groups". +2. Creates the `allow-group-directory-creation-role` client role on the hub client. +3. Enables `serviceAccountsEnabled` on the hub client and binds + `realm-management.{view-clients,view-groups,view-realm}` to its service account. +4. Assigns the shared-mount role to the groups listed in `rbac.bootstrap.sharedMountGroups`. + +Set `enabled: false` for BYO-Keycloak or local development. Override `namespace`, +`kcAdminCredentialSecret`, and `kcHost` for non-bitnami Keycloak layouts. + +## Integrations + +- **[Nebi](/nebi-integration/)** — the environment manager. Ships into user pods via an init + container and needs a matching OIDC client for token exchange. +- **[MLflow](/mlflow-integration/)** — experiment tracking. Two values, one of which is a + NetworkPolicy that has to name the pod port rather than the service port. +- **[NebariApp](/nebariapp-integration/)** — the CRD fields this chart sets and why. + +## User-facing configuration + +- **[Server profiles](/server-profiles/)** — sizes, images, and per-group gating. +- **[Shared Storage](/shared-storage/)** — per-group directories and RWX requirements. + +## Verify a deployment + +```bash +kubectl -n data-science get pods +kubectl -n data-science get nebariapp,httproute,certificate + +# The operator only acts on labeled namespaces +kubectl get namespace data-science -o jsonpath='{.metadata.labels}' + +# The hub reads its OAuth client from this Secret; absent means dummy auth +kubectl -n data-science get secret data-science-pack-nebari-data-science-pack-oidc-client + +# Did the Keycloak bootstrap hook succeed? +kubectl -n keycloak get jobs -l app.kubernetes.io/instance=data-science-pack +``` + +Then log in through Keycloak and check that the profile selector appears with the sizes you +expect. An empty or unexpectedly short list usually means the `groups` claim is missing — +see [Server profiles](/server-profiles/#gating-profiles-by-group). diff --git a/docs/src/content/docs/configuration.md b/docs/src/content/docs/configuration.md index 1af1dcb..f25dd47 100644 --- a/docs/src/content/docs/configuration.md +++ b/docs/src/content/docs/configuration.md @@ -23,7 +23,21 @@ Every derived value can still be overridden explicitly. Values under For the full set of fields and their defaults, read [`values.yaml`](https://github.com/nebari-dev/data-science-pack/blob/main/values.yaml) -directly — it is heavily commented and is the source of truth. +directly — it is heavily commented and is the source of truth. The +[Values reference](/values-reference/) covers the same ground field by field, including the +derivation rules and the defaults worth knowing about before you override them. + +## Detailed guides + +- [Admin setup](/admin-setup/) — cluster prerequisites, the derivation model, and what the + chart creates. +- [Values reference](/values-reference/) — every field in every section. +- [Server profiles](/server-profiles/) — `jupyterhub.custom.profiles`, image choices, and + per-group gating. +- [Nebi integration](/nebi-integration/) — the `nebi.*` values and the token exchange behind + them. +- [MLflow integration](/mlflow-integration/) — `jupyterhub.singleuser.extraEnv` plus an + egress rule. ## Local development diff --git a/docs/src/content/docs/index.md b/docs/src/content/docs/index.md index 3a4ea1d..fc05a58 100644 --- a/docs/src/content/docs/index.md +++ b/docs/src/content/docs/index.md @@ -28,9 +28,17 @@ Nebari's custom images, per-group shared storage, and integration with the - [Architecture](/architecture/) — how the proxy, hub, jhub-apps, and user pods fit together. - [Shared Storage](/shared-storage/) — per-group directories, StorageClass requirements, and the transitional NFS mode. +## Administration + +- [Admin setup](/admin-setup/) — cluster prerequisites, the one required value, and what the chart creates. +- [Server profiles](/server-profiles/) — sizing JupyterLab servers and gating profiles by group. +- [Nebi integration](/nebi-integration/) — wiring the environment manager: images, OIDC, registries. +- [MLflow integration](/mlflow-integration/) — letting notebooks log experiments to MLflow. + ## Reference - [Configuration](/configuration/) — the top-level `values.yaml` sections. +- [Values reference](/values-reference/) — field-by-field detail for every value. - [NebariApp Integration](/nebariapp-integration/) — the CRD fields this chart sets and why. Source, issues, and the full `values.yaml` live in the diff --git a/docs/src/content/docs/mlflow-integration.md b/docs/src/content/docs/mlflow-integration.md new file mode 100644 index 0000000..7cb4721 --- /dev/null +++ b/docs/src/content/docs/mlflow-integration.md @@ -0,0 +1,172 @@ +--- +title: MLflow integration +description: Letting notebooks log experiments to an MLflow deployment — tracking URI, NetworkPolicy, and the client library. +--- + +[mlflow-pack](https://packs.nebari.dev/mlflow-pack/) deploys MLflow with its own hostname, +Keycloak SSO, and PostgreSQL backend. Connecting notebooks to it takes two values in this +chart, plus a client library in the user's environment. + +Nothing here is enabled by default — the two packs are independent, and this is the wiring +between them. + +## Why notebooks bypass the gateway + +MLflow's browser UI sits behind Envoy's OIDC filter: unauthenticated requests get a 302 to +Keycloak. The MLflow Python client has no cookie jar and no interactive login, so pointing +`MLFLOW_TRACKING_URI` at `https://mlflow.example.com` gets a redirect the client cannot +follow — usually surfacing as a parse error rather than an auth error. + +Notebooks therefore talk to MLflow's ClusterIP service directly, over the cluster network. + +``` + browser ──► Envoy Gateway ──► MLflow UI (Keycloak login) + notebook ─────────────────► mlflow-pack.mlflow.svc:80 (no auth) +``` + +## Configuration + +Both halves go under `jupyterhub.singleuser`: + +```yaml +jupyterhub: + singleuser: + extraEnv: + MLFLOW_TRACKING_URI: "http://mlflow-pack.mlflow.svc.cluster.local:80" + networkPolicy: + egress: + - ports: + - port: 5000 + protocol: TCP + to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: mlflow +``` + +Adjust `mlflow-pack` and `mlflow` to the release name and namespace MLflow was installed +under. The MLflow service is named after its **release**, not `-mlflow` — the +community chart's fullname helper collapses when the release name contains the chart name. + +:::caution[The two port numbers are different, and both are correct] +The tracking URI uses **80**, the Service port. The NetworkPolicy uses **5000**, the pod +port. + +NetworkPolicy is enforced at the pod IP level, *after* kube-proxy has already translated the +ClusterIP's 80 to the container's 5000. A rule written against port 80 matches nothing and +every connection times out with no error anywhere. +::: + +This is the single most common reason the integration silently fails. + +### Why a NetworkPolicy is needed at all + +Zero to JupyterHub defaults `singleuser.networkPolicy.enabled: true` with +`egressAllowRules.privateIPs: false`. User pods can reach the public internet and DNS, and +nothing else in the cluster. Every in-cluster destination needs an explicit rule. + +The blunt alternative opens all private addresses: + +```yaml +jupyterhub: + singleuser: + networkPolicy: + egressAllowRules: + privateIPs: true +``` + +That works, and it also lets user code reach every other in-cluster service — databases, +internal APIs, the Kubernetes API's private endpoints. Prefer the targeted rule unless you +have a reason not to. + +`extraEnv` is a dict, so adding `MLFLOW_TRACKING_URI` merges with the chart's existing +entries. `networkPolicy.egress` is a list, but its default is empty, so setting it is +additive in practice — entries there are rendered verbatim into the generated policy and +unioned with the built-in rules. + +## Restart to pick it up + +Running servers keep the environment and the policy they started with: + +```bash +kubectl -n data-science rollout restart deployment/hub +``` + +Then users must stop and start their server from the hub control panel. A pod that predates +the change has neither the variable nor the new egress rule. + +## The client library is not in the image + +`mlflow` is **not** part of the JupyterLab image's package set. Setting the tracking URI +does nothing on its own — `import mlflow` fails. + +Users install it in their own environment. With [Nebi](/nebi-integration/), add it to a +workspace: + +```toml +[dependencies] +mlflow = ">=3" +``` + +or, for a quick check in the base environment: + +```bash +pip install mlflow +``` + +For a cluster where MLflow is standard, adding it to a shared registry environment (see +[Nebi integration](/nebi-integration/#admin-provisioned-registries)) means users get it +without asking. + +## Verify + +From a notebook, confirm the environment reached the pod: + +```python +import os +print(os.environ.get("MLFLOW_TRACKING_URI")) +``` + +Empty means the server predates the change — restart it. Then log a run: + +```python +import mlflow + +mlflow.set_experiment("connectivity-check") +with mlflow.start_run(): + mlflow.log_param("framework", "pytorch") + mlflow.log_metric("accuracy", 0.95) + +print("Run ID:", mlflow.last_active_run().info.run_id) +``` + +It should appear in the MLflow UI under `connectivity-check`. A hang that eventually times +out is the NetworkPolicy port. + +From outside the notebook: + +```bash +kubectl -n data-science exec -- \ + python -c "import urllib.request; print(urllib.request.urlopen('http://mlflow-pack.mlflow.svc.cluster.local:80/health').status)" +``` + +## What this does not give you + +**Identity.** In-cluster access to MLflow is unauthenticated — any pod the NetworkPolicy +permits can read and write every experiment, with no user attached. Runs are attributed by +whatever the client sets, not by who is logged into JupyterHub. Scope the egress rule to +the namespaces that genuinely need it and treat MLflow as a shared, trusted-network +service. + +**Durable artifacts.** By default mlflow-pack stores run metadata in PostgreSQL but writes +artifacts to a path inside the MLflow pod, with no volume behind it. `log_artifact()` and +`log_model()` from a notebook succeed and are lost on the next MLflow restart — leaving runs +that reference models which no longer exist. Configure an artifact bucket on the MLflow side +before people start logging models; see +[Artifact storage](https://packs.nebari.dev/mlflow-pack/artifact-storage/). + +## Other in-cluster services + +The same two-part pattern — an env var plus a targeted egress rule naming the **pod** port — +works for any in-cluster endpoint you want notebooks to reach. The port trap applies every +time. diff --git a/docs/src/content/docs/nebi-integration.md b/docs/src/content/docs/nebi-integration.md new file mode 100644 index 0000000..2223cc8 --- /dev/null +++ b/docs/src/content/docs/nebi-integration.md @@ -0,0 +1,225 @@ +--- +title: Nebi integration +description: Wiring the Nebi environment manager into JupyterHub — images, OIDC clients, token exchange, and registries. +--- + +[Nebi](https://github.com/nebari-dev/nebi) is the environment manager users reach from +inside JupyterLab. It runs as its own service (`nebari-nebi-pack`, in its own namespace) and +this chart wires JupyterHub to it: the binary ships into every user pod, and the hub +performs a token exchange so each user's Nebi session is scoped to them. + +This page is the administrator's side. For what users *do* with it once it works, see the +Nebi in JupyterLab page added by +[PR #204](https://github.com/nebari-dev/data-science-pack/pull/204). + +:::note[Nebi is not NebariApp] +Similar names, unrelated things. **Nebi** is the environment manager described here. +**NebariApp** is the CRD the operator reconciles into routing and auth — see +[NebariApp Integration](/nebariapp-integration/). +::: + +## What has to exist + +1. **nebi-pack deployed**, in its own namespace (`nebi` by default), from + [nebari-nebi-pack](https://github.com/nebari-dev/nebari-nebi-pack). +2. **Its own Keycloak OIDC client**, provisioned by the operator from nebi-pack's own + `NebariApp`. +3. **Network reachability** — the hub must reach Nebi in-cluster, and user pods must reach + it through the gateway. + +This chart contributes the third and the wiring; it does not deploy Nebi itself. + +## Minimum configuration + +With `keycloak.hostname` set and nebi-pack installed under its default release name and +namespace, nothing else is required. The defaults derive: + +| Value | Derived as | +|---|---| +| `nebi.remoteURL` | `https://.` | +| `nebi.internalURL` | `http://nebi-pack-nebari-nebi-pack..svc.cluster.local` | +| Nebi OIDC client ID | `nebi--nebari-nebi-pack` | +| Hub OIDC client ID | `jupyterhub--` | + +Override any of them when your layout differs: + +```yaml +nebi: + namespace: nebi + releaseName: nebi-pack + remoteURL: https://nebi.example.com + internalURL: http://nebi-pack-nebari-nebi-pack.nebi.svc.cluster.local + port: 8460 +``` + +:::caution[The derived client ID follows nebi-pack's *release* name] +`nebi.releaseName` is not this chart's release name — it is the Helm release nebi-pack was +installed under, because the operator names the client after it. Install nebi-pack as +anything other than `nebi-pack` and this must match, or the token exchange fails with an +invalid-client error from Keycloak. +::: + +## How the binary reaches user pods + +The Nebi binary is **not** baked into the JupyterLab image. An init container copies it out +of `nebi.image` into an `emptyDir` mounted at `/usr/local/bin/nebi` in the user pod. + +That means the Nebi version is a deploy-time decision: + +```yaml +nebi: + image: + repository: quay.io/nebari/nebi + tag: "sha-bd25921" + pullPolicy: IfNotPresent +``` + +Leaving `tag` empty disables the init container entirely — no Nebi in user pods. It is +pinned per chart release (`scripts/bump_image_tags.py` only handles the JupyterLab images), +so override it to test a PR build or to roll forward between chart releases. + +## The token exchange + +Nebi needs a per-user credential, and the hub is the only component holding the user's +Keycloak tokens. At spawn, the hub runs a three-step exchange: + +1. **Refresh** the user's access token at Keycloak, using the hub client's ID and secret. +2. **Exchange** that access token for an ID token with the *Nebi* audience — a standard + Keycloak token-exchange grant from the hub client to the Nebi client. +3. **Exchange** the Nebi ID token at Nebi's own `/api/v1/auth/session` for a Nebi JWT. + +The resulting JWT is injected into the user pod, so Nebi acts as that user rather than as a +shared service account. The same helper is reused by jhub-apps to populate its environment +dropdown. + +Five inputs must all resolve or the exchange aborts: + +| Input | Source | +|---|---| +| `keycloak-token-url` | derived from `keycloak.hostname` / `serviceHost` + realm | +| `nebi-client-id` | derived from `nebi.releaseName` | +| `jupyterhub-client-id` | derived from the release and chart names | +| `JUPYTERHUB_OIDC_CLIENT_SECRET` | env var from the operator-provisioned Secret | +| `nebi-internal-url` | derived from `nebi.namespace` | + +The hub logs exactly which are missing: + +```bash +kubectl -n data-science logs deploy/hub | grep -i "token-exchange\|nebi-envs" +``` + +## NetworkPolicy + +Two paths, two policies. + +**Hub → Nebi** is rendered automatically whenever `nebi.internalURL` and `nebi.namespace` +are both set (`hub-nebi-networkpolicy.yaml`), allowing egress to `nebi.port` in the Nebi +namespace. It is needed for the token exchange at spawn time. + +**User pods → Nebi** goes through the gateway, not the Service, because users reach Nebi at +its external URL. That is what `singleuser.networkPolicy.allowEgressToGateway` (default +`true`) covers — an additional NetworkPolicy permitting egress to the Envoy Gateway pod: + +```yaml +singleuser: + networkPolicy: + allowEgressToGateway: true + gatewayNamespace: envoy-gateway-system + gatewayName: nebari-gateway + gatewayPort: 10443 +``` + +Kubernetes unions egress rules across policies selecting the same pod, so both are additive +to the z2jh subchart's own policy. + +## Workspace storage + +Each user gets a dedicated RWO PVC, `nebi-workspaces-{slug}`, created by the spawner and +mounted at `/var/lib/nebi/workspaces`: + +```yaml +jupyterhub: + custom: + workspace-storage-class: "" # empty = cluster default + workspace-storage-capacity: "20Gi" +``` + +Pixi environments run 2–5 GiB each, so 20 GiB is a handful of environments, not dozens. +Size it against how many environments you expect a user to keep. + +## Admin-provisioned registries + +Registries set here are rendered into a ConfigMap mounted at `/etc/nebi/config.yaml` in +every user pod, which Nebi reads at boot. They appear locked in the UI rather than +per-user editable: + +```yaml +nebi: + seedDefaultRegistry: true + registries: + - name: acme-registry + url: registry.acme.com + namespace: acme-envs + default: true +``` + +Each entry follows Nebi's own `registries.entries` schema (`name`, `url`, `namespace`, +`default`). + +:::caution[Public registries only] +Entries carry no credentials — this is a plain ConfigMap, and Nebi does not accept +authentication here. Private registries are out of scope for this mechanism. +::: + +Set `seedDefaultRegistry: false` to remove the built-in `quay.io/nebari_environments` +registry Nebi seeds by default. + +:::note[Both settings need a hub restart] +The mount wiring lives in the hub ConfigMap (`01-spawner.py`), so changes take effect only +for servers started *after* the hub pod restarts: + +```bash +kubectl -n data-science rollout restart deployment/hub +``` +::: + +## jhub-apps and Nebi environments + +jhub-apps offers a Nebi environment when deploying an app. Two things gate it: + +- The chart auto-injects a Nebi card into `japps-config.additional_services` when + `nebi.remoteURL` is set or derivable. Overriding `additional_services` replaces that + default — re-include it if you add your own. +- `jupyterhub.custom.jhub-app-proxy-version` must be **≥ v0.2.3**. Older versions only + activate conda environments and silently fall back to the base environment, so an app + launched into a Nebi environment comes up missing its packages. + +## Behind a TLS-inspecting proxy + +The init container that pre-pulls Nebi environments makes its own outbound HTTPS calls +(`nebi pull`, then `pixi install` from PyPI and conda). When +`jupyterhub.custom.trust-bundle-enabled` is on, it receives the merged CA bundle too — the +CA merge step is ordered before it, so the bundle is ready. See +[Values reference](/values-reference/#enterprise-ca-bundle). + +## Troubleshooting + +| Symptom | Check | +|---|---| +| No Nebi in the pod at all | `nebi.image.tag` is empty, so the init container is not wired. | +| Token exchange aborts | Hub logs name the missing input — usually `nebi-client-id` from a non-default `nebi.releaseName`. | +| `invalid_client` from Keycloak | The derived Nebi client ID does not match the client the operator created for nebi-pack. | +| Empty environment dropdown in jhub-apps | Exchange failure, or `auth_state` missing — `kubectl logs deploy/hub \| grep nebi-envs`. | +| Registry changes not visible | Hub not restarted since the change. | +| Apps missing packages in a Nebi env | `jhub-app-proxy-version` below v0.2.3. | + +```bash +# Is the binary in the pod? +kubectl -n data-science exec -- nebi --version + +# Did the workspace PVC bind? +kubectl -n data-science get pvc -l app=nebi-workspaces + +# What did the hub resolve? +kubectl -n data-science logs deploy/hub | grep -i "token-exchange" +``` diff --git a/docs/src/content/docs/server-profiles.md b/docs/src/content/docs/server-profiles.md new file mode 100644 index 0000000..7693945 --- /dev/null +++ b/docs/src/content/docs/server-profiles.md @@ -0,0 +1,201 @@ +--- +title: Server profiles +description: Sizing JupyterLab servers, offering image choices, and gating profiles by group. +--- + +Profiles are the server sizes users pick from at spawn. Each entry in +`jupyterhub.custom.profiles` maps directly to a +[KubeSpawner](https://jupyterhub-kubespawner.readthedocs.io/) `profile_list` item, so +anything KubeSpawner accepts works without a chart change. + +The chart ships two: + +| Profile | Slug | Resources | +|---|---|---| +| Small Instance *(default)* | `small-instance` | 1 CPU / 2 GB limit, 0.5 CPU / 1 GB guarantee | +| Medium Instance | `medium-instance` | 4 CPU / 8 GB limit, 2 CPU / 4 GB guarantee | + +Set `profiles: []` to remove the selector entirely and run in single-instance mode. + +## Adding a profile + +```yaml +jupyterhub: + custom: + profiles: + - slug: large-instance + display_name: "Large Instance" + description: "16 CPU / 64 GB RAM — large in-memory datasets." + kubespawner_override: + image: quay.io/nebari/nebari-data-science-pack-jupyterlab:sha-16c1922 + cpu_limit: 16 + cpu_guarantee: 8 + mem_limit: "64G" + mem_guarantee: "32G" +``` + +`slug` is a stable identifier independent of the human-facing `display_name`; omit it and +KubeSpawner slugifies the display name (`"Large Instance"` → `large-instance`). Set it +explicitly — the slug is what `access: keycloak` gating matches on, and renaming a display +name would otherwise silently change it. + +`default: true` marks the pre-selected profile. Exactly one should have it. + +`cpu_guarantee` and `mem_guarantee` become the pod's *requests*; `cpu_limit` and `mem_limit` +become its *limits*. Guarantees drive scheduling, so a guarantee larger than any node can +satisfy leaves the server `Pending` forever with no message in the UI. + +`kubespawner_override` accepts any KubeSpawner trait — `node_selector`, `image`, +`extra_resource_limits`, `tolerations`, `environment`, and the rest. + +## Image choices within a profile + +`profile_options` adds a second dropdown under the selected profile: + +```yaml + - slug: small-instance + display_name: "Small Instance" + default: true + kubespawner_override: + image: quay.io/nebari/nebari-data-science-pack-jupyterlab:sha-16c1922 + cpu_limit: 1 + mem_limit: "2G" + profile_options: + image: + display_name: Image + choices: + default: + display_name: "nebari-data-science-pack-jupyterlab:sha-16c1922" + default: true + kubespawner_override: + image: quay.io/nebari/nebari-data-science-pack-jupyterlab:sha-16c1922 + rlang: + display_name: "R" + kubespawner_override: + image: quay.io/nebari/nebari-data-science-pack-jupyterlab-r:sha-16c1922 +``` + +:::caution[The image tag appears in three places per profile] +The outer `kubespawner_override.image` is what jhub-apps' Create App form reads for its +image field; the inner `profile_options.image.choices.default` is what the JupyterLab +profile selector shows. Both must be bumped alongside `jupyterhub.singleuser.image.tag`. + +z2jh values cannot reference other values, so the duplication is unavoidable. +`scripts/bump_image_tags.py` syncs all three on an automated bump — hand edits have to keep +up on their own. +::: + +## Gating profiles by group + +Each profile can declare an `access` mode controlling who sees it. This is parity with +classic Nebari. + +| `access` | Visible to | +|---|---| +| `all` *(or omitted)* | everyone | +| `yaml` | users whose Keycloak groups intersect `groups`, or whose `preferred_username` is in `users` | +| `keycloak` | users whose `jupyterlab-profiles` Keycloak role lists this profile's `slug` | + +```yaml + - slug: gpu-instance + display_name: "G4 GPU Instance" + access: yaml + groups: + - gpu-access + users: + - alice + kubespawner_override: + extra_resource_limits: + nvidia.com/gpu: 1 +``` + +:::note[Unknown access modes fail closed] +Anything other than `all`, `yaml`, or `keycloak` hides the profile and logs a warning. +Restricted profiles gate expensive resources, so a typo must not expose a GPU instance to +the whole cluster. +::: + +The `access`, `groups`, and `users` keys are gating-only — they are stripped before the +profile reaches KubeSpawner. + +### `access: keycloak` + +Moves the allow-list out of the values file and into Keycloak, which is what you want when +the people granting access are not the people editing Helm values. + +Create a `jupyterlab-profiles` client role on the hub client with: + +- attribute `profiles` — the allowed slugs +- attribute `component=jupyterhub-profiles` + +then assign the role to users or groups. The authenticator resolves it at login through the +Keycloak Admin API and stamps the result into `auth_state`, where the spawner reads it. + +Note this requires the hub client's service account to hold the `realm-management` view +roles — which is exactly what the [Keycloak bootstrap Job](/admin-setup/#the-keycloak-bootstrap-job) +provisions. + +### Groups come from the token + +Both `yaml` gating and shared storage read the user's Keycloak groups from the `groups` +claim. If that claim is empty, `access: yaml` profiles are invisible to everyone and shared +directories do not mount. + +The usual cause is a missing `oidc-group-membership-mapper` on the `groups` client scope — +which is the first thing the bootstrap Job fixes. Check with: + +```bash +kubectl -n data-science logs deploy/hub | grep -i "profiles:\|groups" +``` + +## GPU profiles + +A GPU profile requests the resource through `extra_resource_limits`, but scheduling onto a +tainted GPU node group also needs a toleration — whether you must add it yourself depends on +whether the cluster runs the `ExtendedResourceToleration` admission controller (EKS and GKE +do; AKS and most self-managed clusters do not). + +That, plus the fact that `tolerations` in `kubespawner_override` *replaces* rather than +appends, is covered in detail on the GPU profiles page added by +[PR #139](https://github.com/nebari-dev/data-science-pack/pull/139). + +## Idle culling + +Two independent cullers, both on by default. + +| | Value | Default | Scope | +|---|---|---|---| +| In-pod | `singleuserCuller.*` | 15 min | Kernels, terminals, and the server itself — fires even with a browser tab open | +| Hub-level | `jupyterhub.cull` | 30 min | Servers the hub sees as inactive | + +The in-pod culler is the one that actually reclaims resources from users who leave a tab +open overnight; the hub-level culler is the backstop. Raising one without the other rarely +does what you want — see [Values reference](/values-reference/#singleuserculler). + +## Storage per user + +Each user gets two PVCs, both `ReadWriteOnce`: + +| PVC | Size value | Default | Contents | +|---|---|---|---| +| `claim-{username}` | `jupyterhub.custom.storage-capacity` | `20Gi` | Home directory, mounted at `/home/jovyan` | +| `nebi-workspaces-{slug}` | `jupyterhub.custom.workspace-storage-capacity` | `20Gi` | Nebi (pixi) environments | + +Set `workspace-storage-class` and let `storage-capacity` follow your home-directory policy. +Pixi environments run 2–5 GiB each, so the workspace PVC fills faster than people expect. + +:::note[Both PVCs are RWO, so a user's pods land on one node] +The chart adds a pod-affinity rule keeping a user's JupyterLab pod and their jhub-apps pods +together for that reason. A user with several running apps is pinned to a single node. +::: + +## After changing profiles + +Profile changes live in the hub ConfigMap, so the hub must restart: + +```bash +kubectl -n data-science rollout restart deployment/hub +``` + +Running servers keep the profile they spawned with. Users see the new list on their next +spawn. diff --git a/docs/src/content/docs/values-reference.md b/docs/src/content/docs/values-reference.md new file mode 100644 index 0000000..0e1b61d --- /dev/null +++ b/docs/src/content/docs/values-reference.md @@ -0,0 +1,303 @@ +--- +title: Values reference +description: Field-by-field reference for every value the chart owns, plus the jupyterhub.custom keys. +--- + +[`values.yaml`](https://github.com/nebari-dev/data-science-pack/blob/main/values.yaml) is +heavily commented and remains the source of truth. This page is the same content organized +for reading, with the derivation rules and the fields whose defaults are worth knowing +about. + +Sections marked *derived* fall back to a value computed from `keycloak.hostname` — see +[Admin setup](/admin-setup/#one-required-field). + +## `keycloak` + +| Field | Default | What it does | +|---|---|---| +| `keycloak.hostname` | `""` | External Keycloak FQDN. The one field a fresh deploy needs; everything else derives from it. | +| `keycloak.realm` | `nebari` | Realm name. | +| `keycloak.serviceHost` | `keycloak-keycloakx-http.keycloak.svc.cluster.local:8080` | In-cluster Keycloak service, used for the hub↔Nebi token exchange when `hostname` is empty. | +| `keycloak.backchannelURL` | `""` | Split-horizon OIDC — see below. | + +### Split-horizon OIDC + +`backchannelURL` exists for private-VPC clusters where in-cluster CoreDNS cannot resolve the +external Keycloak hostname. The hub pod cannot reach `hostname` at all, so token exchange +over the primary URL fails. + +Set it to the in-cluster URL and the hub uses it for the backchannel legs (`token_url`, +`userdata_url`) while the browser keeps using `hostname` for authorize and end-session. The +`/realms/` suffix is appended for you. + +:::caution[Keycloak needs `KC_HOSTNAME_BACKCHANNEL_DYNAMIC=true`] +Otherwise Keycloak mints tokens whose `iss` claim matches the backchannel URL, and any +validator keyed to the browser-facing issuer rejects them. +::: + +## `subdomains` + +| Field | Default | What it does | +|---|---|---| +| `subdomains.hub` | `hub` | Label prepended to the base domain to derive the hub hostname. | +| `subdomains.nebi` | `nebi` | Same, for Nebi. | + +## `nebariapp` + +Rendered into the `NebariApp` CRD. Semantics of each CRD field are in +[NebariApp Integration](/nebariapp-integration/). + +| Field | Default | What it does | +|---|---|---| +| `nebariapp.enabled` | `true` | Render the `NebariApp`. False outside Nebari. | +| `nebariapp.hostname` | `""` *(derived)* | External hub FQDN. | +| `nebariapp.service.name` | `proxy-public` | Backend service (created by the z2jh subchart). | +| `nebariapp.service.port` | `80` | Backend port. | +| `nebariapp.routing.routes` | `[{pathPrefix: /}]` | Sends every path to the proxy. | +| `nebariapp.auth.enabled` | `true` | Provision a Keycloak client. | +| `nebariapp.auth.provider` | `keycloak` | Identity provider. | +| `nebariapp.auth.provisionClient` | `true` | Operator creates the client and its Secret. | +| `nebariapp.auth.redirectURI` | `/hub/oauth_callback` | JupyterHub's own callback path. | +| `nebariapp.auth.scopes` | `openid, profile, email, groups` | The `groups` scope is what shared storage and profile gating read. | +| `nebariapp.auth.enforceAtGateway` | `false` | See below. | +| `nebariapp.auth.forwardAccessToken` | `false` | The hub persists tokens to `auth_state`; nothing upstream needs an injected Bearer. | +| `nebariapp.landingPage.*` | enabled, "JupyterHub" | Landing-page card: `displayName`, `description`, `icon`/`iconLight`/`iconDark`, `category`, `priority`, `externalUrl`, `healthCheck`. | + +:::note[Why `enforceAtGateway` is false] +JupyterHub is its own OAuth client (`GenericOAuthenticator` with refresh-token grants). +Running Envoy's OIDC filter in front adds nothing, and its cookie-rotation lag stales out +`auth_state` for `/services/japps/*` paths. The operator drops the `SecurityPolicy` on this +flip; the Keycloak client and Secret stay provisioned because `provisionClient` is +independent of enforcement. +::: + +`routing` must stay present. Remove it and the operator reports `RoutingNotConfigured` — +no HTTPRoute, no TLS listener, no reachable hub. + +## `singleuser` (chart-level) + +Not to be confused with `jupyterhub.singleuser`, which is the upstream passthrough. + +| Field | Default | What it does | +|---|---|---| +| `singleuser.networkPolicy.allowEgressToGateway` | `true` | Renders a NetworkPolicy letting user pods reach the Envoy Gateway pod. | +| `singleuser.networkPolicy.gatewayNamespace` | `envoy-gateway-system` | Where the gateway runs. | +| `singleuser.networkPolicy.gatewayName` | `nebari-gateway` | Gateway name, matched on `gateway.envoyproxy.io/owning-gateway-name`. | +| `singleuser.networkPolicy.gatewayPort` | `10443` | Gateway port. | + +Required on Hetzner k3s and any cluster where kube-proxy DNATs the LoadBalancer VIP to the +proxy pod IP *before* the subchart's pod-level policy is evaluated — without it, user pods +cannot reach `https:///services/japps` or the Nebi host. Kubernetes unions egress rules +across policies selecting the same pod, so it is harmless where it is not needed. + +## `singleuserCuller` + +In-pod idle culling, separate from the hub-level `jupyterhub.cull`. This one fires even when +a browser tab is left open. Defaults match classic Nebari. + +| Field | Default | What it does | +|---|---|---| +| `singleuserCuller.kernel.cullConnected` | `true` | Cull kernels despite open browser connections. | +| `singleuserCuller.kernel.cullIdleTimeout` | `900` | Seconds before an idle kernel is culled. | +| `singleuserCuller.kernel.cullInterval` | `300` | Check interval. | +| `singleuserCuller.kernel.cullBusy` | `false` | Never cull a kernel running code. | +| `singleuserCuller.terminal.cullInactiveTimeout` | `900` | Seconds before an idle terminal is culled. | +| `singleuserCuller.terminal.cullInterval` | `300` | Check interval. | +| `singleuserCuller.server.shutdownNoActivityTimeout` | `900` | Seconds after the last kernel/terminal before the server self-terminates. | + +The hub-level culler is `jupyterhub.cull` (`timeout: 1800`, `every: 600`). The two work +together: the in-pod culler shuts an idle server down at 15 minutes; the hub-level culler is +the backstop for servers that stop reporting activity. + +## `sharedStorage` + +Full treatment in [Shared Storage](/shared-storage/). + +| Field | Default | What it does | +|---|---|---| +| `sharedStorage.enabled` | `true` | Mount `/shared/` in user pods. | +| `sharedStorage.storageClass` | `""` | RWX StorageClass when `nfsServer.enabled: false`. Empty uses the cluster default, which must support RWX. | +| `sharedStorage.size` | `10Gi` | Shared PVC size. | +| `sharedStorage.accessModes` | `[ReadWriteMany]` | — | +| `sharedStorage.groups` | `[]` | Allowlist; empty mounts every group from the user's token. | +| `sharedStorage.mountPathPrefix` | `/shared` | Mount prefix in user pods. | +| `sharedStorage.nfsServer.enabled` | `true` | Transitional in-cluster NFS server re-exporting an RWO PVC as RWX. | +| `sharedStorage.nfsServer.storageClass` | `""` | Backing RWO class for the NFS server. | +| `sharedStorage.nfsServer.image.*` | `quay.io/nebari/volume-nfs:0.8-repack` | Repack of an abandoned upstream image. | +| `sharedStorage.nfsServer.installClient` | `false` | DaemonSet installing `nfs-common` on nodes that lack it (k3s, minimal OS images). | +| `sharedStorage.nfsServer.nodeSelector` | `{}` | Pin the NFS pod — RWO reattachment can take 30–120s when it reschedules. | +| `sharedStorage.nfsServer.nodeAffinity` | `{}` | Full affinity spec; overrides `nodeSelector`. | +| `sharedStorage.nfsServer.mountOptions` | `[]` | Set `["nfsvers=3"]` on overlayfs nodes (kind, k3d) where the image's NFSv4 export of `/` is broken. | + +:::caution[Prefer a native RWX class] +`nfsServer` is a transitional fallback tracked for removal in +[issue #29](https://github.com/nebari-dev/data-science-pack/issues/29). On NIC-managed +clusters set `sharedStorage.storageClass: longhorn` and `nfsServer.enabled: false`. +::: + +## `nebi` + +Admin guide: [Nebi integration](/nebi-integration/). + +| Field | Default | What it does | +|---|---|---| +| `nebi.image.repository` | `quay.io/nebari/nebi` | Binary copied into user pods by an init container. | +| `nebi.image.tag` | `sha-bd25921` | Pinned per chart release. Empty disables the init container. | +| `nebi.image.pullPolicy` | `IfNotPresent` | — | +| `nebi.remoteURL` | `""` *(derived)* | Browser-facing Nebi URL, used for the OIDC redirect. | +| `nebi.internalURL` | `""` *(derived)* | In-cluster URL for the token-exchange path. | +| `nebi.namespace` | `nebi` | Where nebi-pack runs; drives the NetworkPolicy and derived URL. | +| `nebi.releaseName` | `nebi-pack` | Drives the derived Nebi OIDC client ID. | +| `nebi.port` | `8460` | Used in the hub→Nebi egress rule. | +| `nebi.seedDefaultRegistry` | `true` | Seed `quay.io/nebari_environments` in each user's Nebi. | +| `nebi.registries` | `[]` | Admin-provisioned public OCI registries. | + +## `rbac.bootstrap` + +The post-install Keycloak Job — see +[Admin setup](/admin-setup/#the-keycloak-bootstrap-job). + +| Field | Default | What it does | +|---|---|---| +| `rbac.bootstrap.enabled` | `true` | Run the Job. False for BYO-Keycloak or local dev. | +| `rbac.bootstrap.namespace` | `keycloak` | Namespace the Job runs in, so it can read the admin Secret without a cross-namespace copy. | +| `rbac.bootstrap.kcAdminCredentialSecret` | `keycloak-admin-credentials` | Secret holding the realm-admin password. Unset makes the Job skip cleanly. | +| `rbac.bootstrap.kcAdminCredentialSecretKey` | `admin-password` | Key within that Secret. | +| `rbac.bootstrap.realmName` | `nebari` | Realm to bootstrap. | +| `rbac.bootstrap.hubClientId` | `""` | Empty reads it at runtime from the operator-provisioned OIDC Secret. | +| `rbac.bootstrap.oidcClientSecretName` | `""` | Empty derives `-oidc-client`. | +| `rbac.bootstrap.sharedMountRoleName` | `allow-group-directory-creation-role` | Must match the hub's `KC_SHARED_MOUNT_ROLE`. | +| `rbac.bootstrap.sharedMountGroups` | `[]` | Keycloak group paths granted the role. Each must already exist. | +| `rbac.bootstrap.hubExternalUrl` | `""` | Empty defaults to `https://{nebariapp.hostname}`. | +| `rbac.bootstrap.kcHost` | in-cluster Keycloak URL | Where the Admin REST API lives. | +| `rbac.bootstrap.image` | `python:3.12-slim` | Runs a stdlib-only script; any small Python image works. | + +:::note[`hubExternalUrl` prevents an "OAuth state mismatch"] +It sets `rootUrl`, `baseUrl`, and `initiate.login.uri` on the hub client. Without them, +Keycloak-initiated SSO flows (account console, third-party launchers) jump straight to +`/hub/oauth_callback` without first hitting `/hub/oauth_login`, so JupyterHub has no +`oauthenticator-state` cookie and returns `400 OAuth state mismatch`. +::: + +## `jupyterhub.custom` + +Read by the Python files in `jupyterhub_config.d/` through `get_chart_config()`. Every URL +and client ID here is optional — leave it empty and the chart derives it. Explicit values +always win. + +| Field | Default | What it does | +|---|---|---| +| `external-url` | `""` *(derived)* | Hub bind hostname. | +| `nebi-image` | `""` *(derived)* | `repository:tag` copied into user pods. | +| `nebi-image-pull-policy` | `IfNotPresent` | — | +| `jhub-app-proxy-version` | `v0.2.3` | Installed at app-spawn time. Must be ≥ v0.2.3 for apps to run inside a Nebi (pixi) environment; older versions only activate conda and fall back to the base env. | +| `nebi-remote-url` | `""` *(derived)* | Browser-facing Nebi URL. | +| `nebi-internal-url` | `""` *(derived)* | In-cluster Nebi URL. | +| `keycloak-token-url` | `""` *(derived)* | Token endpoint for hub↔Nebi exchange. | +| `keycloak-backchannel-issuer-url` | `""` *(derived)* | Full backchannel issuer including `/realms/`. | +| `nebi-client-id` | `""` *(derived)* | — | +| `jupyterhub-client-id` | `""` *(derived)* | — | +| `trust-bundle-enabled` | `false` | Merge an org CA into user pods — see below. | +| `trust-bundle-configmap` | `nebari-trust-bundle` | ConfigMap holding the org CA. | +| `trust-bundle-key` | `ca-certificates.crt` | Key within it. | +| `profiles` | two profiles | Server sizes — see [Server profiles](/server-profiles/). | +| `terminal-customization` | `true` | Starship prompt in JupyterLab terminals. | +| `shared-storage-groups` | `[]` | Allowlist; empty = every group in the token. | +| `shared-storage-mount-prefix` | `/shared` | — | +| `storage-capacity` | `20Gi` | Per-user home PVC size (`claim-{username}`, RWO). | +| `workspace-storage-class` | `""` | Class for the per-user Nebi workspace PVC. Empty uses the cluster default. | +| `workspace-storage-capacity` | `20Gi` | Pixi environments run 2–5 GiB each — size accordingly. | +| `japps-config` | `{hub_host: hub, service_workers: 1}` | Attributes set on `c.JAppsConfig`. | + +:::note[`service_workers: 1` is deliberate] +Four uvicorn workers take ~12s to bind, past the hub's hardcoded 10s +`wait_for_http_server` timeout — jhub-apps then crash-loops. One worker boots in ~3s. +Raise it only if you have many concurrent users, and watch the hub logs on the first +restart. +::: + +### Enterprise CA bundle + +`trust-bundle-enabled` covers clusters behind a TLS-inspecting proxy, where NIC core's +trust-manager projects the org CA into every namespace as a ConfigMap. Enabling it merges +that CA with the image's system bundle via an init container and sets +`REQUESTS_CA_BUNDLE`, `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`, `CURL_CA_BUNDLE`, and +`GIT_SSL_CAINFO` on singleuser and app pods — so pip, conda, and git work without +`--trusted-host` or `ssl_verify` flags. + +The **hub-side** equivalent is always on and lives in `jupyterhub.hub.extraVolumes`, +`extraVolumeMounts`, `initContainers`, and `extraEnv` (search `values.yaml` for +`merge-ca-bundle`). It mounts the ConfigMap `optional: true`, so it is a no-op where +trust-manager is absent. + +The `trust-bundle-configmap` and `trust-bundle-key` values drive both sides, but the +hub-side entries also hardcode the ConfigMap name in `jupyterhub.hub.extraVolumes`. Renaming +the bundle means editing both places. + +## `jupyterhub` (upstream passthrough) + +Everything else goes verbatim to the +[Zero to JupyterHub](https://z2jh.jupyter.org/) chart, version 4.4.0. Values this chart sets +that are worth knowing about: + +| Field | Set to | Why | +|---|---|---| +| `jupyterhub.hub.image` | `quay.io/nebari/nebari-data-science-pack-jupyterhub` | Nebari hub image with jhub-apps pre-installed. | +| `jupyterhub.hub.config.JupyterHub.authenticator_class` | `dummy` | Local development. Real OAuth is wired by `00-gateway-auth.py` from the mounted OIDC Secret. | +| `jupyterhub.hub.config.JupyterHub.admin_access` | `true` | — | +| `jupyterhub.hub.service.extraPorts` | `10202` | jhub-apps. | +| `jupyterhub.singleuser.image` | `quay.io/nebari/nebari-data-science-pack-jupyterlab` | — | +| `jupyterhub.singleuser.defaultUrl` | `/lab` | — | +| `jupyterhub.singleuser.storage.type` | `none` | jhub-apps' `JHubSpawner` expects volumes as a list; the subchart's dynamic storage generates a dict. The home PVC is configured in `01-spawner.py` instead. | +| `jupyterhub.proxy.service.type` | `ClusterIP` | Routing is the `NebariApp`'s job. | +| `jupyterhub.scheduling.userScheduler.enabled` | `false` | — | +| `jupyterhub.cull` | `enabled: true`, `timeout: 1800`, `every: 600` | Matches classic Nebari. | + +:::caution[`extraVolumes` and `extraVolumeMounts` replace on override] +Both carry required entries — `custom-config` (the `jupyterhub_config.d` ConfigMap) and +`oauth-client` (the OIDC Secret), plus the CA-bundle volumes. Overriding either list without +re-including them leaves the hub with an empty config directory and dummy auth. +::: + +`jupyterhub.singleuser.extraEnv` is a **dict**, so adding a key merges rather than replaces. +That is the supported hook for injecting things like `MLFLOW_TRACKING_URI` — see +[MLflow integration](/mlflow-integration/). + +### Egress from user pods + +z2jh defaults `singleuser.networkPolicy.enabled: true` with +`egressAllowRules.privateIPs: false`, so **user pods cannot reach other in-cluster services +by default**. Reaching an in-cluster endpoint takes an explicit rule: + +```yaml +jupyterhub: + singleuser: + networkPolicy: + egress: + - ports: + - port: + protocol: TCP + to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: +``` + +Entries under `egress` are rendered verbatim into the generated policy and unioned with the +built-in rules. The port must be the **pod** port, not the Service port — NetworkPolicy is +evaluated after kube-proxy has already translated the ClusterIP. + +## Inspecting + +```bash +helm template data-science-pack . --set keycloak.hostname=keycloak.example.com | less +helm -n data-science get values data-science-pack +helm -n data-science get values data-science-pack --all +``` + +To check a derived value actually landed, read it off the running hub rather than the +values: + +```bash +kubectl -n data-science get cm nebari-data-science-pack-hub-config -o yaml | head -40 +```