Discussion: evolve the pack for group RayService, user RayClusters, RayJobs and a Ray gateway

[nenb]

Definitions

RayService: Runs a long-lived Ray cluster for serving models and manages safe updates to its Ray Serve applications.

RayCluster: Runs a standalone Ray cluster for interactive or shared compute.

RayJob: Unlike a RayCluster, it is not primarily a cluster that a user connects to interactively. It is used for durable UI or API submissions and can create and clean up a temporary Ray cluster for the job.

Current topology

flowchart TB
    UI["Deployed UI and API"]
    NotebookA["User A JupyterLab"]
    NotebookB["User B JupyterLab"]
    Models["Model callers"]
    Shared["One shared RayService<br/>and managed RayCluster"]

    UI --> Shared
    NotebookA --> Shared
    NotebookB --> Shared
    Models --> Shared

Loading

The current pack provides one shared Ray environment for serving, UI compute and every notebook. There is no group separation, no user-owned RayCluster and no ephemeral RayJob cluster.

Proposed topology

Single group

flowchart TB
    Gateway["One platform Ray gateway<br/>authentication, authorization, routing and lifecycle"]
    UI["User through the shared UI"]
    NotebookA["User A JupyterLab"]
    NotebookB["User B JupyterLab"]

    subgraph Group["Trusted group's Ray namespace"]
        Serve["One shared RayService<br/>group model serving"]
        Job["RayJob<br/>ephemeral cluster"]
        ClusterA["User A RayCluster<br/>Ray Client endpoint"]
        ClusterB["User B RayCluster<br/>Ray Client endpoint"]
    end

    UI --> Gateway
    NotebookA --> Gateway
    NotebookB --> Gateway
    Gateway -->|authorized model requests| Serve
    Gateway -->|create and manage| Job
    Gateway -->|create and manage| ClusterA
    Gateway -->|create and manage| ClusterB
    NotebookA -.->|authorized Ray Client connection| ClusterA
    NotebookB -.->|authorized Ray Client connection| ClusterB

Loading

Why both RayJob and RayCluster?

A RayJob runs a complete command in a separate process. A notebook cannot use it to keep live Python objects, ObjectRef values or actor handles between cells. A RayCluster with Ray Client keeps the notebook connected, so users can submit work and reuse state interactively.

Multiple groups

flowchart TB
    IdP["OIDC identity and group membership"] --> Gateway["One platform Ray gateway"]

    UsersA["Group A users"] --> Gateway
    UsersB["Group B users"] --> Gateway

    subgraph GroupA["Group A namespace"]
        ServeA["RayService A<br/>models for Group A"]
        ClusterA["User RayClusters"]
        JobA["Ephemeral RayJobs"]
    end

    subgraph GroupB["Group B namespace"]
        ServeB["RayService B<br/>models for Group B"]
        ClusterB["User RayClusters"]
        JobB["Ephemeral RayJobs"]
    end

    Gateway -->|authorized Group A model requests| ServeA
    Gateway -->|authorized Group B model requests| ServeB
    Gateway -->|create and manage| ClusterA
    Gateway -->|create and manage| JobA
    Gateway -->|create and manage| ClusterB
    Gateway -->|create and manage| JobB

    Boundary["Namespaces, quotas and NetworkPolicy<br/>block cross-group access"]
    Boundary -.-> GroupA
    Boundary -.-> GroupB

    Nodes["Shared Kubernetes node pools"]
    ServeA -.-> Nodes
    ClusterA -.-> Nodes
    JobA -.-> Nodes
    ServeB -.-> Nodes
    ClusterB -.-> Nodes
    JobB -.-> Nodes

Loading

There is one Ray gateway for the platform, not one gateway per group. It derives the group from the authenticated identity, routes model requests to that group's RayService, and creates lifecycle resources inside the correct group namespace. Groups have separate RayServices, RayClusters, RayJobs, quotas and network rules, but their pods may still run on the same Kubernetes nodes.

The group is the boundary for model serving, permissions and quotas. Interactive compute is isolated further by giving each user or notebook session its own Ray cluster.

Desired user experience

We want the Ray pack to support these simple workflows:

• Each trusted group has one shared RayService for the models used by that group.
• Every model request is authenticated, and the caller must belong to the group that owns the RayService.
• Direct access to the internal Ray Serve Service is blocked so callers cannot bypass group authorization.
• A group's serving environment is separate from notebook work and UI jobs, so user compute cannot shut it down or consume its Ray resources.
• UI requests always create a RayJob. The job receives an ephemeral Ray cluster and that cluster is removed after the job finishes.
• A JupyterLab user can request their own RayCluster from a gateway and receive a compatible Ray Client address.
• Different users do not share a Ray head, workers, actors or object store unless they explicitly choose a shared cluster in the future.
• Users can list and stop their own clusters. Idle clusters are cleaned up safely.
• Group administrators can control which profiles, images, CPU, memory, GPU and maximum worker counts are allowed.
• Users choose approved options rather than submitting arbitrary Kubernetes manifests or service accounts.
• Users cannot connect to another group's Ray Client, dashboard, Jobs API or GCS endpoint.
• The gateway records ownership and can recover the state of existing RayCluster and RayJob resources after restarting.

This gives us three distinct KubeRay lifecycle patterns:

Resource	Purpose	Lifecycle
RayService	Shared models for one group	Long-running and GitOps-managed
RayCluster	One user's interactive Ray Client session	Created and deleted on demand
RayJob	One durable UI job	Creates an ephemeral cluster and cleans it up

Authentication and group access

The platform gateway authenticates every user and uses group membership to decide what they can access. Users can call models, submit jobs, create clusters, and view resources only within their groups. Internal Ray endpoints are not exposed directly.

The dashboard for a shared group RayService is an operational interface, so it is available only to platform administrators and approved group operators. Ordinary users call group models through the deployed application UI. If users need model health or status, expose a limited group status view rather than the full Ray dashboard.

Users may access the dashboard for their own RayCluster because they own the workloads running there.

---

The following section was removed after discussion as it was premature.

## Evolving the current pack

The current repository can evolve into a more general Ray platform pack rather than creating unrelated implementations. A future name such as nebari-ray-pack would better reflect the wider scope, but existing chart and resource names should remain compatible during migration.

A possible structure is:

<PROJECT_LAYOUT_REMOVED>

RayService, RayCluster and RayJob all need many of the same cluster settings. The pack should define those settings once and reuse them, with different profiles for serving, interactive CPU, interactive GPU and UI jobs.

The platform part of the pack would provide the shared Ray services and one gateway. The group part would create the group's model-serving environment and access boundaries. When a user asks for notebook compute or submits a UI job, the gateway would create the right Ray resource from an approved profile and clean it up when it is no longer needed.

KubeRay would continue to manage the Ray resources underneath. This approach lets the current pack grow to support groups and user-owned compute without creating separate solutions that can drift apart.

[kcpevey]

Users may access the dashboard for their own RayCluster because they own the workloads running there.

How does a user access this? The equivalent of this in dask is a juptyerlab extension.

Just to note:

• groups are derived from keycloak groups.
• users can specify the server that the cluster is running on (within their accessible servers)

[brandonrc]

What Mobula covers (the dynamic layer)

• Cluster lifecycle: create/delete/suspend/resume RayClusters on demand, TTL/idle cleanup, restart recovery
• Identity: SSO in front of Ray, per-user RBAC, holds the cluster tokens, audit trail
• Self-service guardrails: pod-shaping catalog (what you can mount/run as), profiles + option schemas (#74), quotas
• Capacity: Kueue pools via API — allocate/resize compute without touching GitOps
• Access: authenticated proxy to Jobs API, dashboards, logs
• Planned: ephemeral RayJobs (#50), Ray Client access for notebooks

What rayserve-pack covers (the static layer)

• KubeRay operator install
• Group namespaces, NetworkPolicies, node pools
• The long-lived per-group RayService (model serving baseline)
• All GitOps-managed, changes on deploy cadence

The overlap (where #34 collides with Mobula)

#34 proposes building a gateway inside the pack that does auth, routing, lifecycle, profiles, and ownership tracking. That's Mobula's entire feature list. Everything in #34 except the RayService/namespace/operator parts is duplicate work.

The clean split

Pack = the arena. Mobula = everything that moves. #34's own lifecycle table already says this: RayService is "long-running, GitOps-managed" (pack), RayCluster/RayJob are "on demand" (Mobula).

[brandonrc]

1. The gateway lives in Mobula, not in a gateway/ dir in the pack
2. Who writes the group RayService — pack via GitOps (Mobula just authorizes routing to it), or Mobula's services API. Pick one writer.
3. Quota authority is Kueue/Mobula — pack shouldn't ship namespace ResourceQuotas as working limits or the two systems fight

[nenb]

On reflection, I should not have added the section 'Evolving the current pack' in the original message. It was premature and unnecessarily focused on implementation details. I am removing it now so that we can focus instead on the schematics/topology and see if we can find consensus.

I think we need some kind of consensus on these schematics before we proceed to the implementation details, otherwise we will all be targeting different use-cases.

@brandonrc What are your thoughts on the single and multiple groups schematics above? Do you feel that they need to be changed or that they don't correctly reflect our proposed userbase? (Once we share an understanding of what we need to build, then I think it makes sense to get more into the details of Mobula. It's tricky for me to evaluate it until I understand what we need to build.)

@kcpevey

How does a user access this? The equivalent of this in dask is a juptyerlab extension.

At the moment, this is just a desired user experience ie it does not exist 😄 But I think a jupyterlab extension would make sense. I would need to look into the Dask method for doing this. The user basically needs to request the dashboard URL from the cluster that they create (the gateway will need to provide a way to pass this information on) and then display it in the notebook. I see no obvious blocker to this right now, but it does not yet exist is the most important point.

Just to note:
groups are derived from keycloak groups.
users can specify the server that the cluster is running on (within their accessible servers)

👍