Gerrit91 · Oct 2, 2026 · Sep 28, 2026 · Sep 28, 2026 · Sep 29, 2026 · Oct 2, 2026
@@ -5,13 +5,10 @@ sidebar_position: 4
 ---
 
 # Multi-Tenancy for the metal-api
-:::info
-This document is work in progress.
-:::
 
-In the past we decided to treat the metal-api as a "low-level API", i.e. the API does not specifically deal with projects and tenants. A user with editor access can for example assign machines to every project he desires, he can see all the machines available and can control them. We tried to keep the metal-api code base as small as possible and we added resource scoping to a "higher-level APIs". From there, a user would be able to only see his own clusters and IP addresses.
+In the past we decided to treat the metal-api as a "low-level API", i.e. the API does not specifically deal with projects and tenants. A user with editor access can for example assign machines to every project they desire, they can see all the machines available and can control them. We tried to keep the metal-api code base as small as possible and we added resource scoping to "higher-level" APIs. From there, a user would be able to only see their own machines, clusters and IP addresses.
 
-As time passed metal-stack has become an open-source project and people are willing to adopt. Adopters who want to put their own technologies on top of the metal-stack infrastructure don't have those "higher-level APIs" that we implemented closed-source for our user base. So, external adopters most likely need to implement resource scoping on their own.
+Over time, metal-stack has become an open-source project that people are willing to adopt. Adopters who want to put their own technologies on top of the metal-stack infrastructure don't have those "higher-level" APIs that we implemented closed-source for our user base. So, external adopters most likely need to implement resource scoping on their own.
 
 Introducing multi-tenancy to the metal-api is a serious chance of making our product better and more successful as it opens the door for:
 
@@ -25,82 +22,80 @@ Introducing multi-tenancy to the metal-api is a serious chance of making our pro
 
 These are some general requirements / higher objectives that MEP-4 has to fulfill.
 
-- Should be able to run with mini-lab without requiring to setup complex auth backends (dex, LDAP, keycloak, ...)
+- Should be able to run with mini-lab
   - Simple to start with, more complex options for production setups
 - Fine-grained access permissions (every endpoint maps to a permission)
 - Tenant scoping (disallow resource access to resources of other tenants)
 - Project scoping (disallow resource access to resources of other projects)
-- Access tokens in self-service for technical user access
+- Access tokens in self-service for technical API access
 
 ## Implementation
 
-We gathered a lot of knowledge while implementing a multi-tenancy-capable backend for metalstack.cloud. The goal is now to use the same technology and adopt that to the metal-api, this includes:
+We gathered a lot of knowledge while implementing a multi-tenancy-capable backend for metalstack.cloud. The goal is now to use the same technology and adopt that to the new API. This includes:
 
 - gRPC in combination with connectrpc
-- OPA for making auth decisions
+- Authz through an authentication middleware
 - REST HTTP only for OIDC login flows
 
-### API Definitions
+### API V2 Definitions
 
-The API definitions should be located on a separate Github repository separate from the server implementation. The proposed repository location is: https://github.com/metal-stack/api.
+We define a new V2 API for metal-stack. The API definitions are located on a separate Github repository separate from the server implementation. The proposed repository location is: https://github.com/metal-stack/api.
 
-This repository contains the `proto3` specification of the exposed metal-stack api. This includes the messages, simple validations, services and the access permission to these services. The input parameters for the authorization in the backend are generated from the `proto3` annotations.
+This repository contains the `proto3` specification of the V2 API. This includes the messages, simple validations, services and the access permission to these services. The input parameters for the authorization in the backend are generated from the `proto3` annotations.
 
-Client implementations for the most relevant languages (go, python) are generated automatically.
+Client implementations for the most relevant languages (go, python, typescript) are generated automatically.
 
-This api is divided into end-user and admin access at the top level. The proposed APIs are:
+The new API is divided into dedicated top-level scopes. The proposed APIs are:
 
 - `metalstack.api.v2`: For end-user facing services
-- `metalstack.admin.v2`: For operators and controllers which need access to unscoped entities
+- `metalstack.admin.v2`: For operators which need access to unscoped entities
+- `metalstack.infra.v2`: For controllers which need access to unscoped entities
 
 The methods of the API can have different role scopes (and can be narrowed down further with fine-grained method permissions):
 
 - `tenant`: Tenant-scoped methods, e.g. project creation (tenant needs to be provided in the request payload)
   - Available roles: VIEWER, EDITOR, OWNER
-- `project`: Project-scoped methods, e.g. machine creation (tenant needs to be provided in the request payload)
+- `project`: Project-scoped methods, e.g. machine creation (project needs to be provided in the request payload)
   - Available roles: VIEWER, EDITOR, OWNER
-- `admin` Admin-scoped methods, e.g. unscoped tenant list or switch register
+- `admin` Admin-scoped methods, e.g. unscoped tenant list or switch register (user must be member of the provider tenant)
   - Available roles: VIEWER, EDITOR
 
-And has methods with different visibility scopes:
+Alternatively, methods can also have more unusual scopes:
 
 - `self`: Methods that only the logged in user can access, e.g. show permissions with the presented token
 - `public`: Methods that do not require any specific authorization
+- `infra`: Required for accessing infra methods (for controllers)
+  - Available roles: VIEWER; EDITOR
+- `machine`: Required for accessing infra machine methods (right now exclusively for metal-hammer in order to narrow down the scope to a minimum)
+  - Available roles: VIEWER; EDITOR
 
-### API
+### API Server Implementation
 
-The API server implements the services defined in the API and validates access to a method using OPA with the JWT tokens passed in the requests. The server is implemented using the connectrpc.com framework.
+The server implementation is put into a dedicated github repo and implements the services defined in the `api` repository. It opens a `https` endpoints where the grpc (via connectrpc.com) and oidc services are exposed.
 
-The API server implements the login flow through OIDC. After successful authentication, the API server derives user permissions from the OIDC provider and issues a new JWT token which is passed on to the user. The tokens including the permissions are stored in a redis compatible backend.
+The API server validates access to a method with the JWT tokens passed in the requests.
 
-With these tokens, users can create Access Tokens for CI/CD or other use cases.
-
-JWT Tokens can be revoked by admins and the user itself.
+The API server implements the login flow through OIDC. After successful authentication, the API server derives user permissions from the OIDC provider and issues a new JWT token which is passed on to the user. The tokens including the permissions are stored in a redis compatible backend. In addition, permissions can be derived from project / tenant memberships in order to be able to immediately revoke existing tokens.
 
-### API Server
+With these tokens, users can create scoped Access Tokens for CI/CD or other use cases.
 
-Is put into a new github repo which implements the services defined in the `api` repository. It opens a `https` endpoints where the grpc (via connectrpc.com) and oidc services are exposed.
+JWT Tokens can be revoked by admins and the user itself.
 
 ### Migration of the Consumers
 
-To allow consumers to migrate to the `v2` API gradually, both apis, the new and the old, are deployed in parallel. In the control-plane both apis are deployed side-by-side behind the ingress. `api.example.com` is forwarded to `metal-api` and `metal.example.com` is forwarded to the new `metal-apiserver`.
+To allow consumers to migrate to the `v2` API gradually, both apis, the new and the old, are deployed in parallel. In the control-plane both apis are deployed side-by-side behind the traffic ingress. `api.example.com` is forwarded to `metal-api` and `metal.example.com` is forwarded to the new `metal-apiserver`.
 
-The api-server will talk to the existing metal-api during the process of migration services away to the new grpc api.
+The new V2 metal-apiserver reimplements the endpoints that were already present in the V1 API. This gives us a chance to re-iterate existing implementations. The metal-apiserver has no connection to the metal-api.
 
 The migration process can be done in the following manner:
 
-for each resource in the metal-api:
+For each resource in the metal-api:
 
-- create a new proto3 based definition in the `api` repo.
-- implement the business logic per service in the new `metal-apiserver` without calling the metal-api.
-- clients must be able to talk to `v1` and `v2` backend in parallel
+- Create a new proto3 based definition in the `api` repo.
+- Implement the business logic per service in the new `metal-apiserver`.
+- Clients must be able to talk to `v1` and `v2` backend and have a transition period for their migration.
 - Deprecate the already migrated service in the swagger route to notify the client that this route should not be used anymore.
-- identify all consumers of this resource and replace them to use the grpc instead of the rest api
-- move the business logic incl. the backend calls to ipam, metal-db, masterdata-api, nsq for this resource from the metal-api to the `metal-apiserver`
-
-We will migrate the rethinkdb backend implementation to a generic approach during this effort.
-
-- Try to enhance the generic rethinkdb interface with `project` scoped methods.
+- Identify all consumers of this resource and replace them to use the grpc instead of the rest api.
 
 There are a lot of consumers of metal-api, which need to be migrated:
 
@@ -179,6 +174,7 @@ Requirements: Project was created, permissions are present
 :::warning
 A user **cannot** list all allocated machines for all projects. The user **must** always switch project context first and can only view the machines inside this project. Only admins can see all machines at once.
 :::
+
 ### Scopes for Resources
 
 The admins / operators of the metal-stack should be able to provide _global_ resources that users are able to use along with their own resources. In particular, users can view and use _global_ resources, but they are not allowed to create, modify or delete them.
@@ -189,23 +185,22 @@ When a project ID field is empty on a resource, the resource is considered _glob
 
 Where possible, users should be capable of creating their own resource entities.
 
-| Resource           | User | Global |
-| :----------------- | :--- | :----- |
-| File System Layout | yes  | yes    |
-| Firewall           | yes  |        |
-| Firmware           |      | yes    |
-| OS Image           |      | yes    |
-| Machine            | yes  |        |
-| Network (Base)     |      | yes    |
-| Network (Children) | yes  |        |
-| IP                 | yes  |        |
-| Partition          |      | yes    |
-| Project            | yes  |        |
-| Project Token      | yes  |        |
-| Size               |      | yes    |
-| Switch             |      |        |
-| Tenant             |      | yes    |
+| Resource                            | User | Global | Infra |
+| :---------------------------------- | :--- | :----- | ----- |
+| File System Layout                  | yes* | yes    |       |
+| Firewall                            | yes  |        |       |
+| OS Image                            | yes* | yes    |       |
+| Machine                             | yes  |        |       |
+| Network (Super, external, underlay) |      | yes    |       |
+| Network (Children)                  | yes  |        |       |
+| IP                                  | yes  |        |       |
+| Partition                           |      | yes    |       |
+| Project                             | yes  |        |       |
+| Token                               | yes  |        |       |
+| Size                                |      | yes    |       |
+| Switch                              |      |        | yes   |
+| Tenant                              |      | yes    |       |
 
 :::info
-Example: A user can make use of the file system layouts provided by the admins, but can also create own layouts. Same applies for images. As soon as a user creates own resources, the user takes over the responsibility for the machine provisioning to succeed.
+(*) This is only a long-term goal and must not be part of the initial V2 server implementation. Example: A user can make use of the file system layouts provided by the admins, but can also create own layouts. Same applies for images. As soon as a user creates own resources, the user takes over the responsibility for the machine provisioning to succeed.
 :::
@@ -14,20 +14,20 @@ You can consume it as-is with our [Plain Flavor](#plain) or use it as foundation
 
 All flavors start with this. This is what you get if you set up metal-stack and stop there.
 
-Using plain metal-stack without additional layer was not a focus in the past. Therefore firewall features and role management are quite basic. There is ongoing work on [improved RBAC in MEP-4](/community/MEP-4-multi-tenancy-for-the-metal-api) and [firewall configuration via metal-api in MEP-16](/community/MEP-16-metal-api-as-an-alternative-configuration-source-for-the-firewall-controller).
+Using plain metal-stack without additional layer was not a focus in the past. Therefore firewall features and role management are quite basic. There is ongoing work on [improved RBAC in MEP-4](/community/04-Proposals/MEP4/README.md) and [firewall configuration via metal-api in MEP-16](/community/MEP-16-metal-api-as-an-alternative-configuration-source-for-the-firewall-controller).
 
 If you want more features, keep reading.
 
 ## Gardener
 
 [Gardener](https://gardener.cloud/) is an open-source managed Kubernetes service. It provides a good "batteries-included" developer experience and should be your first choice for a Kubernetes-as-a-service solution.
 
-Gardener is vendor agnostic and can be used with a wide selection of infrastructure providers. One big advantage are its containerized control planes. These allow for control planes to not require three machines for each managed cluster, called `Shoot`. This makes operating many smaller clusters more economical, compared to bare-metal control planes. 
+Gardener is vendor agnostic and can be used with a wide selection of infrastructure providers. One big advantage are its containerized control planes. These allow for control planes to not require three machines for each managed cluster, called `Shoot`. This makes operating many smaller clusters more economical, compared to bare-metal control planes.
 
 We provide support to run Gardener on metal-stack via [Gardener extensions](../05-Concepts/04-Kubernetes/01-gardener.md). This integration is production-hardened, well documented, used by many organizations in production and build on top of the open-source project [Gardener](https://gardener.cloud/).
 
 ## Cluster API
 
 Our [Cluster API integration](../05-Concepts/04-Kubernetes/02-cluster-api.md) is a more verbose approach to provide Kubernetes clusters with metal-stack. Our implementation is still in early development. It is based on the [Cluster API](https://cluster-api.sigs.k8s.io/) project.
 
-Configuring Cluster API is very verbose. It requires additional tooling to provide a developer experience. Cluster API will give you building blocks to build a Kubernetes-as-a-service platform on top of it, but no more. We do not recommend you use Cluster API, unless you already have a large platform engineering team, that is very experienced in bare-metal K8s operations and they agree that your specific requirements cannot be modelled with Gardener. In any other case, you will have more success with Gardener.
+Configuring Cluster API is very verbose. It requires additional tooling to provide a developer experience. Cluster API will give you building blocks to build a Kubernetes-as-a-service platform on top of it, but no more. We do not recommend you use Cluster API, unless you already have a large platform engineering team, that is very experienced in bare-metal K8s operations and they agree that your specific requirements cannot be modelled with Gardener. In any other case, you will have more success with Gardener.