diff --git a/community/04-Proposals/MEP4/README.md b/community/04-Proposals/MEP4/README.md index 389a02d4..84c0c982 100644 --- a/community/04-Proposals/MEP4/README.md +++ b/community/04-Proposals/MEP4/README.md @@ -5,13 +5,10 @@ sidebar_position: 4 --- # Multi-Tenancy for the metal-api -:::info -This document is work in progress. -::: -In the past we decided to treat the metal-api as a "low-level API", i.e. the API does not specifically deal with projects and tenants. A user with editor access can for example assign machines to every project he desires, he can see all the machines available and can control them. We tried to keep the metal-api code base as small as possible and we added resource scoping to a "higher-level APIs". From there, a user would be able to only see his own clusters and IP addresses. +In the past we decided to treat the metal-api as a "low-level API", i.e. the API does not specifically deal with projects and tenants. A user with editor access can for example assign machines to every project they desire, they can see all the machines available and can control them. We tried to keep the metal-api code base as small as possible and we added resource scoping to "higher-level" APIs. From there, a user would be able to only see their own machines, clusters and IP addresses. -As time passed metal-stack has become an open-source project and people are willing to adopt. Adopters who want to put their own technologies on top of the metal-stack infrastructure don't have those "higher-level APIs" that we implemented closed-source for our user base. So, external adopters most likely need to implement resource scoping on their own. +Over time, metal-stack has become an open-source project that people are willing to adopt. Adopters who want to put their own technologies on top of the metal-stack infrastructure don't have those "higher-level" APIs that we implemented closed-source for our user base. So, external adopters most likely need to implement resource scoping on their own. Introducing multi-tenancy to the metal-api is a serious chance of making our product better and more successful as it opens the door for: @@ -25,82 +22,80 @@ Introducing multi-tenancy to the metal-api is a serious chance of making our pro These are some general requirements / higher objectives that MEP-4 has to fulfill. -- Should be able to run with mini-lab without requiring to setup complex auth backends (dex, LDAP, keycloak, ...) +- Should be able to run with mini-lab - Simple to start with, more complex options for production setups - Fine-grained access permissions (every endpoint maps to a permission) - Tenant scoping (disallow resource access to resources of other tenants) - Project scoping (disallow resource access to resources of other projects) -- Access tokens in self-service for technical user access +- Access tokens in self-service for technical API access ## Implementation -We gathered a lot of knowledge while implementing a multi-tenancy-capable backend for metalstack.cloud. The goal is now to use the same technology and adopt that to the metal-api, this includes: +We gathered a lot of knowledge while implementing a multi-tenancy-capable backend for metalstack.cloud. The goal is now to use the same technology and adopt that to the new API. This includes: - gRPC in combination with connectrpc -- OPA for making auth decisions +- Authz through an authentication middleware - REST HTTP only for OIDC login flows -### API Definitions +### API V2 Definitions -The API definitions should be located on a separate Github repository separate from the server implementation. The proposed repository location is: https://github.com/metal-stack/api. +We define a new V2 API for metal-stack. The API definitions are located on a separate Github repository separate from the server implementation. The proposed repository location is: https://github.com/metal-stack/api. -This repository contains the `proto3` specification of the exposed metal-stack api. This includes the messages, simple validations, services and the access permission to these services. The input parameters for the authorization in the backend are generated from the `proto3` annotations. +This repository contains the `proto3` specification of the V2 API. This includes the messages, simple validations, services and the access permission to these services. The input parameters for the authorization in the backend are generated from the `proto3` annotations. -Client implementations for the most relevant languages (go, python) are generated automatically. +Client implementations for the most relevant languages (go, python, typescript) are generated automatically. -This api is divided into end-user and admin access at the top level. The proposed APIs are: +The new API is divided into dedicated top-level scopes. The proposed APIs are: - `metalstack.api.v2`: For end-user facing services -- `metalstack.admin.v2`: For operators and controllers which need access to unscoped entities +- `metalstack.admin.v2`: For operators which need access to unscoped entities +- `metalstack.infra.v2`: For controllers which need access to unscoped entities The methods of the API can have different role scopes (and can be narrowed down further with fine-grained method permissions): - `tenant`: Tenant-scoped methods, e.g. project creation (tenant needs to be provided in the request payload) - Available roles: VIEWER, EDITOR, OWNER -- `project`: Project-scoped methods, e.g. machine creation (tenant needs to be provided in the request payload) +- `project`: Project-scoped methods, e.g. machine creation (project needs to be provided in the request payload) - Available roles: VIEWER, EDITOR, OWNER -- `admin` Admin-scoped methods, e.g. unscoped tenant list or switch register +- `admin` Admin-scoped methods, e.g. unscoped tenant list or switch register (user must be member of the provider tenant) - Available roles: VIEWER, EDITOR -And has methods with different visibility scopes: +Alternatively, methods can also have more unusual scopes: - `self`: Methods that only the logged in user can access, e.g. show permissions with the presented token - `public`: Methods that do not require any specific authorization +- `infra`: Required for accessing infra methods (for controllers) + - Available roles: VIEWER; EDITOR +- `machine`: Required for accessing infra machine methods (right now exclusively for metal-hammer in order to narrow down the scope to a minimum) + - Available roles: VIEWER; EDITOR -### API +### API Server Implementation -The API server implements the services defined in the API and validates access to a method using OPA with the JWT tokens passed in the requests. The server is implemented using the connectrpc.com framework. +The server implementation is put into a dedicated github repo and implements the services defined in the `api` repository. It opens a `https` endpoints where the grpc (via connectrpc.com) and oidc services are exposed. -The API server implements the login flow through OIDC. After successful authentication, the API server derives user permissions from the OIDC provider and issues a new JWT token which is passed on to the user. The tokens including the permissions are stored in a redis compatible backend. +The API server validates access to a method with the JWT tokens passed in the requests. -With these tokens, users can create Access Tokens for CI/CD or other use cases. - -JWT Tokens can be revoked by admins and the user itself. +The API server implements the login flow through OIDC. After successful authentication, the API server derives user permissions from the OIDC provider and issues a new JWT token which is passed on to the user. The tokens including the permissions are stored in a redis compatible backend. In addition, permissions can be derived from project / tenant memberships in order to be able to immediately revoke existing tokens. -### API Server +With these tokens, users can create scoped Access Tokens for CI/CD or other use cases. -Is put into a new github repo which implements the services defined in the `api` repository. It opens a `https` endpoints where the grpc (via connectrpc.com) and oidc services are exposed. +JWT Tokens can be revoked by admins and the user itself. ### Migration of the Consumers -To allow consumers to migrate to the `v2` API gradually, both apis, the new and the old, are deployed in parallel. In the control-plane both apis are deployed side-by-side behind the ingress. `api.example.com` is forwarded to `metal-api` and `metal.example.com` is forwarded to the new `metal-apiserver`. +To allow consumers to migrate to the `v2` API gradually, both apis, the new and the old, are deployed in parallel. In the control-plane both apis are deployed side-by-side behind the traffic ingress. `api.example.com` is forwarded to `metal-api` and `metal.example.com` is forwarded to the new `metal-apiserver`. -The api-server will talk to the existing metal-api during the process of migration services away to the new grpc api. +The new V2 metal-apiserver reimplements the endpoints that were already present in the V1 API. This gives us a chance to re-iterate existing implementations. The metal-apiserver has no connection to the metal-api. The migration process can be done in the following manner: -for each resource in the metal-api: +For each resource in the metal-api: -- create a new proto3 based definition in the `api` repo. -- implement the business logic per service in the new `metal-apiserver` without calling the metal-api. -- clients must be able to talk to `v1` and `v2` backend in parallel +- Create a new proto3 based definition in the `api` repo. +- Implement the business logic per service in the new `metal-apiserver`. +- Clients must be able to talk to `v1` and `v2` backend and have a transition period for their migration. - Deprecate the already migrated service in the swagger route to notify the client that this route should not be used anymore. -- identify all consumers of this resource and replace them to use the grpc instead of the rest api -- move the business logic incl. the backend calls to ipam, metal-db, masterdata-api, nsq for this resource from the metal-api to the `metal-apiserver` - -We will migrate the rethinkdb backend implementation to a generic approach during this effort. - -- Try to enhance the generic rethinkdb interface with `project` scoped methods. +- Identify all consumers of this resource and replace them to use the grpc instead of the rest api. There are a lot of consumers of metal-api, which need to be migrated: @@ -179,6 +174,7 @@ Requirements: Project was created, permissions are present :::warning A user **cannot** list all allocated machines for all projects. The user **must** always switch project context first and can only view the machines inside this project. Only admins can see all machines at once. ::: + ### Scopes for Resources The admins / operators of the metal-stack should be able to provide _global_ resources that users are able to use along with their own resources. In particular, users can view and use _global_ resources, but they are not allowed to create, modify or delete them. @@ -189,23 +185,22 @@ When a project ID field is empty on a resource, the resource is considered _glob Where possible, users should be capable of creating their own resource entities. -| Resource | User | Global | -| :----------------- | :--- | :----- | -| File System Layout | yes | yes | -| Firewall | yes | | -| Firmware | | yes | -| OS Image | | yes | -| Machine | yes | | -| Network (Base) | | yes | -| Network (Children) | yes | | -| IP | yes | | -| Partition | | yes | -| Project | yes | | -| Project Token | yes | | -| Size | | yes | -| Switch | | | -| Tenant | | yes | +| Resource | User | Global | Infra | +| :---------------------------------- | :--- | :----- | ----- | +| File System Layout | yes* | yes | | +| Firewall | yes | | | +| OS Image | yes* | yes | | +| Machine | yes | | | +| Network (Super, external, underlay) | | yes | | +| Network (Children) | yes | | | +| IP | yes | | | +| Partition | | yes | | +| Project | yes | | | +| Token | yes | | | +| Size | | yes | | +| Switch | | | yes | +| Tenant | | yes | | :::info -Example: A user can make use of the file system layouts provided by the admins, but can also create own layouts. Same applies for images. As soon as a user creates own resources, the user takes over the responsibility for the machine provisioning to succeed. +(*) This is only a long-term goal and must not be part of the initial V2 server implementation. Example: A user can make use of the file system layouts provided by the admins, but can also create own layouts. Same applies for images. As soon as a user creates own resources, the user takes over the responsibility for the machine provisioning to succeed. ::: diff --git a/docs/02-General/04-flavors-of-metalstack.md b/docs/02-General/04-flavors-of-metalstack.md index 18c0671f..faa034ca 100644 --- a/docs/02-General/04-flavors-of-metalstack.md +++ b/docs/02-General/04-flavors-of-metalstack.md @@ -14,7 +14,7 @@ You can consume it as-is with our [Plain Flavor](#plain) or use it as foundation All flavors start with this. This is what you get if you set up metal-stack and stop there. -Using plain metal-stack without additional layer was not a focus in the past. Therefore firewall features and role management are quite basic. There is ongoing work on [improved RBAC in MEP-4](/community/MEP-4-multi-tenancy-for-the-metal-api) and [firewall configuration via metal-api in MEP-16](/community/MEP-16-metal-api-as-an-alternative-configuration-source-for-the-firewall-controller). +Using plain metal-stack without additional layer was not a focus in the past. Therefore firewall features and role management are quite basic. There is ongoing work on [improved RBAC in MEP-4](/community/04-Proposals/MEP4/README.md) and [firewall configuration via metal-api in MEP-16](/community/MEP-16-metal-api-as-an-alternative-configuration-source-for-the-firewall-controller). If you want more features, keep reading. @@ -22,7 +22,7 @@ If you want more features, keep reading. [Gardener](https://gardener.cloud/) is an open-source managed Kubernetes service. It provides a good "batteries-included" developer experience and should be your first choice for a Kubernetes-as-a-service solution. -Gardener is vendor agnostic and can be used with a wide selection of infrastructure providers. One big advantage are its containerized control planes. These allow for control planes to not require three machines for each managed cluster, called `Shoot`. This makes operating many smaller clusters more economical, compared to bare-metal control planes. +Gardener is vendor agnostic and can be used with a wide selection of infrastructure providers. One big advantage are its containerized control planes. These allow for control planes to not require three machines for each managed cluster, called `Shoot`. This makes operating many smaller clusters more economical, compared to bare-metal control planes. We provide support to run Gardener on metal-stack via [Gardener extensions](../05-Concepts/04-Kubernetes/01-gardener.md). This integration is production-hardened, well documented, used by many organizations in production and build on top of the open-source project [Gardener](https://gardener.cloud/). @@ -30,4 +30,4 @@ We provide support to run Gardener on metal-stack via [Gardener extensions](../0 Our [Cluster API integration](../05-Concepts/04-Kubernetes/02-cluster-api.md) is a more verbose approach to provide Kubernetes clusters with metal-stack. Our implementation is still in early development. It is based on the [Cluster API](https://cluster-api.sigs.k8s.io/) project. -Configuring Cluster API is very verbose. It requires additional tooling to provide a developer experience. Cluster API will give you building blocks to build a Kubernetes-as-a-service platform on top of it, but no more. We do not recommend you use Cluster API, unless you already have a large platform engineering team, that is very experienced in bare-metal K8s operations and they agree that your specific requirements cannot be modelled with Gardener. In any other case, you will have more success with Gardener. \ No newline at end of file +Configuring Cluster API is very verbose. It requires additional tooling to provide a developer experience. Cluster API will give you building blocks to build a Kubernetes-as-a-service platform on top of it, but no more. We do not recommend you use Cluster API, unless you already have a large platform engineering team, that is very experienced in bare-metal K8s operations and they agree that your specific requirements cannot be modelled with Gardener. In any other case, you will have more success with Gardener. diff --git a/docs/04-For Operators/07-migrating-to-gatewayapi.md b/docs/04-For Operators/07-migrating-to-gatewayapi copy.md similarity index 100% rename from docs/04-For Operators/07-migrating-to-gatewayapi.md rename to docs/04-For Operators/07-migrating-to-gatewayapi copy.md diff --git a/docs/04-For Operators/08-v2-api-guide.md b/docs/04-For Operators/08-v2-api-guide.md new file mode 100644 index 00000000..6786f984 --- /dev/null +++ b/docs/04-For Operators/08-v2-api-guide.md @@ -0,0 +1,225 @@ +--- +slug: /v2-api-guide +title: Operator guide for the metal-stack V2 API +sidebar_position: 8 +--- + +# The metal-stack V2 API + +With [MEP-4](/community/04-Proposals/MEP4/README.md), we defined a new metal-stack V2 API definition based on proto. + +This guide is meant for operators to become familiar with the concepts of the new V2 API. This includes design pattern how V2 services are getting deployed and how to initially bootstrap the solution. + +With the V2 API, the `metal-apiserver` derives the effective permissions of a request from the presented token and the memberships stored in the database. The following sections explain the resulting permission model, how operators get admin access, and how the components in the infrastructure are bootstrapped with infra tokens. All examples use [metalctlv2](https://github.com/metal-stack/cli), the CLI for the V2 API. + +:::info +This document primarily describes how to work with the V2 API and not how to deploy it. An example for a working V2 deployment can be found in the [mini-lab](https://github.com/metal-stack/mini-lab). +::: + +## Basic Permission Principles + +The V2 API defines an own permission model that can be used to create fine-grained API tokens that allow only minimal privileges to server methods for consumers. In this section we will go through the basic idea behind the permission model and explain the relevant terms. + +The V2 API server implements the login flow through OIDC. We recommend [Zitadel](https://zitadel.com/) as the OIDC provider, which can be federated with a company-wide OIDC, so that operators log in with their company identity. After successful authentication the API server derives the user's permissions and issues its own JWT token for the user. + +```bash +$ metalctlv2 login +Starting server at http://127.0.0.1:44351... +✔ login successful! Updated and activated context "demo" +``` + +The login stores the issued user token in the CLI context, which can be named with the `--context` flag. + +Note that the resulting token contains only very minimal information (not the permissions itself). The effective permissions are stored in the backend only. + +### User Permission Scopes + +For regular users, the V2 API defines the following three scopes: `Tenant`, `Project` and `Self`. + +**Tenants** are the top-level scoping entity and represent a user or an organization on the platform (or maybe we can just call it a "namespace" for resources). A *default tenant* is created automatically for every user after the first successful login: the OIDC callback creates a tenant with the user's login as its ID and adds the user as an owner. Because every user owns their default tenant, every user is able to create user-scoped resources such as projects immediately after login, without any operator intervention. A user cannot leave or delete their own default tenant. Tenant-scoped service methods require the `login` field in their request payload. + +**Projects** group the resources belonging to a tenant. A tenant can own many projects. Projects are the primary scope for resources such as machines, IPs and child networks, which is why project-scoped service methods require the `project` field in their request payload. Before acting on a project, select it as the default project for the CLI with `metalctlv2 context set-project ` (this value can be overwritten if the `--project` flag is explicitly provided). + +The **Self** scope provides users access to server methods that every valid token owner can access. These methods are for example managing their own tokens, listing global resources such as partitions, images and sizes or other basic methods like listing the permissions currently hold by the token. + +### Memberships + +Project and tenant memberships are how users gain permissions on project and tenant scopes. They can be established by using **invites**. Invites are used to onboard a user to a tenant or project. A member with sufficient permissions creates an invite that contains a secret and the role the user will hold after accepting it. The invited user accepts the invite to gain the membership. Pending invites can be listed and deleted, and every invite expires after a defined time. + +As an alternative to invites, it is also possible to declare memberships statically such that they can be applied through automation. This functionality is provided by the `AddMember` methods for tenants and projects. For this, it is necessary that the users were created already (either through deployment automation or through login). + +Each membership associates a user with a role, either on a tenant (`OWNER`, `EDITOR`, `VIEWER`, `GUEST`) or on a project (`OWNER`, `EDITOR`, `VIEWER`). + +Tenant memberships inherit permissions on projects. For example, a tenant `EDITOR` is automatically the `EDITOR` for all projects of this tenant. With project invites it is possible to give a user only permissions on a particular project of a tenant. We call this _direct project membership_. A direct project membership creates an implicit `GUEST` membership in the tenant. A tenant `GUEST` has a minimum amount of permissions on a tenant. + +```bash +$ metalctlv2 project invite generate-join-secret +You can share this secret with the member to join, it expires in 6 days from now: + + + +$ metalctlv2 project invite list +SECRET PROJECT ROLE EXPIRES IN + 11111111-1111-1111-1111-111111111111 PROJECT_ROLE_VIEWER 6 days from now + +$ metalctlv2 project invite delete + +# the invited user accepts with +$ metalctlv2 project join +✔ successfully joined project "example-project" +``` + +Memberships can also be removed again, however, a default tenant user cannot leave their own tenant namespace and a tenant must always have at least one owner (to prevent orphanage). + +### Tokens Types and Validation + +Tokens authenticate requests to the API. There are two distinct kinds of tokens: **user** tokens and **API** tokens. + +User tokens are created automatically after a successful login. A user token contains no explicit permissions or roles. Instead, the `metal-apiserver` expands its effective permissions on every request from the user's current project and tenant memberships in the database. As a consequence, removing a membership takes effect immediately, even for tokens that were issued earlier. + +Scoped API tokens are created by users or services for CI/CD and other technical use cases. An API token references explicit project and tenant roles, optional admin, infra and machine roles, and individual method permissions. For access checks, the roles and individual permissions are flattened into a set of method permissions. The server refuses to create or update a token that would grant more than the calling token and the user's memberships already allow, so a token cannot be used to elevate privileges. API tokens are always validated against the effective user permission, too, such that requests are declined in case the token holds more permissions than the user has at the point of the request. + +:::info +Note that roles internally are flattened to method permissions for request validation. In other words, roles are grouping method permissions. The current API defines roles statically directly in the API. It is currently not possible to dynamically create own roles. Dynamically composable roles could be implemented in the future if there is demand for this functionality. +::: + +The API server validates the token's signature and expiration against its signing keys, and then looks the token up in the token store by subject and JWT id. If the token is no longer present, it has been revoked or deleted and the request is rejected as unauthenticated. This way a token can be revoked at any time and stops working immediately, regardless of its remaining JWT lifetime. + +Rate limiting is applied to every request by an interceptor. Authenticated requests are limited per token and unauthenticated requests per client IP address. If a client exceeds the configured limit, the request is rejected with a `RESOURCE_EXHAUSTED` error. Please make sure in your deployment that the real client IP addresses are passed to the metal-apiserver as otherwise unauthenticated requests cannot be rate-limited properly. + +```bash +$ metalctlv2 token create \ + --description "Project scoped" \ + --project-roles 11111111-1111-1111-1111-111111111111=PROJECT_ROLE_EDITOR \ + --expires 8h +Make sure to copy your personal access token now as you will not be able to see this again. + +eyJhbGciOiJFUzUxMiIs... + +uuid: 22222222-2222-2222-2222-222222222222 +user: user@example.com@openid-connect +description: Project scoped +expires: "2026-09-28T19:08:15.175078586Z" +issuedAt: "2026-09-28T11:08:15.175078586Z" +tokenType: TOKEN_TYPE_API +projectRoles: + 11111111-1111-1111-1111-111111111111: PROJECT_ROLE_EDITOR + +$ metalctlv2 token revoke 22222222-2222-2222-2222-222222222222 +uuid: 22222222-2222-2222-2222-222222222222 +``` + +`revoke` is an alias of `token delete`. The token stops working immediately afterwards. + +## Admin and Infra API + +The V2 API defines additional scopes that are not intended for regular users. These are the `Admin`, `Infra` and `Machine` API. + +On startup, the metal-apiserver creates a so-called _provider tenant_. This tenant is special because every member of this tenant is allowed to gain permissions for the admin API. The `metal-apiserver` (respectively its deployment) creates an admin API token for the provider tenant and writes its secret back into a Kubernetes secret in the control-plane namespace. + +```bash +$ kubectl get secret -n metal-control-plane metal-apiserver-admin-token -o yaml +apiVersion: v1 +data: + admin_editor_token: + admin_viewer_token: +kind: Secret +metadata: + labels: + app: metal-apiserver + name: metal-apiserver-admin-token + namespace: metal-control-plane +type: Opaque +``` + +This token has the highest privileges possible and should in general not be used by operators. Its purpose is to provide emergency access and allow deployment bootstrapping. It is rotated automatically every eight hours. Another reason why operators should not use this token is that it will not audit their actual username, which is usually undesirable. + +Platform operators can become tenant members of the provider tenant after their first login (creates the user) and then by adding the provider tenant membership `metalctlv2 admin tenant add-member` (e.g. by another provider tenant user or with the provider tenant secret through deployment). The role the user holds in the provider tenant is mapped to an admin role: an `OWNER` maps to the admin `EDITOR` role, and an `EDITOR` or `VIEWER` maps to the admin `VIEWER` role. + +```bash +$ metalctlv2 tenant member list --tenant metal-stack +ID ROLE SINCE +operator-a@example.com@openid-connect TENANT_ROLE_OWNER 7 months ago +operator-b@example.com@openid-connect TENANT_ROLE_OWNER 2 months ago +metal-stack TENANT_ROLE_OWNER 7 months ago + +$ metalctlv2 admin tenant add-member \ + --tenant-id metal-stack \ + --member-id operator-c@example.com@openid-connect \ + --role TENANT_ROLE_OWNER +``` + +The provider tenant is named `metal-stack` by default and can be configured on initial deployment. Operators can issue an admin token for their own context with the hidden `--admin-role` flag of the login command. The CLI then creates a short-lived admin API token from the login token and stores it in the context. + +```bash +$ metalctlv2 login --admin-role ADMIN_ROLE_EDITOR +``` + +### Infra Components + +As metal-stack consists of microservices, these individual services require API tokens. The components of the infrastructure do not use user credentials but their own dedicated tenant tokens, which are bootstrapped with a deployment token. + +The deployment (Ansible) uses the provider tenant token from the Kubernetes secret (for example through the `metal-deployment-token` role in metal-roles). Note that the deployment admin token is only used to bootstrap the environment and is capable of creating tokens **for other tenants**. The deployment creates dedicated tenants per service for which it issues individual API tokens. Each of these sub-tokens only has those roles and permissions that the component actually needs, for example `metal-core`, `pixiecore`, `metal-bmc`, or the `metal-hammer`. The sub-token is written to the target host's filesystem with restrictive permissions and used by the component from there, following the principle of least privilege. + +```bash +$ metalctlv2 admin token describe 33333333-3333-3333-3333-333333333333 +uuid: 33333333-3333-3333-3333-333333333333 +user: metal-core-user +meta: + labels: + labels: + ci.metal-stack.io/id: metal-core-a-r01leaf01 + ci.metal-stack.io/manager: ansible +description: metal-core token r01leaf01 in partition a +permissions: + - methods: + - /metalstack.api.v2.TokenService/Refresh + - methods: + - /metalstack.infra.v2.ComponentService/Ping + - /metalstack.infra.v2.EventService/Send + - /metalstack.infra.v2.SwitchService/Get + - /metalstack.infra.v2.SwitchService/Heartbeat + - /metalstack.infra.v2.SwitchService/Register +expires: "2026-09-28T19:08:15.175078586Z" +issuedAt: "2026-09-28T11:08:15.175078586Z" +tokenType: TOKEN_TYPE_API +``` + +The example shows a token created by the deployment for `metal-core` using minimal permissions. The `user` points to a tenant specifically created for the `metal-core` service. It has the `/metalstack.api.v2.TokenService/Refresh` method permission allowing the service to rotate the token automatically. + +:::info +For deployments inside the partition it might not be desired that the partition runner has access to the control plane Kubernetes cluster. In this case, we recommend issuing a long-lived provider tenant admin token (maximum is 365 days) and provide this in the partition deployment (e.g. through the `defaults_partition_metal_apiserver_admin_token` variable or the `METAL_APIV2_TOKEN` env variable). This way it is not necessary to read the secret from the Kubernetes cluster for deploying partition components. Currently, this token needs to be renewed manually. + +As every token is associated with the user who issued the token, the audit log will contain this user name, too. Hence, if you create a dedicated deployment token for a partition, you might consider issuing the token using provider tenant secret and not from your own user login. This avoids embedding your own user identity, which would then associate the deployment actions with your user and show up in the audit logs with your user name. This is one of the few use-cases where you would need the provider tenant secret because in general you always want an audit log to show a real user account. +::: + +Every service that talks to the API calls the `/metalstack.infra.v2.ComponentService/Ping` method periodically at a configurable interval. Each ping is stored and registers the component with its type, identifier, version, start time and the token it uses. This gives operators a global overview over which services are currently connected to the infrastructure API and which token each of them uses, which is available through the admin component endpoints. + +```bash +$ metalctlv2 admin component ls --type COMPONENT_TYPE_METAL_CORE +ID TYPE IDENTIFIER STARTED AGE VERSION TOKEN TOKEN EXPIRES IN +55555555-5555-5555-5555-555555555555 metal-core leaf01 7d 1h 2m 22s v0.20.0 33333333-3333-3333-3333-333333333333 2d 14h +66666666-6666-6666-6666-666666666666 metal-core leaf02 7d 1h 2m 40s v0.20.0 44444444-4444-4444-4444-444444444444 2d 14h +``` + +Components use the API client with token renewal enabled. On every request the client checks whether the token has passed three quarters of its lifetime, and once that is the case it calls the `/metalstack.api.v2.TokenService/Refresh` endpoint to obtain a new token with exactly the same permissions, roles and lifetime. The check happens lazily on the next request, which for a component is typically its periodic ping, so no separate rotation job is required. The refresh endpoint is what makes this possible, and the same mechanism is used by the [metal-token-refresher](https://github.com/metal-stack/metal-token-refresher) to keep tokens that are stored in Kubernetes secrets up to date. + +The API server also rotates its own token signing certificate automatically before it expires, so that rotation never interrupts running services. + +## Tasks API + +Non-atomic operations in the V2 API are wrapped in tasks. In the V1 implementation this was an opaque design implemented by NSQ. Now, in V2, the backend handles this functionality through [asynq](https://github.com/hibiken/asynq). A task has an idempotent handle function that can get retried in case of runtime execution failure. Examples for a task are: + +- Machine decommission (cleans up a machine and its associated resources, configures it back to a state where it can re-register again through the metal-hammer) +- Machine BMC command (not retried, waits for a [metal-bmc](https://github.com/metal-stack/metal-bmc) to pickup the requested BMC task and waits for the execution result) +- ... + +Tasks can be observed by operators through the admin API. Usually, a task id is returned in the response payload for operations that enqueue tasks. + +``` +❯ metalctlv2 admin task list +ID QUEUE WHEN TYPE STATE +01a08afd-80c2-7629-8f7c-e886c7f18961 default 18d 2h ago machine:delete completed +``` + +Describing a task shows the last error in case it occurred and the amount of retries that were necessary for the task to execute. Operators should inspect tasks in the state `archived` as these tasks were not able to be executed successfully. In these cases, a manual cleanup or investigation of the environment might be necessary. diff --git a/docs/06-For CISOs/Security/01-principles.md b/docs/06-For CISOs/Security/01-principles.md index 652053e0..992dc46f 100644 --- a/docs/06-For CISOs/Security/01-principles.md +++ b/docs/06-For CISOs/Security/01-principles.md @@ -15,7 +15,7 @@ The minimal need to know principle is a security concept that restricts access t ### RBAC :::info -As of now metal-stack does not implement fine-grained Role-Based Access Control (RBAC) within the `metal-api` but this is worked on in [MEP-4](/community/MEP-14-independence-from-external-sources). +As of now metal-stack does not implement fine-grained Role-Based Access Control (RBAC) within the `metal-api` but this is worked on in [MEP-4](/community/04-Proposals/MEP4/README.md). ::: As described in our [User Management](../../05-Concepts/02-user-management.md) concept the [metal-api](https://github.com/metal-stack/metal-api) currently offers three different user roles for authorization: diff --git a/docs/06-For CISOs/rbac.md b/docs/06-For CISOs/rbac.md index 617434aa..da9be52b 100644 --- a/docs/06-For CISOs/rbac.md +++ b/docs/06-For CISOs/rbac.md @@ -31,4 +31,4 @@ To ensure that internal components interact securely with the metal-api, metal-s Users can interact with the metal-api using [metalctl](https://github.com/metal-stack/metalctl), the command-line interface provided by metal-stack. Depending on the required operations, users should authenticate with the appropriate role to match their level of access. -As part of [MEP-4](/community/MEP-14-independence-from-external-sources), significant work is underway to introduce more fine-grained access control mechanisms within metal-stack, enhancing the precision and flexibility of permission management. +As part of [MEP-4](/community/04-Proposals/MEP4/README.md), significant work is underway to introduce more fine-grained access control mechanisms within metal-stack, enhancing the precision and flexibility of permission management. diff --git a/docs/06-For CISOs/remote-access.md b/docs/06-For CISOs/remote-access.md index 9e8a7cf4..f2e25d11 100644 --- a/docs/06-For CISOs/remote-access.md +++ b/docs/06-For CISOs/remote-access.md @@ -26,4 +26,4 @@ This approach uses the [`metal-console`](../08-References/Control%20Plane/metal- Both methods ensure secure and controlled access to machines without exposing them unnecessarily to the network, maintaining the integrity and safety of the infrastructure. -Connecting directly to a machine without a clear plan of action can have unintended consequences and negatively impact stability. For this reason, administrative privileges are required. This restriction ensures that only authorized personnel with the necessary expertise can perform actions that affect the underlying infrastructure. These principles will evolve with the introduction of [MEP-4](/community/MEP-14-independence-from-external-sources). +Connecting directly to a machine without a clear plan of action can have unintended consequences and negatively impact stability. For this reason, administrative privileges are required. This restriction ensures that only authorized personnel with the necessary expertise can perform actions that affect the underlying infrastructure. These principles will evolve with the introduction of [MEP-4](/community/04-Proposals/MEP4/README.md).