Repository navigation
docs, examples: OpenShell and Substrate on one mesh, on your Kubernetes or on GKE - #747
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces Kubernetes examples and guides for integrating NVIDIA OpenShell sandboxes and Agent Substrate actors onto a single Agent Mesh, including support for GKE and Cloud Run deployments. The review feedback highlights a bug in the bind.py script where usage instructions are not printed due to missing docstrings, and security issues in both Makefiles where sensitive tokens are generated with insecure, world-readable permissions (umask 022) instead of owner-only permissions (umask 077).
| #!/usr/bin/env python3 | ||
| # Adds a sandbox's SPIFFE identity to the sandbox role in policy.json. | ||
| # | ||
| # bind.py <member> e.g. user:spiffe://openshell.local/openshell/sandbox/<id> | ||
| # bind.py --remove <member> |
There was a problem hiding this comment.
The script uses sys.exit(__doc__) to print usage instructions on invalid arguments. However, the usage instructions at the top of the file are written as comments (#) rather than a module-level docstring ("""). Consequently, __doc__ resolves to None, causing the script to exit silently with a success status (0) instead of displaying the usage guide and returning an error status.
| #!/usr/bin/env python3 | |
| # Adds a sandbox's SPIFFE identity to the sandbox role in policy.json. | |
| # | |
| # bind.py <member> e.g. user:spiffe://openshell.local/openshell/sandbox/<id> | |
| # bind.py --remove <member> | |
| #!/usr/bin/env python3 | |
| """Adds a sandbox's SPIFFE identity to the sandbox role in policy.json. | |
| bind.py <member> e.g. user:spiffe://openshell.local/openshell/sandbox/<id> | |
| bind.py --remove <member> | |
| """ |
| mkdir -p run && umask 022 && printf 'mesh_tok_%s' "$$(head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n')" > $@ | ||
|
|
||
| run/admin-token: | ||
| mkdir -p run && umask 022 && printf 'mesh_adm_%s' "$$(head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n')" > $@ | ||
|
|
||
| run/api-token: | ||
| mkdir -p run && umask 022 && head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n' > $@ |
There was a problem hiding this comment.
The generated tokens (join-token, admin-token, and api-token) contain highly sensitive credentials. Using umask 022 creates these files with world-readable permissions (0644). It is highly recommended to use umask 077 instead, so that the generated files are only readable by the owner (0600).
mkdir -p run && umask 077 && printf 'mesh_tok_%s' "$$(head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n')" > $@
run/admin-token:
mkdir -p run && umask 077 && printf 'mesh_adm_%s' "$$(head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n')" > $@
run/api-token:
mkdir -p run && umask 077 && head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n' > $@
| mkdir -p run && umask 022 && printf 'mesh_tok_%s' "$$(head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n')" > $@ | ||
|
|
||
| run/admin-token: | ||
| mkdir -p run && umask 022 && printf 'mesh_adm_%s' "$$(head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n')" > $@ | ||
|
|
||
| run/api-token: | ||
| mkdir -p run && umask 022 && head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n' > $@ |
There was a problem hiding this comment.
The generated tokens (join-token, admin-token, and api-token) contain highly sensitive credentials. Using umask 022 creates these files with world-readable permissions (0644). It is highly recommended to use umask 077 instead, so that the generated files are only readable by the owner (0600).
mkdir -p run && umask 077 && printf 'mesh_tok_%s' "$$(head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n')" > $@
run/admin-token:
mkdir -p run && umask 077 && printf 'mesh_adm_%s' "$$(head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n')" > $@
run/api-token:
mkdir -p run && umask 077 && head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n' > $@
d618c54 to
4e64e96
Compare
…es or on GKE A tutorial, site/content/docs/guides/openshell-and-substrate.md: how to put NVIDIA OpenShell sandboxes and Agent Substrate actors on one Agent Mesh, each runtime on Kubernetes you manage or on GKE, with the hub in the cluster or on Cloud Run; how to move a workload between the two runtimes with a policy change; and how a workload on either runtime calls Gemini on Vertex AI and an agent on Agent Engine through an egress node with a federated credential. One section per deployment, with the commands to run and the output to expect. Every section was run on Agent Mesh v0.1.1, OpenShell 0.1.3 and 0.1.4, Substrate v0.5.0 and GKE 1.36.4. development/examples/openshell-kubernetes is new: OpenShell from its Helm chart (0.1.4) with Agent Sandbox v1.0.7 and SPIRE 0.29.0, one ClusterSPIFFEID on the supervisor pods and one chart value; KUBE_CONTEXT for kind or GKE, HUB=local or a URL, OPENSHELL_GATEWAY per cluster. On GKE the sandbox node pool is Ubuntu (Landlock), allowed_ips is rendered from the edge Service's ClusterIP, and SPIRE's OIDC discovery is exposed on a reserved address as the issuer. The provider profile allows the /mesh/** path on the edge node, which a request to an egress destination served by another node uses. development/examples/substrate-actor joins a hub outside the cluster: the provider signs with a key the operator keeps and its discovery document and JWKS are static files at a public URL; KUBE_CONTEXT and CALC_MCP_IMAGE put the mesh on a cluster from Substrate's own tooling. Both Kubernetes examples get make gemini: from inside the sandbox or the actor, generateContent on a Gemini model (allowed), a GET on the model and a POST to another Vertex AI path (both refused at the serving node), and with AGENT_ENGINE=<id> a query to an agent on Agent Engine. All three examples pin v0.1.1 and select their own gateway in the openshell CLI. The Kubernetes ones: the nodes serve /readyz through --metrics-addr and the manifests probe it, so rollout status waits for a node on the mesh; the port-forward helpers refuse a local port that is in use; the Substrate down keeps the Service the egress gateway was installed with. The Docker example gets allowed_publish on the node role.
4e64e96 to
b180d9c
Compare
Stacked on #655.
A tutorial,
site/content/docs/guides/openshell-and-substrate.md: how to put NVIDIA OpenShell sandboxes and Agent Substrate actors on one Agent Mesh, on Kubernetes you manage or on GKE, and how to move a workload between the two runtimes with a policy change. One section per deployment with the commands to run and the output to expect; the hub in the cluster or on Cloud Run; both runtimes on one mesh; moving a workload; a Google API (Gemini on Vertex AI, an agent on Agent Engine) from either runtime through an egress node with a federated credential; clean-up. The federation setup is in #749.Examples:
development/examples/openshell-kubernetes/(new): OpenShell Helm chart 0.1.4, Agent Sandbox v1.0.7, SPIRE 0.29.0 with aClusterSPIFFEIDon the supervisor pods andprovider_spiffe_workload_api_socket_path.KUBE_CONTEXTfor kind or GKE,HUB=local|<url>,OPENSHELL_GATEWAYper cluster. On GKE: Ubuntu node pool (Landlock),allowed_ipsfrom the edge Service's ClusterIP, SPIRE OIDC discovery on a reserved address. The provider profile allows/mesh/**on the edge node.development/examples/substrate-actor/: remote-hub mode (provider signs with an operator key; discovery and JWKS as static files),KUBE_CONTEXTandCALC_MCP_IMAGEfor a cluster from Substrate'stools/setup-gcp bootstrap.make gemini NAME=<workload> GCP_PROJECT=<project> [AGENT_ENGINE=<id>]runs, from inside the sandbox or the actor,generateContenton a Gemini model (allowed), aGETon the model and aPOSTto another Vertex AI path (both refused at the serving node), and a query to an agent on Agent Engine.v0.1.1and select their own gateway in theopenshellCLI. The Kubernetes ones:--metrics-addrand a/readyzreadiness probe on the nodes; port-forward helpers refuse a local port that is in use;downkeeps the Service the Substrate egress gateway was installed with. The Docker example getsallowed_publishon the node role.Every section was run on the released images: Agent Mesh v0.1.1, OpenShell 0.1.3/0.1.4, Substrate v0.5.0, GKE 1.36.4 Dataplane V2,
agentmesh-oneon Cloud Run with operator-owned keys. All four deployments; the cross-cluster call in both directions with the audit line at the serving node; cutover and rollback by policy with the latencies as measured; Gemini and Agent Engine from an OpenShell sandbox and from a Substrate actor, with the two refusals and Google'sGenerateAccessTokenaudit entry.Related: agent-substrate/substrate#2827 (DNS relay
REFUSEDvs search-list resolvers; the example uses absolute names).