From ab499c62121c0f2714beef3a2a3af804f70d5076 Mon Sep 17 00:00:00 2001 From: Rajiv Shah Date: Fri, 2 Oct 2026 22:48:43 -0500 Subject: [PATCH 1/5] docs: add Google Vertex AI and Gemini gateway guide --- docs.json | 1 + .../integrations/google-llm-gateway.mdx | 288 ++++++++++++++++++ enterprise/integrations/overview.mdx | 3 + enterprise/k8s-install/installation.mdx | 3 + .../admin-console-configuration.mdx | 1 + llms-full.txt | 287 +++++++++++++++++ llms.txt | 1 + 7 files changed, 584 insertions(+) create mode 100644 enterprise/integrations/google-llm-gateway.mdx diff --git a/docs.json b/docs.json index 9babff37f..08dfb9c13 100644 --- a/docs.json +++ b/docs.json @@ -603,6 +603,7 @@ }, "enterprise/integrations/slack", "enterprise/integrations/external-llm-gateways", + "enterprise/integrations/google-llm-gateway", "enterprise/integrations/observability-platforms" ] }, diff --git a/enterprise/integrations/google-llm-gateway.mdx b/enterprise/integrations/google-llm-gateway.mdx new file mode 100644 index 000000000..f3466c209 --- /dev/null +++ b/enterprise/integrations/google-llm-gateway.mdx @@ -0,0 +1,288 @@ +--- +title: Google LLM Gateway +description: Connect Google AI Studio and Vertex AI models through the OpenHands Enterprise LLM gateway. +icon: google +--- + +OpenHands Enterprise connects to Google models through its bundled LiteLLM +gateway. These configurations apply whether OpenHands runs on GKE or another +supported Kubernetes platform. The Google credential belongs to the bundled +LiteLLM gateway; the sandbox uses the gateway model alias. + +Choose your installation method: + +- **Replicated**: configure the gateway in the Admin Console. +- **Helm**: configure the gateway through your installation values. + +## Choose the Provider Route + +Choose the route that matches your Google credentials: + +| Route | Google Authentication | LiteLLM Model Prefix | +| --- | --- | --- | +| Google AI Studio (Gemini API) | Gemini API key | `gemini/` | +| Google Cloud Platform (Vertex AI) | Google Cloud project, location, and service account | `vertex_ai/` | + +These are separate APIs, even when both serve a model named +`gemini-2.5-flash`. In Replicated, the Admin Console selects one Google API +type. A Helm installation can expose both as different gateway aliases. + +## Configure the Gateway + + + + +1. Open the [Admin Console LLM configuration](/enterprise/vm-install/admin-console-configuration#llm-configuration) + and select `Google` as the provider. +2. Under `Google API Type`, choose one route: + - `Google AI Studio (Gemini API)`: enter the `Google Gemini API Key` from + [Google AI Studio](https://ai.google.dev/gemini-api/docs/api-key). In + `Gemini Models`, enter one model ID per line. + - `Google Cloud Platform (Vertex AI)`: enter the `Google Cloud Project ID` + and `Google Cloud Location`, then upload the `Google Cloud Service Account + JSON file`. In `Vertex AI Models`, enter one model ID per line. Enable the + Vertex AI API in the project and grant the service account the + [Vertex AI User role](https://docs.cloud.google.com/iam/docs/roles-permissions/aiplatform) + or equivalent model-inference permissions. +3. Enter the raw model IDs, without a `gemini/` or `vertex_ai/` prefix. For + example, the Vertex field can contain: + + ```text + gemini-2.5-flash + gemini-2.5-pro + ``` + + The Admin Console creates one bundled-gateway route per line, and the first + line becomes the installation default. Confirm that each model is available + to your account and, for Vertex, in your selected location. +4. Save the configuration and deploy the updated version. + +The uploaded service-account file is used by the bundled LiteLLM pod. Keep it +out of source control. The `Allow users to configure their own LLM providers +(BYOK)` checkbox is separate from these administrator-managed models. + + + + +### Prerequisites + +- A working [OpenHands Enterprise Helm installation](/enterprise/k8s-install/installation). +- A Google model that supports tool use and is available through your chosen + API. The examples use `gemini-2.5-flash`. +- HTTPS access from the bundled LiteLLM pod to `generativelanguage.googleapis.com` + for Gemini API, or your Vertex API endpoint and `oauth2.googleapis.com` for Vertex. +- For Vertex, a project with billing and the Vertex AI API enabled, a supported + model/location, and service-account inference permissions. +- Enough model quota for agent prompts and tool definitions. + +Choose either route below, or add both to your **complete** installation +`values.yaml`. Keep your existing `litellm-helm.proxy_config.model_list`, +`environmentSecrets`, `volumes`, and `volumeMounts` entries when adding new +ones: Helm replaces lists when applying overrides. Use distinct `model_name` +aliases when both routes expose the same Google model ID. + +### Google AI Studio (Gemini API) + +Create a Kubernetes Secret from a private file containing your +[Gemini API key](https://ai.google.dev/gemini-api/docs/api-key): + +```bash +kubectl -n openhands create secret generic google-ai-studio-gateway \ + --from-file=GOOGLE_API_KEY=/path/to/private/gemini-api-key +``` + +Add that Secret to the gateway's existing `environmentSecrets` list and append +one route per model to `model_list`: + +```yaml +litellm-helm: + environmentSecrets: + - litellm-env-secrets + - google-ai-studio-gateway + proxy_config: + model_list: + # Retain your existing model entries here. + - model_name: google-ai-studio-flash + litellm_params: + model: gemini/gemini-2.5-flash + api_key: os.environ/GOOGLE_API_KEY +``` + +The API key belongs to the LiteLLM gateway. It is separate from an OpenHands +API key for conversations or automations. + +### Google Cloud Platform (Vertex AI) + +Enable Vertex AI in your Google Cloud project. Grant a dedicated service +account the `roles/aiplatform.user` role or equivalent model-inference +permissions, and confirm that `gemini-2.5-flash` is available in your chosen +location. This example uses a service-account JSON file, matching the +Replicated Admin Console path. Follow your organization's policy for +[service-account key creation and rotation](https://cloud.google.com/iam/docs/keys-create-delete). +A workstation `gcloud` login does not supply credentials to the LiteLLM pod. +Create the Secret from a private file: + +```bash +kubectl -n openhands create secret generic google-vertex-gateway \ + --from-file=credentials.json=/path/to/private/service-account.json +``` + +Mount that Secret only in the bundled LiteLLM pod. Add one gateway route per +model. Replace the project ID and location with your own: + +```yaml +litellm-helm: + volumes: + # Retain any existing volumes here. + - name: google-vertex-credentials + secret: + secretName: google-vertex-gateway + volumeMounts: + # Retain any existing volume mounts here. + - name: google-vertex-credentials + mountPath: /etc/gcloud/vertex-credentials.json + subPath: credentials.json + readOnly: true + envVars: + GOOGLE_APPLICATION_CREDENTIALS: /etc/gcloud/vertex-credentials.json + proxy_config: + model_list: + # Retain your existing model entries here. + - model_name: google-vertex-flash + litellm_params: + model: vertex_ai/gemini-2.5-flash + vertex_project: + vertex_location: us-central1 +``` + +The credentials file, project, and location are used by LiteLLM to call +Vertex AI. OpenHands and its sandboxes use the gateway alias; they do not +need this file mounted into their pods. + +### Apply the Updated Values + +Use the licensed chart URL and version from your installation, then confirm +the bundled gateway is ready: + +```bash +helm upgrade openhands "$OPENHANDS_CHART_URL" \ + --namespace openhands \ + --version "$OPENHANDS_CHART_VERSION" \ + --values values.yaml \ + --wait --timeout 10m + +kubectl -n openhands rollout status deployment/openhands-litellm +``` + +Adjust the namespace, release name, and Deployment name if they differ in your +installation. To make one of these aliases the installation default, set +`env.LITELLM_DEFAULT_MODEL` to `litellm_proxy/` in the same values file. + + + + +## Select the Model in OpenHands + +For Replicated, select the Google models in the Admin Console and deploy the +configuration. Users do not need to open LiteLLM or enter the Google credential. + +For Helm, set the desired gateway alias as the installation default in your +complete values file before upgrading. For the Vertex example: + +```yaml +env: + LITELLM_DEFAULT_MODEL: litellm_proxy/google-vertex-flash +``` + +For the Gemini API example, use `litellm_proxy/google-ai-studio-flash` instead. +Keep the provider credential in LiteLLM; users do not need a personal Google key +to use an administrator-managed route. + +On the tested chart `0.74.0` / OpenHands `1.67.0`, a fresh user's `Default` +profile resolved to `openhands/google-vertex-flash` with the internal gateway +base URL `http://openhands-litellm.openhands.svc.cluster.local:4000`. No user +profile override or provider key entry was required. Existing users may retain +previously selected profiles; confirm the model selected for the new conversation. + +To offer both Helm aliases as named profiles, save a profile for each in +`Settings` → `LLM`, using your installation's internal gateway URL: + +| Field | Vertex AI | Gemini API | +| --- | --- | --- | +| Profile Name | `Google-Vertex-Flash` | `Google-Gemini-Flash` | +| Model | `openhands/google-vertex-flash` | `openhands/google-ai-studio-flash` | +| Base URL | `http://openhands-litellm.openhands.svc.cluster.local:4000` | `http://openhands-litellm.openhands.svc.cluster.local:4000` | +| Canonical Model Name | `gemini-2.5-flash` | `gemini-2.5-flash` | +| API Mode | `chat` | `chat` | +| API Key | Leave unset; use the managed gateway credential | Leave unset; use the managed gateway credential | + +Use your actual gateway Service name and namespace. The selected model points to +the bundled gateway alias; the Google provider route is configured in LiteLLM. + +The gateway calls Vertex AI. The sandbox calls the gateway, so this path does +not require mounting Google credentials into the sandbox or rebuilding the +agent-server image with `ENABLE_VERTEX=1`. That build flag applies when the +agent-server calls `vertex_ai/*` directly; see +[Vertex AI dependencies](/openhands/usage/llms/google-llms#vertex-ai-dependencies). + +## Start Using the Model + +1. Sign in and confirm the conversation UI loads without an additional backend + URL or API-key prompt. If it prompts, check the Canvas configuration in the + [Helm installation guide](/enterprise/k8s-install/installation). +2. Start a new conversation using the installation's `Default` profile. +3. Ask the agent to run `pwd`, create a small workspace file and read it back. +4. Confirm the sandbox reaches `READY`, tool results contain the expected path + and file contents, and the agent completes its response. A successful direct + gateway request alone does not validate the OpenHands conversation path. + + +The Vertex route passed end-to-end tests on GKE with Helm chart `0.74.0`, +OpenHands `1.67.0`, agent-server `1.49.6-python`, and `gemini-2.5-flash` in +`us-central1`. The unmodified agent-server image ran terminal and file tools +and completed the conversation. On Replicated release `0.74.0`, the Admin +Console Vertex configuration also passed GitHub login, terminal file operations +and a read-only repository conversation. The Gemini API examples were checked +against the provider and chart configuration; they have not yet been validated +with an end-to-end conversation in this evaluation. + + +## Troubleshooting + + + +Check the Gemini API key in the Kubernetes Secret and confirm that it can use +the selected model. The model route must use `gemini/`, not `vertex_ai/`. + + +Check that Vertex AI is enabled, the service account can call the model, the +Secret contains a valid JSON file, and the LiteLLM pod mounts it at the path +in `GOOGLE_APPLICATION_CREDENTIALS`. + + +Check the model ID and the Google API type. For Vertex AI, also check the +project and location. Model availability can differ between Google AI Studio +and Vertex AI and across locations. + + +Check the internal LiteLLM Service DNS name, namespace, and selected model alias. +Confirm the LiteLLM Deployment is ready. + + +Check request and token quotas, billing, and model capacity for the chosen API. +A small direct completion can succeed while a larger agent prompt is rate limited. + + +Check runtime registration, pod readiness and Kubernetes events using the +[Troubleshooting guide](/enterprise/troubleshooting). For GKE, verify your Sysbox +installation. A runtime startup failure does not by itself establish a Google +credential problem. + + + +For provider-specific configuration, see LiteLLM's +[Google AI Studio](https://docs.litellm.ai/docs/providers/gemini) and +[Vertex AI](https://docs.litellm.ai/docs/providers/vertex) references. + +If your organization uses an existing external gateway, follow +[External LLM Gateways](/enterprise/integrations/external-llm-gateways). diff --git a/enterprise/integrations/overview.mdx b/enterprise/integrations/overview.mdx index b1e323419..31decf978 100644 --- a/enterprise/integrations/overview.mdx +++ b/enterprise/integrations/overview.mdx @@ -140,6 +140,9 @@ See [MCP Settings](/openhands/usage/settings/mcp-settings) to add and configure Route LLM traffic through your existing LiteLLM or Bifrost gateway for routing, cost tracking, and audit. + + Connect Vertex AI and Gemini API models through the bundled gateway on Replicated or Helm. + Send conversation traces to your own OTLP-compatible platform, such as Langfuse, Honeycomb, or Tempo. diff --git a/enterprise/k8s-install/installation.mdx b/enterprise/k8s-install/installation.mdx index e84ac3f77..120061e88 100644 --- a/enterprise/k8s-install/installation.mdx +++ b/enterprise/k8s-install/installation.mdx @@ -358,6 +358,9 @@ overrides on the same release — edit your `values.yaml` and apply with Enable conversation analytics with Laminar. + + Configure Vertex AI or Gemini API routes in the bundled gateway. + Run scheduled or event-triggered tasks on a Helm installation. diff --git a/enterprise/vm-install/admin-console-configuration.mdx b/enterprise/vm-install/admin-console-configuration.mdx index 2052f18b0..711cee992 100644 --- a/enterprise/vm-install/admin-console-configuration.mdx +++ b/enterprise/vm-install/admin-console-configuration.mdx @@ -111,6 +111,7 @@ Select the administrator-managed LLM provider. The Admin Console shows only the - For AWS Bedrock, use an EC2 instance profile where possible. Pods must be able to reach the instance metadata service, and the role needs model invocation permissions. - For custom OpenAI-compatible endpoints, prefix model names with `openai/`. - Model lists accept one model per line. +- For Google, see [Google LLM Gateway](/enterprise/integrations/google-llm-gateway) for Vertex AI and Gemini API configuration and verification. ### Bring Your Own Key diff --git a/llms-full.txt b/llms-full.txt index e5697256d..b3213886a 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -50014,6 +50014,292 @@ when the job starts and when it completes. | Bitbucket webhook deliveries do not reach OpenHands | Confirm the Bitbucket Data Center network can reach the OpenHands app URL. | | Bitbucket API calls fail with TLS errors | Upload the Bitbucket Data Center CA certificate in **Additional Trusted CA Certificates** and redeploy. | +### Google LLM Gateway +Source: https://docs.openhands.dev/enterprise/integrations/google-llm-gateway.md + +OpenHands Enterprise connects to Google models through its bundled LiteLLM +gateway. These configurations apply whether OpenHands runs on GKE or another +supported Kubernetes platform. The Google credential belongs to the bundled +LiteLLM gateway; the sandbox uses the gateway model alias. + +Choose your installation method: + +- **Replicated**: configure the gateway in the Admin Console. +- **Helm**: configure the gateway through your installation values. + +## Choose the Provider Route + +Choose the route that matches your Google credentials: + +| Route | Google Authentication | LiteLLM Model Prefix | +| --- | --- | --- | +| Google AI Studio (Gemini API) | Gemini API key | `gemini/` | +| Google Cloud Platform (Vertex AI) | Google Cloud project, location, and service account | `vertex_ai/` | + +These are separate APIs, even when both serve a model named +`gemini-2.5-flash`. In Replicated, the Admin Console selects one Google API +type. A Helm installation can expose both as different gateway aliases. + +## Configure the Gateway + + + + +1. Open the [Admin Console LLM configuration](/enterprise/vm-install/admin-console-configuration#llm-configuration) + and select `Google` as the provider. +2. Under `Google API Type`, choose one route: + - `Google AI Studio (Gemini API)`: enter the `Google Gemini API Key` from + [Google AI Studio](https://ai.google.dev/gemini-api/docs/api-key). In + `Gemini Models`, enter one model ID per line. + - `Google Cloud Platform (Vertex AI)`: enter the `Google Cloud Project ID` + and `Google Cloud Location`, then upload the `Google Cloud Service Account + JSON file`. In `Vertex AI Models`, enter one model ID per line. Enable the + Vertex AI API in the project and grant the service account the + [Vertex AI User role](https://docs.cloud.google.com/iam/docs/roles-permissions/aiplatform) + or equivalent model-inference permissions. +3. Enter the raw model IDs, without a `gemini/` or `vertex_ai/` prefix. For + example, the Vertex field can contain: + + ```text + gemini-2.5-flash + gemini-2.5-pro + ``` + + The Admin Console creates one bundled-gateway route per line, and the first + line becomes the installation default. Confirm that each model is available + to your account and, for Vertex, in your selected location. +4. Save the configuration and deploy the updated version. + +The uploaded service-account file is used by the bundled LiteLLM pod. Keep it +out of source control. The `Allow users to configure their own LLM providers +(BYOK)` checkbox is separate from these administrator-managed models. + + + + +### Prerequisites + +- A working [OpenHands Enterprise Helm installation](/enterprise/k8s-install/installation). +- A Google model that supports tool use and is available through your chosen + API. The examples use `gemini-2.5-flash`. +- HTTPS access from the bundled LiteLLM pod to `generativelanguage.googleapis.com` + for Gemini API, or your Vertex API endpoint and `oauth2.googleapis.com` for Vertex. +- For Vertex, a project with billing and the Vertex AI API enabled, a supported + model/location, and service-account inference permissions. +- Enough model quota for agent prompts and tool definitions. + +Choose either route below, or add both to your **complete** installation +`values.yaml`. Keep your existing `litellm-helm.proxy_config.model_list`, +`environmentSecrets`, `volumes`, and `volumeMounts` entries when adding new +ones: Helm replaces lists when applying overrides. Use distinct `model_name` +aliases when both routes expose the same Google model ID. + +### Google AI Studio (Gemini API) + +Create a Kubernetes Secret from a private file containing your +[Gemini API key](https://ai.google.dev/gemini-api/docs/api-key): + +```bash +kubectl -n openhands create secret generic google-ai-studio-gateway \ + --from-file=GOOGLE_API_KEY=/path/to/private/gemini-api-key +``` + +Add that Secret to the gateway's existing `environmentSecrets` list and append +one route per model to `model_list`: + +```yaml +litellm-helm: + environmentSecrets: + - litellm-env-secrets + - google-ai-studio-gateway + proxy_config: + model_list: + # Retain your existing model entries here. + - model_name: google-ai-studio-flash + litellm_params: + model: gemini/gemini-2.5-flash + api_key: os.environ/GOOGLE_API_KEY +``` + +The API key belongs to the LiteLLM gateway. It is separate from an OpenHands +API key for conversations or automations. + +### Google Cloud Platform (Vertex AI) + +Enable Vertex AI in your Google Cloud project. Grant a dedicated service +account the `roles/aiplatform.user` role or equivalent model-inference +permissions, and confirm that `gemini-2.5-flash` is available in your chosen +location. This example uses a service-account JSON file, matching the +Replicated Admin Console path. Follow your organization's policy for +[service-account key creation and rotation](https://cloud.google.com/iam/docs/keys-create-delete). +A workstation `gcloud` login does not supply credentials to the LiteLLM pod. +Create the Secret from a private file: + +```bash +kubectl -n openhands create secret generic google-vertex-gateway \ + --from-file=credentials.json=/path/to/private/service-account.json +``` + +Mount that Secret only in the bundled LiteLLM pod. Add one gateway route per +model. Replace the project ID and location with your own: + +```yaml +litellm-helm: + volumes: + # Retain any existing volumes here. + - name: google-vertex-credentials + secret: + secretName: google-vertex-gateway + volumeMounts: + # Retain any existing volume mounts here. + - name: google-vertex-credentials + mountPath: /etc/gcloud/vertex-credentials.json + subPath: credentials.json + readOnly: true + envVars: + GOOGLE_APPLICATION_CREDENTIALS: /etc/gcloud/vertex-credentials.json + proxy_config: + model_list: + # Retain your existing model entries here. + - model_name: google-vertex-flash + litellm_params: + model: vertex_ai/gemini-2.5-flash + vertex_project: + vertex_location: us-central1 +``` + +The credentials file, project, and location are used by LiteLLM to call +Vertex AI. OpenHands and its sandboxes use the gateway alias; they do not +need this file mounted into their pods. + +### Apply the Updated Values + +Use the licensed chart URL and version from your installation, then confirm +the bundled gateway is ready: + +```bash +helm upgrade openhands "$OPENHANDS_CHART_URL" \ + --namespace openhands \ + --version "$OPENHANDS_CHART_VERSION" \ + --values values.yaml \ + --wait --timeout 10m + +kubectl -n openhands rollout status deployment/openhands-litellm +``` + +Adjust the namespace, release name, and Deployment name if they differ in your +installation. To make one of these aliases the installation default, set +`env.LITELLM_DEFAULT_MODEL` to `litellm_proxy/` in the same values file. + + + + +## Select the Model in OpenHands + +For Replicated, select the Google models in the Admin Console and deploy the +configuration. Users do not need to open LiteLLM or enter the Google credential. + +For Helm, set the desired gateway alias as the installation default in your +complete values file before upgrading. For the Vertex example: + +```yaml +env: + LITELLM_DEFAULT_MODEL: litellm_proxy/google-vertex-flash +``` + +For the Gemini API example, use `litellm_proxy/google-ai-studio-flash` instead. +Keep the provider credential in LiteLLM; users do not need a personal Google key +to use an administrator-managed route. + +On the tested chart `0.74.0` / OpenHands `1.67.0`, a fresh user's `Default` +profile resolved to `openhands/google-vertex-flash` with the internal gateway +base URL `http://openhands-litellm.openhands.svc.cluster.local:4000`. No user +profile override or provider key entry was required. Existing users may retain +previously selected profiles; confirm the model selected for the new conversation. + +To offer both Helm aliases as named profiles, save a profile for each in +`Settings` → `LLM`, using your installation's internal gateway URL: + +| Field | Vertex AI | Gemini API | +| --- | --- | --- | +| Profile Name | `Google-Vertex-Flash` | `Google-Gemini-Flash` | +| Model | `openhands/google-vertex-flash` | `openhands/google-ai-studio-flash` | +| Base URL | `http://openhands-litellm.openhands.svc.cluster.local:4000` | `http://openhands-litellm.openhands.svc.cluster.local:4000` | +| Canonical Model Name | `gemini-2.5-flash` | `gemini-2.5-flash` | +| API Mode | `chat` | `chat` | +| API Key | Leave unset; use the managed gateway credential | Leave unset; use the managed gateway credential | + +Use your actual gateway Service name and namespace. The selected model points to +the bundled gateway alias; the Google provider route is configured in LiteLLM. + +The gateway calls Vertex AI. The sandbox calls the gateway, so this path does +not require mounting Google credentials into the sandbox or rebuilding the +agent-server image with `ENABLE_VERTEX=1`. That build flag applies when the +agent-server calls `vertex_ai/*` directly; see +[Vertex AI dependencies](/openhands/usage/llms/google-llms#vertex-ai-dependencies). + +## Start Using the Model + +1. Sign in and confirm the conversation UI loads without an additional backend + URL or API-key prompt. If it prompts, check the Canvas configuration in the + [Helm installation guide](/enterprise/k8s-install/installation). +2. Start a new conversation using the installation's `Default` profile. +3. Ask the agent to run `pwd`, create a small workspace file and read it back. +4. Confirm the sandbox reaches `READY`, tool results contain the expected path + and file contents, and the agent completes its response. A successful direct + gateway request alone does not validate the OpenHands conversation path. + + +The Vertex route passed end-to-end tests on GKE with Helm chart `0.74.0`, +OpenHands `1.67.0`, agent-server `1.49.6-python`, and `gemini-2.5-flash` in +`us-central1`. The unmodified agent-server image ran terminal and file tools +and completed the conversation. On Replicated release `0.74.0`, the Admin +Console Vertex configuration also passed GitHub login, terminal file operations +and a read-only repository conversation. The Gemini API examples were checked +against the provider and chart configuration; they have not yet been validated +with an end-to-end conversation in this evaluation. + + +## Troubleshooting + + + +Check the Gemini API key in the Kubernetes Secret and confirm that it can use +the selected model. The model route must use `gemini/`, not `vertex_ai/`. + + +Check that Vertex AI is enabled, the service account can call the model, the +Secret contains a valid JSON file, and the LiteLLM pod mounts it at the path +in `GOOGLE_APPLICATION_CREDENTIALS`. + + +Check the model ID and the Google API type. For Vertex AI, also check the +project and location. Model availability can differ between Google AI Studio +and Vertex AI and across locations. + + +Check the internal LiteLLM Service DNS name, namespace, and selected model alias. +Confirm the LiteLLM Deployment is ready. + + +Check request and token quotas, billing, and model capacity for the chosen API. +A small direct completion can succeed while a larger agent prompt is rate limited. + + +Check runtime registration, pod readiness and Kubernetes events using the +[Troubleshooting guide](/enterprise/troubleshooting). For GKE, verify your Sysbox +installation. A runtime startup failure does not by itself establish a Google +credential problem. + + + +For provider-specific configuration, see LiteLLM's +[Google AI Studio](https://docs.litellm.ai/docs/providers/gemini) and +[Vertex AI](https://docs.litellm.ai/docs/providers/vertex) references. + +If your organization uses an existing external gateway, follow +[External LLM Gateways](/enterprise/integrations/external-llm-gateways). + ### External LLM Gateways Source: https://docs.openhands.dev/enterprise/integrations/external-llm-gateways.md @@ -55981,6 +56267,7 @@ Select the administrator-managed LLM provider. The Admin Console shows only the - For AWS Bedrock, use an EC2 instance profile where possible. Pods must be able to reach the instance metadata service, and the role needs model invocation permissions. - For custom OpenAI-compatible endpoints, prefix model names with `openai/`. - Model lists accept one model per line. +- For Google, see [Google LLM Gateway](/enterprise/integrations/google-llm-gateway) for Vertex AI and Gemini API configuration and verification. ### Bring Your Own Key diff --git a/llms.txt b/llms.txt index 638b3c95d..d6dcdded1 100644 --- a/llms.txt +++ b/llms.txt @@ -267,6 +267,7 @@ from the OpenHands Software Agent SDK. - [Custom Sandbox Images](https://docs.openhands.dev/enterprise/custom-sandbox-image.md): Preload repos, dependencies, and tooling into custom sandbox images, and run multiple images side by side with warm runtime pools. - [DNS and TLS](https://docs.openhands.dev/enterprise/k8s-install/dns-and-tls.md): Automate DNS records and TLS certificates with external-dns and cert-manager - [Enterprise vs. Open Source](https://docs.openhands.dev/enterprise/enterprise-vs-oss.md): Compare OpenHands Enterprise and Open Source offerings to choose the right option for your team +- [Google LLM Gateway](https://docs.openhands.dev/enterprise/integrations/google-llm-gateway.md): Connect Google AI Studio and Vertex AI models through the OpenHands Enterprise LLM gateway. - [External LLM Gateways](https://docs.openhands.dev/enterprise/integrations/external-llm-gateways.md): Chain OpenHands Enterprise to an existing LiteLLM or Bifrost gateway so LLM traffic flows through your existing routing, cost tracking, and audit layer. - [External Observability Platforms](https://docs.openhands.dev/enterprise/integrations/observability-platforms.md): Send OpenHands Enterprise conversation traces to your own OTLP-compatible observability platform such as Langfuse, Honeycomb, or Tempo. - [External PostgreSQL](https://docs.openhands.dev/enterprise/external-postgres.md): Configure OpenHands Enterprise to use your own PostgreSQL database From ab9fa3fde0311b15d1fd58bf927ecfb2ed796818 Mon Sep 17 00:00:00 2001 From: Rajiv Shah Date: Sun, 4 Oct 2026 16:41:01 -0500 Subject: [PATCH 2/5] Preserve measured validation and clarify supported configuration scope --- enterprise/integrations/google-llm-gateway.mdx | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/enterprise/integrations/google-llm-gateway.mdx b/enterprise/integrations/google-llm-gateway.mdx index 3f768fa00..3bf60686f 100644 --- a/enterprise/integrations/google-llm-gateway.mdx +++ b/enterprise/integrations/google-llm-gateway.mdx @@ -221,9 +221,10 @@ To offer both Helm aliases as named profiles, save a profile for each in | API Key | Leave unset; use the managed gateway credential | Leave unset; use the managed gateway credential | Use your actual gateway Service name and namespace. The selected model points to -the bundled gateway alias; the Google provider route is configured in LiteLLM. +the bundled gateway alias; configure its provider route through Helm values +or the Replicated Admin Console. -The gateway calls Vertex AI. The sandbox calls the gateway, so this path does +The Helm gateway calls Vertex AI. The sandbox calls the gateway, so this path does not require mounting Google credentials into the sandbox or rebuilding the agent-server image with `ENABLE_VERTEX=1`. That build flag applies when the agent-server calls `vertex_ai/*` directly; see @@ -244,7 +245,10 @@ agent-server calls `vertex_ai/*` directly; see The Vertex route passed end-to-end tests on GKE with Helm chart `0.74.0`, OpenHands `1.67.0`, agent-server `1.49.6-python`, and `gemini-2.5-flash` in `us-central1`. The unmodified agent-server image ran terminal and file tools -and completed the conversation. On Replicated release `0.74.0`, the Admin +and completed the conversation. A fresh conversation using a named organization +profile also passed terminal file operations through the OpenHands API on this +release. The personal Settings UI named-profile workflow was not separately +validated by that API check. On Replicated release `0.74.0`, the Admin Console Vertex configuration also passed GitHub login, terminal file operations and a read-only repository conversation. The Gemini API examples were checked against the provider and chart configuration; they have not yet been validated From 1e0db1b764bad9ecdd7e0dfaf5b2291c51a80cb7 Mon Sep 17 00:00:00 2001 From: Rajiv Shah Date: Sun, 4 Oct 2026 16:57:00 -0500 Subject: [PATCH 3/5] Use verified profile UI fields and tested organization API overrides --- .../integrations/google-llm-gateway.mdx | 22 +++++++++++-------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/enterprise/integrations/google-llm-gateway.mdx b/enterprise/integrations/google-llm-gateway.mdx index 3bf60686f..ce677d41d 100644 --- a/enterprise/integrations/google-llm-gateway.mdx +++ b/enterprise/integrations/google-llm-gateway.mdx @@ -208,17 +208,20 @@ base URL `http://openhands-litellm.openhands.svc.cluster.local:4000`. No user profile override or provider key entry was required. Existing users may retain previously selected profiles; confirm the model selected for the new conversation. -To offer both Helm aliases as named profiles, save a profile for each in -`Settings` → `LLM`, using your installation's internal gateway URL: +To offer both Helm aliases as administrator-managed profiles, open the +organization's **Language Model (LLM)** defaults at `/settings/org-defaults`. +Choose **Add LLM Profile → Advanced** and use the internal gateway URL: + | Field | Vertex AI | Gemini API | | --- | --- | --- | -| Profile Name | `Google-Vertex-Flash` | `Google-Gemini-Flash` | -| Model | `openhands/google-vertex-flash` | `openhands/google-ai-studio-flash` | +| Name (Optional) | `Google-Vertex-Flash` | `Google-Gemini-Flash` | +| Custom Model | `openhands/google-vertex-flash` | `openhands/google-ai-studio-flash` | | Base URL | `http://openhands-litellm.openhands.svc.cluster.local:4000` | `http://openhands-litellm.openhands.svc.cluster.local:4000` | -| Canonical Model Name | `gemini-2.5-flash` | `gemini-2.5-flash` | -| API Mode | `chat` | `chat` | -| API Key | Leave unset; use the managed gateway credential | Leave unset; use the managed gateway credential | + +OpenHands supplies the managed gateway credential, so no key entry is required. +On OpenHands `1.67.0`, saving on this page also makes the profile the +organization's active default. Re-activate the intended default after testing. Use your actual gateway Service name and namespace. The selected model points to the bundled gateway alias; configure its provider route through Helm values @@ -247,8 +250,9 @@ OpenHands `1.67.0`, agent-server `1.49.6-python`, and `gemini-2.5-flash` in `us-central1`. The unmodified agent-server image ran terminal and file tools and completed the conversation. A fresh conversation using a named organization profile also passed terminal file operations through the OpenHands API on this -release. The personal Settings UI named-profile workflow was not separately -validated by that API check. On Replicated release `0.74.0`, the Admin +release. The organization defaults UI also saved the named Vertex profile with the +expected URL. Its subsequent fresh-start check did not complete; this is not +counted as an additional inference pass. On Replicated release `0.74.0`, the Admin Console Vertex configuration also passed GitHub login, terminal file operations and a read-only repository conversation. The Gemini API examples were checked against the provider and chart configuration; they have not yet been validated From ad3f6557c14f04326021c84660e2b052061cc59a Mon Sep 17 00:00:00 2001 From: Rajiv Shah Date: Sun, 4 Oct 2026 16:58:02 -0500 Subject: [PATCH 4/5] State Google profile validation scope precisely --- enterprise/integrations/google-llm-gateway.mdx | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/enterprise/integrations/google-llm-gateway.mdx b/enterprise/integrations/google-llm-gateway.mdx index ce677d41d..38b0b0da0 100644 --- a/enterprise/integrations/google-llm-gateway.mdx +++ b/enterprise/integrations/google-llm-gateway.mdx @@ -251,8 +251,7 @@ OpenHands `1.67.0`, agent-server `1.49.6-python`, and `gemini-2.5-flash` in and completed the conversation. A fresh conversation using a named organization profile also passed terminal file operations through the OpenHands API on this release. The organization defaults UI also saved the named Vertex profile with the -expected URL. Its subsequent fresh-start check did not complete; this is not -counted as an additional inference pass. On Replicated release `0.74.0`, the Admin +expected URL. Fresh startup for that UI-created profile remains unvalidated. On Replicated release `0.74.0`, the Admin Console Vertex configuration also passed GitHub login, terminal file operations and a read-only repository conversation. The Gemini API examples were checked against the provider and chart configuration; they have not yet been validated From 11ae17260d222935cac423d3dc060b1582c383be Mon Sep 17 00:00:00 2001 From: Rajiv Shah Date: Thu, 8 Oct 2026 07:37:00 -0500 Subject: [PATCH 5/5] Clarify Replicated Google credential distribution --- enterprise/integrations/google-llm-gateway.mdx | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/enterprise/integrations/google-llm-gateway.mdx b/enterprise/integrations/google-llm-gateway.mdx index 38b0b0da0..ae4818bbe 100644 --- a/enterprise/integrations/google-llm-gateway.mdx +++ b/enterprise/integrations/google-llm-gateway.mdx @@ -199,8 +199,9 @@ env: ``` For the Gemini API example, use `litellm_proxy/google-ai-studio-flash` instead. -Keep the provider credential in LiteLLM; users do not need a personal Google key -to use an administrator-managed route. +Users do not need a personal Google key to use an administrator-managed route. +Configure the provider credential through your installation's Helm values or +Admin Console; the Replicated credential distribution described above still applies. On the tested chart `0.74.0` / OpenHands `1.67.0`, a fresh user's `Default` profile resolved to `openhands/google-vertex-flash` with the internal gateway