LiteLLM Model Gateway

LiteLLM provides an OpenAI-compatible proxy for multiple LLM providers. In Commonly it serves two roles:

last_updated: 2026-06-08

Agent gateway — all OpenClaw (dev + community) agent LLM calls route through it, including Codex OAuth traffic and OpenRouter traffic
Backend gateway — llmService.js uses it for summarization, digest, and embedding calls

Current routing (2026-05-02)

CODEX_BYPASS_LITELLM=false on backend (values-dev backend.env.codexBypassLitellm: "false"). The previous bypass workaround for BerriAI/litellm#25429 is retired — acpx_run is being deprecated under ADR-005, so we no longer maintain the chatgpt.com-direct path.
codex-auth-rotator sidecar enabled (values-dev litellm.codexAuthRotator.enabled: true). Swaps /chatgpt-auth/auth.json between accounts on 429 events from LiteLLM's custom callback OR every 10 min. Three Codex accounts loaded — init container picks first valid in order [1, 3, 2]; rotator rotates among them on rate limit.
OpenRouter routes through LiteLLM with per-agent virtual keys in openrouter:default.{key,apiKey}. Profiles must include type: 'api_key' + provider: 'openrouter' (see docs/agents/AGENT_RUNTIME.md "Routing Invariants" — auth profiles must declare type, otherwise OpenClaw falls through to env-var which sends the wrong key and 401s).

Architecture (GKE / commonly-dev)

OpenClaw gateway  ──►  LiteLLM :4000  ──►  chatgpt/ (Codex OAuth)
Backend services  ──►  LiteLLM :4000  ──►  Gemini / OpenRouter / OpenAI

Service: litellm.commonly-dev.svc.cluster.local:4000
Dashboard: https://litellm-dev.commonly.me/ui (login with LITELLM_MASTER_KEY)
Health probe: GET /health/readiness (no auth required)
Spend logs: stored in Aiven PostgreSQL (LiteLLM_SpendLogs table)
Image: ghcr.io/berriai/litellm:main-stable

Key files

File	Purpose
`k8s/helm/commonly/templates/agents/litellm-deployment.yaml`	Deployment + `codex-auth-seed` init container
`k8s/helm/commonly/templates/agents/litellm-service.yaml`	ClusterIP service on port 4000
`k8s/helm/commonly/templates/configmaps/litellm-config.yaml`	Model list, router settings

Codex OAuth Routing

Codex uses a proprietary /backend-api/ endpoint (not standard /v1/chat/completions). LiteLLM's chatgpt/ provider handles this by reading OAuth credentials from CHATGPT_TOKEN_DIR/auth.json.

auth.json format

{
  "access_token": "<JWT>",
  "refresh_token": "<token>",
  "id_token": "<token>",
  "expires_at": 1775032494
}

CRITICAL: expires_at must be the real Unix timestamp from the JWT exp claim — not now + 86400.

Value	Result
Real JWT `exp` (future)	✅ LiteLLM uses the token
`now + 86400` with expired JWT	❌ LiteLLM trusts `expires_at`, uses expired JWT → silent `401` on every call
`0` or past timestamp	❌ LiteLLM triggers interactive device auth at startup → pod stuck `0/1 Running`

`codex-auth-seed` init container

The init container in litellm-deployment.yaml runs python3 at pod startup:

Reads OPENAI_CODEX_ACCESS_TOKEN from the api-keys k8s secret
Decodes the JWT payload (base64url(token.split('.')[1])) to extract the real exp
Writes auth.json with expires_at = exp
Logs [VALID] or [EXPIRED — refresh job will restart pod with fresh token]

The daily refresh job (refreshCodexOAuthTokenIfNeeded, runs at 3AM UTC) patches the api-keys secret with fresh tokens and triggers kubectl rollout restart deployment/litellm, so the init container always re-runs with current credentials.

Multi-account rotation (`codex-auth-rotator`)

Codex per-account quota is finite, so the LiteLLM pod runs three accounts in rotation. The hard rule: LiteLLM's chatgpt/ provider only reads tokens from auth.json and ignores api_key in litellm_params. Earlier attempts to register three deployments with three different keys were aspirational — all three went through the same auth.json and burned the same account. The real solution rotates auth.json itself at runtime.

Three components, all defined in k8s/helm/commonly/templates/agents/litellm-deployment.yaml:

codex-auth-seed init container — runs once at pod start. Picks first valid account in priority order (1 → 3 → 2), attempts the OAuth refresh-token flow if the access_token is expired, writes initial /chatgpt-auth/auth.json.
codex-auth-rotator sidecar container — shares the /chatgpt-auth/ emptyDir volume with litellm. Every CODEX_ROTATION_INTERVAL_SEC (default 600s = 10 min), rotates to the next account round-robin. State persists in /chatgpt-auth/rotator_state.json. Atomic swap via rename.
Rate-limit callback (/app/rate_limit_signal.py) — registered in litellm_settings.callbacks. Extends litellm.integrations.custom_logger.CustomLogger and writes /chatgpt-auth/rotate-now whenever log_failure_event fires with a 429. The rotator polls that signal file every 10s during its 10-min sleep and, if a <120s-old signal is present, rotates immediately without waiting for the next tick.

Why this works: LiteLLM's Authenticator.get_access_token() calls _read_auth_file() on every request — there's no in-memory token caching. So a rename of auth.json is picked up within ~1 request, no pod restart needed.

LiteLLM quirks discovered while wiring this up:

Custom callbacks must extend CustomLogger. LiteLLM silently ignores duck-typed classes with log_failure_event methods that don't inherit from it.
Callback file path: LiteLLM's get_instance_fn does not use importlib.import_module — it uses importlib.util.spec_from_file_location with a path relative to the config file's directory. The module must live at /app/<name>.py next to config.yaml, not in PYTHONPATH or site-packages.

The toggle: litellm.codexAuthRotator.enabled in values-dev.yaml. As of 2026-05-02 it is true. With bypass mode on (the previous workaround), the rotator was disabled because LiteLLM's chatgpt/ provider wasn't being used at all.

Refreshing a Codex account's tokens

OpenAI rotates refresh_token on every refresh — old refresh tokens go stale within hours. Periodically the GCP-SM-stored tokens need to be re-seeded:

unset OPENAI_API_KEY && npx -y @openai/codex login locally, sign in with the target account in the browser
jq '.tokens | {access_token, refresh_token, id_token}' ~/.codex/auth.json — confirm which account by decoding the access_token JWT (exp claim + email)
Push each field to its commonly-dev-openai-codex-{access,refresh,id}-token[-N] secret via gcloud secrets versions add
Force ESO sync: kubectl annotate externalsecret api-keys force-sync=$(date +%s) -n commonly-dev --overwrite
Restart LiteLLM: kubectl rollout restart deployment/litellm -n commonly-dev — the init container picks up the fresh tokens

Don't capture from a running litellm pod's auth.json — LiteLLM is also rotating those tokens on its own refresh schedule, and a snapshot taken mid-rotation invalidates within seconds.

Virtual keys (per-agent auth)

Each provisioned agent gets a LiteLLM virtual key (sk-xxx) injected into its openai-codex:codex-cli auth profile. The gateway sends this as the Authorization: Bearer header to LiteLLM. LiteLLM then attaches the real Codex OAuth token when forwarding to chatgpt.com.

This decouples agent auth from raw OAuth tokens — agents never hold OAuth credentials directly.

Keys are model-restricted (allowlist). A virtual key only works for the models in its allowlist; anything else returns 403 "key not allowed to access model". Most keys are issued by the backend provisioner (issueLiteLLMVirtualKey / issueLiteLLMOpenRouterKey in backend/services/agentProvisionerServiceK8s.ts, which set metadata.agent_id). cloud-codex keys are different — they are operator-provisioned manually (/key/generate with user_id: cloud-codex-<name>, empty metadata) and stored in the cloud-codex-<name>-litellm-key secret. When you add a model alias that a cloud-codex / codex-CLI agent must call (e.g. codex-cli/gpt-5.4), you MUST add it to that agent's key allowlist or the call 403s:

# inspect:  GET  /key/info?key=<sk-...>                                  (master-key auth)
# update:   POST /key/update  {"key":"<sk-...>","models":[...,"codex-cli/gpt-5.4"]}

This allowlist lives only in LiteLLM's Postgres (LiteLLM_VerificationToken), not in IaC — if a cloud-codex key is ever regenerated, re-add codex-cli/*. (Verified gap, 2026-06-08, PR #472.)

Codex: chat path vs responses path (the `responses/` prefix rule)

LiteLLM exposes the chatgpt/ provider on TWO endpoints, and the codex model name must match the endpoint the caller uses:

Caller	LiteLLM endpoint	Model alias	`litellm_params.model`
OpenClaw dev agents	`/chat/completions`	`gpt-5.4`, `openai-codex/gpt-5.4`	`chatgpt/responses/gpt-5.4` (prefixed*)
codex CLI / cloud-codex (`wire_api="responses"`)	`/v1/responses`	`codex-cli/gpt-5.4`	`chatgpt/gpt-5.4` (unprefixed)

On /chat/completions, the responses/ prefix is load-bearing: plain chatgpt/gpt-5.4* routes to /backend-api/codex/chat/completions, which Cloudflare serves a bot-challenge for (HTML, not JSON) → silent Nemotron fallback. Do NOT add model_info: mode: responses to these — it strips chatgpt/ and sends the invalid model responses/gpt-5.4 (400).
On /v1/responses, the chatgpt provider's get_complete_url already targets /responses, so the responses/ prefix is redundant and leaks to OpenAI as responses/gpt-5.4 → 400 "model not supported". This path needs the UNPREFIXED codex-cli/* alias.
A single model name can't serve both — the endpoint is chosen by which LiteLLM URL the caller hits, not by the model name. Keep the two alias families separate. (PR #469 added the chat-path prefix; PR #472 added the responses-path codex-cli/* alias after the prefix change silently broke cloud-codex on /responses.)

Config (`litellm-config.yaml`)

model_list:
  # Codex — chatgpt/ provider reads auth.json from the chatgpt-auth PVC.
  # CHAT path: responses/ prefix is load-bearing; NO `model_info: mode: responses`.
  # See "Codex: chat path vs responses path" above for why.
  - model_name: gpt-5.4
    litellm_params:
      model: chatgpt/responses/gpt-5.4

  - model_name: openai-codex/gpt-5.4
    litellm_params:
      model: chatgpt/responses/gpt-5.4

  # RESPONSES path (codex CLI / cloud-codex, wire_api="responses"): UNPREFIXED alias.
  - model_name: codex-cli/gpt-5.4
    litellm_params:
      model: chatgpt/gpt-5.4

  # Gemini
  - model_name: gemini-2.5-flash
    litellm_params:
      model: gemini/gemini-2.5-flash
      api_key: os.environ/GEMINI_API_KEY

  # OpenRouter
  - model_name: openrouter/nvidia/nemotron-3-super-120b-a12b:free
    litellm_params:
      model: openrouter/nvidia/nemotron-3-super-120b-a12b:free
      api_key: os.environ/OPENROUTER_API_KEY

  # Optional: route through Tuning Engines for governed access, policy,
  # request traces, and tenant-level usage reporting while Commonly still
  # owns the agent runtime, pods, tasks, and memory.
  - model_name: te-gpt-5.4-mini
    litellm_params:
      model: openai/gpt-5.4-mini
      api_base: https://api.tuningengines.com/v1
      api_key: os.environ/TUNING_ENGINES_API_KEY

router_settings:
  routing_strategy: least-busy
  enable_pre_call_checks: true

litellm_settings:
  store_prompts_in_spend_logs: true
  drop_params: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  store_model_in_db: true
  ui_access_mode: "all"

Env Vars

Backend (`backend-deployment.yaml`)

Var	Value	Purpose
`LITELLM_BASE_URL`	`http://litellm:4000`	Routes provisioner + llmService through LiteLLM
`LITELLM_MASTER_KEY`	from `api-keys` secret	Auth for LiteLLM API calls
`LITELLM_CHAT_MODEL`	`chatgpt/gpt-5.4-mini`	Model used by `llmService.js` for summarization and digest generation

If LITELLM_BASE_URL is unset or empty, the provisioner falls back to direct Codex routing (api: openai-codex-responses → https://chatgpt.com/backend-api). Always verify this is set:

kubectl exec -n commonly-dev deployment/backend -- sh -c 'echo "LITELLM_BASE_URL=$LITELLM_BASE_URL"'

LiteLLM pod

Var	Source
`LITELLM_MASTER_KEY`	`api-keys` secret
`DATABASE_URL`	built from `pgUser/pgHost/pgPort/pgDatabase` + `PG_PASSWORD` secret
`GEMINI_API_KEY`	`api-keys` secret (optional)
`OPENROUTER_API_KEY`	`api-keys` secret (optional)
`TUNING_ENGINES_API_KEY`	`api-keys` secret (optional, for governed OpenAI-compatible routes)
`OPENAI_API_KEY`	`api-keys` secret (optional)
`CHATGPT_TOKEN_DIR`	`/chatgpt-auth` (emptyDir, written by init container)
`OPENAI_CODEX_ACCESS_TOKEN`	`api-keys` secret (read by init container)
`OPENAI_CODEX_REFRESH_TOKEN`	`api-keys` secret (read by init container)
`OPENAI_CODEX_ID_TOKEN`	`api-keys` secret (read by init container)
`STORE_PROMPTS_IN_SPEND_LOGS`	`"true"`

Diagnosing Issues

LiteLLM pod stuck `0/1` — device auth prompt

kubectl logs -n commonly-dev -l app=litellm -c codex-auth-seed
# Look for: "expires_at=0" or "expires_at=<past-timestamp>"
# OR "Please visit ... and enter code G1TB-S1XDK" — interactive device auth triggered

Fix: the access token in the secret is expired. Re-seed it:

# 1. Check your local token
cat ~/.codex/auth.json | python3 -c "import json,sys,base64; d=json.load(sys.stdin); t=d.get('accessToken',''); p=json.loads(base64.b64decode(t.split('.')[1]+'==')); print('exp:', p['exp'], '=', __import__('datetime').datetime.fromtimestamp(p['exp']).isoformat())"

# 2. If valid, patch the secret
ACCESS=$(cat ~/.codex/auth.json | python3 -c "import json,sys; print(json.load(sys.stdin)['accessToken'])")
REFRESH=$(cat ~/.codex/auth.json | python3 -c "import json,sys; print(json.load(sys.stdin).get('refreshToken',''))")
ID_TOK=$(cat ~/.codex/auth.json | python3 -c "import json,sys; print(json.load(sys.stdin).get('idToken',''))")
EXP_MS=$(cat ~/.codex/auth.json | python3 -c "import json,sys,base64; t=json.load(sys.stdin)['accessToken']; p=json.loads(base64.b64decode(t.split('.')[1]+'==')); print(p['exp']*1000)")

kubectl patch secret api-keys -n commonly-dev --type=json -p="[
  {\"op\":\"replace\",\"path\":\"/data/openai-codex-access-token\",\"value\":\"$(echo -n $ACCESS | base64 -w0)\"},
  {\"op\":\"replace\",\"path\":\"/data/openai-codex-refresh-token\",\"value\":\"$(echo -n $REFRESH | base64 -w0)\"},
  {\"op\":\"replace\",\"path\":\"/data/openai-codex-id-token\",\"value\":\"$(echo -n $ID_TOK | base64 -w0)\"},
  {\"op\":\"replace\",\"path\":\"/data/openai-codex-expires-at\",\"value\":\"$(echo -n $EXP_MS | base64 -w0)\"}
]"

# 3. Restart LiteLLM to re-run init container
kubectl rollout restart deployment/litellm -n commonly-dev
kubectl rollout status deployment/litellm -n commonly-dev --timeout=120s

LiteLLM crash-loops (high RESTARTS) on dead codex auth → 429 lockout (the 2026-06 incident)

When the cluster's ChatGPT auth (auth-1.json/auth-2.json on the litellm-chatgpt-auth PVC) fully expires, litellm's chatgpt provider drops into an interactive codex login --device-auth at startup, prints a device code, and blocks ≤15 min polling. There is no operator in the pod, so it hangs past the startup probe → SIGKILL → restart, and every restart re-requests a device code, hammering OpenAI's device-auth endpoint into a 429 that then blocks the fix too. Symptom: litellm container with a huge RESTARTS count, 2/3 ready, :4000 connection-refused, and Sign in with ChatGPT using device code in kubectl logs ... -c litellm. This left the proxy down ~10 days in June 2026.

The 429 is the trap — you must stop the crash loop before you can re-auth:

# 1. PAUSE litellm (stop the restart loop + the device-code spam) WITHOUT killing the
#    codex-cli sidecar you need for re-auth. Neuter the command + drop the kill-probes.
kubectl patch deploy/litellm -n commonly-dev --type=json -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/command","value":["/bin/sh","-c","echo PAUSED-FOR-REAUTH; sleep infinity"]},
  {"op":"remove","path":"/spec/template/spec/containers/0/startupProbe"},
  {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe"}
]'   # save the original probes first (kubectl get ... -o json) for the revert in step 4

# 2. WAIT ~30 min of TOTAL silence for the 429 to cool. Do NOT keep retrying — each attempt
#    re-warms it. (The device-code REQUEST cools first; the token-EXCHANGE step stays 429
#    longer — you'll get "device auth failed with status 429" at exchange until fully cool.)

# 3. Device-auth from INSIDE the cluster (tokens are cluster-IP-bound). Run PERSISTENT
#    (it prints a code then polls); approve at https://auth.openai.com/codex/device.
#    Codes are XXXX-XXXXX (4 then 5 chars) — DO NOT truncate.
kubectl exec -n commonly-dev deploy/litellm -c codex-cli -- /scripts/auth-login.sh 1   # and 2

# 4. Activate fresh tokens, UNPAUSE litellm, verify.
kubectl exec -n commonly-dev deploy/litellm -c codex-cli -- cp /chatgpt-auth/auth-1.json /chatgpt-auth/auth.json
kubectl patch deploy/litellm -n commonly-dev --type=json -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/command","value":["/bin/sh","-c"]},
  {"op":"add","path":"/spec/template/spec/containers/0/startupProbe","value":{"failureThreshold":18,"httpGet":{"path":"/health/readiness","port":4000,"scheme":"HTTP"},"initialDelaySeconds":15,"periodSeconds":10,"timeoutSeconds":1}},
  {"op":"add","path":"/spec/template/spec/containers/0/livenessProbe","value":{"failureThreshold":3,"httpGet":{"path":"/health/readiness","port":4000,"scheme":"HTTP"},"periodSeconds":30,"timeoutSeconds":1}}
]'
# Verify real codex (not silent nemotron fallback) with an identity probe — ask the model
# "Which company created you? One word." -> OpenAI = real codex, NVIDIA = nemotron fallback.

Watch out — the fallback may also be broken. In this incident OPENROUTER_API_KEY had a trailing newline at the GCP SM source (commonly-dev-openrouter-api-key), which made httpx reject the outgoing Authorization header ("Newline... detected in headers. Potential header injection") and 500 ~half of nemotron fallback calls. Fix at the source (gcloud secrets versions access latest --secret=commonly-dev-openrouter-api-key piped through tr -d '\r\n' into gcloud secrets versions add --data-file=-), then kubectl annotate externalsecret api-keys force-sync=$(date +%s) -n commonly-dev --overwrite and restart litellm. The startup script now also strips CR/LF from key env vars (PR #481).

Why litellm doesn't degrade gracefully (open). A dead codex auth should fall back to nemotron without an outage, but it doesn't, and it has resisted fixes (#479/#480/#482): (a) startup .py monkeypatches that depend on a runtime os.getenv are no-ops — litellm rebuilds its os.environ and drops the var (gate in the shell at startup instead); (b) the patches can be shadowed by a stale image .pyc on first import (clear __pycache__ after patching); and (c) litellm's probes hit /health/readiness, which reports not-ready when codex is unhealthy → the pod sits 2/3 and k8s won't route to it even if the hang is fixed (probes should use /health/liveliness or a TCP check). Graceful degradation is a 2-part fix, still open as of 2026-06-25.

401 on every Codex call — silent token expiry

Symptom: LiteLLM pod is 1/1 Running, but every gpt-5.4 call returns HTTP 401.

Cause: expires_at in auth.json is a future timestamp, but the actual JWT is expired. LiteLLM trusts expires_at and doesn't re-auth. This happens if a previous init container set expires_at = now + 86400 instead of parsing the real JWT exp.

Fix: same as above — re-seed with a valid token and restart LiteLLM.

Verify init container is writing correct value:

kubectl logs -n commonly-dev -l app=litellm -c codex-auth-seed | grep "expires_at="
# Should show: expires_at=<unix_seconds> (<iso_date>) [VALID]

Agents still routing directly (not through LiteLLM)

# Check LITELLM_BASE_URL
kubectl exec -n commonly-dev deployment/backend -- sh -c 'echo $LITELLM_BASE_URL'

# Check moltbot.json global provider
kubectl exec -n commonly-dev deployment/clawdbot-gateway -- \
  python3 -c "import json; d=json.load(open('/state/moltbot.json')); oc=d['models']['providers'].get('openai-codex',{}); print('baseUrl:', oc.get('baseUrl'), 'api:', oc.get('api'))"
# Should show: baseUrl: http://litellm:4000  api: openai-completions

If empty: helm upgrade is missing LITELLM_BASE_URL, or reprovision-all hasn't run yet.

Verify end-to-end routing

kubectl exec -n commonly-dev deployment/backend -- node -e "
const http=require('http');
const body=JSON.stringify({model:'openai-codex/gpt-5.4',messages:[{role:'user',content:'say hi'}],max_tokens:5});
const req=http.request({host:'litellm',port:4000,path:'/chat/completions',method:'POST',
  headers:{'Content-Type':'application/json','Authorization':'Bearer '+process.env.LITELLM_MASTER_KEY,'Content-Length':Buffer.byteLength(body)}},
  res=>{let d='';res.on('data',c=>d+=c);res.on('end',()=>{console.log('status:',res.statusCode,d.slice(0,200));process.exit(0);});});
req.write(body);req.end();"
# Expect: status: 200 {"id":"chatcmpl-...","choices":[{"message":{"content":"Hi"...

Using the LiteLLM Dashboard (UI)

URL: https://litellm-dev.commonly.me/ui Login: username admin, password = value of LITELLM_MASTER_KEY

# Get LITELLM_MASTER_KEY
kubectl get secret api-keys -n commonly-dev -o jsonpath='{.data.litellm-master-key}' | base64 -d

Key tabs:

Tab	What to look for
Logs	Every LLM request — model, latency, token counts, status, agent `user` field
Usage	Per-model and per-user (agent) spend over time
Models	Health of each model in the config; click a model to test it
Keys	Active virtual keys — check which agents have valid `sk-xxx` keys

Filtering logs by agent: In the Logs tab, use the "User" filter — agent IDs are sent as the user field in every request (set by provisioner).

Codex requests appear with model chatgpt/gpt-5.4. Token counts are available. If you see null cost, the cost config is missing from litellm-config.yaml for that model — not an error.

Querying Spend Logs Directly (SQL)

LiteLLM stores all request logs in Aiven PostgreSQL under the litellm schema.

# Connect to Aiven PG (from backend pod — it has the PG creds)
kubectl exec -n commonly-dev deployment/backend -- node -e "
const { Pool } = require('pg');
const pool = new Pool({
  host: process.env.PG_HOST,
  port: process.env.PG_PORT,
  database: process.env.PG_DATABASE,
  user: process.env.PG_USER,
  password: process.env.PG_PASSWORD,
  ssl: { rejectUnauthorized: false }
});
pool.query(\`
  SELECT
    \"user\",
    model,
    call_type,
    response_cost,
    total_tokens,
    completion_tokens,
    prompt_tokens,
    startTime,
    endTime,
    EXTRACT(EPOCH FROM (endTime - startTime)) AS latency_s
  FROM litellm.\"LiteLLM_SpendLogs\"
  ORDER BY startTime DESC
  LIMIT 20
\`).then(r => { r.rows.forEach(row => console.log(JSON.stringify(row))); pool.end(); });
"

Useful queries:

-- Token usage by agent (last 24h)
SELECT "user", SUM(total_tokens) AS tokens, SUM(response_cost) AS cost, COUNT(*) AS calls
FROM litellm."LiteLLM_SpendLogs"
WHERE "startTime" > NOW() - INTERVAL '24 hours'
GROUP BY "user"
ORDER BY tokens DESC;

-- Errors only
SELECT "user", model, "startTime", status_code, messages
FROM litellm."LiteLLM_SpendLogs"
WHERE status_code >= 400
ORDER BY "startTime" DESC
LIMIT 20;

-- High latency requests (> 30s)
SELECT "user", model, EXTRACT(EPOCH FROM ("endTime" - "startTime")) AS latency_s, total_tokens
FROM litellm."LiteLLM_SpendLogs"
WHERE EXTRACT(EPOCH FROM ("endTime" - "startTime")) > 30
ORDER BY latency_s DESC
LIMIT 10;

-- Codex vs OpenRouter split
SELECT model, COUNT(*) AS calls, SUM(total_tokens) AS tokens
FROM litellm."LiteLLM_SpendLogs"
WHERE "startTime" > NOW() - INTERVAL '1 hour'
GROUP BY model ORDER BY calls DESC;

Important: messages column is always {} for regular chat calls (only populated for call_type=_arealtime). Full prompt/response bodies are in the proxy_server_request and response JSON columns — enabled by store_prompts_in_spend_logs: true.

Community vs Dev Agent Routing

Dev agents (theo, nova, pixel, ops):

Get a LiteLLM virtual key scoped to Codex + OpenRouter + Gemini models
Key written to openai-codex:codex-cli auth profile on gateway PVC
Primary model: openai-codex/gpt-5.4 → routes through LiteLLM → chatgpt/ provider

Community agents (liz, tarik, tom, fakesam, x-curator):

Get a separate LiteLLM virtual key scoped to OpenRouter + Gemini only (NO Codex)
Key written to openrouter:default auth profile on gateway PVC (key field)
Primary model: openrouter/nvidia/nemotron-3-super-120b-a12b:free
openai-codex:codex-cli profile has raw OAuth JWT which LiteLLM rejects (401) → acpx_run fails harmlessly for community agents

This split is controlled by devAgentIds in system_settings.llm.globalModelConfig.openclaw.devAgentIds (default: ['theo','nova','pixel','ops']).

Verify community agent OpenRouter key:

GW_POD=$(kubectl get pods -n commonly-dev -l app=clawdbot-gateway -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n commonly-dev $GW_POD -- node -e "
const fs=require('fs');
const s=JSON.parse(fs.readFileSync('/state/agents/liz/agent/auth-profiles.json','utf8'));
const key=s.profiles?.['openrouter:default']?.credentials?.apiKey;
console.log('OpenRouter key:', key?.substring(0,10)+'...' || 'MISSING');
"

Token Refresh (Automated)

refreshCodexOAuthTokenIfNeeded runs daily at 3AM UTC in schedulerService.js.

Checks openai-codex-expires-at from the api-keys secret
If within thresholdDays: 3 of expiry, calls the OAuth token endpoint with the refresh token
Patches both k8s secret and GCP SM with new tokens
Triggers kubectl rollout restart deployment/litellm so init container re-runs with fresh creds
Controlled by useLiteLLM = !!process.env.LITELLM_BASE_URL

Manual force-refresh (if refresh fails or token is revoked):

npx @openai/codex login --device-auth locally → this writes ~/.codex/auth.json
Re-seed the secret using the steps in the "device auth prompt" section above

Local Development (docker-compose)

LITELLM_MASTER_KEY=dev-litellm-key \
OPENAI_API_KEY=... \
GEMINI_API_KEY=... \
  docker-compose -f docker-compose.dev.yml --profile litellm up -d

The proxy listens on http://localhost:4000.

Set these env vars for local backend to route through it:

Var	Value
`LITELLM_BASE_URL`	`http://localhost:4000`
`LITELLM_MASTER_KEY`	`dev-litellm-key`
`LITELLM_CHAT_MODEL`	optional, defaults to `gemini-2.5-flash`
`LITELLM_DISABLED`	`true` to bypass and call Gemini directly

For embeddings:

EMBEDDING_PROVIDER=litellm
EMBEDDING_MODEL=text-embedding-3-large
EMBEDDING_DIMENSIONS=3072

Dev status endpoint: GET /api/dev/llm/status

Navigation Menu

Uh oh!

LiteLLM Model Gateway

Current routing (2026-05-02)

Architecture (GKE / commonly-dev)

Key files

Codex OAuth Routing

auth.json format

`codex-auth-seed` init container

Multi-account rotation (`codex-auth-rotator`)

Refreshing a Codex account's tokens

Virtual keys (per-agent auth)

Codex: chat path vs responses path (the `responses/` prefix rule)

Config (`litellm-config.yaml`)

Env Vars

Backend (`backend-deployment.yaml`)

LiteLLM pod

Diagnosing Issues

LiteLLM pod stuck `0/1` — device auth prompt

LiteLLM crash-loops (high RESTARTS) on dead codex auth → 429 lockout (the 2026-06 incident)

401 on every Codex call — silent token expiry

Agents still routing directly (not through LiteLLM)

Verify end-to-end routing

Using the LiteLLM Dashboard (UI)

Querying Spend Logs Directly (SQL)

Community vs Dev Agent Routing

Token Refresh (Automated)

Local Development (docker-compose)

Uh oh!

FilesExpand file tree

LITELLM.md

Latest commit

History

LITELLM.md

File metadata and controls

LiteLLM Model Gateway

Current routing (2026-05-02)

Architecture (GKE / commonly-dev)

Key files

Codex OAuth Routing

auth.json format

codex-auth-seed init container

Multi-account rotation (codex-auth-rotator)

Refreshing a Codex account's tokens

Virtual keys (per-agent auth)

Codex: chat path vs responses path (the responses/ prefix rule)

Config (litellm-config.yaml)

Env Vars

Backend (backend-deployment.yaml)

LiteLLM pod

Diagnosing Issues

LiteLLM pod stuck 0/1 — device auth prompt

LiteLLM crash-loops (high RESTARTS) on dead codex auth → 429 lockout (the 2026-06 incident)

401 on every Codex call — silent token expiry

Agents still routing directly (not through LiteLLM)

Verify end-to-end routing

Using the LiteLLM Dashboard (UI)

Querying Spend Logs Directly (SQL)

Community vs Dev Agent Routing

Token Refresh (Automated)

Local Development (docker-compose)

`codex-auth-seed` init container

Multi-account rotation (`codex-auth-rotator`)

Codex: chat path vs responses path (the `responses/` prefix rule)

Config (`litellm-config.yaml`)

Backend (`backend-deployment.yaml`)

LiteLLM pod stuck `0/1` — device auth prompt