From 5d5dbd4e6c6655397fd3b33ca350480611e2cfcd Mon Sep 17 00:00:00 2001 From: odinwang <48194321+odinwang@users.noreply.github.com> Date: Thu, 1 Oct 2026 18:30:31 +0800 Subject: [PATCH] Docs: lead with the agentcore_gateway backend; diagram the private path README no longer describes llm-edge as the only way to keep the model key out of containers. It now presents both gateway modes, with agentcore_gateway (AgentCore Gateway in front of LiteLLM, key in the token vault, per-session SigV4) first, and llm-edge plus the fixed NAT EIP as the alternative. The quick start's step 2 branches by backend instead of always storing a key for llm-edge. New diagram docs/images/agentcore-gateway-litellm.svg and an architecture.md section that walks every hop: loopback shim and per-session STS, the gateway interface endpoint, what the inference provider target does (match path and model, swap the key, relay verbatim, no translation), and the managed VPC Lattice private endpoint, including what provider.endpoint and routingDomain each do and the 350-second Lattice idle timeout that hangs long non-streaming calls. The hero diagram caption and deployment.md Option A2 point at it. Co-Authored-By: Claude Opus 5.5 (1M context) --- README.md | 48 ++++-- docs/architecture.md | 48 ++++++ docs/deployment.md | 4 + docs/images/agentcore-gateway-litellm.svg | 191 ++++++++++++++++++++++ docs/images/architecture.svg | 2 +- 5 files changed, 277 insertions(+), 16 deletions(-) create mode 100644 docs/images/agentcore-gateway-litellm.svg diff --git a/README.md b/README.md index 5b20fb9..118de19 100644 --- a/README.md +++ b/README.md @@ -14,19 +14,31 @@ It shows how a platform team can offer two hosting models behind one portal: `/invocations` contract. Both kernels route model traffic through a configurable **LLM gateway** -(e.g. LiteLLM) with a **fixed egress IP** (VPC mode + NAT Gateway), so the -platform works in enterprises that enforce model allow-lists, budgets and -source-IP restrictions. The gateway key lives in a platform-side service and -**never enters a session container** — a session's user is root in its own -microVM, so a kernel gets a short-lived, per-session grant rather than a -credential. Direct Bedrock access (cross-region inference) is supported as an +(e.g. LiteLLM), so the platform works in enterprises that enforce model +allow-lists, budgets and cost attribution. The gateway key **never enters a +session container** — a session's user is root in its own microVM, so a +kernel gets a short-lived, per-session grant rather than a credential. Two +ways to get there: + +- **AgentCore Gateway in front of LiteLLM** (`agentcore_gateway`, + recommended). The LiteLLM key lives in AgentCore Identity's token vault, so + the platform holds no key and runs no broker service. Kernels SigV4-sign + with per-session STS credentials (revoked by an IAM Deny on the session + tag); runtime → gateway can stay on PrivateLink and gateway → LiteLLM on a + managed VPC Lattice path, so LiteLLM needs no public endpoint. + [How it works](docs/architecture.md#how-the-agentcore_gateway-backend-reaches-a-private-litellm). +- **`llm-edge`** (`litellm`). A platform-side service holds the key and + forwards session-scoped calls; runtimes egress through a NAT Gateway with a + **fixed EIP** you can put on the gateway's source-IP allow-list. + +Direct Bedrock access (cross-region inference) is supported as an alternative. ![portal overview](docs/images/portal-overview.png) -![architecture](docs/images/architecture.svg?v=3) +![architecture](docs/images/architecture.svg?v=4) The hero diagram above is the map; two focused diagrams zoom into the parts that carry the security story: the @@ -35,7 +47,10 @@ that carry the security story: the traffic) and the [customer-owned MCP hub chains](docs/images/mcp-hub-chains.svg) (how production applications and the Dev Workbench reach a self-hosted tool -backend with per-application HMAC signatures and a forwarded user token). +backend with per-application HMAC signatures and a forwarded user token). A third, the +[agentcore_gateway model path](docs/images/agentcore-gateway-litellm.svg), +shows how a kernel reaches a private LiteLLM through AgentCore Gateway over +PrivateLink and VPC Lattice. ## What's inside @@ -46,7 +61,7 @@ backend with per-application HMAC signatures and a forwarded user token). │ └── mcp-tools-kernel/ # Demo MCP server (protocol=MCP): mock internal tools on AgentCore Runtime ├── backend/ # FastAPI control plane: sessions, terminal URLs, kernel catalog, MCP/skill registry, workflow engine ├── frontend/ # React portal: Workbench, Publish, Debug, Scheduler, MCP & Skills, Gateway, Channels, Memory, Observability, Eval, Workflow, Governance -├── services/ # llm-edge (sole holder of the gateway key) + the optional Keycloak IdP and team APIs +├── services/ # llm-edge (key holder for the `litellm` backend) + the optional Keycloak IdP and team APIs ├── terraform/ # Terraform (the maintained path): network, platform resources, AgentCore runtimes, EKS, portal hosting + scheduler engine ├── infrastructure/ # CDK (Python): the legacy ECS Fargate variant of the same stacks, kept for reference ├── deploy-cli/ # AWS-CLI-only deployment port for accounts that cannot run Terraform or CDK @@ -151,24 +166,27 @@ cp terraform.tfvars.example terraform.tfvars # then edit terraform init terraform apply -var enable_runtime=false -var enable_portal=false -# 2. Store your LLM gateway key (skip for Bedrock direct or AgentCore Gateway). -# Readable only by the llm-edge service; it never enters a kernel container. -# Gateway mode also needs enable_llm_edge=true — see docs/deployment.md §2, -# which also covers the agentcore_gateway backend (no key on this side). +# 2. Model backend. Pick one (docs/deployment.md §2): +# - agentcore_gateway: create the AgentCore Gateway + LiteLLM inference +# target and the per-session caller role (Option A2). No key on this side; +# set enable_gateway_vpce = true to keep runtime -> gateway on PrivateLink. +# - litellm via llm-edge: enable_llm_edge = true and store the key, which is +# readable only by llm-edge and never enters a kernel container: aws secretsmanager put-secret-value \ --secret-id agent-platform/llm-gateway-key \ --secret-string '{"api_key":"sk-..."}' +# - Bedrock direct: nothing to store. # 3. Build & push the images (ARM64) ../scripts/build-and-push.sh -# 4. Allow-list the NAT EIP on your LLM gateway, then create the AgentCore +# 4. (llm-edge only) allow-list the NAT EIP on your LLM gateway. Then create the AgentCore # runtimes (VPC mode, fixed egress IP), the EKS cluster and the portal terraform apply # or: run backend + frontend locally, see docs/deployment.md ../scripts/deploy-frontend.sh ``` -The containers (backend, llm-edge, and the optional Keycloak + team APIs) run +The containers (backend, llm-edge if enabled, and the optional Keycloak + team APIs) run on a dedicated EKS cluster with Graviton nodes; every pod authenticates to AWS through IRSA and carries its own security group. The CDK stacks in `infrastructure/` are the legacy ECS Fargate variant, kept for reference. diff --git a/docs/architecture.md b/docs/architecture.md index 94a47fd..89ac5a5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -285,6 +285,54 @@ variables (clearing families the catalog lacks). This keeps Claude Code's it, a gateway session would inherit the container's baked-in Bedrock profile IDs, which the gateway rejects. +#### How the `agentcore_gateway` backend reaches a private LiteLLM + +![agentcore_gateway backend in front of a private LiteLLM](images/agentcore-gateway-litellm.svg) + +Every hop is inside AWS; neither the gateway nor LiteLLM needs a public +endpoint, and no LiteLLM key exists anywhere on the platform side. + +1. **Kernel → shim.** Claude Code (or the Agent SDK) points + `ANTHROPIC_BASE_URL` at the kernel's loopback shim with a per-invocation + local token. The shim holds the session's STS credential — minted by the + backend at connect via `AssumeRole` with a `session_id` tag and a session + policy that allows `InvokeGateway` on this one gateway — and SigV4-signs + each request. Revoking a session is an IAM Deny conditioned on that tag. +2. **Runtime → gateway over PrivateLink.** With `enable_gateway_vpce` the + platform VPC has a `bedrock-agentcore.gateway` interface endpoint with + private DNS, so the gateway's normal URL resolves to endpoint ENIs and the + call never touches the NAT. +3. **Gateway: match, swap, relay.** The IAM authorizer checks the signature + and session policy. The inference *provider* target matches the request + path (`/v1/messages`, `/v1/chat/completions` or `/v1/responses` — the + only three it accepts) and the body's `model` against its declared + models, replaces `Authorization` with the LiteLLM virtual key held in + AgentCore Identity's token vault, and relays the body and the SSE stream + **verbatim**. The gateway translates nothing; which model works on which + path is LiteLLM's decision. +4. **Gateway → LiteLLM over VPC Lattice.** The target's top-level + `privateEndpoint.managedVpcResource` makes AgentCore place a VPC Lattice + resource gateway (ENIs plus a security group you choose) in the LiteLLM + VPC's subnets. Two names do two jobs: + - `provider.endpoint` (`https://`) is the name on the ALB's + **publicly trusted** certificate. It is sent as TLS SNI and `Host`, so + the handshake succeeds; it needs no public DNS record. + - `routingDomain` is the **internal ALB's DNS name**: where Lattice + actually connects. +5. **ALB → LiteLLM.** The internal ALB terminates TLS and forwards to the + LiteLLM pods. `drop_params` on the LiteLLM model entries strips the + first-party-only fields Claude Code sends (`context_management`), which + is why no gateway interceptor is needed (one would also cap request + bodies at about 4.5 MB, Lambda's invoke payload after base64). + +**The one limit that changes client behaviour:** VPC Lattice closes a TCP +connection that carries no data for **350 seconds** (VPC Lattice quotas, +"Connection idle time per connection for VPC Lattice resources"), and the +gateway does not report it — a non-streaming call whose response takes +longer simply never returns. Streaming keeps bytes on the wire and is not +affected (a 10-minute stream completes). Claude Code and the SDK kernel +always stream; other clients on this path must too. + The gateway pattern is why **networking matters** (next section): enterprise gateways typically enforce source-IP allow-lists. diff --git a/docs/deployment.md b/docs/deployment.md index 34913da..ef902f5 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -112,6 +112,10 @@ multi-provider routing, cost accounting and model catalog exactly as it does for every other client — this option changes only how our platform reaches LiteLLM, not what LiteLLM does behind it. +How each hop works, including the VPC Lattice private path and its 350-second +idle limit, is drawn and explained in +[architecture.md](architecture.md#how-the-agentcore_gateway-backend-reaches-a-private-litellm). + 1. **Mint a LiteLLM virtual key** scoped to the models this backend will serve. The Prod-LiteLLM master key stays where it is; the key stored in the vault below is the virtual one, so a revocation there does not touch diff --git a/docs/images/agentcore-gateway-litellm.svg b/docs/images/agentcore-gateway-litellm.svg new file mode 100644 index 0000000..427aaf1 --- /dev/null +++ b/docs/images/agentcore-gateway-litellm.svg @@ -0,0 +1,191 @@ + +agentcore_gateway model backend · AgentCore Gateway in front of a private LiteLLM + + + + + + +agentcore_gateway model backend · AgentCore Gateway in front of a private LiteLLM +No model key on the platform side or in any container · per-session SigV4 in, API key injected out · runtime → gateway over PrivateLink, gateway → LiteLLM over a managed VPC Lattice resource gateway + +AWS account · Region + +Platform VPC · private subnets + +AgentCore Runtime microVM + +AgentCore service + +LiteLLM VPC · private subnets + + + + + + + + + + + +loopback token + +SigV4 /inference/v1/* + +PrivateLink + +Bearer · LiteLLM key + +:443 + +:4000 + +grant at connect + +authorize + +GetResourceApiKey + + + + + + + + + + + + + + + + +Claude Code / SDKbase URL = 127.0.0.1 +kernel shimSigV4 · per-session STS + + + + + + +Gateway VPC endpointprivate DNS + + + + + + + + + + +AgentCore Gatewayinference provider target + + + + + + +Lattice resource gatewaymanaged ENIs · 350 s idle timeout + + + + + + + + + + + + + + + + + +internal ALBpublic ACM cert +LiteLLMdrop_params · routing + + + + + + + + + + + +Upstream modelsBedrock · others + + + + + + + + + + + + + + + + + +Backend · STSAssumeRole + session tag + + + + + + + + + + + + + + + + + +Token vaultAPI-key provider · LiteLLM key + + + + + + + + + + + + + + + + + +IAM authorizersession policy · Deny on tag +1 +2 +3 +4 +5 +Flows +request → (SSE streams back verbatim) +response ← +credentials +authorization +1–2 The shim signs each call with the session's STS credential (session policy: InvokeGateway on this gateway only; revoke = IAM Deny on the session tag). With private DNS on the interface endpoint, *.gateway.bedrock-agentcore.<region> resolves to endpoint ENIs: no NAT, no URL change. +3 No protocol translation: the gateway matches path (/v1/messages, /v1/chat/completions, /v1/responses only) and the body's model, replaces Authorization with the vault-held LiteLLM key, and relays the body and SSE stream verbatim. +4–5 provider.endpoint = https://<name on the public ACM cert> (TLS SNI + Host) · privateEndpoint.managedVpcResource.routingDomain = internal ALB DNS (where Lattice connects). The cert name needs no public DNS record. +Limit Lattice closes TCP connections idle for 350 s and the gateway returns no error, so a non-streaming call slower than that hangs: stream instead (Claude Code and the SDK kernel do). LiteLLM drop_params strips first-party-only fields. + \ No newline at end of file diff --git a/docs/images/architecture.svg b/docs/images/architecture.svg index ca507c6..5d233cc 100644 --- a/docs/images/architecture.svg +++ b/docs/images/architecture.svg @@ -329,7 +329,7 @@ reply ← tools/memory scheduling -① portal → ② REST /api/v1 → ③ InvokeAgentRuntime (warmup) → ④ per-session grant → llm-edge → ⑤ NAT egress → LLM gateway · EventBridge Scheduler → Lambda fires schedules (retries + DLQ) · workspace + conversation resume from S3 +① portal → ② REST /api/v1 → ③ InvokeAgentRuntime (warmup) → ④ per-session grant → llm-edge → ⑤ NAT egress → LLM gateway (or via AgentCore Gateway: agentcore-gateway-litellm.svg) · Scheduler → Lambda (retries + DLQ) · workspace + conversation resume from S3 machine callers enter via a private SigV4 API Gateway (per-channel caller allowlist · optional robot SSO token) into a data-plane twin of the backend (management routes not mounted) · AgentCore Memory = last-10-turn replay + cross-session recall · detail: data-plane-split.svg + mcp-hub-chains.svg