diff --git a/README.md b/README.md index 5b20fb9..118de19 100644 --- a/README.md +++ b/README.md @@ -14,19 +14,31 @@ It shows how a platform team can offer two hosting models behind one portal: `/invocations` contract. Both kernels route model traffic through a configurable **LLM gateway** -(e.g. LiteLLM) with a **fixed egress IP** (VPC mode + NAT Gateway), so the -platform works in enterprises that enforce model allow-lists, budgets and -source-IP restrictions. The gateway key lives in a platform-side service and -**never enters a session container** — a session's user is root in its own -microVM, so a kernel gets a short-lived, per-session grant rather than a -credential. Direct Bedrock access (cross-region inference) is supported as an +(e.g. LiteLLM), so the platform works in enterprises that enforce model +allow-lists, budgets and cost attribution. The gateway key **never enters a +session container** — a session's user is root in its own microVM, so a +kernel gets a short-lived, per-session grant rather than a credential. Two +ways to get there: + +- **AgentCore Gateway in front of LiteLLM** (`agentcore_gateway`, + recommended). The LiteLLM key lives in AgentCore Identity's token vault, so + the platform holds no key and runs no broker service. Kernels SigV4-sign + with per-session STS credentials (revoked by an IAM Deny on the session + tag); runtime → gateway can stay on PrivateLink and gateway → LiteLLM on a + managed VPC Lattice path, so LiteLLM needs no public endpoint. + [How it works](docs/architecture.md#how-the-agentcore_gateway-backend-reaches-a-private-litellm). +- **`llm-edge`** (`litellm`). A platform-side service holds the key and + forwards session-scoped calls; runtimes egress through a NAT Gateway with a + **fixed EIP** you can put on the gateway's source-IP allow-list. + +Direct Bedrock access (cross-region inference) is supported as an alternative. ![portal overview](docs/images/portal-overview.png) -![architecture](docs/images/architecture.svg?v=3) +![architecture](docs/images/architecture.svg?v=4) The hero diagram above is the map; two focused diagrams zoom into the parts that carry the security story: the @@ -35,7 +47,10 @@ that carry the security story: the traffic) and the [customer-owned MCP hub chains](docs/images/mcp-hub-chains.svg) (how production applications and the Dev Workbench reach a self-hosted tool -backend with per-application HMAC signatures and a forwarded user token). +backend with per-application HMAC signatures and a forwarded user token). A third, the +[agentcore_gateway model path](docs/images/agentcore-gateway-litellm.svg), +shows how a kernel reaches a private LiteLLM through AgentCore Gateway over +PrivateLink and VPC Lattice. ## What's inside @@ -46,7 +61,7 @@ backend with per-application HMAC signatures and a forwarded user token). │ └── mcp-tools-kernel/ # Demo MCP server (protocol=MCP): mock internal tools on AgentCore Runtime ├── backend/ # FastAPI control plane: sessions, terminal URLs, kernel catalog, MCP/skill registry, workflow engine ├── frontend/ # React portal: Workbench, Publish, Debug, Scheduler, MCP & Skills, Gateway, Channels, Memory, Observability, Eval, Workflow, Governance -├── services/ # llm-edge (sole holder of the gateway key) + the optional Keycloak IdP and team APIs +├── services/ # llm-edge (key holder for the `litellm` backend) + the optional Keycloak IdP and team APIs ├── terraform/ # Terraform (the maintained path): network, platform resources, AgentCore runtimes, EKS, portal hosting + scheduler engine ├── infrastructure/ # CDK (Python): the legacy ECS Fargate variant of the same stacks, kept for reference ├── deploy-cli/ # AWS-CLI-only deployment port for accounts that cannot run Terraform or CDK @@ -151,24 +166,27 @@ cp terraform.tfvars.example terraform.tfvars # then edit terraform init terraform apply -var enable_runtime=false -var enable_portal=false -# 2. Store your LLM gateway key (skip for Bedrock direct or AgentCore Gateway). -# Readable only by the llm-edge service; it never enters a kernel container. -# Gateway mode also needs enable_llm_edge=true — see docs/deployment.md §2, -# which also covers the agentcore_gateway backend (no key on this side). +# 2. Model backend. Pick one (docs/deployment.md §2): +# - agentcore_gateway: create the AgentCore Gateway + LiteLLM inference +# target and the per-session caller role (Option A2). No key on this side; +# set enable_gateway_vpce = true to keep runtime -> gateway on PrivateLink. +# - litellm via llm-edge: enable_llm_edge = true and store the key, which is +# readable only by llm-edge and never enters a kernel container: aws secretsmanager put-secret-value \ --secret-id agent-platform/llm-gateway-key \ --secret-string '{"api_key":"sk-..."}' +# - Bedrock direct: nothing to store. # 3. Build & push the images (ARM64) ../scripts/build-and-push.sh -# 4. Allow-list the NAT EIP on your LLM gateway, then create the AgentCore +# 4. (llm-edge only) allow-list the NAT EIP on your LLM gateway. Then create the AgentCore # runtimes (VPC mode, fixed egress IP), the EKS cluster and the portal terraform apply # or: run backend + frontend locally, see docs/deployment.md ../scripts/deploy-frontend.sh ``` -The containers (backend, llm-edge, and the optional Keycloak + team APIs) run +The containers (backend, llm-edge if enabled, and the optional Keycloak + team APIs) run on a dedicated EKS cluster with Graviton nodes; every pod authenticates to AWS through IRSA and carries its own security group. The CDK stacks in `infrastructure/` are the legacy ECS Fargate variant, kept for reference. diff --git a/docs/architecture.md b/docs/architecture.md index 94a47fd..89ac5a5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -285,6 +285,54 @@ variables (clearing families the catalog lacks). This keeps Claude Code's it, a gateway session would inherit the container's baked-in Bedrock profile IDs, which the gateway rejects. +#### How the `agentcore_gateway` backend reaches a private LiteLLM + +![agentcore_gateway backend in front of a private LiteLLM](images/agentcore-gateway-litellm.svg) + +Every hop is inside AWS; neither the gateway nor LiteLLM needs a public +endpoint, and no LiteLLM key exists anywhere on the platform side. + +1. **Kernel → shim.** Claude Code (or the Agent SDK) points + `ANTHROPIC_BASE_URL` at the kernel's loopback shim with a per-invocation + local token. The shim holds the session's STS credential — minted by the + backend at connect via `AssumeRole` with a `session_id` tag and a session + policy that allows `InvokeGateway` on this one gateway — and SigV4-signs + each request. Revoking a session is an IAM Deny conditioned on that tag. +2. **Runtime → gateway over PrivateLink.** With `enable_gateway_vpce` the + platform VPC has a `bedrock-agentcore.gateway` interface endpoint with + private DNS, so the gateway's normal URL resolves to endpoint ENIs and the + call never touches the NAT. +3. **Gateway: match, swap, relay.** The IAM authorizer checks the signature + and session policy. The inference *provider* target matches the request + path (`/v1/messages`, `/v1/chat/completions` or `/v1/responses` — the + only three it accepts) and the body's `model` against its declared + models, replaces `Authorization` with the LiteLLM virtual key held in + AgentCore Identity's token vault, and relays the body and the SSE stream + **verbatim**. The gateway translates nothing; which model works on which + path is LiteLLM's decision. +4. **Gateway → LiteLLM over VPC Lattice.** The target's top-level + `privateEndpoint.managedVpcResource` makes AgentCore place a VPC Lattice + resource gateway (ENIs plus a security group you choose) in the LiteLLM + VPC's subnets. Two names do two jobs: + - `provider.endpoint` (`https://`) is the name on the ALB's + **publicly trusted** certificate. It is sent as TLS SNI and `Host`, so + the handshake succeeds; it needs no public DNS record. + - `routingDomain` is the **internal ALB's DNS name**: where Lattice + actually connects. +5. **ALB → LiteLLM.** The internal ALB terminates TLS and forwards to the + LiteLLM pods. `drop_params` on the LiteLLM model entries strips the + first-party-only fields Claude Code sends (`context_management`), which + is why no gateway interceptor is needed (one would also cap request + bodies at about 4.5 MB, Lambda's invoke payload after base64). + +**The one limit that changes client behaviour:** VPC Lattice closes a TCP +connection that carries no data for **350 seconds** (VPC Lattice quotas, +"Connection idle time per connection for VPC Lattice resources"), and the +gateway does not report it — a non-streaming call whose response takes +longer simply never returns. Streaming keeps bytes on the wire and is not +affected (a 10-minute stream completes). Claude Code and the SDK kernel +always stream; other clients on this path must too. + The gateway pattern is why **networking matters** (next section): enterprise gateways typically enforce source-IP allow-lists. diff --git a/docs/deployment.md b/docs/deployment.md index 34913da..ef902f5 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -112,6 +112,10 @@ multi-provider routing, cost accounting and model catalog exactly as it does for every other client — this option changes only how our platform reaches LiteLLM, not what LiteLLM does behind it. +How each hop works, including the VPC Lattice private path and its 350-second +idle limit, is drawn and explained in +[architecture.md](architecture.md#how-the-agentcore_gateway-backend-reaches-a-private-litellm). + 1. **Mint a LiteLLM virtual key** scoped to the models this backend will serve. The Prod-LiteLLM master key stays where it is; the key stored in the vault below is the virtual one, so a revocation there does not touch diff --git a/docs/images/agentcore-gateway-litellm.svg b/docs/images/agentcore-gateway-litellm.svg new file mode 100644 index 0000000..427aaf1 --- /dev/null +++ b/docs/images/agentcore-gateway-litellm.svg @@ -0,0 +1,191 @@ + +agentcore_gateway model backend · AgentCore Gateway in front of a private LiteLLM + + + + + + +agentcore_gateway model backend · AgentCore Gateway in front of a private LiteLLM +No model key on the platform side or in any container · per-session SigV4 in, API key injected out · runtime → gateway over PrivateLink, gateway → LiteLLM over a managed VPC Lattice resource gateway + +AWS account · Region + +Platform VPC · private subnets + +AgentCore Runtime microVM + +AgentCore service + +LiteLLM VPC · private subnets + + + + + + + + + + + +loopback token + +SigV4 /inference/v1/* + +PrivateLink + +Bearer · LiteLLM key + +:443 + +:4000 + +grant at connect + +authorize + +GetResourceApiKey + + + + + + + + + + + + + + + + +Claude Code / SDKbase URL = 127.0.0.1 +kernel shimSigV4 · per-session STS + + + + + + +Gateway VPC endpointprivate DNS + + + + + + + + + + +AgentCore Gatewayinference provider target + + + + + + +Lattice resource gatewaymanaged ENIs · 350 s idle timeout + + + + + + + + + + + + + + + + + +internal ALBpublic ACM cert +LiteLLMdrop_params · routing + + + + + + + + + + + +Upstream modelsBedrock · others + + + + + + + + + + + + + + + + + +Backend · STSAssumeRole + session tag + + + + + + + + + + + + + + + + + +Token vaultAPI-key provider · LiteLLM key + + + + + + + + + + + + + + + + + +IAM authorizersession policy · Deny on tag +1 +2 +3 +4 +5 +Flows +request → (SSE streams back verbatim) +response ← +credentials +authorization +1–2 The shim signs each call with the session's STS credential (session policy: InvokeGateway on this gateway only; revoke = IAM Deny on the session tag). With private DNS on the interface endpoint, *.gateway.bedrock-agentcore.<region> resolves to endpoint ENIs: no NAT, no URL change. +3 No protocol translation: the gateway matches path (/v1/messages, /v1/chat/completions, /v1/responses only) and the body's model, replaces Authorization with the vault-held LiteLLM key, and relays the body and SSE stream verbatim. +4–5 provider.endpoint = https://<name on the public ACM cert> (TLS SNI + Host) · privateEndpoint.managedVpcResource.routingDomain = internal ALB DNS (where Lattice connects). The cert name needs no public DNS record. +Limit Lattice closes TCP connections idle for 350 s and the gateway returns no error, so a non-streaming call slower than that hangs: stream instead (Claude Code and the SDK kernel do). LiteLLM drop_params strips first-party-only fields. + \ No newline at end of file diff --git a/docs/images/architecture.svg b/docs/images/architecture.svg index ca507c6..5d233cc 100644 --- a/docs/images/architecture.svg +++ b/docs/images/architecture.svg @@ -329,7 +329,7 @@ reply ← tools/memory scheduling -① portal → ② REST /api/v1 → ③ InvokeAgentRuntime (warmup) → ④ per-session grant → llm-edge → ⑤ NAT egress → LLM gateway · EventBridge Scheduler → Lambda fires schedules (retries + DLQ) · workspace + conversation resume from S3 +① portal → ② REST /api/v1 → ③ InvokeAgentRuntime (warmup) → ④ per-session grant → llm-edge → ⑤ NAT egress → LLM gateway (or via AgentCore Gateway: agentcore-gateway-litellm.svg) · Scheduler → Lambda (retries + DLQ) · workspace + conversation resume from S3 machine callers enter via a private SigV4 API Gateway (per-channel caller allowlist · optional robot SSO token) into a data-plane twin of the backend (management routes not mounted) · AgentCore Memory = last-10-turn replay + cross-session recall · detail: data-plane-split.svg + mcp-hub-chains.svg