Docs: lead with the agentcore_gateway backend; diagram the private Lattice path

README.md

@@ -14,19 +14,31 @@ It shows how a platform team can offer two hosting models behind one portal:
   `/invocations` contract.
 
 Both kernels route model traffic through a configurable **LLM gateway**
-(e.g. LiteLLM) with a **fixed egress IP** (VPC mode + NAT Gateway), so the
-platform works in enterprises that enforce model allow-lists, budgets and
-source-IP restrictions. The gateway key lives in a platform-side service and
-**never enters a session container** — a session's user is root in its own
-microVM, so a kernel gets a short-lived, per-session grant rather than a
-credential. Direct Bedrock access (cross-region inference) is supported as an
+(e.g. LiteLLM), so the platform works in enterprises that enforce model
+allow-lists, budgets and cost attribution. The gateway key **never enters a
+session container** — a session's user is root in its own microVM, so a
+kernel gets a short-lived, per-session grant rather than a credential. Two
+ways to get there:
+
+- **AgentCore Gateway in front of LiteLLM** (`agentcore_gateway`,
+  recommended). The LiteLLM key lives in AgentCore Identity's token vault, so
+  the platform holds no key and runs no broker service. Kernels SigV4-sign
+  with per-session STS credentials (revoked by an IAM Deny on the session
+  tag); runtime → gateway can stay on PrivateLink and gateway → LiteLLM on a
+  managed VPC Lattice path, so LiteLLM needs no public endpoint.
+  [How it works](docs/architecture.md#how-the-agentcore_gateway-backend-reaches-a-private-litellm).
+- **`llm-edge`** (`litellm`). A platform-side service holds the key and
+  forwards session-scoped calls; runtimes egress through a NAT Gateway with a
+  **fixed EIP** you can put on the gateway's source-IP allow-list.
+
+Direct Bedrock access (cross-region inference) is supported as an
 alternative.
 
 ![portal overview](docs/images/portal-overview.png)
 
 <!-- ?v= bumps the URL so GitHub's image cache serves the current diagram
      instead of a stale copy at the same path. Bump it whenever the SVG changes. -->
-![architecture](docs/images/architecture.svg?v=3)
+![architecture](docs/images/architecture.svg?v=4)
 
 The hero diagram above is the map; two focused diagrams zoom into the parts
 that carry the security story: the
@@ -35,7 +47,10 @@ that carry the security story: the
 traffic) and the
 [customer-owned MCP hub chains](docs/images/mcp-hub-chains.svg)
 (how production applications and the Dev Workbench reach a self-hosted tool
-backend with per-application HMAC signatures and a forwarded user token).
+backend with per-application HMAC signatures and a forwarded user token). A third, the
+[agentcore_gateway model path](docs/images/agentcore-gateway-litellm.svg),
+shows how a kernel reaches a private LiteLLM through AgentCore Gateway over
+PrivateLink and VPC Lattice.
 
 ## What's inside
 
@@ -46,7 +61,7 @@ backend with per-application HMAC signatures and a forwarded user token).
 │   └── mcp-tools-kernel/     # Demo MCP server (protocol=MCP): mock internal tools on AgentCore Runtime
 ├── backend/                  # FastAPI control plane: sessions, terminal URLs, kernel catalog, MCP/skill registry, workflow engine
 ├── frontend/                 # React portal: Workbench, Publish, Debug, Scheduler, MCP & Skills, Gateway, Channels, Memory, Observability, Eval, Workflow, Governance
-├── services/                 # llm-edge (sole holder of the gateway key) + the optional Keycloak IdP and team APIs
+├── services/                 # llm-edge (key holder for the `litellm` backend) + the optional Keycloak IdP and team APIs
 ├── terraform/                # Terraform (the maintained path): network, platform resources, AgentCore runtimes, EKS, portal hosting + scheduler engine
 ├── infrastructure/           # CDK (Python): the legacy ECS Fargate variant of the same stacks, kept for reference
 ├── deploy-cli/               # AWS-CLI-only deployment port for accounts that cannot run Terraform or CDK
@@ -151,24 +166,27 @@ cp terraform.tfvars.example terraform.tfvars   # then edit
 terraform init
 terraform apply -var enable_runtime=false -var enable_portal=false
 
-# 2. Store your LLM gateway key (skip for Bedrock direct or AgentCore Gateway).
-#    Readable only by the llm-edge service; it never enters a kernel container.
-#    Gateway mode also needs enable_llm_edge=true — see docs/deployment.md §2,
-#    which also covers the agentcore_gateway backend (no key on this side).
+# 2. Model backend. Pick one (docs/deployment.md §2):
+#    - agentcore_gateway: create the AgentCore Gateway + LiteLLM inference
+#      target and the per-session caller role (Option A2). No key on this side;
+#      set enable_gateway_vpce = true to keep runtime -> gateway on PrivateLink.
+#    - litellm via llm-edge: enable_llm_edge = true and store the key, which is
+#      readable only by llm-edge and never enters a kernel container:
 aws secretsmanager put-secret-value \
   --secret-id agent-platform/llm-gateway-key \
   --secret-string '{"api_key":"sk-..."}'
+#    - Bedrock direct: nothing to store.
 
 # 3. Build & push the images (ARM64)
 ../scripts/build-and-push.sh
 
-# 4. Allow-list the NAT EIP on your LLM gateway, then create the AgentCore
+# 4. (llm-edge only) allow-list the NAT EIP on your LLM gateway. Then create the AgentCore
 #    runtimes (VPC mode, fixed egress IP), the EKS cluster and the portal
 terraform apply               # or: run backend + frontend locally, see docs/deployment.md
 ../scripts/deploy-frontend.sh
 ```
 
-The containers (backend, llm-edge, and the optional Keycloak + team APIs) run
+The containers (backend, llm-edge if enabled, and the optional Keycloak + team APIs) run
 on a dedicated EKS cluster with Graviton nodes; every pod authenticates to AWS
 through IRSA and carries its own security group. The CDK stacks in
 `infrastructure/` are the legacy ECS Fargate variant, kept for reference.

docs/architecture.md

@@ -285,6 +285,54 @@ variables (clearing families the catalog lacks). This keeps Claude Code's
 it, a gateway session would inherit the container's baked-in Bedrock profile
 IDs, which the gateway rejects.
 
+#### How the `agentcore_gateway` backend reaches a private LiteLLM
+
+![agentcore_gateway backend in front of a private LiteLLM](images/agentcore-gateway-litellm.svg)
+
+Every hop is inside AWS; neither the gateway nor LiteLLM needs a public
+endpoint, and no LiteLLM key exists anywhere on the platform side.
+
+1. **Kernel → shim.** Claude Code (or the Agent SDK) points
+   `ANTHROPIC_BASE_URL` at the kernel's loopback shim with a per-invocation
+   local token. The shim holds the session's STS credential — minted by the
+   backend at connect via `AssumeRole` with a `session_id` tag and a session
+   policy that allows `InvokeGateway` on this one gateway — and SigV4-signs
+   each request. Revoking a session is an IAM Deny conditioned on that tag.
+2. **Runtime → gateway over PrivateLink.** With `enable_gateway_vpce` the
+   platform VPC has a `bedrock-agentcore.gateway` interface endpoint with
+   private DNS, so the gateway's normal URL resolves to endpoint ENIs and the
+   call never touches the NAT.
+3. **Gateway: match, swap, relay.** The IAM authorizer checks the signature
+   and session policy. The inference *provider* target matches the request
+   path (`/v1/messages`, `/v1/chat/completions` or `/v1/responses` — the
+   only three it accepts) and the body's `model` against its declared
+   models, replaces `Authorization` with the LiteLLM virtual key held in
+   AgentCore Identity's token vault, and relays the body and the SSE stream
+   **verbatim**. The gateway translates nothing; which model works on which
+   path is LiteLLM's decision.
+4. **Gateway → LiteLLM over VPC Lattice.** The target's top-level
+   `privateEndpoint.managedVpcResource` makes AgentCore place a VPC Lattice
+   resource gateway (ENIs plus a security group you choose) in the LiteLLM
+   VPC's subnets. Two names do two jobs:
+   - `provider.endpoint` (`https://<name>`) is the name on the ALB's
+     **publicly trusted** certificate. It is sent as TLS SNI and `Host`, so
+     the handshake succeeds; it needs no public DNS record.
+   - `routingDomain` is the **internal ALB's DNS name**: where Lattice
+     actually connects.
+5. **ALB → LiteLLM.** The internal ALB terminates TLS and forwards to the
+   LiteLLM pods. `drop_params` on the LiteLLM model entries strips the
+   first-party-only fields Claude Code sends (`context_management`), which
+   is why no gateway interceptor is needed (one would also cap request
+   bodies at about 4.5 MB, Lambda's invoke payload after base64).
+
+**The one limit that changes client behaviour:** VPC Lattice closes a TCP
+connection that carries no data for **350 seconds** (VPC Lattice quotas,
+"Connection idle time per connection for VPC Lattice resources"), and the
+gateway does not report it — a non-streaming call whose response takes
+longer simply never returns. Streaming keeps bytes on the wire and is not
+affected (a 10-minute stream completes). Claude Code and the SDK kernel
+always stream; other clients on this path must too.
+
 The gateway pattern is why **networking matters** (next section): enterprise
 gateways typically enforce source-IP allow-lists.
 

docs/deployment.md

@@ -112,6 +112,10 @@ multi-provider routing, cost accounting and model catalog exactly as it does
 for every other client — this option changes only how our platform reaches
 LiteLLM, not what LiteLLM does behind it.
 
+How each hop works, including the VPC Lattice private path and its 350-second
+idle limit, is drawn and explained in
+[architecture.md](architecture.md#how-the-agentcore_gateway-backend-reaches-a-private-litellm).
+
 1. **Mint a LiteLLM virtual key** scoped to the models this backend will
    serve. The Prod-LiteLLM master key stays where it is; the key stored in
    the vault below is the virtual one, so a revocation there does not touch