Flow D — Lambda MicroVM Agent Sandbox (KRO RGD + ACK, label-branched, suspend/resume)

docs/dark-factory/README.md

@@ -26,6 +26,7 @@ spokes as normal deployments.
 2. [Two flows at a glance](#2-two-flows-at-a-glance)
 3. [Flow A — Agent Sandbox capability](#3-flow-a--agent-sandbox-capability-permanent-platform-feature)
 4. [Flow B — the Dark Factory pipeline](#4-flow-b--the-dark-factory-pipeline)
+   - [Flow D — Lambda MicroVM substrate (alternative to Flow A)](#45-flow-d--lambda-microvm-substrate-alternative-to-flow-a)
 5. [The pluggable coding assistant](#5-the-pluggable-coding-assistant)
 6. [Independent verification](#6-independent-verification-the-heart-of-the-pattern)
 7. [Live status in the PR](#7-live-status-in-the-pr)
@@ -84,6 +85,15 @@ useful on its own and the factory is a consumer of it.
 > an isolated VM). The Dark Factory is one *consumer* of that capability. Keeping them separate
 > means the isolation substrate can ship, be tested, and be reused independently of the factory.
 
+> **Flow D — a second substrate.** Flow A's isolation boundary is a **Kata micro-VM pod** on a
+> platform-owned nested-virt node group. **Flow D** offers an *alternative* Flow-A substrate — an
+> **AWS Lambda MicroVM** (serverless micro-VM, no node group) provisioned via the ACK `lambdamicrovms`
+> controller and composed by a single **KRO `ResourceGraphDefinition`**. Flow B is unchanged and can
+> target either substrate through the same `SandboxClaim` contract. See
+> [§4.5](#45-flow-d--lambda-microvm-substrate-alternative-to-flow-a) and
+> [`diagrams/flow-d-microvm-sandbox.md`](diagrams/flow-d-microvm-sandbox.md). *(Flow C is reserved for
+> other work.)*
+
 > **Why the hub, not a spoke?** The Dark Factory is a **pre-dev build/author** activity: it *writes*
 > code and needs GitHub write access. That belongs on the **hub — the control/build plane** — not on
 > a spoke, which is the **deploy/run plane** hosting real enterprise workloads (putting a
@@ -256,6 +266,102 @@ UI — the substrate for scaling across many concurrent issues.
 
 ---
 
+## 4.5. Flow D — Lambda MicroVM substrate (alternative to Flow A)
+
+> 📊 **See the diagrams:** [`diagrams/flow-d-microvm-sandbox.md`](diagrams/flow-d-microvm-sandbox.md)
+> (substrate architecture + platform/app ownership split + the RuntimeClass-shim bridge).
+
+Flow A's isolation boundary is a **Kata micro-VM pod** on a platform-owned nested-virt node group.
+**Flow D is a second Flow-A substrate**: an **AWS Lambda MicroVM** — a *serverless* micro-VM with no
+node group to run or pay for while idle, per-claim lifecycle, and sub-second warm starts. Flow B is
+unchanged: it still creates a `SandboxClaim`, a pod still shows up, and the **same `dark-factory-coder`**
+runs its coding/testing loop — except the coder executes inside a Lambda MicroVM. *(Flow C is reserved
+for other work; this substrate is Flow D.)*
+
+### How it's built — KRO RGD over ACK primitives
+
+| Layer | Mechanism | Notes |
+|---|---|---|
+| **Composition** | **Managed KRO** (EKS Capability) + one `MicrovmSandbox` `ResourceGraphDefinition` | One CR expands into the IMAGE primitives below (built once); the running `Microvm` is NOT in the graph — the shim runs it imperatively |
+| **GA primitives** | **Managed ACK** (EKS Capability) — `iam` Role, `s3` Bucket | AWS-run; the image store + build/exec roles |
+| **Image primitive** | **Self-managed ACK** — the pre-GA `lambdamicrovms` controller | `MicrovmImage` CRD (`lambdamicrovms.services.k8s.aws/v1alpha1`); the `Microvm` is created via SDK by the shim, not as a graph resource |
+
+> **Why self-managed for the MicroVM controller?** Managed ACK bundles only controllers whose service
+> is **GA upstream** (see the [ACK community services / GA list](https://aws-controllers-k8s.github.io/community/docs/community/services/)).
+> `lambdamicrovms` is **pre-GA** (`v1alpha1`, not on that list), so it isn't in Managed ACK yet — it
+> runs as its own GitOps addon. **Managed ACK + self-managed lambdamicrovms coexist** (different CRD
+> groups → no conflict). When `lambdamicrovms` goes GA, delete the self-managed addon and Managed ACK
+> adopts it — **the RGD is unchanged**. This "install both now" posture is deliberate and futuristic.
+>
+> The **Managed KRO + Managed ACK capabilities themselves** are enabled at the platform layer in the
+> **appmod-blueprints** repo (an EKS Capability toggle) — see that repo's
+> `docs/EKS-Capabilities-KRO-ACK-Setup.md`. This repo owns only the **self-managed `lambdamicrovms`
+> controller + the KRO `MicrovmSandbox` RGD + the sandbox shim** (Flow D).
+
+### The split: KRO builds the image ONCE; the shim runs the VM per session
+
+This is the load-bearing design decision (and it matches the ACK controller's own guidance —
+image build is slow/declarative, running a VM is fast/imperative):
+
+- **Platform image — declarative, built ONCE by KRO/ACK.** The `MicrovmSandbox` RGD
+  (`agent-sandbox-lambda/templates/image/`) composes only the slow-changing infra: `MicrovmImage`
+  (`baseImageARN`, `buildRoleARN`, `codeArtifact.uri` — an **S3 zip** of the arm64 `dark-factory-coder`
+  + a Dockerfile) plus its **build role**, **execution role**, and **S3 artifact bucket** (ACK GA
+  controllers). A **single committed `MicrovmSandbox` instance** (GitOps-applied) is reconciled once;
+  KRO gates the handoff on a successful build (`readyWhen state == CREATED||UPDATED`). Its status
+  surfaces `imageARN` + `executionRoleARN`. The RGD **does not** contain a `Microvm`.
+- **Per-session VM — imperative, driven by the shim.** Running a MicroVM (`RunMicrovm`), and its
+  `suspend` / `resume` / `TerminateMicrovm`, are request-time SDK ops the ACK controller does **not**
+  reconcile. So the shim owns them — never a `Microvm` CR per claim.
+
+### The RuntimeClass shim (claim → pod → MicroVM)
+
+A literal K8s `RuntimeClass` (like `kata-clh`) maps to a **node-local containerd handler**; Lambda
+MicroVM is a **remote AWS service**, so a true node-level RuntimeClass would require a virtual-kubelet
+provider (a large Go runtime — **out of scope**). Flow D instead ships a **`lambda-microvm`
+SandboxTemplate variant** (`agent-sandbox-lambda/templates/shim/`) whose pod is a lightweight
+**bridge**: on claim it **reads the platform image handoff** (`imageARN` + `executionRoleARN` from the
+one built `MicrovmSandbox`) and calls **`RunMicrovm`** (SDK) to launch this session's VM, records the
+`microvmID` as an annotation on the owning `Sandbox`, and holds the pod so its lifecycle mirrors the
+MicroVM's. On real teardown it calls `TerminateMicrovm`. To Flow B and the user the UX is identical to
+Flow A. Interactive exec/attach passthrough is **best-effort**; full fidelity is a virtual-kubelet follow-up.
+
+### Suspend / resume / terminate — the coder VM persists across the review loop
+
+Because the substrate is a Lambda MicroVM (not a pod), Flow D uses **suspend/resume through the Agent
+Sandbox CRD** to keep the coder's context across the whole review→fix→re-review loop — the payoff of
+this substrate over Kata (where each fix round claims a fresh pod):
+
+1. **Coder finishes the coding task → SUSPEND** (`df-run` flips `Sandbox.operatingMode=Suspended`; the
+   `microvm-lifecycle` reconcile loop calls `suspend-microvm` by the annotated id). Compute is freed;
+   the VM's memory/disk are snapshotted.
+2. DevOps + Security agents review the PR while the coder is suspended.
+3. **Findings + "fix" → RESUME the SAME VM** (df-iterate sets `operatingMode=Running` → `resume-microvm`).
+   Context intact — no cold re-implement.
+4. Coder fixes → SUSPEND again; loop 2–4 until both agents clear.
+5. **Final exit (merge) → TERMINATE** (`df-merge-teardown` calls `TerminateMicrovm`, then deletes the
+   claim). This is the **only** place the VM is destroyed — `df-run`'s onExit is substrate-aware and
+   **keeps** the suspended Lambda VM (unlike Kata, which frees its pod on df-run exit).
+
+The ACK `Microvm` has no suspend field, so the `microvm-lifecycle` loop supplies the intent→SDK
+translation — pure shim, no virtual-kubelet. See
+[`diagrams/flow-d-microvm-sandbox.md` §D.3a](diagrams/flow-d-microvm-sandbox.md).
+
+### Delivery & status
+
+Shipped as GitOps in its **own chart** — `gitops/addons/charts/agent-sandbox-lambda/` (separate from
+the Kata `agent-sandbox` chart), structured as `templates/image/` (KRO RGD + the one platform
+`MicrovmSandbox`) and `templates/shim/` (bridge SandboxTemplate + warm pool + `microvm-lifecycle`
+controller). **Disabled by default** (`microvm.enabled=false`); the hub overlay
+(`clusters/hub/addons/agent-sandbox-lambda/values.yaml`) carries cluster-specific values, and a gated
+`agent-sandbox-lambda` addon entry deploys it hub-only. The platform-capability enablement (Managed ACK
++ Managed KRO) lands separately in the **appmod-blueprints** platform repo (they're EKS Capabilities,
+like the Managed ArgoCD the hub already runs). This PR delivers the **design + GitOps scaffold**; the
+live end-to-end path (enable capabilities → sync controller → publish the arm64 artifact → run a MicroVM
+coder with suspend/resume) is the follow-up.
+
+---
+
 ## 5. The pluggable coding assistant
 
 The coder is behind a **thin, swappable interface** — a deliberate choice (the industry lesson is

docs/dark-factory/diagrams/flow-d-microvm-sandbox.md

@@ -0,0 +1,138 @@
+# Flow D — Lambda MicroVM–backed Agent Sandbox (alternative substrate)
+
+**Flow D is a second Flow-A substrate.** Where Flow A hands out **Kata micro-VM pods** on a
+self-managed nested-virt EKS node group, Flow D hands out **AWS Lambda MicroVMs** provisioned by the
+ACK `lambdamicrovms` controller and composed by a single **KRO `ResourceGraphDefinition`**. The Agent
+Sandbox UX is unchanged: a consumer (notably **Flow B — Dark Factory**) creates a `SandboxClaim`, a
+pod shows up, and the **same `dark-factory-coder`** runs its coding/testing loop — except the coder
+executes inside a Lambda MicroVM instead of on the Kata node.
+
+> **Flow C is reserved for other work** — this substrate is **Flow D**.
+
+> **Why a second substrate?** Kata (Flow A) needs a dedicated nested-virt node group the platform owns
+> and pays for while idle. Lambda MicroVM is a **serverless** micro-VM: no node group to run, per-claim
+> lifecycle, sub-second warm starts, and a clean **platform-owns-the-image / app-owns-the-instance**
+> split that maps directly onto the two ACK CRDs. Flow B can target either substrate with no pipeline
+> change — it only ever sees the Agent Sandbox `SandboxClaim` contract.
+
+> 🎨 Diagrams are editable draw.io — sources in [`src/`](./src/), rendered PNGs in [`img/`](./img/).
+> *(Flow D diagram sources are added alongside the Flow A/B ones; see `src/flow-d-*.drawio`.)*
+
+---
+
+## D.1 — Substrate architecture (KRO RGD over ACK primitives)
+
+The platform installs two controllers and one composition layer, then exposes **one** custom
+resource to consumers:
+
+- **Managed ACK** (EKS Capability) runs the **GA** controllers — `iam.services.k8s.aws` (Role) and
+  `s3.services.k8s.aws` (Bucket) — that the MicroVM image + instance depend on.
+- **Self-managed ACK** runs **only** the pre-GA `lambdamicrovms.services.k8s.aws` controller (its own
+  Helm chart / ArgoCD addon), because Managed ACK bundles GA controllers only.
+- **Managed KRO** (EKS Capability) runs the `ResourceGraphDefinition` engine.
+- A single **`MicrovmSandbox` RGD** ties it all together: one CR expands into `MicrovmImage` +
+  `Microvm` + IAM `Role`(s) + S3 `Bucket`.
+
+```
+consumer (Flow B / any agent)
+        │  creates
+        ▼
+  MicrovmSandbox  (kro.run/v1alpha1 — the single abstraction)
+        │  expands into
+        ├── MicrovmImage        (lambdamicrovms.services.k8s.aws)   ── platform-owned inputs
+        ├── S3 Bucket           (s3.services.k8s.aws)               ── image codeArtifact store
+        ├── IAM Role (build)    (iam.services.k8s.aws)              ── MicrovmImage.buildRoleArn
+        ├── IAM Role (exec)     (iam.services.k8s.aws)              ── Microvm.executionRoleArn
+        └── Microvm             (lambdamicrovms.services.k8s.aws)   ── app-owned instance lifecycle
+```
+
+*Edit: `src/flow-d-substrate.drawio` → `img/flow-d-substrate.png`.*
+
+---
+
+## D.2 — Platform-owned vs app-owned split (inside one RGD)
+
+The two ACK CRDs encode the ownership boundary the platform team and application teams care about;
+the RGD schema surfaces each half to the right owner:
+
+| Layer | Owner | ACK resource | Key fields |
+|---|---|---|---|
+| **Image / substrate** | Platform | `MicrovmImage` | `baseImageARN`, `buildRoleArn`, `codeArtifact.uri` (S3), egress connectors |
+| **Instance / run** | App team | `Microvm` | `imageIdentifier`, `executionRoleArn`, `ingress/egressNetworkConnectors`, `idlePolicy` |
+
+- **Platform** sets the image once (built **from the existing `dark-factory-coder` image** + its
+  `entrypoint.js`, published to the S3 `codeArtifact` bucket) — declarative, ACK-managed, GitOps.
+- **App teams / Flow B** create per-claim `Microvm` instances referencing that image, and own the
+  instance lifecycle (`RunMicrovm` / `TerminateMicrovm`, idle policy) via the same claim they use today.
+
+*Edit: `src/flow-d-ownership.drawio` → `img/flow-d-ownership.png`.*
+
+---
+
+## D.3 — The RuntimeClass shim (claim → pod → MicroVM)
+
+A literal Kubernetes `RuntimeClass` (like `kata-clh`) maps to a **node-local containerd handler**.
+Lambda MicroVM is a **remote AWS service**, so a true node-level RuntimeClass isn't possible without a
+virtual-kubelet provider (a large Go runtime component — explicitly **out of scope**). Flow D uses a
+**RuntimeClass-marked bridge pod** instead, preserving the exact Agent Sandbox UX:
+
+```
+SandboxClaim (Flow B injects DF_ISSUE_NUMBER, repo, branch — unchanged)
+        │
+        ▼
+Sandbox → Pod from the `lambda-microvm` SandboxTemplate variant
+        │   (bridge container; lands on a normal Auto-Mode node, NOT the kata pool)
+        ▼
+bridge applies a MicrovmSandbox (KRO) CR
+        │
+        ▼
+Microvm RUNNING  ── runs the SAME dark-factory-coder entrypoint (node /app/entrypoint.js)
+        │
+        ├── bridge streams MicroVM logs → pod logs   (pod Running  ⇔ Microvm RUNNING)
+        └── pod exit / claim teardown → TerminateMicrovm
+```
+
+To Flow B and the user this is identical to Flow A — "a sandbox pod appeared and ran the coder" — but
+the coder actually executed in the Lambda MicroVM. Log streaming is straightforward; interactive
+exec/attach passthrough is **best-effort** (full fidelity would need virtual-kubelet).
+
+*Edit: `src/flow-d-shim.drawio` → `img/flow-d-shim.png`.*
+
+---
+
+## D.3a — Suspend / resume (Sandbox.operatingMode → MicroVM)
+
+The Agent Sandbox CRD exposes `spec.operatingMode ∈ {Running, Suspended}` — the declarative
+suspend/resume intent. But the ACK `Microvm` CR has **no suspend field**: its spec is create-time
+only, `State` is status-only, and `suspend-microvm`/`resume-microvm` are **imperative SDK ops the ACK
+controller deliberately does not reconcile**. So flipping `operatingMode` does nothing on its own — a
+controller must translate intent into the SDK call.
+
+Flow D closes that gap with a tiny always-on **`microvm-lifecycle`** reconcile loop (a ConfigMap
+script on `alpine/k8s`, same pattern as the pool-manager — **no virtual-kubelet, no new image**):
+
+```
+Sandbox.operatingMode: Running   → Suspended :  aws lambda-microvms suspend-microvm --microvm-identifier <id>
+Sandbox.operatingMode: Suspended → Running   :  aws lambda-microvms resume-microvm  --microvm-identifier <id>
+```
+
+- `<id>` (the `microvmID`) is resolved from the `MicrovmSandbox` (KRO) status; the loop is idempotent
+  (stamps a `last-mode` annotation, acts only on transitions).
+- The `MicrovmSandbox` is **kept** across suspend (the bridge's `preStop` detects `operatingMode:
+  Suspended` and skips teardown), so the VM survives suspend/resume; it's deleted only on real claim
+  teardown → `TerminateMicrovm`.
+- Chosen over bridge `preStop` hooks alone because a reconcile loop is **robust to pod/node loss** and
+  resume needs no live pod. This is the open-source **Sandbox-CRD-driven** suspend/resume you get with
+  the MicroVM substrate.
+
+*Edit: `src/flow-d-suspend-resume.drawio` → `img/flow-d-suspend-resume.png`.*
+
+---
+
+## D.4 — Future: when `lambdamicrovms` goes GA
+
+`lambdamicrovms` is currently **pre-GA** (`v1alpha1`), so its controller is self-managed. When it
+graduates to GA upstream, **Managed ACK adopts it automatically** — the self-managed chart is deleted
+and the `MicrovmSandbox` RGD is **unchanged** (it references the same `lambdamicrovms.services.k8s.aws`
+CRDs regardless of who runs the controller). The design deliberately keeps the RGD independent of the
+controller install method so this migration is a one-line addon removal.

examples/dark-factory/coder/entrypoint.js

@@ -256,11 +256,25 @@ function runCoder(repoDir) {
   // Inherit stdio so the coder CLI's own output + errors stream into the pod
   // logs (kubectl logs), instead of being swallowed by execFileSync's exception.
   const opts = { cwd: repoDir, env, stdio: "inherit", maxBuffer: 64 * 1024 * 1024 };
-  const prompt = `Implement the change described in ${WORKSPACE}/SPEC.md. Build and run unit tests until green. Commit your work.`;
+  // The coder also writes a concise, human-readable summary of WHAT it changed to
+  // artifacts/description.md — this becomes the "Changes" section of the PR body
+  // (in addition to the verification section). Keep it short: what changed + why,
+  // as reviewer-facing markdown bullets.
+  const descPath = `${WORKSPACE}/artifacts/description.md`;
+  const prompt =
+    `Implement the change described in ${WORKSPACE}/SPEC.md. Build and run unit tests until green. Commit your work. ` +
+    `Then write a concise description of the changes you made (what changed and why, as a few markdown bullet points, ` +
+    `reviewer-facing — no preamble) to ${descPath}.`;
   if (ENGINE === "kiro") {
     // Kiro CLI headless — the coder image carries the `kiro` binary; it reads the
     // same Bifrost/Bedrock env above. --headless drives it non-interactively.
+    // Append the description instruction so kiro produces the same artifact.
     console.log("[coder] engine=kiro (kiro run --headless)");
+    try {
+      fs.appendFileSync(`${WORKSPACE}/SPEC.md`,
+        `\n\n---\n\n## After implementing\n\nWrite a concise description of the changes you made ` +
+        `(what changed and why, a few reviewer-facing markdown bullets, no preamble) to ${descPath}.\n`);
+    } catch { /* non-fatal */ }
     return execFileSync("kiro", ["run", "--headless", "--spec", `${WORKSPACE}/SPEC.md`], opts);
   }
   console.log("[coder] engine=claude (claude -p)");
@@ -429,9 +443,17 @@ async function main() {
         ? "- ⏳ **Security review (AWS Security Agent):** _queued (DevOps cleared)…_"
         : "- ⬜ **Security review (AWS Security Agent):** _waiting on DevOps clearance_";
     }
+    // Coder-authored description of the changes (artifacts/description.md). Shown
+    // as a "Changes" section ahead of the verification block. Falls back to a
+    // neutral line if the coder didn't produce one, so the PR body is never empty.
+    const desc = readSecret(`${WORKSPACE}/artifacts/description.md`);
+    const changesSection = desc
+      ? ["### 📝 Changes", "", desc, ""]
+      : ["### 📝 Changes", "", "_Implemented per the linked issue; see the diff for details._", ""];
     const prBody = [
       `Closes #${ISSUE}.`,
       "",
+      ...changesSection,
       "<!-- dark-factory:status -->",
       "### 🏭 Dark Factory — verification",
       `- ✅ **Build + unit tests:** ${test.summary}`,