Flow D — Lambda MicroVM Agent Sandbox (KRO RGD + ACK, label-branched, suspend/resume) - #41
Open
elamaran11 wants to merge 19 commits into
Open
Conversation
Flow D is a second Flow-A substrate: AWS Lambda MicroVM via the ACK lambdamicrovms controller, composed by a single KRO ResourceGraphDefinition. Mirrors the flow-a-sandbox-capability.md style. Covers substrate architecture (KRO RGD over ACK primitives), the platform/app ownership split inside one RGD, the RuntimeClass-marked bridge shim (claim -> pod -> MicroVM), and the GA migration path (self-managed controller -> Managed ACK adopts it, RGD unchanged). Flow C is reserved for other work; this substrate is Flow D.
…ctory README New §4.5 'Flow D — Lambda MicroVM substrate (alternative to Flow A)': KRO RGD over ACK primitives (Managed KRO + Managed ACK for GA iam/s3 + self-managed lambdamicrovms pre-GA controller), the platform-owned/app-owned split encoded in the MicrovmImage/Microvm CRDs, the RuntimeClass-marked bridge shim, and disabled-by-default GitOps delivery. Adds the TOC entry and a 'second substrate' note to the Two-flows-at-a-glance section. Reuses dark-factory-coder image.
…isabled) New addons.yaml entry installing the pre-GA Lambda MicroVM ACK controller (oci://public.ecr.aws/aws-controllers-k8s/lambdamicrovms-chart v0.1.1) that reconciles the MicrovmImage + Microvm CRDs. Self-managed because Managed ACK only bundles GA controllers; coexists with Managed ACK (GA iam/s3) since CRD groups differ. enabled:false (Flow D dormant), hub-only via alwaysSelector, sync-wave 0 so CRDs+controller precede the MicrovmSandbox RGD (wave 2). Pod-identity auth (no IRSA annotation), cluster install scope.
templates/50-rgd-microvm-sandbox.yaml: one kro.run/v1alpha1 ResourceGraphDefinition that expands a single MicrovmSandbox CR into S3 Bucket + build/exec IAM Roles (GA -> Managed ACK) + MicrovmImage + Microvm (pre-GA -> self-managed ACK). The platform/app ownership split is encoded in the schema (spec.image.* platform-owned, spec.run.* app-owned); status surfaces microvmID/state for the bridge to mirror. RGD is controller-install-agnostic (unchanged when lambdamicrovms goes GA). values.yaml: new microvm: block (enabled:false, region, apiGroup, bridgeImage, idle/connector defaults; cluster-specific ARNs left blank for the overlay). Verified: helm template renders 0 RGD when disabled, 1 valid RGD (5 resources: bucket/buildRole/execRole/image/microvm) when enabled.
…eClass shim) templates/51-sandboxtemplate-microvm.yaml: the shim that keeps the Agent-Sandbox UX while the coder runs in a remote Lambda MicroVM. A SandboxTemplate (<templateName>-microvm) whose pod is a thin BRIDGE on a normal Auto-Mode node (no kata runtimeClass/nodeSelector) — it idles until a claim injects DF_ISSUE_NUMBER (same as Flow A), applies a MicrovmSandbox (KRO) CR, waits for Microvm RUNNING, mirrors lifecycle (pod Running <-> Microvm RUNNING), and deletes the CR on exit (-> TerminateMicrovm). Ships its ServiceAccount + least-priv Role/RoleBinding (microvmsandboxes only) + the bridge.sh ConfigMap. Both substrates coexist; a claim picks one by which template it references. Verified: helm template renders 5 objects when enabled, 0 when disabled; bridge.sh passes sh -n.
Hub per-cluster overlay gets a microvm: block (enabled:false) alongside the kata nodepool block. Cluster/account-specific ARNs (baseImageARN, codeArtifactUri, ingress/egress connectors) left blank with guidance to fill before enabling. Flow D stays fully dormant on the hub until the Managed KRO + Managed ACK capabilities and the self-managed lambdamicrovms controller are in place. Verified full-chart render with the overlay: 0 Flow D objects when dormant (existing Kata objects intact), clean inventory when enabled (1 RGD + 2 SandboxTemplates + bridge SA/Role/RoleBinding/ConfigMap), valid YAML in both modes.
…into Flow D The ACK GA-services-list reference and the pre-GA-vs-GA / self-managed rationale belong with the consumer (Flow D), not the platform capability-enablement doc. Add the authoritative ACK community services (GA list) link backing the 'GA upstream' claim, and clarify the ownership split: the Managed KRO + Managed ACK CAPABILITIES are enabled in appmod-blueprints (EKS Capability toggle, see its docs/EKS-Capabilities-KRO-ACK-Setup.md); THIS repo owns the self-managed lambdamicrovms controller + the KRO MicrovmSandbox RGD + the sandbox shim.
Align the MicrovmSandbox RGD with the actual lambda-microvms API (verified against the live service + AWS docs in us-west-2): - IAM trust principal: lambda.amazonaws.com (+ sts:TagSession) — NOT the guessed lambdamicrovms.amazonaws.com. buildRole gets inline ECR-read + S3-read + logs (Lambda pulls the coder image from private ECR at build). - Drop network connectors entirely: Lambda MicroVMs have PUBLIC egress by default, which is all the coder needs (git/gh/registry). Ingress/VPC-egress connectors are optional add-ons (the latter created via a different service, aws lambda-core) — omitted from v1. Removes ingress/egressConnectorARN from schema + both resources. - codeArtifactUri: document it accepts an ECR image URI or S3 path (Flow D uses the arm64 dark-factory-coder ECR image directly). - base image ARN format corrected to arn:aws:lambda:<region>:aws:microvm-image:al2023-1 (ARM_64-only) in values guidance. Verified: helm template renders 5 resources, 0 connector refs, correct principal.
Set the hub Flow D overlay to the confirmed live values: - baseImageARN: arn:aws:lambda:us-west-2:aws:microvm-image:al2023-1 (verified available; ARM_64-only). - codeArtifactUri: dark-factory-coder:v0.2.3-arm64 ECR URI (accepts ECR directly; arm64 tag to be built — MicroVM is ARM_64-only, existing amd64 tags untouched). - Drop the connector fields (MicroVMs default to public egress). Still enabled:false (dormant) until the ack-lambdamicrovms controller is synced.
…ifecycle)
Implements the REQUIRED declarative suspend/resume: Sandbox.spec.operatingMode has
{Running,Suspended} but the ACK Microvm CR has no suspend field (suspend/resume are
imperative SDK ops the controller doesn't reconcile). Close the gap with a tiny
always-on reconcile loop (templates/52) — pure shim, no virtual-kubelet, no new image:
- microvm-lifecycle Deployment (alpine/k8s, ConfigMap reconcile.sh, matches
pool-manager pattern): watches Sandboxes labeled substrate=lambda-microvm; on
operatingMode Running->Suspended calls suspend-microvm, Suspended->Running calls
resume-microvm, by microvmID resolved from the MicrovmSandbox status. Idempotent
via a last-mode annotation. + SA/Role/RoleBinding (pod-identity auth).
- Bridge (51) made suspend-aware: labels its MicrovmSandbox agent-sandbox.io/sandbox
for correlation; preStop checks operatingMode and, if Suspended, writes a marker so
cleanup() KEEPS the MicrovmSandbox (VM survives suspend); real teardown only on
claim end. Create is idempotent (resume-or-create). +sandboxes get RBAC.
- values: microvm.lifecycle.intervalSeconds (15s).
Robust vs preStop-only: survives pod/node loss; resume needs no live pod. Verified:
0 objects disabled; 10 when enabled; reconcile.sh + bridge.sh pass sh -n.
Add §D.3a (diagram doc) + a Suspend/resume subsection (README §4.5) describing the microvm-lifecycle reconcile loop that translates Sandbox.operatingMode -> Lambda MicroVM suspend/resume-microvm, why it's needed (ACK Microvm has no suspend field), and why a reconcile loop beats preStop hooks (robust to pod/node loss). Matches the templates/52 implementation.
Per reviewer request: the coder now writes a concise, reviewer-facing description of WHAT it changed (and why) to artifacts/description.md as part of its run, and the PR body renders it as a '### 📝 Changes' section ABOVE the existing verification block. Falls back to a neutral line if absent so the body is never empty. - claude path: description instruction appended to the prompt. - kiro path: same instruction appended to SPEC.md (kiro reads --spec). - PR body: new Changes section placed before the <!-- dark-factory:status --> marker, so status.js (which rewrites from the marker down) preserves it on every update. node --check passes.
Two fixes for the 'security review waits ~15 min while the PR already shows a result' problem: 1. Early-exit: the code-review job's status field flips to COMPLETED long AFTER the AWS Security Agent App bot has already posted its findings on the PR. So in addition to polling status, each iteration now probes list-findings; the moment it returns a well-formed result (findingsSummaries present) we proceed — no waiting for the status flip. Reuses those findings for the report (no re-fetch). 2. Graceful timeout: this step is advisory (the App bot posts the authoritative result regardless), so exceeding the poll timeout now posts a neutral 'pending' status + a pointer to the bot comment, instead of a misleading red 'error'. This also fixes the stale sticky board: the sticky-status step depends on security-agent (runs once, after it), so a fast security step means the board is rewritten with accurate rows within seconds instead of ~15 min later. sh -n passes.
… substrate One pipeline, substrate chosen by label (per design decision): - sensor: accept BOTH dark-factory (Kata) + darkfactory-lambda (MicroVM) labels; pass the firing label to df-run as the trigger-label param (index 6). - df-run: new trigger-label param (default dark-factory); claim-sandbox takes a warm-pool input; the DAG computes it via an Argo expression (darkfactory-lambda -> coder-warmpool-microvm, else coder-warmpool). Everything downstream (coder, holdout, devops, security, PR, merge, teardown) is identical. - agent-sandbox: add the Lambda-MicroVM SandboxWarmPool (claim.warmPoolRef is required, so Flow D needs its own pool) referencing the -microvm SandboxTemplate; microvm.warmPool values (name, targetIdle). dark-factory: warmPool.lambdaName. Verified: df-run + sensor render to valid YAML; warm-pool expression resolves correctly; lambda warmpool renders only when microvm.enabled (0 when disabled).
Flow D visible suspend/resume via Sandbox.operatingMode (per design decision):
- New microvm-suspend DAG step (gated microvmSuspend.enabled + lambda substrate +
PR exists): after drive-coder, flips Sandbox.spec.operatingMode=Suspended so the
agent-sandbox microvm-lifecycle controller calls suspend-microvm — compute freed
while the gates run (coder is idle during review). No-op for Kata.
- microvm-set-mode script template: patches operatingMode, best-effort verifies the
MicrovmSandbox microvmID (advisory). RBAC: workflow SA gets sandboxes 'patch'.
- Resume: happens naturally — df-iterate (PR comment) needs the VM, and the MicroVM
idlePolicy.autoResumeEnabled resumes on next request; the lifecycle controller
also resumes on operatingMode=Running.
- values: microvmSuspend {enabled, image}.
Verified: full chart renders valid YAML; suspend step present when enabled, absent
when disabled.
elamaran11
changed the title
Applied field-level corrections from the lambdamicrovms-controller reference
(mmeckes/lambdamicrovms-controller, 06-kro example + CRD source):
- codeArtifact.uri is S3-ONLY (s3://bucket/key of a zip w/ Dockerfile), NOT an
ECR image URI. Fixed the misleading 'ECR or S3' comments in the RGD schema +
values (the Dockerfile inside MAY pull ECR base layers — build role keeps ecr:Get*).
- image.readyWhen: gate on status.state == CREATED||UPDATED so the Microvm never
launches from a half-built/failed image (KRO holds the instance until ready).
- MicrovmImage.cpuConfigurations: [{architecture: ARM_64}] — the sole supported
arch; make it explicit rather than relying on a default.
- MicrovmImage.logging.cloudWatch.logGroup: /aws/lambda/microvms/<name>-image so
CREATE_FAILED build output is retrievable (aws logs tail ...).
- status.microvmState now reads Microvm.status.state (PENDING/RUNNING/SUSPENDED/
...) not conditions[0].status; added imageState passthrough.
- build role S3 policy scoped to the actual bucket via CEL ref (bucket.spec.name)
+ ListBucket, and logs scoped to the microvms log-group prefix.
Design unchanged: Sandbox CRDs → ACK Lambda MicroVM for create/delete + the
microvm-lifecycle shim for suspend/resume (imperative ops, not controller-managed
— confirmed by the reference). Still gated microvm.enabled=false; render-verified.
…bda chart
Clean folder separation — Flow D (Lambda MicroVM) is now a standalone chart instead
of files dumped into the Kata agent-sandbox chart:
gitops/addons/charts/agent-sandbox-lambda/
Chart.yaml, values.yaml, templates/_helpers.tpl
templates/image/10-rgd-and-image.yaml — KRO RGD (image-only) + the ONE platform
MicrovmSandbox instance (built ONCE)
templates/shim/20-bridge-sandboxtemplate.yaml — bridge: reads image handoff,
RunMicrovm (SDK), TerminateMicrovm
templates/shim/30-microvm-lifecycle.yaml — suspend/resume controller (SDK)
- Moved the 3 microvm templates out of agent-sandbox (Kata chart) → now Kata-only.
- Moved the microvm: values block out of agent-sandbox/values.yaml + the hub overlay
into agent-sandbox-lambda/{values.yaml, clusters/hub overlay}. Corrected stale docs
(codeArtifact is S3-only, not ECR).
- New gated addon entry agent-sandbox-lambda (hub-only, sync-wave 2); dormant until
microvm.enabled=true.
ARCHITECTURE (per design): KRO builds the platform image ONCE; the shim runs/suspends/
resumes/terminates the per-session VM imperatively (ACK doesn't reconcile those). RGD no
longer contains a Microvm resource. Renders: Kata chart microvm-free; Lambda chart 0
disabled / 13 enabled. Flow D only — dark-factory (PR #32) untouched.
… across review loop) Implements the required Lambda coder lifecycle: coder codes → SUSPEND; findings + fix → RESUME the SAME VM (keeps context); terminate ONLY at final exit (merge). - df-run onExit teardown is now SUBSTRATE-AWARE (script, not a blind resource delete): Kata deletes the claim (frees pool); LAMBDA KEEPS the claim + suspended VM so the same VM survives the whole review→fix→re-review loop (df-iterate resumes it). Without this, df-run's exit deleted the claim → orphaned/terminated the suspended VM. - df-merge-teardown adds a microvm-terminate step (gated microvmSuspend.enabled) that resolves the per-session VM id from the Sandbox annotation and TerminateMicrovm on merge — the ONLY place the Lambda VM is destroyed. Ordered before the claim delete so the bridge's suspend-marker can't keep a leaked VM. - microvm-set-mode verify reads the VM id from the Sandbox annotation (post image-only RGD refactor — no per-session MicrovmSandbox status anymore). Flow-D-branch changes to the dark-factory template (base of PR #41); gated by microvmSuspend.enabled so Flow B (Kata, PR #32) is unaffected. Renders clean both ways.
…d/resume/terminate lifecycle Aligns the Flow D docs with the refactor: KRO builds the image ONCE (RGD has no Microvm); the shim runs the per-session VM imperatively (RunMicrovm/suspend/resume/ TerminateMicrovm); the coder VM PERSISTS suspended across the review→fix loop and is terminated only at merge. Updated the composition table, platform/app split, shim description, the suspend/resume/terminate lifecycle section, and the delivery section (now the standalone agent-sandbox-lambda chart with image/ + shim/ subfolders). NOTE: diagrams/flow-d-microvm-sandbox.md ASCII still shows the old per-claim model — follow-up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flow D — Lambda MicroVM Agent Sandbox (+ bundled Flow B improvements)
Adds Flow D: a second Agent-Sandbox substrate where the coder runs in an AWS Lambda MicroVM instead of a Kata pod. One pipeline, substrate chosen by label. Base branch:
dark-factory-autonomous-agent-coding-pattern(PR #32).Flow D — substrate
MicrovmSandboxRGD ties MicrovmImage + Microvm + IAM roles + S3 bucket (one CR). API corrected to the reallambda-microvmsservice (verified live; ARM_64-only; public egress default → no connectors; ECR codeArtifact).ack-lambdamicrovmscontroller addon (pre-GA; Managed ACK can't bundle it) — disabled by default.lambda-microvmSandboxTemplate + bridge (RuntimeClass shim) + its SandboxWarmPool.al2023-1, arm64 coder ECR URI).Flow D — pipeline (one pipeline, label-branched)
dark-factory(Kata) +darkfactory-lambda(MicroVM); passes the label to df-run.claimbinds the Kata pool or the MicroVM pool based on the label; everything downstream identical.Flow D — suspend/resume (required)
microvm-lifecyclecontroller reconcilesSandbox.operatingMode→suspend/resume-microvm(the ACK Microvm CR has no suspend field).microvm-suspenddf-run step: after the coder pushes, suspend the MicroVM while gates run (compute freed); resumes on iterate / next request.Bundled Flow B improvements (deployed + verified on PR #56)
Verification
helm templateclean; scriptssh -n/node --check.Platform prereq (separate, merged): appmod-blueprints #796/#800 enabled Managed KRO + ACK on the hub.
Generated with Claude Code.