From 35b29833256dd271a2f7a5d16586fe02b32cb92f Mon Sep 17 00:00:00 2001 From: Ruben Koster Date: Tue, 11 Aug 2026 07:38:35 +0200 Subject: [PATCH] Add research note: Firecracker microVM isolation --- research/firecracker-microvm.md | 139 ++++++++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100644 research/firecracker-microvm.md diff --git a/research/firecracker-microvm.md b/research/firecracker-microvm.md new file mode 100644 index 0000000..ce52593 --- /dev/null +++ b/research/firecracker-microvm.md @@ -0,0 +1,139 @@ +--- +title: "Firecracker — Hardware-Isolated microVMs for Untrusted, Ephemeral Compute" +author: Ruben Koster (@rkoster) +date: 2026-08-10 +tags: [sandboxing-isolation, runtime-lifecycle, ecosystem-survey] +cf_areas: [] +status: draft +sources: + - https://github.com/firecracker-microvm/firecracker + - https://github.com/firecracker-microvm/firecracker/blob/main/docs/design.md + - https://github.com/firecracker-microvm/firecracker/blob/main/CHARTER.md + - https://github.com/firecracker-microvm/firecracker/blob/main/SPECIFICATION.md + - https://github.com/firecracker-microvm/firecracker/blob/main/docs/snapshotting/snapshot-support.md + - https://firecracker-microvm.github.io/ + - https://fly.io/blog/sandboxing-and-workload-isolation/ + - https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing +--- + +## Summary + +Firecracker is an open-source Virtual Machine Monitor (VMM), written in Rust, that AWS built +and open-sourced in 2018 (Apache-2.0) to power AWS Lambda and Fargate's multi-tenant +serverless infrastructure. It creates "microVMs" — minimal, KVM-based virtual machines that +combine the strong isolation boundary of hardware virtualization with boot times and memory +overhead close to that of a container. Architecturally it is the deliberate opposite of QEMU: +rather than emulating a general-purpose PC (BIOS, legacy devices, broad OS compatibility), +Firecracker exposes only five devices and boots a pre-configured Linux kernel directly, cutting +both attack surface and startup latency. It sits at the "full hardware VM" end of the +container→microVM→full-VM isolation spectrum already touched on in `k8s-agent-sandbox.md` +(gVisor/Kata) and `wasmcloud.md` (WASM components), and is now the substrate several +agent-hosting platforms — notably AWS Bedrock AgentCore (`aws-agents.md`) — use for +per-session/per-tenant agent sandboxing, since agent code execution is treated as inherently +untrusted and often long-lived-but-idle. + +## Key findings + +- **Origin**: a VMM using Linux KVM, written in Rust, developed inside AWS and open-sourced in + November 2018, purpose-built to replace per-customer EC2 instances as the isolation unit for + Lambda and Fargate. It originated as a fork of Chromium OS's `crosvm` VMM, since diverged. +- **License and governance**: Apache License 2.0 (crosvm-derived sections under BSD-3-Clause). + Governance is AWS-controlled, not an independent foundation — the repo's charter states + maintainers are subject to the project's mission and tenets, with maintainers predominantly + AWS-affiliated. There is no CNCF/Linux Foundation transfer; it's "an AWS open source project + that encourages contributions" — closer to a benevolent-vendor model than an independent + foundation, unlike Kata Containers (OpenInfra/CNCF-adjacent) or gVisor (Google-led but under + a more neutral GitHub org). +- **Minimal device model**: Firecracker deliberately exposes only 5 emulated devices — + virtio-net, virtio-block, virtio-vsock, a serial console, and a partial i8042 keyboard + controller (used only to signal guest-requested reboot/reset) — plus KVM's own + PIC/IOAPIC/PIT. All legacy PC/BIOS emulation is excluded by design, the direct architectural + contrast with QEMU. +- **Performance characteristics** (enforced by CI against a formal specification): boot time + ≤125ms from API call to guest init; memory overhead ≤5 MiB per microVM (1 vCPU/128MiB + config); microVM creation rate up to 150/second per host (5/core/sec); network throughput up + to 14.5–25 Gbps depending on CPU budget; ~0.06ms added virtualization latency. +- **Security model — layered defense in depth**: (1) the KVM hardware-virtualization boundary + itself — a stronger isolation guarantee than a shared-kernel container, architecturally + distinct from gVisor's approach of reimplementing a subset of Linux syscalls in userspace Go + rather than running a separate kernel (`k8s-agent-sandbox.md`); (2) per-thread seccomp-bpf + filters restricting Firecracker's own process to a minimal syscall set (independent research + counted ~40 allowed syscalls with tight argument filters); (3) the optional `jailer` + companion process, which sets up cgroups, namespaces, and chroot with elevated privileges, + then drops privileges and execs into the unprivileged Firecracker binary — a second line of + defense if the VM boundary itself is ever compromised. +- **Threat containment model**: the design doc explicitly treats every vCPU thread as running + malicious code the moment it starts, defining nested "trust zones" from least-trusted (guest + vCPUs) to most-trusted (host), with enforced barriers at I/O boundaries (e.g., all network + I/O passes through a rate-limited copy from the emulated NIC to the host TAP device). +- **Snapshot/restore (pause-resume)**: Firecracker supports pausing a running microVM, then + serializing full guest memory + emulated device/KVM state to disk as a snapshot (full or + incremental "diff" snapshots using KVM dirty-page tracking), which can later be loaded into a + fresh Firecracker process to resume execution from that exact point. Snapshot loading uses + copy-on-write memory mapping for fast resume, and a `VMGenID` device notifies the guest + kernel PRNG that it has resumed from a snapshot (avoiding randomness reuse across clones). + Directly relevant to agent-session lifecycle: pause an idle agent session to reclaim compute, + resume later with full memory/process state intact rather than re-initializing the + container/kernel. Caveats: network/vsock connections don't survive resume cleanly, and + re-using the same snapshot for multiple resumed VMs is explicitly called out as insecure + unless the integrator handles unique-identifier de-duplication. +- **Adoption beyond Lambda/Fargate**: production/integration adopters include Fly.io + (positions itself as using Firecracker for its container-hosting platform, chosen over + gVisor/Kata/nsjail for the isolation-vs-performance tradeoff), Kata Containers (uses + Firecracker as one of its VMM backends alongside QEMU/cloud-hypervisor), containerd (via + `firecracker-containerd`), plus smaller PaaS/CI vendors. +- **Comparison to gVisor/Kata (`k8s-agent-sandbox.md`)**: gVisor is a userspace reimplementation + of the Linux kernel interface in memory-safe Go — it still shares the host kernel underneath + but intercepts syscalls, trading a performance hit for eliminating most kernel-attack-surface + risk without needing virtualization. Kata Containers is architecturally closer to + Firecracker's approach (real Linux kernel in a lightweight VM) and can alternatively use + Firecracker itself as its VMM backend — Firecracker and Kata are not strictly competing but + sometimes composable (Kata-on-Firecracker), mirroring the "sandbox API decoupled from + isolation runtime" pattern already noted in `k8s-agent-sandbox.md`. +- **Comparison to WASM isolation (`wasmcloud.md`)**: WASM sandboxes sit at the opposite, + lighter-weight end of the spectrum — isolating by running code inside a language-level + virtual machine with a restricted capability surface, rather than giving each tenant its own + kernel. Cheaper and denser than even microVMs, but only works for code compiled to WASM; + can't run arbitrary Linux binaries/POSIX workloads the way Firecracker can. Firecracker's + approach is heavier but universal — any Linux guest workload, no code changes required. +- **Relevance to agent runtimes**: the per-session microVM pattern (spin up a fresh, + disposable, hardware-isolated VM per agent/session, optionally snapshot-pause it when idle, + resume it later) is exactly the pattern AWS Bedrock AgentCore uses (`aws-agents.md`), and is + a natural fit for agentic workloads because (a) agent sessions run untrusted/dynamically + generated code (tool calls, generated scripts) benefiting from VM-grade isolation rather than + shared-kernel container isolation; (b) sessions are often long-lived-but-idle (waiting on a + human or an LLM round-trip), making snapshot/pause-resume economically attractive versus + keeping a live pod/VM running; (c) the boot-time/density numbers make "one microVM per + session" tractable at scale in a way "one full EC2 instance per session" never was. + +## CF relevance + +Firecracker is the underlying isolation technology implicitly referenced wherever this +research set discusses "microVM-based session isolation" (AWS Bedrock AgentCore, +`aws-agents.md`) — this note fills that gap with the actual mechanics. For CF, the key +takeaway is that the container→microVM→full-VM isolation spectrum has a well-established, +open-source, hardware-virtualization point (Firecracker) that's specifically optimized for +the "many short-lived, untrusted, possibly-idle sessions" shape that agent workloads exhibit — +a different profile than CF's existing long-lived app-instance model was originally designed +for. The snapshot/pause/resume mechanism is particularly relevant to any CF exploration of +scale-to-zero for agent sessions (complementing `keda.md`'s autoscaling angle): rather than +terminating and cold-starting a full app instance, a Firecracker-style pause/resume could +preserve in-memory agent state across idle periods at container-like density. + +## Open questions + +- Would a CF-native agent runtime benefit from Firecracker-style per-session microVM + isolation directly (e.g., via Kata-on-Firecracker on Diego cells), or is this a level of + isolation only relevant for genuinely untrusted/multi-tenant code execution (tool calls, + generated scripts) rather than the agent's own orchestration logic? +- Firecracker's snapshot/restore explicitly warns against reusing one snapshot across multiple + resumed VMs without careful unique-ID handling — how would this constraint interact with a + CF-native "pause an idle agent session, resume it later" feature if built on similar + snapshotting technology? +- Given Firecracker's AWS-controlled (not foundation) governance model, is there a risk in + building CF-native tooling around it directly, versus using it indirectly via + foundation-governed layers like Kata Containers? +- Is the container→gVisor/Kata→Firecracker→full-VM isolation spectrum something CF should + formalize as a tiered sandboxing model (letting operators or workload authors choose a + point on the spectrum per use case), similar to how `k8s-agent-sandbox.md` already frames + gVisor/Kata as swappable backends behind one sandbox API?