Re-snapshot corruption: ext4 rootfs left dirty by vm.kill() produces EBADMSG and broken binaries

[jrimmer]

Problem

When forkd snapshot (or forkd from-image, which delegates to snapshot_cmd) boots a parent VM from an ext4 rootfs, the VM runs read-write and writes to the filesystem (journal entries, atime updates, apt package installations). When the snapshot is complete, vm.kill() SIGKILLs the firecracker process without giving the guest a chance to unmount the ext4 filesystem cleanly.

The guest's PID 1 (forkd-init.sh) uses exec to become the Python agent directly — there is no init system (no systemd, no sysvinit) to process shutdown signals (CtrlAltDel/SIGINT) or perform a clean filesystem unmount. Even if vm.shutdown() were called, the guest has no mechanism to handle it.

The rootfs ext4 file is left on disk with:

• Uncommitted journal transactions
• Potentially corrupted metadata (partial block writes interrupted by SIGKILL)
• A stale clean flag (the journal may have been committed but writes not synced to the host file)

Impact

When forkd snapshot --rootfs <existing.ext4> re-boots from the dirty rootfs, the guest kernel's ext4 driver replays the journal. A partially-written or corrupted journal produces severe filesystem corruption:

• EBADMSG / "Bad message" errors on file access
• Structure needs cleaning on directory operations
• apt directories vanish (/var/lib/apt/lists/partial missing)
• Binaries become unexecutable — file reports them as "data" file type, git returns "Exec format error"
• go list std reports false missing packages (package crypto/mlkem is not in std)

This corruption has been observed across multiple image types:

• Rust images: corrupted after re-snapshot with different --mem-size-mib
• Go images: /go/pkg/mod/ directory returns EBADMSG, mkdir /go/pkg/mod/cache fails
• Custom dev images: corrupted after re-snapshot to adjust memory

The workaround (setting GOPATH=/tmp/gopath and GOMODCACHE=/tmp/gopath/pkg/mod to bypass corrupted directories) is fragile and image-specific. The root cause must be fixed in forkd's snapshot path.

Root Cause

forkd from-image / forkd snapshot
  → BootConfig::ext4_rw(rootfs)     // guest mounts ext4 read-write
  → VM runs, writes to rootfs       // journal, atime, apt, etc.
  → snapshot captured               // memory + vmstate saved
  → vm.kill()                       // SIGKILL firecracker — NO clean unmount
  → rootfs file left dirty on disk  // uncommitted journal, possibly corrupt metadata

Next forkd snapshot --rootfs <same file>
  → ext4 journal replay              // may fail on corrupted journal
  → severe filesystem corruption     // EBADMSG, missing dirs, broken binaries

Solution

Run e2fsck -fy on the ext4 rootfs file before booting in snapshot_cmd:

• -f: Force a full check even when the clean flag is set (the flag can be stale after an unclean kill — the journal may have been committed but the host file's metadata blocks weren't synced)
• -y: Answer yes to all repair prompts non-interactively (this is a non-interactive CLI tool)
• Exit codes are handled as a bitmask: 0=clean, 1=corrected, 4=uncorrectable (warn but continue), 8/16=operational/usage error
• If e2fsck is not available on the host, a warning is printed and boot continues (graceful degradation)
• This is a no-op on freshly-built rootfs — mkfs.ext4 -d produces a clean filesystem, so e2fsck -fy finds nothing to fix

Why e2fsck on the host file is correct

Firecracker uses the ext4 file as a block device. Running e2fsck on the host file is equivalent to running fsck on a block device before mounting it — it repairs the filesystem at the metadata level, ensuring the guest boots from a consistent filesystem. e2fsck is part of e2fsprogs, the same package that provides mkfs.ext4 (already required by build-rootfs.sh).

Why not clean shutdown instead?

A clean guest shutdown (sync + unmount before kill) would be ideal, but is architecturally infeasible with the current guest init:

• forkd-init.sh is PID 1 and execs into the Python agent — no init system to handle signals
• Adding a signal handler to the Python agent wouldn't help — it can't unmount the root filesystem it's running from
• vm.shutdown() (CtrlAltDel via firecracker API) requires guest-side handling that doesn't exist
• Replacing the guest init with a proper init system is a much larger change

e2fsck before boot is the pragmatic fix that works with the existing architecture.

Related

• PR: fix(snapshot): repair ext4 rootfs before boot to prevent re-snapshot corruption #295

[WaylandYang]

Maintainer status: #295 remains CHANGES_REQUESTED. The immutable-baseline direction is sound, but the current head can adopt an already-dirty cached rootfs as the baseline, removes the previous
ootfs.ext4\ before the replacement is known-good (including a source/destination self-delete edge), and duplicates host-absolute rootfs paths across the main pack and sidecar. Keep this issue open until those safety and portability blockers are fixed and the rebased branch passes latest-main CI.