|
| 1 | +--- |
| 2 | +title: "The KVM lab" |
| 3 | +order: 2 |
| 4 | +part: "Setting Up" |
| 5 | +description: "Provision two disposable Fedora KVM guests with cloud-init — an observation-first toolset, snapshot/revert discipline, and a deploy script whose output proves your code ran on the guest kernel, not yours." |
| 6 | +duration: 45 minutes |
| 7 | +--- |
| 8 | + |
| 9 | +Chapter 1 proved that three toolchains build and run the same program on your |
| 10 | +workstation. This chapter builds the other half of the book's infrastructure: |
| 11 | +two disposable Fedora VMs — `systems-target` and `systems-peer` — that absorb |
| 12 | +everything you should never do to the machine you work on. A systems-programming |
| 13 | +book keeps asking you to run code as root, watch the kernel from the inside, |
| 14 | +fill filesystems, kill process trees, and misconfigure networks on purpose. |
| 15 | +The lab is where all of that happens, and one snapshot command makes any |
| 16 | +mistake reversible in seconds. |
| 17 | + |
| 18 | +The scripts are in `scripts/lab/`. They are parameterized by `LAB_PREFIX` |
| 19 | +(default `systems`), so the same set provisions, starts, snapshots, and |
| 20 | +destroys the guests by name. |
| 21 | + |
| 22 | +{% include excalidraw.html |
| 23 | + file="02-lab-topology" |
| 24 | + alt="Lab topology: a host band with libvirtd/KVM, the ~/.cache/systems-lab image cache holding the pristine Fedora Cloud Base image plus one qcow2 overlay per guest, and the scripts/lab directory; below it, the libvirt NAT network 192.168.124.0/24 containing systems-target (4 GB, .7) and systems-peer (2 GB, .95), both on kernel 6.19.10. An amber deploy arrow runs from the scripts to systems-target; a dashed line links the two guests for later two-host chapters; a note marks the lab-ready snapshot taken on both." |
| 25 | + caption="Figure 2.1 — the systems-* lab: cached base image + per-VM overlays on the host, two guests on libvirt NAT, one amber deploy path" %} |
| 26 | + |
| 27 | +## Why a disposable lab |
| 28 | + |
| 29 | +Three kinds of chapters force the issue. **Privileged demos** — capabilities, |
| 30 | +namespaces, cgroup writes, mount tricks — need root, and root that fat-fingers |
| 31 | +a path on your workstation is a bad afternoon. **Kernel observation** — bpftrace |
| 32 | +one-liners, perf profiles, the BCC tools — attaches instrumentation to a live |
| 33 | +kernel, and you want that kernel to be one you can reboot without losing your |
| 34 | +editor. **Destructive demos** — OOM kills, disk-full behavior, deliberately |
| 35 | +broken network configs — only teach if you actually break something, which |
| 36 | +means you need something safe to break. |
| 37 | + |
| 38 | +A VM answers all three at once, and adds the property no container gives you: |
| 39 | +**its own kernel**. Namespaces share the host kernel, so a container can't show |
| 40 | +you kernel-version drift, can't be probed as an isolated kernel, and can't be |
| 41 | +snapshot-reverted with its kernel state. The lab guests run their own |
| 42 | +`6.19.10-300.fc44` kernel while the host runs `7.1.3-200.fc44` — a visible |
| 43 | +difference this chapter turns into its cross-check. |
| 44 | + |
| 45 | +That said, the VM is not the default. The book's decision rule, applied at the |
| 46 | +top of every example: |
| 47 | + |
| 48 | +> **Run locally unless the demo needs root, needs to observe the kernel, or |
| 49 | +> needs two hosts.** Everything else — ordinary syscalls, file I/O, unprivileged |
| 50 | +> process work — runs faster and debugs easier on your workstation. |
| 51 | +
|
| 52 | +## Host prerequisites |
| 53 | + |
| 54 | +Provisioning needs four things on the host, and |
| 55 | +`scripts/lab/provision-vm.sh` checks for `virt-install` and `cloud-localds` |
| 56 | +before doing anything: |
| 57 | + |
| 58 | +```bash |
| 59 | +[host]$ sudo dnf install virt-install qemu-img cloud-utils libvirt |
| 60 | +[host]$ sudo usermod -aG libvirt "$USER" # then log out/in, or: sg libvirt -c '...' |
| 61 | +[host]$ ls -l /dev/kvm # hardware virtualization is available |
| 62 | +``` |
| 63 | + |
| 64 | +`virt-install` defines and boots the domain, `qemu-img` builds overlay disks, |
| 65 | +`cloud-localds` (from `cloud-utils`) packs the cloud-init seed, and membership |
| 66 | +in the `libvirt` group lets you talk to the system libvirt instance — every |
| 67 | +script exports `LIBVIRT_DEFAULT_URI=qemu:///system` so the guests live in the |
| 68 | +system instance, not a per-user session that vanishes with your login. |
| 69 | + |
| 70 | +## How provisioning works |
| 71 | + |
| 72 | +`scripts/lab/provision-vm.sh` is a straight pipeline: cached base image → |
| 73 | +per-VM overlay → cloud-init seed → `virt-install`. Each stage exists for a |
| 74 | +reason worth spelling out. |
| 75 | + |
| 76 | +**The cached base image.** The script downloads |
| 77 | +`Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2` once into |
| 78 | +`~/.cache/systems-lab/` and never modifies it. The version is pinned in the |
| 79 | +script ("Pin the exact image you tested; bump deliberately") — an unpinned |
| 80 | +"latest" would silently change the guest kernel under the whole book. |
| 81 | + |
| 82 | +**The overlay disk.** Instead of copying the 400 MB base per guest, the script |
| 83 | +layers a qcow2 overlay on top of it: |
| 84 | + |
| 85 | +```bash |
| 86 | +qemu-img create -f qcow2 -F qcow2 -b "$CACHE_DIR/$BASE_IMG" "$OVERLAY" "${DISK_GB}G" |
| 87 | +``` |
| 88 | + |
| 89 | +The overlay records only the blocks the guest changes; the base stays pristine |
| 90 | +and shared. This is what makes the guests *disposable* — destroying and |
| 91 | +re-provisioning a guest deletes one thin overlay and rebuilds it in seconds, |
| 92 | +with no re-download. |
| 93 | + |
| 94 | +**The cloud-init seed.** Fedora Cloud Base has no users, no packages, no SSH |
| 95 | +access — cloud-init is how it becomes yours on first boot. The script fills |
| 96 | +two placeholders (`__HOSTNAME__`, `__SSH_PUBKEY__`) in |
| 97 | +`scripts/lab/cloud-init/user-data.tmpl` with `sed`, then `cloud-localds` packs |
| 98 | +the result into a tiny seed image that is attached as a CD-ROM. On first boot |
| 99 | +the guest reads it, creates the `fedora` user with your SSH key and |
| 100 | +passwordless sudo, grows the root filesystem into the overlay, and installs |
| 101 | +the toolset. |
| 102 | + |
| 103 | +**Booting.** `virt-install --import` boots the overlay directly (no installer), |
| 104 | +with `--cpu host-passthrough` so the guest sees your real CPU features, |
| 105 | +virtio disk and network for near-native I/O, and `--graphics none |
| 106 | +--noautoconsole` because everything from here on is SSH. The script also |
| 107 | +refuses to clobber: an existing domain of the same name is an error, and the |
| 108 | +fix is an explicit `./destroy-vm.sh <name>` — rebuilds are deliberate, never |
| 109 | +accidental. |
| 110 | + |
| 111 | +## What goes in the guest — and what deliberately doesn't |
| 112 | + |
| 113 | +The package list in `scripts/lab/cloud-init/user-data.tmpl` *is* the book's |
| 114 | +tooling philosophy, so read it as an argument, not a manifest. The first block |
| 115 | +is **observation tooling**: `bcc-tools`, `bpftrace`, `bpftool`, `libbpf-tools` |
| 116 | +(precompiled CO-RE tools — `opensnoop`, `execsnoop`, `biolatency`), `perf`, |
| 117 | +`strace`, `ltrace`, `gdb` plus `gdb-gdbserver`, `valgrind`, `sysstat`. That |
| 118 | +block is the point of the lab: this book uses eBPF **as tooling, not as a |
| 119 | +build target**, which is why there is *no* clang/llvm/libbpf-devel eBPF build |
| 120 | +chain in the guest — the template says so in a comment right above the list. |
| 121 | +You will trace with bpftrace constantly; you will not compile BPF programs. |
| 122 | + |
| 123 | +The second block is **per-language toolchains** (`gcc-c++`, `clang`, `cmake`, |
| 124 | +`ninja-build`, `golang`, `rust`/`cargo`) so demos that must compile in-VM — |
| 125 | +privileged code, namespace experiments, glibc-parity fallbacks — can. The rest |
| 126 | +is `podman`/`crun` for the container chapters, networking tools (`tcpdump`, |
| 127 | +`socat`, `nmap-ncat`) for the two-host chapters, and `cockpit`, whose socket |
| 128 | +cloud-init enables so the web console is reachable at |
| 129 | +`https://<guest-ip>:9090` — a useful visual second opinion on guest load. |
| 130 | + |
| 131 | +The `runcmd` section ends with the **readiness stamp**: it writes |
| 132 | +`bpftrace --version` and `uname -r` into `/var/log/lab-ready`. That file is |
| 133 | +the lab's contract — if it exists and names a bpftrace, cloud-init finished |
| 134 | +and the observation tooling works. Every later chapter can check one file |
| 135 | +instead of guessing whether provisioning completed. |
| 136 | + |
| 137 | +Packages a single demo needs (fio, rt-tests, numactl) stay *out* of the base |
| 138 | +on purpose: install them on demand, then re-snapshot to bake them in. A lean |
| 139 | +base keeps provisioning fast and the snapshot meaningful. |
| 140 | + |
| 141 | +## Snapshots: the discipline that makes "destructive" safe |
| 142 | + |
| 143 | +`scripts/lab/snapshot-vm.sh <vm>` captures a snapshot named `lab-ready` — |
| 144 | +RAM *and* disk, because the guest is running, so a revert restores it |
| 145 | +*running*, tools warm, in seconds. The discipline has two halves: |
| 146 | + |
| 147 | +- **Snapshot immediately after provisioning** (and after any deliberate |
| 148 | + addition you want to keep). This is your known-good tooled baseline. |
| 149 | +- **Revert after every destructive demo**: `scripts/lab/revert-vm.sh |
| 150 | + systems-target` puts the guest back to `lab-ready`, no forensics, no |
| 151 | + "did that experiment leave state behind?" doubt. |
| 152 | + |
| 153 | +Re-running `snapshot-vm.sh` replaces the same-named snapshot, so refreshing |
| 154 | +the baseline is idempotent. `destroy-vm.sh` is the nuclear option — it |
| 155 | +undefines the domain and deletes its overlay and seed, leaving only the cached |
| 156 | +base image, from which a fresh guest is minutes away. |
| 157 | + |
| 158 | +## The daily lifecycle |
| 159 | + |
| 160 | +Day to day you touch two scripts. `scripts/lab/lab-up.sh` starts whichever of |
| 161 | +`systems-target`/`systems-peer` are defined but stopped, waits for DHCP |
| 162 | +leases, and prints IPs; `eval "$(./lab-up.sh --export)"` drops `TARGET_IP` and |
| 163 | +`PEER_IP` straight into your shell for scripted tests. `scripts/lab/lab-down.sh` |
| 164 | +sends ACPI shutdowns, waits up to 60 seconds, and — deliberately — deletes |
| 165 | +nothing; disks and snapshots survive, and `FORCE=1` hard-stops a guest that |
| 166 | +ignores ACPI. `vm-ip.sh` answers "what address did libvirt lease this guest?" |
| 167 | +by reading the DHCP lease (guest-agent fallback). One host-specific note: on |
| 168 | +this machine the libvirt `default` network is `192.168.124.0/24` with gateway |
| 169 | +`192.168.124.1` — not the stock `192.168.122.0/24` you'll see in most libvirt |
| 170 | +docs — so never hardcode an address; always ask `vm-ip.sh`. |
| 171 | + |
| 172 | +## Deploying host builds |
| 173 | + |
| 174 | +`scripts/lab/deploy-to-vm.sh <vm> <binary> [-- args]` is how host-built code |
| 175 | +reaches the guest: it resolves the IP, `scp`s the binary to |
| 176 | +`/home/fedora/`, marks it executable, and `exec`s it over `ssh -t` so Ctrl-C |
| 177 | +reaches the remote process. Two toggles matter. `SUDO=1` runs the binary under |
| 178 | +sudo for the demos that genuinely need root — the default is unprivileged, |
| 179 | +matching the decision rule. And if `OTEL_ENDPOINT` is set, the script forwards |
| 180 | +it as `OTEL_EXPORTER_OTLP_ENDPOINT` through `env` explicitly, because `sudo` |
| 181 | +strips the environment — without that, a privileged guest binary would fall |
| 182 | +back to its own localhost and its telemetry would never reach the host's |
| 183 | +observability stack. This works because host builds and the guest are both |
| 184 | +Fedora 44 (glibc parity); the in-VM toolchains are the fallback when that |
| 185 | +assumption breaks. |
| 186 | + |
| 187 | +## Build, run, observe |
| 188 | + |
| 189 | +Provision both guests — target at the 4 GB default, peer smaller: |
| 190 | + |
| 191 | +```bash |
| 192 | +[host]$ cd scripts/lab |
| 193 | +[host]$ ./provision-vm.sh systems-target |
| 194 | +[host]$ RAM_MB=2048 ./provision-vm.sh systems-peer |
| 195 | +``` |
| 196 | + |
| 197 | +Give cloud-init a minute or two (package installation dominates), then read |
| 198 | +the readiness stamp: |
| 199 | + |
| 200 | +```bash |
| 201 | +[host]$ ssh fedora@"$(./vm-ip.sh systems-target)" cat /var/log/lab-ready |
| 202 | +bpftrace v0.24.2 |
| 203 | +6.19.10-300.fc44.x86_64 |
| 204 | +``` |
| 205 | + |
| 206 | +That one file confirms the whole chain: the guest booted, cloud-init ran to |
| 207 | +completion, the observation tooling works, and you can see which kernel the |
| 208 | +lab runs. On this run the guests came up at `192.168.124.7` (target) and |
| 209 | +`192.168.124.95` (peer). Now freeze the baseline on both: |
| 210 | + |
| 211 | +```bash |
| 212 | +[host]$ ./snapshot-vm.sh systems-target |
| 213 | +[host]$ ./snapshot-vm.sh systems-peer |
| 214 | +``` |
| 215 | + |
| 216 | +`virsh snapshot-list` on each should show one `lab-ready` snapshot. That's the |
| 217 | +lab: built once, reverted forever. |
| 218 | + |
| 219 | +## Cross-check: prove the code runs *there* |
| 220 | + |
| 221 | +Chapter 1's example prints the kernel it runs on — which makes it a perfect |
| 222 | +independent check that `deploy-to-vm.sh` really executes on the guest rather |
| 223 | +than, say, quietly running the local binary. Run it both ways: |
| 224 | + |
| 225 | +```bash |
| 226 | +[host]$ cd examples/01-hello-syscalls |
| 227 | +[host]$ ./demo.sh cpp |
| 228 | +pid <PID> on Linux 7.1.3-200.fc44.x86_64 (x86_64) |
| 229 | +[host]$ TARGET=systems-target ./demo.sh cpp |
| 230 | +→ copying app to fedora@192.168.124.7:/home/fedora/app |
| 231 | +→ running on systems-target (Ctrl-C to stop): |
| 232 | +pid <PID> on Linux 6.19.10-300.fc44.x86_64 (x86_64) |
| 233 | +``` |
| 234 | + |
| 235 | +Same binary, two different kernel strings: `7.1.3-200.fc44` is your host, |
| 236 | +`6.19.10-300.fc44` is the guest. The example's per-language `demo.sh` wires |
| 237 | +this up — with `TARGET` set, its `run` step calls |
| 238 | +`scripts/lab/deploy-to-vm.sh "$TARGET" <binary>` instead of executing locally, |
| 239 | +so every example in the book gets remote execution for free. The Go build |
| 240 | +behaves identically (`TARGET=systems-target ./demo.sh go`); both were run on |
| 241 | +this lab and printed the guest kernel string. The kernel version in the output |
| 242 | +is the observable proof — a claim like "it deployed" is worthless next to the |
| 243 | +guest's own `uname` in your terminal. |
| 244 | + |
| 245 | +## What you learned |
| 246 | + |
| 247 | +- The lab exists for exactly three reasons — root, kernel observation, and |
| 248 | + two-host demos — and the decision rule says everything else runs locally. |
| 249 | +- Provisioning is a cheap pipeline: pinned cached base image → qcow2 overlay |
| 250 | + per guest → cloud-init seed → `virt-install`; guests are disposable because |
| 251 | + only the overlay is theirs. |
| 252 | +- The guest toolset is observation-first (bcc/bpftrace/perf/gdbserver), with |
| 253 | + in-VM toolchains as a fallback and deliberately no eBPF build chain. |
| 254 | +- Snapshot `lab-ready` after provisioning, revert after destructive demos — |
| 255 | + and trust `/var/log/lab-ready` plus a guest kernel string in your output, |
| 256 | + not assumptions. |
| 257 | + |
| 258 | +Next, the third and final piece of setup: the Podman **LGTM observability |
| 259 | +stack**, so every later demo has somewhere to send its telemetry. |
| 260 | + |
| 261 | +--- |
| 262 | + |
| 263 | +<p><span class="status status--verified">verified</span> — both guests were provisioned from Fedora-Cloud-Base-Generic-44-1.7 on this host (target 4 GB at 192.168.124.7, peer 2 GB at 192.168.124.95); <code>/var/log/lab-ready</code> read back <code>bpftrace v0.24.2</code> and kernel <code>6.19.10-300.fc44</code> on both; <code>lab-ready</code> snapshots exist on both; and <code>deploy-to-vm.sh</code> ran the host-built C++ and Go binaries on <code>systems-target</code>, printing the guest kernel string <code>6.19.10-300.fc44.x86_64</code> versus <code>7.1.3-200.fc44.x86_64</code> locally.</p> |
0 commit comments