docs(§2): document host/guest runtime drift in the KVM lab chapter

_docs/02-lab-setup.md

@@ -185,8 +185,80 @@ it as `OTEL_EXPORTER_OTLP_ENDPOINT` through `env` explicitly, because `sudo`
 strips the environment — without that, a privileged guest binary would fall
 back to its own localhost and its telemetry would never reach the host's
 observability stack. This works because host builds and the guest are both
-Fedora 44 (glibc parity); the in-VM toolchains are the fallback when that
-assumption breaks.
+Fedora 44 — but that parity is a *starting* condition, not a standing one, and
+the next section is about what happens when it lapses.
+
+### When the guest falls behind the host
+
+Deploying a host build assumes the guest can still run it. Both machines start
+as Fedora 44, so they can. Then the host gets updated — as workstations do —
+and the guest does not, because a snapshot froze it. From then on your host
+compiler emits binaries against runtime symbols the guest has never heard of.
+
+Here is that failure, exactly as it appeared on this lab:
+
+```
+[host]$ TARGET=systems-target ./demo.sh cpp run
+→ copying app to fedora@192.168.124.7:/home/fedora/app
+/home/fedora/app: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.36' not found (required by /home/fedora/app)
+```
+
+**Note what this does not look like.** Run through a demo's own harness rather
+than by hand, that binary simply never starts, and every assertion downstream
+fails describing the demo instead of the cause — a port that never opened, a
+Landlock line that never printed, an OOM kill that never happened. The message
+above is the only place the truth appears, and you see it only if you run the
+binary directly.
+
+**The tell is which languages fail.** On the run that found this, six examples
+failed on the guest and every one of them failed in C++ alone, with Go and Rust
+passing on the same guest in the same run. That asymmetry is not a coincidence
+and it is not a C++ bug: Go and Rust link their runtimes statically, so they do
+not care what the guest ships. If one language fails on the guest and the others
+pass, suspect the runtime before you suspect your code.
+
+**Do not diagnose this by comparing compiler versions.** They will agree and
+prove nothing, because the compiler and the runtime are separate packages that
+move independently. On this host, `g++ --version` said 16.1.1 and the guest's
+`rpm -q libstdc++` said 16.1.1 — while the *host's* libstdc++ had moved to
+16.2.1 underneath both. Compare the symbol versions the loader actually cares
+about, on each side:
+
+```bash
+[host]$ strings /lib64/libstdc++.so.6 | grep -oE 'GLIBCXX_3\.4\.[0-9]+' | sort -V | tail -1
+GLIBCXX_3.4.36
+[vm]$  strings /lib64/libstdc++.so.6 | grep -oE 'GLIBCXX_3\.4\.[0-9]+' | sort -V | tail -1
+GLIBCXX_3.4.35
+```
+
+Guest lower than host means host builds cannot run there. The same check works
+for glibc with `GLIBC_` in place of `GLIBCXX_`.
+
+The fix is to bring the guest forward, and then — the half that is easy to
+forget — **to re-snapshot, because otherwise the next revert undoes it**:
+
+```bash
+[vm]$   sudo dnf -y update libstdc++ libgcc
+[host]$ ./scripts/lab/snapshot-vm.sh systems-target lab-ready
+```
+
+Do it on **every** guest, not just the one that failed. On this lab
+`systems-peer` carried the identical drift and had shown no symptom at all,
+because no C++ binary is ever deployed to it — re-snapshotting it unchanged
+would have preserved a defect that had simply not been triggered yet.
+
+Then prove it stuck, by reverting and re-running rather than by assuming:
+
+```bash
+[host]$ ./scripts/lab/revert-vm.sh systems-target lab-ready
+[host]$ python3 scripts/test-all-examples.py --mode vm --lang cpp
+8 passed, 0 failed, 0 skipped
+```
+
+A fix that lives only in the running guest is not a fix; it is a countdown to
+the next revert. And if you would rather not track this at all, the in-VM
+toolchains are the standing fallback — build on the guest and the question of
+whose runtime is newer never arises.
 
 ## Build, run, observe
 
@@ -258,10 +330,17 @@ guest's own `uname` in your terminal.
 - Snapshot `lab-ready` after provisioning, revert after destructive demos —
   and trust `/var/log/lab-ready` plus a guest kernel string in your output,
   not assumptions.
+- **Host/guest parity is a starting condition, not a standing one.** A snapshot
+  freezes the guest while your workstation keeps updating, and host-built
+  binaries then fail on the guest for a missing `GLIBCXX_`/`GLIBC_` symbol —
+  presenting as a broken demo, never as a version problem. The tell is C++
+  failing where Go and Rust pass, because those two link statically. Compare
+  symbol versions rather than compiler versions, fix every guest rather than the
+  one that failed, and **re-snapshot** — otherwise the next revert undoes it.
 
 Next, the third and final piece of setup: the Podman **LGTM observability
 stack**, so every later demo has somewhere to send its telemetry.
 
 ---
 
-<p><span class="status status--verified">verified</span> — both guests were provisioned from Fedora-Cloud-Base-Generic-44-1.7 on this host (target 4 GB at 192.168.124.7, peer 2 GB at 192.168.124.95); <code>/var/log/lab-ready</code> read back <code>bpftrace v0.24.2</code> and kernel <code>6.19.10-300.fc44</code> on both; <code>lab-ready</code> snapshots exist on both; and <code>deploy-to-vm.sh</code> ran the host-built C++ and Go binaries on <code>systems-target</code>, printing the guest kernel string <code>6.19.10-300.fc44.x86_64</code> versus <code>7.1.3-200.fc44.x86_64</code> locally.</p>
+<p><span class="status status--verified">verified</span> — both guests were provisioned from Fedora-Cloud-Base-Generic-44-1.7 on this host (target 4 GB at 192.168.124.7, peer 2 GB at 192.168.124.95); <code>/var/log/lab-ready</code> read back <code>bpftrace v0.24.2</code> and kernel <code>6.19.10-300.fc44</code> on both; <code>lab-ready</code> snapshots exist on both; and <code>deploy-to-vm.sh</code> ran the host-built C++ and Go binaries on <code>systems-target</code>, printing the guest kernel string <code>6.19.10-300.fc44.x86_64</code> versus <code>7.1.3-200.fc44.x86_64</code> locally. The <strong>&ldquo;When the guest falls behind the host&rdquo;</strong> section was added later and is verified separately, on the same lab: the host&rsquo;s <code>libstdc++</code> reached <code>16.2.1-2.fc44</code> (<code>libstdc++.so.6.0.36</code>, max <code>GLIBCXX_3.4.36</code>) while both guests sat at <code>16.1.1-2.fc44</code> (<code>libstdc++.so.6.0.35</code>, max <code>GLIBCXX_3.4.35</code>) behind snapshots taken before the update, and every host-built C++ binary then failed on the guest with the quoted <code>version `GLIBCXX_3.4.36&#39; not found</code>. A full-matrix run reproduced the asymmetry the section describes — six VM examples failing in C++ while Go and Rust passed on the same guest in the same run — and the quoted diagnostic commands, the <code>dnf</code> fix, and the re-snapshot were all run here: after <code>revert-vm.sh systems-target lab-ready</code> the guest reported <code>GLIBCXX_3.4.36</code> and <code>test-all-examples.py --mode vm --lang cpp</code> reported the quoted <code>8 passed, 0 failed, 0 skipped</code>, including the two-guest <code>41-capstone-fleet</code>. <code>systems-peer</code> was confirmed to carry the same drift while showing no symptom. Not exercised: <span class="status status--unverified">unverified</span> — the <code>GLIBC_</code> form of the symbol check is stated by analogy and was not the failure encountered here, which was <code>GLIBCXX_</code> only; and building inside the guest with the in-VM toolchains is named as the standing fallback but is not exercised by any example in this book.</p>