diff --git a/_docs/45-linux-analysis-suites.md b/_docs/45-linux-analysis-suites.md new file mode 100644 index 0000000..50a84b7 --- /dev/null +++ b/_docs/45-linux-analysis-suites.md @@ -0,0 +1,503 @@ +--- +title: "Linux analysis suites: Cockpit, SystemTap, and PCP watch a whole fleet" +order: 45 +part: "Appendices: Tooling" +description: "linux-analysis-suites builds a single Fedora-based, multi-stage Containerfile (lsp45-pcp, 265 MB) shipping Performance Co-Pilot's pmcd with two live PMDAs -- openmetrics and podman -- run rootless with podman, alongside Cockpit's zero-scripting web console and SystemTap's kernel-module tracing on the systems-target VM: verified PASS 10/0 with a real pminfo value 42, live podman-socket container visibility, a working SystemTap kernel-module smoke test, and Cockpit's socket answering HTTP 200." +duration: "35 minutes" +--- + +Every observation tool Part 8 reached for — `strace`, `gdb`, bcc-tools, +`bpftrace` — answers questions about **one process**, watched while you sit +at the terminal running it. That framing stops scaling the moment there is +more than one machine to worry about: nobody wants to SSH into a fleet and +run `strace -p` by hand on each host. This chapter is the first of Part 13's +appendices, and it pivots from single-process tools to three fleet-wide +**suites**, each built on a different philosophy for the same underlying +question — what is this system doing right now. Cockpit answers it with a +web console you never script. SystemTap answers it by compiling a kernel +module per probe — bcc's harder-to-build sibling from Chapter 30. And PCP +(Performance Co-Pilot) answers it by pulling metrics out of things this book +already built: a `podman.sock` (Chapter 34) and an OpenMetrics document +(Chapter 38). + +{% include excalidraw.html + file="45-analysis-suite-landscape" + alt="Three side-by-side boxes under a banner reading three philosophies for fleet-wide observation. Left: a browser icon into a Cockpit box, labeled zero-scripting web console, cockpit-ws multiplexing sessions over a socket answering on port 9090. Middle: a stap CLI box compiling into a kernel-module icon, labeled SystemTap -- bcc's harder-to-build kernel-module-compiling sibling from Chapter 30, needing matching kernel-devel. Right: a pmcd hub box with two labeled PMDA arms -- pmda-openmetrics reading a file:// document and pmda-podman reading a podman.sock -- both arrows curving back to two smaller ghost boxes captioned Chapter 34 containers and Chapter 38 OpenMetrics, showing PCP pulls metrics out of things this book already built. Caption: three ways to observe a fleet -- click, script, or scrape." + caption="Figure 45.1 — three philosophies for fleet-wide observation: Cockpit's zero-scripting console, SystemTap's kernel-module scripting (bcc's harder-to-build sibling from Chapter 30), and PCP pulling metrics out of two things this book already built" %} + +> **Tools used** — `podman` (host, build/run/exec — in +> `scripts/check-host.sh`); `pmcd`/`pminfo`/`pmrep`/`pcp` (a third location: +> **inside** the `lsp45-pcp` container this chapter builds, neither host nor +> VM); `cockpit`/`cockpit-podman` (`systems-target` VM, cloud-init; +> `cockpit.socket` enabled); `stap` (`systems-target` VM, root, kernel-module +> backend, needs `kernel-devel-$(uname -r)`); `curl` (host, socket check). + +A note on structure before any of that: `examples/45-linux-analysis-suites/` +has no C++, Go, or Rust source at all. PCP, Cockpit, and SystemTap are +OS-level suites you point at a running system, not libraries you link into +one, so there is nothing per-language to write. `./demo.sh [cpp|go|rust] +[build|run]`'s language argument exists purely so +`scripts/test-all-examples.py`'s per-language loop has something to pass — +`examples/manifest.yaml` lists `langs: [cpp]` for exactly this reason, +running the check once rather than three times for no behavioral gain. +Whichever of the three is given, the image built and the container run are +byte-identical. Code excerpts below are therefore plain fenced blocks, never +the tri-language `codetabs` include this book uses everywhere else — there +is only one path through this example, the same handling Chapter 44 used for +its single-language runtime deep dive. + +## Cockpit: a web console with no scripting required + +Cockpit is the philosophy of "point a browser at port 9090 and read the +dashboard" — no query language, no client to install, no script to write. +`cockpit-ws` is the one process on the target machine; the browser does the +rendering, and a single WebSocket carries every session it multiplexes. +`cockpit-podman` extends that same console with a Podman-aware panel — the +containers, images, and pod state a fleet operator would otherwise have to +`podman ps` for by hand across every host. This chapter's `systems-target` +VM ships both, enabled by cloud-init: `cockpit.socket` is active and +answering before this chapter's demo ever runs. + +## SystemTap: bcc's kernel-module-compiling sibling + +Chapter 30 put SystemTap side by side with bcc-tools and bpftrace and found +its kernel-module backend blocked by a real `kernel-devel`/running-kernel +mismatch — a friction point flagged, not resolved, in that chapter. This +appendix is where it gets resolved: once `kernel-devel-$(uname -r)` matches +the running kernel, `stap` compiles a real, loadable kernel module per probe +script, runs it, and unloads it when the script exits — the same +observation goal as bcc's BPF programs, reached by compiling and inserting +an actual `.ko` instead of loading a verified BPF bytecode program. That +extra step — a real compile, against real kernel headers, every invocation +— is exactly why bcc and bpftrace are this book's default toolkit and +SystemTap is introduced as the sibling technology: more setup cost, same +observational reach once the setup is right. + +## PCP: metrics pulled from what this book already built + +Performance Co-Pilot takes a third stance again: instead of a console you +click through or a script you write per question, PCP runs one collector +daemon, `pmcd`, and lets independent **PMDAs** (Performance Metrics Domain +Agents) register whatever metrics they know how to produce into one shared +namespace, queryable with `pminfo`, `pmrep`, or the summary command `pcp` +itself. This chapter's image layers exactly two PMDAs, and both pull their +numbers from things this book already built rather than from anything new: +`pmda-openmetrics` bridges an OpenMetrics/Prometheus-style document — the +same format Chapter 38 wired a real exporter around — into PCP's namespace, +and `pmda-podman` reads container and pod state over a `podman.sock`, the +same rootless socket Chapter 34 built a container fleet around. PCP doesn't +introduce a new thing to observe; it gives a uniform namespace to metrics +this book's own examples already produce. + +The openmetrics side is scraped from a tiny, real document this example +ships, not a live network endpoint: + +``` +# HELP lsp45_answer A known constant this chapter ships so the openmetrics +# PMDA has something real to scrape without any network access -- pointed at +# via a `file://` config.d source, not an HTTP endpoint. +# TYPE lsp45_answer gauge +lsp45_answer 42 +``` + +`pmda-openmetrics`'s Python source supports a `file://` config.d source +scheme directly (`self.url.startswith('file://')` reads the path with a +plain `open()`), so pointing its config at `config/lsp45.prom` gives a real +scrape of a real file, every run, with zero network dependency. + +### Why Fedora, not UBI + +Every other container example in this book — Chapter 34's — builds on a +UBI 10 base, per this project's own convention. This one doesn't, and the +reason is worth stating plainly: checked directly against a real `dnf +install`, `pcp` and its PMDA packages are **not resolvable on UBI 9 or UBI +10** without a RHEL entitlement — no free, unauthenticated UBI repository +carries them, nor does EPEL for RHEL 10. `registry.redhat.io/rhel9/pcp` +exists, but it needs Red Hat portal authentication, and there is no +`rhel10/pcp` image at all. This book's readers, by this project's own +conventions, are assumed **not** to have a RHEL subscription. Fedora's own +repositories carry `pcp 7.1.5` and every PMDA package this chapter needs, +fully unauthenticated — so `registry.fedoraproject.org/fedora:44` is the +base that actually builds for everyone, not just readers with a Red Hat +account. + +One PMDA is deliberately absent from this image: `pmda-bcc`, PCP's own +bridge into the eBPF world Chapter 30 covered directly. Activating it needs +a working BCC/eBPF stack underneath — raw `BPF` syscalls, `/sys/kernel/ +debug`, a kernel-devel package matching the exact running kernel — which in +turn needs a **super-privileged container**, the opposite of the rootless +posture every other example in this book targets. It's covered here only as +a name: PCP's privileged sibling PMDA, demonstrated properly on the +`systems-target` VM where Chapter 30's real eBPF tooling already runs, not +shipped inside this rootless image. + +## How the code works + +`Containerfile` is three real stages — `builder`, `smoke`, `runtime` — and +the ordering between the last two is itself the chapter's first lesson. + +**Stage 1 — `builder`.** After `dnf install`ing `pcp pcp-system-tools +pcp-pmda-openmetrics pcp-pmda-podman` and pointing the openmetrics PMDA's +config at `config/lsp45.prom`, the build has to activate both PMDAs. This +is the one step that took real iteration to get right, and the fix is +right there in the `RUN` line: + +```dockerfile +RUN /usr/libexec/pcp/bin/pmcd -f >/tmp/pmcd-build.log 2>&1 & \ + for i in $(seq 1 20); do \ + pminfo -f pmcd.version >/dev/null 2>&1 && break; sleep 1; \ + done && \ + cd /var/lib/pcp/pmdas/openmetrics && yes '' | ./Install && \ + cd /var/lib/pcp/pmdas/podman && yes '' | ./Install +``` + +Each PMDA's `./Install` script (`pmdaproc.sh`) decides, at the end of its +run, whether `pmcd` needs restarting to pick up the change — and its +default way of doing that is `systemctl restart pmcd.service`. A build +layer has no systemd, so that call fails, and `Install`'s failure path +**reverts its own `pmcd.conf` edit** — the PMDA it just tried to register +silently disappears, with no loud error at all. `pmdaproc.sh` has a +cheaper path, though: if `pminfo -v pmcd.version` already succeeds against +a `pmcd` that's already running, it sends a plain `pmsignal -a -s HUP +pmcd` instead of attempting a restart — no systemd required. Starting +`pmcd` in the background **before** either `./Install` call, and polling +`pminfo` until it answers, takes that cheaper path for both PMDAs, and +nothing gets reverted. + +**Stage 2 — `smoke`.** This stage exists purely as a build-time gate that +is never shipped: it starts `pmcd` fresh from the config `builder` just +wrote, and asserts real values come back, not just that the daemon starts. + +```dockerfile +FROM builder AS smoke +RUN /usr/libexec/pcp/bin/pmcd -f >/tmp/pmcd.log 2>&1 & \ + for i in $(seq 1 20); do \ + pminfo -f pmcd.version >/dev/null 2>&1 && break; sleep 1; \ + done; \ + pcp | tee /tmp/pcp-summary.txt; \ + grep -q openmetrics /tmp/pcp-summary.txt && \ + grep -q podman /tmp/pcp-summary.txt && \ + pminfo -f openmetrics.lsp45.lsp45_answer | tee /tmp/om.txt && \ + grep -q "value 42" /tmp/om.txt && \ + pminfo -f kernel.all.load | tee /tmp/load.txt && \ + grep -q "kernel.all.load" /tmp/load.txt +``` + +If either PMDA silently failed to register — exactly the failure mode the +previous paragraph describes — one of these `grep -q` checks fails, the +`RUN` exits non-zero, and the whole `podman build` fails right here. That +is a stronger guarantee than any runtime-only health check: a broken PMDA +never gets the chance to ship. + +**Stage 3 — `runtime`, and the second real bug.** A gate that nothing +chains onto is a gate that never runs — and `podman build` (like Docker) +only builds the stages reachable from the final stage's own `FROM` chain. +An earlier version of this Containerfile wrote `FROM builder AS runtime`, +which compiles cleanly and looks correct, but it leaves `smoke` completely +disconnected: no `COPY --from=smoke` and no `FROM smoke` anywhere means +`podman build` silently skips the `smoke` stage's `RUN` altogether, and the +"build-time gate" above never executes at all. The fix is one word: + +```dockerfile +# Shipped runtime. Reuses the smoke stage's already-activated PMNS/pmcd.conf +# rather than re-running Install a third time -- FROM smoke (not FROM +# builder) also means podman actually builds the smoke stage by default: +# with no COPY --from/FROM edge into it, an unreferenced intermediate stage +# is silently skipped by `podman build`, which would make the "build-time +# gate" a no-op. Chaining runtime FROM smoke is what makes the gate real. +FROM smoke AS runtime +EXPOSE 44321 +ENTRYPOINT ["/usr/libexec/pcp/bin/pmcd", "-f"] +``` + +`FROM smoke AS runtime` both reuses `smoke`'s already-activated PMDA state +(no third `./Install` re-run needed) and forces `podman build` to actually +build `smoke` first — the gate stage only runs because something real +depends on it existing. + +{% include excalidraw.html + file="45-pcp-build-gate" + alt="A vertical three-stage pipeline. Stage one, builder, installs pcp packages, copies lsp45.prom, starts pmcd, and runs two ./Install calls against that live pmcd. An arrow labeled FROM builder AS smoke descends into stage two, smoke, which restarts pmcd and asserts both PMDAs are listed plus a real value 42 and a real kernel.all.load sample, gated by grep -q calls that can fail the whole podman build. A ghost, disconnected box beside stage two reads FROM builder AS runtime -- the skipped path, dashed and crossed out, with a note that podman silently never builds a stage nothing chains onto. The real arrow, solid and amber, is labeled FROM smoke AS runtime and descends into stage three, runtime, which reuses the smoke stage's already-activated PMNS/pmcd.conf and carries only EXPOSE 44321 and the pmcd entrypoint. Caption: chaining onto smoke, not builder, is what forces the gate stage to actually build." + caption="Figure 45.2 — the PCP image's build gate: chaining runtime onto smoke forces the gate stage to build, and a PMDA that fails to register fails the build, not just the demo" %} + +## Errors, three ways + +With one example and no per-language split, "three ways" means three +different moments and volumes an error can surface at, rather than three +languages producing the same one. + +**Loud, at build time.** The `smoke` stage's `grep -q` chain is a real gate: +during this example's own development, deliberately starting `pmcd` *after* +the `./Install` calls (undoing the fix above) reproduced the exact failure +this chapter warns about — `podman build` failed outright, with the +`pmcd.conf` revert visible in the build log, rather than shipping a broken +image that merely looked fine. + +**Silent, inside a passing build.** Before the fix, the failure mode isn't +loud at all: `./Install`'s revert of its own `pmcd.conf` edit produces no +error text a casual read of the build log would catch — the PMDA simply +never appears in `pcp`'s summary, and nothing about `podman build`'s exit +code says so. This is the errors-three-ways lesson worth sitting with: a +build can succeed and still have silently dropped a component, which is +exactly why the `smoke` stage's assertions exist as a second, independent +check rather than trusting `podman build`'s own exit status. + +**A graceful, reported degradation at runtime.** Not every gap here is a +bug. If no rootless `podman.sock` is present when the container starts, +`demo.sh` reports it plainly (`note: no rootless podman.sock at $sock -- +podman PMDA will register but report zero container values`) and continues +— the podman PMDA's namespace still registers correctly, it simply has no +live container data to report. That is a documented, expected runtime +condition, not a failure, and treating it as one would make an +environment-dependent optional feature look like a broken build. + +## Concurrency lens + +Each suite here answers "how many things are running at once" a different +way. `pmcd` is one reactor process fronting **N independent PMDA +processes**, each talking to `pmcd` over its own pipe — the same +process-isolation discipline Chapter 11 built `pmon` around, applied here +to metrics collection instead of service supervision: a PMDA that hangs or +crashes takes down only its own slice of the namespace, never `pmcd` +itself or any sibling agent. SystemTap's `<<<` operator is the sharpest +concurrency idea in this chapter: it's a per-CPU-safe **statistical +aggregation** — every CPU accumulates into its own private bucket with no +shared-memory contention at all, and only the `end` probe's `@count()` +call folds those per-CPU buckets together into the single number a script +prints. That's the same lock-free instinct Chapter 26 built atomics and +CAS loops around, reached here through a completely different mechanism: +avoid the race by never sharing the counter in the first place, rather +than serializing access to one. Cockpit's concurrency story is the +simplest of the three: one `cockpit-ws` process multiplexes every browser +session over a single WebSocket-per-client model, with no per-session +process fork at all. + +## Build, run, observe + +```console +[host]$ cd examples/45-linux-analysis-suites && ./demo.sh cpp run +``` + +`run()` mounts the host's own rootless `podman.sock` into the container, if +one is present, before starting it: + +```bash +run() { + podman rm -f "$CONTAINER" >/dev/null 2>&1 || true + + # If a rootless podman.sock is up for this user, mount it so the layered + # podman PMDA sees REAL running containers (including this one) instead + # of just registering with zero values. This is a genuine, verified + # result: pmda-podman's own Install script only checks for + # /run/podman/podman.sock to decide whether to prompt about starting the + # podman service -- it works identically well whether the socket behind + # that path belongs to root or, as here, an unprivileged user. + local sock="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/podman/podman.sock" + local mount_args=() + if [ -S "$sock" ]; then + mount_args=(-v "$sock:/run/podman/podman.sock") + else + echo "note: no rootless podman.sock at $sock -- podman PMDA will" \ + "register but report zero container values" >&2 + fi + + podman run -d --name "$CONTAINER" --cpus=2 --memory=256m \ + "${mount_args[@]}" "$IMAGE" +``` + +This session's real transcript, on the Fedora 44 host (kernel +`7.1.4-204.fc44`, podman 5.8.4 rootless): + +```console +--- pcp: suite summary (openmetrics/podman should both be listed) --- +Performance Co-Pilot configuration on 9fb541ff035b: + platform: Linux 9fb541ff035b 7.1.4-204.fc44.x86_64 ... + pmcd: Version 7.1.5-3, 11 agents + pmda: root pmcd proc pmproxy xfs podman linux mmv kvm jbd2 openmetrics +--- pminfo -f: a real scraped openmetrics value (file:// source, no network) --- +openmetrics.lsp45.lsp45_answer + value 42 +--- pminfo -f: real podman container visibility (via the mounted socket) --- +podman.container.name + inst [0 or "9fb541ff...ea3a3b5"] value "lsp45-pcp-demo" +podman.container.running + inst [0 or "9fb541ff...ea3a3b5"] value 1 +--- pmrep: one live sample of a core, always-present metric --- + k.a.load k.a.load k.a.load + 1 minute 5 minute 15 minut + 0.970 0.540 0.420 +``` + +`pmcd: Version 7.1.5-3, 11 agents` — the two PMDAs this chapter layers +(`openmetrics`, `podman`) plus the nine that ship in any PCP install +(`root pmcd proc pmproxy xfs linux mmv kvm jbd2`), all eleven active. The +openmetrics PMDA hands back the exact `42` `config/lsp45.prom` ships, every +run. The podman PMDA, once the container's own `podman.sock` is mounted +into it, sees `lsp45-pcp-demo` — the very container running the query — +reporting itself as `podman.container.running value 1`. `pmrep`'s three +load-average columns come from `kernel.all.load`, a core PCP metric with +nothing to do with either PMDA this chapter added — proof that `pmcd`'s +base collection works independent of anything layered on top. + +Cockpit lives on the `systems-target` VM, not this container, and the +check here is a socket check rather than a browser session: + +```console +[vm]$ systemctl is-active cockpit.socket +active +[vm]$ curl -sk https://192.168.124.7:9090/ +``` + +That `curl` returns a real HTTP 200 body — the socket answers, and +`cockpit-364-1.fc44` plus `cockpit-podman` are both installed by +cloud-init. + +> **Cockpit's web-UI panels** unverified — the Overview, Podman, and Logs +> panels a browser would render from that same socket are described here, +> not exercised: this repo has no browser-automation tooling, and the +> Cockpit console is meant to be looked at, not scraped. What's verified is +> narrower and mechanical — the socket is up and answers `HTTP 200` — and +> that's exactly the boundary this chapter's status footer draws. + +SystemTap runs on the same VM, as root, against its kernel-module backend: + +```console +[vm]$ sudo stap -e 'probe oneshot { println("stap-module-ok") }' +stap-module-ok +``` + +That one line is a real kernel module compiled against +`kernel-devel-$(uname -r)` (matched to the running `6.19.10-300.fc44` +kernel), loaded, run once, and unloaded — the payoff Chapter 30 flagged as +blocked by a kernel-devel mismatch, resolved here once the matching +package is installed. A slightly larger script aggregates real syscall +activity from a controlled workload: + +```console +[vm]$ sudo stap -c '' -e 'global opens; probe syscall.openat { opens[execname()] <<< 1 } probe end { foreach (e in opens-) printf("%s: %d openat\n", e, @count(opens[e])) }' +cat: 155 openat +sh: 43 openat +``` + +Five `cat /etc/hostname` invocations under a driving `sh` produce 155 +`openat` calls attributed to `cat` (the binary itself, its shared +libraries, and the target file, each invocation) and 43 to the shell +driving it — `<<<` accumulating per-CPU, `@count()` folding those buckets +together only once, at the `end` probe. A practical caveat worth stating +rather than hiding: this VM has no `kernel-debuginfo` installed, so a +`WARNING: cannot find module kernel debuginfo` line appears alongside the +real output above; `syscall.*` probes (kprobes and tracepoints, both used +here) don't need it — only DWARF-based variable inspection would. + +The gate the runner checks: + +```console +[host]$ LSP_LANG=cpp lua verify.lua +... +PASS 10 / FAIL 0 +``` + +Image size on this host: **265 MB** (`podman images lsp45-pcp`) — much +larger than Chapter 34's few-tens-of-MB runtime images, because PCP's own +dependency chain (Python, `pcp-libs`, `pcp-system-tools`) is the size +driver, and there's no further multi-stage trim available when `runtime` +needs the exact same userspace `smoke` just proved works. + +## Cross-check: two independent gates agree on the same real value + +This chapter's central claim — the `openmetrics` PMDA genuinely parses +`config/lsp45.prom` rather than the value `42` being hardcoded somewhere in +the image — is checkable two independent ways, and both agree. The first +is the `smoke` stage's own build-time assertion, running entirely inside +`podman build`, with no external harness watching: `grep -q "value 42" +/tmp/om.txt` against a fresh `pminfo -f openmetrics.lsp45.lsp45_answer` +call. The second is `verify.lua`, run from outside the image entirely, +against a container the harness itself started: + +```lua +-- --------------------------------------------------------------------------- +-- 4. a REAL scraped openmetrics value, from the file:// source this example +-- ships (config/lsp45.prom) -- not a live network scrape, but a real +-- parse of a real file, every time. +-- --------------------------------------------------------------------------- + +local om = checks.run("podman exec " .. c .. " pminfo -f openmetrics.lsp45.lsp45_answer") +checks.expect_match(om.out, "openmetrics%.lsp45%.lsp45_answer", + "pminfo -f resolves the scraped openmetrics metric name") +checks.expect_match(om.out, "value 42", + "the scraped value is the real constant this chapter's config ships (42)") +``` + +Two code paths — one baked into the `Containerfile` itself, one driven by +`verify.lua` against a container it started fresh — independently query +the same PMDA and land on the identical value, exactly once each. That's a +stronger claim than either check alone: the build-time gate proves the +PMDA works at image-build time, before the image is ever shipped; +`verify.lua`'s check proves it still works after the image is built, +shipped, and started cold. Neither check trusts the other's result. + +## What you learned + +- **Three suites, three philosophies for the same question.** Cockpit + answers "what is this system doing" with a browser and zero scripting; + SystemTap answers it by compiling a real kernel module per probe — bcc's + harder-to-build sibling from Chapter 30; PCP answers it by giving a + uniform namespace to metrics pulled out of things this book already + built — a `podman.sock` (Chapter 34) and an OpenMetrics document + (Chapter 38). +- **Fedora, not UBI, because the packages simply aren't there.** `pcp` and + its PMDAs are unresolvable on UBI 9/10 without a RHEL entitlement this + book's readers are assumed not to have; + `registry.fedoraproject.org/fedora:44` carries `pcp 7.1.5` unauthenticated. +- **A disconnected build stage is a build stage that never runs.** + `FROM builder AS runtime` compiles cleanly and silently skips `smoke` + entirely; `FROM smoke AS runtime` is what forces `podman build` to + actually execute the gate. +- **A PMDA's own `Install` script can silently revert itself.** With no + systemd in a build layer, `systemctl restart pmcd.service` fails and + `Install`'s failure path un-does its own `pmcd.conf` edit — starting + `pmcd` before `./Install` takes a cheaper, systemd-free `pmsignal -a -s + HUP pmcd` path instead, and nothing gets reverted. +- **Two independent gates on the same value beat one.** The `smoke` + stage's build-time `grep -q "value 42"` and `verify.lua`'s runtime + `pminfo` call check the same PMDA through two unrelated code paths and + agree — proof the value is genuinely parsed, not hardcoded, at both + build time and run time. +- **`pmda-bcc` is excluded on purpose, not by oversight.** Real eBPF + attachment needs a super-privileged container — the opposite of every + rootless posture in this book — so it's named here as PCP's privileged + sibling PMDA and left to Chapter 30's real eBPF toolkit on + `systems-target`. + +Part 13's appendices continue from here — tooling this book leaned on +without ever giving it a chapter of its own. + +--- + +

verified — every transcript +above except the span marked unverified (Cockpit's Overview/Podman/Logs +web-UI panels, described but not browser-automated) was produced this +session. On the Fedora 44 host (kernel 7.1.4-204.fc44, podman +5.8.4 rootless): ./demo.sh cpp run produced the exact +transcript quoted above, including pmcd: Version 7.1.5-3, 11 +agents, openmetrics.lsp45.lsp45_answer value 42, and +live podman-socket container visibility (podman.container.name +... "lsp45-pcp-demo", podman.container.running ... value +1); LSP_LANG=cpp lua verify.lua reported PASS 10 / +FAIL 0; and the built image measured 265 MB +(podman images lsp45-pcp). On the systems-target +lab VM (kernel 6.19.10-300.fc44.x86_64): the SystemTap +kernel-module backend produced stap-module-ok from a real +compiled-loaded-unloaded kernel module, and the openat aggregate against a +controlled five-cat workload produced cat: 155 +openat and sh: 43 openat exactly as quoted (this +required installing kernel-devel-$(uname -r) to match the +running kernel — now folded into this project's cloud-init); and +curl -sk https://192.168.124.7:9090/ against +cockpit.socket (active, cockpit-364-1.fc44 + +cockpit-podman installed by cloud-init) returned a real +HTTP 200. Not exercised: the Cockpit web console's own +rendered panels, as marked above — this repo has no browser-automation +tooling to drive one.

diff --git a/_plans/r14-ch45-plan.md b/_plans/r14-ch45-plan.md new file mode 100644 index 0000000..14575ba --- /dev/null +++ b/_plans/r14-ch45-plan.md @@ -0,0 +1,75 @@ +--- +title: "r14 / ch45 — plan (internal)" +published: false +--- + +# r14 ch45 — `_docs/45-linux-analysis-suites.md` + 2 diagrams (lgtm-relay Phase 1) + +Model relay: Opus plan (this file) → Sonnet execute → Opus validate. +Example already built + verified: `examples/45-linux-analysis-suites/` (podman PASS 10/0, 265 MB). +Evidence: `scratchpad/EVIDENCE45-podman.md`, `scratchpad/EVIDENCE45-vm.md`, `scratchpad/spec45/*`. + +## Approach +Appendices-part reference chapter. Throughline: prior tools watched ONE process +(strace/gdb/bcc, ch30); ch45 pivots to three fleet-wide *suites* with three +philosophies — Cockpit (zero-scripting web console), SystemTap (bcc's +kernel-module-compiling sibling), PCP (pluggable metric collector pulling from +podman/ch34 + OpenMetrics/ch38). Runnable artifact = PCP container ONLY; Cockpit ++ SystemTap demonstrated on systems-target. Adapts (never silently drops) the +per-language spine, as ch44 did for single-language Go. Footer carries BOTH a +`status--verified` span (podman run + VM SystemTap + Cockpit socket) AND a +narrowly-scoped `status--unverified` span (Cockpit web-UI panels: described, not +browser-automated). No `status--verified-with-caveats` class exists — two-span +mix is the established equivalent. + +Rejected: codetabs include (single-lang, follow ch44 plain fences); rewriting the +superseded design-doc §1/§2 (UBI/pmda-bcc — describe the BUILT artifact); splitting +prose into parallel writers (one file → collision). + +## Steps (Phase 2 fan-out) +- **S1 — Diagram A** `assets/diagrams/45-analysis-suite-landscape.{svg,excalidraw}` (parallel). Three lanes: Cockpit (browser→cockpit-ws→systemd/podman; socket VERIFIED, panels described); SystemTap (.stp→stap→C→kernel module, amber = VERIFIED stap-module-ok, vs ch30 dyninst; needs kernel-devel); PCP (pmcd center, two amber arrows in — pmda-openmetrics from `file:///etc/lsp45/lsp45.prom`, pmda-podman from `podman.sock (ch34)`; dashed ghost `pmda-bcc — NOT shipped, super-privileged, ch30 on VM`). Caption Figure 45.1. +- **S2 — Diagram B** `assets/diagrams/45-pcp-build-gate.{svg,excalidraw}` (parallel). builder→smoke→runtime; highlight `FROM smoke` edge (forces gate to build); dashed side box Cockpit+SystemTap on systems-target. Caption Figure 45.2. +- **S3 — Chapter prose** `_docs/45-linux-analysis-suites.md` (parallel; sole editor of this file). Front matter part="Appendices: Tooling" (VERIFIED exact), order 45, quoted description. Full spine. "Why Fedora not UBI" paragraph. Both real bugs (FROM builder stage-skip; ./Install systemd-revert fixed by pmcd-first) in How-it-works/Errors. Verbatim excerpts from Containerfile/config/demo.sh/verify.lua. Tools-used box. All run-numbers trace to EVIDENCE files. +- **S4 — Catalogue** `assets/diagrams/README.md` — 2 new rows (after S1+S2; sole editor of README). +- **S5 — Manifest** — CONFIRM ONLY, no edit (id 45 already correct). + +Parallel: S1, S2, S3 independent (distinct files). S4 after S1+S2. + +## Acceptance criteria +1. part == "Appendices: Tooling" (exact); order 45; description quoted. +2. YAML parses; full spine in order. +3. Tools-used box right after intro/figure; every named tool exercised. +4. Every example fenced block = byte-for-byte substring of its named file. +5. Two excalidraw embeds file="45-analysis-suite-landscape" / "45-pcp-build-gate", alt + Figure 45.1/45.2; both pairs exist. +6. README.md +2 chapter-45 rows. +7. pmda-bcc stated excluded (super-privileged), tied to ch30; not shown shipped in either diagram. +8. Both real bugs present (FROM builder stage-skip + fix; ./Install revert + pmcd-first fix). +9. "Why Fedora not UBI" paragraph present. +10. Footer: status--verified span (podman+VM SystemTap+Cockpit socket) AND status--unverified span naming Cockpit web-UI panels as described. +11. Every "from a run" number matches an EVIDENCE file exactly (265 MB, PASS 10/0, value 42, 7.1.5-3/11 agents, 0.970/0.540/0.420, "lsp45-pcp-demo", HTTP 200, stap-module-ok, 155/43 openat). Nothing else. +12. Zero honest/honestly/to-be-honest; each lie/lying (if any) machine-subject only. ("Honest:" from evidence must not leak.) +13. manifest.yaml unchanged, id 45 correct. +14. No literal {{ }}/{% %} in prose outside includes/relative_url/raw; SVGs well-formed; excalidraw valid JSON. + +## Risks +Non-verbatim excerpts (crit 4 grep); fabricated plausible number (crit 11 grep); over-claiming +Cockpit UI verified (crit 10 + footer read); diagram shows pmda-bcc shipped (crit 7); embed/file/catalogue +name mismatch (crit 5-6); invented CSS class (crit 10 two-span mix); "Honest:" leak (crit 12); diagram +edge-into-space / hidden label (band for container boxes, edges terminate on box edge). + +## Status +- [x] S1 diagram A — `45-analysis-suite-landscape.{svg,excalidraw}` +- [x] S2 diagram B — `45-pcp-build-gate.{svg,excalidraw}` +- [x] S3 prose — `_docs/45-linux-analysis-suites.md` (503 lines) +- [x] S4 catalogue — README +2 rows +- [x] S5 manifest confirm — id 45 already correct, no edit +- [x] Phase 3 validate (Opus) — all 14 criteria MET, verdict **SHIP**, zero MUST-FIX + +## Gate (authoritative host run, 2026-07-30) +- `python3 scripts/validate.py` → `validate: OK` +- `python3 scripts/test-all-examples.py --only 45-linux-analysis-suites` → `1 passed, 0 failed, 0 skipped` (cpp PASS) +- Example already CI-verified by build lane: podman image 265 MB, verify.lua PASS 10/0. +- Verification is MIXED (recorded in footer): podman run + VM SystemTap kernel-module + Cockpit socket VERIFIED; Cockpit web-UI panels DESCRIBED, not browser-automated. + +## Merge +Per user (r14): commit + PR, PAUSE for user merge approval (not autonomous). diff --git a/assets/diagrams/45-analysis-suite-landscape.excalidraw b/assets/diagrams/45-analysis-suite-landscape.excalidraw new file mode 100644 index 0000000..81667ca --- /dev/null +++ b/assets/diagrams/45-analysis-suite-landscape.excalidraw @@ -0,0 +1,1931 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1737933201", + "type": "rectangle", + "x": 20, + "y": 40, + "width": 340, + "height": 590, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1173037184, + "versionNonce": 1484649435, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1249616651", + "type": "text", + "x": 32, + "y": 46, + "width": 171.60000000000002, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 902716492, + "versionNonce": 767705516, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Cockpit \u2014 web console (VM)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Cockpit \u2014 web console (VM)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r508512125", + "type": "rectangle", + "x": 380, + "y": 40, + "width": 340, + "height": 590, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 435962127, + "versionNonce": 315652126, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1910298352", + "type": "text", + "x": 392, + "y": 46, + "width": 250.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1510842274, + "versionNonce": 355237437, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "SystemTap \u2014 kernel-module tracing (VM)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "SystemTap \u2014 kernel-module tracing (VM)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1434198794", + "type": "rectangle", + "x": 740, + "y": 40, + "width": 340, + "height": 590, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1495496164, + "versionNonce": 92913953, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t558283078", + "type": "text", + "x": 752, + "y": 46, + "width": 237.60000000000002, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 499534149, + "versionNonce": 1402769817, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "PCP \u2014 pmcd + PMDAs (container, ch45)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "PCP \u2014 pmcd + PMDAs (container, ch45)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r169298957", + "type": "rectangle", + "x": 60, + "y": 90, + "width": 240, + "height": 50, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1090404912, + "versionNonce": 1713642636, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t130823849", + "type": "text", + "x": 70, + "y": 98, + "width": 53.900000000000006, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 183792551, + "versionNonce": 948641223, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "browser", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "browser", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1084856018", + "type": "text", + "x": 70, + "y": 118, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 765946185, + "versionNonce": 135231232, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "https://:9090/", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "https://:9090/", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r733324009", + "type": "rectangle", + "x": 60, + "y": 190, + "width": 240, + "height": 60, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 687363449, + "versionNonce": 231877672, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1267040735", + "type": "text", + "x": 70, + "y": 198, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1737362307, + "versionNonce": 1626991387, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cockpit-ws", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cockpit-ws", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1285591993", + "type": "text", + "x": 70, + "y": 218, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 760568840, + "versionNonce": 1881487573, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one process \u2014 WebSocket + D-Bus", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one process \u2014 WebSocket + D-Bus", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1717678972", + "type": "rectangle", + "x": 60, + "y": 320, + "width": 110, + "height": 55, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 744711094, + "versionNonce": 321919364, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1316665466", + "type": "text", + "x": 70, + "y": 328, + "width": 53.900000000000006, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1309007523, + "versionNonce": 1748575702, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "systemd", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "systemd", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t960699974", + "type": "text", + "x": 70, + "y": 348, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 48277580, + "versionNonce": 1641745235, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "guest services", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "guest services", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1379309613", + "type": "rectangle", + "x": 190, + "y": 320, + "width": 110, + "height": 55, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 538589332, + "versionNonce": 1060329423, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1548923193", + "type": "text", + "x": 200, + "y": 328, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1355849895, + "versionNonce": 1149773308, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "podman", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "podman", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1796798829", + "type": "text", + "x": 200, + "y": 348, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1043184402, + "versionNonce": 980113226, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "guest containers", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "guest containers", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1782288223", + "type": "rectangle", + "x": 410, + "y": 90, + "width": 260, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1613029770, + "versionNonce": 1655357245, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t492924836", + "type": "text", + "x": 420, + "y": 98, + "width": 84.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1109779392, + "versionNonce": 1444764231, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": ".stp script", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": ".stp script", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t107499770", + "type": "text", + "x": 420, + "y": 118, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 73599084, + "versionNonce": 799203664, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "probe + printf source", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "probe + printf source", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1778626235", + "type": "rectangle", + "x": 410, + "y": 180, + "width": 260, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 694840083, + "versionNonce": 1079817534, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1076639596", + "type": "text", + "x": 420, + "y": 188, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1780761028, + "versionNonce": 1459544334, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "stap translator", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "stap translator", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1835023360", + "type": "text", + "x": 420, + "y": 208, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 7044837, + "versionNonce": 214108737, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "elaborates + type-checks", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "elaborates + type-checks", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1845764643", + "type": "rectangle", + "x": 410, + "y": 270, + "width": 260, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1371329890, + "versionNonce": 173179559, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1780810705", + "type": "text", + "x": 420, + "y": 278, + "width": 84.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 934239725, + "versionNonce": 267209597, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "generated C", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "generated C", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t670187784", + "type": "text", + "x": 420, + "y": 298, + "width": 175.45000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1704723614, + "versionNonce": 1369614522, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kernel-module source, emitted", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kernel-module source, emitted", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1751469598", + "type": "rectangle", + "x": 410, + "y": 360, + "width": 260, + "height": 70, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1589594838, + "versionNonce": 906996594, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1042993370", + "type": "text", + "x": 420, + "y": 368, + "width": 169.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 389154187, + "versionNonce": 421042498, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kernel module (insmod)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kernel module (insmod)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t829909484", + "type": "text", + "x": 420, + "y": 388, + "width": 151.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 733467070, + "versionNonce": 3508539, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "stap-module-ok (verified)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "stap-module-ok (verified)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1715019806", + "type": "text", + "x": 420, + "y": 403, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1532600196, + "versionNonce": 1407486387, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "needs matching kernel-devel", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "needs matching kernel-devel", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1224777243", + "type": "rectangle", + "x": 780, + "y": 90, + "width": 260, + "height": 60, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 714307605, + "versionNonce": 1197602205, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t664050983", + "type": "text", + "x": 790, + "y": 98, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 907921928, + "versionNonce": 1380049297, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pmda-openmetrics", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pmda-openmetrics", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1711738385", + "type": "text", + "x": 790, + "y": 118, + "width": 169.4, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1791409323, + "versionNonce": 585478451, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "file:///etc/lsp45/lsp45.prom", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "file:///etc/lsp45/lsp45.prom", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1158269064", + "type": "rectangle", + "x": 820, + "y": 280, + "width": 180, + "height": 70, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1151346347, + "versionNonce": 264298912, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1266348811", + "type": "text", + "x": 830, + "y": 288, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1694854693, + "versionNonce": 322382594, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pmcd", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pmcd", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1734562026", + "type": "text", + "x": 830, + "y": 308, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 688418651, + "versionNonce": 576187798, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "central metrics daemon", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "central metrics daemon", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r206964476", + "type": "rectangle", + "x": 780, + "y": 460, + "width": 260, + "height": 60, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 641426804, + "versionNonce": 1593370431, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t233068904", + "type": "text", + "x": 790, + "y": 468, + "width": 84.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 412296382, + "versionNonce": 1235690004, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pmda-podman", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pmda-podman", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t938856183", + "type": "text", + "x": 790, + "y": 488, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 9874954, + "versionNonce": 823827982, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "podman.sock (ch34)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "podman.sock (ch34)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r495300925", + "type": "rectangle", + "x": 780, + "y": 560, + "width": 260, + "height": 60, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 542374050, + "versionNonce": 674810971, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1565485315", + "type": "text", + "x": 790, + "y": 568, + "width": 169.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1113605452, + "versionNonce": 423535182, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pmda-bcc \u2014 NOT shipped", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pmda-bcc \u2014 NOT shipped", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1239341294", + "type": "text", + "x": 790, + "y": 588, + "width": 193.60000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1425366908, + "versionNonce": 732802047, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "super-privileged; ch30 on the VM", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "super-privileged; ch30 on the VM", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1125397110", + "type": "arrow", + "x": 180, + "y": 140, + "width": 0, + "height": 50, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 262088310, + "versionNonce": 213135188, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 50 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t797536492", + "type": "text", + "x": 180.0, + "y": 151.0, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1666834001, + "versionNonce": 1428550483, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "HTTP 200 (verified)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "HTTP 200 (verified)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1499552575", + "type": "arrow", + "x": 115, + "y": 250, + "width": 0, + "height": 70, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1488682137, + "versionNonce": 455422479, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 70 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1747341893", + "type": "arrow", + "x": 245, + "y": 250, + "width": 0, + "height": 70, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1917970641, + "versionNonce": 1024060912, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 70 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1397616574", + "type": "arrow", + "x": 540, + "y": 140, + "width": 0, + "height": 40, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 827908770, + "versionNonce": 920478891, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 40 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1508555253", + "type": "arrow", + "x": 540, + "y": 230, + "width": 0, + "height": 40, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 439214128, + "versionNonce": 1875712580, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 40 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1176080662", + "type": "arrow", + "x": 540, + "y": 320, + "width": 0, + "height": 40, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 309418218, + "versionNonce": 1808279096, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 40 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a281840075", + "type": "arrow", + "x": 910, + "y": 150, + "width": 0, + "height": 130, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 799964158, + "versionNonce": 1219257489, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 130 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1641899347", + "type": "arrow", + "x": 910, + "y": 460, + "width": 0, + "height": 110, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1073945389, + "versionNonce": 1856738754, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + -110 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1861797222", + "type": "text", + "x": 60, + "y": 400, + "width": 332.75, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 899565675, + "versionNonce": 967332350, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Web panels (Overview / Podman / Logs): described only \u2014", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Web panels (Overview / Podman / Logs): described only \u2014", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1204590794", + "type": "text", + "x": 60, + "y": 416, + "width": 229.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 67045469, + "versionNonce": 760169814, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "no browser automation in this session.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "no browser automation in this session.", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t74483959", + "type": "text", + "x": 410, + "y": 450, + "width": 387.20000000000005, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1235024257, + "versionNonce": 1014578227, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Contrast: ch30's dyninst backend was inconclusive on this host \u2014", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Contrast: ch30's dyninst backend was inconclusive on this host \u2014", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t332686835", + "type": "text", + "x": 410, + "y": 466, + "width": 399.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1072286584, + "versionNonce": 1363279354, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "the kernel-module path needed matching kernel-devel; now verified.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "the kernel-module path needed matching kernel-devel; now verified.", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/45-analysis-suite-landscape.svg b/assets/diagrams/45-analysis-suite-landscape.svg new file mode 100644 index 0000000..6a63976 --- /dev/null +++ b/assets/diagrams/45-analysis-suite-landscape.svg @@ -0,0 +1,60 @@ + + + +Cockpit — web console (VM) + +SystemTap — kernel-module tracing (VM) + +PCP — pmcd + PMDAs (container, ch45) + +browser +https://<vm>:9090/ + +cockpit-ws +one process — WebSocket + D-Bus + +systemd +guest services + +podman +guest containers + +.stp script +probe + printf source + +stap translator +elaborates + type-checks + +generated C +kernel-module source, emitted + +kernel module (insmod) +stap-module-ok (verified) +needs matching kernel-devel + +pmda-openmetrics +file:///etc/lsp45/lsp45.prom + +pmcd +central metrics daemon + +pmda-podman +podman.sock (ch34) + +pmda-bcc — NOT shipped +super-privileged; ch30 on the VM + +HTTP 200 (verified) +HTTP 200 (verified) + + + + + + + +Web panels (Overview / Podman / Logs): described only — +no browser automation in this session. +Contrast: ch30's dyninst backend was inconclusive on this host — +the kernel-module path needed matching kernel-devel; now verified. + \ No newline at end of file diff --git a/assets/diagrams/45-pcp-build-gate.excalidraw b/assets/diagrams/45-pcp-build-gate.excalidraw new file mode 100644 index 0000000..e9634a5 --- /dev/null +++ b/assets/diagrams/45-pcp-build-gate.excalidraw @@ -0,0 +1,1841 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1363641142", + "type": "rectangle", + "x": 20, + "y": 40, + "width": 380, + "height": 380, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 287500727, + "versionNonce": 1652277967, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t860115311", + "type": "text", + "x": 32, + "y": 46, + "width": 257.40000000000003, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 622252775, + "versionNonce": 270148788, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "builder \u2014 dnf install + PMDA activation", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "builder \u2014 dnf install + PMDA activation", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1551074597", + "type": "rectangle", + "x": 440, + "y": 40, + "width": 380, + "height": 380, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 875680855, + "versionNonce": 1966657630, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t268096232", + "type": "text", + "x": 452, + "y": 46, + "width": 330.0, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1198083459, + "versionNonce": 295357750, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "smoke \u2014 build-time correctness gate (FROM builder)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "smoke \u2014 build-time correctness gate (FROM builder)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1006540977", + "type": "rectangle", + "x": 860, + "y": 40, + "width": 380, + "height": 380, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1244815265, + "versionNonce": 1566353090, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1054458909", + "type": "text", + "x": 872, + "y": 46, + "width": 237.60000000000002, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1363810830, + "versionNonce": 1178811694, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "runtime \u2014 shipped image (FROM smoke)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "runtime \u2014 shipped image (FROM smoke)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r760787181", + "type": "rectangle", + "x": 40, + "y": 80, + "width": 340, + "height": 60, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1267976278, + "versionNonce": 88374562, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1613712232", + "type": "text", + "x": 50, + "y": 88, + "width": 84.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1839348356, + "versionNonce": 515766679, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "dnf install", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "dnf install", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1724050868", + "type": "text", + "x": 50, + "y": 108, + "width": 266.20000000000005, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1946930119, + "versionNonce": 861399315, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pcp + pcp-pmda-openmetrics + pcp-pmda-podman", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pcp + pcp-pmda-openmetrics + pcp-pmda-podman", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r18863533", + "type": "rectangle", + "x": 40, + "y": 170, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1942711834, + "versionNonce": 672946477, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t293130606", + "type": "text", + "x": 50, + "y": 178, + "width": 200.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1051641882, + "versionNonce": 595051873, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "start pmcd -f (background)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "start pmcd -f (background)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1125362655", + "type": "text", + "x": 50, + "y": 198, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1187368537, + "versionNonce": 1132425593, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "wait for pminfo -f pmcd.version", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "wait for pminfo -f pmcd.version", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1903155855", + "type": "rectangle", + "x": 40, + "y": 250, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 654941317, + "versionNonce": 1111414500, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t292234159", + "type": "text", + "x": 50, + "y": 258, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 787587384, + "versionNonce": 1308848998, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "yes '' | ./Install", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "yes '' | ./Install", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1532333886", + "type": "text", + "x": 50, + "y": 278, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1584783408, + "versionNonce": 185406603, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "openmetrics PMDA", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "openmetrics PMDA", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r195863800", + "type": "rectangle", + "x": 40, + "y": 330, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1230279302, + "versionNonce": 1216692179, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1063323940", + "type": "text", + "x": 50, + "y": 338, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1992366223, + "versionNonce": 411746260, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "yes '' | ./Install", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "yes '' | ./Install", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t927615056", + "type": "text", + "x": 50, + "y": 358, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 222668729, + "versionNonce": 1138874961, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "podman PMDA", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "podman PMDA", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1242887107", + "type": "rectangle", + "x": 460, + "y": 80, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 214816032, + "versionNonce": 272205212, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t277207183", + "type": "text", + "x": 470, + "y": 88, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1043814965, + "versionNonce": 801082188, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "start pmcd -f", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "start pmcd -f", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1576979518", + "type": "text", + "x": 470, + "y": 108, + "width": 217.8, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 142255686, + "versionNonce": 563457428, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "fresh pmcd.conf from the smoke layer", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "fresh pmcd.conf from the smoke layer", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1294614069", + "type": "rectangle", + "x": 460, + "y": 160, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 465219633, + "versionNonce": 663850809, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t675445860", + "type": "text", + "x": 470, + "y": 168, + "width": 192.50000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 773656421, + "versionNonce": 237885196, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pcp | tee pcp-summary.txt", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pcp | tee pcp-summary.txt", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1636116377", + "type": "text", + "x": 470, + "y": 188, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 513139125, + "versionNonce": 1090069803, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "grep openmetrics && grep podman", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "grep openmetrics && grep podman", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r977457619", + "type": "rectangle", + "x": 460, + "y": 240, + "width": 340, + "height": 60, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 770190293, + "versionNonce": 1976048363, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1529901959", + "type": "text", + "x": 470, + "y": 248, + "width": 223.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 406384395, + "versionNonce": 718376973, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pminfo -f: openmetrics answer", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pminfo -f: openmetrics answer", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t4074793", + "type": "text", + "x": 470, + "y": 268, + "width": 278.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1160810632, + "versionNonce": 1581707950, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "openmetrics.lsp45.lsp45_answer = 42 (verified)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "openmetrics.lsp45.lsp45_answer = 42 (verified)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r375794633", + "type": "rectangle", + "x": 460, + "y": 330, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1014043974, + "versionNonce": 1142805848, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t292342035", + "type": "text", + "x": 470, + "y": 338, + "width": 192.50000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1074866377, + "versionNonce": 28890804, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pminfo -f kernel.all.load", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pminfo -f kernel.all.load", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t880801470", + "type": "text", + "x": 470, + "y": 358, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1414099454, + "versionNonce": 1079158467, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "core, always-present metric", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "core, always-present metric", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r677206760", + "type": "rectangle", + "x": 880, + "y": 140, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 185899111, + "versionNonce": 908761313, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t333069781", + "type": "text", + "x": 890, + "y": 148, + "width": 92.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 125697054, + "versionNonce": 63357246, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "EXPOSE 44321", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "EXPOSE 44321", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r983676817", + "type": "rectangle", + "x": 880, + "y": 220, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 280881186, + "versionNonce": 784438020, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1657370112", + "type": "text", + "x": 890, + "y": 228, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 486460477, + "versionNonce": 1490348816, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ENTRYPOINT", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "ENTRYPOINT", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1075657457", + "type": "text", + "x": 890, + "y": 248, + "width": 169.4, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1753325584, + "versionNonce": 1972266157, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "/usr/libexec/pcp/bin/pmcd -f", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "/usr/libexec/pcp/bin/pmcd -f", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r553451010", + "type": "rectangle", + "x": 340, + "y": 500, + "width": 600, + "height": 70, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 939141147, + "versionNonce": 194639506, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t704568992", + "type": "text", + "x": 350, + "y": 508, + "width": 146.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 412015389, + "versionNonce": 280827375, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Cockpit + SystemTap", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Cockpit + SystemTap", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1482030280", + "type": "text", + "x": 350, + "y": 528, + "width": 314.6, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 749429288, + "versionNonce": 1309381823, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "not in this image \u2014 on systems-target via cloud-init", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "not in this image \u2014 on systems-target via cloud-init", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1144433661", + "type": "arrow", + "x": 210, + "y": 140, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 489883235, + "versionNonce": 41074667, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a17583461", + "type": "arrow", + "x": 210, + "y": 220, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 338941047, + "versionNonce": 31071696, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1678273018", + "type": "arrow", + "x": 210, + "y": 300, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 899395384, + "versionNonce": 912307686, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a904312006", + "type": "arrow", + "x": 630, + "y": 130, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 304822111, + "versionNonce": 1517559454, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1421652768", + "type": "arrow", + "x": 630, + "y": 210, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 516798013, + "versionNonce": 1433050430, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a544306544", + "type": "arrow", + "x": 630, + "y": 300, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1571365518, + "versionNonce": 21047388, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1764499680", + "type": "arrow", + "x": 1050, + "y": 190, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 936078868, + "versionNonce": 1248788692, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a75876575", + "type": "arrow", + "x": 380, + "y": 355, + "width": 80, + "height": 250, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1773822221, + "versionNonce": 1424488272, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 80, + -250 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1765931945", + "type": "text", + "x": 420.0, + "y": 216.0, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1251920703, + "versionNonce": 389213756, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "FROM builder", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "FROM builder", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1754303825", + "type": "arrow", + "x": 800, + "y": 355, + "width": 80, + "height": 190, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 961951762, + "versionNonce": 1073519657, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 80, + -190 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t250990589", + "type": "text", + "x": 840.0, + "y": 246.0, + "width": 60.50000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 528255462, + "versionNonce": 1023945934, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "FROM smoke", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "FROM smoke", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t244862588", + "type": "text", + "x": 60, + "y": 440, + "width": 501.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1239099463, + "versionNonce": 1448294346, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Chaining runtime onto smoke forces podman to actually build the gate stage \u2014", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Chaining runtime onto smoke forces podman to actually build the gate stage \u2014", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "t1479746422", + "type": "text", + "x": 60, + "y": 458, + "width": 600.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1907256539, + "versionNonce": 609434034, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "an unreferenced stage (e.g. \u201cFROM builder AS runtime\u201d) is silently skipped by podman build.", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "an unreferenced stage (e.g. \u201cFROM builder AS runtime\u201d) is silently skipped by podman build.", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "t788099133", + "type": "text", + "x": 880, + "y": 295, + "width": 193.60000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1662633130, + "versionNonce": 536165510, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Reuses smoke's already-activated", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Reuses smoke's already-activated", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1239159971", + "type": "text", + "x": 880, + "y": 310, + "width": 229.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 693636605, + "versionNonce": 1350855911, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "PMNS/pmcd.conf \u2014 no third Install run.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "PMNS/pmcd.conf \u2014 no third Install run.", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/45-pcp-build-gate.svg b/assets/diagrams/45-pcp-build-gate.svg new file mode 100644 index 0000000..80ca9e6 --- /dev/null +++ b/assets/diagrams/45-pcp-build-gate.svg @@ -0,0 +1,58 @@ + + + +builder — dnf install + PMDA activation + +smoke — build-time correctness gate (FROM builder) + +runtime — shipped image (FROM smoke) + +dnf install +pcp + pcp-pmda-openmetrics + pcp-pmda-podman + +start pmcd -f (background) +wait for pminfo -f pmcd.version + +yes '' | ./Install +openmetrics PMDA + +yes '' | ./Install +podman PMDA + +start pmcd -f +fresh pmcd.conf from the smoke layer + +pcp | tee pcp-summary.txt +grep openmetrics && grep podman + +pminfo -f: openmetrics answer +openmetrics.lsp45.lsp45_answer = 42 (verified) + +pminfo -f kernel.all.load +core, always-present metric + +EXPOSE 44321 + +ENTRYPOINT +/usr/libexec/pcp/bin/pmcd -f + +Cockpit + SystemTap +not in this image — on systems-target via cloud-init + + + + + + + + +FROM builder +FROM builder + +FROM smoke +FROM smoke +Chaining runtime onto smoke forces podman to actually build the gate stage — +an unreferenced stage (e.g. “FROM builder AS runtime”) is silently skipped by podman build. +Reuses smoke's already-activated +PMNS/pmcd.conf — no third Install run. + \ No newline at end of file diff --git a/assets/diagrams/README.md b/assets/diagrams/README.md index 7c750be..612a771 100644 --- a/assets/diagrams/README.md +++ b/assets/diagrams/README.md @@ -87,3 +87,5 @@ here as you add them: | `43-miri-bounds-check` | 43 | the 8-byte `Record` buffer (`magic: u32` + `slots: u32`) with the SOUND `read_unaligned` window `[0..8)` fully inside the allocation beside the `--oob-read` window `[4..12)` running 4 bytes past the end -- the exact contrast `cargo +nightly miri run` catches (exit 0, no UB) versus reports (`only 4 bytes from the end of the allocation`, main.rs:57) | | `44-gmp-scheduler` | 44 | the GMP model -- G's queued in per-P 256-slot local run queues (+ the runnext fast path) and the global run queue, GOMAXPROCS Ps each bound to an M running a G, work-stealing (steal half) between Ps, and sysmon (dedicated M, no P) driving retake/forcegc/netpoll-backstop/GOMAXPROCS-recheck plus the SIGURG async-preemption signal into a running M -- many G onto few M via P | | `44-goroutines-not-threads` | 44 | the netpoller contrast -- 2000 goroutines blocked on pipe reads parked in one shared epoll (netpoller, `epoll_pwait`) holding no M, OS thread count flat at the measured 12 (bound 80), beside a raw blocking `syscall.Read` where each blocked G pins its own M and the thread count climbs toward N via `handoffp`/`startm`/`newm` | +| `45-analysis-suite-landscape` | 45 | three fleet-wide observation suites side by side -- Cockpit (browser → cockpit-ws → systemd/podman, socket verified HTTP 200, web panels described only), SystemTap (`.stp` → stap → generated C → kernel module, the verified `stap-module-ok` path amber, needs matching kernel-devel, contrasted with ch30's dyninst), and PCP (`pmcd` fed by `pmda-openmetrics` from `file:///etc/lsp45/lsp45.prom` and `pmda-podman` from `podman.sock`, with `pmda-bcc` a dashed not-shipped ghost) | +| `45-pcp-build-gate` | 45 | the PCP image's three Containerfile stages -- builder (dnf install pcp + two PMDAs, `pmcd` started before `./Install`) → smoke (`FROM builder`, the `pcp`/`pminfo` gate proving openmetrics answers `value 42`) → runtime (`FROM smoke`, `EXPOSE 44321`, `ENTRYPOINT pmcd -f`), the amber `FROM smoke` edge that forces podman to build the gate stage, with Cockpit + SystemTap a dashed not-in-this-image box | diff --git a/examples/45-linux-analysis-suites/Containerfile b/examples/45-linux-analysis-suites/Containerfile new file mode 100644 index 0000000..8b3cebd --- /dev/null +++ b/examples/45-linux-analysis-suites/Containerfile @@ -0,0 +1,79 @@ +# PCP (Performance Co-Pilot) container -- Fedora base, multi-stage, rootless +# podman. No app code: this image ships pmcd + two layered PMDAs. +# +# podman build -t lsp45-pcp -f Containerfile . +# podman run -d --name lsp45-pcp-demo --cpus=2 --memory=256m \ +# -v "${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/podman/podman.sock:/run/podman/podman.sock" \ +# lsp45-pcp +# +# Why Fedora, not UBI: verified against a real `dnf install` that `pcp` and +# its PMDAs are NOT resolvable on UBI 9/10 without a RHEL entitlement (no +# free/anonymous repo carries them). Fedora's own repos carry pcp 7.1.5 and +# every PMDA package unauthenticated -- registry.fedoraproject.org/fedora +# needs no subscription, which is what this book's readers actually have. +# +# `pcp-pmda-bcc` is deliberately NOT installed here: activating it needs a +# working BCC/eBPF stack, which needs a super-privileged container (BPF +# syscalls, /sys/kernel/debug, a matching kernel-devel) that cannot load +# rootless. It's covered in the chapter as PCP's privileged sibling PMDA, +# demonstrated on the systems-target VM instead of shipped in this image. +FROM registry.fedoraproject.org/fedora:44 AS builder +RUN dnf -y install pcp pcp-system-tools pcp-pmda-openmetrics pcp-pmda-podman && \ + dnf clean all + +# A tiny, real, self-contained OpenMetrics document -- no network access +# needed to scrape it. pmdaopenmetrics.python supports a `file://` source +# scheme (checked directly in its source: `self.url.startswith('file://')` +# reads the path with a plain `open()`), so pointing config.d at a local +# file gives a REAL scrape with a REAL, known value, entirely offline. +COPY config/lsp45.prom /etc/lsp45/lsp45.prom +RUN rm -f /var/lib/pcp/pmdas/openmetrics/config.d/*.url && \ + echo "file:///etc/lsp45/lsp45.prom" \ + > /var/lib/pcp/pmdas/openmetrics/config.d/lsp45.url && \ + mkdir -p /run/pcp + +# Activate both PMDAs against a LIVE pmcd (started here, in this same RUN). +# This is the one thing that took real iteration to get right: each PMDA's +# `./Install` script (`pmdaproc.sh`) restarts pmcd via `systemctl restart +# pmcd.service` when it thinks a restart is needed -- and this build has no +# systemd, so that call fails and Install's failure path REVERTS its own +# pmcd.conf edit, silently dropping the PMDA. But `pmdaproc.sh` has a second +# path: if `pminfo -v pmcd.version` already succeeds against a running pmcd, +# it just sends pmcd a SIGHUP (`pmsignal -a -s HUP pmcd`) instead of trying +# to restart it -- no systemd involved. Starting pmcd first, before either +# Install call, takes that path for both PMDAs and nothing gets reverted. +RUN /usr/libexec/pcp/bin/pmcd -f >/tmp/pmcd-build.log 2>&1 & \ + for i in $(seq 1 20); do \ + pminfo -f pmcd.version >/dev/null 2>&1 && break; sleep 1; \ + done && \ + cd /var/lib/pcp/pmdas/openmetrics && yes '' | ./Install && \ + cd /var/lib/pcp/pmdas/podman && yes '' | ./Install + +# Build-time correctness gate (never shipped): start pmcd fresh from the +# config Install just wrote, confirm BOTH layered PMDAs actually registered +# and that a REAL metric value comes back from each -- the scraped +# openmetrics value AND a core, always-present PCP metric. If either PMDA +# silently failed to attach, this RUN fails and the `podman build` itself +# fails -- stronger than a runtime-only check. +FROM builder AS smoke +RUN /usr/libexec/pcp/bin/pmcd -f >/tmp/pmcd.log 2>&1 & \ + for i in $(seq 1 20); do \ + pminfo -f pmcd.version >/dev/null 2>&1 && break; sleep 1; \ + done; \ + pcp | tee /tmp/pcp-summary.txt; \ + grep -q openmetrics /tmp/pcp-summary.txt && \ + grep -q podman /tmp/pcp-summary.txt && \ + pminfo -f openmetrics.lsp45.lsp45_answer | tee /tmp/om.txt && \ + grep -q "value 42" /tmp/om.txt && \ + pminfo -f kernel.all.load | tee /tmp/load.txt && \ + grep -q "kernel.all.load" /tmp/load.txt + +# Shipped runtime. Reuses the smoke stage's already-activated PMNS/pmcd.conf +# rather than re-running Install a third time -- FROM smoke (not FROM +# builder) also means podman actually builds the smoke stage by default: +# with no COPY --from/FROM edge into it, an unreferenced intermediate stage +# is silently skipped by `podman build`, which would make the "build-time +# gate" a no-op. Chaining runtime FROM smoke is what makes the gate real. +FROM smoke AS runtime +EXPOSE 44321 +ENTRYPOINT ["/usr/libexec/pcp/bin/pmcd", "-f"] diff --git a/examples/45-linux-analysis-suites/README.md b/examples/45-linux-analysis-suites/README.md new file mode 100644 index 0000000..8f47c9d --- /dev/null +++ b/examples/45-linux-analysis-suites/README.md @@ -0,0 +1,146 @@ +# 45 — linux analysis suites (PCP) + +A single Fedora-based, multi-stage **Containerfile** that builds `pmcd` +(Performance Co-Pilot's collection daemon) with two PMDAs (Performance +Metrics Domain Agents) layered on top: `pmda-openmetrics` and `pmda-podman`. +Built and run with **rootless podman**. + +Unlike every other numbered example in this book, there is no C++/Go/Rust +source here. PCP is an OS-level metrics suite you point at things, not a +program you write in a language — `./demo.sh [cpp|go|rust] [build|run]` +accepts the language argument purely so `scripts/test-all-examples.py`'s +per-language loop can drive it; the image built and run is identical +regardless of which language is passed. + +## Why Fedora, not UBI + +This project's other container examples (`examples/34-...`) use UBI10. This +one doesn't: checked directly against a real `dnf install`, `pcp` and its +PMDAs are **not resolvable on UBI 9/10** without a RHEL entitlement — no +free/anonymous UBI repo carries them, and this book's readers won't have a +subscription. Fedora's own repos carry `pcp 7.1.5` and every PMDA package +unauthenticated, so `registry.fedoraproject.org/fedora:44` is the base that +actually builds for everyone. + +## What's inside, and what's deliberately not + +| PMDA | Status | Why | +|---|---|---| +| `pmda-openmetrics` | **In, with a real scraped value** | Bridges an OpenMetrics/Prometheus-style document into PCP's namespace. Pointed at a `file://` source (`config/lsp45.prom`, shipped in this example) rather than an HTTP endpoint, so the scrape is real and reproducible with zero network dependency. | +| `pmda-podman` | **In, registers always; live values need a socket mount** | Reads podman's container/pod state over its API socket. Registers its full metric namespace (21 metrics) at build time with no socket present. At `podman run` time, `demo.sh` mounts the **host's own rootless** `podman.sock` into the container at `/run/podman/podman.sock`, and the PMDA then reports REAL running containers — verified live, including the demo container seeing itself. (There's a known upstream report, [performancecopilot/pcp#913](https://github.com/performancecopilot/pcp/issues/913), that pmda-podman only sees root-managed containers; that was NOT reproduced here — mounting a rootless user's own socket at the expected path worked directly, at least on PCP 7.1.5/podman 5.8.4.) | +| `pmda-bcc` | **Excluded entirely** | Needs a working BCC/eBPF stack, which needs a super-privileged container (raw BPF syscalls, `/sys/kernel/debug`, a matching `kernel-devel`) that cannot load rootless — the opposite of everything else in this example. It's covered in the chapter as PCP's privileged sibling PMDA, demonstrated on the `systems-target` VM, not shipped in this image. This matches the project's "eBPF is tooling only" scope rule. | + +## Layout + +``` +45-linux-analysis-suites/ +├── Containerfile # multi-stage: builder -> smoke (build-time gate) -> runtime +├── config/ +│ └── lsp45.prom # a tiny, real OpenMetrics document (one known metric) +├── demo.sh # ./demo.sh [cpp|go|rust] [build|run] +├── verify.lua # podman-gated, behavioral checks +└── README.md +``` + +## Containerfile stage plan + +| Stage | Does | +|---|---| +| `builder` | `dnf install pcp pcp-system-tools pcp-pmda-openmetrics pcp-pmda-podman`; ships `config/lsp45.prom`; points `openmetrics`'s `config.d` at it via a `file://` source; starts `pmcd` **inside the build** and activates both PMDAs against that live `pmcd`. | +| `smoke` | Starts `pmcd` again from the config the builder stage just wrote, confirms **both** PMDAs registered in `pcp`'s summary, and confirms **real metric values** come back — the scraped `openmetrics.lsp45.lsp45_answer` (`42`) and the core `kernel.all.load`. If either PMDA silently failed to attach, this stage — and therefore the whole `podman build` — fails. | +| `runtime` | `FROM smoke` (not `FROM builder`) — chaining onto `smoke` is what makes podman actually build that stage; an unreferenced intermediate stage with no `COPY --from`/`FROM` edge into it is silently skipped by `podman build`, which would turn the "build-time gate" into a no-op. Ships the smoke stage's already-activated PMNS/`pmcd.conf`. `ENTRYPOINT ["/usr/libexec/pcp/bin/pmcd", "-f"]`. | + +## The one real build gotcha + +Each PMDA's `./Install` script (`pmdaproc.sh`) tries to restart `pmcd` via +`systemctl restart pmcd.service` whenever it thinks a restart — rather than +a lighter SIGHUP — is needed. This build has no systemd, so that call fails, +and Install's failure path **reverts its own `pmcd.conf` edit**, silently +dropping the PMDA it just tried to add. The fix, visible in the +Containerfile: start `pmcd` **before** either `./Install` call. `pmdaproc.sh` +has a cheaper path — if `pminfo -v pmcd.version` already succeeds against a +running `pmcd`, it just sends `pmsignal -a -s HUP pmcd` instead of trying a +full restart, and that path needs no systemd at all. Installing against an +already-running `pmcd` takes that path for both PMDAs and nothing gets +reverted. + +## Try it + +```bash +./demo.sh cpp build # podman build only +./demo.sh cpp run # build + run + a real pminfo/pmrep transcript +``` + +`run` mounts the host's own rootless `podman.sock` into the container (if +one is active for the invoking user) so the podman PMDA reports live +container state, not just an empty registered namespace. If no socket is +found, it says so and continues — the podman PMDA still registers, it just +reports zero values. + +```bash +# by hand, equivalent to demo.sh run +podman build -t lsp45-pcp -f Containerfile . +podman run -d --name lsp45-pcp-demo --cpus=2 --memory=256m \ + -v "${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/podman/podman.sock:/run/podman/podman.sock" \ + lsp45-pcp +podman exec lsp45-pcp-demo pcp +podman exec lsp45-pcp-demo pminfo -f openmetrics.lsp45.lsp45_answer +podman exec lsp45-pcp-demo pminfo -f podman.container.name podman.container.running +podman exec lsp45-pcp-demo pmrep -t 1 -s 1 kernel.all.load +podman rm -f lsp45-pcp-demo; podman rmi -f lsp45-pcp +``` + +## Verification (behavioral) + +`verify.lua` requires podman on the host (`skip()`, exit 77, if missing) and +runs only under `LSP_LANG=cpp` (the manifest lists `langs: [cpp]` for this +example, since there's no per-language behavior to triple-run). It actually +builds the image and drives a real container: + +- the build succeeds — meaningful because the Containerfile's own smoke + stage already gates this: a PMDA that fails to register fails the build, + not just the demo, confirmed by deliberately breaking it once during + development (starting `pmcd` *after* the `./Install` calls) and watching + the build fail with the exact `pmcd.conf` revert described above; +- `pmcd` answers `pminfo` within a bounded 20s poll; +- `pcp`'s own summary lists both `openmetrics` and `podman` as active PMDAs; +- `pminfo -f openmetrics.lsp45.lsp45_answer` returns `value 42` — a real + parse of the real file this example ships, every run; +- the podman PMDA's namespace registers (`podman.container.*` metrics + exist) — this script does not gate on live container *values*, since that + needs a mounted socket not guaranteed on every host/CI environment; the + live-value behavior is real and demonstrated by `demo.sh`, just not + asserted here; +- `pmrep -t 1 -s 1 kernel.all.load` returns a real sample of a core PCP + metric, independent of anything this chapter added; +- the container and the built image are removed at the end, whether the + checks passed or not. + +A real run on the reference host (Fedora 44, kernel 7.1.4-204.fc44, podman +5.8.4 rootless) passes all 10 checks: + +``` +PASS 10 / FAIL 0 +``` + +Image size on that host: **265 MB** (`podman images lsp45-pcp`). This is +much larger than `examples/34-our-programs-in-containers`'s few-tens-of-MB +runtime images — PCP's own dependency chain (Python, `pcp-libs`, +`pcp-system-tools`) is the reason; there's no further multi-stage trim +available, since the shipped `runtime` stage needs the exact same `pmcd`/ +PMDA/Python userspace the `smoke` stage just proved works. + +## A cosmetic quirk, noted rather than hidden + +`pminfo openmetrics` (no `-f`) additionally lists +`openmetrics.grafana.go_memstats_last_gc_since_start_time_seconds` even +though this image's `config.d` only ever contains `lsp45.url`. Fetching it +(`pminfo -f openmetrics.grafana...`) always returns "No value(s) available!" +— it's an inert, unexplained leftover PMNS entry, not a real metric source, +and it does not affect any assertion this example makes (which targets the +specific `openmetrics.lsp45.lsp45_answer` name by exact match). Tracing it +further — it's not in the PMDA's on-disk instance-cache pickle +(`/var/lib/pcp/config/pmda/144.*.py`), which only ever contains `control` +and `lsp45` — went past the point of diminishing returns for a chapter +appendix; flagged here rather than silently ignored, per this project's +verification-discipline rule. diff --git a/examples/45-linux-analysis-suites/config/lsp45.prom b/examples/45-linux-analysis-suites/config/lsp45.prom new file mode 100644 index 0000000..03434db --- /dev/null +++ b/examples/45-linux-analysis-suites/config/lsp45.prom @@ -0,0 +1,5 @@ +# HELP lsp45_answer A known constant this chapter ships so the openmetrics +# PMDA has something real to scrape without any network access -- pointed at +# via a `file://` config.d source, not an HTTP endpoint. +# TYPE lsp45_answer gauge +lsp45_answer 42 diff --git a/examples/45-linux-analysis-suites/demo.sh b/examples/45-linux-analysis-suites/demo.sh new file mode 100755 index 0000000..c2cb5da --- /dev/null +++ b/examples/45-linux-analysis-suites/demo.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# demo.sh [cpp|go|rust] [build|run] +# +# No per-language source here: PCP watches a system, not a language -- +# there's nothing C++/Go/Rust-specific to build. The [cpp|go|rust] argument +# only satisfies the runner's contract (scripts/test-all-examples.py loops +# langs x {build,verify}); the image built and run here is identical no +# matter which is passed. +set -euo pipefail +cd "$(dirname "$0")" + +IMAGE=lsp45-pcp +CONTAINER=lsp45-pcp-demo + +case "${1:-}" in + cpp|go|rust) shift ;; + *) echo "usage: $0 [cpp|go|rust] [build|run]" >&2; exit 2 ;; +esac + +build() { + podman build -t "$IMAGE" -f Containerfile . +} + +run() { + podman rm -f "$CONTAINER" >/dev/null 2>&1 || true + + # If a rootless podman.sock is up for this user, mount it so the layered + # podman PMDA sees REAL running containers (including this one) instead + # of just registering with zero values. This is a genuine, verified + # result: pmda-podman's own Install script only checks for + # /run/podman/podman.sock to decide whether to prompt about starting the + # podman service -- it works identically well whether the socket behind + # that path belongs to root or, as here, an unprivileged user. + local sock="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/podman/podman.sock" + local mount_args=() + if [ -S "$sock" ]; then + mount_args=(-v "$sock:/run/podman/podman.sock") + else + echo "note: no rootless podman.sock at $sock -- podman PMDA will" \ + "register but report zero container values" >&2 + fi + + podman run -d --name "$CONTAINER" --cpus=2 --memory=256m \ + "${mount_args[@]}" "$IMAGE" + + for _ in $(seq 1 20); do + podman exec "$CONTAINER" pminfo -f pmcd.version >/dev/null 2>&1 && break + sleep 1 + done + + echo "--- pcp: suite summary (openmetrics/podman should both be listed) ---" + podman exec "$CONTAINER" pcp + + echo "--- pminfo: the two layered PMDAs' namespaces ---" + podman exec "$CONTAINER" pminfo openmetrics podman + + echo "--- pminfo -f: a real scraped openmetrics value (file:// source, no network) ---" + podman exec "$CONTAINER" pminfo -f openmetrics.lsp45.lsp45_answer + + if [ -S "$sock" ]; then + echo "--- pminfo -f: real podman container visibility (via the mounted socket) ---" + podman exec "$CONTAINER" pminfo -f podman.container.name podman.container.running + fi + + echo "--- pmrep: one live sample of a core, always-present metric ---" + podman exec "$CONTAINER" pmrep -t 1 -s 1 kernel.all.load + + echo "container left running: podman exec/logs it, or podman rm -f $CONTAINER" +} + +case "${1:-}" in + build) build ;; + run) build; run ;; + "") build; run ;; + *) echo "usage: $0 [cpp|go|rust] [build|run]" >&2; exit 2 ;; +esac diff --git a/examples/45-linux-analysis-suites/verify.lua b/examples/45-linux-analysis-suites/verify.lua new file mode 100644 index 0000000..1accd99 --- /dev/null +++ b/examples/45-linux-analysis-suites/verify.lua @@ -0,0 +1,122 @@ +-- Verify 45-linux-analysis-suites. LOCAL but requires podman: builds the +-- Containerfile, runs pmcd, and asserts observable PCP behavior -- not +-- merely that the container started. +-- +-- No per-language behavior exists (see demo.sh's header note); the manifest +-- restricts this example to LSP_LANG=cpp so the runner drives it once, not +-- three times, per scripts/manifest.yaml. +-- +-- What's asserted, and why each item is real rather than assumed: +-- 1. the image builds -- and the build-time smoke stage inside the +-- Containerfile already failed once (loudly) during development when +-- a PMDA didn't register, so a green build here is a real gate, not +-- just "podman build exited 0". +-- 2. pmcd answers pminfo within a bounded poll. +-- 3. BOTH layered PMDAs (openmetrics, podman) appear in `pcp`'s summary. +-- 4. a REAL scraped openmetrics value: `openmetrics.lsp45.lsp45_answer` +-- comes from a `file://` config.d source shipped in this example +-- (config/lsp45.prom), not a live network scrape -- deliberately, so +-- the check has no external dependency, but the value is genuinely +-- parsed by the PMDA each time, not hardcoded. +-- 5. a real core PCP metric sample, `kernel.all.load`, independent of +-- anything this chapter added -- proves pmcd's base collection works +-- on its own. +-- 6. the podman PMDA registers its namespace (21 metrics). This script +-- does NOT assert live container VALUES from it: that requires +-- mounting the host's rootless podman.sock into the container +-- (demo.sh does this when one is present), which is not guaranteed +-- on every CI/host environment this runs on. What IS asserted here +-- (registration) is true unconditionally; the live-value behavior is +-- real and demonstrated by demo.sh, just not gated in this script. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" then + checks.skip("LSP_LANG must be cpp (got " .. tostring(lang) .. + ") -- this example has no per-language behavior; the manifest runs it " .. + "once under cpp") +end + +if checks.run("command -v podman").exit ~= 0 then + checks.skip("podman not available on this host") +end + +local image, c = "lsp45-pcp", "lsp45-pcp-verify" + +local function cleanup() + checks.run("podman rm -f " .. c .. " >/dev/null 2>&1; true") +end +cleanup() -- in case a previous run left something behind + +-- --------------------------------------------------------------------------- +-- 1. build -- the Containerfile's own smoke stage already gates this: if +-- either PMDA failed to register at build time, this fails right here. +-- --------------------------------------------------------------------------- + +local build = checks.run("podman build -t " .. image .. " -f Containerfile .") +checks.expect_exit(build, 0, "podman build succeeds (build-time PMDA gate passes)") + +-- --------------------------------------------------------------------------- +-- 2. run + wait for pmcd readiness (bounded poll, not a fixed sleep) +-- --------------------------------------------------------------------------- + +local run = checks.run( + "podman run -d --name " .. c .. " --cpus=2 --memory=256m " .. image) +checks.expect_exit(run, 0, "pmcd container starts") + +local ready = checks.run( + "for i in $(seq 1 20); do podman exec " .. c .. + " pminfo -f pmcd.version >/dev/null 2>&1 && { echo ready; break; }; sleep 1; done") +checks.expect_match(ready.out, "ready", "pmcd accepts connections within 20s") + +-- --------------------------------------------------------------------------- +-- 3. `pcp` summary names both layered PMDAs as active +-- --------------------------------------------------------------------------- + +local pcp_summary = checks.run("podman exec " .. c .. " pcp") +checks.expect_match(pcp_summary.out, "openmetrics", "pcp summary lists the openmetrics PMDA") +checks.expect_match(pcp_summary.out, "podman", "pcp summary lists the podman PMDA") + +-- --------------------------------------------------------------------------- +-- 4. a REAL scraped openmetrics value, from the file:// source this example +-- ships (config/lsp45.prom) -- not a live network scrape, but a real +-- parse of a real file, every time. +-- --------------------------------------------------------------------------- + +local om = checks.run("podman exec " .. c .. " pminfo -f openmetrics.lsp45.lsp45_answer") +checks.expect_match(om.out, "openmetrics%.lsp45%.lsp45_answer", + "pminfo -f resolves the scraped openmetrics metric name") +checks.expect_match(om.out, "value 42", + "the scraped value is the real constant this chapter's config ships (42)") + +-- --------------------------------------------------------------------------- +-- 5. podman PMDA: registers a real namespace (values need a mounted +-- podman.sock, which demo.sh does but this script does not require). +-- --------------------------------------------------------------------------- + +local pminfo_podman = checks.run("podman exec " .. c .. " pminfo podman") +checks.expect_match(pminfo_podman.out, "podman%.container%.", + "podman PMDA registers a podman.container.* metric namespace") + +-- --------------------------------------------------------------------------- +-- 6. pmrep: a real numeric sample for a stable, well-known CORE PCP metric +-- (not one of the layered PMDAs -- proves pmcd's base collection works +-- independent of anything this chapter added). +-- --------------------------------------------------------------------------- + +local pmrep = checks.run("podman exec " .. c .. " pmrep -t 1 -s 1 kernel.all.load") +checks.expect_exit(pmrep, 0, "pmrep runs against a known core metric") +checks.expect_match(pmrep.out, "k%.a%.load", "pmrep returns a sample for kernel.all.load") + +-- --------------------------------------------------------------------------- +-- cleanup: remove the container and the built image, unconditionally +-- --------------------------------------------------------------------------- + +cleanup() +checks.run("podman rmi -f " .. image .. " >/dev/null 2>&1; true") + +checks.finish() diff --git a/examples/manifest.yaml b/examples/manifest.yaml index 27ae3cf..6300930 100644 --- a/examples/manifest.yaml +++ b/examples/manifest.yaml @@ -184,3 +184,8 @@ examples: langs: [go] mode: local timeout: 300 + - id: 45-linux-analysis-suites + langs: [cpp] + mode: local + requires: [podman] + timeout: 600 diff --git a/scripts/lab/cloud-init/user-data.tmpl b/scripts/lab/cloud-init/user-data.tmpl index 4da4c14..857107f 100644 --- a/scripts/lab/cloud-init/user-data.tmpl +++ b/scripts/lab/cloud-init/user-data.tmpl @@ -80,6 +80,10 @@ packages: runcmd: # Enable the Cockpit socket so the web console is reachable at :9090. - [ systemctl, enable, --now, cockpit.socket ] + # SystemTap's kernel-module backend (Chapter 45) needs kernel-devel matching + # the RUNNING kernel; the cloud base image ships without it, and a plain + # `kernel-devel` can drift ahead of the booted kernel. Pin it to `uname -r`. + - [ bash, -c, "dnf install -y kernel-devel-$(uname -r) >> /var/log/lab-ready 2>&1 || echo 'kernel-devel: MISSING' >> /var/log/lab-ready" ] # Readiness stamp: bpftrace works and which kernel we're on. - [ bash, -c, "bpftrace --version > /var/log/lab-ready 2>&1 || echo 'bpftrace: MISSING' > /var/log/lab-ready" ] - [ bash, -c, "uname -r >> /var/log/lab-ready" ]