From 502836e3b24341dd70f39cb594e21a6d41035c42 Mon Sep 17 00:00:00 2001
From: rishabh <rishank69@gmail.com>
Date: Sat, 15 Aug 2026 22:51:29 +0800
Subject: [PATCH 1/3] Rewrite NVIDIA Fabric Manager post, add post-format CI
 check

- Restructure the NVIDIA Fabric Manager post into a two-part format:
  concept explanations with analogies, then the exact runbook
- Add tools/check_post_format.rb: validates every post's filename/
  folder date convention and required front matter fields
  (title, date, categories, tags)
- Wire it into the PR build workflow so it runs on every pull request
- Document the local command for it in README.md

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---
 .github/workflows/pr-build.yaml               |   3 +
 README.md                                     |  12 +
 ...c-manager-air-gapped-harvester-hgx-h200.md | 824 ++++++++++++------
 tools/check_post_format.rb                    | 100 +++
 4 files changed, 682 insertions(+), 257 deletions(-)
 create mode 100644 tools/check_post_format.rb

diff --git a/.github/workflows/pr-build.yaml b/.github/workflows/pr-build.yaml
index c50b02e..da14dca 100644
--- a/.github/workflows/pr-build.yaml
+++ b/.github/workflows/pr-build.yaml
@@ -22,6 +22,9 @@ jobs:
           ruby-version: 3.3
           bundler-cache: true
 
+      - name: Check post format
+        run: ruby tools/check_post_format.rb
+
       - name: Build site
         run: bundle exec jekyll b -d "_site"
         env:
diff --git a/README.md b/README.md
index 1f014a1..113f929 100644
--- a/README.md
+++ b/README.md
@@ -63,3 +63,15 @@ Then write the post in Markdown below the front matter. A few notes:
   on per post.
 - To link out to an externally-hosted post instead of writing one here, see any of the existing
   Medium-redirect posts in `_posts/` for the pattern (a `<script>` redirect plus a fallback link).
+
+### Checking your post is formatted correctly
+
+Before opening a PR, verify your post's filename, folder, and front matter are all correct:
+
+```bash
+docker compose exec site ruby tools/check_post_format.rb
+```
+
+It checks that every post: has a `YYYY-MM-DD-slug.md` filename, lives in a matching `YYYY-MM-DD/`
+folder, has valid YAML front matter, and has a non-empty `title`, `date`, `categories`, and `tags`.
+This also runs automatically as a CI check on every pull request.
diff --git a/_posts/2026-08-12/2026-08-12-installing-nvidia-fabric-manager-air-gapped-harvester-hgx-h200.md b/_posts/2026-08-12/2026-08-12-installing-nvidia-fabric-manager-air-gapped-harvester-hgx-h200.md
index a6fd5d7..2c3e25e 100644
--- a/_posts/2026-08-12/2026-08-12-installing-nvidia-fabric-manager-air-gapped-harvester-hgx-h200.md
+++ b/_posts/2026-08-12/2026-08-12-installing-nvidia-fabric-manager-air-gapped-harvester-hgx-h200.md
@@ -5,223 +5,445 @@ categories: [GPU Infrastructure]
 tags: [nvidia, fabric-manager, hgx, h200, nvswitch, harvester, kubevirt, gpu, air-gapped, vfio]
 ---
 
-Getting CUDA working inside passthrough VMs on an 8× H200 SXM HGX system running Harvester,
-where the host OS is immutable and the environment has no internet access and no NVIDIA vGPU licence.
+Getting CUDA working inside passthrough VMs on an 8× H200 SXM HGX system, where the host OS is
+immutable, the environment has no internet access, and there is no NVIDIA vGPU licence.
+
+The first half explains **what all the pieces are**, using everyday analogies. The second half is
+the **procedure**, with exact commands. Read the first half once; use the second half as a runbook.
 
 ---
 
-## Part 1 — Background: the NVIDIA concepts that matter here
+# Part One — Understanding the pieces
 
-### NVLink and NVSwitch
+## The building and the workers
 
-On an HGX baseboard, the GPUs are not only connected by PCIe. They also have **NVLink** ports —
-a much faster, GPU-to-GPU interconnect used for peer-to-peer memory access and collective
-operations in multi-GPU training.
+### Story
 
-On small systems, NVLink is a fixed mesh of direct GPU-to-GPU cables. On an HGX baseboard, every
-GPU's NVLink ports instead connect to **NVSwitch** — a crossbar switch ASIC that routes NVLink
-traffic between any pair of GPUs at full bandwidth. An 8-GPU HGX board typically carries four
-NVSwitch ASICs forming a single fabric domain across all eight GPUs.
+Imagine a big office building. Inside, there are **eight extremely fast workers**. Each is a genius
+at maths, but only maths. Give them a huge pile of calculations and they finish in seconds.
 
-Two things follow from this, and they drive everything else in this document:
+These workers are your **eight H200 GPUs**.
 
-1. **NVSwitch is not a per-GPU resource.** It is shared infrastructure belonging to the whole
-   baseboard. It has its own routing tables and memory-mapped address space, and appears as its
-   own PCI device — distinct from the GPUs.
-2. **NVSwitch is not a PCIe switch.** It switches NVLink traffic. The PCIe switches on the board
-   (e.g. Broadcom PEX-series) are a completely separate layer, and they are what determines your
-   IOMMU groupings. Don't conflate the two — they are different fabrics with different constraints.
+You rent out office space to tenants, and each tenant gets some of these workers.
 
-> **Further reading**
-> - [NVIDIA NVLink and NVSwitch overview](https://www.nvidia.com/en-us/data-center/nvlink/) — what the interconnect is and why it exists
-> - [NVIDIA HGX Software User Guide](https://docs.nvidia.com/datacenter/tesla/hgx-software-guide/index.html) — the full software stack for an HGX baseboard, bottom to top
+The **tenants are Virtual Machines**.
+
+### Technical
+
+```
+        ONE PHYSICAL SERVER (the "host")
+        ┌──────────────────────────────────────────┐
+        │  CPU + RAM + disks                       │
+        │                                          │
+        │  GPU0  GPU1  GPU2  GPU3                  │   8 × H200 SXM
+        │  GPU4  GPU5  GPU6  GPU7                  │
+        └──────────────────────────────────────────┘
+                        │
+             VMs run on top of this host
+        ┌───────────┐  ┌───────────┐  ┌───────────┐
+        │   VM  A   │  │   VM  B   │  │   VM  C   │
+        │  gets 4   │  │  gets 2   │  │  gets 2   │
+        │   GPUs    │  │   GPUs    │  │   GPUs    │
+        └───────────┘  └───────────┘  └───────────┘
+```
+
+---
+
+## Two kinds of roads
+
+### Story
+
+The workers need to talk to each other. Worker one finishes half a calculation and hands it to
+worker two.
+
+There are **two roads** in the building.
+
+**The normal corridor (PCIe).** Everyone uses it — the CPU, the disks, the network card. It works,
+but it's busy.
+
+**The private expressway (NVLink).** Built *only* for the workers, to hand work to each other. Many
+times faster than the corridor.
+
+For big AI training jobs, the expressway is the whole point. Without it, eight GPUs are just eight
+separate GPUs instead of one giant one.
 
-### Fabric Manager
+### Technical
 
-The NVSwitch ASICs do not configure themselves. **Fabric Manager (FM)** is a host-side service
-that:
+| | PCIe | NVLink |
+|---|---|---|
+| Who uses it | Everything — CPU, disk, NIC, GPU | GPU-to-GPU only |
+| Speed | Fast | Much faster |
+| Purpose | General system bus | Peer-to-peer GPU memory access |
+
+---
+
+## The junction — NVSwitch
 
-- discovers the NVSwitches and GPUs on the baseboard,
-- loads the correct topology file for the board,
-- programs the switch routing tables,
-- and reports fabric health.
+### Story
 
-Until FM has run successfully, GPUs on an NVSwitch system report:
+Now a problem. If every worker needs a private expressway to every other worker, eight workers means
+twenty-eight separate roads. Messy, expensive, doesn't scale.
+
+So the building has **one big junction in the middle** — like a traffic roundabout, or a telephone
+exchange. Every worker's expressway leads into it. The junction decides: "traffic from worker three
+heading to worker six — send it out that exit."
+
+**That junction is NVSwitch.** On this board there are four NVSwitch chips working together as one
+junction system for all eight GPUs.
+
+### Diagram
 
 ```
-Fabric State  : In Progress
+   Without a junction (mesh)          With NVSwitch
+
+   GPU ─────── GPU                     GPU   GPU   GPU   GPU
+    │ ╲       ╱ │                        ╲    │     │    ╱
+    │   ╲   ╱   │                         ╲   │     │   ╱
+    │     ╳     │                        ┌─────────────────┐
+    │   ╱   ╲   │                        │    NVSwitch     │  ← 4 chips
+    │ ╱       ╲ │                        │  (the junction) │
+   GPU ─────── GPU                       └─────────────────┘
+                                           ╱   │     │   ╲
+   Too many roads.                       GPU   GPU   GPU   GPU
+
+                                       Every GPU reaches every
+                                       other GPU through the middle.
 ```
 
-and CUDA refuses to initialise, typically with:
+### Two things that matter enormously
+
+**The junction belongs to the whole building — not to any one tenant.** You cannot hand it to
+Tenant A; if you did, Tenant A could redirect traffic belonging to Tenants B and C. NVSwitch
+**stays with the host** and is never given to a VM.
+
+**NVSwitch is not a PCIe switch.** The server also has PCIe switches (Broadcom PEX chips) — those
+are junctions for the *normal corridor*. Two completely different road systems. The PCIe switches
+are what determine IOMMU grouping; NVSwitch is a separate story entirely. Don't conflate them.
+
+> **Further reading**
+> - [NVIDIA NVLink and NVSwitch overview](https://www.nvidia.com/en-us/data-center/nvlink/)
+> - [NVIDIA HGX Software User Guide](https://docs.nvidia.com/datacenter/tesla/hgx-software-guide/index.html)
+
+---
+
+## The traffic policeman — Fabric Manager
+
+### Story
+
+Here's the thing about that junction: **when you switch the building on, the junction is dead.**
+
+No signs. No signals. No routing table. A worker sends traffic into it and it goes nowhere.
+
+Someone has to walk up and program it: "this entrance connects to that exit", "traffic from GPU
+three to GPU six goes this way", "check all the roads are actually working."
+
+**That someone is Fabric Manager.** A program that runs on the host, wakes up the junction, programs
+all the routes, then keeps watching for faults.
+
+Until it has done its job, the GPUs sit there saying *"I'm not ready, the fabric isn't set up"* —
+which is exactly the error seen here:
 
 ```
+Fabric State : In Progress          ← "still waiting for the policeman"
 Error 802: system not yet initialized
 ```
 
-FM depends on two other pieces:
+### Technical — its four jobs
+
+1. Coordinate with the NVSwitch driver to initialise and train switch-to-switch NVLinks
+2. Coordinate with the GPU driver to initialise and train switch-to-GPU NVLinks
+3. Configure routing among NVSwitch ports
+4. Monitor the fabric for NVLink and NVSwitch errors
+
+### What it depends on
 
 | Component | Role |
 |---|---|
 | NVIDIA kernel driver | Provides `nvidia-nvswitch` — the kernel-side driver that claims the NVSwitch PCI devices |
-| `libnvidia-nscq` | NSCQ = NVSwitch Configuration and Query. The library FM uses to talk to the switches |
+| `libnvidia-nscq` | NVSwitch Configuration and Query — the library Fabric Manager uses to talk to the switches |
 
-So the dependency chain is strictly: **driver → NSCQ → Fabric Manager**. Installing FM alone
-achieves nothing; it will start, fail to query NVSwitch device information, and exit.
+The chain is strictly **driver → NSCQ → Fabric Manager**. Installing Fabric Manager alone achieves
+nothing; it will start, fail to query NVSwitch device information, and exit.
 
-FM's four documented responsibilities are worth internalising, since they explain every symptom in
-this document:
+> **Further reading**
+> - [NVIDIA Fabric Manager User Guide](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html)
+> - [Fabric Manager — full contents](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/contents.html)
+> - [Fabric Manager Client (GitHub)](https://github.com/NVIDIA/Fabric-Manager-Client)
 
-1. Coordinate with the NVSwitch driver to initialise and train switch-to-switch NVLinks
-2. Coordinate with the GPU driver to initialise and train switch-to-GPU NVLinks
-3. Configure routing among NVSwitch ports
-4. Monitor the fabric for NVLink and NVSwitch errors
+---
 
-> **Further reading**
-> - [NVIDIA Fabric Manager User Guide](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html) — the primary reference for everything in this document
-> - [Fabric Manager config options](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/contents.html) — full table of contents, including config options, startup options and the service file
-> - [Fabric Manager Client (GitHub)](https://github.com/NVIDIA/Fabric-Manager-Client) — CLI for the partition-management API used in Shared NVSwitch mode
+## Who stands where — the most important rule
 
-### Where each piece runs in a virtualised setup
+### Story
 
-This is the single most important design point, and getting it wrong is a very common mistake.
+The policeman must stand **at the junction**, in the building's own control room.
 
-| Component | Host | Guest VM |
-|---|---|---|
-| NVIDIA kernel driver (for NVSwitch) | ✅ | — |
-| Fabric Manager + NSCQ | ✅ | ❌ **never** |
-| GPU driver | — | ✅ |
-| CUDA toolkit, PyTorch, workloads | — | ✅ |
+You cannot put him inside a rented office. From in there he can't even *see* the junction — there's
+a wall in the way. He'll stand confused, unable to do anything, and give up.
+
+**This was the original mistake here.** Fabric Manager had been installed inside the guest VM. It
+failed every time:
+
+```
+request to query NVSwitch device information from NVSwitch driver ... failed
+```
+
+Of course it failed. There was no junction in that room to look at.
+
+### Diagram — the correct split
+
+```
+┌─────────────────────────────────────────────────────────┐
+│  HOST  (the building's control room)                    │
+│                                                         │
+│   ✅ NVIDIA kernel driver  → grabs the NVSwitch chips   │
+│   ✅ libnvidia-nscq        → the "language" FM speaks   │
+│   ✅ Fabric Manager        → programs the junction      │
+│                                                         │
+│   ┌──── NVSwitch × 4 ────┐   ┌──── GPU × 8 ────┐        │
+│   │ owned by the host    │   │ handed to VMs   │        │
+│   └──────────────────────┘   └────────┬────────┘        │
+└───────────────────────────────────────┼─────────────────┘
+                                        │ passthrough
+                    ┌───────────────────┴──────────────────┐
+                    │  GUEST VM                            │
+                    │                                      │
+                    │   ✅ NVIDIA GPU driver               │
+                    │   ✅ CUDA toolkit, PyTorch           │
+                    │   ❌ Fabric Manager — NEVER HERE     │
+                    └──────────────────────────────────────┘
+```
+
+### How the two sides ever agree
+
+They don't talk to each other directly. **The GPU chip itself carries the message.**
 
-A guest VM that receives individual GPUs via PCI passthrough **cannot see the NVSwitch hardware**
-— and should not. Running FM inside such a guest fails immediately, because there is no NVSwitch
-device for it to query.
+1. Host Fabric Manager programs the junction → the GPU hardware records "fabric is ready"
+2. Guest starts, its GPU driver loads, reads that flag off the GPU
+3. Guest reports `Fabric State: Completed` and CUDA works
 
-The handshake between the two sides happens through the GPU hardware itself: host-side FM programs
-the fabric, and when the guest's GPU driver initialises, it reads that state from the GPU and
-reports `Fabric State: Completed`.
+This is why **a VM already running when Fabric Manager starts stays broken**. It read the flag too
+early, saw "not ready", and never checks again. Power-cycle the VM.
 
-### The two supported virtualisation models
+---
+
+## The two supported virtualisation models
 
 | Model | GPUs | NVSwitch | Fabric Manager | Use when |
 |---|---|---|---|---|
 | **Full baseboard passthrough** | All 8 to one VM | Also passed through | Runs **inside the guest** | One workload needs all 8 GPUs with full NVLink |
 | **Shared NVSwitch** | Passed through individually to different VMs | Stays host-owned | Runs **on the host** | GPUs split across multiple VMs |
 
-Passing through *some* GPUs plus the NVSwitch is not a valid configuration. The switch is either
-entirely host-owned, or it goes to a single VM along with every GPU on the board.
+Passing through *some* GPUs plus the NVSwitch is not valid. The switch is either entirely
+host-owned, or it goes to a single VM along with every GPU on the board.
 
 This document covers the **Shared NVSwitch** model.
 
 > **Further reading**
-> - [Full Passthrough Virtualization Model](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html#full-passthrough-virtualization-model) — supported VM configurations for 1, 2, 4, 8 and 16 GPUs, and their limitations
-> - [Shared NVSwitch Virtualization Model](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html#shared-nvswitch-virtualization-model) — service VM design, guest lifecycle, and the partition APIs
-> - [Bare Metal Mode](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html#bare-metal-mode) — the `FABRIC_MODE=0` behaviour this setup ended up relying on
+> - [Full Passthrough Virtualization Model](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html#full-passthrough-virtualization-model)
+> - [Shared NVSwitch Virtualization Model](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html#shared-nvswitch-virtualization-model)
+> - [Bare Metal Mode](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html#bare-metal-mode)
 
 ---
 
-## Part 2 — The problem
+## Passthrough — handing over the keys
 
-**Symptom (inside the guest VM):**
+### Story
 
-```
-Error 802: system not yet initialized
-torch.cuda.is_available() -> False
-```
+When you give a worker to Tenant A, you don't supervise them. You hand over the key and step back.
+The tenant brings their own tools and instructions, and works with that worker directly.
 
-```
-Fabric
-    State   : In Progress
-    Status  : N/A
-GPU Fabric GUID : N/A
-```
+Building management now has **no access to that worker at all**.
 
-**First wrong turn:** Fabric Manager had been installed *inside the guest VM*, where it failed with:
+### Technical
 
+Each GPU gets bound to a driver called `vfio-pci` — a placeholder meaning "this device is rented
+out, hands off."
+
+```bash
+lspci -k -s <gpu-bdf>
 ```
-request to query NVSwitch device information from NVSwitch driver ... failed
 ```
+3D controller: NVIDIA Corporation GH100 [H200 SXM 141GB]
+	Kernel driver in use: vfio-pci      ← rented out to a VM
+```
+
+That's correct and healthy. A side effect worth knowing: since the host no longer controls the GPUs,
+running `nvidia-smi` on the host shows nothing. That is expected, not a fault.
+
+---
+
+## What was actually broken
+
+### Story
+
+You looked at the junction and found something odd. It wasn't misconfigured. It wasn't broken.
 
-That is correct behaviour — a passthrough guest has no NVSwitch. FM was in the wrong place.
+**Nobody had even switched it on.** No electrician had ever wired it up.
 
-**Actual root cause**, found by inspecting the host:
+### Technical
 
 ```bash
 lspci -k
 ```
-
 ```
-xx:00.0 Bridge: NVIDIA Corporation GH100 [H100 NVSwitch] (rev a1)
+xx:00.0 Bridge: NVIDIA Corporation GH100 [H100 NVSwitch]
 	Subsystem: NVIDIA Corporation Device 1796
-        <-- no "Kernel driver in use" line at all
+        ← NOTHING. No "Kernel driver in use" line at all.
 ```
 
-All four NVSwitch bridges had **no kernel driver bound**. Meanwhile the GPUs were correctly bound
-to `vfio-pci` for passthrough, and NVMe was on `nvme`. Only the switches were orphaned.
+Compare with everything else on the machine:
+
+| Device | Driver | Status |
+|---|---|---|
+| GPUs | `vfio-pci` | correct — rented to VMs |
+| NVMe disks | `nvme` | correct |
+| Network cards | `mlx5_core` / `vfio-pci` | correct |
+| **NVSwitch × 4** | **(none)** | ← **the bug** |
+
+No driver on the switches → Fabric Manager can't reach them → the fabric never initialises → every
+GPU in every VM waits forever.
 
-No driver on the switches → FM cannot initialise the fabric → every passthrough GPU waits forever
-on a handshake that never comes.
+**One missing driver. That was the whole problem.** Everything after this was the fight to install it.
 
 ---
 
-## Part 3 — Approaches that did not work
+## Why installing it was so hard
 
-Documented so they are not retried.
+Three walls, all at once.
 
-| Attempt | Outcome |
-|---|---|
-| Harvester `nvidia-driver-toolkit` add-on | Requires a **licensed vGPU/GRID `.run` driver** from NVIDIA's licensing portal. Also vGPU-specific — the documentation makes no mention of NVSwitch or Fabric Manager. Not applicable to full PCI passthrough. |
-| NVIDIA Container Toolkit | Wrong layer entirely. It exposes GPUs to *containers* and explicitly assumes a working host driver already exists. Irrelevant when GPUs go straight into VMs via VFIO. |
-| `zypper install` on host | `The target filesystem is mounted as read-only.` |
-| `transactional-update pkg install` | `command not found` — removed from this build. |
-| `mount -o remount,rw /` | `cannot remount /dev/loopN read-write, is write-protected` — the root is a loop-mounted OS **image**, not merely a read-only mount flag. |
-| `.run` driver installer | Designed for a normal writable filesystem; incompatible with an immutable/transactional root. |
-| Passing NVSwitch through to a guest | Breaks the shared-fabric model; guest gains control over routing affecting GPUs owned by other VMs. |
+### Wall one — the sealed room
+
+**Story:** a hotel room where every piece of furniture is bolted to the floor and the cupboards are
+welded shut. You cannot add anything. That's how Harvester's OS is built — deliberately, so nobody
+breaks the cluster by fiddling.
+
+**Technical:**
+```bash
+zypper install nvidia-open
+# The target filesystem is mounted as read-only.
 
-### Why no package can be installed
+transactional-update
+# command not found
 
-The host is SUSE Linux Micro with an elemental-style immutable root. From `/proc/cmdline`:
+mount -o remount,rw /
+# cannot remount /dev/loopN read-write, is write-protected
+```
+
+That last one is the killer. The root filesystem isn't just *flagged* read-only — it's a
+**loop-mounted image file** (`/cOS/active.img`). Like a sealed CD. There's no flag to flip.
 
+**But** one drawer *is* unlocked. From the boot line:
 ```
 root=LABEL=COS_STATE cos-img/filename=/cOS/active.img
 rd.cos.mount=LABEL=COS_OEM:/oem
 rd.cos.mount=LABEL=COS_PERSISTENT:/usr/local
 ```
+`/usr/local` is writable **and** survives reboots. That drawer became the whole solution.
+
+### Wall two — no internet
+
+**Story:** the building is in a submarine. No deliveries. To get anything, you send a courier to the
+surface with a shopping list, and they bring back parcels.
+
+**Technical:** air-gapped network. The courier was a separate VM with internet access.
+
+### Wall three — no licence
+
+**Story:** the official tool for this job needs a paid membership card. You don't have one.
+
+**Technical:** Harvester's NVIDIA driver toolkit add-on requires a licensed **vGPU/GRID** driver from
+NVIDIA's licensing portal. It's also built for vGPU sharing rather than full passthrough, and its
+documentation never mentions NVSwitch or Fabric Manager. Dead end on two counts.
+
+**Good news:** NVIDIA's datacenter driver has been **open source and free since 2022**. No licence
+needed. That's the `nvidia-open` line — exactly what passthrough wants.
+
+---
 
-The root filesystem is a read-only image file. But note the last line: **`/usr/local` is a
-writable, persistent partition.** That is the opening.
+## Approaches that did not work
+
+Documented so they aren't retried.
+
+| Attempt | Outcome |
+|---|---|
+| Harvester NVIDIA driver toolkit add-on | Requires a **licensed vGPU/GRID `.run` driver**. Also vGPU-specific — no mention of NVSwitch or Fabric Manager anywhere in its docs. |
+| NVIDIA Container Toolkit | Wrong layer entirely. Exposes GPUs to *containers*, and explicitly assumes a working host driver already exists. Irrelevant when GPUs go straight into VMs via VFIO. |
+| `zypper install` on host | `The target filesystem is mounted as read-only.` |
+| `transactional-update pkg install` | `command not found` — removed from this build. |
+| `mount -o remount,rw /` | Refused; the root is a loop-mounted OS **image**, not merely a read-only mount flag. |
+| `.run` driver installer | Designed for a normal writable filesystem; incompatible with an immutable root. |
+| Passing NVSwitch through to a guest | Breaks the shared-fabric model; the guest gains control over routing affecting GPUs owned by other VMs. |
 
 > **Further reading**
-> - [Harvester PCI Devices add-on](https://docs.harvesterhci.io/v1.4/advanced/addons/pcidevices/) — how `PCIDevice` / `PCIDeviceClaim` bind host devices to `vfio-pci`
-> - [`harvester/pcidevices` controller (GitHub)](https://github.com/harvester/pcidevices) — the controller doing the runtime binding, and the CRD shapes
-> - [Harvester NVIDIA driver toolkit add-on](https://docs.harvesterhci.io/v1.8/advanced/addons/nvidiadrivertoolkit/) — the vGPU-oriented add-on, and why it doesn't apply here
-> - [Harvester CloudInit CRD](https://docs.harvesterhci.io/v1.8/advanced/cloudinitcrd/) — the supported way to persist host-level config on an immutable OS
-> - [NVIDIA Container Toolkit install guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) — note its prerequisite that a host driver already exists; it is a container-layer tool, not a passthrough one
+> - [Harvester PCI Devices add-on](https://docs.harvesterhci.io/v1.4/advanced/addons/pcidevices/)
+> - [`harvester/pcidevices` controller (GitHub)](https://github.com/harvester/pcidevices)
+> - [Harvester NVIDIA driver toolkit add-on](https://docs.harvesterhci.io/v1.8/advanced/addons/nvidiadrivertoolkit/)
+> - [Harvester CloudInit CRD](https://docs.harvesterhci.io/v1.8/advanced/cloudinitcrd/)
+> - [NVIDIA Container Toolkit install guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
+
+---
+
+## The trick that made it work
+
+### Story
+
+You can't install furniture in the sealed room. **But you can carry a toolbox in and use it.**
+
+So instead of *installing* the driver: unzip the package into the one unlocked drawer, then pick up
+the driver file and plug it into the kernel directly.
+
+The system is never modified. Nothing is installed. It just works.
+
+### Why this is legitimate, not a hack
+
+A kernel module doesn't care where it lives. "Installing" it just means copying it to a standard
+folder so the system finds it automatically. If you already know where it is, `insmod` loads it from
+any path. The standard folder is a convenience, not a requirement.
 
 ---
 
-## Part 4 — The approach that worked
+# Part Two — The procedure
 
-> Kernel modules do not have to be installed to be loaded. `insmod` accepts any file path.
-> So: extract the RPMs into the writable partition and load the module by hand.
+## Target environment
 
-No package installation, no filesystem remount, no OS modification.
+| Item | Value |
+|---|---|
+| Host OS | SUSE Linux Micro (SLE16-based), Harvester v1.8 |
+| Kernel | `6.12.0-160000.28-default` |
+| Platform | AMD Genoa/Bergamo |
+| GPUs | 8 × NVIDIA H200 SXM 141GB (`10de:2335`), HGX baseboard |
+| NVSwitch | 4 × GH100 NVSwitch bridges |
+| Model | Individual GPU passthrough to KubeVirt VMs; NVSwitch host-owned |
+| Constraint | Air-gapped; no NVIDIA vGPU licence |
+| Driver version used | **580.126.09** |
 
 ---
 
-## Part 5 — Phase 1: obtaining the packages (air-gapped)
+## Obtaining the packages
+
+Performed on a separate staging VM with internet access and Docker. The staging VM ran Ubuntu, which
+has no `zypper`, so a SUSE container was used to resolve dependencies properly rather than scraping
+HTML directory listings.
+
+### Story
 
-Performed on a separate staging VM that has internet access and Docker. The staging VM ran Ubuntu,
-which has no `zypper`, so a SUSE container was used to resolve dependencies properly rather than
-scraping HTML directory listings.
+You send your courier with a shopping list. But the shop is confusing:
 
-### 5.1 Start a SUSE container
+- the **box** for the part is in Shop A
+- the **actual part** is in Shop B — a completely different shop
+- the **tools** to fit it are in Shop C
+
+And every piece must be the **exact same model number**, or nothing fits together.
+
+### Start a SUSE container
 
 ```bash
 mkdir -p rpms
 docker run -it --rm -v $(pwd)/rpms:/rpms registry.opensuse.org/opensuse/leap:16.0 bash
 ```
 
-### 5.2 Add the NVIDIA repositories
+### Add the NVIDIA repositories
 
 ```bash
 zypper ar -G https://download.nvidia.com/suse/sle16/ nvidia-gfx
@@ -229,7 +451,7 @@ zypper ar -G https://developer.download.nvidia.com/compute/cuda/repos/sles15/x86
 zypper refresh
 ```
 
-### 5.3 Find what actually exists
+### Find what actually exists
 
 ```bash
 zypper se -s nvidia-open-driver-G06-signed-kmp
@@ -237,47 +459,46 @@ zypper se -s nvidia-fabricmanager
 zypper se -s libnvidia-nscq
 ```
 
-**Key discovery — the packages are spread across three repositories:**
-
-| Package | Repository |
-|---|---|
-| `nvidia-open-driver-G06-signed-kmp-default` — *the actual kernel module* | **Leap `repo-oss`** (the distribution repo) |
-| `nvidia-open-driver-G06-signed-kmp-meta` — a 36 KB pointer only | `nvidia-gfx` |
-| `nvidia-fabricmanager`, `libnvidia-nscq` | `nvidia-cuda` |
-| `nvidia-compute-utils-G06`, `nvidia-common-G06`, `nvidia-compute-G06` | `nvidia-gfx` / `nvidia-cuda` |
+**The confusing bit:** NVIDIA's own SUSE repo contains only a **36 KB meta-package** — an empty box,
+a pointer. The real 9.5 MB kernel module comes from **openSUSE Leap's `repo-oss`**, the Linux
+distribution's own repo. On SUSE, **SUSE builds and signs kernel modules**, not NVIDIA.
 
-NVIDIA's SUSE repo ships **only** userspace packages and meta-packages. SUSE builds and signs the
-kernel module itself, so the real binary comes from the distribution repository. Scraping NVIDIA's
-repo alone will never find it.
+You could search NVIDIA's site forever and never find it.
 
-### 5.4 Choosing a version
+```
+   ┌── nvidia-gfx repo ────────┐   meta-package (empty box, 36 KB)
+   │                           │   userspace tools
+   └───────────────────────────┘
 
-Two constraints must hold simultaneously:
+   ┌── nvidia-cuda repo ───────┐   nvidia-fabricmanager
+   │                           │   libnvidia-nscq
+   └───────────────────────────┘
 
-1. The version must exist in **all** the repos involved — FM must match the driver version exactly,
-   or it refuses to attach.
-2. The kernel module package (KMP) must be **kABI-compatible** with the running host kernel.
+   ┌── Leap repo-oss ──────────┐   ★ nvidia-open-driver...kmp-default
+   │  (the DISTRO's own repo)  │     ← the actual kernel module, 9.5 MB
+   └───────────────────────────┘
+```
 
-The KMP filename encodes the kernel it was built against, e.g.:
+### Choosing the version — a three-way match
 
-```
-nvidia-open-driver-G06-signed-kmp-default-<driver>_k6.12.0_<kernel-abi>-...
-```
+Every candidate had to satisfy three conditions at once: present in the driver repo, matching
+Fabric Manager and NSCQ available, and kernel-ABI compatible with the host.
 
-Candidate matrix (host kernel ABI was `160000.28`):
+| Version | In driver repo? | FM + NSCQ exist? | Kernel match? | Verdict |
+|---|---|---|---|---|
+| 580.159.03 | yes | yes | built for `.29` — **newer than host** | risky |
+| 580.126.18 | yes | **no** | `.26` | unusable |
+| **580.126.09** | yes | **yes** | built for `.9` — older, same family | ✅ **chosen** |
+| 580.119.02 | yes | **no** | `.7` | unusable |
 
-| Driver | Built against | FM + NSCQ available? | Verdict |
-|---|---|---|---|
-| 580.159.03 | `160000.29` — **newer than host** | yes | risky |
-| 580.126.18 | `160000.26` | **no** | unusable |
-| **580.126.09** | `160000.9` — older, same codestream | **yes** | **chosen** |
-| 580.119.02 | `160000.7` | no | unusable |
+**Story version of the kernel rule:** a spare part made for last year's model usually fits this
+year's car, because the manufacturer keeps the mounting points the same. A part made for *next*
+year's model might need a bracket that doesn't exist yet.
 
-**Rule of thumb:** within a single SUSE codestream, kernel ABI is stable, so a module built against
-an *older* kernel in that stream loads fine on a newer one. The reverse direction is the risky one —
-a module built against a newer kernel may reference symbols the running kernel does not have.
+The host kernel was `160000.28`. The module built for `160000.9` was made earlier in the same family
+→ safe. The one built for `160000.29` was made later → risky.
 
-### 5.5 Download without installing
+### Download without installing
 
 ```bash
 zypper --pkg-cache-dir /rpms install -y --download-only \
@@ -289,25 +510,30 @@ exit
 ```
 
 Syntax notes:
-
 - The version goes after `=`. Appending it to the package name fails.
-- `nvidia-fabricmanager` and `libnvidia-nscq` have **no** `-580` branch suffix; modern packages
-  carry the branch in the version field.
+- `nvidia-fabricmanager` and `libnvidia-nscq` have **no** `-580` branch suffix; modern packages carry
+  the branch in the version field.
+
+### ⚠️ The parcel you must throw away
 
-### ⚠️ 5.6 Exclude the kernel package
+One of the 65 downloaded files is **a whole new kernel** (`kernel-default`), newer than the host's.
+The container has no kernel, so the resolver helpfully adds one.
 
-The resolver pulls **`kernel-default`** (a full kernel, newer than the host's) because the container
-has no kernel installed. This must **never** reach the host — the platform manages its own kernel,
-and replacing it out-of-band risks breaking the node.
+**Story:** the courier came back with a replacement engine you never ordered. Fitting it would wreck
+the car.
+
+The platform manages its own kernel. Replacing it out-of-band risks breaking the node.
 
 ---
 
-## Part 6 — Phase 2: internal repository
+## Building the internal repository
 
 ```bash
 mkdir -p nvidia-repo
 find rpms -name '*.rpm' ! -name 'kernel-default*' -exec cp {} nvidia-repo/ \;
+
 ls nvidia-repo | grep kernel     # MUST return nothing
+
 createrepo_c nvidia-repo/
 cd nvidia-repo && python3 -m http.server 8000
 ```
@@ -322,23 +548,25 @@ curl -sI http://<staging-vm-ip>:8000/repodata/repomd.xml    # expect 200 OK
 
 ---
 
-## Part 7 — Phase 3: load the driver on the host
+## Loading the driver on the host
 
-### 7.1 Extract into the writable partition
+### Extract into the writable partition
 
 ```bash
 mkdir -p /usr/local/nvidia && cd /usr/local/nvidia
+
 curl -O http://<staging-vm-ip>:8000/nvidia-open-driver-G06-signed-kmp-default-580.126.09_k6.12.0_160000.9-160000.1.1.x86_64.rpm
 curl -O http://<staging-vm-ip>:8000/nvidia-fabricmanager-580.126.09-1.x86_64.rpm
 curl -O http://<staging-vm-ip>:8000/libnvidia-nscq-580.126.09-1.x86_64.rpm
 curl -O http://<staging-vm-ip>:8000/nvidia-modprobe-580.126.09-160000.22.1.x86_64.rpm
+
 for f in *.rpm; do rpm2cpio $f | cpio -idm; done
 ```
 
-`rpm2cpio | cpio -idm` unpacks an RPM's contents into the current directory without touching the
-RPM database or the system.
+`rpm2cpio | cpio -idm` unpacks an RPM's contents into the current directory without touching the RPM
+database or the system. It's just "unzip".
 
-### 7.2 Decompress and load the module
+### Decompress and load the module
 
 SUSE ships kernel modules zstd-compressed; `insmod` needs them decompressed.
 
@@ -355,47 +583,54 @@ Success:
 NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  580.126.09  Release Build
 ```
 
-No symbol or version errors — the kABI compatibility assumption held.
+No symbol or version errors — the compatibility gamble paid off.
 
-> If Secure Boot is enforced, an unenrolled module signature will be rejected here. Check with
+> If Secure Boot is enforced, an unenrolled module signature is rejected here. Check with
 > `mokutil --sb-state`, or `dmesg | grep -i "secure boot"` if `mokutil` is absent.
 
-### 7.3 Verify driver binding — both conditions must hold
+### The moment of truth — both must be true
 
 ```bash
-lspci -k -s <nvswitch-bdf>     # e.g. 09:00.0
-lspci -k -s <gpu-bdf>          # e.g. 23:00.0
+lspci -k -s <nvswitch-bdf>
+lspci -k -s <gpu-bdf>
 ```
 
 ```
 Bridge: NVIDIA Corporation GH100 [H100 NVSwitch]
-	Kernel driver in use: nvidia-nvswitch     <-- switch claimed ✅
+	Kernel driver in use: nvidia-nvswitch     ← ✅ junction switched on
+
 3D controller: NVIDIA Corporation GH100 [H200 SXM 141GB]
-	Kernel driver in use: vfio-pci            <-- passthrough intact ✅
+	Kernel driver in use: vfio-pci            ← ✅ tenants undisturbed
 ```
 
-If a GPU shows `nvidia`, **stop**. The driver has stolen a passthrough device. See Part 10.
+The new driver could have grabbed the GPUs away from the VMs. If a GPU shows `nvidia`, **stop** —
+see the binding-protection note in the persistence section.
 
 ---
 
-## Part 8 — Phase 4: create NVSwitch device nodes
+## Making door handles — device nodes
+
+### Story
+
+The junction is powered on. But there's no door into the control room — no handle, no keyhole. The
+policeman can't get in. You have to fit the handles yourself.
 
-There are no udev rules for these on a minimal host, and `nvidia-modprobe -c 0` will not help — it
-creates *GPU* nodes, and all GPUs are vfio-bound and invisible to the NVIDIA driver.
+### Technical
 
-Find the character device major number and confirm the switches registered:
+Linux programs reach hardware through files in `/dev`. Normally `udev` creates these automatically,
+but this minimal OS has no rules for NVSwitch. And `nvidia-modprobe -c 0` won't help — it creates
+*GPU* nodes, and all GPUs are vfio-bound and invisible to the NVIDIA driver.
 
 ```bash
 cat /proc/devices | grep -i nvidia
 ls /proc/driver/nvidia-nvswitch/devices/
 ```
-
 ```
-507 nvidia-nvswitch
-0000:09:00.0  0000:0a:00.0  0000:0b:00.0  0000:0c:00.0
+507 nvidia-nvswitch                                        ← the "house number"
+0000:xx:00.0  0000:xx:00.0  0000:xx:00.0  0000:xx:00.0     ← all 4 switches present
 ```
 
-Create the nodes — one per switch, plus a control node at minor 255:
+One door per switch, plus a master control door at minor 255:
 
 ```bash
 MAJOR=$(awk '/nvidia-nvswitch/{print $1}' /proc/devices)
@@ -404,12 +639,19 @@ for i in 0 1 2 3; do mknod -m 666 /dev/nvidia-nvswitch$i c $MAJOR $i; done
 ls -la /dev/nvidia-nvswitch*
 ```
 
-> Read the major from `/proc/devices` rather than hardcoding it — dynamic majors can change
-> between boots.
+> Read the major from `/proc/devices` rather than hardcoding it — dynamic majors can change between
+> boots.
 
 ---
 
-## Part 9 — Phase 5: configure and start Fabric Manager
+## Sending in the policeman — Fabric Manager
+
+### Story
+
+The policeman arrives with a map of the building, but his map has the wrong address printed on it —
+it points to a control room that doesn't exist here. Correct the address, and he walks straight in.
+
+### Technical
 
 The stock config references absolute paths under `/usr/share` and `/var`, which don't exist here.
 Copy it and repoint everything at the extracted tree.
@@ -417,18 +659,12 @@ Copy it and repoint everything at the extracted tree.
 ```bash
 cd /usr/local/nvidia
 cp ./usr/share/nvidia/nvswitch/fabricmanager.cfg fm.cfg
+
 sed -i 's|^TOPOLOGY_FILE_PATH=.*|TOPOLOGY_FILE_PATH=/usr/local/nvidia/usr/share/nvidia/nvswitch|' fm.cfg
 sed -i 's|^LOG_FILE_NAME=.*|LOG_FILE_NAME=/usr/local/nvidia/fabricmanager.log|'                   fm.cfg
 sed -i 's|^STATE_FILE_NAME=.*|STATE_FILE_NAME=/usr/local/nvidia/fabricmanager.state|'             fm.cfg
 sed -i 's|^DAEMONIZE=.*|DAEMONIZE=0|'                                                             fm.cfg
-```
-
-The topology directory contains a file per supported board; the HGX H100/H200 baseboard uses
-`dgxh100_hgxh100_topology`. FM selects it automatically once the path is correct.
 
-Start it:
-
-```bash
 export LD_LIBRARY_PATH=/usr/local/nvidia/usr/lib64:/usr/local/nvidia/usr/lib:$LD_LIBRARY_PATH
 ./usr/bin/nv-fabricmanager -c /usr/local/nvidia/fm.cfg
 ```
@@ -439,49 +675,131 @@ NVLink Peer-to-Peer support will be enabled once the GPUs are successfully
 registered with the NVLink fabric.
 ```
 
-### A note on FABRIC_MODE
+**That sentence is the goal of the entire exercise.**
+
+### What's a topology file?
+
+The junction's blueprint. Different server models are wired differently, so NVIDIA ships one
+blueprint per board — the HGX H100/H200 baseboard uses `dgxh100_hgxh100_topology`. Fabric Manager
+picks the right one automatically once the folder path is correct.
+
+### FABRIC_MODE — the one lucky break
 
 | Value | Meaning |
 |---|---|
 | `0` | Bare metal — FM expects to see the GPUs itself |
-| `1` | Shared NVSwitch — FM configures switches, then waits for a hypervisor service to call its partition-activation API |
+| `1` | Shared NVSwitch — FM configures switches, then **waits for a hypervisor service to call its partition-activation API** |
+
+By the book, this setup looks like mode `1`. But mode `1` requires the virtualisation platform to
+actively call Fabric Manager's partition API — and KubeVirt-based platforms have no such feature.
+That would have been a wall with no way around it.
+
+**Mode `0` worked anyway**, even with the GPUs vfio-bound and invisible to the host driver. The
+switches initialise, and each GPU registers with the fabric when its driver starts inside its guest.
+
+> ⚠️ Worth testing before trusting this generally: mode `1` exists to *enforce isolation* between
+> tenants. Mode `0` may be leaving the fabric open rather than partitioned. Fine for a single tenant;
+> check `nvidia-smi topo -m` and a peer-to-peer bandwidth test before relying on it for multi-tenant
+> work.
+
+---
+
+## Verification
+
+### On the host
+
+```bash
+systemctl status nvidia-fabricmanager-local     # active (running)
+pgrep -af nv-fabricmanager                      # plain pgrep fails — name exceeds 15 chars
+tail -30 /usr/local/nvidia/fabricmanager.log
+journalctl -u nvidia-fabricmanager-local -n 30
+
+lspci -k -s <nvswitch-bdf>    # nvidia-nvswitch
+lspci -k -s <gpu-bdf>         # vfio-pci
+```
+
+### In the guest VM
+
+Power-cycle the VM after Fabric Manager starts. The fabric handshake happens during GPU driver init
+inside the guest and is not retried, so a VM that was already running stays broken.
+
+```bash
+nvidia-smi -q | grep -A5 Fabric
+```
+```
+Fabric
+    State   : Completed        ← was "In Progress"
+    Status  : Success
+    CliqueId: 0
+```
+```bash
+python -c "import torch; print(torch.cuda.is_available())"     # True
+python -c "import torch; print(torch.__version__, torch.version.cuda)"
+```
 
-**`FABRIC_MODE=0` worked**, even though the GPUs are vfio-bound and invisible to the host driver.
-The switches initialise, and each GPU registers with the fabric when its driver starts inside its
-guest. Mode `1` was *not* required — which matters, because it expects a hypervisor integration
-that KubeVirt-based platforms do not provide out of the box.
+### The full chain, finally connected
+
+```
+  ┌─ HOST ──────────────────────────────────────────────┐
+  │                                                     │
+  │  1. insmod nvidia.ko                                │
+  │         ↓                                           │
+  │  2. NVSwitch chips claimed by nvidia-nvswitch       │
+  │         ↓                                           │
+  │  3. /dev/nvidia-nvswitch* door handles created      │
+  │         ↓                                           │
+  │  4. Fabric Manager programs all routes  ✅          │
+  │         ↓                                           │
+  │  5. GPU hardware records "fabric ready"             │
+  └───────────────────┬─────────────────────────────────┘
+                      │  (passthrough)
+  ┌─ GUEST VM ────────┴─────────────────────────────────┐
+  │  6. GPU driver starts, reads the flag               │
+  │  7. Fabric State: Completed                         │
+  │  8. torch.cuda.is_available() → True    🎉          │
+  └─────────────────────────────────────────────────────┘
+```
 
 ---
 
-## Part 10 — Persistence
+## Making it survive a reboot
+
+### Story
+
+Everything above was written in pencil. Reboot the machine and it's rubbed out — the driver unplugs,
+the door handles vanish, the policeman goes home.
 
-Nothing loaded at runtime survives a reboot: the module, the device nodes and the FM process all
-disappear. Only the extracted files under `/usr/local` persist.
+The *parts* stay in the drawer (`/usr/local` survives). But somebody has to redo the assembly each
+morning. So you write the instructions down and hire someone to follow them at 6 a.m. daily.
 
-### 10.1 Startup script
+### The startup script
 
 ```bash
 cat > /usr/local/nvidia/start-fm.sh << 'EOF'
 #!/bin/bash
 set -e
 cd /usr/local/nvidia
+
 # 1. Load the NVIDIA kernel module (claims the NVSwitch bridges)
 lsmod | grep -q '^nvidia ' || \
   insmod ./usr/lib/modules/6.12.0-160000.9-default/updates/nvidia.ko
+
 # 2. Create NVSwitch device nodes
 MAJOR=$(awk '/nvidia-nvswitch/{print $1}' /proc/devices)
 [ -e /dev/nvidia-nvswitchctl ] || mknod -m 666 /dev/nvidia-nvswitchctl c $MAJOR 255
 for i in 0 1 2 3; do
   [ -e /dev/nvidia-nvswitch$i ] || mknod -m 666 /dev/nvidia-nvswitch$i c $MAJOR $i
 done
+
 # 3. Start Fabric Manager
 export LD_LIBRARY_PATH=/usr/local/nvidia/usr/lib64:/usr/local/nvidia/usr/lib:$LD_LIBRARY_PATH
 exec ./usr/bin/nv-fabricmanager -c /usr/local/nvidia/fm.cfg
 EOF
+
 chmod +x /usr/local/nvidia/start-fm.sh
 ```
 
-### 10.2 systemd unit
+### The systemd unit
 
 ```bash
 cat > /etc/systemd/system/nvidia-fabricmanager-local.service << 'EOF'
@@ -498,23 +816,40 @@ RestartSec=10
 [Install]
 WantedBy=multi-user.target
 EOF
+
 systemctl daemon-reload
 systemctl enable --now nvidia-fabricmanager-local
 systemctl status nvidia-fabricmanager-local
 ```
 
-> **`DAEMONIZE=0` is required** (set in Part 9). FM daemonises by default, which makes a
-> `Type=simple` unit report `inactive (dead)` moments after start — systemd sees the parent exit
-> while FM keeps running untracked, and `Restart=on-failure` becomes useless. With `DAEMONIZE=0`
-> the service shows `active (running)` and logs land in journald.
-> (Alternative: keep `DAEMONIZE=1` and use `Type=forking`.)
+### The DAEMONIZE puzzle
+
+The first attempt showed `inactive (dead)` — but the logs said it succeeded. Confusing.
+
+**Story:** you hired a supervisor to watch the worker. The worker walked in the front door, slipped
+out the back, and got on with the job through a side entrance. The supervisor saw him leave and
+wrote "gone home." Work was happening; supervision wasn't.
 
-### 10.3 Surviving OS upgrades
+**Technical:** Fabric Manager daemonises itself by default. With `Type=simple`, systemd watches the
+original process, sees it exit, and marks the service dead — while FM runs on untracked.
+`Restart=on-failure` becomes useless, because systemd can't tell when it dies.
+
+Setting `DAEMONIZE=0` keeps it in the foreground:
+
+```
+Active: active (running)
+Main PID: <pid> (nv-fabricmanage)
+```
+
+Properly supervised, restartable, and logs go to journald.
+
+> Alternative: keep `DAEMONIZE=1` and use `Type=forking`. The `DAEMONIZE=0` route is cleaner.
+
+### Surviving OS upgrades
 
 On an immutable OS, `/etc` may be reset from the image at boot, and is definitely replaced on an OS
 upgrade. To make the unit file reappear every boot, write it via the platform's own cloud-init
-mechanism (which writes into the persistent `/oem` partition) rather than relying on the manual
-`/etc` write:
+mechanism, which writes into the persistent `/oem` partition:
 
 ```yaml
 apiVersion: node.harvesterhci.io/v1beta1
@@ -536,11 +871,13 @@ spec:
                 [Unit]
                 Description=NVIDIA Fabric Manager (local extracted)
                 After=network.target
+
                 [Service]
                 Type=simple
                 ExecStart=/usr/local/nvidia/start-fm.sh
                 Restart=on-failure
                 RestartSec=10
+
                 [Install]
                 WantedBy=multi-user.target
           commands:
@@ -558,7 +895,7 @@ elemental run-stage boot           # re-execute the boot stage now
 systemctl status nvidia-fabricmanager-local
 ```
 
-### 10.4 Protecting the passthrough binding
+### Protecting the passthrough binding
 
 If GPU→`vfio-pci` binding is performed at runtime by a controller pod rather than at boot, there is
 now a race: the `nvidia` driver exists on the host and may claim a GPU before the controller binds
@@ -570,63 +907,27 @@ options vfio-pci ids=10de:2335
 softdep nvidia pre: vfio-pci
 ```
 
-`10de:2335` is the H200 SXM PCI ID. Do **not** add the NVSwitch ID here — the switches must be
-claimed by `nvidia`.
+`10de:2335` is the H200 SXM PCI ID. Do **not** add the NVSwitch ID — the switches must be claimed by
+`nvidia`.
 
 ---
 
-## Part 11 — Verification
+# References
 
-### Host
-
-```bash
-systemctl status nvidia-fabricmanager-local     # active (running)
-pgrep -af nv-fabricmanager                      # note: plain pgrep fails, name > 15 chars
-tail -30 /usr/local/nvidia/fabricmanager.log
-journalctl -u nvidia-fabricmanager-local -n 30
-lspci -k -s <nvswitch-bdf>    # nvidia-nvswitch
-lspci -k -s <gpu-bdf>         # vfio-pci
-```
-
-### Guest VM
-
-Power-cycle the VM after FM starts. The fabric handshake happens during GPU driver init inside the
-guest and is not retried, so a VM that was already running will stay broken.
-
-```bash
-nvidia-smi -q | grep -A5 Fabric
-```
-
-```
-Fabric
-    State   : Completed
-    Status  : Success
-    CliqueId: 0
-```
-
-```bash
-python -c "import torch; print(torch.cuda.is_available())"     # True
-python -c "import torch; print(torch.__version__, torch.version.cuda)"
-```
-
----
-
-## Part 12 — References
-
-### NVIDIA — fabric, drivers, concepts
+## NVIDIA — fabric, drivers, concepts
 
 | Link | What it covers |
 |---|---|
 | [Fabric Manager User Guide](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html) | The primary reference. Deployment models, config options, service management, SDK, high-availability modes, troubleshooting |
-| [Fabric Manager — full contents](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/contents.html) | Section-level index; useful for jumping straight to config options or a specific virtualisation model |
+| [Fabric Manager — full contents](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/contents.html) | Section-level index; jump straight to config options or a specific virtualisation model |
 | [HGX Software User Guide](https://docs.nvidia.com/datacenter/tesla/hgx-software-guide/index.html) | The complete HGX software stack and install order — driver, NSCQ, FM, DCGM |
 | [NVLink & NVSwitch product page](https://www.nvidia.com/en-us/data-center/nvlink/) | Conceptual overview of the interconnect and switch generations |
-| [Fabric Manager Client (GitHub)](https://github.com/NVIDIA/Fabric-Manager-Client) | Partition management CLI for Shared NVSwitch mode; also documents `FABRIC_MODE_RESTART` and resiliency |
+| [Fabric Manager Client (GitHub)](https://github.com/NVIDIA/Fabric-Manager-Client) | Partition management CLI for Shared NVSwitch mode; documents `FABRIC_MODE_RESTART` and resiliency |
 | [NVIDIA open GPU kernel modules (GitHub)](https://github.com/NVIDIA/open-gpu-kernel-modules) | Source of the open driver — the same code SUSE builds into the KMP used here |
 | [NVIDIA CUDA repo for SLES](https://developer.download.nvidia.com/compute/cuda/repos/sles15/x86_64/) | Where `nvidia-fabricmanager` and `libnvidia-nscq` come from |
 | [NVIDIA driver repo for SLE](https://download.nvidia.com/suse/sle16/) | Userspace and meta-packages (note: **not** the kernel module) |
 
-### Harvester / KubeVirt / immutable OS
+## Harvester / KubeVirt / immutable OS
 
 | Link | What it covers |
 |---|---|
@@ -637,7 +938,7 @@ python -c "import torch; print(torch.__version__, torch.version.cuda)"
 | [KubeVirt host devices / GPU assignment](https://kubevirt.io/user-guide/compute/host-devices/) | `permittedHostDevices`, resource names, VM spec syntax |
 | [SUSE Linux Micro documentation](https://documentation.suse.com/sl-micro/6.2/) | Transactional/immutable OS model underlying the Harvester host |
 
-### VFIO, IOMMU and passthrough background
+## VFIO, IOMMU and passthrough background
 
 | Link | What it covers |
 |---|---|
@@ -648,21 +949,30 @@ python -c "import torch; print(torch.__version__, torch.version.cuda)"
 
 ---
 
-## Part 13 — Summary
+# Summary
 
 | | |
 |---|---|
 | **Symptom** | CUDA Error 802 in guest; `Fabric State: In Progress` |
 | **Root cause** | NVSwitch bridges on the host had no kernel driver bound |
 | **Why it was hard** | Immutable read-only root; no package installation possible; no vGPU licence; air-gapped; the real kernel module ships from the distro repo, not NVIDIA's |
-| **Solution** | Extract RPMs into the writable persistent partition, `insmod` the module directly, create NVSwitch device nodes manually, run FM from the extracted tree |
+| **Solution** | Extract RPMs into the writable persistent partition, `insmod` the module directly, create NVSwitch device nodes manually, run Fabric Manager from the extracted tree |
 | **Result** | `Fabric State: Completed`; `torch.cuda.is_available() == True` |
 
-### Lessons worth carrying forward
+## Lessons worth carrying forward
 
 - Fabric Manager belongs on the **host**, never in a passthrough guest.
-- FM alone is useless — it needs the kernel driver and NSCQ, all at the **same version**.
+- Fabric Manager alone is useless — it needs the kernel driver and NSCQ, all at the **same version**.
 - On SUSE, NVIDIA ships meta-packages; the **distribution** ships the signed kernel module.
 - Version selection is a three-way intersection: driver repo × CUDA repo × kernel ABI.
 - An immutable root blocks *installation*, not *module loading*. `insmod` takes a path.
 - NVSwitch (NVLink fabric) and PCIe switches (IOMMU grouping) are different problems. Don't conflate them.
+
+## The five-line version
+
+1. NVSwitch is a junction that lets eight GPUs talk at full speed. It belongs to the **host**, never a VM.
+2. Fabric Manager is the policeman who programs that junction. It runs on the **host**, never in a VM.
+3. The junction had **no driver at all** — so Fabric Manager could never run, so every VM's CUDA was dead.
+4. The OS refused all installation, so the driver was **unzipped into the one writable folder and
+   loaded by hand.**
+5. Fabric Manager started, the fabric came up, and CUDA returned `True`.
\ No newline at end of file
diff --git a/tools/check_post_format.rb b/tools/check_post_format.rb
new file mode 100644
index 0000000..4ff219d
--- /dev/null
+++ b/tools/check_post_format.rb
@@ -0,0 +1,100 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+#
+# Validates every post in _posts/ against this repo's conventions:
+#   - filename matches YYYY-MM-DD-slug.md
+#   - lives in a _posts/YYYY-MM-DD/ folder matching that same date
+#   - has valid YAML front matter with title, date, categories, tags
+#   - categories/tags are non-empty arrays
+#
+# Usage: ruby tools/check_post_format.rb
+
+require "yaml"
+require "date"
+
+POSTS_DIR = File.join(__dir__, "..", "_posts")
+FILENAME_RE = /\A(\d{4}-\d{2}-\d{2})-.+\.md\z/
+
+errors = []
+checked = 0
+
+Dir.glob(File.join(POSTS_DIR, "**", "*.md")).sort.each do |path|
+  checked += 1
+  rel = path.sub("#{POSTS_DIR}/", "")
+  filename = File.basename(path)
+  folder = File.basename(File.dirname(path))
+
+  # filename convention
+  match = FILENAME_RE.match(filename)
+  unless match
+    errors << "#{rel}: filename must match YYYY-MM-DD-slug.md"
+    next
+  end
+
+  file_date = match[1]
+
+  # folder convention
+  if folder != file_date
+    errors << "#{rel}: lives in folder '#{folder}/' but filename date is '#{file_date}' — should be _posts/#{file_date}/#{filename}"
+  end
+
+  content = File.read(path)
+
+  unless content.start_with?("---\n") || content.start_with?("---\r\n")
+    errors << "#{rel}: missing front matter (must start with '---')"
+    next
+  end
+
+  parts = content.split(/^---\s*$/, 3)
+  if parts.length < 3
+    errors << "#{rel}: front matter block is not closed with a second '---'"
+    next
+  end
+
+  front_matter = begin
+    YAML.safe_load(parts[1], permitted_classes: [Date, Time])
+  rescue Psych::SyntaxError => e
+    errors << "#{rel}: invalid YAML in front matter — #{e.message}"
+    next
+  end
+
+  unless front_matter.is_a?(Hash)
+    errors << "#{rel}: front matter did not parse to a set of key/value pairs"
+    next
+  end
+
+  # required fields
+  title = front_matter["title"]
+  if title.nil? || !title.is_a?(String) || title.strip.empty?
+    errors << "#{rel}: missing or empty 'title'"
+  end
+
+  date = front_matter["date"]
+  if date.nil?
+    errors << "#{rel}: missing 'date'"
+  end
+
+  categories = front_matter["categories"]
+  if categories.nil? || !categories.is_a?(Array) || categories.empty?
+    errors << "#{rel}: 'categories' must be a non-empty list, e.g. [Category Name]"
+  end
+
+  tags = front_matter["tags"]
+  if tags.nil? || !tags.is_a?(Array) || tags.empty?
+    errors << "#{rel}: 'tags' must be a non-empty list, e.g. [tag-one, tag-two]"
+  end
+end
+
+if checked.zero?
+  puts "No posts found under #{POSTS_DIR} — nothing to check."
+  exit 0
+end
+
+if errors.empty?
+  puts "All #{checked} post(s) passed format checks."
+  exit 0
+else
+  puts "#{errors.size} problem(s) found across #{checked} post(s):"
+  errors.each { |e| puts "  ✗ #{e}" }
+  exit 1
+end

From 2249e65e0fe8d2df1c006654b0c23d95cf9d6dd1 Mon Sep 17 00:00:00 2001
From: rishabh <rishank69@gmail.com>
Date: Sat, 15 Aug 2026 22:58:49 +0800
Subject: [PATCH 2/3] Self-host theme static assets instead of CDN

- Add assets/lib as a proper git submodule (chirpy-static-assets),
  replacing the plain untracked clone used for local testing
- Enable assets.self_host in _config.yml so FontAwesome, Day.js,
  GLightbox, MathJax, etc. serve from this domain instead of jsdelivr
- Checkout with submodules: true in both GitHub Actions workflows so
  CI/deploy actually pulls the submodule content
- Stop excluding assets/lib in .dockerignore so local Docker builds
  include it too
- Document cloning with --recurse-submodules in README.md

Note: Bootstrap's CSS/JS is hardcoded to jsdelivr separately in the
theme and isn't covered by this toggle.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---
 .dockerignore                      |  1 -
 .github/workflows/pages-deploy.yml |  1 +
 .github/workflows/pr-build.yaml    |  1 +
 README.md                          | 15 +++++++++++++++
 _config.yml                        |  2 +-
 assets/lib                         |  1 +
 6 files changed, 19 insertions(+), 2 deletions(-)
 create mode 160000 assets/lib

diff --git a/.dockerignore b/.dockerignore
index 5aa2070..3b96552 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -6,4 +6,3 @@ vendor
 .bundle
 Gemfile.lock
 node_modules
-assets/lib
diff --git a/.github/workflows/pages-deploy.yml b/.github/workflows/pages-deploy.yml
index 4283b03..0e5dc66 100644
--- a/.github/workflows/pages-deploy.yml
+++ b/.github/workflows/pages-deploy.yml
@@ -29,6 +29,7 @@ jobs:
         uses: actions/checkout@v4
         with:
           fetch-depth: 0
+          submodules: true
 
       - name: Setup Pages
         id: pages
diff --git a/.github/workflows/pr-build.yaml b/.github/workflows/pr-build.yaml
index da14dca..e79d37e 100644
--- a/.github/workflows/pr-build.yaml
+++ b/.github/workflows/pr-build.yaml
@@ -15,6 +15,7 @@ jobs:
         uses: actions/checkout@v4
         with:
           fetch-depth: 0
+          submodules: true
 
       - name: Setup Ruby
         uses: ruby/setup-ruby@v1
diff --git a/README.md b/README.md
index 113f929..5c79ac6 100644
--- a/README.md
+++ b/README.md
@@ -10,6 +10,21 @@ Built with [Jekyll](https://jekyllrb.com) + [Chirpy](https://github.com/cotes202
 
 Requires [Docker](https://www.docker.com/) — no local Ruby/Jekyll install needed.
 
+The theme's static assets (fonts, JS libs) are self-hosted via a git submodule
+([`assets/lib`](assets/lib)), so clone with `--recurse-submodules`:
+
+```bash
+git clone --recurse-submodules https://github.com/allexistence/allexistence.github.io.git
+```
+
+If you already cloned without it:
+
+```bash
+git submodule update --init
+```
+
+Then:
+
 ```bash
 docker compose up
 ```
diff --git a/_config.yml b/_config.yml
index 0d6b340..32845ca 100644
--- a/_config.yml
+++ b/_config.yml
@@ -113,7 +113,7 @@ comments:
 # Self-hosted static assets, optional › https://github.com/cotes2020/chirpy-static-assets
 assets:
   self_host:
-    enabled: # boolean, keep empty means false
+    enabled: true # boolean, keep empty means false
     # specify the Jekyll environment, empty means both
     # only works if `assets.self_host.enabled` is 'true'
     env: # [development | production]
diff --git a/assets/lib b/assets/lib
new file mode 160000
index 0000000..5cde3f0
--- /dev/null
+++ b/assets/lib
@@ -0,0 +1 @@
+Subproject commit 5cde3f0076b62e45fc68291893cf7d93e122adb7

From 5cdd77f2c3895f90874691a3fbbf8d4bceebf4f5 Mon Sep 17 00:00:00 2001
From: rishabh <rishank69@gmail.com>
Date: Sat, 15 Aug 2026 23:21:07 +0800
Subject: [PATCH 3/3] Improving kubevirt FM notes

---
 ...c-manager-air-gapped-harvester-hgx-h200.md | 26 ++++++++++++-------
 1 file changed, 16 insertions(+), 10 deletions(-)

diff --git a/_posts/2026-08-12/2026-08-12-installing-nvidia-fabric-manager-air-gapped-harvester-hgx-h200.md b/_posts/2026-08-12/2026-08-12-installing-nvidia-fabric-manager-air-gapped-harvester-hgx-h200.md
index 2c3e25e..35f7934 100644
--- a/_posts/2026-08-12/2026-08-12-installing-nvidia-fabric-manager-air-gapped-harvester-hgx-h200.md
+++ b/_posts/2026-08-12/2026-08-12-installing-nvidia-fabric-manager-air-gapped-harvester-hgx-h200.md
@@ -687,20 +687,26 @@ picks the right one automatically once the folder path is correct.
 
 | Value | Meaning |
 |---|---|
-| `0` | Bare metal — FM expects to see the GPUs itself |
-| `1` | Shared NVSwitch — FM configures switches, then **waits for a hypervisor service to call its partition-activation API** |
+| `0` | Bare metal — FM assumes one owner for the whole baseboard |
+| `1` | Shared NVSwitch — FM configures the switches, then **waits for the hypervisor to call its partition-activation API** as VMs come and go |
 
-By the book, this setup looks like mode `1`. But mode `1` requires the virtualisation platform to
-actively call Fabric Manager's partition API — and KubeVirt-based platforms have no such feature.
-That would have been a wall with no way around it.
+By the book, this setup is mode `1`: GPUs split across separate VMs is exactly what Shared NVSwitch
+is for. The problem is that mode `1` only completes if the virtualisation platform actively calls
+Fabric Manager's partition API — and KubeVirt has no such integration. Mode `1` would have been a
+wall with nothing on the other side.
 
-**Mode `0` worked anyway**, even with the GPUs vfio-bound and invisible to the host driver. The
+**Mode `0` worked anyway**, even with every GPU vfio-bound and invisible to the host driver. The
 switches initialise, and each GPU registers with the fabric when its driver starts inside its guest.
 
-> ⚠️ Worth testing before trusting this generally: mode `1` exists to *enforce isolation* between
-> tenants. Mode `0` may be leaving the fabric open rather than partitioned. Fine for a single tenant;
-> check `nvidia-smi topo -m` and a peer-to-peer bandwidth test before relying on it for multi-tenant
-> work.
+> ⚠️ **What I tested, and what I didn't.**
+> Mode `0` brought the fabric up and CUDA returned `True` — that part is reproducible.
+> What I have *not* verified is isolation. Mode `1` exists specifically to partition the fabric
+> between tenants; mode `0` makes no such claim, and may well leave NVLink routing open across all
+> eight GPUs regardless of which VM holds them.
+>
+> For a single-tenant box this probably doesn't matter. For separately-owned workloads sharing one
+> baseboard, check `nvidia-smi topo -m` and run a peer-to-peer bandwidth test **between GPUs in
+> different VMs** before trusting it. If you've done that test, I'd like to hear the result.
 
 ---