Guard the notebook cache: pin Pluto, fail fast on drift, document it

.gitattributes

@@ -1,2 +1,16 @@
 page/* linguist-vendored
-* text=auto
+* text=auto
+
+# Notebooks are keyed into _cache/ by sha256 of their exact bytes, so a CRLF
+# checkout (git's default on Windows, since `text=auto` above means the eol
+# attribute falls back to core.eol=native) changes every hash and misses every
+# cache entry, cold-running the whole site. Pin them to LF everywhere.
+# All 209 tracked .jl files are already LF in the index, so this renormalizes
+# nothing and invalidates no cache entry.
+*.jl text eol=lf
+
+# Cached notebook states are binary. git already detects that from content
+# today, but that is a heuristic over the first 8000 bytes: a mostly-ASCII
+# state could flip to "text" and get line-ending mangled on checkout, silently
+# corrupting the cache. Say it explicitly instead of relying on the guess.
+*.plutostate binary

.github/workflows/ExportNotebooks.yml

@@ -12,7 +12,21 @@ concurrency:
     cancel-in-progress: true
 
 jobs:
+    # Cheap gate on an expensive job: a cache that cannot serve this build means
+    # cold-running every notebook, which does not fit in the 6h job limit. Find
+    # out in seconds rather than after six hours of runner time.
+    cache-preflight:
+        runs-on: ubuntu-latest
+        timeout-minutes: 5
+        steps:
+            - name: Checkout source
+              uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+
+            - name: 🔎 Check notebook cache against src/ and the pinned Pluto version
+              run: python3 tools/check_cache.py
+
     build-and-deploy:
+        needs: cache-preflight
         runs-on: ubuntu-latest
         timeout-minutes: 360
         steps:

.gitignore

@@ -697,3 +697,8 @@ Manifest.toml
 *.jpg
 *.svg
 # _cache is on purpose omitted to avoid again cold run of github actions that results in timeout
+
+# ...but the deployment Manifest must stay tracked: it is the only thing that
+# pins Pluto to the version baked into the _cache/*.plutostate filenames.
+# Keep this last - .gitignore applies the last matching rule.
+!pluto-deployment-environment/Manifest.toml

pluto-deployment-environment/Project.toml

@@ -37,3 +37,12 @@ URIs = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4"
 Unicode = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"
 Unitful = "1986cc42-f94f-5a68-af5c-568840ba703d"
 YAML = "ddb6d928-2868-570f-bddf-ab3f9cf99eb6"
+
+[compat]
+# EXACT pins, on purpose. The cached notebook states in _cache/ are named
+# "<pluto version><content hash>.plutostate" (see PlutoSliderServer's
+# Export.jl), so bumping Pluto invalidates all of them at once and the next
+# build cold-runs every notebook - which exceeds GitHub's 6h job limit.
+# Bumping these is a deliberate act: see website_maintenance.md first.
+Pluto = "=0.20.13"
+PlutoSliderServer = "=1.4.0"

tools/check_cache.py

@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+"""Check that _cache/ matches the notebooks in src/ and the pinned Pluto version.
+
+Run before the expensive export job so a stale cache fails in seconds instead
+of after a six-hour build that gets killed by the GitHub job limit.
+
+The cached notebook states are named by PlutoSliderServer as
+
+    <pluto version><content hash>.plutostate      e.g. 0_20_13AbC...xyz.plutostate
+
+    src/Export.jl:56    escapeuri(string(pluto_version, hash)), "." => "_"
+    src/PlutoHash.jl    plutohash = base64urlencode . sha256   (of the file bytes)
+
+Both halves matter:
+
+  * content hash -- edit a notebook and only that notebook re-runs. Normal, cheap.
+  * pluto version -- bump Pluto and EVERY entry is invalidated at once, so the
+    next build cold-runs all notebooks. That does not fit in 6h. This is the
+    case the script hard-fails on.
+
+Deliberately plain Python: no Julia, no Pkg.instantiate, runs in ~2 seconds.
+Keep it in sync with PlutoSliderServer if that naming ever changes.
+"""
+
+import base64
+import fnmatch
+import hashlib
+import pathlib
+import re
+import sys
+
+try:
+    import tomllib  # stdlib since Python 3.11; ubuntu-latest runners ship 3.12+
+except ModuleNotFoundError:
+    sys.exit(f"FAIL: needs Python 3.11+ for tomllib, got {sys.version.split()[0]}")
+
+ROOT = pathlib.Path(__file__).resolve().parent.parent
+MANIFEST = ROOT / "pluto-deployment-environment" / "Manifest.toml"
+DEPLOY_TOML = ROOT / "pluto-deployment-environment" / "PlutoDeployment.toml"
+SRC = ROOT / "src"
+CACHE = ROOT / "_cache"
+
+PLUTO_HEADER = b"### A Pluto.jl notebook ###"
+
+
+def pluto_version() -> str:
+    """Read the exact Pluto version the Manifest pins."""
+    text = MANIFEST.read_text()
+    m = re.search(r'^\[\[deps\.Pluto\]\]$(?:(?!^\[\[).)*?^version = "([^"]+)"',
+                  text, re.S | re.M)
+    if not m:
+        sys.exit(f"FAIL: no Pluto version found in {MANIFEST.relative_to(ROOT)}")
+    return m.group(1)
+
+
+def ignored_globs() -> list[str]:
+    """Notebooks PlutoDeployment.toml tells PlutoSliderServer not to cache."""
+    if not DEPLOY_TOML.is_file():
+        return []
+    with DEPLOY_TOML.open("rb") as f:
+        return tomllib.load(f).get("Export", {}).get("ignore_cache", [])
+
+
+def is_notebook(path: pathlib.Path) -> bool:
+    """Same test PlutoPages uses: Pluto.is_pluto_notebook, i.e. the header."""
+    with path.open("rb") as f:
+        return f.read(len(PLUTO_HEADER)) == PLUTO_HEADER
+
+
+def statefile(path: pathlib.Path, prefix: str) -> str:
+    digest = hashlib.sha256(path.read_bytes()).digest()
+    h = base64.urlsafe_b64encode(digest).decode().rstrip("=")
+    return f"{prefix}{h}.plutostate"
+
+
+def main() -> int:
+    version = pluto_version()
+    prefix = version.replace(".", "_")
+    skip = ignored_globs()
+
+    notebooks = [
+        p for p in sorted(SRC.rglob("*.jl"))
+        if is_notebook(p)
+        and not any(fnmatch.fnmatch(str(p.relative_to(SRC)), g)
+                    or fnmatch.fnmatch(p.name, g) for g in skip)
+    ]
+    cache = {p.name for p in CACHE.glob("*.plutostate")} if CACHE.is_dir() else set()
+    want = {statefile(p, prefix): p for p in notebooks}
+
+    missing = {fn: p for fn, p in want.items() if fn not in cache}
+    orphans = sorted(cache - want.keys())
+    wrong_version = [fn for fn in cache if not fn.startswith(prefix)]
+
+    print(f"Pluto {version} | {len(notebooks)} notebooks | {len(cache)} cache entries")
+
+    # Total invalidation: a Pluto bump, or an empty cache. Unrecoverable in CI.
+    if cache and len(wrong_version) == len(cache):
+        sys.exit(
+            f"\nFAIL: every cache entry was built by a different Pluto version.\n"
+            f"  Manifest pins Pluto {version}, so entries must start with '{prefix}'\n"
+            f"  but _cache/ holds e.g. {sorted(wrong_version)[0]}\n\n"
+            f"  Cold-running all {len(notebooks)} notebooks exceeds the 6h job limit.\n"
+            f"  Either revert the Pluto bump, or regenerate the cache locally:\n"
+            f"      rm -rf _cache && julia --project=pluto-deployment-environment generate.jl\n"
+            f"  and commit the result. See website_maintenance.md."
+        )
+    if not cache:
+        sys.exit(
+            f"\nFAIL: _cache/ is empty or missing.\n"
+            f"  Cold-running all {len(notebooks)} notebooks exceeds the 6h job limit.\n"
+            f"  See website_maintenance.md for how to regenerate it."
+        )
+
+    for fn in orphans:
+        print(f"  orphan  {fn}")
+    for fn, p in sorted(missing.items(), key=lambda kv: str(kv[1])):
+        print(f"  cold    {p.relative_to(ROOT)}")
+
+    if orphans:
+        print(f"\n{len(orphans)} orphaned entr{'y' if len(orphans) == 1 else 'ies'} "
+              f"(notebook edited or deleted). Safe to delete; do it with the next commit "
+              f"so _cache/ does not grow without bound.")
+    if missing:
+        print(f"\n{len(missing)} notebook(s) will run cold this build. That is expected "
+              f"after editing a notebook. Commit the regenerated _cache/ entries to keep "
+              f"later builds fast.")
+    if not missing and not orphans:
+        print("\nCache is complete and current.")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

website_maintenance.md

@@ -36,7 +36,74 @@ Pluto notebooks will be rendered to HTML and included in the page. What you see
 
 On a separate system, we are running a PlutoSliderServer that is synchronized to the `Fall23` brach. This makes our notebooks interactive!
 
-Notebook outputs are **cached** (for a long time) by the file hash. This means that a notebook file will only ever run once, which makes it much faster to work on the website. If you need to re-run your notebook, add a space somewhere in the code :)
+Notebook outputs are **cached** (for a long time) by the file hash *and the Pluto version*. This means that a notebook file will only ever run once, which makes it much faster to work on the website. If you need to re-run your notebook, add a space somewhere in the code :)
+
+See [The notebook cache](#the-notebook-cache-_cache) below before you touch the Pluto version.
+
+## The notebook cache (`_cache/`)
+
+Running all notebooks from scratch takes well over six hours, which is more than a
+GitHub Actions job is allowed to run. The build only finishes because `_cache/` is
+committed to the repository, so every checkout starts warm. Treat that directory as
+build infrastructure, not as generated junk.
+
+Each file is named by PlutoSliderServer as `<pluto version><content hash>.plutostate`,
+for example `0_20_13AbC...xyz.plutostate`. Both halves matter, and they fail very
+differently.
+
+### Editing a notebook: nothing to worry about
+
+The content hash changes, so CI re-runs that one notebook (a few minutes) and
+everything else is served from cache. The old `.plutostate` is left behind as an
+orphan. Regenerate locally and commit the new cache entries, deleting the orphans, so
+the directory does not grow without bound:
+
+```
+julia --project=pluto-deployment-environment generate.jl
+python3 tools/check_cache.py     # lists orphans and notebooks that would run cold
+```
+
+### ⚠️ Bumping Pluto: read this first
+
+A new Pluto version invalidates **every** entry at once, because the version is part
+of every filename. The next build then cold-runs all notebooks, exceeds the job
+limit, and gets killed. That is how `main` broke before PR #81, and it does not
+recover on its own.
+
+`Pluto` and `PlutoSliderServer` are therefore pinned with `=` in
+`pluto-deployment-environment/Project.toml`, so `Pkg.update()` cannot do this by
+accident. To bump on purpose:
+
+1. Edit the `[compat]` pins, then `Pkg.update()` in that environment.
+2. Locally: `rm -rf _cache && julia --project=pluto-deployment-environment generate.jl`.
+   Budget several hours.
+3. Commit the regenerated `_cache/` **in the same commit** as the `Manifest.toml`
+   change. The two must never be out of step.
+4. `python3 tools/check_cache.py` must pass before you push.
+
+`pluto-deployment-environment/Manifest.toml` is the only thing that pins Pluto, so it
+is force-tracked in `.gitignore`. Do not untrack it.
+
+### The inverse problem: stale output
+
+Because the cache key is the notebook's *content*, bumping any other package
+(ModelingToolkit, Turing, ...) does **not** invalidate anything. The site keeps
+serving output produced by the old versions until a notebook is edited. After a
+significant SciML upgrade, force a full refresh by hand:
+
+```
+rm -rf _cache && julia --project=pluto-deployment-environment generate.jl
+```
+
+This also proves every notebook still runs, which a warm build never checks. Worth
+doing once a term.
+
+### The preflight check
+
+`ExportNotebooks.yml` runs `tools/check_cache.py` as a separate `cache-preflight`
+job before the expensive one. It fails in seconds when the cache cannot serve the
+build (Pluto bump, empty `_cache/`) instead of after six hours of runner time. It
+only warns for the normal cases, edited notebooks and orphans. Run it locally too.
 
 ## `.css`, `.html`, `.gif`, etc