diff --git a/.gitattributes b/.gitattributes index bb05dd6c..19b79535 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,2 +1,16 @@ page/* linguist-vendored -* text=auto \ No newline at end of file +* text=auto + +# Notebooks are keyed into _cache/ by sha256 of their exact bytes, so a CRLF +# checkout (git's default on Windows, since `text=auto` above means the eol +# attribute falls back to core.eol=native) changes every hash and misses every +# cache entry, cold-running the whole site. Pin them to LF everywhere. +# All 209 tracked .jl files are already LF in the index, so this renormalizes +# nothing and invalidates no cache entry. +*.jl text eol=lf + +# Cached notebook states are binary. git already detects that from content +# today, but that is a heuristic over the first 8000 bytes: a mostly-ASCII +# state could flip to "text" and get line-ending mangled on checkout, silently +# corrupting the cache. Say it explicitly instead of relying on the guess. +*.plutostate binary diff --git a/.github/workflows/ExportNotebooks.yml b/.github/workflows/ExportNotebooks.yml index 73f29192..cd15a0b5 100644 --- a/.github/workflows/ExportNotebooks.yml +++ b/.github/workflows/ExportNotebooks.yml @@ -12,7 +12,21 @@ concurrency: cancel-in-progress: true jobs: + # Cheap gate on an expensive job: a cache that cannot serve this build means + # cold-running every notebook, which does not fit in the 6h job limit. Find + # out in seconds rather than after six hours of runner time. + cache-preflight: + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - name: Checkout source + uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 + + - name: 🔎 Check notebook cache against src/ and the pinned Pluto version + run: python3 tools/check_cache.py + build-and-deploy: + needs: cache-preflight runs-on: ubuntu-latest timeout-minutes: 360 steps: diff --git a/.gitignore b/.gitignore index 5976268b..ea0d29da 100644 --- a/.gitignore +++ b/.gitignore @@ -697,3 +697,8 @@ Manifest.toml *.jpg *.svg # _cache is on purpose omitted to avoid again cold run of github actions that results in timeout + +# ...but the deployment Manifest must stay tracked: it is the only thing that +# pins Pluto to the version baked into the _cache/*.plutostate filenames. +# Keep this last - .gitignore applies the last matching rule. +!pluto-deployment-environment/Manifest.toml diff --git a/pluto-deployment-environment/Project.toml b/pluto-deployment-environment/Project.toml index dffa0591..789c1cb3 100644 --- a/pluto-deployment-environment/Project.toml +++ b/pluto-deployment-environment/Project.toml @@ -37,3 +37,12 @@ URIs = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4" Unicode = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" Unitful = "1986cc42-f94f-5a68-af5c-568840ba703d" YAML = "ddb6d928-2868-570f-bddf-ab3f9cf99eb6" + +[compat] +# EXACT pins, on purpose. The cached notebook states in _cache/ are named +# ".plutostate" (see PlutoSliderServer's +# Export.jl), so bumping Pluto invalidates all of them at once and the next +# build cold-runs every notebook - which exceeds GitHub's 6h job limit. +# Bumping these is a deliberate act: see website_maintenance.md first. +Pluto = "=0.20.13" +PlutoSliderServer = "=1.4.0" diff --git a/tools/check_cache.py b/tools/check_cache.py new file mode 100755 index 00000000..fbe19384 --- /dev/null +++ b/tools/check_cache.py @@ -0,0 +1,133 @@ +#!/usr/bin/env python3 +"""Check that _cache/ matches the notebooks in src/ and the pinned Pluto version. + +Run before the expensive export job so a stale cache fails in seconds instead +of after a six-hour build that gets killed by the GitHub job limit. + +The cached notebook states are named by PlutoSliderServer as + + .plutostate e.g. 0_20_13AbC...xyz.plutostate + + src/Export.jl:56 escapeuri(string(pluto_version, hash)), "." => "_" + src/PlutoHash.jl plutohash = base64urlencode . sha256 (of the file bytes) + +Both halves matter: + + * content hash -- edit a notebook and only that notebook re-runs. Normal, cheap. + * pluto version -- bump Pluto and EVERY entry is invalidated at once, so the + next build cold-runs all notebooks. That does not fit in 6h. This is the + case the script hard-fails on. + +Deliberately plain Python: no Julia, no Pkg.instantiate, runs in ~2 seconds. +Keep it in sync with PlutoSliderServer if that naming ever changes. +""" + +import base64 +import fnmatch +import hashlib +import pathlib +import re +import sys + +try: + import tomllib # stdlib since Python 3.11; ubuntu-latest runners ship 3.12+ +except ModuleNotFoundError: + sys.exit(f"FAIL: needs Python 3.11+ for tomllib, got {sys.version.split()[0]}") + +ROOT = pathlib.Path(__file__).resolve().parent.parent +MANIFEST = ROOT / "pluto-deployment-environment" / "Manifest.toml" +DEPLOY_TOML = ROOT / "pluto-deployment-environment" / "PlutoDeployment.toml" +SRC = ROOT / "src" +CACHE = ROOT / "_cache" + +PLUTO_HEADER = b"### A Pluto.jl notebook ###" + + +def pluto_version() -> str: + """Read the exact Pluto version the Manifest pins.""" + text = MANIFEST.read_text() + m = re.search(r'^\[\[deps\.Pluto\]\]$(?:(?!^\[\[).)*?^version = "([^"]+)"', + text, re.S | re.M) + if not m: + sys.exit(f"FAIL: no Pluto version found in {MANIFEST.relative_to(ROOT)}") + return m.group(1) + + +def ignored_globs() -> list[str]: + """Notebooks PlutoDeployment.toml tells PlutoSliderServer not to cache.""" + if not DEPLOY_TOML.is_file(): + return [] + with DEPLOY_TOML.open("rb") as f: + return tomllib.load(f).get("Export", {}).get("ignore_cache", []) + + +def is_notebook(path: pathlib.Path) -> bool: + """Same test PlutoPages uses: Pluto.is_pluto_notebook, i.e. the header.""" + with path.open("rb") as f: + return f.read(len(PLUTO_HEADER)) == PLUTO_HEADER + + +def statefile(path: pathlib.Path, prefix: str) -> str: + digest = hashlib.sha256(path.read_bytes()).digest() + h = base64.urlsafe_b64encode(digest).decode().rstrip("=") + return f"{prefix}{h}.plutostate" + + +def main() -> int: + version = pluto_version() + prefix = version.replace(".", "_") + skip = ignored_globs() + + notebooks = [ + p for p in sorted(SRC.rglob("*.jl")) + if is_notebook(p) + and not any(fnmatch.fnmatch(str(p.relative_to(SRC)), g) + or fnmatch.fnmatch(p.name, g) for g in skip) + ] + cache = {p.name for p in CACHE.glob("*.plutostate")} if CACHE.is_dir() else set() + want = {statefile(p, prefix): p for p in notebooks} + + missing = {fn: p for fn, p in want.items() if fn not in cache} + orphans = sorted(cache - want.keys()) + wrong_version = [fn for fn in cache if not fn.startswith(prefix)] + + print(f"Pluto {version} | {len(notebooks)} notebooks | {len(cache)} cache entries") + + # Total invalidation: a Pluto bump, or an empty cache. Unrecoverable in CI. + if cache and len(wrong_version) == len(cache): + sys.exit( + f"\nFAIL: every cache entry was built by a different Pluto version.\n" + f" Manifest pins Pluto {version}, so entries must start with '{prefix}'\n" + f" but _cache/ holds e.g. {sorted(wrong_version)[0]}\n\n" + f" Cold-running all {len(notebooks)} notebooks exceeds the 6h job limit.\n" + f" Either revert the Pluto bump, or regenerate the cache locally:\n" + f" rm -rf _cache && julia --project=pluto-deployment-environment generate.jl\n" + f" and commit the result. See website_maintenance.md." + ) + if not cache: + sys.exit( + f"\nFAIL: _cache/ is empty or missing.\n" + f" Cold-running all {len(notebooks)} notebooks exceeds the 6h job limit.\n" + f" See website_maintenance.md for how to regenerate it." + ) + + for fn in orphans: + print(f" orphan {fn}") + for fn, p in sorted(missing.items(), key=lambda kv: str(kv[1])): + print(f" cold {p.relative_to(ROOT)}") + + if orphans: + print(f"\n{len(orphans)} orphaned entr{'y' if len(orphans) == 1 else 'ies'} " + f"(notebook edited or deleted). Safe to delete; do it with the next commit " + f"so _cache/ does not grow without bound.") + if missing: + print(f"\n{len(missing)} notebook(s) will run cold this build. That is expected " + f"after editing a notebook. Commit the regenerated _cache/ entries to keep " + f"later builds fast.") + if not missing and not orphans: + print("\nCache is complete and current.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/website_maintenance.md b/website_maintenance.md index cb3ddadf..f919fedb 100644 --- a/website_maintenance.md +++ b/website_maintenance.md @@ -36,7 +36,74 @@ Pluto notebooks will be rendered to HTML and included in the page. What you see On a separate system, we are running a PlutoSliderServer that is synchronized to the `Fall23` brach. This makes our notebooks interactive! -Notebook outputs are **cached** (for a long time) by the file hash. This means that a notebook file will only ever run once, which makes it much faster to work on the website. If you need to re-run your notebook, add a space somewhere in the code :) +Notebook outputs are **cached** (for a long time) by the file hash *and the Pluto version*. This means that a notebook file will only ever run once, which makes it much faster to work on the website. If you need to re-run your notebook, add a space somewhere in the code :) + +See [The notebook cache](#the-notebook-cache-_cache) below before you touch the Pluto version. + +## The notebook cache (`_cache/`) + +Running all notebooks from scratch takes well over six hours, which is more than a +GitHub Actions job is allowed to run. The build only finishes because `_cache/` is +committed to the repository, so every checkout starts warm. Treat that directory as +build infrastructure, not as generated junk. + +Each file is named by PlutoSliderServer as `.plutostate`, +for example `0_20_13AbC...xyz.plutostate`. Both halves matter, and they fail very +differently. + +### Editing a notebook: nothing to worry about + +The content hash changes, so CI re-runs that one notebook (a few minutes) and +everything else is served from cache. The old `.plutostate` is left behind as an +orphan. Regenerate locally and commit the new cache entries, deleting the orphans, so +the directory does not grow without bound: + +``` +julia --project=pluto-deployment-environment generate.jl +python3 tools/check_cache.py # lists orphans and notebooks that would run cold +``` + +### ⚠️ Bumping Pluto: read this first + +A new Pluto version invalidates **every** entry at once, because the version is part +of every filename. The next build then cold-runs all notebooks, exceeds the job +limit, and gets killed. That is how `main` broke before PR #81, and it does not +recover on its own. + +`Pluto` and `PlutoSliderServer` are therefore pinned with `=` in +`pluto-deployment-environment/Project.toml`, so `Pkg.update()` cannot do this by +accident. To bump on purpose: + +1. Edit the `[compat]` pins, then `Pkg.update()` in that environment. +2. Locally: `rm -rf _cache && julia --project=pluto-deployment-environment generate.jl`. + Budget several hours. +3. Commit the regenerated `_cache/` **in the same commit** as the `Manifest.toml` + change. The two must never be out of step. +4. `python3 tools/check_cache.py` must pass before you push. + +`pluto-deployment-environment/Manifest.toml` is the only thing that pins Pluto, so it +is force-tracked in `.gitignore`. Do not untrack it. + +### The inverse problem: stale output + +Because the cache key is the notebook's *content*, bumping any other package +(ModelingToolkit, Turing, ...) does **not** invalidate anything. The site keeps +serving output produced by the old versions until a notebook is edited. After a +significant SciML upgrade, force a full refresh by hand: + +``` +rm -rf _cache && julia --project=pluto-deployment-environment generate.jl +``` + +This also proves every notebook still runs, which a warm build never checks. Worth +doing once a term. + +### The preflight check + +`ExportNotebooks.yml` runs `tools/check_cache.py` as a separate `cache-preflight` +job before the expensive one. It fails in seconds when the cache cannot serve the +build (Pluto bump, empty `_cache/`) instead of after six hours of runner time. It +only warns for the normal cases, edited notebooks and orphans. Run it locally too. ## `.css`, `.html`, `.gif`, etc