From 953e12b5957f1b79aa10141bd8d8c1a9e5873d67 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 6 Aug 2026 16:51:07 +1000 Subject: [PATCH 1/2] Add repoint rule 3: the published site lags main, so publish before deleting Rules 1 and 2 protect the repositories. Neither protects the PUBLISHED SITE, and that gap is where the first real reader-facing breakage happened. lecture-python-intro publishes on a `publish*` tag, not on push to main. So merging a repoint does not refresh the live site -- the already-published notebooks keep the old URL, and if the same PR deleted the file, that URL 404s. Set 1 demonstrated it: after lecture-python-intro#823 merged, the notebook served at intro.quantecon.org still carried the deleted mpd2020.xlsx path, and stayed broken until a publish was tagged. What makes it nasty is that nothing reports it. The rendered HTML is fine -- figures are baked at build time, so the pages look correct -- and neither repo's CI exercises a published notebook. The breakage lands only on readers who download or open the lecture in Colab, which for a computational series is the audience that matters most. So a repoint set is two phases: repoint the URLs keeping the files, publish, then delete in a follow-up. Costs one extra PR per set; leaves no window. Publish triggers differ per repo and the table says which need the split -- lecture-wasm publishes on push to main and self-heals, and deleting a mirror-orphan nothing reads never needs it. Rule 2 is also sharpened. It was written as a constraint on the tracker flip, but 2026-08-06 showed it binds the lecture PRs equally: lecture-wasm#53 merged ahead of lecture-python-intro#824 and left main failing on both files until the second landed. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 7 ++++--- PLAN.md | 30 ++++++++++++++++++++++++++++-- 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index ed172f0..c6f3ba0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,12 +36,13 @@ The Feb 2025 migration left files that cannot fully satisfy the rules above. The - **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that says why (`status: unverifiable` with a `note`). - **`builder_status: unrecovered` — constructed without a recoverable builder.** A constructed dataset ships its builder, and one that omits it *silently* is the bug. Several inherited files are constructed with no recoverable extraction steps (PLAN Phase 9 tracks them). Keep `class: constructed` — reclassifying to `verbatim` to dodge the rule is misclassification — set `builder: null` and `builder_status: unrecovered`, and the gap stays visible for Phase 9 to recover. `unrecovered` is for **inherited files only**; never introduce a *new* constructed file without its builder. -### Repointing a lecture — two ordering traps +### Repointing a lecture — three ordering traps -Both are cheap to follow and expensive to discover. `PLAN.md` carries the reasoning and the current counts. +All cheap to follow and expensive to discover. `PLAN.md` carries the reasoning and the current counts. - **Never delete a file a sibling repo reads.** `lecture-wasm` fetches `lecture-python-intro`'s *committed blobs* by URL, so deleting intro's copy in a repoint PR 404s the wasm build immediately. "Delete the lecture repo's own copy in the same repoint PR" applies only where no sibling reads it; where one does, the sibling's repoint lands first or in the same set. -- **Repoint every consumer of a dataset together.** The strict audit has no green state for a partially-repointed dataset — `pending`/`landed` fails once any consumer reads data-lectures, and `repointed`/`final` fails while any consumer still does not. Land the lecture repoints first, then flip `migration.yml`; that flip is the push that re-runs the audit, so reality and the tracker agree by the time it runs. +- **Repoint every consumer of a dataset together.** The strict audit has no green state for a partially-repointed dataset — `pending`/`landed` fails once any consumer reads data-lectures, and `repointed`/`final` fails while any consumer still does not. Land the lecture repoints first, then flip `migration.yml`; that flip is the push that re-runs the audit, so reality and the tracker agree by the time it runs. This binds the **lecture PRs too**: merging one half of a set while the other sits open opens the same window. +- **Repoint, publish, then delete — the published site lags `main`.** A lecture repo that publishes on a **tag** (`lecture-python-intro` uses `publish*`) does not refresh its site when a repoint merges, so the already-published notebooks keep the old URL. Delete the file in the same PR and that URL 404s for every reader who downloads or opens the lecture in Colab, until someone tags a publish. Rendered HTML is unaffected — figures are baked at build time — so nothing will alert you. **Split it: repoint the URLs and keep the files, publish, then delete in a follow-up PR.** Repos that publish on push to `main` (`lecture-wasm`) self-heal and need no split, and neither does deleting a mirror-orphan nothing reads. - **A migration moves bytes; it does not update them.** Land the copy the lectures already consume, validated byte-identical — that is what makes a repoint provably unable to change a figure. If the committed file differs from what upstream publishes today, migrate it unchanged anyway, record the delta in `integrity.upstream` **and** in the register at [#39](https://github.com/QuantEcon/data-lectures/issues/39), and leave the decision for after the migration. Adopting a newer vintage changes lecture output and is an author's call, not an infrastructure one — and per "Corrections vs vintages" below it gets a **new filename**, never a silent replacement. diff --git a/PLAN.md b/PLAN.md index 33516a2..9f84599 100644 --- a/PLAN.md +++ b/PLAN.md @@ -42,7 +42,7 @@ This repository is being shaped into the **single canonical repository for data ## Repoint rules -Three rules learned the hard way. The first two are about *ordering*, both cheap to follow and expensive to discover, and neither is enforced by CI — the strict audit catches the second only after the fact. The third is about *scope*. +Four rules learned the hard way, three of them the hard way twice. Rules 1-3 are about *ordering* and none is enforced by CI — the strict audit catches rule 2 only after the fact, and cannot see rule 3 at all. Rule 4 is about *scope*. ### 1. Repoint a sibling reader before deleting the file it reads @@ -60,7 +60,33 @@ The strict audit has **no green state for a partially-repointed dataset**. `scri Practically: one branch name across data-lectures + every consuming repo, PRs opened together, lecture repoints merged first, then the `migration.yml` flip to `repointed` — that last push is what re-runs the audit, and by then reality and the tracker agree. -### 3. A migration moves bytes; it does not update them +This constrains the **lecture PRs**, not only the tracker flip. Merging one half of a set while the other sits open partially repoints the dataset and opens the same window — observed on 2026-08-06, when `lecture-wasm#53` merged ahead of `lecture-python-intro#824` and left `main` failing on both files until the second landed. + +### 3. Repoint, publish, *then* delete — the published site lags `main` + +Rules 1 and 2 protect the **repositories**. Neither protects the **published site**, and that gap is where the first real breakage happened. + +`lecture-python-intro` publishes on a **`publish*` tag**, not on push to `main`. So merging a repoint does not refresh the live site: the already-published notebooks keep the *old* URL, and if the same PR deleted the file, that URL now 404s. Set 1 proved it — after [lecture-python-intro#823](https://github.com/QuantEcon/lecture-python-intro/pull/823) merged, the notebook served at `intro.quantecon.org` still carried `…/lecture-python-intro/raw/main/lectures/datasets/mpd2020.xlsx`, which had just been deleted. The window stayed open until a publish was tagged. + +**The rendered HTML is fine** — figures are baked at build time, so a reader browsing the site sees nothing wrong. The breakage is confined to the downloadable notebook, the Colab link, and `{download}` targets: i.e. every reader who actually *runs* the lecture. + +So a repoint set is **two phases**: + +1. **Repoint the URLs, keep the files.** Merge, then publish. Now the published notebooks fetch from data-lectures while the old paths still resolve — neither the old site nor the new one can break. +2. **Delete the old copies.** Nothing references them in the repo *or* on the live site. + +Cost is one extra PR per set and a slower orphan cleanup; the benefit is that no reader-facing window exists at any point. + +**Per-repo publish triggers matter**, so check before assuming: + +| Repo | Trigger | Needs two phases? | +| --- | --- | --- | +| `lecture-python-intro` | `publish*` tag (manual) | **yes** | +| `lecture-wasm` | push to `main` | no — self-heals on merge | + +A repo that publishes on push needs no split, and neither does the deletion of a *mirror-orphan* nothing reads in either repo. + +### 4. A migration moves bytes; it does not update them The copy that lands here is the copy the lectures **already consume**, validated byte-identical in the repoint PR. That is what makes a repoint safe to merge: it provably cannot change a single figure. From 805f263ff88f9b9cacfb5071baaca9783c8e9a4f Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 6 Aug 2026 17:46:09 +1000 Subject: [PATCH 2/2] Address Copilot: fix the rule count, and define "mirror-orphan" Three valid comments, all documentation defects introduced by the rule-3 commit. AGENTS.md said "three ordering traps" above a list of four bullets. Adding rule 3 pushed the count without updating the header. PLAN.md already had this right -- rules 1-3 are ordering, rule 4 is scope, numbered separately -- so AGENTS.md now matches that split rather than merely correcting the numeral: the scope rule gets its own subheading, because it is a different kind of rule and burying it in an ordering list was the actual mistake. "mirror-orphan" appeared in both files and was defined in neither. It now reads as what it is -- a copy no lecture reads in either repo, typically one a repo committed alongside its mirrored sources while the lecture fetches the other repo's copy by URL -- with the term kept parenthetically for anyone who has met it in the audit output. Checking that turned up slightly more than the comments flagged: plain "orphan" is not defined in AGENTS.md either, so a reader previously hit two undefined terms in one sentence. The rewrite removes the dependency on both. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 8 +++++--- PLAN.md | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c6f3ba0..3fa8f73 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,12 +42,14 @@ All cheap to follow and expensive to discover. `PLAN.md` carries the reasoning a - **Never delete a file a sibling repo reads.** `lecture-wasm` fetches `lecture-python-intro`'s *committed blobs* by URL, so deleting intro's copy in a repoint PR 404s the wasm build immediately. "Delete the lecture repo's own copy in the same repoint PR" applies only where no sibling reads it; where one does, the sibling's repoint lands first or in the same set. - **Repoint every consumer of a dataset together.** The strict audit has no green state for a partially-repointed dataset — `pending`/`landed` fails once any consumer reads data-lectures, and `repointed`/`final` fails while any consumer still does not. Land the lecture repoints first, then flip `migration.yml`; that flip is the push that re-runs the audit, so reality and the tracker agree by the time it runs. This binds the **lecture PRs too**: merging one half of a set while the other sits open opens the same window. -- **Repoint, publish, then delete — the published site lags `main`.** A lecture repo that publishes on a **tag** (`lecture-python-intro` uses `publish*`) does not refresh its site when a repoint merges, so the already-published notebooks keep the old URL. Delete the file in the same PR and that URL 404s for every reader who downloads or opens the lecture in Colab, until someone tags a publish. Rendered HTML is unaffected — figures are baked at build time — so nothing will alert you. **Split it: repoint the URLs and keep the files, publish, then delete in a follow-up PR.** Repos that publish on push to `main` (`lecture-wasm`) self-heal and need no split, and neither does deleting a mirror-orphan nothing reads. - -- **A migration moves bytes; it does not update them.** Land the copy the lectures already consume, validated byte-identical — that is what makes a repoint provably unable to change a figure. If the committed file differs from what upstream publishes today, migrate it unchanged anyway, record the delta in `integrity.upstream` **and** in the register at [#39](https://github.com/QuantEcon/data-lectures/issues/39), and leave the decision for after the migration. Adopting a newer vintage changes lecture output and is an author's call, not an infrastructure one — and per "Corrections vs vintages" below it gets a **new filename**, never a silent replacement. +- **Repoint, publish, then delete — the published site lags `main`.** A lecture repo that publishes on a **tag** (`lecture-python-intro` uses `publish*`) does not refresh its site when a repoint merges, so the already-published notebooks keep the old URL. Delete the file in the same PR and that URL 404s for every reader who downloads or opens the lecture in Colab, until someone tags a publish. Rendered HTML is unaffected — figures are baked at build time — so nothing will alert you. **Split it: repoint the URLs and keep the files, publish, then delete in a follow-up PR.** Repos that publish on push to `main` (`lecture-wasm`) self-heal and need no split — and neither does deleting a copy that no lecture reads, such as one a repo committed while its lecture fetches another repo's copy by URL (a *mirror-orphan*). Cross-repo repoints are worked from [`QuantEcon/workspace-lectures`](https://github.com/QuantEcon/workspace-lectures) — same branch name in each repo, one PR per repo, no aggregate PR. +### Repointing a lecture — one scope rule + +- **A migration moves bytes; it does not update them.** Land the copy the lectures already consume, validated byte-identical — that is what makes a repoint provably unable to change a figure. If the committed file differs from what upstream publishes today, migrate it unchanged anyway, record the delta in `integrity.upstream` **and** in the register at [#39](https://github.com/QuantEcon/data-lectures/issues/39), and leave the decision for after the migration. Adopting a newer vintage changes lecture output and is an author's call, not an infrastructure one — and per "Corrections vs vintages" below it gets a **new filename**, never a silent replacement. + ### Corrections vs vintages - **Corrections** (bad parse, wrong units, corrupt rows): fix **in place**, same filename — every consumer should get the fix. Use the manifest's `consumers` list to know which lectures to rebuild/review. diff --git a/PLAN.md b/PLAN.md index 9f84599..b43c94e 100644 --- a/PLAN.md +++ b/PLAN.md @@ -84,7 +84,7 @@ Cost is one extra PR per set and a slower orphan cleanup; the benefit is that no | `lecture-python-intro` | `publish*` tag (manual) | **yes** | | `lecture-wasm` | push to `main` | no — self-heals on merge | -A repo that publishes on push needs no split, and neither does the deletion of a *mirror-orphan* nothing reads in either repo. +A repo that publishes on push needs no split. Neither does deleting a copy that **no lecture reads in either repo** — typically one a repo committed alongside its mirrored sources while the lecture itself fetches the *other* repo's copy by URL (a *mirror-orphan*); `lecture-wasm` holds a dozen of these. ### 4. A migration moves bytes; it does not update them