From a2d4683038d65e39d006c6e0519ea13af664e2b9 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Wed, 12 Aug 2026 15:26:44 +1000 Subject: [PATCH 1/2] Record graph.txt's disposition now that wasm embeds it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit QuantEcon/lecture-wasm#63 stopped lecture-wasm fetching lecture-python-intro's committed graph.txt over the network and gave it a `%%file` cell like every sibling repo. That was the only URL read of that blob anywhere in the organisation, so three records here are now wrong. `scripts/audit_annotations.yml`: - the graph.txt note said lecture-wasm "fetches intro's committed copy by URL (requests.get), making that copy load-bearing". No longer true. Every consumer now embeds the data in its own lecture, and graph.txt has dropped out of the scanned dataset set entirely — it is `embedded` in four repos and a shadowed orphan in the rest. - the `lecture-wasm:lectures/graph.txt` mirror-orphan entry names a file that #63 deleted. Dead key, removed. - intro's copy became a BARE orphan the moment that read went away — `kind: orphan`, no note — which reads as an unexplained stray file. It is annotated `shadowed`, matching lecture-dp's identical case: short_path regenerates it via `%%file` before reading it, so the committed bytes are never consumed (the build prints "Overwriting graph.txt" where lecture-jax, which commits no copy, prints "Writing graph.txt"). `PLAN.md`: Track A is complete. The last two CSVs landed as wave A4 (#74, flipped in #75), and graph.txt is recorded as a deliberate NON-migration rather than a remaining item — synthetic teaching data, null in every provenance field, and the exercise teaches its format by quoting the first line, so the data has to stay visible on the page. Hosting it here would have put a toy in a registry that exists to carry provenance. The note also carries the deletion caveat, because the obvious next step is wrong at scale: intro's copy is deletable as Track X, but the same blob sits in 8 repos and is regenerated at 17 `%%file` sites including archived `.rst` ancestors that `gh search code` cannot see. That deletion needs a per-repo reader sweep, not an org-wide one. Figures re-read from audit.json rather than carried: 40 static files (was 41 — graph.txt is no longer a scanned dataset), 38 committed files (was 43 — five deletions across wasm and intro), 24 orphans, 26 migrated. Strict audit exit 0, zero warnings, zero problems in all four categories; catalog freshness gate green. --- PLAN.md | 4 +++- scripts/audit_annotations.yml | 19 +++++++++++++------ 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/PLAN.md b/PLAN.md index 3097fc5..bfbed80 100644 --- a/PLAN.md +++ b/PLAN.md @@ -213,7 +213,7 @@ The remaining work decomposes by **consuming series** rather than by hosting pat | Track | Datasets | Coupling | Blocked on | | --- | --- | --- | --- | -| **A — `intro` + `wasm`** | 17, of which **3 remain**: `life-expectancy…`, `usa-gini…`, `graph.txt` (the 8 intro statics and the 6 `high_dim_data` files are done) | **paired — repoint together, always**; `lecture-intro.zh-cn` and `test-actions-lecture-intro` are third and fourth consumers of the two CSVs | nothing — `usa-gini`'s SCF dependency was discharged by P3 | +| **A — `intro` + `wasm`** | 17, **all done**. The last two CSVs landed as wave A4 ([#74](https://github.com/QuantEcon/data-lectures/pull/74), flipped in [#75](https://github.com/QuantEcon/data-lectures/pull/75)); `graph.txt` was never a migration — see below | — | — | | **B — `python.myst`** | 7: `maketable1/2/4.dta`, `fp.dta`, `hansen_singleton_1982/1983_data.csv`, `NEWQDATA.csv` | none | nothing | | **C — `advanced.myst`** | 6: `dataBHS.mat`, `acs_data_summary.csv`, `bbh` ×2, `fred_data.csv`, `hansen_jagannathan_1991_data.json` | none | nothing (builder recovery is in-wave work, not a gate) | | **D — `programming`** | 1: `test_pwt.csv` | none | nothing — a single-PR track | @@ -221,6 +221,8 @@ The remaining work decomposes by **consuming series** rather than by hosting pat | **X — orphan sweep** | 26 committed orphans across 6 repos — dp 10, programming 5, wasm 5, intro 3, python.myst 2, `continuous_time_mcs` 1 | per repo | that repo's repoints landing first | | **Y — consumer interface (`qeld`)** | the `qeld` package, Q1–Q7 of `PLAN-QELD-PACKAGE.md` — audit support, the package, pilots, then adoption by win; QEP graduation stays | — | nothing — re-scoped 2026-08-12 (D11): the DNS → custom domain → URL-sweep sequence this row used to carry is retired | +**`graph.txt` was closed out as a non-migration (2026-08-12).** It is synthetic teaching data — `provenance: toy`, null in every real provenance field — and the shortest-path exercise teaches its format by quoting the first line, so the data has to stay visible on the page. Hosting it here would have put a toy in a registry that exists to carry provenance. Instead `lecture-wasm` stopped fetching intro's committed copy over the network and embeds it with `%%file` like every sibling ([QuantEcon/lecture-wasm#63](https://github.com/QuantEcon/lecture-wasm/pull/63)), which retired the last cross-repo read of that blob anywhere in the organisation. `graph.txt` consequently no longer appears as a scanned dataset at all: it is `embedded` in four repos and a shadowed orphan in the rest. Intro's committed copy is now deletable as Track X — but the same blob sits in 8 repos and is regenerated at 17 `%%file` sites, including archived `.rst` ancestors that `gh search code` cannot see, so that deletion needs its own per-repo reader sweep rather than an org-wide sweep. + `lecture-dp`, `lecture-jax` and `continuous_time_mcs` are **not data consumers** — dp's 10 committed files are inherited orphans, jax embeds `graph.txt` via `%%file`, and continuous_time_mcs has one orphan scratch file. They appear only in Track X. **Tracks A–D are independent of each other and can run in any order or in parallel.** The only hard dependencies in the whole programme are: `usa-gini-nwealth-tincome-lincome.csv` is built from `SCF_plus_mini.csv` (so it follows the SCF migration inside Track A); Track E's rollout needs its own template proven first; Track X follows its repo's repoints; and Track Y's adoption sweep (qeld Q7) is last. diff --git a/scripts/audit_annotations.yml b/scripts/audit_annotations.yml index a023c31..cee051e 100644 --- a/scripts/audit_annotations.yml +++ b/scripts/audit_annotations.yml @@ -72,9 +72,12 @@ datasets: description: 100-node weighted digraph for the shortest-path problem provenance: toy note: > - maintained as %%file blocks in intro, dp and jax; lecture-wasm instead - fetches intro's committed copy by URL (requests.get), making that copy - load-bearing — it was merely "shadowed" in the 2026-07-15 audit + maintained as %%file blocks in intro, dp, jax and (since + QuantEcon/lecture-wasm#63) lecture-wasm — every consumer now embeds the + data in its own lecture, so no repo reads another's copy. wasm used to + fetch intro's committed copy by URL, which made that copy load-bearing; + with that read gone the file is genuinely shadowed everywhere and + graph.txt no longer appears as a scanned dataset at all hansen_jagannathan_1991_data.json: description: Hansen–Jagannathan (1991) asset-returns bundle provenance: constructed-lost @@ -277,6 +280,13 @@ committed_unreferenced: lecture-python.myst:lectures/web_graph_data.txt: kind: shadowed note: shadowed duplicate at lectures/ root + lecture-python-intro:lectures/graph.txt: + kind: shadowed + note: > + short_path regenerates it via %%file before reading it, so the committed + bytes are never consumed — the build prints "Overwriting graph.txt". It + became an orphan when QuantEcon/lecture-wasm#63 removed the only URL read + of it in the organisation; deletable as Track X lecture-dp:lectures/graph.txt: kind: shadowed note: short_path regenerates it via %%file @@ -307,9 +317,6 @@ committed_unreferenced: lecture-dp:lectures/_static/lecture_specific/pandas_panel/realwage.csv: kind: orphan note: inherited copy; the consuming lectures live in other repos and now read data-lectures - lecture-wasm:lectures/graph.txt: - kind: mirror-orphan - note: wasm's short_path fetches intro's committed graph.txt by URL, not this copy lecture-wasm:lectures/_static/lecture_specific/inequality/usa-gini-nwealth-tincome-lincome.csv: kind: mirror-orphan note: wasm's inequality reads intro's copy by URL From b2c2a8849c017c2945f7d34dea2ea466bc5e68ca Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Wed, 12 Aug 2026 15:33:03 +1000 Subject: [PATCH 2/2] Copilot: distinguish 'shadowed' from 'embedded', and fix a contradiction in rule 2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two review comments, both correct, plus a third instance of the first one that the review did not flag. **'shadowed everywhere' was wrong** (audit_annotations.yml). Shadowed means a committed copy exists and is overwritten by the %%file cell before being read. That is true of two of the four embedding repos, not all of them: lecture-python-intro %%file + committed copy -> shadowed lecture-dp %%file + committed copy -> shadowed lecture-jax %%file, no copy -> just embedded lecture-wasm %%file, no copy -> just embedded The note implied a committed copy exists everywhere, which is backwards for the two repos that model the shape we actually want. Reworded to split the cases. **PLAN.md rule 2 contradicted the rest of the document.** It still read "(graph.txt is Track A but single-consumer: only lecture-wasm reads intro's committed copy.)" — false twice over after this PR, since wasm no longer reads it and it is not a Track A item. Rule 2 is where someone scoping a repoint set looks, so the parenthetical is rewritten to say graph.txt is out of scope and why, rather than deleted. **The same imprecision was in PLAN.md's own disposition note**, which the review did not comment on: "embedded in four repos and a shadowed orphan in the rest" puts intro and dp in both sets. Now states which repos embed, which of those also commit a copy, and that the remaining committed copies (lecture-intro.zh-cn, the canary, lecture-python.zh-cn, lecture-dp.monorepo, ipynb_pdf_constructor) are read by nothing. Verified against the repos rather than restated: all four embed via %%file; only intro and dp commit a copy. Strict audit exit 0 with zero warnings, catalog gate green, audit_annotations.yml parses with the dead wasm key gone and the intro key present. --- PLAN.md | 4 ++-- scripts/audit_annotations.yml | 6 ++++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/PLAN.md b/PLAN.md index bfbed80..38713bb 100644 --- a/PLAN.md +++ b/PLAN.md @@ -70,7 +70,7 @@ Sweep by cloning and grepping, not with `gh search code` on a URL — code searc The strict audit has **no green state for a partially-repointed dataset**. `scripts/build_audit.py` fails a record marked `pending`/`landed` while any consumer already reads data-lectures, *and* fails one marked `repointed`/`final` while any consumer still does not. That is deliberate — it is what makes the tracker trustworthy — but it means a dataset with two consuming repos cannot be moved one repo at a time without the drift alarm firing in the gap. -**2 of the 17 remaining datasets have two consuming repos, and both are `lecture-python-intro` + `lecture-wasm`** — `life-expectancy-vs-gdp-per-capita.csv` and `usa-gini-nwealth-tincome-lincome.csv`. (Six of the eight this line used to name were the `high_dim_data` files, repointed 2026-08-11.) There is no other cross-series coupling left; the last one was the P2 `pandas_panel` trio, already done. (`graph.txt` is Track A but single-consumer: only `lecture-wasm` reads intro's committed copy.) +**2 of the 17 remaining datasets have two consuming repos, and both are `lecture-python-intro` + `lecture-wasm`** — `life-expectancy-vs-gdp-per-capita.csv` and `usa-gini-nwealth-tincome-lincome.csv`. (Six of the eight this line used to name were the `high_dim_data` files, repointed 2026-08-11.) There is no other cross-series coupling left; the last one was the P2 `pandas_panel` trio, already done. (`graph.txt` is **not** in this count and is no longer a Track A item at all — `lecture-wasm` used to read intro's committed copy, which is what made it look like a one-consumer dataset; that read is gone and it is now embedded in every consuming lecture. See the Track A row below.) **"Two consuming repos" is the audit's count, not the consumer set.** `SCAN_REPOS` is the eight Python-family repos, so a dataset the dashboard shows with two consumers may have four or five in reality. Measured 2026-08-12, both remaining pairs are read by **five** reference-holders each: intro, `lecture-wasm`, `lecture-intro.zh-cn`, `QuantEcon/test-actions-lecture-intro`, and the generated `lecture-python-intro.notebooks` mirror. The last three are invisible to every audit run — see rule 1. @@ -221,7 +221,7 @@ The remaining work decomposes by **consuming series** rather than by hosting pat | **X — orphan sweep** | 26 committed orphans across 6 repos — dp 10, programming 5, wasm 5, intro 3, python.myst 2, `continuous_time_mcs` 1 | per repo | that repo's repoints landing first | | **Y — consumer interface (`qeld`)** | the `qeld` package, Q1–Q7 of `PLAN-QELD-PACKAGE.md` — audit support, the package, pilots, then adoption by win; QEP graduation stays | — | nothing — re-scoped 2026-08-12 (D11): the DNS → custom domain → URL-sweep sequence this row used to carry is retired | -**`graph.txt` was closed out as a non-migration (2026-08-12).** It is synthetic teaching data — `provenance: toy`, null in every real provenance field — and the shortest-path exercise teaches its format by quoting the first line, so the data has to stay visible on the page. Hosting it here would have put a toy in a registry that exists to carry provenance. Instead `lecture-wasm` stopped fetching intro's committed copy over the network and embeds it with `%%file` like every sibling ([QuantEcon/lecture-wasm#63](https://github.com/QuantEcon/lecture-wasm/pull/63)), which retired the last cross-repo read of that blob anywhere in the organisation. `graph.txt` consequently no longer appears as a scanned dataset at all: it is `embedded` in four repos and a shadowed orphan in the rest. Intro's committed copy is now deletable as Track X — but the same blob sits in 8 repos and is regenerated at 17 `%%file` sites, including archived `.rst` ancestors that `gh search code` cannot see, so that deletion needs its own per-repo reader sweep rather than an org-wide sweep. +**`graph.txt` was closed out as a non-migration (2026-08-12).** It is synthetic teaching data — `provenance: toy`, null in every real provenance field — and the shortest-path exercise teaches its format by quoting the first line, so the data has to stay visible on the page. Hosting it here would have put a toy in a registry that exists to carry provenance. Instead `lecture-wasm` stopped fetching intro's committed copy over the network and embeds it with `%%file` like every sibling ([QuantEcon/lecture-wasm#63](https://github.com/QuantEcon/lecture-wasm/pull/63)), which retired the last cross-repo read of that blob anywhere in the organisation. `graph.txt` consequently no longer appears as a scanned dataset at all. Four repos embed it via `%%file` — intro, dp, jax and wasm — and two of those (intro, dp) also commit a copy the cell overwrites before reading, so those two are shadowed orphans; jax and wasm commit none, which is the cleaner shape. The remaining committed copies (`lecture-intro.zh-cn`, the canary, `lecture-python.zh-cn`, `lecture-dp.monorepo`, `ipynb_pdf_constructor`) are read by nothing. Intro's committed copy is now deletable as Track X — but the same blob sits in 8 repos and is regenerated at 17 `%%file` sites, including archived `.rst` ancestors that `gh search code` cannot see, so that deletion needs its own per-repo reader sweep rather than an org-wide sweep. `lecture-dp`, `lecture-jax` and `continuous_time_mcs` are **not data consumers** — dp's 10 committed files are inherited orphans, jax embeds `graph.txt` via `%%file`, and continuous_time_mcs has one orphan scratch file. They appear only in Track X. diff --git a/scripts/audit_annotations.yml b/scripts/audit_annotations.yml index cee051e..dd05da1 100644 --- a/scripts/audit_annotations.yml +++ b/scripts/audit_annotations.yml @@ -76,8 +76,10 @@ datasets: QuantEcon/lecture-wasm#63) lecture-wasm — every consumer now embeds the data in its own lecture, so no repo reads another's copy. wasm used to fetch intro's committed copy by URL, which made that copy load-bearing; - with that read gone the file is genuinely shadowed everywhere and - graph.txt no longer appears as a scanned dataset at all + with that read gone graph.txt no longer appears as a scanned dataset at + all. Two of the four repos also commit a copy the %%file cell overwrites + before reading (intro, dp) — those are shadowed; jax and wasm commit no + copy at all, which is the cleaner shape hansen_jagannathan_1991_data.json: description: Hansen–Jagannathan (1991) asset-returns bundle provenance: constructed-lost