diff --git a/PLAN.md b/PLAN.md index 3097fc5..38713bb 100644 --- a/PLAN.md +++ b/PLAN.md @@ -70,7 +70,7 @@ Sweep by cloning and grepping, not with `gh search code` on a URL — code searc The strict audit has **no green state for a partially-repointed dataset**. `scripts/build_audit.py` fails a record marked `pending`/`landed` while any consumer already reads data-lectures, *and* fails one marked `repointed`/`final` while any consumer still does not. That is deliberate — it is what makes the tracker trustworthy — but it means a dataset with two consuming repos cannot be moved one repo at a time without the drift alarm firing in the gap. -**2 of the 17 remaining datasets have two consuming repos, and both are `lecture-python-intro` + `lecture-wasm`** — `life-expectancy-vs-gdp-per-capita.csv` and `usa-gini-nwealth-tincome-lincome.csv`. (Six of the eight this line used to name were the `high_dim_data` files, repointed 2026-08-11.) There is no other cross-series coupling left; the last one was the P2 `pandas_panel` trio, already done. (`graph.txt` is Track A but single-consumer: only `lecture-wasm` reads intro's committed copy.) +**2 of the 17 remaining datasets have two consuming repos, and both are `lecture-python-intro` + `lecture-wasm`** — `life-expectancy-vs-gdp-per-capita.csv` and `usa-gini-nwealth-tincome-lincome.csv`. (Six of the eight this line used to name were the `high_dim_data` files, repointed 2026-08-11.) There is no other cross-series coupling left; the last one was the P2 `pandas_panel` trio, already done. (`graph.txt` is **not** in this count and is no longer a Track A item at all — `lecture-wasm` used to read intro's committed copy, which is what made it look like a one-consumer dataset; that read is gone and it is now embedded in every consuming lecture. See the Track A row below.) **"Two consuming repos" is the audit's count, not the consumer set.** `SCAN_REPOS` is the eight Python-family repos, so a dataset the dashboard shows with two consumers may have four or five in reality. Measured 2026-08-12, both remaining pairs are read by **five** reference-holders each: intro, `lecture-wasm`, `lecture-intro.zh-cn`, `QuantEcon/test-actions-lecture-intro`, and the generated `lecture-python-intro.notebooks` mirror. The last three are invisible to every audit run — see rule 1. @@ -213,7 +213,7 @@ The remaining work decomposes by **consuming series** rather than by hosting pat | Track | Datasets | Coupling | Blocked on | | --- | --- | --- | --- | -| **A — `intro` + `wasm`** | 17, of which **3 remain**: `life-expectancy…`, `usa-gini…`, `graph.txt` (the 8 intro statics and the 6 `high_dim_data` files are done) | **paired — repoint together, always**; `lecture-intro.zh-cn` and `test-actions-lecture-intro` are third and fourth consumers of the two CSVs | nothing — `usa-gini`'s SCF dependency was discharged by P3 | +| **A — `intro` + `wasm`** | 17, **all done**. The last two CSVs landed as wave A4 ([#74](https://github.com/QuantEcon/data-lectures/pull/74), flipped in [#75](https://github.com/QuantEcon/data-lectures/pull/75)); `graph.txt` was never a migration — see below | — | — | | **B — `python.myst`** | 7: `maketable1/2/4.dta`, `fp.dta`, `hansen_singleton_1982/1983_data.csv`, `NEWQDATA.csv` | none | nothing | | **C — `advanced.myst`** | 6: `dataBHS.mat`, `acs_data_summary.csv`, `bbh` ×2, `fred_data.csv`, `hansen_jagannathan_1991_data.json` | none | nothing (builder recovery is in-wave work, not a gate) | | **D — `programming`** | 1: `test_pwt.csv` | none | nothing — a single-PR track | @@ -221,6 +221,8 @@ The remaining work decomposes by **consuming series** rather than by hosting pat | **X — orphan sweep** | 26 committed orphans across 6 repos — dp 10, programming 5, wasm 5, intro 3, python.myst 2, `continuous_time_mcs` 1 | per repo | that repo's repoints landing first | | **Y — consumer interface (`qeld`)** | the `qeld` package, Q1–Q7 of `PLAN-QELD-PACKAGE.md` — audit support, the package, pilots, then adoption by win; QEP graduation stays | — | nothing — re-scoped 2026-08-12 (D11): the DNS → custom domain → URL-sweep sequence this row used to carry is retired | +**`graph.txt` was closed out as a non-migration (2026-08-12).** It is synthetic teaching data — `provenance: toy`, null in every real provenance field — and the shortest-path exercise teaches its format by quoting the first line, so the data has to stay visible on the page. Hosting it here would have put a toy in a registry that exists to carry provenance. Instead `lecture-wasm` stopped fetching intro's committed copy over the network and embeds it with `%%file` like every sibling ([QuantEcon/lecture-wasm#63](https://github.com/QuantEcon/lecture-wasm/pull/63)), which retired the last cross-repo read of that blob anywhere in the organisation. `graph.txt` consequently no longer appears as a scanned dataset at all. Four repos embed it via `%%file` — intro, dp, jax and wasm — and two of those (intro, dp) also commit a copy the cell overwrites before reading, so those two are shadowed orphans; jax and wasm commit none, which is the cleaner shape. The remaining committed copies (`lecture-intro.zh-cn`, the canary, `lecture-python.zh-cn`, `lecture-dp.monorepo`, `ipynb_pdf_constructor`) are read by nothing. Intro's committed copy is now deletable as Track X — but the same blob sits in 8 repos and is regenerated at 17 `%%file` sites, including archived `.rst` ancestors that `gh search code` cannot see, so that deletion needs its own per-repo reader sweep rather than an org-wide sweep. + `lecture-dp`, `lecture-jax` and `continuous_time_mcs` are **not data consumers** — dp's 10 committed files are inherited orphans, jax embeds `graph.txt` via `%%file`, and continuous_time_mcs has one orphan scratch file. They appear only in Track X. **Tracks A–D are independent of each other and can run in any order or in parallel.** The only hard dependencies in the whole programme are: `usa-gini-nwealth-tincome-lincome.csv` is built from `SCF_plus_mini.csv` (so it follows the SCF migration inside Track A); Track E's rollout needs its own template proven first; Track X follows its repo's repoints; and Track Y's adoption sweep (qeld Q7) is last. diff --git a/scripts/audit_annotations.yml b/scripts/audit_annotations.yml index a023c31..dd05da1 100644 --- a/scripts/audit_annotations.yml +++ b/scripts/audit_annotations.yml @@ -72,9 +72,14 @@ datasets: description: 100-node weighted digraph for the shortest-path problem provenance: toy note: > - maintained as %%file blocks in intro, dp and jax; lecture-wasm instead - fetches intro's committed copy by URL (requests.get), making that copy - load-bearing — it was merely "shadowed" in the 2026-07-15 audit + maintained as %%file blocks in intro, dp, jax and (since + QuantEcon/lecture-wasm#63) lecture-wasm — every consumer now embeds the + data in its own lecture, so no repo reads another's copy. wasm used to + fetch intro's committed copy by URL, which made that copy load-bearing; + with that read gone graph.txt no longer appears as a scanned dataset at + all. Two of the four repos also commit a copy the %%file cell overwrites + before reading (intro, dp) — those are shadowed; jax and wasm commit no + copy at all, which is the cleaner shape hansen_jagannathan_1991_data.json: description: Hansen–Jagannathan (1991) asset-returns bundle provenance: constructed-lost @@ -277,6 +282,13 @@ committed_unreferenced: lecture-python.myst:lectures/web_graph_data.txt: kind: shadowed note: shadowed duplicate at lectures/ root + lecture-python-intro:lectures/graph.txt: + kind: shadowed + note: > + short_path regenerates it via %%file before reading it, so the committed + bytes are never consumed — the build prints "Overwriting graph.txt". It + became an orphan when QuantEcon/lecture-wasm#63 removed the only URL read + of it in the organisation; deletable as Track X lecture-dp:lectures/graph.txt: kind: shadowed note: short_path regenerates it via %%file @@ -307,9 +319,6 @@ committed_unreferenced: lecture-dp:lectures/_static/lecture_specific/pandas_panel/realwage.csv: kind: orphan note: inherited copy; the consuming lectures live in other repos and now read data-lectures - lecture-wasm:lectures/graph.txt: - kind: mirror-orphan - note: wasm's short_path fetches intro's committed graph.txt by URL, not this copy lecture-wasm:lectures/_static/lecture_specific/inequality/usa-gini-nwealth-tincome-lincome.csv: kind: mirror-orphan note: wasm's inequality reads intro's copy by URL