Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ Sweep by cloning and grepping, not with `gh search code` on a URL — code searc

The strict audit has **no green state for a partially-repointed dataset**. `scripts/build_audit.py` fails a record marked `pending`/`landed` while any consumer already reads data-lectures, *and* fails one marked `repointed`/`final` while any consumer still does not. That is deliberate — it is what makes the tracker trustworthy — but it means a dataset with two consuming repos cannot be moved one repo at a time without the drift alarm firing in the gap.

**2 of the 17 remaining datasets have two consuming repos, and both are `lecture-python-intro` + `lecture-wasm`** — `life-expectancy-vs-gdp-per-capita.csv` and `usa-gini-nwealth-tincome-lincome.csv`. (Six of the eight this line used to name were the `high_dim_data` files, repointed 2026-08-11.) There is no other cross-series coupling left; the last one was the P2 `pandas_panel` trio, already done. (`graph.txt` is Track A but single-consumer: only `lecture-wasm` reads intro's committed copy.)
**2 of the 17 remaining datasets have two consuming repos, and both are `lecture-python-intro` + `lecture-wasm`** — `life-expectancy-vs-gdp-per-capita.csv` and `usa-gini-nwealth-tincome-lincome.csv`. (Six of the eight this line used to name were the `high_dim_data` files, repointed 2026-08-11.) There is no other cross-series coupling left; the last one was the P2 `pandas_panel` trio, already done. (`graph.txt` is **not** in this count and is no longer a Track A item at all — `lecture-wasm` used to read intro's committed copy, which is what made it look like a one-consumer dataset; that read is gone and it is now embedded in every consuming lecture. See the Track A row below.)

**"Two consuming repos" is the audit's count, not the consumer set.** `SCAN_REPOS` is the eight Python-family repos, so a dataset the dashboard shows with two consumers may have four or five in reality. Measured 2026-08-12, both remaining pairs are read by **five** reference-holders each: intro, `lecture-wasm`, `lecture-intro.zh-cn`, `QuantEcon/test-actions-lecture-intro`, and the generated `lecture-python-intro.notebooks` mirror. The last three are invisible to every audit run — see rule 1.

Expand Down Expand Up @@ -213,14 +213,16 @@ The remaining work decomposes by **consuming series** rather than by hosting pat

| Track | Datasets | Coupling | Blocked on |
| --- | --- | --- | --- |
| **A — `intro` + `wasm`** | 17, of which **3 remain**: `life-expectancy…`, `usa-gini…`, `graph.txt` (the 8 intro statics and the 6 `high_dim_data` files are done) | **paired — repoint together, always**; `lecture-intro.zh-cn` and `test-actions-lecture-intro` are third and fourth consumers of the two CSVs | nothing — `usa-gini`'s SCF dependency was discharged by P3 |
| **A — `intro` + `wasm`** | 17, **all done**. The last two CSVs landed as wave A4 ([#74](https://github.com/QuantEcon/data-lectures/pull/74), flipped in [#75](https://github.com/QuantEcon/data-lectures/pull/75)); `graph.txt` was never a migration — see below | — | — |
| **B — `python.myst`** | 7: `maketable1/2/4.dta`, `fp.dta`, `hansen_singleton_1982/1983_data.csv`, `NEWQDATA.csv` | none | nothing |
| **C — `advanced.myst`** | 6: `dataBHS.mat`, `acs_data_summary.csv`, `bbh` ×2, `fred_data.csv`, `hansen_jagannathan_1991_data.json` | none | nothing (builder recovery is in-wave work, not a gate) |
| **D — `programming`** | 1: `test_pwt.csv` | none | nothing — a single-PR track |
| **E — dynamic / live-API** | the UNRATE twin, then the 15 incidental API lectures | wasm is the forcing customer | [#14](https://github.com/QuantEcon/data-lectures/issues/14) schema decisions, [#26](https://github.com/QuantEcon/data-lectures/issues/26) fetch layer |
| **X — orphan sweep** | 26 committed orphans across 6 repos — dp 10, programming 5, wasm 5, intro 3, python.myst 2, `continuous_time_mcs` 1 | per repo | that repo's repoints landing first |
| **Y — consumer interface (`qeld`)** | the `qeld` package, Q1–Q7 of `PLAN-QELD-PACKAGE.md` — audit support, the package, pilots, then adoption by win; QEP graduation stays | — | nothing — re-scoped 2026-08-12 (D11): the DNS → custom domain → URL-sweep sequence this row used to carry is retired |

**`graph.txt` was closed out as a non-migration (2026-08-12).** It is synthetic teaching data — `provenance: toy`, null in every real provenance field — and the shortest-path exercise teaches its format by quoting the first line, so the data has to stay visible on the page. Hosting it here would have put a toy in a registry that exists to carry provenance. Instead `lecture-wasm` stopped fetching intro's committed copy over the network and embeds it with `%%file` like every sibling ([QuantEcon/lecture-wasm#63](https://github.com/QuantEcon/lecture-wasm/pull/63)), which retired the last cross-repo read of that blob anywhere in the organisation. `graph.txt` consequently no longer appears as a scanned dataset at all. Four repos embed it via `%%file` — intro, dp, jax and wasm — and two of those (intro, dp) also commit a copy the cell overwrites before reading, so those two are shadowed orphans; jax and wasm commit none, which is the cleaner shape. The remaining committed copies (`lecture-intro.zh-cn`, the canary, `lecture-python.zh-cn`, `lecture-dp.monorepo`, `ipynb_pdf_constructor`) are read by nothing. Intro's committed copy is now deletable as Track X — but the same blob sits in 8 repos and is regenerated at 17 `%%file` sites, including archived `.rst` ancestors that `gh search code` cannot see, so that deletion needs its own per-repo reader sweep rather than an org-wide sweep.

`lecture-dp`, `lecture-jax` and `continuous_time_mcs` are **not data consumers** — dp's 10 committed files are inherited orphans, jax embeds `graph.txt` via `%%file`, and continuous_time_mcs has one orphan scratch file. They appear only in Track X.

**Tracks A–D are independent of each other and can run in any order or in parallel.** The only hard dependencies in the whole programme are: `usa-gini-nwealth-tincome-lincome.csv` is built from `SCF_plus_mini.csv` (so it follows the SCF migration inside Track A); Track E's rollout needs its own template proven first; Track X follows its repo's repoints; and Track Y's adoption sweep (qeld Q7) is last.
Expand Down
21 changes: 15 additions & 6 deletions scripts/audit_annotations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,14 @@ datasets:
description: 100-node weighted digraph for the shortest-path problem
provenance: toy
note: >
maintained as %%file blocks in intro, dp and jax; lecture-wasm instead
fetches intro's committed copy by URL (requests.get), making that copy
load-bearing — it was merely "shadowed" in the 2026-07-15 audit
maintained as %%file blocks in intro, dp, jax and (since
QuantEcon/lecture-wasm#63) lecture-wasm — every consumer now embeds the
data in its own lecture, so no repo reads another's copy. wasm used to
fetch intro's committed copy by URL, which made that copy load-bearing;
with that read gone graph.txt no longer appears as a scanned dataset at
all. Two of the four repos also commit a copy the %%file cell overwrites
before reading (intro, dp) — those are shadowed; jax and wasm commit no
copy at all, which is the cleaner shape
hansen_jagannathan_1991_data.json:
description: Hansen–Jagannathan (1991) asset-returns bundle
provenance: constructed-lost
Expand Down Expand Up @@ -277,6 +282,13 @@ committed_unreferenced:
lecture-python.myst:lectures/web_graph_data.txt:
kind: shadowed
note: shadowed duplicate at lectures/ root
lecture-python-intro:lectures/graph.txt:
kind: shadowed
note: >
short_path regenerates it via %%file before reading it, so the committed
bytes are never consumed — the build prints "Overwriting graph.txt". It
became an orphan when QuantEcon/lecture-wasm#63 removed the only URL read
of it in the organisation; deletable as Track X
lecture-dp:lectures/graph.txt:
kind: shadowed
note: short_path regenerates it via %%file
Expand Down Expand Up @@ -307,9 +319,6 @@ committed_unreferenced:
lecture-dp:lectures/_static/lecture_specific/pandas_panel/realwage.csv:
kind: orphan
note: inherited copy; the consuming lectures live in other repos and now read data-lectures
lecture-wasm:lectures/graph.txt:
kind: mirror-orphan
note: wasm's short_path fetches intro's committed graph.txt by URL, not this copy
lecture-wasm:lectures/_static/lecture_specific/inequality/usa-gini-nwealth-tincome-lincome.csv:
kind: mirror-orphan
note: wasm's inequality reads intro's copy by URL
Expand Down
Loading