From 1f1f7ed5675461d7ae42d2ac543c14c287934a96 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Wed, 12 Aug 2026 09:39:45 +1000 Subject: [PATCH] PLAN: bring the record up to date now the P3 fold is complete MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seven statements in PLAN.md and one comment in migration.yml still described the six high_dim_data files as unrepointed. They were true when written on 2026-08-10 and false by the following afternoon, when PR set C landed and #69 flipped the records. PLAN.md's own line 9 is the diagnosis: "every figure below that restates one is a copy that can drift." Figures re-read from audit.json (2026-08-11), not carried forward: - the headline: 18 of 41 migrated / 23 to go / 5 URL forms -> **24 of 41 / 17 to go / 4 URL forms** - rule 2: "8 of the 23 remaining datasets have two consuming repos" -> **2 of 17**, both intro + wasm, both in step 4 - Track A: "17, of which 9 remain" -> **3 remain**; usa-gini's SCF dependency is discharged - Phase 6: 21 non-.yml files / 18 manifests -> **27 / 24** - Phase 9: 23 datasets to repoint -> **17** - the CATALOG-vs-`migrated` paragraph explained a six-file gap that has since closed; it now explains when the gap appears rather than asserting one Two boxes were unchecked for work that had landed — the 28 consuming repoints (Phase 3) and P3 itself (Phase 8). Both are ticked, with the four repoint PRs named, and P3 gains the record of its consumer half: the C0 -> C1 -> C2 ordering, the flip as its acceptance test (dry-run `landed` -> exit 1 with 6 warnings, `repointed` -> exit 0), and the independent validation against QuantEcon/workspace-lectures#36. Two things worth more than the arithmetic: **Rule 6's tables are now labelled as the pre-fold record.** The "Current host" column said `media` for four `data.ipynb:37` rows that have read raw.githubusercontent since 2026-08-11 — a table titled "current" that describes a state no repo is in is worse than a stale number, because it reads as a description of today. It is the worked example now, not open work. **Rule 2 gains the distinction the audit cannot make.** "Two consuming repos" is `SCAN_REPOS`' count, not the consumer set: both remaining pairs have five reference-holders each, and three of the five are invisible to every audit run. That gap is exactly what rule 1 exists for, and rule 2 previously read as though the dashboard's number were the whole answer. P3's "three things it proved" list is now five, adding the two findings that outlived the step: the `# i18n` markers were never exercised, so the hand-diff is still what protects a localisation; and the translation sync is `.md`-only, so no hand-localised `_static` asset can be carried by it (QuantEcon/action-translation#271). Also drops migration.yml's verbatim quote of a render_audit.py label that #70 deleted — a quoted string that outlives its source is an unfindable false record. Strict audit green. --- PLAN.md | 42 ++++++++++++++++++++++++++---------------- migration.yml | 25 ++++++++++++++++--------- 2 files changed, 42 insertions(+), 25 deletions(-) diff --git a/PLAN.md b/PLAN.md index 06332d2..2d40c1f 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1,10 +1,10 @@ # PLAN — `data-lectures` (formerly `QuantEcon/data`) -**Status:** active roadmap (last updated 2026-08-10) — **the repo is LIVE**: the first repoint merged 2026-07-17 (P1, `lingcod_msy_recovery.csv` → `msy_fishery`), so published filenames are an API from here on +**Status:** active roadmap (last updated 2026-08-12) — **the repo is LIVE**: the first repoint merged 2026-07-17 (P1, `lingcod_msy_recovery.csv` → `msy_fishery`), so published filenames are an API from here on -**Where the numbers stand (`audit.json`, 2026-08-10):** 18 of 41 static datasets migrated and repointed, 23 to go; 22 lectures still fetch live API data; 26 committed orphans; 0 legacy-repo references; 5 URL forms in use. +**Where the numbers stand (`audit.json`, 2026-08-11):** 24 of 41 static datasets migrated and repointed, 17 to go; 22 lectures still fetch live API data; 26 committed orphans; 0 legacy-repo references; 4 URL forms in use. -**`CATALOG.md` says 24 datasets, and both numbers are right.** `migrated` counts datasets whose *consumers* read this repo; the catalog counts datasets that *live* here. The six `high_dim_data` files landed 2026-08-10 at `status: landed` with `consumers: []`, so they are published and documented but nothing reads them yet — they close the gap only when PR set C repoints them. Expect the same six-file gap between the two figures for the duration of any wave that lands ahead of its repoints, which is the convention here. +**`CATALOG.md` and `migrated` both say 24 today, and that agreement is temporary.** They count different things: `migrated` counts datasets whose *consumers* read this repo; the catalog counts datasets that *live* here. They agree only when no wave is in flight. The P3 fold is the worked example — the six `high_dim_data` files landed 2026-08-10 at `status: landed` with `consumers: []`, opening a six-file gap that closed on 2026-08-11 when PR set C repointed them and [#69](https://github.com/QuantEcon/data-lectures/pull/69) flipped the records. Expect that gap again for the duration of any wave that lands ahead of its repoints, which is the convention here. Every figure on that line comes from `stats` in the generated `audit.json`, and every figure below that restates one is a copy that can drift — as all seven of them had by 2026-08-07, each understating progress by three repoint sets. Re-read them from `audit.json` before quoting them, and prefer citing the generated file over this document. @@ -54,13 +54,13 @@ Six rules learned the hard way, three of them the hard way twice. Rules 1-3 are So the general rule "delete the lecture repo's own copy in the same repoint PR" holds **only** when no sibling reads that copy. Where one does, the sibling's repoint must land **first or together**, and the deletion goes in the same set — never in an earlier PR with the sibling's fix scheduled later. -This affects every `intro` + `wasm` dataset, which is all 8 of the multi-consumer files remaining below. +This affects every `intro` + `wasm` dataset, which is both of the multi-consumer files remaining below. **The consumer set is the org, not `manifest.yml` and not `SCAN_REPOS`.** A repo that fetches another repo's committed blobs by URL is a rule-1 consumer regardless of how its content is produced, so the sweep before a deletion must enumerate the organisation. Three classes sit outside this repo's eight scanned repos and have each already been missed once: | Class | Repos | Why it is missed | | --- | --- | --- | -| **Translations** | five of the six live editions read another repo's blobs — everything except `lecture-python-programming.ml` | excluded from `SCAN_REPOS` by decision (`scripts/build_audit.py:46-47`), so no audit run can ever see them. `lecture-intro.zh-cn` alone holds 7 reads of the `high_dim_data` six | +| **Translations** | five of the six live editions read another repo's blobs — everything except `lecture-python-programming.ml` | excluded from `SCAN_REPOS` by decision (`scripts/build_audit.py:46-47`), so no audit run can ever see them. `lecture-intro.zh-cn` alone held 7 reads of the `high_dim_data` six, repointed by hand in QuantEcon/lecture-intro.zh-cn#292 | | **Generated mirrors** | `lecture-python-intro.notebooks` | auto-published, so it self-heals on the next publish — but only *after* one | | **Course forks and canaries** | `tom-econ370-2025`, `test-actions-lecture-intro` | not in any manifest, publish their own Pages sites, and no CI in the family covers them | @@ -70,7 +70,9 @@ Sweep by cloning and grepping, not with `gh search code` on a URL — code searc The strict audit has **no green state for a partially-repointed dataset**. `scripts/build_audit.py` fails a record marked `pending`/`landed` while any consumer already reads data-lectures, *and* fails one marked `repointed`/`final` while any consumer still does not. That is deliberate — it is what makes the tracker trustworthy — but it means a dataset with two consuming repos cannot be moved one repo at a time without the drift alarm firing in the gap. -**8 of the 23 remaining datasets have two consuming repos, and every one of them is `lecture-python-intro` + `lecture-wasm`** — the six `high_dim_data` files plus `life-expectancy-vs-gdp-per-capita.csv` and `usa-gini-nwealth-tincome-lincome.csv`. There is no other cross-series coupling left; the last one was the P2 `pandas_panel` trio, already done. (`graph.txt` is Track A but single-consumer: only `lecture-wasm` reads intro's committed copy.) +**2 of the 17 remaining datasets have two consuming repos, and both are `lecture-python-intro` + `lecture-wasm`** — `life-expectancy-vs-gdp-per-capita.csv` and `usa-gini-nwealth-tincome-lincome.csv`. (Six of the eight this line used to name were the `high_dim_data` files, repointed 2026-08-11.) There is no other cross-series coupling left; the last one was the P2 `pandas_panel` trio, already done. (`graph.txt` is Track A but single-consumer: only `lecture-wasm` reads intro's committed copy.) + +**"Two consuming repos" is the audit's count, not the consumer set.** `SCAN_REPOS` is the eight Python-family repos, so a dataset the dashboard shows with two consumers may have four or five in reality. Measured 2026-08-12, both remaining pairs are read by **five** reference-holders each: intro, `lecture-wasm`, `lecture-intro.zh-cn`, `QuantEcon/test-actions-lecture-intro`, and the generated `lecture-python-intro.notebooks` mirror. The last three are invisible to every audit run — see rule 1. Practically: one branch name across data-lectures + every consuming repo, PRs opened together, lecture repoints merged first, then the `migration.yml` flip to `repointed` — that last push is what re-runs the audit, and by then reality and the tracker agree. @@ -142,7 +144,9 @@ Phase 4 inherits the requirement: `data.quantecon.org` must serve `access-contro Both hosts send `access-control-allow-origin: *`, so this is **host routing, not CORS** — a distinct failure from rule 5, and it bites CPython consumers too, not only the browser. -This matters for exactly one piece of remaining work, and it matters a lot. `high_dim_data` tracks `*.csv` **and** `*.dta` under a blanket LFS rule, so **every** consuming lecture reads its datasets through the media host today. The storage decision lands those six datasets here as **plain git** (both SCF minis fit under the 100 MiB blob limit). After the fold the media host will 404 for them, so **every consuming read must change host as well as org and repo** — a mechanical org/repo swap that preserves the host breaks all of them. +**The one piece of work this rule was written for is DONE — everything from here to the end of the rule is the record of it, kept because the mechanism recurs.** `high_dim_data` tracked `*.csv` **and** `*.dta` under a blanket LFS rule, so every consuming lecture read its datasets through the media host. The storage decision landed those six datasets here as **plain git** (both SCF minis fit under the 100 MiB blob limit), which made the media host 404 for them, so **every consuming read had to change host as well as org and repo** — a mechanical org/repo swap preserving the host would have broken all of them. All 28 were repointed on 2026-08-11 and the flip landed as [#69](https://github.com/QuantEcon/data-lectures/pull/69); the acceptance greps at the end of this rule both return nothing today. + +The rule itself still binds on any future fold out of an LFS-tracked repo, and the two CI assertions it earned ([#55](https://github.com/QuantEcon/data-lectures/pull/55)) are permanent. Read the tables below as the worked example, not as open work. **28 source reads are affected across four repos: 22 on the media host, 6 on the `github.com/*/raw/` redirect form.** The `github.com///raw/…` form is a *smart* redirect that routes per path by LFS status, so those 6 survive an org/repo/path swap with no host decision; **the other 22 must change host as well.** Two of the four consumers are invisible to every audit run: `lecture-intro.zh-cn` is excluded from `SCAN_REPOS` by decision (`scripts/build_audit.py:48-57`) — see rule 1 — and `QuantEcon/test-actions-lecture-intro`, the `quantecon/actions` canary, is not a Python-family repo and was never in scope for it. @@ -156,9 +160,9 @@ Counting by repo. Two superseded figures are recorded here because **each has al | `test-actions-lecture-intro` | 5 | 2 | 7 | **nothing** | | **all** | **22** | **6** | **28** | | -Line numbers below were measured against each repo's `main` on **2026-08-11**. Treat them as a snapshot, not a fact — `lecture-intro.zh-cn`'s drift by +1 in the course of one afternoon, from a `[translation-sync]` PR that never touched a data-read line. **Re-derive immediately before editing**, with `bin/zh-fold-lines` in `QuantEcon/workspace-lectures` for zh-cn (no clone needed) and a plain `grep -rn high_dim_data lectures/` for the rest. The bare needle is deliberate here and in the acceptance check below — see the note there. +**The table below is the PRE-FOLD state — the reads as they stood immediately before PR set C, with the host each was on.** It is kept as the record of what was changed, not as a description of any repo today: all 28 now read `raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/`, and the four `data.ipynb:37` rows in particular have said `raw` since 2026-08-11. Line numbers were measured against each repo's `main` on that date and several have already moved. **Re-derive immediately before editing anything** — `lecture-intro.zh-cn`'s drifted by +1 in the course of one afternoon, from a `[translation-sync]` PR that never touched a data-read line, and its `inequality.md` moved by −5 later the same day. Use `bin/zh-fold-lines` in `QuantEcon/workspace-lectures` for zh-cn (no clone needed) and a plain `grep -rn high_dim_data lectures/` for the rest. The bare needle is deliberate here and in the acceptance check below — see the note there. -| Repo | File | Lines | Current host | +| Repo | File | Lines | Host **before** the fold | | --- | --- | --- | --- | | `lecture-python-intro` | `lectures/heavy_tails.md` | 827, 854, 855, 879 | media | | `lecture-python-intro` | `lectures/_static/lecture_specific/inequality/data.ipynb` | 37 | media | @@ -209,7 +213,7 @@ The remaining work decomposes by **consuming series** rather than by hosting pat | Track | Datasets | Coupling | Blocked on | | --- | --- | --- | --- | -| **A — `intro` + `wasm`** | 17, of which **9 remain**: the 6 `high_dim_data` files, `life-expectancy…`, `usa-gini…`, `graph.txt` (the 8 intro statics are done) | **paired — repoint together, always**, and `lecture-intro.zh-cn` is a third consumer of 8 of the 9 | nothing to start; `usa-gini` needs the SCF files first | +| **A — `intro` + `wasm`** | 17, of which **3 remain**: `life-expectancy…`, `usa-gini…`, `graph.txt` (the 8 intro statics and the 6 `high_dim_data` files are done) | **paired — repoint together, always**; `lecture-intro.zh-cn` and `test-actions-lecture-intro` are third and fourth consumers of the two CSVs | nothing — `usa-gini`'s SCF dependency was discharged by P3 | | **B — `python.myst`** | 7: `maketable1/2/4.dta`, `fp.dta`, `hansen_singleton_1982/1983_data.csv`, `NEWQDATA.csv` | none | nothing | | **C — `advanced.myst`** | 6: `dataBHS.mat`, `acs_data_summary.csv`, `bbh` ×2, `fred_data.csv`, `hansen_jagannathan_1991_data.json` | none | nothing (builder recovery is in-wave work, not a gate) | | **D — `programming`** | 1: `test_pwt.csv` | none | nothing — a single-PR track | @@ -265,9 +269,9 @@ Only one file genuinely forces LFS, and it is not a dataset: - [x] Add `sources/` for builder inputs, with `sources/README.md` as the **audit trail**: a **`## ` section per committed file**, recording where it came from, when, its licence, the upstream identifier (DOI where one exists), its `sha256`, and which builder consumes it. That shape is **enforced, not conventional** — `check_sources()` splits the README on `## ` headings and reads the first 64-hex token in each filename-shaped section, so a file documented as a row in a shared table has no recorded hash and fails the required check. (This line previously said "one row per committed file", which was written before the format existed and would now send you into a red build.) A file in `sources/` is not a published dataset and gets no sidecar manifest — this README is its provenance record. Landed in [#63](https://github.com/QuantEcon/data-lectures/pull/63), which also made the README **load-bearing rather than documentary**: `check_consumed_files.py` now asserts that every file here is captured by the LFS rule and hashes to a `sha256` recorded under a `## ` heading, and fails on a README entry naming a file that is not there. It reads the pointer's `oid` rather than the object, so it verifies ~100 MiB under `lfs: false` at zero bandwidth - [x] Per-path LFS via `.gitattributes`, scoped to `sources/**` only — never a blanket rule like `high_dim_data`'s `*.csv` **and** `*.dta` (data#1). Landed in [#57](https://github.com/QuantEcon/data-lectures/pull/57), with `sources/README.md` excluded so the audit trail stays readable text -- [x] Fold in `high_dim_data` content (data#2; coordinate with meta#337 for consuming-lecture repoints) — the **data** side is done: six datasets plus six manifests and six `migration.yml` records at `landed` in [#62](https://github.com/QuantEcon/data-lectures/pull/62), `sources/SCF_plus.dta` in [#63](https://github.com/QuantEcon/data-lectures/pull/63). The consuming repoints are the open box below +- [x] Fold in `high_dim_data` content (data#2; coordinate with meta#337 for consuming-lecture repoints) — the **data** side is done: six datasets plus six manifests and six `migration.yml` records at `landed` in [#62](https://github.com/QuantEcon/data-lectures/pull/62), `sources/SCF_plus.dta` in [#63](https://github.com/QuantEcon/data-lectures/pull/63). The consuming repoints landed 2026-08-11 — see the box below - [x] ~~**Repoint `generating_mini.md`'s input URL**~~ — **superseded, do not do this.** [#14](https://github.com/QuantEcon/data-lectures/issues/14) settled the question this box was waiting on, and it settled it the other way: the builder lands as **provenance**, `builder_status: committed-frozen`, committed verbatim and not edited at all. The worry behind the box was that archiving `high_dim_data` while `pd.read_stata('https://github.com/QuantEcon/high_dim_data/…')` stands re-introduces a retired-repo dependency. It does not, because the rule requiring a builder to read from `sources/` binds builders that **run** — a never-executed record has no live dependency to re-introduce, and editing it destroys the one property that makes it worth keeping. The substitution is recorded as prose in `sources/README.md`: the input is now committed at `sources/SCF_plus.dta`, byte-identical to what that URL served. The rule still binds fully on any builder that does run -- [ ] **Repoint all 28 consuming reads, moving the 22 media-host ones off `media.githubusercontent.com`**, in the same set as the fold — they are LFS-tracked in `high_dim_data` and land here as plain git, so the media host 404s for them and changing only org and repo breaks every read. The reads span **four** repos, not three: `lecture-python-intro` (7), `lecture-wasm` (7), `lecture-intro.zh-cn` (7) and `QuantEcon/test-actions-lecture-intro` (7) — the last two by hand, neither visible to any CI. See **repoint rule 6** for the enumerated reads, the acceptance check, and why CI does not cover it. *(The "21 reads / three repos" figure this line carried is the first three repos only; the canary was found later and has been mistaken for out-of-scope once already.)* +- [x] **Repoint all 28 consuming reads, moving the 22 media-host ones off `media.githubusercontent.com`** — **done 2026-08-11.** They were LFS-tracked in `high_dim_data` and landed here as plain git, so the media host 404s for them and changing only org and repo would have broken every read. The reads span **four** repos, not three: `lecture-python-intro` (7, QuantEcon/lecture-python-intro#832), `lecture-wasm` (7, QuantEcon/lecture-wasm#60), `lecture-intro.zh-cn` (7, QuantEcon/lecture-intro.zh-cn#292) and `QuantEcon/test-actions-lecture-intro` (7, QuantEcon/test-actions-lecture-intro#53) — the last two hand-written and hand-verified, neither visible to any CI. All 28 landed on `raw.githubusercontent.com`, including the 6 redirect-form reads that would have survived a bare org swap: one spelling across four repos, so the "harmonise these two forms" class of fix that broke wasm in [#46](https://github.com/QuantEcon/data-lectures/issues/46) cannot recur. See **repoint rule 6** for the enumeration and the acceptance check, both of which return nothing today. *(The "21 reads / three repos" figure this line once carried is the first three repos only; the canary was found later and has been mistaken for out-of-scope once already.)* - [x] Set `lfs: false` on **both** workflows that check this repo out, not only the Pages job — landed in [#57](https://github.com/QuantEcon/data-lectures/pull/57). `.github/workflows/audit-dashboard.yml:51` (push to `main` plus weekly, and its paths filter includes `migration.yml` and `lectures/*.yml`, so the fold PR's own files trigger it) and `.github/workflows/consumed-file-check.yml:26` (**every** pull request). `lfs: false` is the *assertion* that nothing published is an LFS object, not a saving: a mis-tracked `lectures/` file then hashes as its pointer and the required check goes red, where `lfs: true` would fetch the real bytes, pass green, and publish a file that resolves only from Pages. On the quota, measured 2026-08-10: the org's net LFS charge for all of 2026 is $0.04 on $1.04 gross, and `high_dim_data`'s bandwidth is 0 GB since June — so the outage scenario is not currently binding. The mechanism is real, though: anonymous public downloads bill the repository owner with no open-source exemption, forks count against the parent, and a **$0 budget blocks LFS downloads** for the rest of the month rather than billing them - [x] Record in `sources/README.md` that `SCF_plus.dta` is 103,934,093 B — **923,507 B, or 0.88%, under GitHub's hard 104,857,600 B blob limit**. It must stay LFS-tracked permanently; an upstream vintage 1% larger could not be pushed as plain git at all. Landed in [#63](https://github.com/QuantEcon/data-lectures/pull/63), and the margin is why the `git check-attr` precondition is a gate rather than a formality: below the limit a mis-scoped rule does not error, so the push succeeds as plain git and the blob is in history permanently @@ -299,7 +303,7 @@ Full automation: ### Phase 6 — Metadata backfill for existing holdings -- [ ] Manifest per dataset for the files now in `lectures/`: source, license, retrieval date, schema, consumers, provenance class. Schema sketched in `manifest-schema.yml` (Phase 2); backfill is per-file work gated on the license check below. **Largely done** — 21 non-`.yml` files, 18 manifests; the only dataset still lacking one is `business_cycle_data.csv` (`business_cycle_info.md` and `business_cycle_metadata.md` are prose, not datasets) +- [ ] Manifest per dataset for the files now in `lectures/`: source, license, retrieval date, schema, consumers, provenance class. Schema sketched in `manifest-schema.yml` (Phase 2); backfill is per-file work gated on the license check below. **Largely done** — 27 non-`.yml` files, 24 manifests; the only dataset still lacking one is `business_cycle_data.csv` (`business_cycle_info.md` and `business_cycle_metadata.md` are prose, not datasets, so the gap is 1 file and not 3) - [ ] Classify: the 8 static intro files are author-assembled or verbatim; `business_cycle_data.csv` is the one dynamic snapshot and needs its cadence declared - [ ] Licence check **per source**, not per file: the question is *"may this source be cached and served publicly, with attribution?"* — a cheap binary gate (`redistribution: permitted | restricted`, see AGENTS.md "Licensing and attribution"), a fast yes for public data sources. Two sources already answered: World Bank is **CC BY-4.0** (`business_cycle_metadata.md`, the model for what a manifest should capture) and RAM Legacy is **CC BY 4.0** (established against its Zenodo DOI record, P1). The remaining sources need the equivalent established by hand @@ -325,16 +329,22 @@ The first end-to-end deployment: one dataset per hosting pattern, each the harde - [x] **P1 — local-path static**: `lingcod_msy_recovery.csv` (`msy_fishery`, intro). Tests: single-PR green build under `-nW`, Colab-unchanged download, catalog metadata for an author-assembled file. **Complete 2026-07-17** — data half #12, repoint QuantEcon/lecture-python-intro#792 (lecture build green in the single repoint PR); served URL verified byte-identical to the manifest `sha256`; Colab holds by construction (the lecture now reads a public URL, where the old relative path was exactly what broke downloaded notebooks); metadata findings recorded in meta#338. **The repo is live from this merge** — the pyodide/CORS check below remains open - [x] **P2 — cross-series shared static**: the `pandas_panel` trio (`realwage.csv`, `countries.csv`, `employ.csv`), consumed by programming **and** python.myst. Tests: flat namespace with two consuming series, one data PR updating two lecture repos; retires 5 of the 8 legacy-repo references as a side effect. **Complete 2026-07-17** — data half #17, repoints QuantEcon/lecture-python-programming#578 and QuantEcon/lecture-python.myst#973; both lecture repos' own stale copies deleted in the repoint PRs. Lifecycle recorded in `migration.yml` -- [ ] **P3 — external-repo static, LFS → plain git**: the `heavy_tails` set (Forbes ×2, cities ×2) plus the SCF pair from `high_dim_data`. **Reframed 2026-08-07** — the Phase 3 storage decision made the published tree 100% plain git, so P3's original tests ("served URL makes the raw-vs-media trap invisible", "Pages handles LFS objects with `lfs: true`") are unrunnable by construction. What P3 now tests: the **host migration** off `media.githubusercontent.com` (repoint rule 6) across four consuming repos, two of which no CI can see; per-path LFS confined to `sources/`; and builders (`webscrape_forbes.ipynb`, `generating_mini.md`) migrating alongside their data. Note P3 **deletes nothing** — neither `intro` nor `wasm` holds a copy of the six files, and archiving `high_dim_data` preserves serving on both hosts, so rule 3's phase 2 does not apply and the set is fully reversible. +- [x] **P3 — external-repo static, LFS → plain git**: the `heavy_tails` set (Forbes ×2, cities ×2) plus the SCF pair from `high_dim_data`. **Complete 2026-08-11. Reframed 2026-08-07** — the Phase 3 storage decision made the published tree 100% plain git, so P3's original tests ("served URL makes the raw-vs-media trap invisible", "Pages handles LFS objects with `lfs: true`") are unrunnable by construction. What P3 now tests: the **host migration** off `media.githubusercontent.com` (repoint rule 6) across four consuming repos, two of which no CI can see; per-path LFS confined to `sources/`; and builders (`webscrape_forbes.ipynb`, `generating_mini.md`) migrating alongside their data. Note P3 **deletes nothing** — neither `intro` nor `wasm` holds a copy of the six files, and archiving `high_dim_data` preserves serving on both hosts, so rule 3's phase 2 does not apply and the set is fully reversible. + + **Data half complete 2026-08-10** — [#62](https://github.com/QuantEcon/data-lectures/pull/62) landed the six datasets, six manifests at `consumers: []`, six `migration.yml` records at `landed` and both frozen builders; [#63](https://github.com/QuantEcon/data-lectures/pull/63) landed `sources/SCF_plus.dta` and the `sources/` hash gate. All six verified byte-identical to the upstream LFS objects and confirmed serving real bytes (not pointer text) from `raw` and Pages, with the media host now 404ing for them — rule 6 confirmed in production. + + **Consumer half complete 2026-08-11**, in the order the ordering constraint required: C0 (QuantEcon/lecture-intro.zh-cn#291, seven `# i18n` markers) → C1 (QuantEcon/lecture-intro.zh-cn#292, zh-cn's 7 repoints) → C2 (QuantEcon/lecture-python-intro#832, QuantEcon/lecture-wasm#60, QuantEcon/test-actions-lecture-intro#53, three independent branches) → the flip ([#69](https://github.com/QuantEcon/data-lectures/pull/69)) → dashboard cleanup ([#70](https://github.com/QuantEcon/data-lectures/pull/70)). `high_dim_data` was then **archived, not deleted**, and still serves on both hosts. Independently validated in a fresh session against QuantEcon/workspace-lectures#36: the consumer set was complete (an org-wide tarball-and-content sweep of all 277 repos found no fifth consumer), all six datasets verify three-way byte-for-byte, and every dataset-driven figure on the live intro site is pixel-identical to the prior publish. + + **The flip was the acceptance test, and it measured as one.** Dry-run locally in both directions before pushing: `landed` → exit 1 with 6 warnings; `repointed` → exit 0. So the red window was real, opened when the last consuming PR merged, and closed with the flip. **Do this both-directions dry-run on every future wave** — it converts "same-day, trust me" into a measurement. - **Data half complete 2026-08-10** — [#62](https://github.com/QuantEcon/data-lectures/pull/62) landed the six datasets, six manifests at `consumers: []`, six `migration.yml` records at `landed` and both frozen builders; [#63](https://github.com/QuantEcon/data-lectures/pull/63) landed `sources/SCF_plus.dta` and the `sources/` hash gate. All six verified byte-identical to the upstream LFS objects and confirmed serving real bytes (not pointer text) from `raw` and Pages, with the media host now 404ing for them — rule 6 confirmed in production. **Open: the 28 consuming repoints and the `migration.yml` flip.** Three things P3 has already proven that were not on its test list: a `constructed` dataset's builder must land in the *same* PR as the data, because `check_consumed_files.py` asserts the `builder:` path resolves; `builders/README.md`'s coverage table is a real coverage report and goes stale silently; and the plain-git decision costs ~10 MB of packed history for 110 MB of working tree, since CSV compresses 5-22× + Five things P3 proved that were not on its test list. A `constructed` dataset's builder must land in the **same** PR as the data, because `check_consumed_files.py` asserts the `builder:` path resolves. `builders/README.md`'s coverage table is a real coverage report and goes stale silently. The plain-git decision costs ~10 MB of packed history for 110 MB of working tree, since CSV compresses 5-22×. The **C0 → C1 → C2 ordering worked and proved less than it looks like** — the sync PR it was designed to defuse (QuantEcon/lecture-intro.zh-cn#293) touched zero data-read lines, zero `# i18n` markers and zero protected localisations, but nothing ever asked the model to rewrite those cells, so the markers remain unexercised, prompt-level protection and **the hand-diff is what protects a localisation**. And the translation sync is **`.md`-only**, so no hand-localised `_static` asset can be created, updated or repaired by it — every `data.ipynb` copy had to be repointed by hand in all four repos, filed upstream as QuantEcon/action-translation#271 - [ ] **P4 — dynamic snapshot twin**: `UNRATE`, consumed today by 4 lectures across 3 repos via 2 access methods. Tests: the full dynamic template — manifest, four-stage builder, refresh-as-PR, canary catching an induced failure — plus the documented live-call ↔ snapshot switch mechanism - [ ] Verify each migrated URL with a pyodide/JupyterLite fetch (CORS, meta#143) - [ ] Fold every validated decision into the draft `styleguide/datasets.md` (manual#108) as it is proven ### Phase 9 — Adoption (broad sweep — the step that stalled in Feb 2025) -- [ ] Repoint the remaining consuming lectures as datasets land here (data#4) — **23 datasets**, organised as tracks A–D above. Mechanical, but see "Repoint rules": repoint all consumers of a dataset together, and never delete a copy a sibling repo reads +- [ ] Repoint the remaining consuming lectures as datasets land here (data#4) — **17 datasets**, organised as tracks A–D above. Mechanical, but see "Repoint rules": repoint all consumers of a dataset together, and never delete a copy a sibling repo reads - [ ] Remove lecture repos' duplicate copies as each repoint merges (tracked with the orphan sweep in meta#337) — 26 orphans today, Track X. Note the wasm mirror copies are only safe to delete **after** wasm reads data-lectures directly, not before - [ ] Intake rule for migrations: constructed datasets arrive **with their builders**; the 5 known constructed-but-unscripted files (`hansen_jagannathan_1991_data.json`, `fred_data.csv`, the two `bbh` extracts, `acs_data_summary.csv`) need their pipelines recovered or rewritten — recorded as QEP follow-ups per meta#338 - [ ] Graduate the convention to a QEP and merge manual#108, with the remaining sweep as its rollout checklist diff --git a/migration.yml b/migration.yml index 0d32b57..d48dfdb 100644 --- a/migration.yml +++ b/migration.yml @@ -314,21 +314,28 @@ datasets: cutover: null # --- P3: the high_dim_data fold ---------------------------------------- - # Six datasets folded in from QuantEcon/high_dim_data, which is retired - # (archive, never delete) once the repoints land. `prior_pattern: external` - # is the audit's own label for these reads — "external data repo, - # QuantEcon/high_dim_data via raw and media (LFS) hosts". + # COMPLETE 2026-08-11. Six datasets folded in from QuantEcon/high_dim_data, + # which is now archived (never deleted) and still serving on both hosts. + # `prior_pattern: external` is the audit's own label for these reads; the + # human-readable text behind that label lives in render_audit.py and is + # deliberately NOT quoted here — the copy that used to sit on this line + # ("...via raw and media (LFS) hosts") outlived the string it quoted by one + # PR (#70) and became a false record of a label nobody could grep for. # - # They land at `landed` with `repoints: []` and their manifests at - # `consumers: []`, which is honest: no lecture reads a data-lectures URL for + # They landed at `landed` with `repoints: []` and their manifests at + # `consumers: []`, which was honest: no lecture read a data-lectures URL for # these until PR set C. The window between those repoints and the flip to # `repointed` is RED by design — build_audit fails on a landed dataset whose - # consumers read this repo — so the flip is a same-day action. + # consumers read this repo — so the flip is a same-day action. Measured on + # this wave: `landed` exits 1 with 6 warnings, `repointed` exits 0. Dry-run + # both directions before pushing a flip; that is the wave's acceptance test. # # They were LFS-tracked upstream and are plain git here, so every consuming - # read must change HOST as well as org and repo (repoint rule 6). 28 reads + # read had to change HOST as well as org and repo (repoint rule 6). 28 reads # across four repos: lecture-python-intro, lecture-wasm, lecture-intro.zh-cn - # and test-actions-lecture-intro. + # and test-actions-lecture-intro — the last two invisible to every audit run, + # which is why all four consumers are recorded per manifest below rather than + # only the two the scan can see. forbes-global2000.csv: pilot: P3