From 2dcfa6fd87f40cb1e4a1b9aaf9cce29c82681437 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Tue, 11 Aug 2026 15:29:19 +1000 Subject: [PATCH 1/2] Retire the dashboard's high_dim_data references now the fold is complete Every consumer was repointed and migration.yml flipped in #69, so the dashboard was the last place still describing a repo the lecture family no longer reads. It named it once on the overview page. Three changes, none of them a plain deletion: - audit_annotations.yml loses the six entries for the folded datasets. Annotations are curated judgment for refs that are NOT yet migrated; all six now carry full manifests, which is the authoritative record. The strict audit fails a ref with neither, so this is only safe *because* the manifests landed first. - render_audit.py's `external` pattern label is generalised rather than removed. It reads "QuantEcon/high_dim_data via raw and media (LFS) hosts", which describes one retired repo, but the entry is still live: migration.yml records `prior_pattern: external` for all six and render_audit.py:610 looks the label up to render it. Deleting it would have degraded those rows to a raw enum string. - The branch-pin finding is restated as history. It described SCF_plus_mini_no_weights.csv as "now reads main" on a repo we have since stopped reading entirely; it now says the file was folded in and every consumer repointed, so the pin is doubly retired. Verified locally: scan --strict exit 0 with zero warnings, render exit 0, and grep for high_dim_data across the three generated pages returns 0, down from 1. Close-out of QuantEcon/workspace-lectures#23 step 3. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/audit_annotations.yml | 27 --------------------------- scripts/render_audit.py | 6 ++++-- 2 files changed, 4 insertions(+), 29 deletions(-) diff --git a/scripts/audit_annotations.yml b/scripts/audit_annotations.yml index 8266c6d..ba87584 100644 --- a/scripts/audit_annotations.yml +++ b/scripts/audit_annotations.yml @@ -28,17 +28,6 @@ datasets: A migration candidate — it is read from lecture-python.myst main by raw URL, and the lecture documents every transformation and rebuilds the post-2000Q4 continuation from FRED - SCF_plus_mini.csv: - description: SCF+ wealth/income survey extract - provenance: constructed-committed - note: built by generating_mini.md (executable MyST) in high_dim_data from SCF_plus.dta - SCF_plus_mini_no_weights.csv: - description: SCF+ extract, no survey weights - provenance: constructed-committed - note: > - variant of the SCF_plus_mini pipeline. Was pinned to feature branch - update_scf_noweights (critical flag in the 2026-07-15 audit); reads - high_dim_data main since lecture-python-intro#793 acs_data_summary.csv: description: ACS occupation summary provenance: constructed-lost @@ -62,14 +51,6 @@ datasets: description: Hyperinflation tables, Sargent "Ends of Four Big Inflations" provenance: author-assembled note: hand-transcribed by authors - cities_brazil.csv: - description: Brazilian city populations - provenance: author-assembled - note: manual download from worldpopulationreview.com, documented in high_dim_data README - cities_us.csv: - description: US city populations - provenance: author-assembled - note: manual download from worldpopulationreview.com, documented in high_dim_data README dataBHS.mat: description: US consumption/income series, MATLAB replication bundle provenance: verbatim @@ -80,14 +61,6 @@ datasets: fig_3.xlsx: description: French Revolution fiscal data provenance: author-assembled - forbes-billionaires.csv: - description: Forbes billionaires list - provenance: constructed-committed - note: webscrape_forbes.ipynb in high_dim_data (Forbes API) - forbes-global2000.csv: - description: Forbes Global 2000 firm size - provenance: constructed-committed - note: webscrape_forbes.ipynb in high_dim_data (Forbes API) fp.dta: description: Treisman (2016) Russia billionaires replication data provenance: verbatim diff --git a/scripts/render_audit.py b/scripts/render_audit.py index ee0c677..fd77082 100644 --- a/scripts/render_audit.py +++ b/scripts/render_audit.py @@ -26,7 +26,7 @@ "own-repo": ("p-own", "own-repo URL", "fetches a file committed in its own repo via a GitHub raw URL"), "local-path": ("p-local", "local path", "pd.read_csv('…') relative path — no URL; breaks in Colab/download"), "sibling": ("p-sib", "sibling repo URL", "fetches another lecture repo's committed copy by URL"), - "external": ("p-ext", "external data repo", "QuantEcon/high_dim_data via raw and media (LFS) hosts"), + "external": ("p-ext", "external data repo", "reads a data file from another QuantEcon repo by URL"), "external-web": ("p-ext", "external web host", "fetches a data file from a non-GitHub host by URL"), "legacy": ("p-legacy", "legacy-repo URL", "fetches from the retired pre-MyST lecture-python repo"), "embedded": ("p-embed", "%%file embedded", "written by the lecture itself, then read back"), @@ -428,7 +428,9 @@ def what_changed(audit: dict) -> str: (lecture-python-programming#576, lecture-python.myst#968) removed every live reference.

Branch-pinned refs: 1 → 0.

The critical flag on SCF_plus_mini_no_weights.csv — pinned to a feature branch of -high_dim_data — was fixed by lecture-python-intro#793; it now reads main.

+the external repo it then lived in — was fixed by lecture-python-intro#793. That file has since been +folded into this repo (data-lectures#62) and every consumer repointed here, so the pin is doubly +retired.

lecture-wasm joined the audit — and changed one fact.

The WASM mirror carries its own copies of intro's data files (all unread — its lectures fetch intro's copies by URL), and its short_path fetches intro's committed From 8cedc1e2eeb3e661cb22a28aac82962f760ba632 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Tue, 11 Aug 2026 15:43:48 +1000 Subject: [PATCH 2/2] Describe `external` by what it excludes, not by whose repo it was Copilot caught that the generalisation in 2dcfa6f was still wrong, just one level up. classify_url reaches `external` by TWO routes: a non-QuantEcon org (build_audit.py:180-181) and a QuantEcon repo that is none of this repo, the consuming repo, a legacy name or a scanned sibling (:190-191). "another QuantEcon repo" describes only the second, which is the branch high_dim_data happened to take. The pattern is defined by what it excludes, so the label now says that: a GitHub repo outside the audited lecture set, any org. Added a comment naming both branches so the next edit does not have to re-derive the definition from classify_url -- this string has now been too narrow twice, first naming one repo and then one org. Note the description is currently dormant: PATTERN_META's label (index 1) renders 30 times in migration.html for `prior_pattern`, but the description (index 2) only surfaces when a live ref classifies external, and none does today. A wrong one would have sat unnoticed until the next external read appeared, which is the argument for fixing it rather than deferring. scan --strict exit 0 / 0 warnings; render exit 0; high_dim_data still 0 across the generated pages. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/render_audit.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/scripts/render_audit.py b/scripts/render_audit.py index fd77082..3116d32 100644 --- a/scripts/render_audit.py +++ b/scripts/render_audit.py @@ -26,7 +26,12 @@ "own-repo": ("p-own", "own-repo URL", "fetches a file committed in its own repo via a GitHub raw URL"), "local-path": ("p-local", "local path", "pd.read_csv('…') relative path — no URL; breaks in Colab/download"), "sibling": ("p-sib", "sibling repo URL", "fetches another lecture repo's committed copy by URL"), - "external": ("p-ext", "external data repo", "reads a data file from another QuantEcon repo by URL"), + # `external` is defined by what it EXCLUDES, and classify_url reaches it by + # two routes: a non-QuantEcon org (build_audit.py:180-181) and a QuantEcon + # repo that is none of this repo, the consuming repo, a legacy name or a + # scanned sibling (:190-191). Describe the exclusion, not whichever repo + # happens to be using it — this label has now been too narrow twice. + "external": ("p-ext", "external data repo", "reads a data file by URL from a GitHub repo outside the audited lecture set — any org"), "external-web": ("p-ext", "external web host", "fetches a data file from a non-GitHub host by URL"), "legacy": ("p-legacy", "legacy-repo URL", "fetches from the retired pre-MyST lecture-python repo"), "embedded": ("p-embed", "%%file embedded", "written by the lecture itself, then read back"),