Land wave C1 — the three advanced.myst datasets that needed no rename (Track C) by mmcky · Pull Request #92 · QuantEcon/data-lectures

mmcky · 2026-08-17T00:35:19Z

Wave C1 of Track C: the three lecture-python-advanced.myst datasets that needed no rename decision. Status is landed, not repointed — the repoint PR in advanced.myst follows this one.

Byte identity

All three land byte-identical to the blobs the lecture reads today, taken from git show HEAD:<path> in advanced.myst at 00057ba rather than from a working tree, and cross-checked against the worktree copies:

file	sha256	bytes
`bbh_macro_quarterly.csv`	`f14f4256434371877f28…`	32,217
`bbh_michigan_monthly.csv`	`567efe5a009a30bcfa32…`	12,196
`hansen_jagannathan_1991_data.json`	`c81333f372a23465a593…`	63,456

Each is recorded in its manifest's integrity.sha256, so the repoint that follows provably cannot change a figure.

Two builders recovered — the annotations they replace were wrong

Both bbh files were recorded as provenance: constructed-lost, "extraction not scripted". They are in fact fully reconstructible from the authors' Zenodo deposit (doi:10.5281/zenodo.10194324, CC BY 4.0), and the builders here reproduce their committed bytes byte for byte. Verified three times each, independently: in the authoring environment, re-run from scratch in a clean clone by a second reviewer, and again in this branch — cmp reports no difference every time, and running them in this branch leaves git diff -- lectures/ empty. They land as builder_status: committed.

builders/bbh_macro_quarterly.py extracts the 169 KB workbook from the 198.8 MB archive using HTTP range requests — four requests, ~296 KB, about six seconds — instead of downloading the archive.

Neither builder reads live FRED, and the reason is not the obvious one. These bytes are on a 2012 base and FRED rebased to 2017 — but the lecture takes log first differences and ratios, which are rebasing-invariant: under a pure rebasing 8 of the 9 VAR inputs are bit-identical and output_gap is invariant to 2.3e-14. The operative risk is upstream revisions, which is why the builders pin an immutable versioned DOI. An earlier draft of this manifest got that reasoning backwards and quantified it with a live-FRED delta table; the table has been removed because FRED's bulk endpoints do not answer from this environment, so no future validator could re-derive it.

The hansen bundle stays `unrecovered`, deliberately

Its construction is fully recovered — all 2,462 cells re-derive from Shiller's two workbooks and seven FRED series to within 4 ULP, with six of eleven columns bitwise identical end to end — but the bytes are not reproducible, so no builder can honestly claim to rebuild it. The manifest records the complete specification so Phase 9 becomes a finishing job rather than an investigation, including the unexplained level factor of 0.8938191876245914 on annual.consumption, which cancels out of everything the lecture prints (consumption enters only as c[t+1]/c[t]).

It is also the one file with no stale-serving grace period: it is already fetched over an own-repo raw URL at cell-execution time, so deleting it later breaks readers instantly rather than after a cache rebuild. Recorded in its migration.yml record.

A licence finding that needs recording elsewhere

bbh_michigan_monthly.csv lands redistribution: restricted. The Zenodo deposit declares CC BY 4.0, but four of its five columns are University of Michigan Surveys of Consumers aggregates, and Michigan's usage agreement states you "agree not to reproduce, retransmit, distribute, sell, publish, or broadcast" the data without written consent. The depositors are not the rights holder and the deposit records no consent. Handled as an inherited exposure per AGENTS.md — the file has been served publicly from advanced.myst since 2026-06-12, so this does not gate the migration — and registered for review.

#35's inventory currently records both bbh files as "already known CC BY 4.0", which is the opposite of this finding and should be corrected.

Also here, because CI enforces it

Three migration.yml records at landed; the three audit_annotations.yml entries retired (a manifest plus an annotation now fails the dual_recorded check added in #90); CATALOG.md regenerated. openpyxl is pinned — both new builders need it, and ames_house_prices.py and japan_population_by_age.py have needed it since they landed without it ever being declared, because CI never runs builders.

Verification

Strict audit exits 0 with all five problem keys empty. migration.yml parses to 36 datasets — 33 repointed, 3 landed — where a grep would say 34 repointed, the known prose-comment inflation. Manifest sha256 fields match the landed bytes. Both builders re-run clean in this branch.

Every manifest was drafted and then attacked by an independent reviewer that re-ran the measurements rather than reading the draft. That pass corrected a fabricated v1 version string, a series count of 29 that is 30, "≤2 ULP" claims that measure 4, a constant printed as 0.89381918762459 that reproduces 0 of 95 values, and the FRED delta table above. Those corrections are in this PR rather than in a follow-up.

… (Track C) Moves bbh_macro_quarterly.csv, bbh_michigan_monthly.csv and hansen_jagannathan_1991_data.json out of lecture-python-advanced.myst. All three land byte-identical to the blobs the lecture reads today, so the repoint that follows cannot change a figure: sha256 f14f4256…, 567efe5a… and c81333f3…, each checked against `git show HEAD:<path>` in advanced.myst at 00057ba and recorded in the manifests' integrity.sha256. Status is `landed`, not `repointed` — the lecture still reads its own copies. The repoint PR follows this one, in that order: advanced.myst sets execute_notebooks: "cache", so the repointed cells re-execute against whatever is on this repo's main at that moment. TWO BUILDERS RECOVERED, refuting the annotations they replace. Both bbh files were recorded as `constructed-lost` with "extraction not scripted". Both are in fact fully reconstructible from the authors' Zenodo deposit (doi:10.5281/zenodo.10194324), and the builders here reproduce their committed bytes BYTE FOR BYTE — verified by three end-to-end runs each, independently re-run in a clean clone and again in this branch, with `cmp` reporting no difference every time. They therefore land as `builder_status: committed`, not `unrecovered`. The macro builder pulls the 169 KB workbook out of the 198.8 MB archive with HTTP range requests — four requests, ~296 KB — rather than downloading the lot. Neither builder reads live FRED, and the reason is not the obvious one. The bytes are on a 2012 base and FRED has since rebased to 2017, but the lecture takes log first differences and ratios, which are rebasing-invariant — under a pure rebasing 8 of the 9 VAR inputs are bit-identical. The operative risk is upstream REVISIONS, which is why the builders pin the immutable versioned DOI. hansen_jagannathan_1991_data.json stays `unrecovered`, and the distinction is worth keeping. Its construction IS fully recovered — all 2,462 cells re-derive from Shiller's workbooks and seven FRED series to within 4 ULP, and six of eleven columns are bitwise identical end to end — but the BYTES are not reproducible, so no builder can claim to rebuild it. The manifest records the full specification so Phase 9 is a finishing job rather than an investigation, including the unexplained 0.8938191876245914 level factor on annual.consumption, which cancels out of everything the lecture prints. bbh_michigan_monthly.csv lands `redistribution: restricted`. The Zenodo deposit declares CC BY 4.0, but four of its five columns are University of Michigan Surveys of Consumers aggregates and Michigan's usage agreement forbids redistribution without written consent; the depositors are not the rights holder and the deposit does not record consent. Handled as an inherited exposure per AGENTS.md — the file has been served publicly since 2026-06-12 — and registered for licence review. #35's inventory currently records the opposite and needs the correction. Also here because CI enforces them: the three migration.yml records, the three audit_annotations.yml entries retired (a manifest and an annotation together fails the dual_recorded check added in #90), and CATALOG.md regenerated. openpyxl is pinned — both new builders need it, and ames_house_prices.py and japan_population_by_age.py have needed it since they landed without it ever being declared, because CI never runs builders. Verified: strict audit exits 0 with all five problem keys empty; both builders re-run in this branch leave `git diff -- lectures/` empty; the manifests' sha256 fields match the landed bytes; migration.yml parses to 36 datasets, 33 repointed and 3 landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR lands Wave C1 (Track C) datasets from QuantEcon/lecture-python-advanced.myst into the lectures/ published tree with new manifests, updates migration tracking to landed, retires now-stale audit annotations, and adds builders for the two BBH datasets that are reconstructible from the Zenodo replication package.

Changes:

Add lectures/ artifacts + sidecar manifests for bbh_macro_quarterly.csv, bbh_michigan_monthly.csv, and hansen_jagannathan_1991_data.json.
Add reconstructing builders for the two BBH datasets, and pin openpyxl in requirements.txt to support .xlsx ingestion.
Update migration.yml, retire corresponding scripts/audit_annotations.yml entries, and regenerate CATALOG.md.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
scripts/audit_annotations.yml	Removes stale annotations for newly-manifested datasets.
requirements.txt	Pins `openpyxl` needed by new/existing builders that read Excel.
migration.yml	Adds three Wave C1 dataset records at `status: landed`.
lectures/hansen_jagannathan_1991_data.json.yml	Adds manifest describing the constructed JSON bundle and provenance.
lectures/bbh_michigan_monthly.csv.yml	Adds manifest for Michigan monthly extract and licensing notes.
lectures/bbh_michigan_monthly.csv	Lands the CSV bytes into the published tree.
lectures/bbh_macro_quarterly.csv.yml	Adds manifest for BBH macro quarterly extract and builder linkage.
lectures/bbh_macro_quarterly.csv	Lands the CSV bytes into the published tree.
CATALOG.md	Regenerated catalog to include the new datasets/manifests.
builders/bbh_michigan_monthly.py	New builder that reconstructs the CSV from the DOI-pinned Zenodo deposit (range reads).
builders/bbh_macro_quarterly.py	New builder that reconstructs the CSV from the Zenodo deposit (range reads).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…in fetch() Copilot review of #92, all four comments valid — and three of them catch this PR breaking a convention it inherited from #91. migration.yml records these three as `landed`, so the lecture still reads its own copies. Under the definition settled in #91, `consumers` answers "what must be rebuilt if these bytes change", and a consumer not yet reading this repo carries a `note` saying so. All three entries were listed without one, which made CATALOG.md's "Used by" column imply a repoint that has not happened. Each now carries a note naming where the lecture actually reads from — a local `_static` path for the two bbh files, an own-repo raw URL for the hansen bundle — and the notes come out in the repoint PR. A worse instance of the same drift, which the review pointed at obliquely: hansen_jagannathan_1991_data.json.yml carried the retired "Every lecture that reads this file FROM THIS REPO" header — the exact wording #91 removed from all eight pre-existing manifests, reintroduced here in a new one because it was drafted from a pre-#91 template. Replaced with the current wording. A grep for that phrase now returns zero manifests again, which is the check worth keeping. builders/bbh_macro_quarterly.py: fetch() read from the ZipFile returned by _open_archive() without closing it, leaking the HTTP range reader's socket to the garbage collector. Now a context manager. This matters more than it looks because the builder is designed to be re-run — the byte-identity check reruns it on every verification pass. bbh_michigan_monthly.py already used `with open_deposit()` and needed no change, which is why the review scoped its comment to the one file. Verified after the changes: both builders still reproduce their committed bytes (git diff over lectures/ empty); the fixed builder raises no ResourceWarning under -W error::ResourceWarning; 36 manifests parse with 6 noted consumers; CATALOG.md regenerated and now renders three warning markers; strict audit exits 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Repoints subjective_beliefs_business_cycles.md and hansen_jagannathan_1991.md at QuantEcon/data-lectures, where the three files landed byte-identical in QuantEcon/data-lectures#92. Both reads are COLLAPSED rather than patched, which is the wave-B2' lesson. subjective_beliefs_business_cycles.md built its path from a shared `data_path` stem, so editing that line alone would have left both filenames appended to a `_static/lecture_specific/<lecture>/` segment that does not exist in the flat published tree. The two bbh files share that variable and therefore repoint together, in one edit. hansen_jagannathan_1991.md already read over a URL, so this is a URL-string edit rather than a path-to-URL conversion — but the old URL was split across three adjacent string literals, and the new one is a single line. That is deliberate: a whole-URL grep returned a confident ZERO against the wrapped form, which is how a sweep misses a live reference. It is now greppable in one piece. URL form is github.com/QuantEcon/data-lectures/raw/main/lectures/<file>, per the URL-forms table in data-lectures AGENTS.md: raw.githubusercontent.com is reserved for lecture-wasm, whose cells execute in the reader's browser and need a CORS-clean host. This repo is a CPython consumer, and wave B2' used the same form in lecture-python.myst. The wave plan QuantEcon/workspace-lectures#45 prescribes the wasm form for this repo, which is wrong; noted there. Proven not to change a figure, by running each lecture's own loading code against the old location and the new URL and comparing frames: bbh_macro_quarterly (260, 14) and bbh_michigan_monthly (507, 5) both `.equals()` True with identical columns and index; the hansen bundle's three tables — annual (95, 4), monthly (334, 3), quarterly (270, 4) — all `.equals()` True, and the raw JSON is byte-identical. The local copies are RETAINED. Deleting them here would 404 the already-published notebooks until someone tags a publish; the deletion is a follow-up PR after this one publishes. That ordering matters more than usual for the hansen bundle, which is fetched over the network at cell-execution time and so has no stale-serving grace period at all. The dead local-path fallback branch in _load_bundle() is left alone deliberately — it is unreachable once DATA_URL is a URL, but removing it is a behaviour-neutral cleanup that does not belong in a repoint whose whole claim is that nothing changed. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

#95) lecture-python-advanced.myst#372 merged, so advanced.myst now reads all three files from this repo. This closes the red window that opens the moment a repoint lands: build_audit fails `--strict` on any dataset marked `landed` while a consumer already reads data-lectures, so main has been red since that merge. Measured in both directions rather than asserted, which is what makes this the wave's acceptance test rather than bookkeeping. Against post-merge main: `landed` gives exit 1 with three `migration_inconsistencies`, one per file; `repointed` gives exit 0 with all five problem buckets empty. The dry-run has to run against the consuming repo's post-merge main, because build_audit reads `origin/main` through git show and cannot see an unmerged branch. The three consumer notes come out. They were added in #92 to record that the lecture still read its own copies while the dataset sat at `landed` — that is now false, and a note claiming an unrepointed consumer would be exactly the kind of stale record #91 added them to prevent. CATALOG.md regenerates with those three warning markers gone; the three that remain are the unrelated ws#46 readers (caron.npy, nom_balances.npy, us_adult_heights.csv), which are genuinely still on local copies. Verified the edit touched nothing else: for each of the three manifests, the only top-level key differing from main is `consumers`, and `source.note`, `license.note` and `integrity.upstream.note` are byte-identical to main. That check is not decorative — the first attempt at this edit matched `integrity.upstream.note` instead, because both sit at the same indent, and silently deleted the provenance record. Anchor on the `consumers:` block. Ledger is now 36 datasets, all `repointed`, 0 `landed`. NOT done here, and the order matters: advanced.myst has not published since this repoint, so its published notebooks still carry the old URLs. The three local copies stay until it does. That is load-bearing for hansen_jagannathan_1991_data.json, which is fetched over the network at cell-execution time and therefore has no stale-serving grace period — its deletion breaks every published notebook and Colab reader the instant the blob leaves advanced.myst's main. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Wave C2 (advanced.myst): fred_data.csv, acs_data_summary.csv, and dataBHS.csv — the format-converted successor to lectures/dataBHS.mat, which the published site 404s while the published notebook calls loadmat on it. Track D rides along: test_pwt.csv from lecture-python-programming, whose rename is on the same phase-2 review (#87). Two builders recovered against annotations recording no build script anywhere: fred_data.py reproduces the committed bytes from live FRED (three runs, 2026-08-18 — yields and the recession dummy are stable history, unlike the BBH national-accounts snapshot), and dataBHS.py converts sources/dataBHS.mat (un-refetchable; searched with positive controls, trail in sources/README.md) with the lecture's hardcoded moments asserted on every run. acs_data_summary.csv and test_pwt.csv are honest unrecovered: the ACS summary has no occupation key, year or recorded filters, and test_pwt.csv matches NO downloadable PWT vintage (7.0/6.3/6.2/6.1 all fetched and eliminated, hashes in the manifest). The dataBHS.mat annotation stays until the repoint lands — the lecture still reads the .mat and the basenames differ, so dual_recorded cannot fire; the flip PR deletes it. The other three annotations are deleted here. builders/README.md coverage table re-derived from parsed manifests (28 constructed, 18 with builders, 10 unrecovered), adding the two bbh rows omitted by #92. Strict audit exits 0 against current clones with all four records landed, and the negative control fires: flipping fred_data.csv to repointed yields exit 1 with the expected migration_inconsistencies. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Copilot AI lite review requested due to automatic review settings August 17, 2026 00:35

Record this PR's number in the three migration.yml records

3665043

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot started reviewing on behalf of mmcky August 17, 2026 00:35 View session

Copilot AI reviewed Aug 17, 2026

View reviewed changes

Comment thread lectures/bbh_macro_quarterly.csv.yml

Comment thread lectures/bbh_michigan_monthly.csv.yml

Comment thread builders/bbh_macro_quarterly.py

Comment thread lectures/hansen_jagannathan_1991_data.json.yml Outdated

mmcky mentioned this pull request Aug 17, 2026

Licensing: record-and-track policy, and the per-dataset inventory for migration #35

Open

mmcky merged commit ca2614c into main Aug 17, 2026
4 checks passed

mmcky deleted the datasets/track-c1 branch August 17, 2026 01:17

mmcky mentioned this pull request Aug 17, 2026

Read the three wave-C1 datasets from data-lectures (Track C) QuantEcon/lecture-python-advanced.myst#372

Merged

mmcky mentioned this pull request Aug 17, 2026

Flip wave C1 to repointed — the three advanced.myst datasets (Track C) #95

Merged

This was referenced Aug 17, 2026

VALIDATION: independent review of the 2026-08-17 Track C wave C1 migration and tidy-up work #96

Open

Land wave C2 + Track D — the four remaining static datasets #98

Merged

Navigation Menu

Uh oh!

Uh oh!

Land wave C1 — the three advanced.myst datasets that needed no rename (Track C) - #92

Land wave C1 — the three advanced.myst datasets that needed no rename (Track C)#92
mmcky merged 3 commits into
mainfrom
datasets/track-c1

mmcky commented Aug 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

mmcky commented Aug 17, 2026

Byte identity

Two builders recovered — the annotations they replace were wrong

The hansen bundle stays unrecovered, deliberately

A licence finding that needs recording elsewhere

Also here, because CI enforces it

Verification

Next

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

The hansen bundle stays `unrecovered`, deliberately