Skip to content

VALIDATION: independent review of the 2026-08-17 Track C wave C1 migration and tidy-up work #96

Description

@mmcky

Everything below was verified during the work, by the same agent that did it — same tools, same environment, same URL forms, same mental model that produced the claims. This issue exists to break that.

Bias to test for

Three verification monocultures ran through this session, and the checks below should avoid all three except where a box explicitly says to use one.

scripts/build_audit.py --strict verified most tracker claims — and was itself modified in the same session (#90 added the dual_recorded check). A tool that both changes and grades cannot establish that it grades correctly. Confirm ledger and consistency claims by parsing the YAML directly.

One interpreter produced nearly every numeric claim: /Users/mmcky/anaconda3/bin/python3, Python 3.13.9, pandas 2.3.3. Every profile, every byte comparison, every builder reproduction. Use a different Python and a different pandas.

Every sweep was /usr/bin/grep over local clones or the GitHub Trees API. Both find what they were pointed at. Confirm reader-facing outcomes by fetching the actual URLs and opening the actual artifacts.

What landed

repo merged PRs
data-lectures #89, #90, #91, #92, #94, #95
lecture-python-advanced.myst #372 (repoint), #373 (deletion)

Tags pushed: lecture-python-advanced.myst publish-2026aug17. Issues closed: workspace-lectures#38, #85, #86, and #93 (auto-filed and auto-closed by the CI alert). Issue opened: workspace-lectures#46. Issue reopened: workspace-lectures#41. Bodies edited: workspace-lectures#14, and the merged workspace-lectures#44's body.

1. Reader-facing outcomes

  • https://python-advanced.quantecon.org/subjective_beliefs_business_cycles.html and /hansen_jagannathan_1991.html both return 200 and render their figures. Use a browser or a fresh fetch, not a cached local copy.
  • The published notebooks at /_notebooks/subjective_beliefs_business_cycles.ipynb and /_notebooks/hansen_jagannathan_1991.ipynb each contain the token data-lectures and zero occurrences of _static/lecture_specific. Grep the basename, never an assembled URL — a URL split across adjacent string literals returns a confident zero to a whole-URL grep, which is how this check gives a false pass.
  • Beyond what was tested: actually execute one of those notebooks — open it in Colab from the published page's own launch button and run the data cell. The session confirmed the notebooks contain the new URL; nobody confirmed they run. This is also the last open box on VALIDATION: independent review of the 2026-08-13 Track B migration work (waves B1' and B2') #84.
  • The three data URLs return 200 with these exact byte counts: bbh_macro_quarterly.csv 32,217; bbh_michigan_monthly.csv 12,196; hansen_jagannathan_1991_data.json 63,456, at https://github.com/QuantEcon/data-lectures/raw/main/lectures/<file>. Include a never-existed control path and confirm it 404s.
  • Beyond what was tested: fetch the same three files over https://raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/<file> and check access-control-allow-origin. The session used only the github.com/…/raw/ form, which is the CPython form; no browser-runtime fetch was exercised for these files at all.
  • The three old paths under python-advanced.quantecon.org/_static/lecture_specific/ still return 200. This is expected, not a regression — the settle policy leaves the site to clear on the weekly cache rebuild plus the next publish. Confirm they are still 200 now, and record the date, so a later check can tell settling from staleness.
  • The same three paths on raw.githubusercontent.com/QuantEcon/lecture-python-advanced.myst/main/… return 404.

2. Artifact integrity

  • Each of the three files in data-lectures/lectures/ hashes to the integrity.sha256 in its sidecar manifest. Triangulate three ways: a live fetch of the URL, git show origin/main:lectures/<file>, and the value recorded in the manifest. Expected: f14f4256434371877f28da520f1e8b7950187dc23fff78dcc27912e89c0a8d5c, 567efe5a009a30bcfa32c30746bdaa97ce82ba611a98c03d127a662792319426, c81333f372a23465a593db592b04070b7b5732b4919348423d9534942f610029.
  • The landed bytes are identical to what lecture-python-advanced.myst held before deletion. Recover the pre-deletion blobs from git history — the deletion merged as f442011, so git show f442011^:lectures/_static/lecture_specific/… gives them — and compare against the data-lectures copies. This is the claim that licenses "the repoint cannot change a figure".
  • The figure-equivalence claim, and the question that could void it. The session compared the HTML archives attached to publish-2026aug02 and publish-2026aug17 and found 11/11 and 6/6 figures byte-identical. Re-download both archives and re-derive those hashes independently. Then answer the question the session did not: do any of those 17 figures actually depend on the migrated data? If none of them plot it, 17/17 identical proves nothing about the migration. Identify at least one figure that is computed from bbh_macro_quarterly.csv, bbh_michigan_monthly.csv or the hansen bundle, and confirm that specific figure is among the byte-identical set.
  • The session established the aug17 pages were genuinely rebuilt rather than restored from a cached _build/html, via the rendered source: data_path 3→0 and data_url 0→3 in subjective_beliefs_business_cycles.html, data-lectures 0→2 in hansen_jagannathan_1991.html. Re-derive those counts from the archives.

3. The highest-value claim to re-test: the two recovered builders

Most future work rests on this. builders/bbh_macro_quarterly.py and builders/bbh_michigan_monthly.py are claimed to reproduce their committed CSVs byte for byte from the Zenodo deposit 10.5281/zenodo.10194324, which is what licenses builder_status: committed and integrity.upstream.status: verified on both manifests — and what retires two of PLAN Phase 9's builder-recovery items.

  • Clone data-lectures fresh, install requirements.txt, and run both builders on a different machine or at least a different Python and pandas version from 3.13.9 / 2.3.3. Confirm git diff -- lectures/ is empty afterwards. A reproduction that only holds at one pandas version is not a reproduction; the float formatting in a to_csv is exactly where that would break.
  • builders/bbh_macro_quarterly.py claims to extract a 169 KB member from a 198.8 MB archive using HTTP range requests — roughly four requests and ~296 KB. Verify it is genuinely ranging rather than silently falling back to a full download: watch the bytes transferred, or read _open_archive()'s fallback path and confirm which branch executes.
  • Confirm the builders read the Zenodo deposit and not a cached local artifact. Run in a container or a clean home directory with no prior downloads.
  • Corroborate the provenance independently of the builders: the Zenodo record 10.5281/zenodo.10194324 should report license cc-by-4.0, publication date 2023-11-22, and a single file replication package.zip of 198,817,944 bytes with md5 6e33f9b9e70135cbf3304275d1c5d604.
  • hansen_jagannathan_1991_data.json is recorded builder_status: unrecovered while its manifest claims the construction is fully specified — all 2,462 cells re-derived from Shiller's workbooks and FRED to within 4 ULP. Spot-check that: pick one of the eleven columns and re-derive it from the named upstream. The manifest also records a recovered level factor of 0.8938191876245914 on annual.consumption; confirm that constant reproduces the column and that the earlier value 0.89381918762459 does not.

4. Records written

  • All 36 manifests in lectures/*.yml parse under yaml.safe_load, and each filename field equals its own sidecar name.
  • For the three new manifests, every measured shape claim reproduces against the bytes: row counts, column names and dtypes, per-column ranges, null counts and their pattern, and the date coverage. Re-measure; do not read the manifest and agree with it.
  • The hansen manifest uses a tables: block rather than schema.columns, because the file is a three-table orient='split' bundle. Confirm the three tables are annual (4 columns × 95 rows), monthly (3 × 334) and quarterly (4 × 270), and that the manifest's per-table descriptions match.
  • bbh_michigan_monthly.csv.yml records redistribution: restricted. Re-read the University of Michigan Surveys of Consumers usage agreement at https://data.sca.isr.umich.edu/agreement.php and judge independently whether the manifest's characterisation is fair — and whether restricted or something stronger is right, given the deposit's CC BY 4.0 grant comes from depositors who are not the rights holder.
  • The dual_recorded check added in Retire 8 stale annotations and make the manifest-or-annotation rule enforceable #90 actually fires. Re-add one retired annotation entry for a manifested dataset, confirm --strict exits 1 naming that file, then revert. This is the one place using build_audit.py is the point.
  • openpyxl==3.1.5 in requirements.txt is a real release and is what the two new builders need; confirm they fail without it and succeed with it.

5. Tracker consistency

6. Known blind spots

  • workspace-lectures#46 records two consumers still reading migrated datasets locally: test-actions-lecture-intro's french_rev.md at lines 709-710 (np.load('datasets/caron.npy') and the same for nom_balances.npy, while the three .xlsx reads at 73-75 in the same file were repointed), and lecture-intro.zh-cn's prob_dist.md:76. Verify both are real, at those lines.
  • Verify the stated reason the canary's case is invisible: french_rev is absent from its lectures/_toc.yml, and _config.yml sets only_build_toc_files: true.
  • Does the sweep that found these share the blind spot it documents? The count came from grepping four repos outside SCAN_REPOS. Ask what a fifth unlisted consumer would look like and whether the method could see it — the session enumerated lecture-intro.zh-cn, lecture-python.zh-cn, test-actions-lecture-intro and lecture-stats, and that list was not itself derived from anything, it was inherited.
  • The deletion-time org sweep covered 277 of 278 repos over the Trees API on default branches only, and Trees matches paths, not content. So a repo that references the files by URL without holding a copy, or holds one on a non-default branch, is invisible to it. Probe that margin: content-search a plausible subset, or check at least one non-default branch.
  • The session claimed a lecture-repo merge can turn data-lectures main red "with nothing reporting it". That is wrong as statedaudit-dashboard: the strict build is failing #93 was auto-filed by github-actions when the strict build failed, and auto-closed when it went green. Confirm that alerting path exists and works, and correct the record in Annotate the new bivariate_dist yfinance read — main is red without it #94's PR body and in this repo's notes if it is misdescribed there.

7. Decisions settled today

  • dataBHS.mat is 5,588 bytes, MATLAB v5, and holds exactly three (236,1) float64 arrays c, rb, rs. It is 404 on the published site while /_notebooks/five_preferences.ipynb is 200 and calls loadmat('dataBHS.mat') — the basis for calling its conversion a bug fix rather than tidying. Re-check both.
  • No filename collision exists in the org for fred_data.csv or acs_data_summary.csv, which is what allowed deferring the Batch rename for the §5.1 generic filenames, paired with their prose edits #87 renames. Re-derive.
  • Byte-identical duplicates of acs_data_summary.csv sit in lecture-dp, lecture-dp.monorepo and 2026-tom-course, and of dataBHS.mat in lecture-tools-techniques, python-lecture-sandpit.myst and lecture-mapping. lecture-tools-techniques is public, published, and reads its own copy from its own five_preferences.md. Confirm by blob SHA, and confirm the reads.
  • The legacy QuantEcon/lecture-python-advanced is not archived and has no Pages site, contrary to what the work plan asserted.

8. Deliberately not done

  • The dead local-path fallback branch in hansen_jagannathan_1991.md's _load_bundle() is unreachable now that DATA_URL is always a URL, and was left in place. Confirm it is genuinely unreachable and that this was recorded as a decision in #372's description rather than missed.
  • The three old paths still served by the published site are a settle-policy decision, recorded on workspace-lectures#40, not an incomplete deletion.
  • workspace-lectures#41's prune fix is implemented in zero repos and the issue is open. Confirm that is still true and that mpd2020.xlsx: record the repoint (hold until the lecture PRs merge) #41 records the current reasoning.
  • The deploy preview on lecture-python-advanced.myst serves no content — every path returns the same 3,449-byte 404, including pages that must exist — so the session could not use it to demonstrate mpd2020.xlsx: record the repoint (hold until the lecture PRs merge) #41. Confirm the preview is genuinely empty rather than that the wrong URL was used.

Where the reasoning lives

data-lectures PLAN.md and AGENTS.md carry the rules and the traps, including the URL-forms table that decides which host each consumer must use. The workspace tracker is workspace-lectures#14; the wave plan and its running commentary are workspace-lectures#45. Per-decision reasoning is in the PR descriptions listed under "What landed" — #92 and #95 in particular carry the reasoning that is not recorded anywhere else.


For the validator: work in a session that did not do this work. Do not use the tool named under "Bias to test for" except where a check explicitly says to run it. Re-derive counts rather than confirming them. Where a check can be run against a surface the original session did not exercise, do that too — the margin beyond the checklist is where regressions hide. Deliver: one comment on this issue with a per-item verdict (confirmed / confirmed with caveat / refuted / not completable, with evidence), a new issue for any regression found (do not bury findings in the comment), and leave the checkboxes to the issue owner unless told otherwise.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions