Give builders their own directory, one per published dataset

.github/scripts/check_consumed_files.py

@@ -11,6 +11,10 @@
   - if `integrity.sha256` is recorded, with or without consumers:
       * the data file must exist
       * the committed bytes must hash to it
+  - the builder record must be internally consistent:
+      * `builder_status` must be a known value
+      * a `committed*` status must name a builder
+      * a named builder must exist on disk
 
 The second clause exists because manifests land *ahead* of their repoints by
 convention, so a dataset arrives with `consumers: []` and is flipped by a
@@ -30,7 +34,15 @@
 
 import yaml
 
-LECTURES = pathlib.Path(__file__).resolve().parents[2] / "lectures"
+REPO = pathlib.Path(__file__).resolve().parents[2]
+LECTURES = REPO / "lectures"
+
+# One builder per published dataset, in builders/ (AGENTS.md, "Builders").
+# `committed` asserts a runnable four-stage builder; `committed-frozen` says the
+# builder is here and deliberately will not run (a frozen vintage, a scraper we
+# will not re-run); `unrecovered` says it is absent; `not-applicable` is for
+# verbatim files.
+BUILDER_STATUSES = {"committed", "committed-frozen", "unrecovered", "not-applicable"}
 
 
 def sha256(path: pathlib.Path) -> str:
@@ -68,6 +80,54 @@ def main() -> int:
             )
             continue
 
+        # Builder: the status must be a known one, a dataset that claims a
+        # builder must name it, and the path it names must be a file in this
+        # repo. Nothing else validates any of this — build_audit only records
+        # the value and the catalog only formats it — so a manifest can assert
+        # a builder that was never committed, or that has been moved.
+        #
+        # Types are checked first because YAML will hand us a list or a mapping
+        # for either field, and both would raise rather than report: `status not
+        # in BUILDER_STATUSES` is a TypeError on an unhashable value, and
+        # `REPO / builder` is a TypeError on anything but a string.
+        status = manifest.get("builder_status")
+        builder = manifest.get("builder")
+        if status is not None and not isinstance(status, str):
+            errors.append(
+                f"{declared}: builder_status must be a string, got "
+                f"{type(status).__name__}"
+            )
+        elif status is not None and status not in BUILDER_STATUSES:
+            errors.append(
+                f"{declared}: unknown builder_status {status!r} — expected one "
+                f"of {', '.join(sorted(BUILDER_STATUSES))}"
+            )
+        elif isinstance(status, str) and status.startswith("committed") and not builder:
+            errors.append(
+                f"{declared}: builder_status is {status!r} but no builder is "
+                f"named — a dataset claiming a committed builder must say "
+                f"where it is"
+            )
+
+        if builder is not None and not isinstance(builder, str):
+            errors.append(
+                f"{declared}: builder must be a string path, got "
+                f"{type(builder).__name__}"
+            )
+        elif builder:
+            # Resolve before checking: `REPO / builder` silently discards REPO
+            # when builder is absolute, and a relative path can climb out with
+            # `../`. Either way the assertion would be satisfied by a file that
+            # is not a builder in this repo, which is the only thing it exists
+            # to establish. A directory passes `.exists()` too, hence is_file.
+            target = (REPO / builder).resolve()
+            if not target.is_relative_to(REPO) or not target.is_file():
+                errors.append(
+                    f"{declared}: builder {builder!r} must be a file inside "
+                    f"this repo — builders live in builders/<stem>.<ext> "
+                    f"(AGENTS.md, 'Builders')"
+                )
+
         consumers = manifest.get("consumers") or []
         integrity = manifest.get("integrity")
         # A present-but-malformed integrity block must fail loudly. Read as

AGENTS.md

@@ -34,6 +34,7 @@ A constructed dataset without its committed builder is a bug. Manifest fields: `
 The Feb 2025 migration left files that cannot fully satisfy the rules above. The manifest records each gap **explicitly** — visible in the generated catalog — rather than burying it by misclassification. Both are provisional decisions from the P1 pilot ([meta#338](https://github.com/QuantEcon/meta/issues/338)), to be folded into [manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108).
 
 - **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that says why (`status: unverifiable` with a `note`).
+- **`builder_status: committed-frozen` — the builder is here, and deliberately will not run.** For a dataset built from a source that must not be refreshed: a frozen vintage, or a scraper we will not re-run. The artifact is kept as the record of what produced these bytes, so it is committed verbatim and not edited — editing it is what would destroy its value as provenance. Distinct from `committed`, which asserts a runnable four-stage builder, and from `unrecovered`, which says the builder is absent.
 - **`builder_status: unrecovered` — constructed without a recoverable builder.** A constructed dataset ships its builder, and one that omits it *silently* is the bug. Several inherited files are constructed with no recoverable extraction steps (PLAN Phase 9 tracks them). Keep `class: constructed` — reclassifying to `verbatim` to dodge the rule is misclassification — set `builder: null` and `builder_status: unrecovered`, and the gap stays visible for Phase 9 to recover. `unrecovered` is for **inherited files only**; never introduce a *new* constructed file without its builder.
 
 ### Repointing a lecture — three ordering traps
@@ -105,7 +106,15 @@ The limits: **50 MiB** warns on push, **100 MiB** (104,857,600 B) is a hard bloc
 
 **Never** put an LFS object under `lectures/`, and never reach for GitHub release assets: they send no `access-control-allow-origin` on any hop, so a browser cannot read them. Parquet is not a size remedy here either — `pyarrow` is absent from the Pyodide `lecture-wasm` pins, and gzipped CSV is smaller than Parquet on this data anyway.
 
-### Dynamic builders
+### Builders
+
+**One builder per published dataset, in `builders/`, named for the dataset it produces:** `builders/<stem>.<ext>` builds `lectures/<stem>.<ext2>`. The stem is the dataset's, not the lecture's — `builders/japan_earthquakes.py` writes `lectures/japan_earthquakes.csv`. That makes the manifest's `builder:` field predictable and lets CI assert it.
+
+Where one builder produces a **set** of files, name it for the set and let each file's manifest point at the same path — `business_cycle.py` writes three. The stem rule is the default, not an invariant; what CI asserts is that every `builder:` path exists, and that a dataset claiming a builder names one.
+
+`scripts/` is repo tooling — the audit dashboard and the catalog generator — and produces no dataset. Keep the two apart.
+
+**Where a builder reads its input from.** The normal case is the third-party upstream, fetched at run time: six of the seven builders here do that, and it is the fetch stage of the contract below. A builder reads from `sources/` **only when the input cannot be re-fetched** — the upstream is gone, unlocatable, or was inherited with no recoverable source. `sources/` is that exception layer, not a general input tree, and it is emphatically not "the big-file directory": the defining property is un-refetchability, not size. What it must never be is a network read from another QuantEcon repo — that is how a retired repo becomes load-bearing again.
 
 Builders follow four stages — **fetch → pre-process → validate → write** — and only write on validation pass (expected columns/dtypes, row-count floor, recency of date range, no all-NaN columns, values unchanged in the overlap window with the previous vintage). Lectures always read the last-good snapshot: an upstream outage may fail a refresh, it must never break a lecture build.
 
@@ -142,12 +151,13 @@ The generated dashboard (`scripts/build_audit.py`, [#20](https://github.com/Quan
 
 ```
 lectures/            # the published tree — flat, live on Pages; data.quantecon.org pending
-                     #   19 datasets (10 with manifests; the 8 static intro files
-                     #   and business_cycle_data.csv still need theirs) plus
-                     #   business_cycle's two upstream metadata dumps (see #13)
-                     #   manifests live here as sidecars: <filename>.yml
-scripts/             # builders + generators — NOT published
-  business_cycle.py  #   writes business_cycle_data.csv into lectures/
+                     #   21 files, 18 with manifests (business_cycle's three still
+                     #   need theirs — see #13). Manifests are sidecars: <filename>.yml
+builders/            # one builder per published dataset — NOT published
+                     #   builders/<stem>.py builds lectures/<stem>.<ext>
+sources/             # inputs a builder cannot re-fetch — NOT published, per-path LFS
+                     #   no manifests; sources/README.md is the audit trail
+scripts/             # repo tooling — NOT published, produces no dataset
   build_catalog.py   #   generates CATALOG.md from the manifests
   build_audit.py     #   the audit dashboard: scan lecture repos → audit.json → site/
   render_audit.py    #   its render stage

PLAN.md

@@ -30,7 +30,7 @@ This repository is being shaped into the **single canonical repository for data
   - **8 duplicates of data in active use** — `mpd2020.xlsx`, `longprices.xls`, `chapter_3.xlsx`, `assignat.xlsx`, `dette.xlsx`, `fig_3.xlsx`, `caron.npy`, `nom_balances.npy` are consumed by intro lectures (`long_run_growth`, `inflation_history`, `french_rev`), but via intro's **own copies** (own-repo URLs, or local paths for the `.npy` pair) — these are the Phase 8 repoint targets
   - **2 dead on both ends** — the World Bank GDP-per-capita CSV and its metadata twin are orphaned in intro too; nothing reads either copy anywhere
   - **2 never adopted** — `business_cycle_data.csv` (the one dynamic snapshot; intro's `business_cycle` still fetches live from wbgapi/FRED) and `fig_3.ods` (a source-format twin of `fig_3.xlsx`, referenced by nothing)
-- one manual refresh script (`scripts/business_cycle.py`), run by hand — it has fetch/transform/write but **no validate stage**
+- one manual refresh script (`builders/business_cycle.py`), run by hand — it has fetch/transform/write but **no validate stage**
 - **no `.github/`** — no CI, no PR validation, no scheduled refresh
 - no LFS, no per-dataset manifests, no license records
 - referenced by **zero lectures** (confirmed by the audit and by live GitHub code search, 2026-07-16) — the Feb 2025 migration (data#5–#7) landed the files but the repoint (data#4) never happened. Until the first repoint merges, everything here can be restructured freely
@@ -269,7 +269,7 @@ Full automation:
 
 - [x] Audit dashboard workflow ([#20](https://github.com/QuantEcon/data-lectures/issues/20), added 2026-07-17): `.github/workflows/audit-dashboard.yml` rebuilds the full-universe data audit + migration tracker from the 8 lecture repos' `main` (push to main / weekly / dispatch) and deploys it with the published tree to Pages. Strict mode fails the build on an unannotated data reference or a `migration.yml` status the scan contradicts
 - [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data. The schema decisions these tests force — column patterns for wide files, `known_nulls` exact-vs-ceiling, a canonical dtype vocabulary — are researched in [#14](https://github.com/QuantEcon/data-lectures/issues/14)
-- [ ] Retrofit `scripts/business_cycle.py` to the four-stage builder contract — it has fetch/transform/write today but **no validate stage**. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14)
+- [ ] Retrofit `builders/business_cycle.py` to the four-stage builder contract — it has fetch/transform/write today but **no validate stage**. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14)
 - [ ] Scheduled refresh workflow for dynamic datasets — cron per cadence class, runs the builder (fetch → pre-process → validate → write), lands the result as a PR whose diff summary (rows added, date-range delta, overlap-window changes) is the review surface; low-risk series may auto-merge on green (first consumer: the UNRATE pilot, meta#338 P4)
 - [ ] Weekly sources-alive canary: fetch + validate, no commit, opens an issue on failure — relocates API fragility from 7 lecture repos' CI into one scheduled job here
 - [ ] Consumer fan-out: a merged refresh or in-place correction dispatches rebuilds of the repos in the dataset's machine-readable `consumers` list
@@ -293,7 +293,7 @@ Verify that what this repo holds is actually the data it claims to be — agains
 
 - [ ] **Byte-compare against the in-use copies**: each file migrated in Feb 2025 must be identical to the copy `lecture-python-intro` currently consumes (git blob hash compare). If a copy diverged, a repoint silently changes lecture output — this check is a hard prerequisite for Phase 8. Recorded **in the repoint PR** as a one-time gate, reproducible later from the manifest's `sha256` — not a manifest field (P1 decision)
 - [ ] **Verbatim files**: re-fetch from the upstream source and compare (e.g. `mpd2020.xlsx` against the published Maddison Project 2020 release); record `sha256`, `status`, what it was compared `against`, and the date in the manifest's `integrity.upstream`
-- [ ] **Constructed / dynamic files**: re-run the committed builder (`scripts/business_cycle.py` → `business_cycle_data.csv`) and confirm values agree in the overlap window with the committed snapshot
+- [ ] **Constructed / dynamic files**: re-run the committed builder (`builders/business_cycle.py` → `business_cycle_data.csv`) and confirm values agree in the overlap window with the committed snapshot
 - [ ] **Author-assembled files** (the French Revolution spreadsheets, `caron.npy`, `nom_balances.npy` — prose-only provenance): spot-check key values against the cited publication and record what was checked; full verification may be impossible, and the manifest should say so (`status: unverifiable` with a one-line `note` — the honest known status, per P1)
 - [ ] **Unverifiable or failing files**: flag in the manifest and open an issue — do not promote a file to the canonical URL namespace with a known-bad or unknown integrity status
 

builders/README.md

@@ -0,0 +1,50 @@
+# builders
+
+One builder per published dataset. This directory sits **outside** the published
+tree — it is never served.
+
+## The naming rule
+
+`builders/<stem>.<ext>` builds `lectures/<stem>.<ext2>`.
+
+The stem is the dataset's, not the lecture's: `builders/japan_earthquakes.py`
+writes `lectures/japan_earthquakes.csv`. That makes the manifest's `builder:`
+field predictable and lets CI assert it.
+
+**Where one builder produces a set of files**, name it for the set and let each
+file's manifest point at the same builder path. `business_cycle.py` writes three
+files; the SCF and Forbes builders each write two. The rule is the default, not
+an invariant — what CI asserts is that every `builder:` path exists, and that a
+dataset claiming a builder names one.
+
+## The contract
+
+Builders follow four stages — **fetch → pre-process → validate → write** — and
+only write on validation pass. Lectures always read the last-good snapshot: an
+upstream outage may fail a refresh, it must never break a lecture build. The
+template and the architecture discussion are in
+[#14](https://github.com/QuantEcon/data-lectures/issues/14).
+
+Most builders fetch from the third-party upstream at run time, which is the
+normal case. A builder reads from `sources/` only when its input **cannot be
+re-fetched** — see `AGENTS.md`.
+
+## What is here
+
+| Builder | Writes to `lectures/` | Status |
+| --- | --- | --- |
+| `ames_house_prices.py` | `ames_house_prices.csv` | committed |
+| `epl_match_goals.py` | `epl_match_goals.csv` | committed |
+| `japan_deaths_by_age.py` | `japan_deaths_by_age.csv` | committed |
+| `japan_earthquakes.py` | `japan_earthquakes.csv` | committed |
+| `japan_population_by_age.py` | `japan_population_by_age.csv` | committed |
+| `us_adult_heights.py` | `us_adult_heights.csv` | committed |
+| `business_cycle.py` | `business_cycle_data.csv`, `business_cycle_info.md`, `business_cycle_metadata.md` | run by hand, no validate stage yet (PLAN Phase 5); its three outputs are the repo's only unmanifested files |
+
+**This listing is the coverage report.** The repo has 13 `constructed` datasets
+and 7 builders; the difference is the Phase 9 recovery backlog, carried as
+`builder_status: unrecovered` in each manifest rather than hidden by
+reclassifying the file as `verbatim`.
+
+Repo tooling — the audit dashboard and the catalog generator — lives in
+`scripts/` and is not a builder.

lectures/ames_house_prices.csv.yml

@@ -71,5 +71,5 @@ consumers:
   - repo: QuantEcon/lecture-python-intro
     file: lectures/fitting_distributions.md
 
-builder: scripts/ames_house_prices.py
+builder: builders/ames_house_prices.py
 builder_status: committed

lectures/epl_match_goals.csv.yml

@@ -65,5 +65,5 @@ consumers:
   - repo: QuantEcon/lecture-python-intro
     file: lectures/fitting_distributions.md
 
-builder: scripts/epl_match_goals.py
+builder: builders/epl_match_goals.py
 builder_status: committed

lectures/japan_deaths_by_age.csv.yml

@@ -74,5 +74,5 @@ consumers:
   - repo: QuantEcon/lecture-python-intro
     file: lectures/fitting_distributions.md
 
-builder: scripts/japan_deaths_by_age.py
+builder: builders/japan_deaths_by_age.py
 builder_status: committed

lectures/japan_earthquakes.csv.yml

@@ -75,5 +75,5 @@ consumers:
   - repo: QuantEcon/lecture-python-intro
     file: lectures/fitting_distributions.md
 
-builder: scripts/japan_earthquakes.py
+builder: builders/japan_earthquakes.py
 builder_status: committed

lectures/japan_population_by_age.csv.yml

@@ -85,5 +85,5 @@ consumers:
   - repo: QuantEcon/lecture-python-intro
     file: lectures/prob_dist.md
 
-builder: scripts/japan_population_by_age.py
+builder: builders/japan_population_by_age.py
 builder_status: committed

lectures/us_adult_heights.csv.yml

@@ -82,5 +82,5 @@ consumers:
   - repo: QuantEcon/lecture-python-intro
     file: lectures/fitting_distributions.md
 
-builder: scripts/us_adult_heights.py
+builder: builders/us_adult_heights.py
 builder_status: committed

manifest-schema.yml

@@ -167,15 +167,15 @@ consumers: []
 # Builder — required for constructed and dynamic-snapshot; omit for verbatim
 # ---------------------------------------------------------------------------
 
-builder: scripts/business_cycle.py      # path to the committed builder, or null
+builder: builders/business_cycle.py      # path to the committed builder, or null
 
 # AGENTS.md says a constructed dataset without its builder is a bug — but the
 # repo has inherited several (PLAN Phase 9), and P1's own pilot file is one.
 # The tempting workaround is to misclassify them as `verbatim`, which buries
 # the gap. This field keeps it visible in the generated catalog instead.
 # `unrecovered` is for inherited files only; a NEW constructed file must ship
 # its builder (AGENTS.md, "Two inherited-file states").
-builder_status: committed               # committed | unrecovered | not-applicable
+builder_status: committed               # committed | committed-frozen | unrecovered | not-applicable
 
 # Dynamic snapshots only. Drives the scheduled refresh-as-PR (PLAN Phase 5).
 cadence: annual                         # e.g. daily | weekly | monthly | annual