From f8e506fc10e06445745136582b4d14da7528164a Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Mon, 10 Aug 2026 12:04:03 +1000 Subject: [PATCH] Give builders their own directory, one per published dataset (#60) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Give builders their own directory, one per published dataset scripts/ was two unrelated things sharing a name: seven builders that produce published datasets, and the dashboard/catalog toolchain that has nothing to do with data production. Its own README already apologised for the split with a "Not builders" section — while listing 1 of the 7. builders/. now builds lectures/.. That is not a new convention: six of the seven already matched their output stem exactly, each with an explicit OUT_FILE = '.csv'. The move makes a latent pattern enforceable. Where one builder produces a SET of files, it is named for the set and several manifests point at the same path — business_cycle.py writes three, and both incoming notebook builders write two. So the stem rule is the default, not an invariant, and what CI asserts is weaker and truer: * builder_status must be a known value * a `committed*` status must name a builder * a named builder must exist on disk None of that was checked anywhere before — build_audit only records the value and build_catalog only formats it, so a manifest could assert a builder that was never committed, or one that had been moved. Which is exactly what this commit does to six of them. Adds `committed-frozen` to the enum: the builder is here and deliberately will not run, for a dataset built from a source that must not be refreshed. `committed` asserts a runnable four-stage builder and is a false claim for a frozen vintage; `unrecovered` says the builder is absent and is a false claim for one arriving in the same PR. This is the answer to #14's notebook question, and it lets the two high_dim_data builders land as what they are. Also records where a builder reads its input from, since sources/ is about to exist and is easy to misread: the normal case is the third-party upstream at run time — six of seven do that — and sources/ is only for an input that cannot be re-fetched. It is not a general input tree and not "the big-file directory"; the defining property is un-refetchability. Safe to do now, and cheaper now than later: no workflow runs any builder, nothing outside the repo references the paths, and PLAN's warning that the restructure window is spent applies to lectures/, where filenames are the public API — not to scripts/, which is never served. Doing it before the high_dim_data fold means its two builders land in the final shape. Verified: real tree 18/18 hash-checked exit 0 (which also proves the six moved paths resolve), strict audit exit 0, CATALOG.md regenerates byte-identical. Six negative cases exercised — missing builder path, a committed status with no builder, an unknown status, committed-frozen, and the verbatim/not-applicable shape that must keep passing. Part of #14. Co-Authored-By: Claude Opus 5 (1M context) * Guardrail: type-check the builder fields, and require an in-repo file Copilot review on #60, confirmed by reproduction — the builder checks had four holes, two of which crashed rather than reported. A non-string `builder_status` (a YAML list, say) raised `TypeError: unhashable type` on the set-membership test, and a non-string `builder` raised on `REPO / builder`. Both surfaced as a traceback with no `::error::` line — the same class of defect the review caught in #56, in a file that already handles it that way for `integrity` and for the manifest itself. The path check was satisfiable by things that are not builders in this repo. `REPO / builder` silently discards REPO when `builder` is absolute, so `/etc/hosts` passed; a relative path can climb out with `../`, so a real file outside the repo passed; and `.exists()` is true for directories, so naming `builders` passed. That last one the review did not name. Resolving and asserting `is_relative_to(REPO)` plus `is_file()` closes all three. Framing note: the read is correctness rather than security — the check only calls `.exists()`, never reads — but an assertion satisfiable by a file that is not a builder here establishes nothing, and the likely trigger is a typo'd relative path that happens to resolve on the runner, not an attack. Verified: real tree 18/18 exit 0; the six cases from the PR body unchanged; five new ones — list status, list builder, escaping relative path, absolute path, directory — all now fail cleanly with no traceback. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- .github/scripts/check_consumed_files.py | 62 ++++++++++++++++++- AGENTS.md | 24 ++++--- PLAN.md | 6 +- builders/README.md | 50 +++++++++++++++ {scripts => builders}/ames_house_prices.py | 0 {scripts => builders}/business_cycle.py | 0 {scripts => builders}/epl_match_goals.py | 0 {scripts => builders}/japan_deaths_by_age.py | 0 {scripts => builders}/japan_earthquakes.py | 0 .../japan_population_by_age.py | 0 {scripts => builders}/us_adult_heights.py | 0 lectures/ames_house_prices.csv.yml | 2 +- lectures/epl_match_goals.csv.yml | 2 +- lectures/japan_deaths_by_age.csv.yml | 2 +- lectures/japan_earthquakes.csv.yml | 2 +- lectures/japan_population_by_age.csv.yml | 2 +- lectures/us_adult_heights.csv.yml | 2 +- manifest-schema.yml | 4 +- scripts/README.md | 22 ++----- 19 files changed, 144 insertions(+), 36 deletions(-) create mode 100644 builders/README.md rename {scripts => builders}/ames_house_prices.py (100%) rename {scripts => builders}/business_cycle.py (100%) rename {scripts => builders}/epl_match_goals.py (100%) rename {scripts => builders}/japan_deaths_by_age.py (100%) rename {scripts => builders}/japan_earthquakes.py (100%) rename {scripts => builders}/japan_population_by_age.py (100%) rename {scripts => builders}/us_adult_heights.py (100%) diff --git a/.github/scripts/check_consumed_files.py b/.github/scripts/check_consumed_files.py index b1516aa..be8859b 100644 --- a/.github/scripts/check_consumed_files.py +++ b/.github/scripts/check_consumed_files.py @@ -11,6 +11,10 @@ - if `integrity.sha256` is recorded, with or without consumers: * the data file must exist * the committed bytes must hash to it + - the builder record must be internally consistent: + * `builder_status` must be a known value + * a `committed*` status must name a builder + * a named builder must exist on disk The second clause exists because manifests land *ahead* of their repoints by convention, so a dataset arrives with `consumers: []` and is flipped by a @@ -30,7 +34,15 @@ import yaml -LECTURES = pathlib.Path(__file__).resolve().parents[2] / "lectures" +REPO = pathlib.Path(__file__).resolve().parents[2] +LECTURES = REPO / "lectures" + +# One builder per published dataset, in builders/ (AGENTS.md, "Builders"). +# `committed` asserts a runnable four-stage builder; `committed-frozen` says the +# builder is here and deliberately will not run (a frozen vintage, a scraper we +# will not re-run); `unrecovered` says it is absent; `not-applicable` is for +# verbatim files. +BUILDER_STATUSES = {"committed", "committed-frozen", "unrecovered", "not-applicable"} def sha256(path: pathlib.Path) -> str: @@ -68,6 +80,54 @@ def main() -> int: ) continue + # Builder: the status must be a known one, a dataset that claims a + # builder must name it, and the path it names must be a file in this + # repo. Nothing else validates any of this — build_audit only records + # the value and the catalog only formats it — so a manifest can assert + # a builder that was never committed, or that has been moved. + # + # Types are checked first because YAML will hand us a list or a mapping + # for either field, and both would raise rather than report: `status not + # in BUILDER_STATUSES` is a TypeError on an unhashable value, and + # `REPO / builder` is a TypeError on anything but a string. + status = manifest.get("builder_status") + builder = manifest.get("builder") + if status is not None and not isinstance(status, str): + errors.append( + f"{declared}: builder_status must be a string, got " + f"{type(status).__name__}" + ) + elif status is not None and status not in BUILDER_STATUSES: + errors.append( + f"{declared}: unknown builder_status {status!r} — expected one " + f"of {', '.join(sorted(BUILDER_STATUSES))}" + ) + elif isinstance(status, str) and status.startswith("committed") and not builder: + errors.append( + f"{declared}: builder_status is {status!r} but no builder is " + f"named — a dataset claiming a committed builder must say " + f"where it is" + ) + + if builder is not None and not isinstance(builder, str): + errors.append( + f"{declared}: builder must be a string path, got " + f"{type(builder).__name__}" + ) + elif builder: + # Resolve before checking: `REPO / builder` silently discards REPO + # when builder is absolute, and a relative path can climb out with + # `../`. Either way the assertion would be satisfied by a file that + # is not a builder in this repo, which is the only thing it exists + # to establish. A directory passes `.exists()` too, hence is_file. + target = (REPO / builder).resolve() + if not target.is_relative_to(REPO) or not target.is_file(): + errors.append( + f"{declared}: builder {builder!r} must be a file inside " + f"this repo — builders live in builders/. " + f"(AGENTS.md, 'Builders')" + ) + consumers = manifest.get("consumers") or [] integrity = manifest.get("integrity") # A present-but-malformed integrity block must fail loudly. Read as diff --git a/AGENTS.md b/AGENTS.md index 01abad1..fb52da9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,6 +34,7 @@ A constructed dataset without its committed builder is a bug. Manifest fields: ` The Feb 2025 migration left files that cannot fully satisfy the rules above. The manifest records each gap **explicitly** — visible in the generated catalog — rather than burying it by misclassification. Both are provisional decisions from the P1 pilot ([meta#338](https://github.com/QuantEcon/meta/issues/338)), to be folded into [manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108). - **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that says why (`status: unverifiable` with a `note`). +- **`builder_status: committed-frozen` — the builder is here, and deliberately will not run.** For a dataset built from a source that must not be refreshed: a frozen vintage, or a scraper we will not re-run. The artifact is kept as the record of what produced these bytes, so it is committed verbatim and not edited — editing it is what would destroy its value as provenance. Distinct from `committed`, which asserts a runnable four-stage builder, and from `unrecovered`, which says the builder is absent. - **`builder_status: unrecovered` — constructed without a recoverable builder.** A constructed dataset ships its builder, and one that omits it *silently* is the bug. Several inherited files are constructed with no recoverable extraction steps (PLAN Phase 9 tracks them). Keep `class: constructed` — reclassifying to `verbatim` to dodge the rule is misclassification — set `builder: null` and `builder_status: unrecovered`, and the gap stays visible for Phase 9 to recover. `unrecovered` is for **inherited files only**; never introduce a *new* constructed file without its builder. ### Repointing a lecture — three ordering traps @@ -105,7 +106,15 @@ The limits: **50 MiB** warns on push, **100 MiB** (104,857,600 B) is a hard bloc **Never** put an LFS object under `lectures/`, and never reach for GitHub release assets: they send no `access-control-allow-origin` on any hop, so a browser cannot read them. Parquet is not a size remedy here either — `pyarrow` is absent from the Pyodide `lecture-wasm` pins, and gzipped CSV is smaller than Parquet on this data anyway. -### Dynamic builders +### Builders + +**One builder per published dataset, in `builders/`, named for the dataset it produces:** `builders/.` builds `lectures/.`. The stem is the dataset's, not the lecture's — `builders/japan_earthquakes.py` writes `lectures/japan_earthquakes.csv`. That makes the manifest's `builder:` field predictable and lets CI assert it. + +Where one builder produces a **set** of files, name it for the set and let each file's manifest point at the same path — `business_cycle.py` writes three. The stem rule is the default, not an invariant; what CI asserts is that every `builder:` path exists, and that a dataset claiming a builder names one. + +`scripts/` is repo tooling — the audit dashboard and the catalog generator — and produces no dataset. Keep the two apart. + +**Where a builder reads its input from.** The normal case is the third-party upstream, fetched at run time: six of the seven builders here do that, and it is the fetch stage of the contract below. A builder reads from `sources/` **only when the input cannot be re-fetched** — the upstream is gone, unlocatable, or was inherited with no recoverable source. `sources/` is that exception layer, not a general input tree, and it is emphatically not "the big-file directory": the defining property is un-refetchability, not size. What it must never be is a network read from another QuantEcon repo — that is how a retired repo becomes load-bearing again. Builders follow four stages — **fetch → pre-process → validate → write** — and only write on validation pass (expected columns/dtypes, row-count floor, recency of date range, no all-NaN columns, values unchanged in the overlap window with the previous vintage). Lectures always read the last-good snapshot: an upstream outage may fail a refresh, it must never break a lecture build. @@ -142,12 +151,13 @@ The generated dashboard (`scripts/build_audit.py`, [#20](https://github.com/Quan ``` lectures/ # the published tree — flat, live on Pages; data.quantecon.org pending - # 19 datasets (10 with manifests; the 8 static intro files - # and business_cycle_data.csv still need theirs) plus - # business_cycle's two upstream metadata dumps (see #13) - # manifests live here as sidecars: .yml -scripts/ # builders + generators — NOT published - business_cycle.py # writes business_cycle_data.csv into lectures/ + # 21 files, 18 with manifests (business_cycle's three still + # need theirs — see #13). Manifests are sidecars: .yml +builders/ # one builder per published dataset — NOT published + # builders/.py builds lectures/. +sources/ # inputs a builder cannot re-fetch — NOT published, per-path LFS + # no manifests; sources/README.md is the audit trail +scripts/ # repo tooling — NOT published, produces no dataset build_catalog.py # generates CATALOG.md from the manifests build_audit.py # the audit dashboard: scan lecture repos → audit.json → site/ render_audit.py # its render stage diff --git a/PLAN.md b/PLAN.md index 69da353..3ba4e08 100644 --- a/PLAN.md +++ b/PLAN.md @@ -30,7 +30,7 @@ This repository is being shaped into the **single canonical repository for data - **8 duplicates of data in active use** — `mpd2020.xlsx`, `longprices.xls`, `chapter_3.xlsx`, `assignat.xlsx`, `dette.xlsx`, `fig_3.xlsx`, `caron.npy`, `nom_balances.npy` are consumed by intro lectures (`long_run_growth`, `inflation_history`, `french_rev`), but via intro's **own copies** (own-repo URLs, or local paths for the `.npy` pair) — these are the Phase 8 repoint targets - **2 dead on both ends** — the World Bank GDP-per-capita CSV and its metadata twin are orphaned in intro too; nothing reads either copy anywhere - **2 never adopted** — `business_cycle_data.csv` (the one dynamic snapshot; intro's `business_cycle` still fetches live from wbgapi/FRED) and `fig_3.ods` (a source-format twin of `fig_3.xlsx`, referenced by nothing) -- one manual refresh script (`scripts/business_cycle.py`), run by hand — it has fetch/transform/write but **no validate stage** +- one manual refresh script (`builders/business_cycle.py`), run by hand — it has fetch/transform/write but **no validate stage** - **no `.github/`** — no CI, no PR validation, no scheduled refresh - no LFS, no per-dataset manifests, no license records - referenced by **zero lectures** (confirmed by the audit and by live GitHub code search, 2026-07-16) — the Feb 2025 migration (data#5–#7) landed the files but the repoint (data#4) never happened. Until the first repoint merges, everything here can be restructured freely @@ -269,7 +269,7 @@ Full automation: - [x] Audit dashboard workflow ([#20](https://github.com/QuantEcon/data-lectures/issues/20), added 2026-07-17): `.github/workflows/audit-dashboard.yml` rebuilds the full-universe data audit + migration tracker from the 8 lecture repos' `main` (push to main / weekly / dispatch) and deploys it with the published tree to Pages. Strict mode fails the build on an unannotated data reference or a `migration.yml` status the scan contradicts - [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data. The schema decisions these tests force — column patterns for wide files, `known_nulls` exact-vs-ceiling, a canonical dtype vocabulary — are researched in [#14](https://github.com/QuantEcon/data-lectures/issues/14) -- [ ] Retrofit `scripts/business_cycle.py` to the four-stage builder contract — it has fetch/transform/write today but **no validate stage**. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14) +- [ ] Retrofit `builders/business_cycle.py` to the four-stage builder contract — it has fetch/transform/write today but **no validate stage**. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14) - [ ] Scheduled refresh workflow for dynamic datasets — cron per cadence class, runs the builder (fetch → pre-process → validate → write), lands the result as a PR whose diff summary (rows added, date-range delta, overlap-window changes) is the review surface; low-risk series may auto-merge on green (first consumer: the UNRATE pilot, meta#338 P4) - [ ] Weekly sources-alive canary: fetch + validate, no commit, opens an issue on failure — relocates API fragility from 7 lecture repos' CI into one scheduled job here - [ ] Consumer fan-out: a merged refresh or in-place correction dispatches rebuilds of the repos in the dataset's machine-readable `consumers` list @@ -293,7 +293,7 @@ Verify that what this repo holds is actually the data it claims to be — agains - [ ] **Byte-compare against the in-use copies**: each file migrated in Feb 2025 must be identical to the copy `lecture-python-intro` currently consumes (git blob hash compare). If a copy diverged, a repoint silently changes lecture output — this check is a hard prerequisite for Phase 8. Recorded **in the repoint PR** as a one-time gate, reproducible later from the manifest's `sha256` — not a manifest field (P1 decision) - [ ] **Verbatim files**: re-fetch from the upstream source and compare (e.g. `mpd2020.xlsx` against the published Maddison Project 2020 release); record `sha256`, `status`, what it was compared `against`, and the date in the manifest's `integrity.upstream` -- [ ] **Constructed / dynamic files**: re-run the committed builder (`scripts/business_cycle.py` → `business_cycle_data.csv`) and confirm values agree in the overlap window with the committed snapshot +- [ ] **Constructed / dynamic files**: re-run the committed builder (`builders/business_cycle.py` → `business_cycle_data.csv`) and confirm values agree in the overlap window with the committed snapshot - [ ] **Author-assembled files** (the French Revolution spreadsheets, `caron.npy`, `nom_balances.npy` — prose-only provenance): spot-check key values against the cited publication and record what was checked; full verification may be impossible, and the manifest should say so (`status: unverifiable` with a one-line `note` — the honest known status, per P1) - [ ] **Unverifiable or failing files**: flag in the manifest and open an issue — do not promote a file to the canonical URL namespace with a known-bad or unknown integrity status diff --git a/builders/README.md b/builders/README.md new file mode 100644 index 0000000..28393be --- /dev/null +++ b/builders/README.md @@ -0,0 +1,50 @@ +# builders + +One builder per published dataset. This directory sits **outside** the published +tree — it is never served. + +## The naming rule + +`builders/.` builds `lectures/.`. + +The stem is the dataset's, not the lecture's: `builders/japan_earthquakes.py` +writes `lectures/japan_earthquakes.csv`. That makes the manifest's `builder:` +field predictable and lets CI assert it. + +**Where one builder produces a set of files**, name it for the set and let each +file's manifest point at the same builder path. `business_cycle.py` writes three +files; the SCF and Forbes builders each write two. The rule is the default, not +an invariant — what CI asserts is that every `builder:` path exists, and that a +dataset claiming a builder names one. + +## The contract + +Builders follow four stages — **fetch → pre-process → validate → write** — and +only write on validation pass. Lectures always read the last-good snapshot: an +upstream outage may fail a refresh, it must never break a lecture build. The +template and the architecture discussion are in +[#14](https://github.com/QuantEcon/data-lectures/issues/14). + +Most builders fetch from the third-party upstream at run time, which is the +normal case. A builder reads from `sources/` only when its input **cannot be +re-fetched** — see `AGENTS.md`. + +## What is here + +| Builder | Writes to `lectures/` | Status | +| --- | --- | --- | +| `ames_house_prices.py` | `ames_house_prices.csv` | committed | +| `epl_match_goals.py` | `epl_match_goals.csv` | committed | +| `japan_deaths_by_age.py` | `japan_deaths_by_age.csv` | committed | +| `japan_earthquakes.py` | `japan_earthquakes.csv` | committed | +| `japan_population_by_age.py` | `japan_population_by_age.csv` | committed | +| `us_adult_heights.py` | `us_adult_heights.csv` | committed | +| `business_cycle.py` | `business_cycle_data.csv`, `business_cycle_info.md`, `business_cycle_metadata.md` | run by hand, no validate stage yet (PLAN Phase 5); its three outputs are the repo's only unmanifested files | + +**This listing is the coverage report.** The repo has 13 `constructed` datasets +and 7 builders; the difference is the Phase 9 recovery backlog, carried as +`builder_status: unrecovered` in each manifest rather than hidden by +reclassifying the file as `verbatim`. + +Repo tooling — the audit dashboard and the catalog generator — lives in +`scripts/` and is not a builder. diff --git a/scripts/ames_house_prices.py b/builders/ames_house_prices.py similarity index 100% rename from scripts/ames_house_prices.py rename to builders/ames_house_prices.py diff --git a/scripts/business_cycle.py b/builders/business_cycle.py similarity index 100% rename from scripts/business_cycle.py rename to builders/business_cycle.py diff --git a/scripts/epl_match_goals.py b/builders/epl_match_goals.py similarity index 100% rename from scripts/epl_match_goals.py rename to builders/epl_match_goals.py diff --git a/scripts/japan_deaths_by_age.py b/builders/japan_deaths_by_age.py similarity index 100% rename from scripts/japan_deaths_by_age.py rename to builders/japan_deaths_by_age.py diff --git a/scripts/japan_earthquakes.py b/builders/japan_earthquakes.py similarity index 100% rename from scripts/japan_earthquakes.py rename to builders/japan_earthquakes.py diff --git a/scripts/japan_population_by_age.py b/builders/japan_population_by_age.py similarity index 100% rename from scripts/japan_population_by_age.py rename to builders/japan_population_by_age.py diff --git a/scripts/us_adult_heights.py b/builders/us_adult_heights.py similarity index 100% rename from scripts/us_adult_heights.py rename to builders/us_adult_heights.py diff --git a/lectures/ames_house_prices.csv.yml b/lectures/ames_house_prices.csv.yml index e42155a..20d8fca 100644 --- a/lectures/ames_house_prices.csv.yml +++ b/lectures/ames_house_prices.csv.yml @@ -71,5 +71,5 @@ consumers: - repo: QuantEcon/lecture-python-intro file: lectures/fitting_distributions.md -builder: scripts/ames_house_prices.py +builder: builders/ames_house_prices.py builder_status: committed diff --git a/lectures/epl_match_goals.csv.yml b/lectures/epl_match_goals.csv.yml index 72c5860..864bc2d 100644 --- a/lectures/epl_match_goals.csv.yml +++ b/lectures/epl_match_goals.csv.yml @@ -65,5 +65,5 @@ consumers: - repo: QuantEcon/lecture-python-intro file: lectures/fitting_distributions.md -builder: scripts/epl_match_goals.py +builder: builders/epl_match_goals.py builder_status: committed diff --git a/lectures/japan_deaths_by_age.csv.yml b/lectures/japan_deaths_by_age.csv.yml index 86a16f6..e70b399 100644 --- a/lectures/japan_deaths_by_age.csv.yml +++ b/lectures/japan_deaths_by_age.csv.yml @@ -74,5 +74,5 @@ consumers: - repo: QuantEcon/lecture-python-intro file: lectures/fitting_distributions.md -builder: scripts/japan_deaths_by_age.py +builder: builders/japan_deaths_by_age.py builder_status: committed diff --git a/lectures/japan_earthquakes.csv.yml b/lectures/japan_earthquakes.csv.yml index a6a86f0..140a2f3 100644 --- a/lectures/japan_earthquakes.csv.yml +++ b/lectures/japan_earthquakes.csv.yml @@ -75,5 +75,5 @@ consumers: - repo: QuantEcon/lecture-python-intro file: lectures/fitting_distributions.md -builder: scripts/japan_earthquakes.py +builder: builders/japan_earthquakes.py builder_status: committed diff --git a/lectures/japan_population_by_age.csv.yml b/lectures/japan_population_by_age.csv.yml index 10f369b..e13cae4 100644 --- a/lectures/japan_population_by_age.csv.yml +++ b/lectures/japan_population_by_age.csv.yml @@ -85,5 +85,5 @@ consumers: - repo: QuantEcon/lecture-python-intro file: lectures/prob_dist.md -builder: scripts/japan_population_by_age.py +builder: builders/japan_population_by_age.py builder_status: committed diff --git a/lectures/us_adult_heights.csv.yml b/lectures/us_adult_heights.csv.yml index 60aaf64..5d2bd40 100644 --- a/lectures/us_adult_heights.csv.yml +++ b/lectures/us_adult_heights.csv.yml @@ -82,5 +82,5 @@ consumers: - repo: QuantEcon/lecture-python-intro file: lectures/fitting_distributions.md -builder: scripts/us_adult_heights.py +builder: builders/us_adult_heights.py builder_status: committed diff --git a/manifest-schema.yml b/manifest-schema.yml index ba522c9..cc69f42 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -167,7 +167,7 @@ consumers: [] # Builder — required for constructed and dynamic-snapshot; omit for verbatim # --------------------------------------------------------------------------- -builder: scripts/business_cycle.py # path to the committed builder, or null +builder: builders/business_cycle.py # path to the committed builder, or null # AGENTS.md says a constructed dataset without its builder is a bug — but the # repo has inherited several (PLAN Phase 9), and P1's own pilot file is one. @@ -175,7 +175,7 @@ builder: scripts/business_cycle.py # path to the committed builder, or null # the gap. This field keeps it visible in the generated catalog instead. # `unrecovered` is for inherited files only; a NEW constructed file must ship # its builder (AGENTS.md, "Two inherited-file states"). -builder_status: committed # committed | unrecovered | not-applicable +builder_status: committed # committed | committed-frozen | unrecovered | not-applicable # Dynamic snapshots only. Drives the scheduled refresh-as-PR (PLAN Phase 5). cadence: annual # e.g. daily | weekly | monthly | annual diff --git a/scripts/README.md b/scripts/README.md index 215296d..a05acf6 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,25 +1,13 @@ # scripts -Builders for the datasets in `lectures/`. This directory sits **outside** the -published tree — it is never served. +Repo tooling. This directory sits **outside** the published tree — it is never +served, and nothing here produces a dataset. -Every **constructed** and **dynamic snapshot** dataset must ship its builder -here (see `AGENTS.md`). A constructed dataset without a committed builder is a -bug. - -| Builder | Writes to `lectures/` | -| --- | --- | -| `business_cycle.py` | `business_cycle_data.csv`, `business_cycle_info.md`, `business_cycle_metadata.md` | - -`business_cycle.py` is run by hand today and has no validate stage; retrofitting -it to the four-stage contract (fetch → pre-process → validate → write) is -PLAN Phase 5. - -## Not builders +Dataset builders live in [`builders/`](../builders/), one per published file. | Script | What | | --- | --- | -| `build_catalog.py` | generates `CATALOG.md` from the manifests (`lectures/*.yml`) | -| `build_audit.py` | the audit dashboard: scans the 8 lecture repos, writes `audit.json`, renders `site/` | +| `build_audit.py` | the audit dashboard: scans the 8 lecture repos, writes `audit.json`, renders `site/`. `--strict` fails on an unannotated data reference, on migration-status drift, and on a repoint that names a host, ref or path this repo does not serve | | `render_audit.py` | the render stage of `build_audit.py` (HTML generation) | +| `build_catalog.py` | generates `CATALOG.md` from the manifests (`lectures/*.yml`) | | `audit_annotations.yml` | curated judgment for not-yet-migrated data references — the strict scan fails when a new reference has no entry here and no manifest |