diff --git a/AGENTS.md b/AGENTS.md index d1123ef..6b00d45 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ Guidance for coding agents (and humans) making changes in this repository. Read ## What this repo is -The canonical home for **data consumed by the QuantEcon lecture series** (renamed from `QuantEcon/data` on 2026-07-16, per [meta#336](https://github.com/QuantEcon/meta/issues/336)). It is **mid-transition**: the current tree is a legacy consumer-keyed layout (`lecture-python-intro/…`) that will become a flat published tree served at `https://data.quantecon.org/lectures/`. The full convention lives in the draft manual page ([QuantEcon.manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108)). +The canonical home for **data consumed by the QuantEcon lecture series** (renamed from `QuantEcon/data` on 2026-07-16, per [meta#336](https://github.com/QuantEcon/meta/issues/336)). Its purpose is **stability**: it snapshots upstream sources — with attribution to each source carried in the manifest — so a lecture build never depends on a live API or a third-party host staying up. It is a **cache, not a content-distribution host**. It is **mid-transition**: the current tree is a legacy consumer-keyed layout (`lecture-python-intro/…`) that will become a flat published tree served at `https://data.quantecon.org/lectures/`. The full convention lives in the draft manual page ([QuantEcon.manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108)). ## Rules @@ -23,7 +23,18 @@ Classify as exactly one of: | **constructed** | built from upstream sources by our processing | all of the above **plus the builder script, committed here** | | **dynamic snapshot** | constructed, tracking a moving source (FRED, World Bank) | all of the above **plus a refresh cadence** | -A constructed dataset without its committed builder is a bug. Manifest fields: `source`, `license`, `retrieved`, `schema`, `consumers` (repo + lecture file, machine-readable), `maintainer`, `cadence` (dynamic only). +A constructed dataset without its committed builder is a bug. Manifest fields: `source`, `license` (with the `verified` date it was established), `retrieved`, `integrity` (`sha256` plus the `upstream` verification status, see Phase 7), `schema` (including `known_nulls`), `consumers` (repo + lecture file, machine-readable), `maintainer`, `builder` / `builder_status`, `cadence` (dynamic only). `manifest-schema.yml` is the authoritative, commented field reference — keep it and this list in step. + +**Verifying `integrity.upstream`, by class** (once here, not repeated per manifest): re-fetch-and-compare for **verbatim**; re-run the builder and compare the overlap window for **constructed / dynamic**; spot-check against the cited publication for author-assembled. When verification is impossible, say so plainly — `status: unverifiable` with a one-line `note` is a known status the catalog can show; silence is not. **Migration safety** (does the file byte-match what the consuming lecture used before a repoint?) is deliberately *not* a manifest field: it is a one-time gate recorded in the repoint PR, and the manifest's `sha256` keeps it reproducible afterwards. + +**Capture what the source gives you; never let a missing field block a useful dataset.** Rich provenance — DOI, upstream version, exact retrieval date, licence id — is always welcome and worth recording whenever it is available, because it makes the data auditable years later at almost no ongoing cost. But effort scales with what the source actually provides: where a field is genuinely unavailable, record it as an explicit, reasoned gap (see the inherited-file states below) rather than fabricating it or refusing the file. A clean, well-documented source should produce a short manifest; only genuinely messy provenance earns a long one. + +#### Two inherited-file states that look like violations but are tracked, not hidden + +The Feb 2025 migration left files that cannot fully satisfy the rules above. The manifest records each gap **explicitly** — visible in the generated catalog — rather than burying it by misclassification. Both are provisional decisions from the P1 pilot ([meta#338](https://github.com/QuantEcon/meta/issues/338)), to be folded into [manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108). + +- **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that says why (`status: unverifiable` with a `note`). +- **`builder_status: unrecovered` — constructed without a recoverable builder.** A constructed dataset ships its builder, and one that omits it *silently* is the bug. Several inherited files are constructed with no recoverable extraction steps (PLAN Phase 9 tracks them). Keep `class: constructed` — reclassifying to `verbatim` to dodge the rule is misclassification — set `builder: null` and `builder_status: unrecovered`, and the gap stays visible for Phase 9 to recover. `unrecovered` is for **inherited files only**; never introduce a *new* constructed file without its builder. ### Corrections vs vintages @@ -59,9 +70,11 @@ Builders follow four stages — **fetch → pre-process → validate → write** Live API calls are for *teaching data access*, not for getting data. Don't propose "the lecture should just call the API" as a fix — the fix is a snapshot here plus an automated refresh. -### Licensing +### Licensing and attribution + +Because this repo is a **stability cache, not a content-distribution host** (see "What this repo is"), the licence question is *"is this source OK to cache and serve publicly, with attribution?"* — not *"may we republish this as our own?"*. Attribution to the upstream source is carried in every manifest (`source`: name, url, series, citation), and that is the primary obligation. -Before adding any file, confirm the upstream license permits redistribution and record it in the manifest. This repo is on track to be a promoted public host; unlicensed rehosting is a blocker, not a nice-to-have. +For the public data sources most snapshots come from (World Bank, FRED, Eurostat, …) the answer is a **known yes, recorded once per source** — permissive terms plus attribution. Record what the source states and move on; don't re-litigate it per snapshot. Treat the manifest's `redistribution` field as a **cheap binary gate** (`permitted` / `restricted`): a fast `permitted` for public statistics agencies, `restricted` blocking only the genuinely restricted source before it goes public — e.g. FRED re-serves third-party series that may not be redistributed, and anything under non-commercial or no-redistribution terms must not be cached here, since attribution alone does not cure those. Capture licence detail richly when the source provides it; where it is genuinely unavailable, record the gap rather than blocking the file. ## Cross-repo hygiene diff --git a/PLAN.md b/PLAN.md index 6a35fa1..819089a 100644 --- a/PLAN.md +++ b/PLAN.md @@ -34,7 +34,7 @@ This repository is being shaped into the **single canonical repository for data ## Where we're going (per the draft convention) - **Flat published tree** served at `https://data.quantecon.org/lectures/` via GitHub Pages (custom domain), CORS-open for pyodide/JupyterLite -- Every dataset classified **verbatim / constructed / dynamic snapshot**, each with a manifest (`source`, `license`, `retrieved`, `schema`, `consumers`, `maintainer`, `cadence`) +- Every dataset classified **verbatim / constructed / dynamic snapshot**, each with a manifest — authoritative field reference in `manifest-schema.yml`, as revised by the P1 pilot (`integrity`, `builder_status`, `known_nulls` and `license.verified` joined the original sketch of `source` / `license` / `retrieved` / `schema` / `consumers` / `maintainer` / `cadence`) - Constructed and dynamic datasets ship their **builder**; dynamic datasets get **scheduled refresh-as-PR** plus a weekly **sources-alive canary** - Per-path LFS for large binaries only; storage choice invisible to consumers because URLs decouple from hosting @@ -77,8 +77,8 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020. ### Phase 5 — Automation (`.github/`) -- [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data -- [ ] Retrofit `scripts/business_cycle.py` to the four-stage builder contract — it has fetch/transform/write today but **no validate stage** +- [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data. The schema decisions these tests force — column patterns for wide files, `known_nulls` exact-vs-ceiling, a canonical dtype vocabulary — are researched in [#14](https://github.com/QuantEcon/data-lectures/issues/14) +- [ ] Retrofit `scripts/business_cycle.py` to the four-stage builder contract — it has fetch/transform/write today but **no validate stage**. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14) - [ ] Scheduled refresh workflow for dynamic datasets — cron per cadence class, runs the builder (fetch → pre-process → validate → write), lands the result as a PR whose diff summary (rows added, date-range delta, overlap-window changes) is the review surface; low-risk series may auto-merge on green (first consumer: the UNRATE pilot, meta#338 P4) - [ ] Weekly sources-alive canary: fetch + validate, no commit, opens an issue on failure — relocates API fragility from 7 lecture repos' CI into one scheduled job here - [ ] Consumer fan-out: a merged refresh or in-place correction dispatches rebuilds of the repos in the dataset's machine-readable `consumers` list @@ -88,7 +88,7 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020. - [ ] Manifest per dataset for the **9** files now in `lectures/`: source, license, retrieval date, schema, consumers, provenance class. Schema sketched in `manifest-schema.yml` (Phase 2); backfill is per-file work gated on the license check below - [ ] Classify: the 8 static intro files are author-assembled or verbatim; `business_cycle_data.csv` is the one dynamic snapshot and needs its cadence declared -- [ ] License check per file before this repo is promoted as the canonical public home. One is already answered: `business_cycle_metadata.md` records the World Bank series as **CC BY-4.0** — that dump is the model for what a manifest should capture, and the other 8 need the equivalent established by hand +- [ ] Licence check **per source**, not per file: the question is *"may this source be cached and served publicly, with attribution?"* — a cheap binary gate (`redistribution: permitted | restricted`, see AGENTS.md "Licensing and attribution"), a fast yes for public data sources. Two sources already answered: World Bank is **CC BY-4.0** (`business_cycle_metadata.md`, the model for what a manifest should capture) and RAM Legacy is **CC BY 4.0** (established against its Zenodo DOI record, P1). The remaining sources need the equivalent established by hand - [x] Keep-or-drop decision for the files with no consumer anywhere — **dropped 2026-07-16** in the Phase 2 restructure, rather than promoting them into the published namespace: - `GDP_per_capita_world_bank.csv` and `Metadata_Country_API_NY.GDP.PCAP.CD_DS2_en_csv_v2_4770417.csv` — an org-wide code search returns **zero** references to either, they are freely re-downloadable from the World Bank, and their licence was never established. Rehosting a stale snapshot nobody reads is the opposite of this repo's purpose - `fig_3.ods` — confirmed to carry no provenance the published `.xlsx` lacks: both parse to a single `Sheet1` of identical shape (34×6) and `DataFrame.equals` returns true, so it is a pure format twin @@ -98,10 +98,10 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020. Verify that what this repo holds is actually the data it claims to be — against upstream sources, and against the copies lectures consume today — before any lecture is repointed here. -- [ ] **Byte-compare against the in-use copies**: each file migrated in Feb 2025 must be identical to the copy `lecture-python-intro` currently consumes (git blob hash compare). If a copy diverged, a repoint silently changes lecture output — this check is a hard prerequisite for Phase 8 -- [ ] **Verbatim files**: re-fetch from the upstream source and compare (e.g. `mpd2020.xlsx` against the published Maddison Project 2020 release); record checksum, verification method, and date in the manifest +- [ ] **Byte-compare against the in-use copies**: each file migrated in Feb 2025 must be identical to the copy `lecture-python-intro` currently consumes (git blob hash compare). If a copy diverged, a repoint silently changes lecture output — this check is a hard prerequisite for Phase 8. Recorded **in the repoint PR** as a one-time gate, reproducible later from the manifest's `sha256` — not a manifest field (P1 decision) +- [ ] **Verbatim files**: re-fetch from the upstream source and compare (e.g. `mpd2020.xlsx` against the published Maddison Project 2020 release); record `sha256`, `status`, what it was compared `against`, and the date in the manifest's `integrity.upstream` - [ ] **Constructed / dynamic files**: re-run the committed builder (`scripts/business_cycle.py` → `business_cycle_data.csv`) and confirm values agree in the overlap window with the committed snapshot -- [ ] **Author-assembled files** (the French Revolution spreadsheets, `caron.npy`, `nom_balances.npy` — prose-only provenance): spot-check key values against the cited publication and record what was checked; full verification may be impossible, and the manifest should say so +- [ ] **Author-assembled files** (the French Revolution spreadsheets, `caron.npy`, `nom_balances.npy` — prose-only provenance): spot-check key values against the cited publication and record what was checked; full verification may be impossible, and the manifest should say so (`status: unverifiable` with a one-line `note` — the honest known status, per P1) - [ ] **Unverifiable or failing files**: flag in the manifest and open an issue — do not promote a file to the canonical URL namespace with a known-bad or unknown integrity status ### Phase 8 — Pilot deployment (meta#338) @@ -129,6 +129,6 @@ The first end-to-end deployment: one dataset per hosting pattern, each the harde | Repo name | **settled 2026-07-16**: renamed `data-lectures` (Phase 1) | | URL form | `data.quantecon.org/lectures/...`; interim `github.com/QuantEcon/data-lectures/raw/main/...` | | Layout | flat | -| Licensing review | per-source check before rehosting, recorded in the manifest | +| Licensing review | per-source cache-and-serve-with-attribution gate (`redistribution: permitted \| restricted`), recorded in the manifest — this repo is a stability cache, not a content host | When one of these settles, update this PLAN and `AGENTS.md` in the same PR that acts on it. diff --git a/lectures/lingcod_msy_recovery.csv b/lectures/lingcod_msy_recovery.csv new file mode 100644 index 0000000..ca84787 --- /dev/null +++ b/lectures/lingcod_msy_recovery.csv @@ -0,0 +1,83 @@ +year,B_over_Bmsy,F_over_Fmsy +1928,3.859289617,0.12815534 +1929,3.842213115,0.177669903 +1930,3.825136612,0.198058252 +1931,3.790983607,0.19223301 +1932,3.756830601,0.142718447 +1933,3.756830601,0.224271845 +1934,3.705601093,0.14368932 +1935,3.705601093,0.170873786 +1936,3.671448087,0.133009709 +1937,3.671448087,0.173786408 +1938,3.654371585,0.146601942 +1939,3.637295082,0.127184466 +1940,3.637295082,0.145631068 +1941,3.637295082,0.117475728 +1942,3.637295082,0.067961165 +1943,3.654371585,0.131067961 +1944,3.654371585,0.132038835 +1945,3.654371585,0.130097087 +1946,3.637295082,0.216504854 +1947,3.603142077,0.430097087 +1948,3.50068306,0.462135922 +1949,3.398224044,0.413592233 +1950,3.31284153,0.45631068 +1951,3.227459016,0.453398058 +1952,3.142076503,0.350485437 +1953,3.090846995,0.246601942 +1954,3.073770492,0.290291262 +1955,3.056693989,0.308737864 +1956,3.039617486,0.355339806 +1957,3.005464481,0.526213592 +1958,2.920081967,0.539805825 +1959,2.851775956,0.466990291 +1960,2.800546448,0.417475728 +1961,2.74931694,0.449514563 +1962,2.698087432,0.386407767 +1963,2.663934426,0.391262136 +1964,2.629781421,0.340776699 +1965,2.612704918,0.413592233 +1966,2.578551913,0.518446602 +1967,2.527322404,0.57961165 +1968,2.476092896,0.612621359 +1969,2.407786885,0.575728155 +1970,2.339480874,0.850485437 +1971,2.237021858,1.077669903 +1972,2.117486339,1.563106796 +1973,1.929644809,1.766990291 +1974,1.741803279,1.951456311 +1975,1.572745902,1.951456311 +1976,1.449795082,2.077669903 +1977,1.321721311,1.485436893 +1978,1.270491803,1.941747573 +1979,1.166325137,2.718446602 +1980,1.00068306,3.378640777 +1981,0.840163934,3.378640777 +1982,0.712090164,3.504854369 +1983,0.594262295,2.67961165 +1984,0.544740437,2.805825243 +1985,0.488387978,3.951456311 +1986,0.403005464,3.854368932 +1987,0.37568306,4.019417476 +1988,0.37397541,4.310679612 +1989,0.358606557,4.834951456 +1990,0.3125,4.553398058 +1991,0.285177596,4.32038835 +1992,0.268101093,4.174757282 +1993,0.256147541,3.485436893 +1994,0.266393443,2.27184466 +1995,0.315915301,2.019417476 +1996,0.365437158,2.038834951 +1997,0.396174863,1.815533981 +1998,0.425204918,1.300970874 +1999,0.479849727,1.398058252 +2000,0.539617486,0.63592233 +2001,0.650614754,0.49223301 +2002,0.800887978,1.029126214 +2003,0.988729508,1.291262136 +2004,1.243169399,0.195145631 +2005,1.656420765,0.261165049 +2006,2.083333333,0.249514563 +2007,2.510245902,0.144660194 +2008,2.885928962,0.093203883 +2009,3.193306011, diff --git a/lectures/lingcod_msy_recovery.csv.yml b/lectures/lingcod_msy_recovery.csv.yml new file mode 100644 index 0000000..8993913 --- /dev/null +++ b/lectures/lingcod_msy_recovery.csv.yml @@ -0,0 +1,99 @@ +# Manifest for lingcod_msy_recovery.csv — the first manifest written against +# the schema sketched in ../manifest-schema.yml (PLAN Phase 8, pilot P1). + +filename: lingcod_msy_recovery.csv +title: Pacific Coast lingcod — biomass and fishing pressure relative to MSY +description: > + Annual B/Bmsy and F/Fmsy for the U.S. Pacific Coast lingcod (Ophiodon + elongatus) fishery, 1928-2009. Illustrates a stock fished below its MSY + reference points and subsequently rebuilt. + +# Constructed, not verbatim: these are two ratio series for a single stock, +# subset out of a database that ships as a multi-table release. Selecting the +# stock and the columns is our processing, so the class is `constructed` — and +# a constructed dataset is supposed to ship its builder. This one does not. +# See `builder` below. +class: constructed + +source: + name: RAM Legacy Stock Assessment Database + url: https://www.ramlegacy.org/ + doi: 10.5281/zenodo.14043031 + version: v4.66-assessment-only # current upstream release; NOT known to be + # the release this extract was taken from + citation: > + Ricard, D., Minto, C., Jensen, O. P., & Baum, J. K. (2012). Evaluating the + knowledge base and status of commercially exploited marine species with the + RAM Legacy Stock Assessment Database. Fish and Fisheries, 13(4), 380-398. + +license: + name: CC BY 4.0 + url: https://creativecommons.org/licenses/by/4.0/ + redistribution: permitted + verified: 2026-07-16 + # Established against the Zenodo record for v4.66, which reports license id + # "cc-by-4.0" and access_right "open". Not stated on ramlegacy.org itself -- + # the homepage gives only the citation, so the Zenodo record is the + # authority. Attribution is the sole condition and the citation above + # carries it, so rehosting this extract is permitted. + +retrieved: null # unrecorded -- see integrity.upstream +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (not a manifest field — it is a one-time repoint gate): +# verified 2026-07-16 by sha256 byte-compare against lecture-python-intro's +# copy at lectures/datasets/lingcod_msy_recovery.csv, so the repoint cannot +# change lecture output. Recorded in data-lectures#12 / lecture-python-intro#792 +# and reproducible from the hash below at any later date. + +integrity: + sha256: fc2f873f97a9b7c6d335ca88203c321b4451cf4b94feb16ca40e99551c49c092 + upstream: + status: unverifiable + date: 2026-07-16 + against: null + note: > + No RAM Legacy stock id, database version, or transformation recorded, and + no builder committed — cannot be re-derived. Trust rests on the + provenance of the author, not an independent check. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- + +schema: + format: csv + columns: + - {name: year, dtype: int64, description: calendar year} + - {name: B_over_Bmsy, dtype: float64, description: stock biomass relative to the biomass supporting MSY} + - {name: F_over_Fmsy, dtype: float64, description: fishing pressure relative to the pressure achieving MSY} + row_count_floor: 82 + date_range: {start: 1928, end: 2009} + # F_over_Fmsy is null in the terminal year (2009) -- expected, and normal for + # a terminal assessment year. Recorded so PR validation does not "fix" it: + # a blanket no-nulls rule would reject this file wrongly. + known_nulls: + F_over_Fmsy: 1 + +# --------------------------------------------------------------------------- +# Consumers -- how a correction knows what to rebuild +# --------------------------------------------------------------------------- + +consumers: + - repo: QuantEcon/lecture-python-intro + file: lectures/msy_fishery.md + +# --------------------------------------------------------------------------- +# Builder +# --------------------------------------------------------------------------- +# AGENTS.md: "A constructed dataset without its committed builder is a bug." +# This file is that bug, inherited rather than introduced -- it is one of the +# constructed-but-unscripted set PLAN Phase 9 tracks. Recorded explicitly +# instead of being quietly misclassified as verbatim, so the gap is visible in +# the generated catalog rather than lost. + +builder: null +builder_status: unrecovered # unrecovered | committed | not-applicable diff --git a/manifest-schema.yml b/manifest-schema.yml index 184bc43..06cd8e7 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -50,26 +50,52 @@ source: license: name: CC BY-4.0 url: https://datacatalog.worldbank.org/public-licenses#cc-by - # Redistribution must be confirmed before a file is served from this repo — - # unlicensed rehosting is a blocker, not a nice-to-have (AGENTS.md). - redistribution: permitted - -retrieved: 2024-04-10 # ISO date the bytes were obtained + # This repo is a stability cache, not a content host: the question is whether + # the source may be cached and served publicly WITH attribution, not whether + # we may republish it as our own. A cheap binary gate (AGENTS.md, "Licensing + # and attribution") — a fast yes for public data sources, blocking only the + # genuinely restricted ones (e.g. FRED third-party series). + redistribution: permitted # permitted | restricted + verified: 2026-07-16 # when, and against what, this was + # established — a licence claim with no + # date is not evidence. P1 found the + # authority is often not the project + # homepage but its Zenodo/DOI record. + +retrieved: 2024-04-10 # ISO date the bytes were obtained, or + # null for inherited-undated files — + # never invent one from git history. + # A null here MUST be paired with an + # integrity.upstream note (AGENTS.md, + # "Two inherited-file states"). maintainer: QuantEcon # who fixes this when it breaks # --------------------------------------------------------------------------- -# Integrity (PLAN Phase 7) — a file is not promoted with unknown status +# Integrity (PLAN Phase 7) — record the status honestly, don't fake it # --------------------------------------------------------------------------- +# One permanent axis: is this file actually what the upstream source says? +# (How to check, by class, lives once in AGENTS.md, not per manifest.) +# +# Migration safety — does the file match what the consuming lecture used before +# a repoint? — is deliberately NOT a manifest field. It is a one-time check that +# lives in the repoint PR (a Phase 8 gate), and the sha256 below is what makes +# it reproducible at any later date. P1 (lingcod) proved the two questions get +# different answers: byte-perfect copy of what the lecture consumed, upstream +# provenance unrecoverable. integrity: sha256: null # of the committed file - verified: null # ISO date of the last check - method: > - How this was checked: re-fetch-and-compare for verbatim; re-run the builder - and compare the overlap window for constructed/dynamic; spot-check against - the cited publication for author-assembled. Say so plainly when full - verification is not possible. - status: unverified # verified | spot-checked | unverified | failing + upstream: + status: unverified # verified | spot-checked | unverifiable | unverified | failing + date: null # ISO date of the last check + against: null # what a verified check compared to (URL, + # DOI, or repo path) — makes "verified" + # auditable rather than a bare assertion + note: null # ONE line, only when status is not a + # clean verified — e.g. why a file is + # unverifiable. An honest `unverifiable` + # with a note is a known status the + # catalog can show; silence is not. # --------------------------------------------------------------------------- # Shape — what a consumer can rely on; also the basis for PR validation tests @@ -81,9 +107,18 @@ schema: - {name: economy, dtype: string, description: ISO3 country code} - {name: Country, dtype: string, description: Country name} - {name: YR1960, dtype: float, description: Annual GDP growth (%), one column per year} - row_count_floor: 5 # PR validation fails below this + row_count_floor: 5 # PR validation fails below this. For a + # frozen historical extract set it to the + # exact row count by design (P1 lingcod = + # 82); a loose floor is for files that grow. date_range: {start: 1960, end: null} # null end = tracks the moving source + # Nulls that are correct and must not fail validation. P1 found a real one: + # lingcod's F/Fmsy is null in its terminal assessment year, which a blanket + # no-nulls rule would reject. Declaring them makes the invariant tests + # (Phase 5) enforceable without hand-waving. + known_nulls: {} # e.g. {F_over_Fmsy: 1} + # --------------------------------------------------------------------------- # Consumers — machine-readable, so a correction knows what to rebuild # --------------------------------------------------------------------------- @@ -100,7 +135,15 @@ consumers: [] # Builder — required for constructed and dynamic-snapshot; omit for verbatim # --------------------------------------------------------------------------- -builder: scripts/business_cycle.py +builder: scripts/business_cycle.py # path to the committed builder, or null + +# AGENTS.md says a constructed dataset without its builder is a bug — but the +# repo has inherited several (PLAN Phase 9), and P1's own pilot file is one. +# The tempting workaround is to misclassify them as `verbatim`, which buries +# the gap. This field keeps it visible in the generated catalog instead. +# `unrecovered` is for inherited files only; a NEW constructed file must ship +# its builder (AGENTS.md, "Two inherited-file states"). +builder_status: committed # committed | unrecovered | not-applicable # Dynamic snapshots only. Drives the scheduled refresh-as-PR (PLAN Phase 5). cadence: annual # e.g. daily | weekly | monthly | annual