From e2d3bf9728f7367948793a4f374bbe1e8c8012dd Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 16 Jul 2026 09:25:25 +1000 Subject: [PATCH 1/6] P1 pilot: add lingcod_msy_recovery.csv with its manifest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first dataset to land in the published tree, and the first manifest written against the schema sketch. Consumed by lecture-python-intro's msy_fishery lecture, which currently reads it from a relative path — so the downloaded notebook cannot resolve it. The repoint is the paired lecture PR. Licence established before adding, per AGENTS.md: RAM Legacy is CC BY 4.0, attribution-only, so rehosting this extract is permitted. Worth recording how that was found — ramlegacy.org states no licence at all; the authority is the Zenodo record behind its DOI (license id "cc-by-4.0", access_right "open"). Meeting a real file changed the schema in three ways, all folded back into manifest-schema.yml: - integrity splits into `migration` and `upstream`. Lingcod is a byte-perfect copy of what the lecture consumes, yet its upstream provenance cannot be re-derived at all. One "verified" field would have to overstate or understate it; two fields tell the truth. - `builder_status` records `unrecovered`. This file is constructed but ships no builder — the bug AGENTS.md names. The tempting fix is to relabel it verbatim, which buries the gap; instead it is now visible in the catalog. - `known_nulls` declares nulls that are correct. F/Fmsy is null in the terminal assessment year, which a blanket no-nulls validation rule would wrongly reject. Part of QuantEcon/meta#338 See QuantEcon/meta#336 Co-Authored-By: Claude Fable 5 --- lectures/lingcod_msy_recovery.csv | 83 +++++++++++++++++++ lectures/lingcod_msy_recovery.csv.yml | 111 ++++++++++++++++++++++++++ manifest-schema.yml | 50 ++++++++++-- 3 files changed, 236 insertions(+), 8 deletions(-) create mode 100644 lectures/lingcod_msy_recovery.csv create mode 100644 lectures/lingcod_msy_recovery.csv.yml diff --git a/lectures/lingcod_msy_recovery.csv b/lectures/lingcod_msy_recovery.csv new file mode 100644 index 0000000..ca84787 --- /dev/null +++ b/lectures/lingcod_msy_recovery.csv @@ -0,0 +1,83 @@ +year,B_over_Bmsy,F_over_Fmsy +1928,3.859289617,0.12815534 +1929,3.842213115,0.177669903 +1930,3.825136612,0.198058252 +1931,3.790983607,0.19223301 +1932,3.756830601,0.142718447 +1933,3.756830601,0.224271845 +1934,3.705601093,0.14368932 +1935,3.705601093,0.170873786 +1936,3.671448087,0.133009709 +1937,3.671448087,0.173786408 +1938,3.654371585,0.146601942 +1939,3.637295082,0.127184466 +1940,3.637295082,0.145631068 +1941,3.637295082,0.117475728 +1942,3.637295082,0.067961165 +1943,3.654371585,0.131067961 +1944,3.654371585,0.132038835 +1945,3.654371585,0.130097087 +1946,3.637295082,0.216504854 +1947,3.603142077,0.430097087 +1948,3.50068306,0.462135922 +1949,3.398224044,0.413592233 +1950,3.31284153,0.45631068 +1951,3.227459016,0.453398058 +1952,3.142076503,0.350485437 +1953,3.090846995,0.246601942 +1954,3.073770492,0.290291262 +1955,3.056693989,0.308737864 +1956,3.039617486,0.355339806 +1957,3.005464481,0.526213592 +1958,2.920081967,0.539805825 +1959,2.851775956,0.466990291 +1960,2.800546448,0.417475728 +1961,2.74931694,0.449514563 +1962,2.698087432,0.386407767 +1963,2.663934426,0.391262136 +1964,2.629781421,0.340776699 +1965,2.612704918,0.413592233 +1966,2.578551913,0.518446602 +1967,2.527322404,0.57961165 +1968,2.476092896,0.612621359 +1969,2.407786885,0.575728155 +1970,2.339480874,0.850485437 +1971,2.237021858,1.077669903 +1972,2.117486339,1.563106796 +1973,1.929644809,1.766990291 +1974,1.741803279,1.951456311 +1975,1.572745902,1.951456311 +1976,1.449795082,2.077669903 +1977,1.321721311,1.485436893 +1978,1.270491803,1.941747573 +1979,1.166325137,2.718446602 +1980,1.00068306,3.378640777 +1981,0.840163934,3.378640777 +1982,0.712090164,3.504854369 +1983,0.594262295,2.67961165 +1984,0.544740437,2.805825243 +1985,0.488387978,3.951456311 +1986,0.403005464,3.854368932 +1987,0.37568306,4.019417476 +1988,0.37397541,4.310679612 +1989,0.358606557,4.834951456 +1990,0.3125,4.553398058 +1991,0.285177596,4.32038835 +1992,0.268101093,4.174757282 +1993,0.256147541,3.485436893 +1994,0.266393443,2.27184466 +1995,0.315915301,2.019417476 +1996,0.365437158,2.038834951 +1997,0.396174863,1.815533981 +1998,0.425204918,1.300970874 +1999,0.479849727,1.398058252 +2000,0.539617486,0.63592233 +2001,0.650614754,0.49223301 +2002,0.800887978,1.029126214 +2003,0.988729508,1.291262136 +2004,1.243169399,0.195145631 +2005,1.656420765,0.261165049 +2006,2.083333333,0.249514563 +2007,2.510245902,0.144660194 +2008,2.885928962,0.093203883 +2009,3.193306011, diff --git a/lectures/lingcod_msy_recovery.csv.yml b/lectures/lingcod_msy_recovery.csv.yml new file mode 100644 index 0000000..c21199c --- /dev/null +++ b/lectures/lingcod_msy_recovery.csv.yml @@ -0,0 +1,111 @@ +# Manifest for lingcod_msy_recovery.csv — the first manifest written against +# the schema sketched in ../manifest-schema.yml (PLAN Phase 8, pilot P1). + +filename: lingcod_msy_recovery.csv +title: Pacific Coast lingcod — biomass and fishing pressure relative to MSY +description: > + Annual B/Bmsy and F/Fmsy for the U.S. Pacific Coast lingcod (Ophiodon + elongatus) fishery, 1928-2009. Illustrates a stock fished below its MSY + reference points and subsequently rebuilt. + +# Constructed, not verbatim: these are two ratio series for a single stock, +# subset out of a database that ships as a multi-table release. Selecting the +# stock and the columns is our processing, so the class is `constructed` — and +# a constructed dataset is supposed to ship its builder. This one does not. +# See `builder` below. +class: constructed + +source: + name: RAM Legacy Stock Assessment Database + url: https://www.ramlegacy.org/ + doi: 10.5281/zenodo.14043031 + version: v4.66-assessment-only # current upstream release; NOT known to be + # the release this extract was taken from + citation: > + Ricard, D., Minto, C., Jensen, O. P., & Baum, J. K. (2012). Evaluating the + knowledge base and status of commercially exploited marine species with the + RAM Legacy Stock Assessment Database. Fish and Fisheries, 13(4), 380-398. + +license: + name: CC BY 4.0 + url: https://creativecommons.org/licenses/by/4.0/ + redistribution: permitted + verified: 2026-07-16 + # Established against the Zenodo record for v4.66, which reports license id + # "cc-by-4.0" and access_right "open". Not stated on ramlegacy.org itself -- + # the homepage gives only the citation, so the Zenodo record is the + # authority. Attribution is the sole condition and the citation above + # carries it, so rehosting this extract is permitted. + +retrieved: null # unrecorded -- see integrity.upstream +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Two independent questions with different answers here, which is why they are +# recorded separately rather than as one status field: +# migration -- does this match what the lecture consumed before the repoint? +# upstream -- is this actually what RAM Legacy says? +# A file can be a perfect copy of a lecture's data and still have unprovable +# provenance. Collapsing both into one "verified" would overstate this file. + +integrity: + sha256: fc2f873f97a9b7c6d335ca88203c321b4451cf4b94feb16ca40e99551c49c092 + + migration: + status: verified + date: 2026-07-16 + method: > + sha256 byte-compare against lecture-python-intro's copy at + lectures/datasets/lingcod_msy_recovery.csv, immediately before the + repoint. Identical, so the repoint cannot change lecture output. + + upstream: + status: unverifiable + date: 2026-07-16 + method: > + Cannot be re-derived. The extract does not record which RAM Legacy stock + id it came from, which database version, or what transformation produced + the two ratio columns, and no builder was committed with it. Verifying + would mean re-identifying the stock in a current release and confirming + the series match -- possible in principle, not done. Treat the values as + trusted-by-provenance-of-the-author, not as independently checked. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- + +schema: + format: csv + columns: + - {name: year, dtype: int64, description: calendar year} + - {name: B_over_Bmsy, dtype: float64, description: stock biomass relative to the biomass supporting MSY} + - {name: F_over_Fmsy, dtype: float64, description: fishing pressure relative to the pressure achieving MSY} + row_count_floor: 82 + date_range: {start: 1928, end: 2009} + # F_over_Fmsy is null in the terminal year (2009) -- expected, and normal for + # a terminal assessment year. Recorded so PR validation does not "fix" it: + # a blanket no-nulls rule would reject this file wrongly. + known_nulls: + F_over_Fmsy: 1 + +# --------------------------------------------------------------------------- +# Consumers -- how a correction knows what to rebuild +# --------------------------------------------------------------------------- + +consumers: + - repo: QuantEcon/lecture-python-intro + file: lectures/msy_fishery.md + +# --------------------------------------------------------------------------- +# Builder +# --------------------------------------------------------------------------- +# AGENTS.md: "A constructed dataset without its committed builder is a bug." +# This file is that bug, inherited rather than introduced -- it is one of the +# constructed-but-unscripted set PLAN Phase 9 tracks. Recorded explicitly +# instead of being quietly misclassified as verbatim, so the gap is visible in +# the generated catalog rather than lost. + +builder: null +builder_status: unrecovered # unrecovered | committed | not-applicable diff --git a/manifest-schema.yml b/manifest-schema.yml index 184bc43..b562040 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -53,6 +53,11 @@ license: # Redistribution must be confirmed before a file is served from this repo — # unlicensed rehosting is a blocker, not a nice-to-have (AGENTS.md). redistribution: permitted + verified: 2026-07-16 # when, and against what, this was + # established — a licence claim with no + # date is not evidence. P1 found the + # authority is often not the project + # homepage but its Zenodo/DOI record. retrieved: 2024-04-10 # ISO date the bytes were obtained maintainer: QuantEcon # who fixes this when it breaks @@ -60,16 +65,33 @@ maintainer: QuantEcon # who fixes this when it breaks # --------------------------------------------------------------------------- # Integrity (PLAN Phase 7) — a file is not promoted with unknown status # --------------------------------------------------------------------------- +# Two independent questions, and P1 (lingcod) proved they get different answers: +# that file is a byte-perfect copy of what the lecture consumed, while its +# upstream provenance cannot be re-derived at all. One combined "verified" field +# would have had to either overstate or understate it, so they are split. integrity: sha256: null # of the committed file - verified: null # ISO date of the last check - method: > - How this was checked: re-fetch-and-compare for verbatim; re-run the builder - and compare the overlap window for constructed/dynamic; spot-check against - the cited publication for author-assembled. Say so plainly when full - verification is not possible. - status: unverified # verified | spot-checked | unverified | failing + + # Does this match what the consuming lecture used before the repoint? + # Guards the migration: a mismatch means repointing silently changes output. + # Hard prerequisite for Phase 8. + migration: + status: unverified # verified | unverified | failing | not-applicable + date: null + method: null # e.g. sha256 compare against 's copy + + # Is this actually what the upstream source says? + # Guards the provenance claim above. + upstream: + status: unverified # verified | spot-checked | unverifiable | unverified | failing + date: null + method: > + re-fetch-and-compare for verbatim; re-run the builder and compare the + overlap window for constructed/dynamic; spot-check against the cited + publication for author-assembled. Say so plainly when verification is + impossible — `unverifiable` with a reason is an honest answer and is + what the catalog should show. Silence is not. # --------------------------------------------------------------------------- # Shape — what a consumer can rely on; also the basis for PR validation tests @@ -84,6 +106,12 @@ schema: row_count_floor: 5 # PR validation fails below this date_range: {start: 1960, end: null} # null end = tracks the moving source + # Nulls that are correct and must not fail validation. P1 found a real one: + # lingcod's F/Fmsy is null in its terminal assessment year, which a blanket + # no-nulls rule would reject. Declaring them makes the invariant tests + # (Phase 5) enforceable without hand-waving. + known_nulls: {} # e.g. {F_over_Fmsy: 1} + # --------------------------------------------------------------------------- # Consumers — machine-readable, so a correction knows what to rebuild # --------------------------------------------------------------------------- @@ -100,7 +128,13 @@ consumers: [] # Builder — required for constructed and dynamic-snapshot; omit for verbatim # --------------------------------------------------------------------------- -builder: scripts/business_cycle.py +builder: scripts/business_cycle.py # path to the committed builder, or null + +# AGENTS.md says a constructed dataset without its builder is a bug — but the +# repo has inherited several (PLAN Phase 9), and P1's own pilot file is one. +# The tempting workaround is to misclassify them as `verbatim`, which buries +# the gap. This field keeps it visible in the generated catalog instead. +builder_status: committed # committed | unrecovered | not-applicable # Dynamic snapshots only. Drives the scheduled refresh-as-PR (PLAN Phase 5). cadence: annual # e.g. daily | weekly | monthly | annual From 8d9a1679a8dca1ac40abcb8c3fbf001bbc6146f8 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Fri, 17 Jul 2026 08:43:06 +1000 Subject: [PATCH 2/6] Codify the two inherited-file manifest states (P1 decisions) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The P1 lingcod manifest declares `retrieved: null` and `class: constructed` with `builder: null` / `builder_status: unrecovered`. Copilot flagged both as out of compliance with AGENTS.md — correctly, because the rules had no wording for inherited files. Rather than patch the manifest to dodge the rules (which would mean inventing a retrieval date or misclassifying a constructed file as verbatim), bless both states explicitly so the gap stays visible in the generated catalog and the rules and the manifest stop contradicting. - AGENTS.md: add "Two inherited-file states" under the manifest rules — `retrieved: null` permitted for inherited-undated bytes (paired with an integrity.upstream reason; never reconstructed from git history), and `builder_status: unrecovered` for inherited constructed files (Phase 9), inherited-only so a new constructed file still must ship its builder. - manifest-schema.yml: document the `retrieved` null policy and scope `unrecovered` to inherited files at the two field sites. Provisional P1 decisions, to be folded into the convention. See QuantEcon/meta#338 and QuantEcon/QuantEcon.manual#108. --- AGENTS.md | 7 +++++++ manifest-schema.yml | 9 ++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index d1123ef..21ef767 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,6 +25,13 @@ Classify as exactly one of: A constructed dataset without its committed builder is a bug. Manifest fields: `source`, `license`, `retrieved`, `schema`, `consumers` (repo + lecture file, machine-readable), `maintainer`, `cadence` (dynamic only). +#### Two inherited-file states that look like violations but are tracked, not hidden + +The Feb 2025 migration left files that cannot fully satisfy the rules above. The manifest records each gap **explicitly** — visible in the generated catalog — rather than burying it by misclassification. Both are provisional decisions from the P1 pilot ([meta#338](https://github.com/QuantEcon/meta/issues/338)), to be folded into [manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108). + +- **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that says why (`status: unverifiable` with a reason). +- **`builder_status: unrecovered` — constructed without a recoverable builder.** A constructed dataset ships its builder, and one that omits it *silently* is the bug. Several inherited files are constructed with no recoverable extraction steps (PLAN Phase 9 tracks them). Keep `class: constructed` — reclassifying to `verbatim` to dodge the rule is misclassification — set `builder: null` and `builder_status: unrecovered`, and the gap stays visible for Phase 9 to recover. `unrecovered` is for **inherited files only**; never introduce a *new* constructed file without its builder. + ### Corrections vs vintages - **Corrections** (bad parse, wrong units, corrupt rows): fix **in place**, same filename — every consumer should get the fix. Use the manifest's `consumers` list to know which lectures to rebuild/review. diff --git a/manifest-schema.yml b/manifest-schema.yml index b562040..d402e13 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -59,7 +59,12 @@ license: # authority is often not the project # homepage but its Zenodo/DOI record. -retrieved: 2024-04-10 # ISO date the bytes were obtained +retrieved: 2024-04-10 # ISO date the bytes were obtained, or + # null for inherited-undated files — + # never invent one from git history. + # A null here MUST be paired with an + # integrity.upstream reason (AGENTS.md, + # "Two inherited-file states"). maintainer: QuantEcon # who fixes this when it breaks # --------------------------------------------------------------------------- @@ -134,6 +139,8 @@ builder: scripts/business_cycle.py # path to the committed builder, or null # repo has inherited several (PLAN Phase 9), and P1's own pilot file is one. # The tempting workaround is to misclassify them as `verbatim`, which buries # the gap. This field keeps it visible in the generated catalog instead. +# `unrecovered` is for inherited files only; a NEW constructed file must ship +# its builder (AGENTS.md, "Two inherited-file states"). builder_status: committed # committed | unrecovered | not-applicable # Dynamic snapshots only. Drives the scheduled refresh-as-PR (PLAN Phase 5). From 39ee267d6621e11b10700b7f69dae391ae1a992a Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Fri, 17 Jul 2026 08:51:18 +1000 Subject: [PATCH 3/6] Tidy manifest docs after the P1 pilot Two small doc corrections surfaced reviewing the pilot: - AGENTS.md's manifest field list was written for the #10 schema sketch and never gained the fields P1 added: license.verified, the integrity migration/upstream split, schema.known_nulls, and builder_status. Bring it current and point to manifest-schema.yml as the authoritative field reference so the two stop drifting. - manifest-schema.yml: note that row_count_floor is the exact row count for a frozen historical extract by design (P1 lingcod = 82), so it is not later "relaxed" as if it were a mistake; a loose floor is for files that grow. See QuantEcon/meta#338 Co-Authored-By: Claude Opus 4.8 (1M context) --- AGENTS.md | 2 +- manifest-schema.yml | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 21ef767..d987d9f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,7 +23,7 @@ Classify as exactly one of: | **constructed** | built from upstream sources by our processing | all of the above **plus the builder script, committed here** | | **dynamic snapshot** | constructed, tracking a moving source (FRED, World Bank) | all of the above **plus a refresh cadence** | -A constructed dataset without its committed builder is a bug. Manifest fields: `source`, `license`, `retrieved`, `schema`, `consumers` (repo + lecture file, machine-readable), `maintainer`, `cadence` (dynamic only). +A constructed dataset without its committed builder is a bug. Manifest fields: `source`, `license` (with the `verified` date it was established), `retrieved`, `integrity` (split into `migration` and `upstream`, see Phase 7), `schema` (including `known_nulls`), `consumers` (repo + lecture file, machine-readable), `maintainer`, `builder` / `builder_status`, `cadence` (dynamic only). `manifest-schema.yml` is the authoritative, commented field reference — keep it and this list in step. #### Two inherited-file states that look like violations but are tracked, not hidden diff --git a/manifest-schema.yml b/manifest-schema.yml index d402e13..1d5c357 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -108,7 +108,10 @@ schema: - {name: economy, dtype: string, description: ISO3 country code} - {name: Country, dtype: string, description: Country name} - {name: YR1960, dtype: float, description: Annual GDP growth (%), one column per year} - row_count_floor: 5 # PR validation fails below this + row_count_floor: 5 # PR validation fails below this. For a + # frozen historical extract set it to the + # exact row count by design (P1 lingcod = + # 82); a loose floor is for files that grow. date_range: {start: 1960, end: null} # null end = tracks the moving source # Nulls that are correct and must not fail validation. P1 found a real one: From a6b02af6e41f5d86f7ad35f203155e397393d818 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Fri, 17 Jul 2026 09:08:27 +1000 Subject: [PATCH 4/6] Reframe licensing as cache-not-host, and metadata as capture-when-available MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two tone corrections to the manifest convention, from review discussion — the fields are unchanged, only the surrounding policy language: - Reframe the repo's identity as a stability CACHE for lecture data (snapshots upstream sources so a build doesn't depend on a live API or third-party host), not a content-distribution host. The licence question follows: "is this source OK to cache and serve publicly, with attribution?", not "may we republish it as our own?". Attribution (the manifest `source` block) is the primary obligation; redistribution becomes a cheap binary gate (permitted | restricted) that is a fast yes for public data sources and blocks only genuinely restricted ones (e.g. FRED third-party series). Softens "promoted public host" / "blocker". - State the capture-when-available principle explicitly: rich provenance is always welcome and worth recording when the source provides it, but a missing field is recorded as a reasoned gap, never a bar to landing a useful dataset. Effort scales with the source — a clean one gets a short manifest. Does not touch the integrity method/upstream field structure — that simplification is still under discussion. See QuantEcon/meta#338 and QuantEcon/QuantEcon.manual#108 Co-Authored-By: Claude Opus 4.8 (1M context) --- AGENTS.md | 10 +++++++--- manifest-schema.yml | 11 +++++++---- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d987d9f..61c52b9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ Guidance for coding agents (and humans) making changes in this repository. Read ## What this repo is -The canonical home for **data consumed by the QuantEcon lecture series** (renamed from `QuantEcon/data` on 2026-07-16, per [meta#336](https://github.com/QuantEcon/meta/issues/336)). It is **mid-transition**: the current tree is a legacy consumer-keyed layout (`lecture-python-intro/…`) that will become a flat published tree served at `https://data.quantecon.org/lectures/`. The full convention lives in the draft manual page ([QuantEcon.manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108)). +The canonical home for **data consumed by the QuantEcon lecture series** (renamed from `QuantEcon/data` on 2026-07-16, per [meta#336](https://github.com/QuantEcon/meta/issues/336)). Its purpose is **stability**: it snapshots upstream sources — with attribution to each source carried in the manifest — so a lecture build never depends on a live API or a third-party host staying up. It is a **cache, not a content-distribution host**. It is **mid-transition**: the current tree is a legacy consumer-keyed layout (`lecture-python-intro/…`) that will become a flat published tree served at `https://data.quantecon.org/lectures/`. The full convention lives in the draft manual page ([QuantEcon.manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108)). ## Rules @@ -25,6 +25,8 @@ Classify as exactly one of: A constructed dataset without its committed builder is a bug. Manifest fields: `source`, `license` (with the `verified` date it was established), `retrieved`, `integrity` (split into `migration` and `upstream`, see Phase 7), `schema` (including `known_nulls`), `consumers` (repo + lecture file, machine-readable), `maintainer`, `builder` / `builder_status`, `cadence` (dynamic only). `manifest-schema.yml` is the authoritative, commented field reference — keep it and this list in step. +**Capture what the source gives you; never let a missing field block a useful dataset.** Rich provenance — DOI, upstream version, exact retrieval date, licence id — is always welcome and worth recording whenever it is available, because it makes the data auditable years later at almost no ongoing cost. But effort scales with what the source actually provides: where a field is genuinely unavailable, record it as an explicit, reasoned gap (see the inherited-file states below) rather than fabricating it or refusing the file. A clean, well-documented source should produce a short manifest; only genuinely messy provenance earns a long one. + #### Two inherited-file states that look like violations but are tracked, not hidden The Feb 2025 migration left files that cannot fully satisfy the rules above. The manifest records each gap **explicitly** — visible in the generated catalog — rather than burying it by misclassification. Both are provisional decisions from the P1 pilot ([meta#338](https://github.com/QuantEcon/meta/issues/338)), to be folded into [manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108). @@ -66,9 +68,11 @@ Builders follow four stages — **fetch → pre-process → validate → write** Live API calls are for *teaching data access*, not for getting data. Don't propose "the lecture should just call the API" as a fix — the fix is a snapshot here plus an automated refresh. -### Licensing +### Licensing and attribution + +Because this repo is a **stability cache, not a content-distribution host** (see "What this repo is"), the licence question is *"is this source OK to cache and serve publicly, with attribution?"* — not *"may we republish this as our own?"*. Attribution to the upstream source is carried in every manifest (`source`: name, url, series, citation), and that is the primary obligation. -Before adding any file, confirm the upstream license permits redistribution and record it in the manifest. This repo is on track to be a promoted public host; unlicensed rehosting is a blocker, not a nice-to-have. +For the public data sources most snapshots come from (World Bank, FRED, Eurostat, …) the answer is a **known yes, recorded once per source** — permissive terms plus attribution. Record what the source states and move on; don't re-litigate it per snapshot. Treat the manifest's `redistribution` field as a **cheap binary gate** (`permitted` / `restricted`): a fast `permitted` for public statistics agencies, `restricted` blocking only the genuinely restricted source before it goes public — e.g. FRED re-serves third-party series that may not be redistributed, and anything under non-commercial or no-redistribution terms must not be cached here, since attribution alone does not cure those. Capture licence detail richly when the source provides it; where it is genuinely unavailable, record the gap rather than blocking the file. ## Cross-repo hygiene diff --git a/manifest-schema.yml b/manifest-schema.yml index 1d5c357..7237cd2 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -50,9 +50,12 @@ source: license: name: CC BY-4.0 url: https://datacatalog.worldbank.org/public-licenses#cc-by - # Redistribution must be confirmed before a file is served from this repo — - # unlicensed rehosting is a blocker, not a nice-to-have (AGENTS.md). - redistribution: permitted + # This repo is a stability cache, not a content host: the question is whether + # the source may be cached and served publicly WITH attribution, not whether + # we may republish it as our own. A cheap binary gate (AGENTS.md, "Licensing + # and attribution") — a fast yes for public data sources, blocking only the + # genuinely restricted ones (e.g. FRED third-party series). + redistribution: permitted # permitted | restricted verified: 2026-07-16 # when, and against what, this was # established — a licence claim with no # date is not evidence. P1 found the @@ -68,7 +71,7 @@ retrieved: 2024-04-10 # ISO date the bytes were obtained, or maintainer: QuantEcon # who fixes this when it breaks # --------------------------------------------------------------------------- -# Integrity (PLAN Phase 7) — a file is not promoted with unknown status +# Integrity (PLAN Phase 7) — record the status honestly, don't fake it # --------------------------------------------------------------------------- # Two independent questions, and P1 (lingcod) proved they get different answers: # that file is a byte-perfect copy of what the lecture consumed, while its From 452c12f39f1f6260a95107f5f2e645770c79f893 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Fri, 17 Jul 2026 09:44:19 +1000 Subject: [PATCH 5/6] Simplify integrity to one axis: upstream + sha256, note not method MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applies the simplification agreed in review discussion. The P1 finding stands — migration and upstream are independent questions with different answers — but only one of them belongs in the manifest: - Drop the `migration` sub-block. Migration safety is a ONE-TIME check that matters at the repoint and never again, so it lives in the repoint PR as a Phase 8 gate rather than as a permanent manifest field. The manifest keeps `sha256`, which makes the check reproducible at any later date. Lingcod's verified migration (byte-compare against intro's copy, 2026-07-16) is preserved as a comment pointing at data-lectures#12 / lecture-python-intro#792. - Replace free-text `method` with an optional one-line `note`, populated only when status is not a clean verified — the valuable prose was always the REASON (why lingcod is unverifiable), not a re-typed procedure. - Move the generic how-to-verify-by-class recipe to AGENTS.md, stated once, instead of being repeated in every manifest. - Add `against`: what a verified check compared to (URL, DOI, or repo path), so "verified" is auditable rather than a bare assertion. Net effect: a clean verbatim file now carries a three-line integrity block with zero prose; only genuinely messy provenance earns a note. See QuantEcon/meta#338 and QuantEcon/QuantEcon.manual#108 Co-Authored-By: Claude Opus 4.8 (1M context) --- AGENTS.md | 6 ++-- lectures/lingcod_msy_recovery.csv.yml | 32 +++++++------------- manifest-schema.yml | 42 ++++++++++++--------------- 3 files changed, 33 insertions(+), 47 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 61c52b9..6b00d45 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,7 +23,9 @@ Classify as exactly one of: | **constructed** | built from upstream sources by our processing | all of the above **plus the builder script, committed here** | | **dynamic snapshot** | constructed, tracking a moving source (FRED, World Bank) | all of the above **plus a refresh cadence** | -A constructed dataset without its committed builder is a bug. Manifest fields: `source`, `license` (with the `verified` date it was established), `retrieved`, `integrity` (split into `migration` and `upstream`, see Phase 7), `schema` (including `known_nulls`), `consumers` (repo + lecture file, machine-readable), `maintainer`, `builder` / `builder_status`, `cadence` (dynamic only). `manifest-schema.yml` is the authoritative, commented field reference — keep it and this list in step. +A constructed dataset without its committed builder is a bug. Manifest fields: `source`, `license` (with the `verified` date it was established), `retrieved`, `integrity` (`sha256` plus the `upstream` verification status, see Phase 7), `schema` (including `known_nulls`), `consumers` (repo + lecture file, machine-readable), `maintainer`, `builder` / `builder_status`, `cadence` (dynamic only). `manifest-schema.yml` is the authoritative, commented field reference — keep it and this list in step. + +**Verifying `integrity.upstream`, by class** (once here, not repeated per manifest): re-fetch-and-compare for **verbatim**; re-run the builder and compare the overlap window for **constructed / dynamic**; spot-check against the cited publication for author-assembled. When verification is impossible, say so plainly — `status: unverifiable` with a one-line `note` is a known status the catalog can show; silence is not. **Migration safety** (does the file byte-match what the consuming lecture used before a repoint?) is deliberately *not* a manifest field: it is a one-time gate recorded in the repoint PR, and the manifest's `sha256` keeps it reproducible afterwards. **Capture what the source gives you; never let a missing field block a useful dataset.** Rich provenance — DOI, upstream version, exact retrieval date, licence id — is always welcome and worth recording whenever it is available, because it makes the data auditable years later at almost no ongoing cost. But effort scales with what the source actually provides: where a field is genuinely unavailable, record it as an explicit, reasoned gap (see the inherited-file states below) rather than fabricating it or refusing the file. A clean, well-documented source should produce a short manifest; only genuinely messy provenance earns a long one. @@ -31,7 +33,7 @@ A constructed dataset without its committed builder is a bug. Manifest fields: ` The Feb 2025 migration left files that cannot fully satisfy the rules above. The manifest records each gap **explicitly** — visible in the generated catalog — rather than burying it by misclassification. Both are provisional decisions from the P1 pilot ([meta#338](https://github.com/QuantEcon/meta/issues/338)), to be folded into [manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108). -- **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that says why (`status: unverifiable` with a reason). +- **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that says why (`status: unverifiable` with a `note`). - **`builder_status: unrecovered` — constructed without a recoverable builder.** A constructed dataset ships its builder, and one that omits it *silently* is the bug. Several inherited files are constructed with no recoverable extraction steps (PLAN Phase 9 tracks them). Keep `class: constructed` — reclassifying to `verbatim` to dodge the rule is misclassification — set `builder: null` and `builder_status: unrecovered`, and the gap stays visible for Phase 9 to recover. `unrecovered` is for **inherited files only**; never introduce a *new* constructed file without its builder. ### Corrections vs vintages diff --git a/lectures/lingcod_msy_recovery.csv.yml b/lectures/lingcod_msy_recovery.csv.yml index c21199c..8993913 100644 --- a/lectures/lingcod_msy_recovery.csv.yml +++ b/lectures/lingcod_msy_recovery.csv.yml @@ -43,34 +43,22 @@ maintainer: QuantEcon # --------------------------------------------------------------------------- # Integrity (PLAN Phase 7) # --------------------------------------------------------------------------- -# Two independent questions with different answers here, which is why they are -# recorded separately rather than as one status field: -# migration -- does this match what the lecture consumed before the repoint? -# upstream -- is this actually what RAM Legacy says? -# A file can be a perfect copy of a lecture's data and still have unprovable -# provenance. Collapsing both into one "verified" would overstate this file. +# Migration check (not a manifest field — it is a one-time repoint gate): +# verified 2026-07-16 by sha256 byte-compare against lecture-python-intro's +# copy at lectures/datasets/lingcod_msy_recovery.csv, so the repoint cannot +# change lecture output. Recorded in data-lectures#12 / lecture-python-intro#792 +# and reproducible from the hash below at any later date. integrity: sha256: fc2f873f97a9b7c6d335ca88203c321b4451cf4b94feb16ca40e99551c49c092 - - migration: - status: verified - date: 2026-07-16 - method: > - sha256 byte-compare against lecture-python-intro's copy at - lectures/datasets/lingcod_msy_recovery.csv, immediately before the - repoint. Identical, so the repoint cannot change lecture output. - upstream: status: unverifiable date: 2026-07-16 - method: > - Cannot be re-derived. The extract does not record which RAM Legacy stock - id it came from, which database version, or what transformation produced - the two ratio columns, and no builder was committed with it. Verifying - would mean re-identifying the stock in a current release and confirming - the series match -- possible in principle, not done. Treat the values as - trusted-by-provenance-of-the-author, not as independently checked. + against: null + note: > + No RAM Legacy stock id, database version, or transformation recorded, and + no builder committed — cannot be re-derived. Trust rests on the + provenance of the author, not an independent check. # --------------------------------------------------------------------------- # Shape diff --git a/manifest-schema.yml b/manifest-schema.yml index 7237cd2..06cd8e7 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -66,40 +66,36 @@ retrieved: 2024-04-10 # ISO date the bytes were obtained, or # null for inherited-undated files — # never invent one from git history. # A null here MUST be paired with an - # integrity.upstream reason (AGENTS.md, + # integrity.upstream note (AGENTS.md, # "Two inherited-file states"). maintainer: QuantEcon # who fixes this when it breaks # --------------------------------------------------------------------------- # Integrity (PLAN Phase 7) — record the status honestly, don't fake it # --------------------------------------------------------------------------- -# Two independent questions, and P1 (lingcod) proved they get different answers: -# that file is a byte-perfect copy of what the lecture consumed, while its -# upstream provenance cannot be re-derived at all. One combined "verified" field -# would have had to either overstate or understate it, so they are split. +# One permanent axis: is this file actually what the upstream source says? +# (How to check, by class, lives once in AGENTS.md, not per manifest.) +# +# Migration safety — does the file match what the consuming lecture used before +# a repoint? — is deliberately NOT a manifest field. It is a one-time check that +# lives in the repoint PR (a Phase 8 gate), and the sha256 below is what makes +# it reproducible at any later date. P1 (lingcod) proved the two questions get +# different answers: byte-perfect copy of what the lecture consumed, upstream +# provenance unrecoverable. integrity: sha256: null # of the committed file - - # Does this match what the consuming lecture used before the repoint? - # Guards the migration: a mismatch means repointing silently changes output. - # Hard prerequisite for Phase 8. - migration: - status: unverified # verified | unverified | failing | not-applicable - date: null - method: null # e.g. sha256 compare against 's copy - - # Is this actually what the upstream source says? - # Guards the provenance claim above. upstream: status: unverified # verified | spot-checked | unverifiable | unverified | failing - date: null - method: > - re-fetch-and-compare for verbatim; re-run the builder and compare the - overlap window for constructed/dynamic; spot-check against the cited - publication for author-assembled. Say so plainly when verification is - impossible — `unverifiable` with a reason is an honest answer and is - what the catalog should show. Silence is not. + date: null # ISO date of the last check + against: null # what a verified check compared to (URL, + # DOI, or repo path) — makes "verified" + # auditable rather than a bare assertion + note: null # ONE line, only when status is not a + # clean verified — e.g. why a file is + # unverifiable. An honest `unverifiable` + # with a note is a known status the + # catalog can show; silence is not. # --------------------------------------------------------------------------- # Shape — what a consumer can rely on; also the basis for PR validation tests From d8110bb09b78278f34bd1da8b3ab46877ab80290 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Fri, 17 Jul 2026 09:52:41 +1000 Subject: [PATCH 6/6] Bring PLAN.md in step with the P1 schema decisions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AGENTS.md's own rule: when a decision settles, update PLAN.md in the same PR that acts on it. This PR revised the manifest schema in ways four PLAN passages still described in their pre-pilot form: - "Where we're going": the manifest field list predated the pilot — point to manifest-schema.yml as authoritative and name the fields P1 added (integrity, builder_status, known_nulls, license.verified). - Phase 5: link both the invariant-tests bullet and the business_cycle.py retrofit to #14, which carries the builder template and the schema decisions the tests will force (column patterns, known_nulls semantics, dtype vocabulary). - Phase 6: the licence check is per SOURCE, not per file — a cheap binary cache-and-serve-with-attribution gate, per the cache-not-host reframe. RAM Legacy (P1) joins World Bank as the second source already established. - Phase 7: the byte-compare is recorded in the repoint PR as a one-time gate (not a manifest field); verification results land in integrity.upstream as sha256/status/against/date, replacing the retired "method" field; and unverifiable-with-note is the honest known status. - Open-decisions table: licensing strawman restated in cache-not-host terms. See QuantEcon/meta#338 and QuantEcon/QuantEcon.manual#108 Co-Authored-By: Claude Opus 4.8 (1M context) --- PLAN.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/PLAN.md b/PLAN.md index 6a35fa1..819089a 100644 --- a/PLAN.md +++ b/PLAN.md @@ -34,7 +34,7 @@ This repository is being shaped into the **single canonical repository for data ## Where we're going (per the draft convention) - **Flat published tree** served at `https://data.quantecon.org/lectures/` via GitHub Pages (custom domain), CORS-open for pyodide/JupyterLite -- Every dataset classified **verbatim / constructed / dynamic snapshot**, each with a manifest (`source`, `license`, `retrieved`, `schema`, `consumers`, `maintainer`, `cadence`) +- Every dataset classified **verbatim / constructed / dynamic snapshot**, each with a manifest — authoritative field reference in `manifest-schema.yml`, as revised by the P1 pilot (`integrity`, `builder_status`, `known_nulls` and `license.verified` joined the original sketch of `source` / `license` / `retrieved` / `schema` / `consumers` / `maintainer` / `cadence`) - Constructed and dynamic datasets ship their **builder**; dynamic datasets get **scheduled refresh-as-PR** plus a weekly **sources-alive canary** - Per-path LFS for large binaries only; storage choice invisible to consumers because URLs decouple from hosting @@ -77,8 +77,8 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020. ### Phase 5 — Automation (`.github/`) -- [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data -- [ ] Retrofit `scripts/business_cycle.py` to the four-stage builder contract — it has fetch/transform/write today but **no validate stage** +- [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data. The schema decisions these tests force — column patterns for wide files, `known_nulls` exact-vs-ceiling, a canonical dtype vocabulary — are researched in [#14](https://github.com/QuantEcon/data-lectures/issues/14) +- [ ] Retrofit `scripts/business_cycle.py` to the four-stage builder contract — it has fetch/transform/write today but **no validate stage**. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14) - [ ] Scheduled refresh workflow for dynamic datasets — cron per cadence class, runs the builder (fetch → pre-process → validate → write), lands the result as a PR whose diff summary (rows added, date-range delta, overlap-window changes) is the review surface; low-risk series may auto-merge on green (first consumer: the UNRATE pilot, meta#338 P4) - [ ] Weekly sources-alive canary: fetch + validate, no commit, opens an issue on failure — relocates API fragility from 7 lecture repos' CI into one scheduled job here - [ ] Consumer fan-out: a merged refresh or in-place correction dispatches rebuilds of the repos in the dataset's machine-readable `consumers` list @@ -88,7 +88,7 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020. - [ ] Manifest per dataset for the **9** files now in `lectures/`: source, license, retrieval date, schema, consumers, provenance class. Schema sketched in `manifest-schema.yml` (Phase 2); backfill is per-file work gated on the license check below - [ ] Classify: the 8 static intro files are author-assembled or verbatim; `business_cycle_data.csv` is the one dynamic snapshot and needs its cadence declared -- [ ] License check per file before this repo is promoted as the canonical public home. One is already answered: `business_cycle_metadata.md` records the World Bank series as **CC BY-4.0** — that dump is the model for what a manifest should capture, and the other 8 need the equivalent established by hand +- [ ] Licence check **per source**, not per file: the question is *"may this source be cached and served publicly, with attribution?"* — a cheap binary gate (`redistribution: permitted | restricted`, see AGENTS.md "Licensing and attribution"), a fast yes for public data sources. Two sources already answered: World Bank is **CC BY-4.0** (`business_cycle_metadata.md`, the model for what a manifest should capture) and RAM Legacy is **CC BY 4.0** (established against its Zenodo DOI record, P1). The remaining sources need the equivalent established by hand - [x] Keep-or-drop decision for the files with no consumer anywhere — **dropped 2026-07-16** in the Phase 2 restructure, rather than promoting them into the published namespace: - `GDP_per_capita_world_bank.csv` and `Metadata_Country_API_NY.GDP.PCAP.CD_DS2_en_csv_v2_4770417.csv` — an org-wide code search returns **zero** references to either, they are freely re-downloadable from the World Bank, and their licence was never established. Rehosting a stale snapshot nobody reads is the opposite of this repo's purpose - `fig_3.ods` — confirmed to carry no provenance the published `.xlsx` lacks: both parse to a single `Sheet1` of identical shape (34×6) and `DataFrame.equals` returns true, so it is a pure format twin @@ -98,10 +98,10 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020. Verify that what this repo holds is actually the data it claims to be — against upstream sources, and against the copies lectures consume today — before any lecture is repointed here. -- [ ] **Byte-compare against the in-use copies**: each file migrated in Feb 2025 must be identical to the copy `lecture-python-intro` currently consumes (git blob hash compare). If a copy diverged, a repoint silently changes lecture output — this check is a hard prerequisite for Phase 8 -- [ ] **Verbatim files**: re-fetch from the upstream source and compare (e.g. `mpd2020.xlsx` against the published Maddison Project 2020 release); record checksum, verification method, and date in the manifest +- [ ] **Byte-compare against the in-use copies**: each file migrated in Feb 2025 must be identical to the copy `lecture-python-intro` currently consumes (git blob hash compare). If a copy diverged, a repoint silently changes lecture output — this check is a hard prerequisite for Phase 8. Recorded **in the repoint PR** as a one-time gate, reproducible later from the manifest's `sha256` — not a manifest field (P1 decision) +- [ ] **Verbatim files**: re-fetch from the upstream source and compare (e.g. `mpd2020.xlsx` against the published Maddison Project 2020 release); record `sha256`, `status`, what it was compared `against`, and the date in the manifest's `integrity.upstream` - [ ] **Constructed / dynamic files**: re-run the committed builder (`scripts/business_cycle.py` → `business_cycle_data.csv`) and confirm values agree in the overlap window with the committed snapshot -- [ ] **Author-assembled files** (the French Revolution spreadsheets, `caron.npy`, `nom_balances.npy` — prose-only provenance): spot-check key values against the cited publication and record what was checked; full verification may be impossible, and the manifest should say so +- [ ] **Author-assembled files** (the French Revolution spreadsheets, `caron.npy`, `nom_balances.npy` — prose-only provenance): spot-check key values against the cited publication and record what was checked; full verification may be impossible, and the manifest should say so (`status: unverifiable` with a one-line `note` — the honest known status, per P1) - [ ] **Unverifiable or failing files**: flag in the manifest and open an issue — do not promote a file to the canonical URL namespace with a known-bad or unknown integrity status ### Phase 8 — Pilot deployment (meta#338) @@ -129,6 +129,6 @@ The first end-to-end deployment: one dataset per hosting pattern, each the harde | Repo name | **settled 2026-07-16**: renamed `data-lectures` (Phase 1) | | URL form | `data.quantecon.org/lectures/...`; interim `github.com/QuantEcon/data-lectures/raw/main/...` | | Layout | flat | -| Licensing review | per-source check before rehosting, recorded in the manifest | +| Licensing review | per-source cache-and-serve-with-attribution gate (`redistribution: permitted \| restricted`), recorded in the manifest — this repo is a stability cache, not a content host | When one of these settles, update this PLAN and `AGENTS.md` in the same PR that acts on it.