P1 pilot: add lingcod_msy_recovery.csv with its manifest

AGENTS.md

@@ -4,7 +4,7 @@ Guidance for coding agents (and humans) making changes in this repository. Read
 
 ## What this repo is
 
-The canonical home for **data consumed by the QuantEcon lecture series** (renamed from `QuantEcon/data` on 2026-07-16, per [meta#336](https://github.com/QuantEcon/meta/issues/336)). It is **mid-transition**: the current tree is a legacy consumer-keyed layout (`lecture-python-intro/…`) that will become a flat published tree served at `https://data.quantecon.org/lectures/`. The full convention lives in the draft manual page ([QuantEcon.manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108)).
+The canonical home for **data consumed by the QuantEcon lecture series** (renamed from `QuantEcon/data` on 2026-07-16, per [meta#336](https://github.com/QuantEcon/meta/issues/336)). Its purpose is **stability**: it snapshots upstream sources — with attribution to each source carried in the manifest — so a lecture build never depends on a live API or a third-party host staying up. It is a **cache, not a content-distribution host**. It is **mid-transition**: the current tree is a legacy consumer-keyed layout (`lecture-python-intro/…`) that will become a flat published tree served at `https://data.quantecon.org/lectures/`. The full convention lives in the draft manual page ([QuantEcon.manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108)).
 
 ## Rules
 
@@ -23,7 +23,18 @@ Classify as exactly one of:
 | **constructed** | built from upstream sources by our processing | all of the above **plus the builder script, committed here** |
 | **dynamic snapshot** | constructed, tracking a moving source (FRED, World Bank) | all of the above **plus a refresh cadence** |
 
-A constructed dataset without its committed builder is a bug. Manifest fields: `source`, `license`, `retrieved`, `schema`, `consumers` (repo + lecture file, machine-readable), `maintainer`, `cadence` (dynamic only).
+A constructed dataset without its committed builder is a bug. Manifest fields: `source`, `license` (with the `verified` date it was established), `retrieved`, `integrity` (`sha256` plus the `upstream` verification status, see Phase 7), `schema` (including `known_nulls`), `consumers` (repo + lecture file, machine-readable), `maintainer`, `builder` / `builder_status`, `cadence` (dynamic only). `manifest-schema.yml` is the authoritative, commented field reference — keep it and this list in step.
+
+**Verifying `integrity.upstream`, by class** (once here, not repeated per manifest): re-fetch-and-compare for **verbatim**; re-run the builder and compare the overlap window for **constructed / dynamic**; spot-check against the cited publication for author-assembled. When verification is impossible, say so plainly — `status: unverifiable` with a one-line `note` is a known status the catalog can show; silence is not. **Migration safety** (does the file byte-match what the consuming lecture used before a repoint?) is deliberately *not* a manifest field: it is a one-time gate recorded in the repoint PR, and the manifest's `sha256` keeps it reproducible afterwards.
+
+**Capture what the source gives you; never let a missing field block a useful dataset.** Rich provenance — DOI, upstream version, exact retrieval date, licence id — is always welcome and worth recording whenever it is available, because it makes the data auditable years later at almost no ongoing cost. But effort scales with what the source actually provides: where a field is genuinely unavailable, record it as an explicit, reasoned gap (see the inherited-file states below) rather than fabricating it or refusing the file. A clean, well-documented source should produce a short manifest; only genuinely messy provenance earns a long one.
+
+#### Two inherited-file states that look like violations but are tracked, not hidden
+
+The Feb 2025 migration left files that cannot fully satisfy the rules above. The manifest records each gap **explicitly** — visible in the generated catalog — rather than burying it by misclassification. Both are provisional decisions from the P1 pilot ([meta#338](https://github.com/QuantEcon/meta/issues/338)), to be folded into [manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108).
+
+- **`retrieved: null` — inherited-undated bytes.** `retrieved` is required, but may be `null` when the bytes were inherited (e.g. from a lecture repo) with **no recorded upstream-retrieval date**. Do **not** reconstruct one from git history — that records when QuantEcon acquired the file, not when it was retrieved from the source, and the false precision is worse than an honest null. A null `retrieved` must be paired with an `integrity.upstream` entry that says why (`status: unverifiable` with a `note`).
+- **`builder_status: unrecovered` — constructed without a recoverable builder.** A constructed dataset ships its builder, and one that omits it *silently* is the bug. Several inherited files are constructed with no recoverable extraction steps (PLAN Phase 9 tracks them). Keep `class: constructed` — reclassifying to `verbatim` to dodge the rule is misclassification — set `builder: null` and `builder_status: unrecovered`, and the gap stays visible for Phase 9 to recover. `unrecovered` is for **inherited files only**; never introduce a *new* constructed file without its builder.
 
 ### Corrections vs vintages
 
@@ -59,9 +70,11 @@ Builders follow four stages — **fetch → pre-process → validate → write**
 
 Live API calls are for *teaching data access*, not for getting data. Don't propose "the lecture should just call the API" as a fix — the fix is a snapshot here plus an automated refresh.
 
-### Licensing
+### Licensing and attribution
+
+Because this repo is a **stability cache, not a content-distribution host** (see "What this repo is"), the licence question is *"is this source OK to cache and serve publicly, with attribution?"* — not *"may we republish this as our own?"*. Attribution to the upstream source is carried in every manifest (`source`: name, url, series, citation), and that is the primary obligation.
 
-Before adding any file, confirm the upstream license permits redistribution and record it in the manifest. This repo is on track to be a promoted public host; unlicensed rehosting is a blocker, not a nice-to-have.
+For the public data sources most snapshots come from (World Bank, FRED, Eurostat, …) the answer is a **known yes, recorded once per source** — permissive terms plus attribution. Record what the source states and move on; don't re-litigate it per snapshot. Treat the manifest's `redistribution` field as a **cheap binary gate** (`permitted` / `restricted`): a fast `permitted` for public statistics agencies, `restricted` blocking only the genuinely restricted source before it goes public — e.g. FRED re-serves third-party series that may not be redistributed, and anything under non-commercial or no-redistribution terms must not be cached here, since attribution alone does not cure those. Capture licence detail richly when the source provides it; where it is genuinely unavailable, record the gap rather than blocking the file.
 
 ## Cross-repo hygiene
 

PLAN.md

@@ -34,7 +34,7 @@ This repository is being shaped into the **single canonical repository for data
 ## Where we're going (per the draft convention)
 
 - **Flat published tree** served at `https://data.quantecon.org/lectures/<filename>` via GitHub Pages (custom domain), CORS-open for pyodide/JupyterLite
-- Every dataset classified **verbatim / constructed / dynamic snapshot**, each with a manifest (`source`, `license`, `retrieved`, `schema`, `consumers`, `maintainer`, `cadence`)
+- Every dataset classified **verbatim / constructed / dynamic snapshot**, each with a manifest — authoritative field reference in `manifest-schema.yml`, as revised by the P1 pilot (`integrity`, `builder_status`, `known_nulls` and `license.verified` joined the original sketch of `source` / `license` / `retrieved` / `schema` / `consumers` / `maintainer` / `cadence`)
 - Constructed and dynamic datasets ship their **builder**; dynamic datasets get **scheduled refresh-as-PR** plus a weekly **sources-alive canary**
 - Per-path LFS for large binaries only; storage choice invisible to consumers because URLs decouple from hosting
 
@@ -77,8 +77,8 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020.
 
 ### Phase 5 — Automation (`.github/`)
 
-- [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data
-- [ ] Retrofit `scripts/business_cycle.py` to the four-stage builder contract — it has fetch/transform/write today but **no validate stage**
+- [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data. The schema decisions these tests force — column patterns for wide files, `known_nulls` exact-vs-ceiling, a canonical dtype vocabulary — are researched in [#14](https://github.com/QuantEcon/data-lectures/issues/14)
+- [ ] Retrofit `scripts/business_cycle.py` to the four-stage builder contract — it has fetch/transform/write today but **no validate stage**. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14)
 - [ ] Scheduled refresh workflow for dynamic datasets — cron per cadence class, runs the builder (fetch → pre-process → validate → write), lands the result as a PR whose diff summary (rows added, date-range delta, overlap-window changes) is the review surface; low-risk series may auto-merge on green (first consumer: the UNRATE pilot, meta#338 P4)
 - [ ] Weekly sources-alive canary: fetch + validate, no commit, opens an issue on failure — relocates API fragility from 7 lecture repos' CI into one scheduled job here
 - [ ] Consumer fan-out: a merged refresh or in-place correction dispatches rebuilds of the repos in the dataset's machine-readable `consumers` list
@@ -88,7 +88,7 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020.
 
 - [ ] Manifest per dataset for the **9** files now in `lectures/`: source, license, retrieval date, schema, consumers, provenance class. Schema sketched in `manifest-schema.yml` (Phase 2); backfill is per-file work gated on the license check below
 - [ ] Classify: the 8 static intro files are author-assembled or verbatim; `business_cycle_data.csv` is the one dynamic snapshot and needs its cadence declared
-- [ ] License check per file before this repo is promoted as the canonical public home. One is already answered: `business_cycle_metadata.md` records the World Bank series as **CC BY-4.0** — that dump is the model for what a manifest should capture, and the other 8 need the equivalent established by hand
+- [ ] Licence check **per source**, not per file: the question is *"may this source be cached and served publicly, with attribution?"* — a cheap binary gate (`redistribution: permitted | restricted`, see AGENTS.md "Licensing and attribution"), a fast yes for public data sources. Two sources already answered: World Bank is **CC BY-4.0** (`business_cycle_metadata.md`, the model for what a manifest should capture) and RAM Legacy is **CC BY 4.0** (established against its Zenodo DOI record, P1). The remaining sources need the equivalent established by hand
 - [x] Keep-or-drop decision for the files with no consumer anywhere — **dropped 2026-07-16** in the Phase 2 restructure, rather than promoting them into the published namespace:
   - `GDP_per_capita_world_bank.csv` and `Metadata_Country_API_NY.GDP.PCAP.CD_DS2_en_csv_v2_4770417.csv` — an org-wide code search returns **zero** references to either, they are freely re-downloadable from the World Bank, and their licence was never established. Rehosting a stale snapshot nobody reads is the opposite of this repo's purpose
   - `fig_3.ods` — confirmed to carry no provenance the published `.xlsx` lacks: both parse to a single `Sheet1` of identical shape (34×6) and `DataFrame.equals` returns true, so it is a pure format twin
@@ -98,10 +98,10 @@ The sidecar naming uses the **full filename** (`mpd2020.xlsx.yml`, not `mpd2020.
 
 Verify that what this repo holds is actually the data it claims to be — against upstream sources, and against the copies lectures consume today — before any lecture is repointed here.
 
-- [ ] **Byte-compare against the in-use copies**: each file migrated in Feb 2025 must be identical to the copy `lecture-python-intro` currently consumes (git blob hash compare). If a copy diverged, a repoint silently changes lecture output — this check is a hard prerequisite for Phase 8
-- [ ] **Verbatim files**: re-fetch from the upstream source and compare (e.g. `mpd2020.xlsx` against the published Maddison Project 2020 release); record checksum, verification method, and date in the manifest
+- [ ] **Byte-compare against the in-use copies**: each file migrated in Feb 2025 must be identical to the copy `lecture-python-intro` currently consumes (git blob hash compare). If a copy diverged, a repoint silently changes lecture output — this check is a hard prerequisite for Phase 8. Recorded **in the repoint PR** as a one-time gate, reproducible later from the manifest's `sha256` — not a manifest field (P1 decision)
+- [ ] **Verbatim files**: re-fetch from the upstream source and compare (e.g. `mpd2020.xlsx` against the published Maddison Project 2020 release); record `sha256`, `status`, what it was compared `against`, and the date in the manifest's `integrity.upstream`
 - [ ] **Constructed / dynamic files**: re-run the committed builder (`scripts/business_cycle.py` → `business_cycle_data.csv`) and confirm values agree in the overlap window with the committed snapshot
-- [ ] **Author-assembled files** (the French Revolution spreadsheets, `caron.npy`, `nom_balances.npy` — prose-only provenance): spot-check key values against the cited publication and record what was checked; full verification may be impossible, and the manifest should say so
+- [ ] **Author-assembled files** (the French Revolution spreadsheets, `caron.npy`, `nom_balances.npy` — prose-only provenance): spot-check key values against the cited publication and record what was checked; full verification may be impossible, and the manifest should say so (`status: unverifiable` with a one-line `note` — the honest known status, per P1)
 - [ ] **Unverifiable or failing files**: flag in the manifest and open an issue — do not promote a file to the canonical URL namespace with a known-bad or unknown integrity status
 
 ### Phase 8 — Pilot deployment (meta#338)
@@ -129,6 +129,6 @@ The first end-to-end deployment: one dataset per hosting pattern, each the harde
 | Repo name | **settled 2026-07-16**: renamed `data-lectures` (Phase 1) |
 | URL form | `data.quantecon.org/lectures/...`; interim `github.com/QuantEcon/data-lectures/raw/main/...` |
 | Layout | flat |
-| Licensing review | per-source check before rehosting, recorded in the manifest |
+| Licensing review | per-source cache-and-serve-with-attribution gate (`redistribution: permitted \| restricted`), recorded in the manifest — this repo is a stability cache, not a content host |
 
 When one of these settles, update this PLAN and `AGENTS.md` in the same PR that acts on it.

lectures/lingcod_msy_recovery.csv

@@ -0,0 +1,83 @@
+year,B_over_Bmsy,F_over_Fmsy
+1928,3.859289617,0.12815534
+1929,3.842213115,0.177669903
+1930,3.825136612,0.198058252
+1931,3.790983607,0.19223301
+1932,3.756830601,0.142718447
+1933,3.756830601,0.224271845
+1934,3.705601093,0.14368932
+1935,3.705601093,0.170873786
+1936,3.671448087,0.133009709
+1937,3.671448087,0.173786408
+1938,3.654371585,0.146601942
+1939,3.637295082,0.127184466
+1940,3.637295082,0.145631068
+1941,3.637295082,0.117475728
+1942,3.637295082,0.067961165
+1943,3.654371585,0.131067961
+1944,3.654371585,0.132038835
+1945,3.654371585,0.130097087
+1946,3.637295082,0.216504854
+1947,3.603142077,0.430097087
+1948,3.50068306,0.462135922
+1949,3.398224044,0.413592233
+1950,3.31284153,0.45631068
+1951,3.227459016,0.453398058
+1952,3.142076503,0.350485437
+1953,3.090846995,0.246601942
+1954,3.073770492,0.290291262
+1955,3.056693989,0.308737864
+1956,3.039617486,0.355339806
+1957,3.005464481,0.526213592
+1958,2.920081967,0.539805825
+1959,2.851775956,0.466990291
+1960,2.800546448,0.417475728
+1961,2.74931694,0.449514563
+1962,2.698087432,0.386407767
+1963,2.663934426,0.391262136
+1964,2.629781421,0.340776699
+1965,2.612704918,0.413592233
+1966,2.578551913,0.518446602
+1967,2.527322404,0.57961165
+1968,2.476092896,0.612621359
+1969,2.407786885,0.575728155
+1970,2.339480874,0.850485437
+1971,2.237021858,1.077669903
+1972,2.117486339,1.563106796
+1973,1.929644809,1.766990291
+1974,1.741803279,1.951456311
+1975,1.572745902,1.951456311
+1976,1.449795082,2.077669903
+1977,1.321721311,1.485436893
+1978,1.270491803,1.941747573
+1979,1.166325137,2.718446602
+1980,1.00068306,3.378640777
+1981,0.840163934,3.378640777
+1982,0.712090164,3.504854369
+1983,0.594262295,2.67961165
+1984,0.544740437,2.805825243
+1985,0.488387978,3.951456311
+1986,0.403005464,3.854368932
+1987,0.37568306,4.019417476
+1988,0.37397541,4.310679612
+1989,0.358606557,4.834951456
+1990,0.3125,4.553398058
+1991,0.285177596,4.32038835
+1992,0.268101093,4.174757282
+1993,0.256147541,3.485436893
+1994,0.266393443,2.27184466
+1995,0.315915301,2.019417476
+1996,0.365437158,2.038834951
+1997,0.396174863,1.815533981
+1998,0.425204918,1.300970874
+1999,0.479849727,1.398058252
+2000,0.539617486,0.63592233
+2001,0.650614754,0.49223301
+2002,0.800887978,1.029126214
+2003,0.988729508,1.291262136
+2004,1.243169399,0.195145631
+2005,1.656420765,0.261165049
+2006,2.083333333,0.249514563
+2007,2.510245902,0.144660194
+2008,2.885928962,0.093203883
+2009,3.193306011,