diff --git a/.github/workflows/consumed-file-check.yml b/.github/workflows/consumed-file-check.yml index 26994aa..6d6a713 100644 --- a/.github/workflows/consumed-file-check.yml +++ b/.github/workflows/consumed-file-check.yml @@ -37,3 +37,18 @@ jobs: python-version: "3.12" - run: pip install pyyaml - run: python .github/scripts/check_consumed_files.py + + # CATALOG.md is generated from the same manifests this job validates, and + # `scripts/build_catalog.py` has claimed since it was written that CI + # asserts its freshness. Nothing did: no workflow ran the generator at + # all, so #69 could populate six manifests' `consumers` and leave the + # committed catalog contradicting them — invisible until an unrelated PR + # inherited the surprise diff (#71). This is that gate, put where the + # manifests are already parsed and pyyaml is already installed. + - name: CATALOG.md is current + run: | + python scripts/build_catalog.py + git diff --exit-code -- CATALOG.md || { + echo "::error file=CATALOG.md::CATALOG.md is stale. A manifest changed without regenerating the catalog — run 'python scripts/build_catalog.py' and commit the result." + exit 1 + } diff --git a/CATALOG.md b/CATALOG.md index 04617f1..53046dc 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -6,25 +6,25 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest, which may be before its consuming lectures are repointed — an empty **Used by** column means the file is here and documented but no lecture reads it from this repo yet. Files still to migrate are tracked in [PLAN.md](PLAN.md). -**24 datasets** · 18 read by lectures today, 6 awaiting repoint · 110.0 MB total · 19 permitted / 5 restricted redistribution +**24 datasets** · 24 read by lectures today · 110.0 MB total · 19 permitted / 5 restricted redistribution | Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| [**SCF_plus_mini.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini.csv)
SCF+ mini — net wealth, income and survey weights, 1950-2016 | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 31.3 MB | — | -| [**SCF_plus_mini_no_weights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini_no_weights.csv)
SCF+ mini, weight-expanded — net wealth and income, 1950-2016 | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 72.4 MB | — | +| [**SCF_plus_mini.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini.csv)
SCF+ mini — net wealth, income and survey weights, 1950-2016 | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 31.3 MB | [lecture-python-intro · inequality.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inequality.md)
[lecture-wasm · inequality.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inequality.md)
[lecture-intro.zh-cn · inequality.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/inequality.md)
[test-actions-lecture-intro · inequality.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/inequality.md) | +| [**SCF_plus_mini_no_weights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini_no_weights.csv)
SCF+ mini, weight-expanded — net wealth and income, 1950-2016 | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 72.4 MB | [lecture-python-intro · mle.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/mle.md)
[lecture-wasm · mle.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/mle.md)
[lecture-intro.zh-cn · mle.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/mle.md)
[test-actions-lecture-intro · mle.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/mle.md) | | [**ames_house_prices.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/ames_house_prices.csv)
Ames, Iowa — residential house sales, 2006-2010 | constructed | [Ames Housing data (De Cock 2011), Journal of Statistics Education](http://jse.amstat.org/v19n3/decock.pdf) | | ✅ permitted | ✅ verified | ✅ committed | 75.2 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | | [**assignat.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/assignat.xlsx)
French Revolution — assignat issues, budgets and seigniorage (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 204.6 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md) | | [**caron.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/caron.npy)
French Revolution — monthly specie value of the assignat, 1791-1796 | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.1 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md) | | [**chapter_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/chapter_3.xlsx)
The Ends of Four Big Inflations — appendix tables, transcribed | constructed | [Sargent, "Rational Expectations and Inflation", chapter 3 appendix tables](https://press.princeton.edu/books/paperback/9780691158709/rational-expectations-and-inflation) | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 71.6 KB | [lecture-python-intro · inflation_history.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inflation_history.md)
[lecture-wasm · inflation_history.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inflation_history.md) | -| [**cities_brazil.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/cities_brazil.csv)
World Population Review — Brazilian city populations, 2023 | verbatim | [World Population Review — cities in Brazil](https://worldpopulationreview.com/countries/cities/brazil) | | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 17.5 KB | — | -| [**cities_us.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/cities_us.csv)
World Population Review — US city populations, 2023 | verbatim | [World Population Review — US cities](https://worldpopulationreview.com/us-cities) | | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 47.0 KB | — | +| [**cities_brazil.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/cities_brazil.csv)
World Population Review — Brazilian city populations, 2023 | verbatim | [World Population Review — cities in Brazil](https://worldpopulationreview.com/countries/cities/brazil) | | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 17.5 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | +| [**cities_us.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/cities_us.csv)
World Population Review — US city populations, 2023 | verbatim | [World Population Review — US cities](https://worldpopulationreview.com/us-cities) | | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 47.0 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | | [**countries.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/countries.csv)
WorldData.info country reference table | verbatim | [WorldData.info — country data downloads](https://www.worlddata.info/downloads/) | Proprietary — © WorldData.info, all rights reserved | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 48.4 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | | [**dette.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/dette.xlsx)
French Revolution — public debt, military spending and revenues (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 617.2 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md) | | [**employ.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/employ.csv)
Eurostat employment in Europe — by age and sex, 2007–2016 | constructed | [Eurostat — Employment database](https://ec.europa.eu/eurostat/data/database) | Eurostat reuse (Commission Decision 2011/833/EU) | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.6 MB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | | [**epl_match_goals.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/epl_match_goals.csv)
English Premier League — full-time scores, 2015-16 to 2024-25 | constructed | [openfootball / football.json](https://github.com/openfootball/football.json) | Public domain | ✅ permitted | ✅ verified | ✅ committed | 203.2 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | | [**fig_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fig_3.xlsx)
French Revolution — figure 3 series (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 9.2 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)
[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md) | -| [**forbes-billionaires.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-billionaires.csv)
Forbes Billionaires — individual net worth | constructed | [Forbes Billionaires](https://www.forbes.com/billionaires/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 775.7 KB | — | -| [**forbes-global2000.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-global2000.csv)
Forbes Global 2000 — firm size measures | constructed | [Forbes Global 2000](https://www.forbes.com/lists/global2000/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 115.6 KB | — | +| [**forbes-billionaires.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-billionaires.csv)
Forbes Billionaires — individual net worth | constructed | [Forbes Billionaires](https://www.forbes.com/billionaires/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 775.7 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | +| [**forbes-global2000.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-global2000.csv)
Forbes Global 2000 — firm size measures | constructed | [Forbes Global 2000](https://www.forbes.com/lists/global2000/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 115.6 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | | [**japan_deaths_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_deaths_by_age.csv)
Japan — deaths by single year of age, 2023 | constructed | [United Nations, Department of Economic and Social Affairs, Population Division — World Population Prospects 2024](https://population.un.org/wpp/downloads) | CC BY 3.0 IGO | ✅ permitted | ✅ verified | ✅ committed | 1.7 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | | [**japan_earthquakes.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_earthquakes.csv)
Japan region — earthquakes of magnitude 5 and above, 2000-2024 | constructed | [Advanced National Seismic System (ANSS) Comprehensive Earthquake Catalog (ComCat), US Geological Survey](https://earthquake.usgs.gov/earthquakes/search/) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 172.8 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | | [**japan_population_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_population_by_age.csv)
Japan — population by single year of age, 2024 | constructed | [Population Estimates, Statistics Bureau of Japan, Ministry of Internal Affairs and Communications](https://www.stat.go.jp/english/data/jinsui/index.html) | Japan Statistics Bureau terms of use | ✅ permitted | ✅ verified | ✅ committed | 1.3 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md) | diff --git a/scripts/build_catalog.py b/scripts/build_catalog.py index b7e6469..ed52a11 100644 --- a/scripts/build_catalog.py +++ b/scripts/build_catalog.py @@ -8,8 +8,11 @@ python scripts/build_catalog.py -CI asserts the catalog is current with `git diff --exit-code CATALOG.md` after -regenerating, so a stale catalog fails the build (PLAN Phase 5). +`consumed-file-check.yml` asserts the catalog is current — it regenerates and +runs `git diff --exit-code -- CATALOG.md`, so a stale catalog fails the build +(PLAN Phase 5). That gate was claimed here long before it existed, which is how +#69 populated six manifests' `consumers` without regenerating and left the +committed catalog contradicting them (#71). A docstring is not a gate. Scope: migrated-only. A dataset appears here once it has a manifest; files in lectures/ without a manifest are not yet migrated and are tracked in PLAN