Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/workflows/consumed-file-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,18 @@ jobs:
python-version: "3.12"
- run: pip install pyyaml
- run: python .github/scripts/check_consumed_files.py

# CATALOG.md is generated from the same manifests this job validates, and
# `scripts/build_catalog.py` has claimed since it was written that CI
# asserts its freshness. Nothing did: no workflow ran the generator at
# all, so #69 could populate six manifests' `consumers` and leave the
# committed catalog contradicting them — invisible until an unrelated PR
# inherited the surprise diff (#71). This is that gate, put where the
# manifests are already parsed and pyyaml is already installed.
- name: CATALOG.md is current
run: |
python scripts/build_catalog.py
git diff --exit-code -- CATALOG.md || {
echo "::error file=CATALOG.md::CATALOG.md is stale. A manifest changed without regenerating the catalog — run 'python scripts/build_catalog.py' and commit the result."
exit 1
}
14 changes: 7 additions & 7 deletions CATALOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,25 +6,25 @@

The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest, which may be before its consuming lectures are repointed — an empty **Used by** column means the file is here and documented but no lecture reads it from this repo yet. Files still to migrate are tracked in [PLAN.md](PLAN.md).

**24 datasets** · 18 read by lectures today, 6 awaiting repoint · 110.0 MB total · 19 permitted / 5 restricted redistribution
**24 datasets** · 24 read by lectures today · 110.0 MB total · 19 permitted / 5 restricted redistribution

| Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| [**SCF_plus_mini.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini.csv)<br><sub>SCF+ mini — net wealth, income and survey weights, 1950-2016</sub> | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 31.3 MB | |
| [**SCF_plus_mini_no_weights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini_no_weights.csv)<br><sub>SCF+ mini, weight-expanded — net wealth and income, 1950-2016</sub> | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 72.4 MB | |
| [**SCF_plus_mini.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini.csv)<br><sub>SCF+ mini — net wealth, income and survey weights, 1950-2016</sub> | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 31.3 MB | [lecture-python-intro · inequality.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inequality.md)<br>[lecture-wasm · inequality.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inequality.md)<br>[lecture-intro.zh-cn · inequality.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/inequality.md)<br>[test-actions-lecture-intro · inequality.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/inequality.md) |
| [**SCF_plus_mini_no_weights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/SCF_plus_mini_no_weights.csv)<br><sub>SCF+ mini, weight-expanded — net wealth and income, 1950-2016</sub> | constructed | [SCF+ (Kuhn, Schularick and Steins) — an extension of the Survey of Consumer Finances](https://www.journals.uchicago.edu/doi/10.1086/708815) | | ✅ permitted | ⚠️ unverifiable | committed-frozen | 72.4 MB | [lecture-python-intro · mle.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/mle.md)<br>[lecture-wasm · mle.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/mle.md)<br>[lecture-intro.zh-cn · mle.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/mle.md)<br>[test-actions-lecture-intro · mle.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/mle.md) |
| [**ames_house_prices.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/ames_house_prices.csv)<br><sub>Ames, Iowa — residential house sales, 2006-2010</sub> | constructed | [Ames Housing data (De Cock 2011), Journal of Statistics Education](http://jse.amstat.org/v19n3/decock.pdf) | | ✅ permitted | ✅ verified | ✅ committed | 75.2 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)<br>[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) |
| [**assignat.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/assignat.xlsx)<br><sub>French Revolution — assignat issues, budgets and seigniorage (Sargent-Velde)</sub> | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 204.6 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)<br>[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md) |
| [**caron.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/caron.npy)<br><sub>French Revolution — monthly specie value of the assignat, 1791-1796</sub> | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.1 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)<br>[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md) |
| [**chapter_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/chapter_3.xlsx)<br><sub>The Ends of Four Big Inflations — appendix tables, transcribed</sub> | constructed | [Sargent, "Rational Expectations and Inflation", chapter 3 appendix tables](https://press.princeton.edu/books/paperback/9780691158709/rational-expectations-and-inflation) | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 71.6 KB | [lecture-python-intro · inflation_history.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/inflation_history.md)<br>[lecture-wasm · inflation_history.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/inflation_history.md) |
| [**cities_brazil.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/cities_brazil.csv)<br><sub>World Population Review — Brazilian city populations, 2023</sub> | verbatim | [World Population Review — cities in Brazil](https://worldpopulationreview.com/countries/cities/brazil) | | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 17.5 KB | |
| [**cities_us.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/cities_us.csv)<br><sub>World Population Review — US city populations, 2023</sub> | verbatim | [World Population Review — US cities](https://worldpopulationreview.com/us-cities) | | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 47.0 KB | |
| [**cities_brazil.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/cities_brazil.csv)<br><sub>World Population Review — Brazilian city populations, 2023</sub> | verbatim | [World Population Review — cities in Brazil](https://worldpopulationreview.com/countries/cities/brazil) | | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 17.5 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)<br>[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)<br>[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)<br>[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) |
| [**cities_us.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/cities_us.csv)<br><sub>World Population Review — US city populations, 2023</sub> | verbatim | [World Population Review — US cities](https://worldpopulationreview.com/us-cities) | | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 47.0 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)<br>[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)<br>[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)<br>[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) |
| [**countries.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/countries.csv)<br><sub>WorldData.info country reference table</sub> | verbatim | [WorldData.info — country data downloads](https://www.worlddata.info/downloads/) | Proprietary — © WorldData.info, all rights reserved | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 48.4 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)<br>[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) |
| [**dette.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/dette.xlsx)<br><sub>French Revolution — public debt, military spending and revenues (Sargent-Velde)</sub> | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 617.2 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)<br>[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md) |
| [**employ.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/employ.csv)<br><sub>Eurostat employment in Europe — by age and sex, 2007–2016</sub> | constructed | [Eurostat — Employment database](https://ec.europa.eu/eurostat/data/database) | Eurostat reuse (Commission Decision 2011/833/EU) | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.6 MB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)<br>[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) |
| [**epl_match_goals.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/epl_match_goals.csv)<br><sub>English Premier League — full-time scores, 2015-16 to 2024-25</sub> | constructed | [openfootball / football.json](https://github.com/openfootball/football.json) | Public domain | ✅ permitted | ✅ verified | ✅ committed | 203.2 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) |
| [**fig_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fig_3.xlsx)<br><sub>French Revolution — figure 3 series (Sargent-Velde)</sub> | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 9.2 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)<br>[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md) |
| [**forbes-billionaires.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-billionaires.csv)<br><sub>Forbes Billionaires — individual net worth</sub> | constructed | [Forbes Billionaires](https://www.forbes.com/billionaires/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 775.7 KB | |
| [**forbes-global2000.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-global2000.csv)<br><sub>Forbes Global 2000 — firm size measures</sub> | constructed | [Forbes Global 2000](https://www.forbes.com/lists/global2000/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 115.6 KB | |
| [**forbes-billionaires.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-billionaires.csv)<br><sub>Forbes Billionaires — individual net worth</sub> | constructed | [Forbes Billionaires](https://www.forbes.com/billionaires/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 775.7 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)<br>[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)<br>[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)<br>[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) |
| [**forbes-global2000.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-global2000.csv)<br><sub>Forbes Global 2000 — firm size measures</sub> | constructed | [Forbes Global 2000](https://www.forbes.com/lists/global2000/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 115.6 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)<br>[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)<br>[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)<br>[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) |
| [**japan_deaths_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_deaths_by_age.csv)<br><sub>Japan — deaths by single year of age, 2023</sub> | constructed | [United Nations, Department of Economic and Social Affairs, Population Division — World Population Prospects 2024](https://population.un.org/wpp/downloads) | CC BY 3.0 IGO | ✅ permitted | ✅ verified | ✅ committed | 1.7 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)<br>[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) |
| [**japan_earthquakes.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_earthquakes.csv)<br><sub>Japan region — earthquakes of magnitude 5 and above, 2000-2024</sub> | constructed | [Advanced National Seismic System (ANSS) Comprehensive Earthquake Catalog (ComCat), US Geological Survey](https://earthquake.usgs.gov/earthquakes/search/) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 172.8 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) |
| [**japan_population_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_population_by_age.csv)<br><sub>Japan — population by single year of age, 2024</sub> | constructed | [Population Estimates, Statistics Bureau of Japan, Ministry of Internal Affairs and Communications](https://www.stat.go.jp/english/data/jinsui/index.html) | Japan Statistics Bureau terms of use | ✅ permitted | ✅ verified | ✅ committed | 1.3 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md) |
Expand Down
7 changes: 5 additions & 2 deletions scripts/build_catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,11 @@

python scripts/build_catalog.py

CI asserts the catalog is current with `git diff --exit-code CATALOG.md` after
regenerating, so a stale catalog fails the build (PLAN Phase 5).
`consumed-file-check.yml` asserts the catalog is current — it regenerates and
runs `git diff --exit-code -- CATALOG.md`, so a stale catalog fails the build
(PLAN Phase 5). That gate was claimed here long before it existed, which is how
#69 populated six manifests' `consumers` without regenerating and left the
committed catalog contradicting them (#71). A docstring is not a gate.

Scope: migrated-only. A dataset appears here once it has a manifest; files in
lectures/ without a manifest are not yet migrated and are tracked in PLAN
Expand Down
Loading