diff --git a/.dev/qeld/README.md b/.dev/qeld/README.md new file mode 100644 index 0000000..2bde288 --- /dev/null +++ b/.dev/qeld/README.md @@ -0,0 +1,20 @@ +# `.dev/qeld/` — working artifacts behind `PLAN-QELD-PACKAGE.md` + +Evidence and worklists produced while designing the `qeld` consumer-side data package. These are **inputs to +the plan, not the plan**. Where anything here disagrees with `PLAN-QELD-PACKAGE.md`, the plan wins. + +Nothing here is published — the Pages job assembles `_site` from `site/`, `lectures/` and `audit.json` only. + +| file | what it is | +|---|---| +| `callsites.yml` | The 115-site call-site worklist: every static data read in six lecture repos, with its idiom, the verdict against the "no more complex than fetching a url" bar, the action to take, and whether the win is structural or cosmetic. Drives the rollout ordering (`PLAN-QELD-PACKAGE.md` §7). | +| `migration-catalog.md` | The worked evidence behind the call-site rule — verbatim before/after per idiom, and the head-to-head comparison of the three candidate syntaxes. This is the spec for what a correct conversion produces. | + +## Both are snapshots dated 2026-08-10 + +They were produced from a source sweep on that date, and **both this repo and the lecture repos moved during +the session that produced them**. The most consequential drift: the A3 `french_rev` set was repointed (#49), +so the two `.npy` files gained consumers and their conversion to CSV is no longer a free replacement. + +**Regenerate the worklist before running the sweep.** Treat the catalog's examples as illustrative of the +*pattern*, and re-read the current source before converting any specific site. diff --git a/.dev/qeld/callsites.yml b/.dev/qeld/callsites.yml new file mode 100644 index 0000000..044e133 --- /dev/null +++ b/.dev/qeld/callsites.yml @@ -0,0 +1,881 @@ +# qeld call-site worklist — audit snapshot +# +# Generated 2026-08-10 from a source sweep of six lecture repos (40 lectures, +# 115 read sites). Input to PLAN-QELD-PACKAGE.md; see that file for the +# call-site rule (section 4.1), the carve-outs, and the rollout ordering. +# +# THIS IS A SNAPSHOT, NOT A LIVE VIEW. lecture-python-intro and this repo both +# moved during the session that produced it (the A3 french_rev set was +# repointed in #49). REGENERATE BEFORE THE SWEEP. +# +# `repo` is always owner/repo and is never empty. Several lectures exist in +# more than one repo -- heavy_tails.md, long_run_growth.md and french_rev.md +# are each present in lecture-python-intro, lecture-wasm and lecture-intro.zh-cn +# -- so (lecture, filename) is NOT a unique key, and entries differing only by +# repo are distinct call sites rather than duplicates. Repoint rule 2 requires +# the intro/wasm pair be converted together. +# +# action: +# convert apply the substitute-in-place rule +# carve-out do not convert - see the note +# blocked needs a format decision first +# exclude not a data-lectures candidate at all +# downstream a read of a variable assigned elsewhere; changes by construction +# +# win: indicative only. `structural` means qeld deletes logic, not just shortens +# a literal. The rollout takes structural sites first. + +callsites: + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/fitting_distributions.md:257-259" + filename: "us_adult_heights.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/fitting_distributions.md:467-469" + filename: "epl_match_goals.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/fitting_distributions.md:667-669" + filename: "japan_earthquakes.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/fitting_distributions.md:753-755" + filename: "japan_deaths_by_age.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/fitting_distributions.md:77-79" + filename: "ames_house_prices.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/observed_distributions.md:137-139" + filename: "us_adult_heights.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/observed_distributions.md:152-154" + filename: "ames_house_prices.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/observed_distributions.md:187-189" + filename: "japan_deaths_by_age.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-advanced.myst" + lecture: "lectures/risk_aversion_or_mistaken_beliefs.md:1613" + filename: "fred_data.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/fitting_distributions.md:277" + filename: "us_adult_heights.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/fitting_distributions.md:522" + filename: "epl_match_goals.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/fitting_distributions.md:72" + filename: "ames_house_prices.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/fitting_distributions.md:757" + filename: "japan_earthquakes.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/fitting_distributions.md:855" + filename: "japan_deaths_by_age.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/observed_distributions.md:133" + filename: "us_adult_heights.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/observed_distributions.md:151" + filename: "ames_house_prices.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/observed_distributions.md:192" + filename: "japan_deaths_by_age.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/prob_dist.md:130" + filename: "japan_population_by_age.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/prob_dist.md:57" + filename: "us_adult_heights.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-programming" + lecture: "lectures/polars.md:160-163" + filename: "test_pwt.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-programming" + lecture: "lectures/polars.md:346-349" + filename: "test_pwt.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-programming" + lecture: "lectures/polars.md:424-428" + filename: "test_pwt.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/phillips_drifts_volatilities.md:102" + filename: "NEWQDATA.csv" + idiom: "A-split-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/french_rev.md:84-89" + filename: "fig_3.xlsx, dette.xlsx, assignat.xlsx" + idiom: "B-url-variable" + verdict: "same" + action: "convert" + win: "structural" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/inequality.md:256-257" + filename: "SCF_plus_mini.csv" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/inequality.md:613-614" + filename: "usa-gini-nwealth-tincome-lincome.csv" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/inflation_history.md:102-107" + filename: "longprices.xls" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/inflation_history.md:374-375" + filename: "chapter_3.xlsx" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/long_run_growth.md:106-108" + filename: "mpd2020.xlsx" + idiom: "B-url-variable" + verdict: "more_complex" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/mle.md:105-106" + filename: "SCF_plus_mini_no_weights.csv" + idiom: "B-url-variable" + verdict: "same" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/simple_linear_regression.md:426-427" + filename: "life-expectancy-vs-gdp-per-capita.csv" + idiom: "B-url-variable" + verdict: "more_complex" + action: "carve-out" + win: "cosmetic" + note: "paired with a download link - see PLAN section 8.2" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/french_rev.md:75" + filename: "fig_3.xlsx" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "structural" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/inequality.md:249" + filename: "SCF_plus_mini.csv" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/inequality.md:621" + filename: "usa-gini-nwealth-tincome-lincome.csv" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/inflation_history.md:358" + filename: "chapter_3.xlsx" + idiom: "B-url-variable" + verdict: "same" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/inflation_history.md:86" + filename: "longprices.xls" + idiom: "B-url-variable" + verdict: "same" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/long_run_growth.md:89" + filename: "mpd2020.xlsx" + idiom: "B-url-variable" + verdict: "same" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/mle.md:93" + filename: "SCF_plus_mini_no_weights.csv" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/msy_fishery.md:596" + filename: "lingcod_msy_recovery.csv" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/simple_linear_regression.md:416" + filename: "life-expectancy-vs-gdp-per-capita.csv" + idiom: "B-url-variable" + verdict: "more_complex" + action: "carve-out" + win: "cosmetic" + note: "paired with a download link - see PLAN section 8.2" + - repo: "QuantEcon/lecture-python-programming" + lecture: "lectures/pandas_panel.md:198,202" + filename: "countries.csv" + idiom: "B-url-variable" + verdict: "same" + action: "carve-out" + win: "cosmetic" + note: "the literal URL is the lesson - see PLAN section 8.1" + - repo: "QuantEcon/lecture-python-programming" + lecture: "lectures/pandas_panel.md:507,526" + filename: "employ.csv" + idiom: "B-url-variable" + verdict: "same" + action: "carve-out" + win: "cosmetic" + note: "the literal URL is the lesson - see PLAN section 8.1" + - repo: "QuantEcon/lecture-python-programming" + lecture: "lectures/pandas_panel.md:78,90" + filename: "realwage.csv" + idiom: "B-url-variable" + verdict: "same" + action: "carve-out" + win: "cosmetic" + note: "the literal URL is the lesson - see PLAN section 8.1" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/pandas_panel.md:188" + filename: "countries.csv" + idiom: "B-url-variable" + verdict: "more_complex" + action: "carve-out" + win: "cosmetic" + note: "the literal URL is the lesson - see PLAN section 8.1" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/pandas_panel.md:505" + filename: "employ.csv" + idiom: "B-url-variable" + verdict: "more_complex" + action: "carve-out" + win: "cosmetic" + note: "the literal URL is the lesson - see PLAN section 8.1" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/pandas_panel.md:68" + filename: "realwage.csv" + idiom: "B-url-variable" + verdict: "more_complex" + action: "carve-out" + win: "cosmetic" + note: "the literal URL is the lesson - see PLAN section 8.1" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:80-85" + filename: "fig_3.xlsx, dette.xlsx, assignat.xlsx" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "structural" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/inequality.md:250-251" + filename: "SCF_plus_mini.csv" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/inequality.md:619-620" + filename: "usa-gini-nwealth-tincome-lincome.csv" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/inflation_history.md:350-351" + filename: "chapter_3.xlsx" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/inflation_history.md:75-80" + filename: "longprices.xls" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/long_run_growth.md:96-99" + filename: "mpd2020.xlsx" + idiom: "B-url-variable" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/mle.md:95-96" + filename: "SCF_plus_mini_no_weights.csv" + idiom: "B-url-variable" + verdict: "same" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/short_path.md:274-277" + filename: "graph.txt" + idiom: "B-url-variable" + verdict: "same" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/simple_linear_regression.md:426-427" + filename: "life-expectancy-vs-gdp-per-capita.csv" + idiom: "B-url-variable" + verdict: "same" + action: "carve-out" + win: "cosmetic" + note: "paired with a download link - see PLAN section 8.2" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/heavy_tails.md:811" + filename: "forbes-global2000.csv" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/heavy_tails.md:838-839" + filename: "cities_us.csv, cities_brazil.csv" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/heavy_tails.md:863" + filename: "forbes-billionaires.csv" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/heavy_tails.md:827" + filename: "forbes-global2000.csv" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/heavy_tails.md:854" + filename: "cities_us.csv" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/heavy_tails.md:855" + filename: "cities_brazil.csv" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/heavy_tails.md:879" + filename: "forbes-billionaires.csv" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python-programming" + lecture: "lectures/pandas.md:165" + filename: "test_pwt.csv" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/mle.md:165" + filename: "fp.dta" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/ols.md:326" + filename: "maketable2.dta" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/ols.md:478" + filename: "maketable4.dta" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/ols.md:606" + filename: "maketable4.dta" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/ols.md:679" + filename: "maketable1.dta" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/ols.md:94" + filename: "maketable1.dta" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/heavy_tails.md:827" + filename: "forbes-global2000.csv" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/heavy_tails.md:854" + filename: "cities_us.csv" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/heavy_tails.md:855" + filename: "cities_brazil.csv" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/heavy_tails.md:879" + filename: "forbes-billionaires.csv" + idiom: "C-inline-literal" + verdict: "simpler" + action: "convert" + win: "cosmetic" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/business_cycle.md:62" + filename: "country_code_cn.csv" + idiom: "D-local-path" + verdict: "impossible" + action: "exclude" + win: "structural" + note: "zh-cn translation asset; no data-lectures key" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/long_run_growth.md:141" + filename: "country_code_cn.csv" + idiom: "D-local-path" + verdict: "impossible" + action: "exclude" + win: "structural" + note: "zh-cn translation asset; no data-lectures key" + - repo: "QuantEcon/lecture-python-advanced.myst" + lecture: "lectures/match_transport.md:2234" + filename: "acs_data_summary.csv" + idiom: "D-local-path" + verdict: "simpler" + action: "convert" + win: "structural" + - repo: "QuantEcon/lecture-python-advanced.myst" + lecture: "lectures/subjective_beliefs_business_cycles.md:149" + filename: "bbh_macro_quarterly.csv" + idiom: "D-local-path" + verdict: "same" + action: "convert" + win: "structural" + - repo: "QuantEcon/lecture-python-advanced.myst" + lecture: "lectures/subjective_beliefs_business_cycles.md:152" + filename: "bbh_michigan_monthly.csv" + idiom: "D-local-path" + verdict: "simpler" + action: "convert" + win: "structural" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/french_rev.md:671-674" + filename: "caron.npy, nom_balances.npy" + idiom: "F-npy-bytesio" + verdict: "same" + action: "blocked" + win: "structural" + note: "npy cannot be read from a URL - see PLAN section 8.3" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/french_rev.md:711" + filename: "caron.npy" + idiom: "F-npy-bytesio" + verdict: "same" + action: "blocked" + win: "structural" + note: "npy cannot be read from a URL - see PLAN section 8.3" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/french_rev.md:712" + filename: "nom_balances.npy" + idiom: "F-npy-bytesio" + verdict: "same" + action: "blocked" + win: "structural" + note: "npy cannot be read from a URL - see PLAN section 8.3" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:725-728" + filename: "caron.npy, nom_balances.npy" + idiom: "F-npy-bytesio" + verdict: "same" + action: "blocked" + win: "structural" + note: "npy cannot be read from a URL - see PLAN section 8.3" + - repo: "QuantEcon/lecture-python-programming" + lecture: "lectures/pandas.md:514" + filename: "fredgraph.csv (FRED query endpoint)" + idiom: "G-live-api" + verdict: "impossible" + action: "exclude" + win: "none" + note: "live parameterised API endpoint, not a file" + - repo: "QuantEcon/lecture-python-programming" + lecture: "lectures/pandas.md:532,552" + filename: "fredgraph.csv (FRED query endpoint)" + idiom: "G-live-api" + verdict: "impossible" + action: "exclude" + win: "none" + note: "live parameterised API endpoint, not a file" + - repo: "QuantEcon/lecture-python-programming" + lecture: "lectures/polars.md:557-572" + filename: "fredgraph.csv (FRED query endpoint)" + idiom: "G-live-api" + verdict: "impossible" + action: "exclude" + win: "none" + note: "live parameterised API endpoint, not a file" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/phillips_drifts_volatilities.md:2213" + filename: "fredgraph.csv (FRED query endpoint, not a file)" + idiom: "G-live-api" + verdict: "impossible" + action: "exclude" + win: "none" + note: "live parameterised API endpoint, not a file" + - repo: "QuantEcon/lecture-python-advanced.myst" + lecture: "lectures/five_preferences.md:1848" + filename: "dataBHS.mat" + idiom: "H-loadmat" + verdict: "impossible" + action: "blocked" + win: "none" + note: "scipy.io.loadmat cannot take a URL - see PLAN section 8.4" + - repo: "QuantEcon/lecture-python-advanced.myst" + lecture: "lectures/hansen_jagannathan_1991.md:182" + filename: "hansen_jagannathan_1991_data.json" + idiom: "I-generated" + verdict: "simpler" + action: "exclude" + win: "structural" + note: "file is generated by the lecture itself" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/french_rev.md:110,115,158,231,233,413,447,492,555,557,615,617" + filename: "dette.xlsx / assignat.xlsx / fig_3.xlsx (12 reuses)" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/long_run_growth.md:486" + filename: "mpd2020.xlsx (reuse)" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/long_run_growth.md:589-592" + filename: "mpd2020.xlsx (reuse)" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/prob_dist.md:152-159" + filename: "japan_population_by_age.xlsx (data-lectures key: japan_population_by_age.csv)" + idiom: "Z-downstream-read" + verdict: "simpler" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/prob_dist.md:74-77" + filename: "us_adult_heights.csv" + idiom: "Z-downstream-read" + verdict: "simpler" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-intro.zh-cn" + lecture: "lectures/simple_linear_regression.md:444" + filename: "life-expectancy-vs-gdp-per-capita.csv (reuse)" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/french_rev.md:76" + filename: "dette.xlsx" + idiom: "Z-downstream-read" + verdict: "simpler" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-python-intro" + lecture: "lectures/french_rev.md:77" + filename: "assignat.xlsx" + idiom: "Z-downstream-read" + verdict: "simpler" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/hansen_singleton_1982.md:993" + filename: "hansen_singleton_1982_data.csv" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-python.myst" + lecture: "lectures/hansen_singleton_1983.md:1432" + filename: "hansen_singleton_1983_data.csv" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:106-108" + filename: "dette.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:111-113" + filename: "dette.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:155-156" + filename: "dette.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:233-234" + filename: "dette.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:235-236" + filename: "dette.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:276-277" + filename: "fig_3.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:438-439" + filename: "dette.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:475-476" + filename: "assignat.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:522-523" + filename: "assignat.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:586-587" + filename: "assignat.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:588-589" + filename: "assignat.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:656-657" + filename: "assignat.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/french_rev.md:658-659" + filename: "assignat.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/inflation_history.md:374-375" + filename: "chapter_3.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/long_run_growth.md:488" + filename: "mpd2020.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/long_run_growth.md:593-596" + filename: "mpd2020.xlsx" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" + - repo: "QuantEcon/lecture-wasm" + lecture: "lectures/simple_linear_regression.md:444" + filename: "life-expectancy-vs-gdp-per-capita.csv" + idiom: "Z-downstream-read" + verdict: "same" + action: "downstream" + win: "none" + note: "read of a variable assigned elsewhere" diff --git a/.dev/qeld/migration-catalog.md b/.dev/qeld/migration-catalog.md new file mode 100644 index 0000000..0433208 --- /dev/null +++ b/.dev/qeld/migration-catalog.md @@ -0,0 +1,274 @@ +# qeld migration catalog — how the lectures consume data today, and what each idiom becomes + +**Date:** 2026-08-10 · **Status:** design input, superseded on decisions by `PLAN-QELD-PACKAGE.md`. +**Purpose:** the worked evidence behind the call-site rule — what each conversion actually looks like. +**Method:** every static data read in six lecture repos (40 lectures, 115 read sites) enumerated from source, +clustered by idiom, with verbatim current code. + +> **Snapshot, not a live view.** Both this repo and `lecture-python-intro` moved after this was produced — +> notably the A3 `french_rev` set was repointed in #64's predecessor #49, which changes §3 F's status from a +> free conversion to a breaking one (`PLAN-QELD-PACKAGE.md` §8.3). Re-verify any example before acting on it. + +--- + +## 1. The unit of work is not 115 reads + +115 read *sites*, but **23 of them are downstream uses of a variable assigned earlier** — +`pd.read_excel(data_url, …)` where `data_url` was set once. `french_rev` reads `dette_url` six times; +`long_run_growth` reads `data_url` three times. + +So the real unit is **~78 URL definitions**. One `qeld.url()` call serves N reads. This is the single most +important fact for choosing syntax: a form that preserves the variable converts one line and leaves N reads +untouched; a form that inlines has to touch all N. + +| idiom | definitions | current verdicts | +|---|---:|---| +| **A** — split string literal across lines | 23 | all simpler | +| **B** — `url` variable = single literal | 32 | 15 simpler, 11 same, **all 6 more_complex** | +| **C** — inline literal URL inside the read | 18 | all simpler | +| **D** — relative / local path | 5 | 2 impossible | +| **F** — `requests`+`BytesIO` → `np.load` | 4 | all same | +| **G** — live parameterised API endpoint | 4 | all impossible | +| **H** — `scipy.io.loadmat` | 1 | impossible | +| *(downstream reads of an existing variable)* | *23* | *unchanged by construction* | + +--- + +## 2. The three candidate syntaxes + +- **S1 — inline:** `pd.read_csv(qeld.url('x.csv'))` +- **S2 — two-step always:** `url = qeld.url('x.csv')` then `pd.read_csv(url)` +- **S3 — mirror the existing shape:** replace only the URL expression; inline stays inline, a variable stays + a variable. + +--- + +## 3. The catalog — verbatim, by idiom + +### A. Split string literal (23) — the pattern the design was built for + +Two sub-shapes. **A1, assigned to a variable** (`polars.md:160-163`): + +```python +# CURRENT +url = ('https://raw.githubusercontent.com/QuantEcon/' + 'lecture-python-programming/main/lectures/_static/' + 'lecture_specific/pandas/data/test_pwt.csv') +df = pl.read_csv(url) + +# S2 / S3 (identical here) +url = qeld.url('test_pwt.csv') +df = pl.read_csv(url) +``` + +**A2, split inline inside the read** (`risk_aversion_or_mistaken_beliefs.md:1613`): + +```python +# CURRENT — 5 lines +data = pd.read_csv( + 'https://raw.githubusercontent.com/QuantEcon/lecture-python-advanced.myst/refs/heads/' + 'main/lectures/_static/lecture_specific/risk_aversion_or_mistaken_beliefs/fred_data.csv', + parse_dates=['DATE'], index_col='DATE' +) + +# S1 / S3 — 1 line +data = pd.read_csv(qeld.url('fred_data.csv'), parse_dates=['DATE'], index_col='DATE') +``` + +**Verdict:** the clearest win in the corpus. `polars.md` carries the *same* three-line literal twice +(`:160` and `:346`), which a student has to re-read to confirm it is the same file. + +### B. `url` variable = single literal (32) — where every failure lives + +```python +# CURRENT (pandas_panel.md:68 / :78) +url1 = 'https://github.com/QuantEcon/data-lectures/raw/main/lectures/realwage.csv' +realwage = pd.read_csv(url1) + +# S1 — destroys the variable; every downstream read must change too +realwage = pd.read_csv(qeld.url('realwage.csv')) + +# S2 / S3 — one-line RHS swap, downstream reads untouched +url1 = qeld.url('realwage.csv') +realwage = pd.read_csv(url1) +``` + +**Verdict:** the six `more_complex` gradings in this group were scored against **S1**, which needlessly +destroys a variable the lecture reuses. Under S2/S3 the complaint largely evaporates — the diff is one +line's right-hand side. + +**What survives the fix:** `pandas_panel`'s three reads still lose something real. The prose immediately +above reads *"The dataset can be accessed with the following link:"* followed by a cell containing nothing +but the URL, so a beginner can paste it into a browser. All three files already point at data-lectures in +the exact form qeld emits off-Pyodide, so there is no host-migration gain to offset it. This is a +**pedagogical** objection, not a syntactic one, and no syntax fixes it. *(Note: `pandas_panel` exists in both +`lecture-python.myst` and `lecture-python-programming` — the two sweeps graded the same content differently, +which is why the `more_complex` count is 4–8 rather than exactly 6.)* + +### C. Inline literal URL inside the read (18) — where S2 loses + +```python +# CURRENT (ols.md:94) — a 150-char literal that overflows every page width +df1 = pd.read_stata('https://github.com/QuantEcon/lecture-python.myst/raw/refs/heads/main/lectures/_static/lecture_specific/ols/maketable1.dta') + +# S1 / S3 — 1 line +df1 = pd.read_stata(qeld.url('maketable1.dta')) + +# S2 — 1 line becomes 2, for nothing +url = qeld.url('maketable1.dta') +df1 = pd.read_stata(url) +``` + +**Verdict:** `ols.md` alone carries five of these. The prose above each already names the file +(`maketable2.dta`), so the qeld key matches the words on the page better than the URL does. **S2 is worse +than the status quo here** — this is the group that rules out "always two-step". + +### D. Relative / local path (5) + +```python +# CURRENT (subjective_beliefs_business_cycles.md:149) +data_path = '_static/lecture_specific/subjective_beliefs_business_cycles/' +macro_q = pd.read_csv(data_path + 'bbh_macro_quarterly.csv', index_col='YYYYQ') + +# any syntax +macro_q = pd.read_csv(qeld.url('bbh_macro_quarterly.csv'), index_col='YYYYQ') +``` + +**Verdict:** these relative paths are **portability bugs today** — they break in Colab and in downloaded +notebooks. Two of the five are `country_code_cn.csv`, a zh-cn translation asset with no data-lectures key +and no business having one. + +### F. `requests` + `BytesIO` → `np.load` (4) — resolved by the format decision, not by syntax + +```python +# CURRENT (french_rev.md:725-728) — plus `import requests` and `from io import BytesIO` at :73-74 +caron_response = requests.get(base_url + 'caron.npy') +nom_balances_response = requests.get(base_url + 'nom_balances.npy') +caron = np.load(BytesIO(caron_response.content)) +nom_balances = np.load(BytesIO(nom_balances_response.content)) + +# AFTER the .npy → .csv conversion (review §6.3) +caron = pd.read_csv(qeld.url('caron.csv')).to_numpy() +nom_balances = pd.read_csv(qeld.url('nom_balances.csv')).to_numpy() +``` + +**Verdict:** graded "same" by the sweep only because it did not know about the CSV conversion. Converted, +four lines and two imports go, and the intro copy's `np.load('datasets/caron.npy')` local path goes with +them. This is the second-best diff in the corpus and it is bought by a **data** change, not a package +feature. + +### G. Live parameterised API endpoints (4) — out of scope, and say so loudly + +FRED graph queries with ~700-character query strings encoding series id, date range and chart options: +`phillips_drifts_volatilities.md:2213`, `pandas.md:514`, `pandas.md:532/552`, `polars.md:557-572`. + +`qeld.url()` keys bare filenames and cannot express `?id=CPIAUCSL%2CUNRATE%2CTB3MS`. **The trap:** three of +these end in `.csv` and carry the ugliest literals in the corpus (`polars.md`'s is 16 continuation lines), so +they are the most tempting to convert by mistake. In `pandas.md` the raw URL is the explicit *subject* of a +teaching section on `requests` and error handling — hiding it would be a pedagogical regression on top of an +impossibility. + +### H. `scipy.io.loadmat` (1) + +```python +# CURRENT (five_preferences.md:1848) +data = loadmat('dataBHS.mat') + +# with qeld and no format change — nobody would ship this +data = loadmat(io.BytesIO(urllib.request.urlopen(qeld.url('dataBHS.mat')).read())) +``` + +**Verdict:** the only true impossibility among static files. Resolved by the format convention (convert to +CSV) or by exclusion — never by syntax. + +--- + +## 4. Syntax verdict + +| idiom | n | S1 inline | S2 two-step always | S3 mirror shape | +|---|---:|---|---|---| +| A1 split → variable | ~15 | ✗ kills the variable | ✅ | ✅ | +| A2 split inline | ~8 | ✅ | ✗ adds a line | ✅ | +| B variable = literal | 32 | ✗ kills the variable, touches N downstream reads | ✅ | ✅ | +| C inline literal | 18 | ✅ | ✗ 1 line → 2 | ✅ | +| D local path | 5 | ✅ | ✅ | ✅ | + +**S3 is the only candidate that wins on every idiom**, and it has a property the others lack: + +> Under S3 the diff is *always and only* the URL expression. A reviewer can verify a migration PR by reading +> the changed lines alone — there is no restructuring to audit. + +That matters for a ~78-site sweep, and it satisfies the maintainer's bar by **construction** rather than by +argument: the lecture's shape is unchanged, so it cannot have become harder to read. + +**The rule, for AGENTS.md:** + +> Lecture code reads published data through `qeld.url('')`, substituted **in place of the URL +> expression the lecture already uses** — an inline literal becomes an inline call, an assigned variable +> keeps its assignment. The reader (`pd.read_*`, `pl.read_*`, …) and all its kwargs stay visible and +> unchanged. + +Carve-outs: live-API reads (group G); reads whose literal URL is itself the lesson (`pandas_panel`); and +reads whose file is also named by a prose or `{download}` link, until that has an answer (§5). + +--- + +## 5. Two problems no syntax solves + +### 5.1 Generic filenames collide in a flat namespace + +The published tree is flat, so the key is a bare filename. Several pending files are far too generic to +survive it: + +| file | repo of origin | problem | +|---|---|---| +| `fred_data.csv` | advanced | names a source, not a dataset | +| `fp.dta` | python.myst | two letters | +| `test_pwt.csv` | programming | "test" | +| `data.csv` | programming (`about_py.md`) | referenced but never exists | +| `acs_data_summary.csv` | advanced | plausible but claims a whole survey | + +These need renaming at migration. **And a rename breaks text qeld cannot reach**: `mle.md:160` names +`mle/fp.dta` in prose. So the rename must be paired with a prose edit, in the same PR, found by grep rather +than by the audit. + +### 5.2 Prose and `{download}` links + +28 prose/`{download}` refs; **11 name a data file in two places**. The acute case is +`simple_linear_regression.md` (intro:411/416, zh-cn:421/426), where a `{download}` role and the code read sit +**five lines apart containing byte-identical 158-character strings**. Convert the code and the page shows a +raw URL in prose and an opaque call in the cell, with no way for a student to see they agree — and they can +drift silently. + +Options: convert both and accept two spellings; leave those lectures alone; or give qeld a markdown-time +story (a MyST substitution, or point the prose at `CATALOG.md` instead of the file). Currently unanswered, +and it is the strongest argument for the "convert both or neither" carve-out. + +--- + +## 6. What this implies for the package + +1. **`url()` returning a string is confirmed as the right primitive** — S3 only works because the return + value can sit anywhere a URL literal sits today. +2. **No second function is needed for the static corpus.** Groups F and H are resolved by the *format* + convention; group G is out of scope. `open()`/`fetch()` would exist only to serve files the format rule + says should not exist. +3. **The win is concentrated, not broad.** Of ~78 definitions, roughly 10 see qeld delete real logic; ~50 are + cosmetic shortenings of literals adopted in recent repoint PRs; the rest are neutral. The case for the + package rests on those ~10, the wasm shim, the host-cutover property, and the portability bugs in group D + — **not** on the read-site tally. +4. **Rollout should be ordered by win, not by migration track.** Structural sites first + (`hansen_jagannathan_1991`'s dual-path loader, `french_rev`'s `base_url` block, `ols.md`'s five literals, + `polars.md`'s duplicated literal, group D's portability bugs). Cosmetic-only sites can wait indefinitely. +5. **Two latent bugs qeld incidentally fixes**, worth their own issue regardless: `inequality.md` imports + `pyodide_http` and never calls `patch_all()`; `short_path.md` calls `requests.get` under Pyodide with no + shim at all. + +--- + +## 7. Open + +- The `pandas_panel` pedagogical carve-out — permanent, or revisited if the prose is rewritten? +- The markdown-time story for `{download}` links (§5.2). +- The rename list and its prose pairings (§5.1) — needs a pass before Tracks B and C migrate. +- `dataBHS.mat`: convert at migration, or exclude. diff --git a/PLAN-QELD-PACKAGE.md b/PLAN-QELD-PACKAGE.md new file mode 100644 index 0000000..0c5fbb8 --- /dev/null +++ b/PLAN-QELD-PACKAGE.md @@ -0,0 +1,378 @@ +# PLAN — `qeld`, the consumer-side data package + +**Status:** design settled, nothing implemented · **Last updated:** 2026-08-10 +**Relationship to `PLAN.md`:** that document migrates *bytes* into this repo. This one gives *consumers* a +stable way to read them. They are independent — the migration completes with or without `qeld` — but the +call-site convention here replaces repoint rules 5–6 for any lecture that adopts it. + +--- + +## 1. What `qeld` is + +A tiny package that hands a lecture a **context-aware URL** for a published dataset. The lecture then reads +it with pandas, in the open: + +```python +url = qeld.url('mpd2020.xlsx') # or inline, see §4.1 +data = pd.read_excel(url, sheet_name='Full data') +``` + +**Purpose: transit assistance and simplified fetching.** Not data management — data history is git's job. + +It exists to solve three concrete problems: + +1. **The host is spelled six ways.** The same lecture reads the same file via + `github.com/…/raw/main/…` in `lecture-python-intro` and `raw.githubusercontent.com/…` in `lecture-wasm`, + because the first fails CORS in the browser. That split *is* repoint rules 5–6. `qeld.url()` erases it. +2. **`pyodide_http.patch_all()` is in every wasm lecture.** Importing `qeld` under emscripten installs the + transport shim, and those two lines leave the lectures. +3. **The host cutover.** When `data.quantecon.org` lands (#37, #15), it is one constant in one package + rather than an edit in every lecture. + +**What it is not:** a cache, a fetcher, a loader, a data-version manager, or an integrity client. See §3.1. + +--- + +## 2. Decisions + +D1–D5 were taken 2026-08-10 against the original design report. D2, D3 and D5 were **revised the same day** +in a working session that re-scoped the package from fetch-and-cache to URL-resolver; D1 and D4 stand +unchanged. D6–D10 are new. + +| # | decision | status | +|---|---|---| +| **D1** | Name `qeld`. Free on PyPI (verified 2026-08-10; `quantecon-data` also free, `qeds` is taken and still installable at 0.7.0) | unchanged | +| **D2** | **Call style: `qeld.url()` substituted in place of the URL expression the lecture already uses.** No `fetch()`, no `load()` | **revised** — see §4.1 | +| **D3** | **Pin policy: semver, `qeld>=1,<2` in install cells; `==` in env files if lockfile determinism is wanted.** CalVer retired | **revised** — see §3.4 | +| **D4** | Home: in-repo `packages/qeld/`. Bytes, manifest and package in one commit | unchanged | +| **D5** | v1 scope: `url()` and `info()` only. Non-goals: live APIs (#26), Julia, datascience/networks, mirrors, a MyST provenance directive | **narrowed** | +| **D6** | **Context detection may change transport; never semantics** | new — §3.2 | +| **D7** | **Catalog bundled in the wheel, advisory not authoritative** | new — §3.3 | +| **D8** | **Data format: tier 1 binds at intake, tier 2 is a forward-looking preference** | new — §4.2 | +| **D9** | **Integrity lives in CI, not at the call site** | new — §6 | +| **D10** | **Rollout is ordered by win, not by migration track** | new — §7 | + +**The scoping principle throughout:** ship the minimum; add sophistication as demand requires. + +### 2.1 Why D3 changed — the `qbn` precedent does not transfer + +The original rationale for exact CalVer pins was "the `qbn==1.6` house pattern for data packages". It fails +on inspection, and the reasoning is recorded here so it is not re-proposed: + +- `quantecon_book_networks` ships its data **inside the wheel** (`data.py:1-16`, `importlib.resources`). Its + pin is immortal because nothing is fetched. `qeld` inverts that exactly. +- As a house pattern it does not generalise: of `lecture-python-intro`'s 16 install cells, the **only** exact + pins are the two `qbn` cells; `quantecon` itself is unpinned in five. `lecture-wasm` pins nothing. +- "The pin names the data vintage" is a category error. The serving tree holds one version per filename, so + a version pins the **key → URL contract**, never the bytes. + +**What `<2` protects, and therefore what the major version means:** catalog keys are append-only, and a +key's meaning never changes within a major version. + +--- + +## 3. Design + +### 3.1 API surface — two functions + +| call | returns | notes | +|---|---|---| +| `qeld.url(name)` | `str` | the context-correct URL. Unknown key → **warning with near-key suggestions, still returns a URL** | +| `qeld.info(name)` | provenance record | title, source, licence, citation, `citation_policy`, `redistribution`, `sha256`. Offline, from the bundled catalog | + +`info()` earns its place because Maddison's `citation_policy` is a live obligation a lecture must discharge, +and the manifest already carries it. + +**No `fetch()`, `load()`, `open()`, cache, or path.** The full static corpus is reachable with `url()` alone +once the format convention (§4.2) is applied — see §5.3. + +### 3.2 Context detection (D6) + +**The rule: context may change transport; it may never change semantics.** Same key, same bytes, same result +everywhere. A design where context changes *what you get* produces "works in my notebook, fails in CI". + +Exactly two jobs: + +1. **URL form** — `raw.githubusercontent.com` under Pyodide (the `github.com/…/raw/` form 302s and fails + CORS); the 302-tolerant form elsewhere; `data.quantecon.org` for both after #37. +2. **Browser transport shim** — under emscripten, `import qeld` installs the fetch shim, because the right + URL is still not enough: `pd.read_excel(url)` goes through urllib, which fails in the browser. + +Explicitly **not** detection's job: caching, fallback hosts, graceful degradation on fetch failure. + +### 3.3 Catalog (D7) + +`url()` does not need a catalog to *resolve* — keys are filenames and the tree is flat, so the URL is +`base + name`. The catalog exists to validate a key and to answer `info()`. + +- **Bundled in the wheel, advisory not authoritative.** Unknown key → warning + near-key suggestions, and the + URL is still returned; the subsequent 404 confirms it. +- **Why bundled:** typo diagnosis is the common failure and `qeld` cannot catch it after the fact — pandas + does the fetch, so a bad key surfaces as a pandas 404 that never mentions `qeld`. Keys carry extensions and + the repo serves both `longprices.xls` and `mpd2020.xlsx`, so `.xls`/`.xlsx` confusion is live. +- **Why advisory:** fail-open means a dataset that landed after your wheel still works. **No new dataset ever + requires a package release.** +- **One compiler, three outputs.** `scripts/build_catalog.py` already emits `CATALOG.md`; it gains + `catalog.json` for the wheel, sharing a freshness gate. This is the CI that PLAN Phase 2 promised. +- **Rejected, on the record:** a catalog-free package (`url()` = `base + name`, `info()` fetching the served + sidecar — the sidecars *are* served, `text/yaml`, `acao: *`). Attractive because the package would only + change when its code changes, but it gives up the one diagnostic that matters day to day. + +### 3.4 Versioning and pinning (D3) + +- **Semver**, frozen at 1.0 after the first pilots pass. +- **Install cells: `qeld>=1,<2`.** Install cells resolve fresh every session (Colab, Binder, wasm/piplite), + so a corrected release is picked up automatically. +- **Env files: `==` if wanted.** These are resolved once and cached, so a maintainer is present to act on a + break. +- **A byte correction behaves as `AGENTS.md` already intends:** same filename, same URL, every consumer gets + the fix, nothing breaks. + +### 3.5 Zero runtime dependencies + +Pure `py3-none-any` wheel, stdlib only, so one artifact serves CPython, Colab and Pyodide via +micropip/piplite. `info()` reads the bundled catalog with `json` — no YAML parser at runtime. + +--- + +## 4. Conventions this creates + +### 4.1 The call-site rule (D2) — substitute in place + +> Lecture code reads published data through `qeld.url('')`, substituted **in place of the URL +> expression the lecture already uses** — an inline literal becomes an inline call, an assigned variable +> keeps its assignment. The reader (`pd.read_*`, `pl.read_*`, …) and all its kwargs stay visible and +> unchanged. + +Chosen by testing three candidates against every idiom in the corpus (§5). Inline-always destroys reused +variables across 47 sites; two-step-always turns one line into two across 18 inline literals. Substitute-in- +place is the only form that wins everywhere, and it has the property that matters for a ~78-site sweep: + +**The diff is always and only the URL expression**, so a reviewer verifies a migration PR by reading the +changed lines alone. It satisfies the maintainer's bar — *"no more complex than it currently is just fetching +a url"* — by construction, since the lecture's shape is unchanged. + +The install cell is **accepted cost**, not added complexity. This answers #58's stated objection by decision +rather than by design. + +**Carve-outs — do not convert:** + +| carve-out | why | sites | +|---|---|---| +| Live parameterised API endpoints | `url()` keys bare filenames and cannot express a query string; in `pandas.md` the raw URL is the *subject* of a teaching section | 4 | +| Reads whose literal URL **is the lesson** | `pandas_panel` prose reads *"The dataset can be accessed with the following link:"* then shows it. All three files already resolve to data-lectures in the form `qeld` emits, so there is no gain to offset it | 3 | +| Reads whose file is also named by a prose/`{download}` link | `url()` cannot appear in markdown; converting the code alone leaves two spellings that drift | 3, until §8.2 | + +### 4.2 Data format (D8) + +> **Tier 1 (binding, at intake):** a published dataset ships in a format its consumer can read **directly +> from a URL**. +> **Tier 2 (preference, forward-looking):** where the format is *not itself part of what is being taught*, +> prefer text. + +Tier 1 binds — stricter than the house pattern elsewhere ("licensing does not gate migration"), deliberately: +a missing licence field is a gap you record, whereas a non-URL-readable format produces a lecture a reader +cannot run. Every file failing tier 1 today is read from a **bare local path** — the Colab breakage the +programme exists to retire. Enforceable from the catalog alone: assert every catalogued extension is on the +readable list. + +Tier 2's test is *"is the format the lesson?"*, not *"does the format carry information?"* — `.dta` carries +no information beyond its values, but `pd.read_stata` **is** the lesson in `ols.md`, so it is never +converted. + +### 4.3 CSV conversion is lossless in the file, not on read + +Verified 2026-08-10. `to_csv` writes shortest-round-trip repr, but **pandas' default CSV parser is fast, not +correctly rounded.** On `dataBHS.mat` → CSV: default `float_precision='high'` differs in 18 of 708 values +(max relative error 2.1e-16); `'legacy'` differs in 164; `'round_trip'` is bit-exact. + +So any binary→text conversion must **state its read**, and its gate must run `np.array_equal` **under the +reader the lecture will actually use**. 2.1e-16 is immaterial to a figure, but PLAN rule 4's discipline is +"provably cannot change output", and this is the difference between provable and negligible. + +--- + +## 5. The call-site audit — input to the package + +**Snapshot: 2026-08-10**, six repos, 40 lectures, 115 read sites. Working artifacts live in `.dev/qeld/`: +the machine-readable worklist (`callsites.yml`) and the worked before/after evidence per idiom +(`migration-catalog.md`, the spec for what a correct conversion produces). **Regenerate the worklist before +the sweep** — `lecture-python-intro` and this repo both moved during the session that produced it. + +### 5.1 The unit of work is ~78 URL definitions, not 115 reads + +23 of the 115 sites are downstream uses of a variable assigned earlier (`pd.read_excel(data_url, …)`). +`french_rev` reads `dette_url` six times. One `qeld.url()` call serves N reads — which is why the call-site +rule preserves variables. + +| idiom | definitions | notes | +|---|---:|---| +| A — split string literal across lines | 23 | the clearest win; `polars.md` carries the same 3-line literal twice | +| B — `url` variable = single literal | 32 | one-line RHS swap | +| C — inline literal inside the read | 18 | `ols.md` alone has five 150-char `read_stata` literals | +| D — relative / local path | 5 | **portability bugs today** — these break in Colab | +| F — `requests`+`BytesIO` → `np.load` | 4 | resolved by §4.2, not by the package | +| G — live API endpoint | 4 | carve-out | +| H — `scipy.io.loadmat` | 1 | `dataBHS.mat`, see §8.4 | + +### 5.2 The honest case for the package + +Classified in the worklist: **13 sites are structural** — `qeld` deletes real logic — of which **6 are +convertible today** and the rest are blocked on the format decision (§8.3, §8.4). **70 are cosmetic** +shortenings of literals adopted in the *recent repoint PRs*; a plain one-line literal would win those back +without a package. 32 are neutral or excluded. + +**The case rests on:** those ~10 structural sites, the wasm shim, the host-cutover property, and group D's +portability bugs. **Not** on the read-site tally. Anyone re-reading this plan should weigh it on that basis. + +The best diff in the corpus is `lecture-python-advanced.myst/lectures/hansen_jagannathan_1991.md:182`, where +`qeld` deletes a hand-rolled version of itself — a 3-fragment URL literal, a local-vs-remote branch and a +`Path("lectures") / url` fallback, 16 lines to 3, plus an unused `pathlib` import. + +### 5.3 `url()` alone covers the corpus + +After the `.npy` conversion (§8.3), **`dataBHS.mat` is the only file in the endgame that cannot be read from +a URL**. That is what justifies dropping `fetch()`. + +### 5.4 Two latent bugs found, worth fixing regardless + +- `inequality.md` imports `pyodide_http` and **never calls `patch_all()`**. +- `short_path.md` calls `requests.get` under Pyodide **with no shim at all**. + +Both are incidentally fixed by §3.2's shim, but neither should wait for it. + +### 5.5 Two problems no syntax solves + +**Generic filenames collide in a flat namespace.** `fred_data.csv`, `fp.dta`, `test_pwt.csv`, +`acs_data_summary.csv` all need renaming at migration — and `mle.md:160` names `mle/fp.dta` **in prose**, so +a rename must be paired with a prose edit found by grep, not by the audit. + +**Prose and `{download}` links.** 28 refs; **11 name a data file in two places**. The acute case is +`simple_linear_regression.md` (intro:411/416, zh-cn:421/426), where a `{download}` role and the code read sit +five lines apart containing byte-identical 158-character strings. + +--- + +## 6. Integrity and CI (D9) + +**Baseline, stated honestly:** a bare URL read has no integrity guarantee today either. Git at the source is +the guarantee, and it remains the guarantee. This design **declines to add a property rather than removing +one** — important, so nobody later re-adds `fetch()` to recover something the series never had. + +| leg | status | +|---|---| +| Committed bytes vs `integrity.sha256`, every manifested file | ✅ **already landed** (#56) — hashes whenever a manifest records a hash, with or without consumers | +| URL *spelling* assertions (`ref == main`, path shape, media host, CORS form) | ✅ **already landed** (#55, #48, #47) — static, from the parsed URL | +| **Live serving-URL fetch vs the manifest hash** | ❌ **the remaining gap** — see below | +| Catalog compile + freshness, shared with `CATALOG.md` | ❌ to build (§3.3) | +| Format tier-1 assertion over the catalog | ❌ to build (§4.2) | + +**The live leg belongs post-merge and on a schedule, never on PRs.** `audit-dashboard.yml` gates deploy on +`if: github.event_name != 'pull_request'`, so a PR branch's bytes are never on the host: a new-dataset PR +would 404 and a correction PR would compare old served bytes against a new manifest hash. Both fail by +construction — and those are exactly the PR classes such a gate would exist for. + +This leg catches what the committed-bytes check structurally cannot: an LFS pointer served instead of +content, a Pages misconfiguration, a stale CDN, a 404. It opens an issue on failure rather than blocking. + +**Recorded gap:** a reader running a downloaded notebook years later against a drifted host gets no warning +in their own session. That is **also true today**; the scheduled leg catches drift centrally, so protection +is indirect. Net position after this work: strictly better than today, short of a hashing client. + +--- + +## 7. Development plan + +Ordering is forced by one constraint: **the audit must learn the call form before any consumer adopts it**, +or every migrated read classifies `local-path` and the dashboard inverts. + +| phase | work | gate | +|---|---|---| +| **Q1 — Audit first** | `build_audit.py` learns `qeld.url('X')` → pattern `qeld`, counted migrated **and terminal**. For `pattern == 'qeld'`, assert the key exists in `lectures/` and is not deprecated — otherwise the qeld path loses every assertion #55/#48/#47 added. `migration.yml`: `final` := canonical-host *or* qeld | `audit.json` `stats` and `problems` unchanged on today's repos (**not** "byte-identical" — the audit stamps `date.today()`) | +| **Q2 — Schema hygiene** | Document `read_as` (used in 6 manifests) and `sheets` (5) in `manifest-schema.yml` — both are in use and neither appears in the file `AGENTS.md` calls "the authoritative, commented field reference". Add `deprecated:` (new, used nowhere yet) since §3.3 warns on it. `shape` is already documented. Delete `then: "iloc[1:]"` from `longprices.xls.yml:70` by moving `iloc[1:]` into the lecture — a post-read transform encoded as a string to evaluate is exactly what D5 excludes | `manifest-schema.yml` covers every field any manifest uses. Needs none of #14's decisions — do not block on it | +| **Q3 — Package** | `packages/qeld/`: `url()`, `info()`, context detection, advisory catalog. Catalog compiler shares a freshness gate with `CATALOG.md`. Format tier-1 assertion. First release to PyPI via trusted publishing | Offline suite green on every PR: catalog compiles and is fresh; unknown key warns and still returns a URL; URL form correct per detected context; suffix fidelity incl. `.csv.gz`; `info()` fields present. CPython matrix | +| **Q4 — Live leg** | Post-merge + scheduled job: fetch each served URL, compare to the manifest hash, open an issue on failure | Green on `main`; an induced failure opens an issue | +| **Q5 — Browser session** | `%pip install qeld==` in a real `lecture-wasm` page (**`%pip` routes through piplite, not micropip** — a console `micropip.install` is a false pass); `pd.read_excel(qeld.url('mpd2020.xlsx'), sheet_name='Regional data', header=[0,1,2], index_col=0)`; a `.csv.gz` read; record observed Pyodide and pyodide-kernel versions | Written pass/fail. Fail ⇒ wasm keeps URLs and the plan proceeds for the CPython repos | +| **Q6 — Pilots, by win** | `hansen_jagannathan_1991` (deletes the dual-path loader — the best diff); `french_rev` (the `base_url` block, intro **and** wasm together per repoint rule 2); `subjective_beliefs_business_cycles` or `match_transport` (group D, a real portability bug) | Diff is the URL expression only; intro figure-hash equality; wasm = re-run Q5's checklist on the pilot page with the console transcript filed. On pass: tag 1.0, freeze the API | +| **Q7 — Sweep by win** | The remaining structural sites, then cosmetic sites at leisure or never. `AGENTS.md` gains §4.1 verbatim, §4.2, and the note that repoint rules 5–6 do not apply to qeld call sites | `migrated` meter reflects qeld refs; no lecture regresses | + +**Effort note:** Q3 is not a weekend. The package is small, but the catalog compiler, the tier-1 assertion, +the offline suite and PyPI trusted-publisher configuration are four separate pieces, and the last is not a +code task. + +**Pyodide version pinning is accidental** — the version ships inside `thebe-lite.min.js` inside the theme +zip, so a CSS-only theme bump moves the Python runtime. Record Q5's verdict as "Pyodide 0.27 as shipped by +thebe-lite in quantecon-theme v2.1.0" and re-run when the theme moves. + +--- + +## 8. Ambiguities — decide before the phase that needs them + +### 8.1 `pandas_panel`'s carve-out — permanent or revisited? + +The prose advertises the URL as something a beginner can paste into a browser. Converting is a pedagogical +loss with no offsetting gain (those files already resolve to data-lectures in the form `qeld` emits). The +alternative is rewriting the prose, which is a larger edit than the one being justified. **Needed by Q7.** + +### 8.2 The markdown-time story for `{download}` links + +`qeld.url()` cannot appear in markdown. Options: convert both and accept two spellings; leave those lectures +alone; or give `qeld` a markdown-time story (a MyST substitution, or point prose at `CATALOG.md`). Until +answered, "convert neither" is the default. **Needed by Q7.** + +### 8.3 The `.npy` pair — now a breaking change + +`caron.npy` and `nom_balances.npy` are (63, 2) and (81, 2) float64 arrays whose manifests already name the +columns (`date`/`specie_value`, `date`/`nominal_balances`). Converting to CSV deletes `requests`, `BytesIO` +and two imports from `french_rev` in every consuming repo. + +**But the window closed.** When this was analysed both files had `consumers: []`; the A3 set has since been +repointed (#49) and both now have two consumers. So this is no longer a free replacement — it needs the +`AGENTS.md` "new vintage → new filename" treatment (`caron.csv` lands alongside, consumers opt in, the `.npy` +is swept later) or a coordinated set under repoint rules 1–3. **Decide before Q6**, since `french_rev` is a +pilot. + +### 8.4 `dataBHS.mat` — convert at migration, or exclude? + +5,588 bytes; `c`, `rb`, `rs`, each (236, 1) float64; the lecture uses only `data['c']` and the read is inside +a `hide-input` cell, so nothing about it is taught. Trivially a 236×3 CSV — but see §4.3 on the read. A Track +C decision; the only true impossibility among static files. + +### 8.5 Is `lecture-intro.zh-cn` in scope? + +It carries data reads, appears in **zero** `consumers` blocks, is excluded from `SCAN_REPOS` by decision, has +no data CI, publishes on a `publish*` tag, and inherits install cells automatically via the `.md`-only sync — +so it acquires whatever intro acquires without anyone deciding. It also has files with no data-lectures key +and no business having one (`country_code_cn.csv`, a translation asset). +**Recommendation: explicit non-goal for v1, with one fixed rule instead of machinery — any sweep touching an +intro file also touches zh-cn.** + +### 8.6 The rename list for generic filenames + +§5.5. Needs a pass before Tracks B and C migrate, and each rename needs its prose pairing found by grep. + +### 8.7 Open from the original report + +- Ask Spencer/Tom what actually retired `qeds` — it is dead as a project but **still installable** (PyPI + returns 200 for 0.7.0, not yanked), one letter-transposition from `qeld`. Consider reserving + `quantecon-data` and `qedata` as stubs. +- `data.quantecon.org` DNS (#37) is the nearest unblocked item and races this work for the same weekend. +- Publishing `qeld` does **not** move #35's licensing gate — it rehosts nothing and fetches the same public + URLs `pandas_panel` reads today. Do not add a release check that fails on `redistribution: restricted`; + `countries.csv` is restricted and unresolved, so it would block every release from day one. + +--- + +## 9. Corrections to the original design report + +Recorded so they are not re-proposed. The report is superseded by this document. + +| claim | status | +|---|---| +| "18/18 manifests", "all 18, not a sample" | stale before the ink dried — 24 manifests, 27 files on `main` | +| Exact CalVer pins, "the install cell names the data vintage" | a version cannot pin bytes — §2.1 | +| "CI fails with `IntegrityError`; that failure *is* the rebuild signal" | wasm CI does not execute; intro is `execute_notebooks: "cache"` and a data correction changes no code, so PRs are cache hits | +| "no lecture-content edits, ever" | the install cell is lecture content, and prose data URLs already exist | +| "six different URL ways" | the audit says five in use | +| "closes #8" | #8's catalog box was already ticked; this adds the freshness CI | +| A `~200-line prototype` and an evidence document | neither exists on disk; `git log --all` has zero hits for `qeld`. Both load-bearing prototype claims re-derive from the manifests |