From a8924d1100d5870c971fa5ed436a0c020182e67f Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Wed, 12 Aug 2026 11:23:09 +1000 Subject: [PATCH] Record D11: invest in qeld, defer data.quantecon.org indefinitely The programme carried two candidate stable interfaces without choosing: a branded host (data.quantecon.org, Track Y / #37 / #15) and the qeld package. Decision taken 2026-08-12: the package is the interface; the custom domain is deferred indefinitely. PLAN-QELD-PACKAGE.md gains D11 with the reasoning recorded in full (section 2.2): tidier lectures, print(url) keeps the URL reachable, backend freedom behind one constant interface point, info() metadata at the call site, and no forever-promise of a branded public host -- which also means #35's promotion gate never comes due unscheduled. The operational condition: qeld's base URL stays on the raw repo forms and never moves to quantecon.github.io, so the deferral stays reversible (a custom domain added later 301s the github.io form, re-creating the redirect-CORS failure for shipped wheels; raw URLs are untouched). Consequences swept through the repo so no document re-proposes the retired plan: - PLAN.md: Track Y re-scoped to the qeld plan; Phase 4's DNS box struck as deferred (measurement preserved for any revisit); rule 5's CORS acceptance criterion restated host-generically; the URL-form open decision marked settled; the licensing promotion-gate note updated - README.md / AGENTS.md: the direct URL forms are standing, not interim; qeld.url() is the consumer interface once the package ships - migration.yml: `final` redefined as qeld adoption (with the section 4.1 carve-outs terminal on the direct form), per Q1 - render_audit.py / build_catalog.py: dashboard and catalog prose no longer promise the data.quantecon.org sweep (text only, no logic) See #37 (stays open as the deferred tracker) and #15 (superseded by qeld Q6-Q7 adoption). Part of #66's decision record for the qeld plan (#65). Co-Authored-By: Claude Fable 5 --- AGENTS.md | 9 +++--- PLAN-QELD-PACKAGE.md | 61 ++++++++++++++++++++++++++++++++++------ PLAN.md | 22 +++++++-------- README.md | 19 ++++++------- migration.yml | 10 +++++-- scripts/build_catalog.py | 4 ++- scripts/render_audit.py | 20 +++++++------ 7 files changed, 97 insertions(+), 48 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b1ad0c9..8e03ebb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ Guidance for coding agents (and humans) making changes in this repository. Read ## What this repo is -The canonical home for **data consumed by the QuantEcon lecture series** (renamed from `QuantEcon/data` on 2026-07-16, per [meta#336](https://github.com/QuantEcon/meta/issues/336)). Its purpose is **stability**: it snapshots upstream sources — with attribution to each source carried in the manifest — so a lecture build never depends on a live API or a third-party host staying up. It is a **cache, not a content-distribution host**. The published tree is **flat** (`lectures/`, since 2026-07-16) and live on GitHub Pages; the remaining transition is the custom domain — files are served at `quantecon.github.io/data-lectures/lectures/` today and will move to `https://data.quantecon.org/lectures/` once DNS is resolved (PLAN Phase 4, [#15](https://github.com/QuantEcon/data-lectures/issues/15)). The full convention lives in the draft manual page ([QuantEcon.manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108)). +The canonical home for **data consumed by the QuantEcon lecture series** (renamed from `QuantEcon/data` on 2026-07-16, per [meta#336](https://github.com/QuantEcon/meta/issues/336)). Its purpose is **stability**: it snapshots upstream sources — with attribution to each source carried in the manifest — so a lecture build never depends on a live API or a third-party host staying up. It is a **cache, not a content-distribution host**. The published tree is **flat** (`lectures/`, since 2026-07-16) and live on GitHub Pages; consumers fetch it over the raw GitHub forms below. **There is no pending host transition**: the `data.quantecon.org` custom domain was deferred indefinitely on 2026-08-12 in favor of the `qeld` consumer package (`PLAN-QELD-PACKAGE.md`, D11) — the stable interface lectures get is a package call, `qeld.url('')`, not a branded host ([#37](https://github.com/QuantEcon/data-lectures/issues/37), [#15](https://github.com/QuantEcon/data-lectures/issues/15)). The full convention lives in the draft manual page ([QuantEcon.manual#108](https://github.com/QuantEcon/QuantEcon.manual/pull/108)). ## Rules @@ -72,7 +72,7 @@ There is **no form that is safe in every case**, and that is the whole trap. The - **CPython consumers** (site notebooks, Colab, every series except `lecture-wasm`): `https://github.com/QuantEcon/data-lectures/raw/main/lectures/`. - **`lecture-wasm`** (code cells execute under Pyodide in the reader's browser — **repoint rule 5**): `https://raw.githubusercontent.com/QuantEcon/data-lectures/main/lectures/`, and nothing else. The strict audit hard-fails any `lecture-wasm` code-cell read via a `github.com/…` form. `{download}` targets and prose links are navigations and are CORS-exempt, so any resolving form is fine there. - **Never `media.githubusercontent.com` for a path in this repo** (**repoint rule 6**). It is the LFS media endpoint and routes per *path*, so it 404s everything `lectures/` publishes — which is all of it, since the published tree is 100% plain git. The strict audit fails on this too. -- Final form once Pages is live: `https://data.quantecon.org/lectures/`. +- There is **no pending final-form swap** — the two direct forms above are standing (decision 2026-08-12, `PLAN-QELD-PACKAGE.md` D11; `data.quantecon.org` deferred, [#37](https://github.com/QuantEcon/data-lectures/issues/37)). Once the `qeld` package ships, lecture code reads `qeld.url('')`, which emits exactly these forms per runtime; prose links, `{download}` targets and URL-is-the-lesson reads keep the direct form permanently. - Never reference a non-default branch in a published URL — the audit fails on any ref that is not `main`. The failure modes are silent in both directions, which is why all three are machine-checked: a `raw` read of an LFS path returns **HTTP 200 with ~133 bytes of pointer text**, and `pd.read_csv` on it raises nothing and yields a 2×1 frame. @@ -128,7 +128,7 @@ Because this repo is a **stability cache, not a content-distribution host** (see For the public data sources most snapshots come from (World Bank, FRED, Eurostat, …) the answer is a **known yes, recorded once per source** — permissive terms plus attribution. Record what the source states and move on; don't re-litigate it per snapshot. Treat the manifest's `redistribution` field as a **cheap binary gate** (`permitted` / `restricted`): a fast `permitted` for public statistics agencies, `restricted` blocking only the genuinely restricted source before it goes public — e.g. FRED re-serves third-party series that may not be redistributed, and anything under non-commercial or no-redistribution terms must not be cached here, since attribution alone does not cure those. Capture licence detail richly when the source provides it; where it is genuinely unavailable, record the gap rather than blocking the file. -**Licensing does not gate migration** (settled 2026-08-06, [#35](https://github.com/QuantEcon/data-lectures/issues/35)). A file the lectures have **already served publicly** migrates here with its licence recorded *as found* — including `redistribution: restricted` and a null licence name where that is the honest answer — and is logged in the inventory ([#35](https://github.com/QuantEcon/data-lectures/issues/35), feeding [workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)) with a `note`. Resolve it (permission, an open replacement, or removal) before `data.quantecon.org` is promoted as a public open-data host: **that promotion is the gate, not the file's move.** Rehosting the same bytes with better provenance and an explicit licence field improves on the status quo, so a licence question is never a reason to stall a migration. +**Licensing does not gate migration** (settled 2026-08-06, [#35](https://github.com/QuantEcon/data-lectures/issues/35)). A file the lectures have **already served publicly** migrates here with its licence recorded *as found* — including `redistribution: restricted` and a null licence name where that is the honest answer — and is logged in the inventory ([#35](https://github.com/QuantEcon/data-lectures/issues/35), feeding [workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)) with a `note`. Resolve it (permission, an open replacement, or removal) before this repo is ever promoted as a branded public open-data host: **that promotion is the gate, not the file's move** — and with the custom domain deferred indefinitely (2026-08-12, D11), no promotion is scheduled, so the gate looms over nothing while the [#35](https://github.com/QuantEcon/data-lectures/issues/35) inventory stays tracked. Rehosting the same bytes with better provenance and an explicit licence field improves on the status quo, so a licence question is never a reason to stall a migration. This covers **inherited** data only. A genuinely new dataset — one with no prior life in a lecture repo — still has its licence established *before* it lands, as the P5 additions all did. @@ -150,7 +150,8 @@ The generated dashboard (`scripts/build_audit.py`, [#20](https://github.com/Quan ## Repo map ``` -lectures/ # the published tree — flat, live on Pages; data.quantecon.org pending +lectures/ # the published tree — flat, live on Pages; read via raw URLs + # today, qeld.url() once the package ships (PLAN-QELD-PACKAGE.md) # 21 files, 18 with manifests (business_cycle's three still # need theirs — see #13). Manifests are sidecars: .yml builders/ # one builder per published dataset — NOT published diff --git a/PLAN-QELD-PACKAGE.md b/PLAN-QELD-PACKAGE.md index 3b29102..4a23afb 100644 --- a/PLAN-QELD-PACKAGE.md +++ b/PLAN-QELD-PACKAGE.md @@ -1,6 +1,6 @@ # PLAN — `qeld`, the consumer-side data package -**Status:** design settled, nothing implemented · **Last updated:** 2026-08-10 +**Status:** design settled, nothing implemented · **Last updated:** 2026-08-12 **Relationship to `PLAN.md`:** that document migrates *bytes* into this repo. This one gives *consumers* a stable way to read them. They are independent — the migration completes with or without `qeld` — but the call-site convention here replaces repoint rules 5–6 for any lecture that adopts it. @@ -26,8 +26,10 @@ It exists to solve three concrete problems: because the first fails CORS in the browser. That split *is* repoint rules 5–6. `qeld.url()` erases it. 2. **`pyodide_http.patch_all()` is in every wasm lecture.** Importing `qeld` under emscripten installs the transport shim, and those two lines leave the lectures. -3. **The host cutover.** When `data.quantecon.org` lands (#37, #15), it is one constant in one package - rather than an edit in every lecture. +3. **The backend stays reworkable.** Any future change of host or serving arrangement is one constant in + one package rather than an edit in every lecture. This property is now load-bearing rather than + incidental: the `data.quantecon.org` cutover it originally anticipated was itself retired in favor of + it (D11) — the package, not a branded host, is the stable interface. **What it is not:** a cache, a fetcher, a loader, a data-version manager, or an integrity client. See §3.1. @@ -37,7 +39,8 @@ It exists to solve three concrete problems: D1–D5 were taken 2026-08-10 against the original design report. D2, D3 and D5 were **revised the same day** in a working session that re-scoped the package from fetch-and-cache to URL-resolver; D1 and D4 stand -unchanged. D6–D10 are new. +unchanged. D6–D10 are new. D11 was taken 2026-08-12 and reaches outside the package: it retires the +`data.quantecon.org` plan this document previously deferred to. | # | decision | status | |---|---|---| @@ -51,6 +54,7 @@ unchanged. D6–D10 are new. | **D8** | **Data format: tier 1 binds at intake, tier 2 is a forward-looking preference** | new — §4.2 | | **D9** | **Integrity lives in CI, not at the call site** | new — §6 | | **D10** | **Rollout is ordered by win, not by migration track** | new — §7 | +| **D11** | **`qeld` replaces the host cutover: `data.quantecon.org` is deferred indefinitely, and the base URL stays on the raw repo forms** | new — §2.2 | **The scoping principle throughout:** ship the minimum; add sophistication as demand requires. @@ -69,6 +73,42 @@ on inspection, and the reasoning is recorded here so it is not re-proposed: **What `<2` protects, and therefore what the major version means:** catalog keys are append-only, and a key's meaning never changes within a major version. +### 2.2 D11 — the package replaces the custom domain (2026-08-12) + +The programme carried two candidate stable interfaces without ever choosing between them: a branded host +(`data.quantecon.org/lectures/` — PLAN Track Y, #37, #15) and this package. **Decision: invest in +`qeld`; defer the domain indefinitely.** Taken by the maintainer 2026-08-12; the recorded reasons: + +1. **Tidier lectures.** A dataset lookup reads better than a URL literal — and the URL is still one + `print(url)` away for any reader who wants it, since `url()` returns a plain string. +2. **Backend freedom.** The serving arrangement can be reworked as project needs evolve behind one + constant interface point in the lectures. A branded host offers the same indirection only for host + moves; the package's indirection covers key naming, transport, and context too. +3. **More than a URL.** `info()` puts provenance, licence and citation metadata at the call site — + something no URL scheme can offer. +4. **No forever-promise.** A branded public data host must be served at that name indefinitely, and #35 + makes its licensing review a gate on promoting it. The package makes no such promise: it points at + whatever the repo already serves, so the #35 promotion gate simply never comes due unless a public + host is someday established after all. + +It is not an effort trade — #37 is two DNS/settings actions while Q3 "is not a weekend" — it is a +commitment trade, and the technical ground is already level: the raw forms and the Pages default URL both +serve `access-control-allow-origin: *` on a direct 200, so nothing consumer-facing waits on a domain. + +**The operational consequence — the base URL stays on the raw repo forms** (`raw.githubusercontent.com` +under Pyodide, the 302-tolerant `github.com/…/raw/` form elsewhere) **and must never move to +`quantecon.github.io`.** The moment a custom domain is attached to a Pages site, GitHub starts +301-redirecting the `github.io` form to it — re-creating for already-shipped wheels exactly the +redirect-CORS failure §3.2 exists to avoid. Raw URLs are untouched by Pages domain changes, so on a +raw-link base the deferral is reversible at any time for one minor release. What is accepted in exchange: +emitted and leaked URLs name `QuantEcon/data-lectures@main`, with GitHub's rename redirects as the only +safety net — the same exposure every lecture URL has today. + +Consequences recorded elsewhere in this document and the repo: the "interim" URL forms are **standing, +not interim**; #15's final-URL sweep is superseded by Q6–Q7 adoption; `migration.yml`'s `final` status is +redefined (Q1); §8.7's note that #37 "races this work" is retracted; PLAN.md Track Y is re-scoped to this +plan. + --- ## 3. Design @@ -94,7 +134,9 @@ everywhere. A design where context changes *what you get* produces "works in my Exactly two jobs: 1. **URL form** — `raw.githubusercontent.com` under Pyodide (the `github.com/…/raw/` form 302s and fails - CORS); the 302-tolerant form elsewhere; `data.quantecon.org` for both after #37. + CORS); the 302-tolerant form elsewhere. These are the standing forms, not interim ones: the + single-host collapse a custom domain would have offered was declined in D11, and the base never moves + to `quantecon.github.io` (redirect trap, §2.2). 2. **Browser transport shim** — under emscripten, `import qeld` installs the fetch shim, because the right URL is still not enough: `pd.read_excel(url)` goes through urllib, which fails in the browser. @@ -222,8 +264,8 @@ convertible today** and the rest are blocked on the format decision (§8.3, §8. shortenings of literals adopted in the *recent repoint PRs*; a plain one-line literal would win those back without a package. 32 are neutral or excluded. -**The case rests on:** those ~10 structural sites, the wasm shim, the host-cutover property, and group D's -portability bugs. **Not** on the read-site tally. Anyone re-reading this plan should weigh it on that basis. +**The case rests on:** those ~10 structural sites, the wasm shim, the backend-rework property (§1, now +load-bearing per D11), and group D's portability bugs. **Not** on the read-site tally. Anyone re-reading this plan should weigh it on that basis. The best diff in the corpus is `lecture-python-advanced.myst/lectures/hansen_jagannathan_1991.md:182`, where `qeld` deletes a hand-rolled version of itself — a 3-fragment URL literal, a local-vs-remote branch and a @@ -290,7 +332,7 @@ or every migrated read classifies `local-path` and the dashboard inverts. | phase | work | gate | |---|---|---| -| **Q1 — Audit first** | `build_audit.py` learns `qeld.url('X')` → pattern `qeld`, counted migrated **and terminal**. For `pattern == 'qeld'`, assert the key exists in `lectures/` and is not deprecated — otherwise the qeld path loses every assertion #55/#48/#47 added. `migration.yml`: `final` := canonical-host *or* qeld | `audit.json` `stats` and `problems` unchanged on today's repos (**not** "byte-identical" — the audit stamps `date.today()`) | +| **Q1 — Audit first** | `build_audit.py` learns `qeld.url('X')` → pattern `qeld`, counted migrated **and terminal**. For `pattern == 'qeld'`, assert the key exists in `lectures/` and is not deprecated — otherwise the qeld path loses every assertion #55/#48/#47 added. `migration.yml`: `final` := every code read via qeld, with §4.1 carve-outs terminal on the direct form (the canonical-host arm was retired with D11) | `audit.json` `stats` and `problems` unchanged on today's repos (**not** "byte-identical" — the audit stamps `date.today()`) | | **Q2 — Schema hygiene** | Document `read_as` (used in 6 manifests) and `sheets` (5) in `manifest-schema.yml` — both are in use and neither appears in the file `AGENTS.md` calls "the authoritative, commented field reference". Add `deprecated:` (new, used nowhere yet) since §3.3 warns on it. `shape` is already documented. Delete `then: "iloc[1:]"` from `longprices.xls.yml:70` by moving `iloc[1:]` into the lecture — a post-read transform encoded as a string to evaluate is exactly what D5 excludes | `manifest-schema.yml` covers every field any manifest uses. Needs none of #14's decisions — do not block on it | | **Q3 — Package** | `packages/qeld/`: `url()`, `info()`, context detection, advisory catalog. Catalog compiler shares a freshness gate with `CATALOG.md`. Format tier-1 assertion. First release to PyPI via trusted publishing | Offline suite green on every PR: catalog compiles and is fresh; unknown key warns and still returns a URL; URL form correct per detected context; suffix fidelity incl. `.csv.gz`; `info()` fields present. CPython matrix | | **Q4 — Live leg** | Post-merge + scheduled job: fetch each served URL, compare to the manifest hash, open an issue on failure | Green on `main`; an induced failure opens an issue | @@ -361,7 +403,8 @@ intro file also touches zh-cn.** - Ask Spencer/Tom what actually retired `qeds` — it is dead as a project but **still installable** (PyPI returns 200 for 0.7.0, not yanked), one letter-transposition from `qeld`. Consider reserving `quantecon-data` and `qedata` as stubs. -- `data.quantecon.org` DNS (#37) is the nearest unblocked item and races this work for the same weekend. +- ~~`data.quantecon.org` DNS (#37) is the nearest unblocked item and races this work for the same + weekend.~~ **Retracted 2026-08-12** — D11 defers #37 indefinitely; nothing races this work. - Publishing `qeld` does **not** move #35's licensing gate — it rehosts nothing and fetches the same public URLs `pandas_panel` reads today. Do not add a release check that fails on `redistribution: restricted`; `countries.csv` is restricted and unresolved, so it would block every release from day one. diff --git a/PLAN.md b/PLAN.md index 591660e..39cf35f 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1,6 +1,6 @@ # PLAN — `data-lectures` (formerly `QuantEcon/data`) -**Status:** active roadmap (last updated 2026-08-10) — **the repo is LIVE**: the first repoint merged 2026-07-17 (P1, `lingcod_msy_recovery.csv` → `msy_fishery`), so published filenames are an API from here on +**Status:** active roadmap (last updated 2026-08-12) — **the repo is LIVE**: the first repoint merged 2026-07-17 (P1, `lingcod_msy_recovery.csv` → `msy_fishery`), so published filenames are an API from here on **Where the numbers stand (`audit.json`, 2026-08-10):** 18 of 41 static datasets migrated and repointed, 23 to go; 22 lectures still fetch live API data; 26 committed orphans; 0 legacy-repo references; 5 URL forms in use. @@ -39,7 +39,7 @@ This repository is being shaped into the **single canonical repository for data ## Where we're going (per the draft convention) -- **Flat published tree** served at `https://data.quantecon.org/lectures/` via GitHub Pages (custom domain), CORS-open for pyodide/JupyterLite +- **Flat published tree** at `lectures/`, served over the raw GitHub forms (and GitHub Pages), CORS-open for pyodide/JupyterLite. **The custom-domain plan (`data.quantecon.org`) was retired 2026-08-12**: the stable consumer interface is the `qeld` package (`PLAN-QELD-PACKAGE.md`, D11), not a branded host — lectures read `qeld.url('')` and the direct URL forms below are standing, not interim - Every dataset classified **verbatim / constructed / dynamic snapshot**, each with a manifest — authoritative field reference in `manifest-schema.yml`, as revised by the P1 pilot (`integrity`, `builder_status`, `known_nulls` and `license.verified` joined the original sketch of `source` / `license` / `retrieved` / `schema` / `consumers` / `maintainer` / `cadence`) - Constructed and dynamic datasets ship their **builder**; dynamic datasets get **scheduled refresh-as-PR** plus a weekly **sources-alive canary** - The published tree is **100% plain git**; per-path LFS is confined to `sources/`, which is never served. Storage does **not** decouple from hosting — the URL a consumer must write is a function of how the file is stored, and there is no browser-safe form invariant under a storage flip (repoint rule 6, and [#58](https://github.com/QuantEcon/data-lectures/issues/58) for the ladder above 100 MiB) @@ -129,7 +129,7 @@ Learned from the independent validation ([#45](https://github.com/QuantEcon/data `{download}` targets and prose links are plain navigations — CORS does not apply, and the `github.com` form is fine there. The audit classifies references by org/repo across all URL forms, so both spellings count as the same pattern — but the strict build now also checks the *form*: any `lecture-wasm` code-cell read via a `github.com/…` URL fails the audit. That is a **post-merge net, not a gate** — the scan reads each lecture repo's `main`, so a violation turns the dashboard red at the next audit run rather than blocking the offending PR; the repoint PR remains the place the rule is actually upheld. Quick test from any `quantecon.github.io` page console: `fetch('')` — the bad form rejects, the good form resolves. -Phase 4 inherits the requirement: `data.quantecon.org` must serve `access-control-allow-origin: *` before `lecture-wasm` can cut over to it — recorded as an acceptance criterion on [#37](https://github.com/QuantEcon/data-lectures/issues/37). +The requirement outlives the plan that created it: any host `lecture-wasm` reads from must serve `access-control-allow-origin: *` on a direct 200, no redirect hop. The `data.quantecon.org` cutover this criterion was recorded against ([#37](https://github.com/QuantEcon/data-lectures/issues/37)) was retired 2026-08-12 in favor of `qeld` (`PLAN-QELD-PACKAGE.md` D11); the criterion binds again in full if a custom domain is ever revisited. ### 6. `media.githubusercontent.com` is LFS-only — a fold changes the *host*, not just the org @@ -197,13 +197,13 @@ The remaining work decomposes by **consuming series** rather than by hosting pat | **D — `programming`** | 1: `test_pwt.csv` | none | nothing — a single-PR track | | **E — dynamic / live-API** | the UNRATE twin, then the 15 incidental API lectures | wasm is the forcing customer | [#14](https://github.com/QuantEcon/data-lectures/issues/14) schema decisions, [#26](https://github.com/QuantEcon/data-lectures/issues/26) fetch layer | | **X — orphan sweep** | 26 committed orphans across 6 repos — dp 10, programming 5, wasm 5, intro 3, python.myst 2, `continuous_time_mcs` 1 | per repo | that repo's repoints landing first | -| **Y — infra / cutover** | DNS → custom domain → interim-to-final URL sweep → QEP | — | nothing external: the name is NXDOMAIN and the record is ours to create | +| **Y — consumer interface (`qeld`)** | the `qeld` package, Q1–Q7 of `PLAN-QELD-PACKAGE.md` — audit support, the package, pilots, then adoption by win; QEP graduation stays | — | nothing — re-scoped 2026-08-12 (D11): the DNS → custom domain → URL-sweep sequence this row used to carry is retired | `lecture-dp`, `lecture-jax` and `continuous_time_mcs` are **not data consumers** — dp's 10 committed files are inherited orphans, jax embeds `graph.txt` via `%%file`, and continuous_time_mcs has one orphan scratch file. They appear only in Track X. -**Tracks A–D are independent of each other and can run in any order or in parallel.** The only hard dependencies in the whole programme are: `usa-gini-nwealth-tincome-lincome.csv` is built from `SCF_plus_mini.csv` (so it follows the SCF migration inside Track A); Track E's rollout needs its own template proven first; Track X follows its repo's repoints; and Track Y's cutover is last. +**Tracks A–D are independent of each other and can run in any order or in parallel.** The only hard dependencies in the whole programme are: `usa-gini-nwealth-tincome-lincome.csv` is built from `SCF_plus_mini.csv` (so it follows the SCF migration inside Track A); Track E's rollout needs its own template proven first; Track X follows its repo's repoints; and Track Y's adoption sweep (qeld Q7) is last. -Track Y no longer has external lead time: the stale A record was deleted and `data.quantecon.org` is now NXDOMAIN at its own authoritative nameserver, so creating it is ours to do. One thing to settle before it: `classify_url` recognises six GitHub-host regexes and neither `quantecon.github.io` nor `data.quantecon.org`, so a consumer on the canonical host classifies as `external-web`, `migrated` goes false, and **the `final` status is a state the audit is structurally guaranteed to report as broken** — which also lapses both of rule 6's assertions at the cutover. Teach the classifier the canonical host before the sweep, not after. +**Track Y was re-scoped 2026-08-12: invest in `qeld`, defer the custom domain indefinitely** (`PLAN-QELD-PACKAGE.md` D11, where the reasoning is recorded in full). The short form: the package delivers everything the domain would have — a stable interface point that survives backend rework — plus tidier lectures and call-site metadata, without the forever-promise of a branded public host, and so without [#35](https://github.com/QuantEcon/data-lectures/issues/35)'s promotion gate ever coming due. It is a commitment decision, not an effort one: the DNS record remains two actions QuantEcon controls (the stale A record was deleted; the name is NXDOMAIN as of 2026-08-10) and [#37](https://github.com/QuantEcon/data-lectures/issues/37) stays open as deferred-not-dead, reopenable at any time because qeld's base URL stays on the raw forms (never `quantecon.github.io` — D11's redirect trap). The classifier constraint this paragraph used to carry moves with the re-scope: `classify_url` must learn the `qeld.url('X')` pattern **before** any consumer adopts it (qeld Q1), for the same structural reason it would have had to learn the canonical host before a URL sweep — otherwise every migrated read classifies as broken and the dashboard inverts. ### Where this work happens @@ -211,7 +211,7 @@ Repoints span data-lectures plus one or two lecture repos and must land together ## Phases -Ordering note: phases 1–3 and 6 can proceed now; phase 4 needs the DNS question resolved; phase 5 follows layout, **except its go-live guardrails, which must precede the first repoint**; phase 7 needs the sources recorded in phase 6; phase 8 (the pilot) is the first end-to-end pass through phases 2–7's machinery and requires phase 7's byte-compare for the files it touches **plus phase 5's go-live guardrails** — the first repoint turns `raw/main` into a production URL, so the repo must not go live unprotected; phase 9 follows the pilot (interim URL form makes repoints churn-tolerant to start earlier). +Ordering note: phases 1–3 and 6 can proceed now; phase 4's DNS question was resolved 2026-08-12 by deferring it (D11 — see the box below); phase 5 follows layout, **except its go-live guardrails, which must precede the first repoint**; phase 7 needs the sources recorded in phase 6; phase 8 (the pilot) is the first end-to-end pass through phases 2–7's machinery and requires phase 7's byte-compare for the files it touches **plus phase 5's go-live guardrails** — the first repoint turns `raw/main` into a production URL, so the repo must not go live unprotected; phase 9 follows the pilot (interim URL form makes repoints churn-tolerant to start earlier). ### Phase 0 — Scaffolding (this PR) @@ -258,8 +258,8 @@ Only one file genuinely forces LFS, and it is not a dataset: ### Phase 4 — Publishing - [x] GitHub Pages deploy of the published tree, **`lfs: false` at checkout** (inverted by [#57](https://github.com/QuantEcon/data-lectures/pull/57): a mis-tracked file must publish as its pointer, so the mistake is visible rather than masked) — landed 2026-07-17 with the audit dashboard (`.github/workflows/audit-dashboard.yml`, [#20](https://github.com/QuantEcon/data-lectures/issues/20)): the default `quantecon.github.io/data-lectures/` site serves the dashboard at `/` and the published tree at `/lectures/`. The custom domain below stays open -- [ ] `data.quantecon.org` DNS + custom domain. **Measured 2026-08-10: the name is NXDOMAIN at `quantecon.org`'s own authoritative nameserver, and the repo's Pages `cname` is null.** The stale A record and the AWS box it pointed at are gone, so this is no longer an external-ownership question — it is two actions QuantEcon controls: create the record, then set the custom domain on the repo -- [x] Verify `access-control-allow-origin: *` on served files (pyodide/JupyterLite, meta#143) — **verified 2026-08-06**: `quantecon.github.io/data-lectures/lectures/lingcod_msy_recovery.csv` returns `access-control-allow-origin: *`. The requirement is met on the default Pages domain today and does **not** wait on the custom domain; re-verify once DNS moves +- [x] ~~`data.quantecon.org` DNS + custom domain~~ — **deferred indefinitely 2026-08-12, do not do this without revisiting D11** (`PLAN-QELD-PACKAGE.md` §2.2): the `qeld` package is the stable consumer interface instead of a branded host, and the direct raw URLs are standing rather than interim. The measurement stands for whenever this is revisited: as of 2026-08-10 the name is NXDOMAIN at `quantecon.org`'s own authoritative nameserver, the repo's Pages `cname` is null, and the stale A record and the AWS box it pointed at are gone — so reviving it is two actions QuantEcon controls (create the record, set the custom domain), plus D11's condition that old wheels keep working because qeld's base never sat on `quantecon.github.io`. [#37](https://github.com/QuantEcon/data-lectures/issues/37) stays open as the deferred tracker +- [x] Verify `access-control-allow-origin: *` on served files (pyodide/JupyterLite, meta#143) — **verified 2026-08-06**: `quantecon.github.io/data-lectures/lectures/lingcod_msy_recovery.csv` returns `access-control-allow-origin: *`. The requirement is met on the default Pages domain today and never waited on a custom domain; re-verify only if DNS is ever revisited (D11) - [ ] Monitor Pages soft limits (~1 GB site, 100 GB/month) ### Phase 5 — Automation (`.github/`) @@ -285,7 +285,7 @@ Full automation: - [ ] Classify: the 8 static intro files are author-assembled or verbatim; `business_cycle_data.csv` is the one dynamic snapshot and needs its cadence declared - [ ] Licence check **per source**, not per file: the question is *"may this source be cached and served publicly, with attribution?"* — a cheap binary gate (`redistribution: permitted | restricted`, see AGENTS.md "Licensing and attribution"), a fast yes for public data sources. Two sources already answered: World Bank is **CC BY-4.0** (`business_cycle_metadata.md`, the model for what a manifest should capture) and RAM Legacy is **CC BY 4.0** (established against its Zenodo DOI record, P1). The remaining sources need the equivalent established by hand - **Licensing does not gate migration** (settled 2026-08-06, [#35](https://github.com/QuantEcon/data-lectures/issues/35)). Inherited data — anything the lecture repos already serve publicly — migrates with its licence recorded **as found**, including `redistribution: restricted` and `name: null` where that is the honest answer. Moving the same bytes to a canonical host with better provenance and an explicit licence field improves on the status quo, so the migration does not wait on review; what needs further thought is tracked in [#35](https://github.com/QuantEcon/data-lectures/issues/35) with alternatives, and resolved before `data.quantecon.org` is promoted as a public open-data host. That promotion is the gate, not each file's move. This generalises the exception AGENTS.md already carried for `countries.csv`, and applies to **inherited** data only — a genuinely new dataset still establishes its licence before it lands + **Licensing does not gate migration** (settled 2026-08-06, [#35](https://github.com/QuantEcon/data-lectures/issues/35)). Inherited data — anything the lecture repos already serve publicly — migrates with its licence recorded **as found**, including `redistribution: restricted` and `name: null` where that is the honest answer. Moving the same bytes to a canonical host with better provenance and an explicit licence field improves on the status quo, so the migration does not wait on review; what needs further thought is tracked in [#35](https://github.com/QuantEcon/data-lectures/issues/35) with alternatives, and resolved before this repo is ever promoted as a branded public open-data host. That promotion is the gate, not each file's move — and with the custom domain deferred indefinitely (2026-08-12, D11), no such promotion is scheduled: the #35 inventory stays open and the gate binds only if a public host is someday established after all. This generalises the exception AGENTS.md already carried for `countries.csv`, and applies to **inherited** data only — a genuinely new dataset still establishes its licence before it lands - [x] Keep-or-drop decision for the files with no consumer anywhere — **dropped 2026-07-16** in the Phase 2 restructure, rather than promoting them into the published namespace: - `GDP_per_capita_world_bank.csv` and `Metadata_Country_API_NY.GDP.PCAP.CD_DS2_en_csv_v2_4770417.csv` — an org-wide code search returns **zero** references to either, they are freely re-downloadable from the World Bank, and their licence was never established. Rehosting a stale snapshot nobody reads is the opposite of this repo's purpose - `fig_3.ods` — confirmed to carry no provenance the published `.xlsx` lacks: both parse to a single `Sheet1` of identical shape (34×6) and `DataFrame.equals` returns true, so it is a pure format twin @@ -326,7 +326,7 @@ The first end-to-end deployment: one dataset per hosting pattern, each the harde | Decision | Current strawman | | --- | --- | | Repo name | **settled 2026-07-16**: renamed `data-lectures` (Phase 1) | -| URL form | `data.quantecon.org/lectures/...`; interim `github.com/QuantEcon/data-lectures/raw/main/...` | +| URL form | **settled 2026-08-12** (D11): lecture code reads `qeld.url('')`; the direct forms are the runtime-dependent raw URLs (repoint rule 5), standing rather than interim. `data.quantecon.org` deferred indefinitely ([#37](https://github.com/QuantEcon/data-lectures/issues/37)) | | Layout | flat | | Licensing review | per-source cache-and-serve-with-attribution gate (`redistribution: permitted \| restricted`), recorded in the manifest — this repo is a stability cache, not a content host | diff --git a/README.md b/README.md index 73d28e6..d449334 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,9 @@ The canonical repository for **data consumed by the QuantEcon lecture series**, ## Referencing data -Until the `data.quantecon.org` Pages deployment is live, use the interim form. **There is no single safe form — it depends on the consumer's runtime** (repoint rule 5): +The stable consumer interface is the **`qeld` package** ([`PLAN-QELD-PACKAGE.md`](PLAN-QELD-PACKAGE.md) — designed, not yet shipped): lecture code reads `qeld.url('')` in place of a URL literal, which resolves to the context-correct direct form below and keeps the URL one `print(url)` away. There is no pending host cutover — the `data.quantecon.org` custom domain was **deferred indefinitely on 2026-08-12** in favor of `qeld` (D11, [#37](https://github.com/QuantEcon/data-lectures/issues/37)), so the direct forms below are standing, not interim. + +Until `qeld` ships (and permanently for prose links, `{download}` targets, and reads whose URL is the lesson), use the direct form. **There is no single safe form — it depends on the consumer's runtime** (repoint rule 5): | Consumer | Use | | --- | --- | @@ -21,12 +23,6 @@ Until the `data.quantecon.org` Pages deployment is live, use the interim form. * The `github.com/…/raw/` form is a 302 whose response carries an **empty** `access-control-allow-origin`, so a browser rejects it before following the redirect. The strict audit fails on any `lecture-wasm` code-cell read that uses it. `{download}` targets and prose links are plain navigations, so any resolving form is fine there. -Once publishing lands (PLAN Phase 4), the canonical form becomes: - -``` -https://data.quantecon.org/lectures/ -``` - **Never** use `media.githubusercontent.com`. It is the LFS media endpoint and routes per path, so it 404s every file this repo publishes — `lectures/` is 100% plain git, and LFS is confined to `sources/`, which is never served ([#58](https://github.com/QuantEcon/data-lectures/issues/58)). Never pin a branch other than `main`. ## Adding a dataset @@ -34,7 +30,7 @@ https://data.quantecon.org/lectures/ 1. Confirm the license permits redistribution. 2. Classify it: **verbatim** (third-party file as distributed), **constructed** (built by our processing — commit the builder too), or **dynamic snapshot** (tracks a moving source — builder plus refresh cadence). 3. Open a PR with the file, its manifest, and any builder. -4. Reference it from the lecture by the canonical URL — the lecture PR builds green immediately, no two-step merge. +4. Reference it from the lecture — `qeld.url('')` once the package ships, the runtime-correct direct URL until then. The lecture PR builds green immediately, no two-step merge. 5. Add the lecture to the dataset's `consumers` list. See the [draft convention](https://github.com/QuantEcon/QuantEcon.manual/pull/108) for the full checklist and manifest schema. @@ -48,9 +44,10 @@ See the [draft convention](https://github.com/QuantEcon/QuantEcon.manual/pull/10 | `manifest-schema.yml` | the per-dataset manifest schema (strawman — see [`PLAN.md`](PLAN.md) Phase 2) | no | | `migration.yml` | the migration lifecycle tracker — which PRs landed and repointed each dataset (transitional; archivable when the migration programme completes) | rendered | -The tree is flat because the URL is the interface: `lectures/` maps to -`data.quantecon.org/lectures/`, so a file can never be re-filed under a -new owner and break its consumers. Anything outside `lectures/` is not served. +The tree is flat because the filename is the interface: `lectures/` is +the served URL's last segment and the `qeld` key (`qeld.url('')`), so a +file can never be re-filed under a new owner and break its consumers. Anything +outside `lectures/` is not served. ## The audit dashboard diff --git a/migration.yml b/migration.yml index 81c9134..4771219 100644 --- a/migration.yml +++ b/migration.yml @@ -14,9 +14,13 @@ # Lifecycle (status): # pending identified for migration; nothing landed here yet # landed file + manifest merged in data-lectures; consumers unchanged -# repointed every consumer reads this repo's interim raw URL -# final every consumer reads https://data.quantecon.org/lectures/… -# (gated on the Phase 4 DNS work — data-lectures#15) +# repointed every consumer reads this repo's raw URL directly +# final every consumer code read resolves through qeld.url(), with the +# PLAN-QELD-PACKAGE.md section 4.1 carve-outs terminal on the direct +# form. (Redefined 2026-08-12, D11 — this status previously meant the +# data.quantecon.org address, a cutover retired in favor of qeld; +# see data-lectures#37 and #15. The audit learns the qeld pattern +# in Q1, before any consumer adopts it.) datasets: lingcod_msy_recovery.csv: diff --git a/scripts/build_catalog.py b/scripts/build_catalog.py index b7e6469..a2c6e44 100644 --- a/scripts/build_catalog.py +++ b/scripts/build_catalog.py @@ -26,7 +26,9 @@ LECTURES = REPO / "lectures" OUT = REPO / "CATALOG.md" -# Interim URL form (AGENTS.md); swaps to data.quantecon.org/lectures/ at Phase 4. +# The standing direct URL form (AGENTS.md). Catalog links are navigations, so the +# 302-tolerant form is fine; the data.quantecon.org swap was retired 2026-08-12 +# in favor of qeld (PLAN-QELD-PACKAGE.md D11). RAW = "https://github.com/QuantEcon/data-lectures/raw/main/lectures" diff --git a/scripts/render_audit.py b/scripts/render_audit.py index ee0c677..91d2081 100644 --- a/scripts/render_audit.py +++ b/scripts/render_audit.py @@ -649,8 +649,8 @@ def stepper(fname: str, rec: dict, verified: bool) -> str: "landed": pr_link(str((rec.get("landed") or {}).get("pr", ""))) + f'
{esc((rec.get("landed") or {}).get("date", ""))}', "repointed": "
".join(pr_link(str(r.get("pr", ""))) for r in rec.get("repoints") or []), - "final": ("awaits " + issue_link("QuantEcon/data-lectures#15")) if status != "final" - else esc((rec.get("cutover") or {}).get("date", "")), + "final": ("awaits qeld adoption" if status != "final" + else esc((rec.get("cutover") or {}).get("date", ""))), } steps = "" for i, name in enumerate(STATUS_STEPS): @@ -667,7 +667,7 @@ def stepper(fname: str, rec: dict, verified: bool) -> str: def milestones(audit: dict) -> str: """The migration programme as reader-facing milestones — completed waves (derived from migration.yml records), upcoming waves, the broad sweep, and - the final-URL switch. This section is what lets the rest of the dashboard + the qeld adoption milestone. This section is what lets the rest of the dashboard stay plan-agnostic: wave codes like P3 mean something only because they are presented here.""" mig = audit["migration"] or {} @@ -716,11 +716,13 @@ def milestones(audit: dict) -> str: per consuming lecture repo.

-○ Final URLs — serve everything from data.quantecon.org -

Migrated lectures currently read this repository's GitHub URL. Once the custom domain -is live, every migrated dataset switches to its permanent -data.quantecon.org/lectures/… address in a single sweep -({issue_link("QuantEcon/data-lectures#15")}). No dataset has made this step yet.

+○ Stable interface — lecture code reads qeld.url('<file>') +

Migrated lectures currently read this repository's GitHub URL directly. Once the +qeld package ships, each code read is switched to a qeld.url() +call that resolves to the same URL — one constant interface point that survives any +future backend rework. (This milestone replaced the data.quantecon.org +URL sweep on 2026-08-12 — {issue_link("QuantEcon/data-lectures#37")}, +{issue_link("QuantEcon/data-lectures#15")}.) No dataset has made this step yet.

""" return f""" @@ -791,7 +793,7 @@ def render_migration(audit: dict) -> str: landed — the file and its metadata are merged into the central repo; lectures unchanged. repointed — every consuming lecture now reads the central copy (shown as ✓ migrated in the table below). -final — the lecture reads the permanent data.quantecon.org address +final — every code read resolves through the qeld package (the last milestone below).

{consistency} {series_manifest(audit)}