From d9297db57eb454843916acf0c343344521e1b3cf Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Mon, 10 Aug 2026 08:56:05 +1000 Subject: [PATCH 1/2] LFS: scope it to sources/**, and stop fetching it in CI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Infra only — no data moves here. This is the groundwork the high_dim_data fold needs, landed on its own so the fold PR is a data diff. `.gitattributes` did not exist, so `sources/` had no rule waiting for it and `lectures/` had no rule keeping it plain. LFS is now scoped to `sources/**`, with `sources/README.md` excluded so the audit trail stays readable text. Never a blanket rule like high_dim_data's `*.csv` + `*.dta` — that is what puts all of its consumers on the media host. Both workflow checkouts go `lfs: true` -> `lfs: false`, and the comments explaining them were inverted. `lfs: false` is the assertion, not a saving: a manifest records the sha256 of the real bytes, so an accidentally LFS-tracked file under `lectures/` gets hashed as its pointer, mismatches, and goes red — the same bytes a reader would get from raw.githubusercontent.com. With `lfs: true` the checker fetches the real bytes, hashes them correctly, passes green, and Pages publishes a file that resolves only from Pages. It also keeps a 99 MiB object out of every PR run once `sources/` is populated. Verified: `git check-attr filter` prints `lfs` for sources/SCF_plus.dta and `unspecified` for sources/README.md and for lectures/ paths; both workflows still parse and now report lfs=False. AGENTS.md's LFS bullet described the pre-change state and prescribed this change, so it is rewritten here rather than left to go stale, plus a new line for the trap this opens: SCF_plus.dta is 923,507 B under GitHub's hard blob limit, so a mis-scoped rule does not error — the push just silently succeeds as plain git. Check `git check-attr` before adding to sources/. PR A of QuantEcon/workspace-lectures#23 step 3. Part of #1. Co-Authored-By: Claude Opus 5 (1M context) --- .gitattributes | 15 +++++++++++++++ .github/workflows/audit-dashboard.yml | 10 +++++++++- .github/workflows/consumed-file-check.yml | 14 +++++++++----- AGENTS.md | 3 ++- 4 files changed, 35 insertions(+), 7 deletions(-) create mode 100644 .gitattributes diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..755cf9e --- /dev/null +++ b/.gitattributes @@ -0,0 +1,15 @@ +# LFS here is per-path, opt-in, and scoped to builder inputs — never a blanket +# rule like `high_dim_data`'s `*.csv` + `*.dta`, which is what puts every one of +# its consumers on the media host. +# +# `lectures/` is the published tree and stays 100% plain git. An LFS-tracked +# path returns HTTP 200 from raw.githubusercontent.com with ~130 bytes of +# pointer text, so a reader gets a parse error rather than a 404 and a +# status-code check reads as green — the failure is silent at both ends. +# +# `sources/` holds builder inputs, is never served, and is the only place an +# LFS object belongs here. Its README is the audit trail for what lives there, +# so it stays plain text. + +sources/** filter=lfs diff=lfs merge=lfs -text +sources/README.md !filter !diff !merge text diff --git a/.github/workflows/audit-dashboard.yml b/.github/workflows/audit-dashboard.yml index bc70a46..f4879e6 100644 --- a/.github/workflows/audit-dashboard.yml +++ b/.github/workflows/audit-dashboard.yml @@ -40,7 +40,15 @@ jobs: steps: - uses: actions/checkout@v4 with: - lfs: true # published files must be bytes, never LFS pointers + # Nothing under `lectures/` may be an LFS object (.gitattributes + # scopes LFS to `sources/**`), and this job publishes `lectures/` to + # Pages. `lfs: false` is what makes a mistake visible: an + # accidentally-tracked file is deployed as its pointer, which is the + # same bytes raw.githubusercontent.com would serve. Fetching the real + # bytes here would publish a file that reads correctly from Pages and + # as pointer text everywhere else. It also keeps `sources/` — a 99 MiB + # LFS object — off every run of this workflow. + lfs: false - uses: actions/setup-python@v5 with: python-version: "3.12" diff --git a/.github/workflows/consumed-file-check.yml b/.github/workflows/consumed-file-check.yml index 3c55a38..5b68704 100644 --- a/.github/workflows/consumed-file-check.yml +++ b/.github/workflows/consumed-file-check.yml @@ -15,11 +15,15 @@ jobs: steps: - uses: actions/checkout@v4 with: - # No LFS-tracked files today (PLAN Phase 3). Set proactively: if a - # consumed file is ever LFS-tracked, this job must hash the bytes, - # not the pointer — a pointer hash would fail loudly here, which is - # the correct behaviour, but only with lfs available to fetch. - lfs: true + # `lfs: false` is the assertion, not a saving. A manifest records the + # sha256 of the real bytes, so if a `lectures/` file is ever + # LFS-tracked by mistake this job hashes the pointer, mismatches, and + # goes red — which is exactly what a reader would get from + # raw.githubusercontent.com. Fetching the real bytes would hash them + # correctly and pass green while every consumer downloads ~130 bytes + # of pointer text. LFS belongs to `sources/**` only (.gitattributes), + # and nothing here reads `sources/`. + lfs: false - uses: actions/setup-python@v5 with: python-version: "3.12" diff --git a/AGENTS.md b/AGENTS.md index 39ef4d9..47a1475 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -84,7 +84,8 @@ Rules that still apply: - LFS is **per-path**, opt-in, large binaries only. Never a blanket rule like `high_dim_data`'s `*.csv` **and** `*.dta`. - Do not LFS-track an **existing** file until you've confirmed no consumer fetches it via `raw.githubusercontent.com` — converting silently turns their download into pointer text. - A builder must read its input from `sources/`, never over the network from another QuantEcon repo. That is how a retired repo becomes load-bearing again. -- **Two** workflows check this repo out, and both say `lfs: true` today — `.github/workflows/audit-dashboard.yml:43` (the Pages deploy) and `.github/workflows/consumed-file-check.yml:22` (every pull request). Both must become `lfs: false` while nothing published is an LFS object, and both must go back to `lfs: true` the moment anything under `lectures/` does, or Pages publishes pointer files. LFS bandwidth is an org-wide quota, so leaving them `true` once `sources/` holds a 99 MiB object spends it on every PR. +- **Two** workflows check this repo out, and both now say `lfs: false` — `.github/workflows/audit-dashboard.yml` (the Pages deploy) and `.github/workflows/consumed-file-check.yml` (every pull request). Leave them that way: `lfs: false` is the assertion that nothing published is an LFS object. If a `lectures/` file is ever tracked by mistake, the checker hashes the pointer and goes red, and Pages deploys the same pointer bytes a reader would get from `raw.githubusercontent.com` — whereas `lfs: true` fetches the real bytes, passes green, and publishes a file that works only from Pages. It also keeps `sources/` (a 99 MiB LFS object) off every run; LFS bandwidth is an org-wide quota. +- `git check-attr filter -- sources/` must print `filter: lfs` **before** you `git add` anything to `sources/`. `SCF_plus.dta` is 103,934,093 B against GitHub's 104,857,600 B hard limit, so a mis-scoped rule does not error — the push succeeds as plain git and the blob is in history permanently. ### Dynamic builders From 34354848db84d4c59e1d4df2e3c4b4fc49a40f13 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Mon, 10 Aug 2026 09:56:02 +1000 Subject: [PATCH 2/2] PLAN: drop the line that says storage is invisible to consumers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Per-path LFS for large binaries only; storage choice invisible to consumers because URLs decouple from hosting" is the position repoint rule 6 overturned on 2026-08-06, still sitting in the plan's own summary section — contradicted by rule 6 itself, by the settled-decision note further down, and by AGENTS.md. It is also the sentence a future session would cite to put an LFS object back into lectures/, which is the one thing this PR exists to prevent. Measured: `https://github.com///raw//` — the form AGENTS.md calls storage-agnostic — is a 302 whose Location is raw.githubusercontent.com for a plain-git path and media.githubusercontent.com for an LFS path, and whose access-control-allow-origin header is present but EMPTY. So it is storage-agnostic only because the server resolves storage on the author's behalf, and it is precisely the form a browser rejects. For a wasm consumer there is no URL form invariant under a storage flip. Reasoning and the >100 MiB ladder recorded in #58. Co-Authored-By: Claude Opus 5 (1M context) --- PLAN.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PLAN.md b/PLAN.md index ac6fc27..c805558 100644 --- a/PLAN.md +++ b/PLAN.md @@ -40,7 +40,7 @@ This repository is being shaped into the **single canonical repository for data - **Flat published tree** served at `https://data.quantecon.org/lectures/` via GitHub Pages (custom domain), CORS-open for pyodide/JupyterLite - Every dataset classified **verbatim / constructed / dynamic snapshot**, each with a manifest — authoritative field reference in `manifest-schema.yml`, as revised by the P1 pilot (`integrity`, `builder_status`, `known_nulls` and `license.verified` joined the original sketch of `source` / `license` / `retrieved` / `schema` / `consumers` / `maintainer` / `cadence`) - Constructed and dynamic datasets ship their **builder**; dynamic datasets get **scheduled refresh-as-PR** plus a weekly **sources-alive canary** -- Per-path LFS for large binaries only; storage choice invisible to consumers because URLs decouple from hosting +- The published tree is **100% plain git**; per-path LFS is confined to `sources/`, which is never served. Storage does **not** decouple from hosting — the URL a consumer must write is a function of how the file is stored, and there is no browser-safe form invariant under a storage flip (repoint rule 6, and [#58](https://github.com/QuantEcon/data-lectures/issues/58) for the ladder above 100 MiB) ## Repoint rules