diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..755cf9e --- /dev/null +++ b/.gitattributes @@ -0,0 +1,15 @@ +# LFS here is per-path, opt-in, and scoped to builder inputs — never a blanket +# rule like `high_dim_data`'s `*.csv` + `*.dta`, which is what puts every one of +# its consumers on the media host. +# +# `lectures/` is the published tree and stays 100% plain git. An LFS-tracked +# path returns HTTP 200 from raw.githubusercontent.com with ~130 bytes of +# pointer text, so a reader gets a parse error rather than a 404 and a +# status-code check reads as green — the failure is silent at both ends. +# +# `sources/` holds builder inputs, is never served, and is the only place an +# LFS object belongs here. Its README is the audit trail for what lives there, +# so it stays plain text. + +sources/** filter=lfs diff=lfs merge=lfs -text +sources/README.md !filter !diff !merge text diff --git a/.github/workflows/audit-dashboard.yml b/.github/workflows/audit-dashboard.yml index bc70a46..f4879e6 100644 --- a/.github/workflows/audit-dashboard.yml +++ b/.github/workflows/audit-dashboard.yml @@ -40,7 +40,15 @@ jobs: steps: - uses: actions/checkout@v4 with: - lfs: true # published files must be bytes, never LFS pointers + # Nothing under `lectures/` may be an LFS object (.gitattributes + # scopes LFS to `sources/**`), and this job publishes `lectures/` to + # Pages. `lfs: false` is what makes a mistake visible: an + # accidentally-tracked file is deployed as its pointer, which is the + # same bytes raw.githubusercontent.com would serve. Fetching the real + # bytes here would publish a file that reads correctly from Pages and + # as pointer text everywhere else. It also keeps `sources/` — a 99 MiB + # LFS object — off every run of this workflow. + lfs: false - uses: actions/setup-python@v5 with: python-version: "3.12" diff --git a/.github/workflows/consumed-file-check.yml b/.github/workflows/consumed-file-check.yml index 3c55a38..5b68704 100644 --- a/.github/workflows/consumed-file-check.yml +++ b/.github/workflows/consumed-file-check.yml @@ -15,11 +15,15 @@ jobs: steps: - uses: actions/checkout@v4 with: - # No LFS-tracked files today (PLAN Phase 3). Set proactively: if a - # consumed file is ever LFS-tracked, this job must hash the bytes, - # not the pointer — a pointer hash would fail loudly here, which is - # the correct behaviour, but only with lfs available to fetch. - lfs: true + # `lfs: false` is the assertion, not a saving. A manifest records the + # sha256 of the real bytes, so if a `lectures/` file is ever + # LFS-tracked by mistake this job hashes the pointer, mismatches, and + # goes red — which is exactly what a reader would get from + # raw.githubusercontent.com. Fetching the real bytes would hash them + # correctly and pass green while every consumer downloads ~130 bytes + # of pointer text. LFS belongs to `sources/**` only (.gitattributes), + # and nothing here reads `sources/`. + lfs: false - uses: actions/setup-python@v5 with: python-version: "3.12" diff --git a/AGENTS.md b/AGENTS.md index 39ef4d9..47a1475 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -84,7 +84,8 @@ Rules that still apply: - LFS is **per-path**, opt-in, large binaries only. Never a blanket rule like `high_dim_data`'s `*.csv` **and** `*.dta`. - Do not LFS-track an **existing** file until you've confirmed no consumer fetches it via `raw.githubusercontent.com` — converting silently turns their download into pointer text. - A builder must read its input from `sources/`, never over the network from another QuantEcon repo. That is how a retired repo becomes load-bearing again. -- **Two** workflows check this repo out, and both say `lfs: true` today — `.github/workflows/audit-dashboard.yml:43` (the Pages deploy) and `.github/workflows/consumed-file-check.yml:22` (every pull request). Both must become `lfs: false` while nothing published is an LFS object, and both must go back to `lfs: true` the moment anything under `lectures/` does, or Pages publishes pointer files. LFS bandwidth is an org-wide quota, so leaving them `true` once `sources/` holds a 99 MiB object spends it on every PR. +- **Two** workflows check this repo out, and both now say `lfs: false` — `.github/workflows/audit-dashboard.yml` (the Pages deploy) and `.github/workflows/consumed-file-check.yml` (every pull request). Leave them that way: `lfs: false` is the assertion that nothing published is an LFS object. If a `lectures/` file is ever tracked by mistake, the checker hashes the pointer and goes red, and Pages deploys the same pointer bytes a reader would get from `raw.githubusercontent.com` — whereas `lfs: true` fetches the real bytes, passes green, and publishes a file that works only from Pages. It also keeps `sources/` (a 99 MiB LFS object) off every run; LFS bandwidth is an org-wide quota. +- `git check-attr filter -- sources/` must print `filter: lfs` **before** you `git add` anything to `sources/`. `SCF_plus.dta` is 103,934,093 B against GitHub's 104,857,600 B hard limit, so a mis-scoped rule does not error — the push succeeds as plain git and the blob is in history permanently. ### Dynamic builders diff --git a/PLAN.md b/PLAN.md index ac6fc27..c805558 100644 --- a/PLAN.md +++ b/PLAN.md @@ -40,7 +40,7 @@ This repository is being shaped into the **single canonical repository for data - **Flat published tree** served at `https://data.quantecon.org/lectures/` via GitHub Pages (custom domain), CORS-open for pyodide/JupyterLite - Every dataset classified **verbatim / constructed / dynamic snapshot**, each with a manifest — authoritative field reference in `manifest-schema.yml`, as revised by the P1 pilot (`integrity`, `builder_status`, `known_nulls` and `license.verified` joined the original sketch of `source` / `license` / `retrieved` / `schema` / `consumers` / `maintainer` / `cadence`) - Constructed and dynamic datasets ship their **builder**; dynamic datasets get **scheduled refresh-as-PR** plus a weekly **sources-alive canary** -- Per-path LFS for large binaries only; storage choice invisible to consumers because URLs decouple from hosting +- The published tree is **100% plain git**; per-path LFS is confined to `sources/`, which is never served. Storage does **not** decouple from hosting — the URL a consumer must write is a function of how the file is stored, and there is no browser-safe form invariant under a storage flip (repoint rule 6, and [#58](https://github.com/QuantEcon/data-lectures/issues/58) for the ladder above 100 MiB) ## Repoint rules