Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# LFS here is per-path, opt-in, and scoped to builder inputs — never a blanket
# rule like `high_dim_data`'s `*.csv` + `*.dta`, which is what puts every one of
# its consumers on the media host.
#
# `lectures/` is the published tree and stays 100% plain git. An LFS-tracked
# path returns HTTP 200 from raw.githubusercontent.com with ~130 bytes of
# pointer text, so a reader gets a parse error rather than a 404 and a
# status-code check reads as green — the failure is silent at both ends.
#
# `sources/` holds builder inputs, is never served, and is the only place an
# LFS object belongs here. Its README is the audit trail for what lives there,
# so it stays plain text.

sources/** filter=lfs diff=lfs merge=lfs -text
sources/README.md !filter !diff !merge text
10 changes: 9 additions & 1 deletion .github/workflows/audit-dashboard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,15 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
lfs: true # published files must be bytes, never LFS pointers
# Nothing under `lectures/` may be an LFS object (.gitattributes
# scopes LFS to `sources/**`), and this job publishes `lectures/` to
# Pages. `lfs: false` is what makes a mistake visible: an
# accidentally-tracked file is deployed as its pointer, which is the
# same bytes raw.githubusercontent.com would serve. Fetching the real
# bytes here would publish a file that reads correctly from Pages and
# as pointer text everywhere else. It also keeps `sources/` — a 99 MiB
# LFS object — off every run of this workflow.
lfs: false
- uses: actions/setup-python@v5
with:
python-version: "3.12"
Expand Down
14 changes: 9 additions & 5 deletions .github/workflows/consumed-file-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,15 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
# No LFS-tracked files today (PLAN Phase 3). Set proactively: if a
# consumed file is ever LFS-tracked, this job must hash the bytes,
# not the pointer — a pointer hash would fail loudly here, which is
# the correct behaviour, but only with lfs available to fetch.
lfs: true
# `lfs: false` is the assertion, not a saving. A manifest records the
# sha256 of the real bytes, so if a `lectures/` file is ever
# LFS-tracked by mistake this job hashes the pointer, mismatches, and
# goes red — which is exactly what a reader would get from
# raw.githubusercontent.com. Fetching the real bytes would hash them
# correctly and pass green while every consumer downloads ~130 bytes
# of pointer text. LFS belongs to `sources/**` only (.gitattributes),
# and nothing here reads `sources/`.
lfs: false
- uses: actions/setup-python@v5
with:
python-version: "3.12"
Expand Down
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,8 @@ Rules that still apply:
- LFS is **per-path**, opt-in, large binaries only. Never a blanket rule like `high_dim_data`'s `*.csv` **and** `*.dta`.
- Do not LFS-track an **existing** file until you've confirmed no consumer fetches it via `raw.githubusercontent.com` — converting silently turns their download into pointer text.
- A builder must read its input from `sources/`, never over the network from another QuantEcon repo. That is how a retired repo becomes load-bearing again.
- **Two** workflows check this repo out, and both say `lfs: true` today — `.github/workflows/audit-dashboard.yml:43` (the Pages deploy) and `.github/workflows/consumed-file-check.yml:22` (every pull request). Both must become `lfs: false` while nothing published is an LFS object, and both must go back to `lfs: true` the moment anything under `lectures/` does, or Pages publishes pointer files. LFS bandwidth is an org-wide quota, so leaving them `true` once `sources/` holds a 99 MiB object spends it on every PR.
- **Two** workflows check this repo out, and both now say `lfs: false` — `.github/workflows/audit-dashboard.yml` (the Pages deploy) and `.github/workflows/consumed-file-check.yml` (every pull request). Leave them that way: `lfs: false` is the assertion that nothing published is an LFS object. If a `lectures/` file is ever tracked by mistake, the checker hashes the pointer and goes red, and Pages deploys the same pointer bytes a reader would get from `raw.githubusercontent.com` — whereas `lfs: true` fetches the real bytes, passes green, and publishes a file that works only from Pages. It also keeps `sources/` (a 99 MiB LFS object) off every run; LFS bandwidth is an org-wide quota.
- `git check-attr filter -- sources/<file>` must print `filter: lfs` **before** you `git add` anything to `sources/`. `SCF_plus.dta` is 103,934,093 B against GitHub's 104,857,600 B hard limit, so a mis-scoped rule does not error — the push succeeds as plain git and the blob is in history permanently.

### Dynamic builders

Expand Down
2 changes: 1 addition & 1 deletion PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ This repository is being shaped into the **single canonical repository for data
- **Flat published tree** served at `https://data.quantecon.org/lectures/<filename>` via GitHub Pages (custom domain), CORS-open for pyodide/JupyterLite
- Every dataset classified **verbatim / constructed / dynamic snapshot**, each with a manifest — authoritative field reference in `manifest-schema.yml`, as revised by the P1 pilot (`integrity`, `builder_status`, `known_nulls` and `license.verified` joined the original sketch of `source` / `license` / `retrieved` / `schema` / `consumers` / `maintainer` / `cadence`)
- Constructed and dynamic datasets ship their **builder**; dynamic datasets get **scheduled refresh-as-PR** plus a weekly **sources-alive canary**
- Per-path LFS for large binaries only; storage choice invisible to consumers because URLs decouple from hosting
- The published tree is **100% plain git**; per-path LFS is confined to `sources/`, which is never served. Storage does **not** decouple from hosting — the URL a consumer must write is a function of how the file is stored, and there is no browser-safe form invariant under a storage flip (repoint rule 6, and [#58](https://github.com/QuantEcon/data-lectures/issues/58) for the ladder above 100 MiB)

## Repoint rules

Expand Down
Loading