feat: v202608 release

.env.example

@@ -1,5 +1,9 @@
 HARVARD_DATAVERSE_GUESTBOOK_JSON = '{"email":Replace with your email address,"institution":Replace with your institution}'
+# INGEST_ROOT and SCRATCH_ROOT take any fsspec URI. A local root must be an absolute path
+# that already exists; a gs:// root needs application-default credentials.
+# INGEST_ROOT = '/absolute/path/to/ingest'
 INGEST_ROOT = 'gs://cornerstone-ingest-us-central1'
 NUMBER_OF_DASK_WORKERS = 12
+# SCRATCH_ROOT = '/absolute/path/to/scratch'
 SCRATCH_ROOT = 'gs://cornerstone-scratch-us-central1'
 USDA_NASS_API_KEY = 'Replace me with value from https://quickstats.nass.usda.gov/api'

.github/workflows/ci.yml

@@ -12,8 +12,6 @@ jobs:
     steps:
       - uses: actions/checkout@v7
       - uses: astral-sh/setup-uv@v8.3.2
-        with:
-          enable-cache: true
       - name: Install dependencies
         run: uv sync
       - uses: actions/cache@v6
@@ -28,9 +26,7 @@ jobs:
     steps:
       - uses: actions/checkout@v7
       - uses: astral-sh/setup-uv@v8.3.2
-        with:
-          enable-cache: true
       - name: Install dependencies
         run: uv sync
       - name: pytest (unit only)
-        run: uv run pytest jdluc/__tests__/
+        run: uv run pytest

.gitignore

@@ -13,3 +13,5 @@ __pycache__/
 
 # Env
 .env
+
+.cache/

.pre-commit-config.yaml

@@ -18,19 +18,30 @@ repos:
       - id: mypy
         name: mypy
         language: system
-        entry: uv run mypy jdluc
+        entry: uv run mypy jdluc validation
         pass_filenames: false
   - repo: https://github.com/astral-sh/uv-pre-commit
     rev: 0.11.26
     hooks:
       - id: uv-lock
+  - repo: https://github.com/hukkin/mdformat
+    rev: 1.0.0
+    hooks:
+      - id: mdformat
+        additional_dependencies:
+          - mdformat-gfm
+        entry: mdformat --number --wrap=no
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v6.0.0
     hooks:
       - id: check-added-large-files
+      - id: check-ast
+      - id: check-json
       - id: check-merge-conflict
-      - id: check-yaml
       - id: check-toml
+      - id: check-yaml
       - id: debug-statements
       - id: end-of-file-fixer
+      - id: pretty-format-json
+        args: [--autofix]
       - id: trailing-whitespace

ATTRIBUTION.md

@@ -1,4 +1,4 @@
-Attribute all emissions factors or other data built on this pipeline as: "Cornerstone Sustainability Data Initiative, http://doi.org/DOI", using the DOI for the data version. If no published DOI exists, link to the relevant code release on GitHub, or simply https://github.com/cornerstone-data/jdluc/.
+Attribute all emissions factors or other data built on this pipeline as: "Cornerstone Sustainability Data Initiative, http://doi.org/DOI", using the DOI for the data version. If no published DOI exists, link to the relevant code release on GitHub, or simply https://github.com/cornerstone-data/luc/.
 
 You must prominently display this attribution in public-facing models, tools, datasets or any other applications that are generated from this code. This includes reimplementations of the code that are built primarily by passing the methodology and/or technical specs to a coding agent.
 

LICENSE.txt

@@ -7,7 +7,7 @@ License
 The licensor licenses you to do everything with this software that would otherwise infringe the licensor’s copyright in it, or any patent claims the licensor can license or becomes able to license. If you make any written claim that the software infringes or contributes to infringement of any patent, your license ends immediately.
 
 Notices on Software
-If you provide a copy of the software to anyone else, or provide or display Produced Data to anyone else, you must provide a copy of this license or a link to https://github.com/cornerstone-data/jdluc/blob/main/LICENSE.txt.
+If you provide a copy of the software to anyone else, or provide or display Produced Data to anyone else, you must provide a copy of this license or a link to https://github.com/cornerstone-data/luc/blob/main/LICENSE.txt.
 
 Notices on Produced Data
 If you calculate data (including summaries or reports of data) using this software, you must deliver an attribution, in a form to be specified by the licensor, in a reasonable and conspicuous manner to anyone to whom you deliver or display the data.

README.md

@@ -1,13 +1,10 @@
-# Cornerstone jdLUC
+# Cornerstone LUC
 
 This repo contains an experimental methodology and data pipeline for estimating the land use change (LUC) related emissions associated with agricultural commodities. This methodology allocates LUC emissions to crops in proportion to their displacement of natural ecosystems, based on high-resolution satellite imagery. It's primarily intended for use in corporate GHG inventories, and follows the new GHGP Land Sector and Removals Standard. It supports two attribution methodologies over a shared per-pixel emissions core: the high-resolution "jurisdictional direct land use change" (jdLUC) calculation where detailed crop maps exist, and a coarser-resolution statistical approach for regions where they don't.
 
-As a proof of concept, the jurisdictional-direct leg focuses on the main row crops grown in the United States (corn, soy, and wheat). Although U.S. land use change emissions are relatively modest contributors to global totals, the U.S. agricultural sector is well studied and has strong data infrastructure, which makes it a good place to start testing methods. The statistical leg extends the same per-pixel emissions core to global crop coverage, using coarser sub-national statistics (IFPRI MapSPAM) where high-resolution crop maps aren't available.
+As a proof of concept, the jurisdictional-direct leg covers eleven crops grown in the United States (barley, dry beans, cotton, maize, potato, rice, sorghum, soybean, sugarbeet, sugarcane and wheat). Although U.S. land use change emissions are relatively modest contributors to global totals, the U.S. agricultural sector is well studied and has strong data infrastructure, which makes it a good place to start testing methods. The statistical leg extends the same per-pixel emissions core to global crop coverage, using coarser sub-national statistics (IFPRI MapSPAM) where high-resolution crop maps aren't available.
 
-> **Assessing LSRS conformance?** The [executive summary](docs/executive_summary.md) gives a
-> concise account of how this methodology maps to the GHGP Land Sector and Removals Standard —
-> which requirements it meets, the key modeling choices behind them, and where it deviates or
-> remains a work in progress.
+> **Assessing LSRS conformance?** The [executive summary](docs/executive_summary.md) gives a concise account of how this methodology maps to the GHGP Land Sector and Removals Standard — which requirements it meets, the key modeling choices behind them, and where it deviates or remains a work in progress.
 
 ![Land conversion and soy expansion drive LUC emissions in Matopiba, Brazil](docs/figures/soy-brazil-matopiba-methodology.png)
 
@@ -25,49 +22,35 @@ The methodology and technical decisions in this repo are intended as a starting
 
 ## Data access
 
-The pipeline publishes three artifacts — see `docs/data.md` for each artifact's full schema, grid, and variable/column reference.
-
-Harmonized inputs and per-pixel emissions (zarr):
+The pipeline's published artifact is the emissions-factor table: one `emissions-factors.parquet` per data version deposited in a public data archive that mints a DOI per version. The archive is still being set up. Each version's git tag names the commit that produced the table.
 
 ```python
->>> import xarray
->>> harmonized = xarray.open_zarr("gs://cornerstone-luc/v3-jdluc-sluc-south-america/harmonize.zarr", consolidated=False)
->>> emissions = xarray.open_zarr("gs://cornerstone-luc/v3-jdluc-sluc-south-america/emit.zarr", consolidated=False)
+>>> import pandas
+>>> emission_factors = pandas.read_parquet("emissions-factors.parquet")
 ```
 
-Emissions factors (parquet):
+See `docs/data.md` for the full column reference and `docs/coverage.md` for the countries and crops a version covers.
 
-```python
->>> import pandas
->>> emission_factors = pandas.read_parquet("gs://cornerstone-luc/v3-jdluc-sluc-south-america/emissions-factors.parquet")
-```
+The harmonized inputs and per-pixel emissions behind that table are pipeline outputs rather than published artifacts — zarr stores far too large to deposit alongside it. Reproduce them with the steps under [Running the pipeline](#running-the-pipeline); sharing the underlying maps is open work.
 
 The data is licensed [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/). Please follow the latest attribution guidance in ATTRIBUTION.md.
 
 ## Methodology and architecture
 
 Once you're ready to look under the hood:
 
-- `docs/executive_summary.md` — the short, non-technical overview: what jdLUC is, why LUC
-  emissions matter, and the headline takeaways. Start here if you're new.
-- `docs/methodology.md` — the high-level overview of the methodology: the datasets behind
-  it, how we quantify per-pixel emissions, and how the jurisdictional-direct and statistical
-  attribution legs produce emissions factors. The *what* we compute and *why*, from a
-  scientific standpoint.
-- `docs/architecture.md` — the system architecture and the rationale behind it: why we
-  build on the [Pangeo](https://pangeo.io/) stack, how the five-stage pipeline is
-  structured, and the tooling, storage, and caching choices that make it reproducible on a
-  single host. The *how* it's built and *why those choices*.
-- `docs/data.md` — the published data products: storage locations, grids, and full schemas
-  for the harmonized inputs, per-pixel emissions, and emissions-factor table.
-- `docs/coverage.md` — the full list of countries and territories the pipeline produces
-  emissions factors for, by ISO 3166-1 alpha-3 code.
+- `docs/executive_summary.md` — the short, non-technical overview: what jdLUC is, why LUC emissions matter, and the headline takeaways. Start here if you're new.
+- `docs/methodology.md` — the high-level overview of the methodology: the datasets behind it, how we quantify per-pixel emissions, and how the jurisdictional-direct and statistical attribution legs produce emissions factors. The *what* we compute and *why*, from a scientific standpoint.
+- `docs/architecture.md` — the system architecture and the rationale behind it: why we build on the [Pangeo](https://pangeo.io/) stack, how the five-stage pipeline is structured, and the tooling, storage, and caching choices that make it reproducible on a single host. The *how* it's built and *why those choices*.
+- `docs/data.md` — how to get the data products, grids, and full schemas for the harmonized inputs, per-pixel emissions, and emissions-factor table.
+- `docs/coverage.md` — the full list of countries and crops the pipeline produces emissions factors for, by ISO 3166-1 alpha-3 code.
+- `docs/validation.md` — how the pipeline is measured against external datasets: how the (country, crop) targets are chosen, which relationships are held as regression controls, and what such a comparison can and cannot conclude. The tooling lives in `validation/` and reports against the gaps in `docs/further_research.md`.
 
 ## Running and contributing
 
 ### Getting set up
 
-You'll need [uv](https://docs.astral.sh/uv/getting-started/installation/) installed as the Python env manager, plus a GCP project with GCS access.
+You'll need [uv](https://docs.astral.sh/uv/getting-started/installation/) installed as the Python env manager. Storage is either local directories or a GCS bucket: `INGEST_ROOT` and `SCRATCH_ROOT` in `.env` take any fsspec-supported URI, and only a `gs://` root needs a GCP project. Every source dataset is fetched over https, so ingest needs no cloud account either way.
 
 ```bash
 # Copy the example env file and fill in your values.
@@ -76,39 +59,54 @@ cp .env.example .env
 
 # Sync Python dependencies into the project venv.
 uv sync
+```
+
+**Local roots.** Point `INGEST_ROOT` and `SCRATCH_ROOT` at absolute paths and create them up front:
 
+```bash
+mkdir -p "${HOME}/luc/ingest" "${HOME}/luc/scratch"
+```
+
+Budget the disk: the two zarr stores dominate, at 12 TiB for the continent-scale benchmark in [`docs/architecture.md`](docs/architecture.md#single-host-by-design).
+
+**A `gs://` root** additionally needs application-default credentials, which gcsfs finds on its own and GDAL's `/vsigs` reader does not:
+
+```bash
 # Authenticate gcloud application-default credentials (for GCS access).
 gcloud auth application-default login --project "${GCP_PROJECT}"
+
+# Point GDAL at those credentials.
+export GOOGLE_APPLICATION_CREDENTIALS="${HOME}/.config/gcloud/application_default_credentials.json"
 ```
 
 ### Running the pipeline
 
-The pipeline runs as a sequence of per-stage entry points. Stages 1–3 operate on a **continent** (`AFRICA`, `ASIA`, `EUROPE`, `NORTH_AMERICA`, `OCEANIA`, `RUSSIA`, `SOUTH_AMERICA`, `UNCLASSIFIED`); stages 4–5 operate on one or more **ISO 3166 country codes** plus a `--methodology-name` (`STATISTICAL` or `JURISDICTIONAL_DIRECT`). The example below reproduces the United States via the statistical leg:
+Every stage is scoped by one or more **ISO 3166 alpha-3 country codes**, or by `--backfill` for a global run. Stages 4–5 additionally take a `--methodology-name` (`STATISTICAL` or `JURISDICTIONAL_DIRECT`). The example below reproduces Honduras via the statistical leg. Honduras is the cheapest end-to-end run in the corpus: the unit of work is a 10° tile, and it falls inside a single one (`20N_090W`). Swap in `USA` for the jurisdictional-direct leg, which needs the US-only CDL and NASS inputs:
 
 ```bash
-# 1. Ingest each source dataset for the continent (positional: continent, dataset).
+# 1. Ingest each source dataset for the countries (positional: dataset, then ISO codes).
 #    Repeat per DATASET in the inventory (see docs/methodology.md).
-uv run python -m jdluc.ingest NORTH_AMERICA GLAD_GLCLUC
+uv run python -m jdluc.ingest GLAD_GLCLUC HND
 
-# 2. Harmonize the continent's tiles onto the common grid (--grid-name defaults to GLAD ~30 m).
-uv run python jdluc/harmonize.py NORTH_AMERICA
+# 2. Harmonize the countries' tiles onto the common grid (--grid-name defaults to GLAD ~30 m).
+uv run python jdluc/harmonize.py HND
 
 # 3. Compute per-pixel land-conversion emissions (writes zarr).
-uv run python jdluc/emit.py NORTH_AMERICA
+uv run python jdluc/emit.py HND
 
 # 4. Attribute emissions to crops for one or more countries (writes the rollup parquet).
-uv run python jdluc/attribute.py USA --methodology-name STATISTICAL
+uv run python jdluc/attribute.py HND --methodology-name STATISTICAL
 
 # 5. Reduce the rollup to the emissions-factor table (writes parquet).
-uv run python jdluc/trace.py USA --methodology-name STATISTICAL
+uv run python jdluc/trace.py HND --methodology-name STATISTICAL
 ```
 
-Each stage pulls its cached upstreams, so re-running a later stage recomputes only what's missing. The artifacts these stages write are documented in `docs/data.md`.
+Stages 1–3 derive their tiles from the country boundaries exactly as stage 4 does, so pre-warming them covers precisely what attribution will request. Each stage pulls its cached upstreams, so re-running a later stage recomputes only what's missing. The artifacts these stages write are documented in `docs/data.md`.
 
 ### Running tests
 
 ```bash
-uv run pytest jdluc
+uv run pytest
 ```
 
 ### Linting the codebase