Land wave C2 + Track D — the four remaining static datasets

builders/README.md

@@ -39,7 +39,11 @@ re-fetched** — see `AGENTS.md`.
 | `japan_earthquakes.py` | `japan_earthquakes.csv` | committed |
 | `japan_population_by_age.py` | `japan_population_by_age.csv` | committed |
 | `us_adult_heights.py` | `us_adult_heights.csv` | committed |
-| `NEWQDATA.py` | `NEWQDATA.csv` | committed — the **only** builder here that reads a committed input (`sources/NEWQDATA.MAT`) instead of fetching. Its upstream is published nowhere; see `sources/README.md`. Reproduces its output byte for byte |
+| `NEWQDATA.py` | `NEWQDATA.csv` | committed — reads a committed input (`sources/NEWQDATA.MAT`) instead of fetching. Its upstream is published nowhere; see `sources/README.md`. Reproduces its output byte for byte |
+| `dataBHS.py` | `dataBHS.csv` | committed — the second `sources/` reader (`sources/dataBHS.mat`, un-refetchable; see `sources/README.md`). A value-preserving MATLAB-to-CSV conversion; validates the consuming lecture's hardcoded moments on every run |
+| `bbh_macro_quarterly.py` | `bbh_macro_quarterly.csv` | committed — range-reads one workbook out of the 198.8 MB Zenodo replication package. Reproduces its output byte for byte (2026-08-17) |
+| `bbh_michigan_monthly.py` | `bbh_michigan_monthly.csv` | committed — same Zenodo package, different workbook. Reproduces its output byte for byte (2026-08-17) |
+| `fred_data.py` | `fred_data.csv` | committed — fetches six FRED series live over a pinned 1953-04..2024-12 window (yields and the recession dummy are stable history, unlike the BBH national-accounts snapshot). Reproduces its output byte for byte (2026-08-18) |
 | `hansen_singleton_1982_data.py` | `hansen_singleton_1982_data.csv` | committed — fetches FRED and the Ken French factors live. Reproduces its output byte for byte (2026-08-13) |
 | `hansen_singleton_1983_data.py` | `hansen_singleton_1983_data.csv` | committed — the same construction plus a T-bill leg, so its output is a strict superset of the 1982 file's. Reproduces its output byte for byte (2026-08-13) |
 | `business_cycle.py` | `business_cycle_data.csv`, `business_cycle_info.md`, `business_cycle_metadata.md` | run by hand, no validate stage yet (PLAN Phase 5); its three outputs are the repo's only unmanifested files |
@@ -67,13 +71,14 @@ here instead of patched — fixing it would mean this file is no longer the thin
 that produced those bytes. The fix belongs in `lecture-python-intro`, which still
 serves that notebook to readers.
 
-**This listing is the coverage report.** The repo has 21 `constructed` datasets.
-Fourteen ship a builder (9 `committed`, 5 `committed-frozen`), carried by **12**
-distinct builder files — fewer than the datasets because `generating_mini.md`
-and `webscrape_forbes.ipynb` each produce two. The remaining **7** have none:
-they carry `builder_status: unrecovered` in their manifests, which is the Phase
-9 recovery backlog, kept visible rather than hidden by reclassifying the file as
-`verbatim`. The table above lists a **13th** builder, `business_cycle.py`, which
+**This listing is the coverage report.** The repo has 28 `constructed` datasets
+(re-derived from the parsed manifests, 2026-08-18). Eighteen ship a builder (13
+`committed`, 5 `committed-frozen`), carried by **16** distinct builder files —
+fewer than the datasets because `generating_mini.md` and
+`webscrape_forbes.ipynb` each produce two. The remaining **10** have none: they
+carry `builder_status: unrecovered` in their manifests, which is the Phase 9
+recovery backlog, kept visible rather than hidden by reclassifying the file as
+`verbatim`. The table above lists a **17th** builder, `business_cycle.py`, which
 no manifest references — its three outputs are the repo's only unmanifested
 files.
 

builders/dataBHS.py

@@ -0,0 +1,111 @@
+"""
+Builder for lectures/dataBHS.csv.
+
+Converts the Barillas-Hansen-Sargent "Doubts or variability?" (JET, 2009)
+MATLAB data file into the CSV the `five_preferences` lecture reads. Three
+quarterly US series over 1948Q1-2006Q4: log real per-capita consumption and
+two gross real asset returns. The file carries no date column; the sample is
+stated in the paper and in the consuming lecture's prose ("1948.I-2006.IV").
+
+This is a value-preserving container conversion and nothing else -- .mat to
+.csv, no filtering, no rescaling, no reordering. The published CSV parses back
+bit-exactly under pandas' correctly-rounded reader, and -- measured, not
+assumed -- the consuming lecture's histogram of consumption growth has
+identical counts and bin edges under pandas' DEFAULT parser, so the lecture
+needs no float_precision flag.
+
+READS ITS INPUT FROM sources/, WHICH IS THE EXCEPTION, NOT THE RULE.
+AGENTS.md permits it only when the input cannot be re-fetched, and this one
+cannot: neither author hosts the replication files (tomsargent.com's source
+page 404s, larspeterhansen.org lists no code or data for the paper), the
+Journal of Economic Theory article carries no data supplement, and a
+GitHub-wide code search finds only QuantEcon's own inherited copies of this
+blob. Searched with positive controls 2026-08-18 -- see sources/README.md.
+
+Stages: fetch -> pre-process -> validate -> write.
+"""
+
+import io
+import os
+
+import pandas as pd
+from scipy.io import loadmat
+
+CURRENT_FILE_DIR = os.path.dirname(os.path.abspath(__file__))
+REPO_ROOT = os.path.dirname(CURRENT_FILE_DIR)
+PUBLISHED_DIR = os.path.join(REPO_ROOT, 'lectures')
+SOURCES_DIR = os.path.join(REPO_ROOT, 'sources')
+
+SOURCE_FILE = 'dataBHS.mat'
+
+# The .mat holds three bare (236, 1) float64 arrays under these names, in this
+# order. The output column order is the input order -- see validate().
+COLUMNS = ['c', 'rb', 'rs']
+
+# 1948Q1 to 2006Q4 inclusive, quarterly, no gaps -- 59 years x 4.
+N_QUARTERS = 236
+
+# The consuming lecture hardcodes the mean and standard deviation of quarterly
+# log consumption growth (five_preferences.md, "Set parameter values"). They
+# are moments of THIS vintage, so they double as its fingerprint: a substituted
+# or truncated input fails here rather than silently mis-plotting the lecture's
+# approximating and worst-case densities against its histogram.
+GROWTH_MEAN = 0.004952
+GROWTH_STD = 0.005050
+
+OUT_FILE = 'dataBHS.csv'
+
+
+def fetch():
+    return loadmat(os.path.join(SOURCES_DIR, SOURCE_FILE))
+
+
+def pre_process(raw):
+    # Each array is (236, 1); ravel to 1-D so the frame is 236 rows, not 236
+    # columns of one element.
+    return pd.DataFrame({name: raw[name].ravel() for name in COLUMNS})
+
+
+def validate(df):
+    """Refuse to write anything that is not the shape we expect."""
+    assert list(df.columns) == COLUMNS
+    assert len(df) == N_QUARTERS, f'expected {N_QUARTERS} quarters, got {len(df)}'
+    assert not df.isnull().values.any()
+    assert (df.dtypes == 'float64').all()
+
+    # c is LOG per-capita consumption; rb and rs are GROSS real returns. A
+    # vintage stored in levels, percentages or net returns would pass the
+    # structural checks above and quietly rescale everything downstream.
+    assert df['c'].between(-5.0, -3.0).all(), 'c is not log consumption'
+    assert df['rb'].between(0.9, 1.1).all(), 'rb is not a gross return'
+    assert df['rs'].between(0.6, 1.4).all(), 'rs is not a gross return'
+
+    # The lecture's hardcoded moments of quarterly log consumption growth,
+    # reproduced to their printed precision.
+    growth = df['c'].to_numpy()[1:] - df['c'].to_numpy()[:-1]
+    assert round(growth.mean(), 6) == GROWTH_MEAN, growth.mean()
+    assert round(growth.std(), 6) == GROWTH_STD, growth.std()
+
+    # The conversion contract: the CSV must parse back bit-exactly under the
+    # correctly-rounded reader. (pandas' default parser is fast, not correctly
+    # rounded -- PLAN-QELD-PACKAGE.md section 4.3 measured 18 of 708 values off
+    # by <= 2.1e-16 relative under 'high'. The lecture's histogram is identical
+    # either way, which is what lets the lecture keep a plain read_csv.)
+    buffer = io.StringIO()
+    df.to_csv(buffer, index=False)
+    buffer.seek(0)
+    back = pd.read_csv(buffer, float_precision='round_trip')
+    for name in COLUMNS:
+        assert (back[name].to_numpy() == df[name].to_numpy()).all(), \
+            f'{name} does not round-trip bit-exactly'
+
+
+def run():
+    df = pre_process(fetch())
+    validate(df)
+    df.to_csv(os.path.join(PUBLISHED_DIR, OUT_FILE), index=False)
+    print(f'wrote {OUT_FILE}: {len(df)} quarters x {len(df.columns)} series')
+
+
+if __name__ == '__main__':
+    run()

builders/fred_data.py

@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+"""
+Builder for lectures/fred_data.csv.
+
+Fetches the six FRED series the `risk_aversion_or_mistaken_beliefs` lecture
+plots -- three nominal Treasury constant-maturity yields (GS1, GS5, GS10), two
+real (TIPS) yields (DFII5, DFII10) and the NBER recession indicator (USREC) --
+monthly, over the fixed window 1953-04-01 to 2024-12-01, and writes them as
+one date-indexed CSV.
+
+Unlike the BBH files this IS a live-FRED read, deliberately: none of these
+series is revised the way the national accounts are. The nominal and real
+yields are historical H.15 market rates and USREC is a dummy built from
+NBER's published turning points, so the live values are stable -- measured
+2026-08-18, a fresh fetch reproduced the committed file byte for byte. The
+window end is pinned; this file is a frozen extract, not a tracking snapshot.
+
+Two fetch details that are easy to get wrong:
+
+- FRED publishes DFII5/DFII10 daily. The lecture's file carries their MONTHLY
+  AVERAGES, which fredgraph serves with `fq=Monthly&fam=avg`. GS1/GS5/GS10 and
+  USREC are monthly at source and need no aggregation.
+- fredgraph.csv now titles its date column `observation_date` (it used to be
+  `DATE`). The committed file predates the rename, so the index is renamed on
+  read; a builder that trusted the served header would change the byte layout.
+
+Stages: fetch -> pre-process -> validate -> write.
+
+Requires pandas.
+"""
+import io
+import os
+import urllib.request
+
+import pandas as pd
+
+CURRENT_FILE_DIR = os.path.dirname(os.path.abspath(__file__))
+REPO_ROOT = os.path.dirname(CURRENT_FILE_DIR)
+PUBLISHED_DIR = os.path.join(REPO_ROOT, 'lectures')
+
+OUT_FILE = 'fred_data.csv'
+
+FRED_CSV = 'https://fred.stlouisfed.org/graph/fredgraph.csv'
+START, END = '1953-04-01', '2024-12-01'
+
+# Monthly at source.
+MONTHLY = ['GS1', 'GS5', 'GS10', 'USREC']
+# Daily at source; fetched as monthly averages.
+DAILY_AVERAGED = ['DFII5', 'DFII10']
+# Column order of the published file.
+COLUMNS = ['GS1', 'GS5', 'GS10', 'DFII5', 'DFII10', 'USREC']
+
+N_MONTHS = 861                 # 1953-04 .. 2024-12 inclusive, no gaps
+# FRED publishes the TIPS yields from 2003-01, so the first 597 months of the
+# window are empty in both DFII columns and every other column is complete.
+KNOWN_NULLS = {'DFII5': 597, 'DFII10': 597}
+TIPS_START = pd.Timestamp('2003-01-01')
+
+
+def _fetch_series(code):
+    url = f'{FRED_CSV}?id={code}&cosd={START}&coed={END}'
+    if code in DAILY_AVERAGED:
+        url += '&fq=Monthly&fam=avg'
+    request = urllib.request.Request(url, headers={'User-Agent': 'qeld-builder'})
+    with urllib.request.urlopen(request) as response:
+        payload = response.read()
+    frame = pd.read_csv(io.BytesIO(payload), index_col=0, parse_dates=True,
+                        na_values='.')
+    frame.columns = [code]
+    return frame
+
+
+def fetch():
+    return pd.concat([_fetch_series(code) for code in COLUMNS], axis=1)
+
+
+def pre_process(fred):
+    fred = fred.loc[START:END]
+    fred.index.name = 'DATE'
+    fred['USREC'] = fred['USREC'].astype('int64')
+    return fred[COLUMNS]
+
+
+def validate(frame):
+    """Refuse to write anything that is not the shape we expect."""
+    assert list(frame.columns) == COLUMNS, list(frame.columns)
+    assert frame.index.name == 'DATE'
+
+    # 1953-04 .. 2024-12 on an unbroken monthly grid of first-of-month stamps.
+    assert len(frame) == N_MONTHS, f'expected {N_MONTHS}, got {len(frame)}'
+    assert frame.index[0] == pd.Timestamp(START)
+    assert frame.index[-1] == pd.Timestamp(END)
+    assert frame.index.is_monotonic_increasing
+    assert (frame.index.day == 1).all()
+    months = frame.index.year * 12 + frame.index.month
+    assert (pd.Series(months).diff().dropna() == 1).all(), 'gap in the grid'
+
+    # Exactly the declared holes, and nowhere else: the TIPS series before
+    # 2003-01, full stop.
+    nulls = frame.isnull().sum()
+    assert dict(nulls[nulls > 0]) == KNOWN_NULLS, dict(nulls[nulls > 0])
+    for code in DAILY_AVERAGED:
+        assert frame.loc[frame.index < TIPS_START, code].isnull().all()
+        assert frame.loc[frame.index >= TIPS_START, code].notnull().all()
+
+    # Units: percent per annum for every yield, 0/1 for the recession dummy.
+    # A fetch that silently switched to decimals or to an index would pass the
+    # grid checks above and rescale the lecture's figure.
+    for code in ['GS1', 'GS5', 'GS10']:
+        assert frame[code].between(0.0, 20.0).all(), f'{code} out of band'
+    for code in DAILY_AVERAGED:
+        assert frame[code].dropna().between(-3.0, 5.0).all(), f'{code} out of band'
+    assert set(frame['USREC'].unique()) <= {0, 1}
+
+
+def run():
+    frame = pre_process(fetch())
+    validate(frame)
+    frame.to_csv(os.path.join(PUBLISHED_DIR, OUT_FILE))
+    print(f'wrote {OUT_FILE}: {frame.shape[0]} months x {frame.shape[1]} series '
+          f'({frame.index[0].date()} .. {frame.index[-1].date()})')
+
+
+if __name__ == '__main__':
+    run()