Derive the 191->detail II and VA series, and give Step 3 an observed column control

[WesIngwersen]

Stacked on #710 (step3_core), which is green on all four checks.

BEA publishes intermediate inputs (UII205-A) and value added (UVA205-A) annually 1997-2024 on its 191-row "underlying" industry frame, in two workbooks sitting beside the GrossOutput.xlsx this repo already reads. Nothing read either one. This extracts both, allocates them to the 402 detail industries, and replaces Step 3's frozen-2017 VAPRO seed.

The 191→402 mapping is derived, not hand-written

UGO205-A and UGO305-A order industries the same way, so the 138 leaves of the 205-A hierarchy partition the 414 rows of 305-A into contiguous runs; the runs are closed by matching gross output in all 28 years. All 138 leaves match, exactly 414/414 detail rows are consumed, and the 402 codes cover the model schema once each.

derive_underlying_line_mapping reproduces the checked-in constant exactly. --mapping on the new diagnostic is how to re-check it when the BEA vintage moves.

VA is allocated; II is the residual

weight_c(t) = VAPRO_c(2017)/GO_c(2017) · GO_c(t), rescaled to the published line total; then T005 = GO − VAPRO.

Taking II as the residual is what makes GO = T005 + VAPRO hold per industry, which is the form T1 imposes — allocating both independently broke it by up to $15.2B a cell at 2024. It costs nothing against BEA:

check	result
VAPRO summed back to the 138 lines vs UVA205-A	0.0 on 3,864 cells
T005 summed back vs UII205-A	$9M (BEA's own GO=II+VA rounding)
2017 vs published detail Use SUT margins	$0.89M (VAPRO), $1.48M (T005) per industry
economy-wide VA vs published GDP, every year	≤ $9M on $29T
GO − T005 − VAPRO at detail	2.9e-11

It also removes the suppression problem: BEA suppresses II in every year on lines 83 (Customs duties) and 176 (Private households), both single-industry lines whose published 2017 T005 is zero, and the residual recovers that zero rather than needing a special case.

Step 3's column control

Now GO − VAPRO with both sides observed. Aggregated to summary and scored against the published summary T005: within 0.00007% economy-wide and 0.00023% weighted MAE by industry in every year 2018-2024, worst summary industry 0.003%. The superseded frozen-ratio seed scored 0.2-2.3% and 2.5-8.0%, with GSLG 18.3% low at 2022.

⚠️ That is a consistency check, not an independent validation — UII205-A and the summary Use SUT's T005 are the same BEA estimate published two ways. What it establishes is that the allocation adds back correctly, so the control is BEA's T005 rather than an approximation of it.

vapro_seed becomes vapro; the now-dead _row helper and published_gross_output import go.

Signs are carried, not clipped

S00201 has a published 2017 VAPRO of −$10,069M and stays negative in all 28 years. The residual T005 goes negative in 11 cells, all in 5191A0 and all in 2002-2015, so the 2017-2024 nowcast span is clear. The other residual direction was worse: 13 spurious negatives across three industries, reaching −$13,117M.

This is not Step 2

VAPRO is the column total. Step 2 owes the split across the five value-added rows, and T4/T6 are still soft placeholders. What changes is that there is now a VA level for every year rather than for 2017 alone, so the balance can run on 2024.

It also arrives off the P1-gated path — a straight Excel read on the shape of the working UGO305-A loader, so map_fbs_sectors_to_model_schema never enters it. plan.md's gating table is updated accordingly.

Checks

Repo-wide: black, ruff, mypy clean (bar the four known Windows-only settings.py errors); 779 passed, 2 skipped, 1 xfailed.

🤖 Generated with Claude Code