Source TRANS for 2023 from AIES, and move TOP's residual onto a purchaser-price base (#611, #580)

[WesIngwersen]

Closes: #611 (transport phase 2, 2023 leg)
Related: #580, #612

What changed? Why?

The supply bridge is now complete and evaluable for 2017–2023 — seven years, up from one. TRANS was the last column blocking it, and sourcing it unblocked a second thing the tax layer had been waiting on.

Three commits, in dependency order:

	commit	issue
1	Source TRANS for 2023 from AIES	#611
2	Move TOP's sales-tax residual onto a purchaser-price base	#580
3	Close air's 2023 seam defect on FAF volume	#611

They are in one PR because each depends on the one before: #2 cannot be built until the margin columns exist, and #3 is a defect in what #1 introduced.

---

1 — TRANS for 2023 from AIES (#611)

SAS stops at data year 2022, so TRANS stopped there too and 2023 had no bridge. The plan listed truck as "source found; concordance to build" and 2023 as unsourced. It turned out better: three of the four modes needing a source came across, and the concordance needed no work at all.

Truck — 67.8%

New Census_AIES_MiscSector extract on timeseries/aies/miscsector. Its eleven RCPT_MOTR_<group>_DVAL items carry the same eleven group names as SAS Table 8 — verified against variables.json, all ten identified groups matching Crosswalk_SAS_Group_to_BEA_2017.csv exactly — so the crosswalk joins unchanged.

The parse emits Table 8's own FlowName strings, so everything below the source dispatch in load_truck_group_revenue stays one implementation across the seam. RCPT_MOTR_HAZRD_DVAL is deliberately not requested: hazardous materials is a cross-cut of the same revenue, not a twelfth group.

Pipeline (11.9%) and water/air freight revenue (3.8%)

Already inside the aies/basic call Census_AIES makes — only _TYPE_OF_OPERATION needed NAICS 48 at TYPOP 00. Those rows are inert for the trade margin, which filters to FlowName == 'Gross margins' and reads NAICS 42/44-45 by name. The four pipeline items still partition NAICS 486 to the dollar in 2023.

⚠️ mode_freight_revenue was a third SAS dependency not in the plan's mode table — the per-mode control total, which water and air also read. It is routed too; without it the column raised rather than built.

The truck splice — carried as observed

Shares step 4× their normal volatility at the seam (mean |Δpp| 1.39 vs 0.34 within SAS). Carried as observed on the same evidence that settled the retail rate step: taxonomy unchanged, the eleven still partition the published total (−65 on 341bn, $1,000 rounding), and the movers — used household goods 6.69→9.94%, coal and petroleum 6.99→9.56% — fit the 2023 freight recession. Only shares are spent; the level is anchored on 2017.

2024 stays unsourced

Both AIES datasets return 204 for 2024. Rail reaches it on STB and water/air on FAF, but truck plus pipeline are 79.7% of the column, so a partial fill would break the identity quietly. Same shape as the TRADE 2024 gap.

---

2 — TOP's residual moves onto a purchaser-price base (#580)

70% of TOP is general sales tax with no named NIPA line, and it sat on frozen 2017 shares. nowcast_product_taxes said the fix was "a purchaser-price base by commodity, which is Step 5's output" — but the supply bridge reaches it first. Once TRADE and TRANS are sourced, T013 + T014 is that base: supply at basic value plus the margins carrying it to the purchaser.

It clears both objections that killed the T007 movers

Objection to T007	This base
S00402 used/secondhand goods carries 15,699 $M of 2017 TOP — eighth largest — and T007 is zero for it by definition, so a top-ten position could not move	Purchaser base 174,312 $M, because secondhand goods are almost entirely trade margin. The margin layer reaches exactly what domestic output cannot
The one-sided mover drifted to 1.86× its 2017 level against a control growing 1.53×	Rescale stays within 0.939–1.008 across 2017–2023 — purchaser value is what sales tax is levied on, so base and control track

The pandemic years are the check

2020 accommodation 721000 falls 7,045 $M against the frozen vector, air transport roughly halves, restaurants fall — the sectors #580 worried the frozen vector would get wrong. The rejected T007 service-side mover put accommodation at −7,511 $M, so two independent bases agree on the size of the effect and only this one reaches it without breaking the control.

Construction notes

• No circularity. The base is assembled from the bridge's components, not from derive_initial_supply_bridge — the bridge calls top_column, so TOP cannot read the finished bridge back.
• Taxes excluded from the base. T015 carries TOP itself; including it would make the tax its own allocator.
• 2024 holds 2023's shares. The margin columns stop at 2023. Reverting to frozen 2017 would undo six years of movement in one step. It is a hold, not a measurement — and a new dependency of TOP on margin coverage, which resolves when AIES publishes 2024.

2017 is unchanged, as it must be. Every year still hits its NIPA control exactly and stays non-negative.

---

3 — Air's 2023 seam defect, closed on FAF volume (#611)

AIES 2023 publishes 481212 nonscheduled chartered freight air at 13,271 $M against 4,846 / 4,857 / 4,987 / 6,045 over 2019–2022 in SAS, unsuppressed. With 481112 that puts air freight revenue at 2.32× its 2017 level.

FAF settles it — and FAF is already air's commodity allocator, so no fourth series was imported to arbitrate:

year	revenue idx	volume idx	implied $/ton-mile
2021	1.18	1.16	1.02
2022	1.68	1.32	1.27
2023 published	2.32	1.06	2.19
2023 corrected	1.35	1.06	1.27

Volume fell back to 1.06× while revenue claimed 2.32× — unit revenue doubling in the year air cargo rates collapsed from their pandemic peak. Wrong size and wrong sign.

From 2023 air's control moves on volume, holding unit revenue at its last observed value: revenue(2022) × ton_miles(year) / ton_miles(2022). 2023 air revenue becomes 14,386 $M rather than 24,734, and air lands at 1.6% of TRANS — the share the mode table states. Uncorrected it took 2.7%.

⚠️ Water is deliberately not treated this way. Its unit revenue moves −4% across the same seam (1.65 → 1.58). Only air breaks, and a test pins that water keeps its published revenue so nobody later "fixes" it by symmetry.

⚠️ The 2022→2023 step alone is a misleading cut. From the 2017 anchor the two air codes grow 2.17× and 2.47×, which looks benign — it took an independent series to show the break. Recorded in the docstring so the next reader does not repeat it.

⚠️ Holding unit revenue flat is conservative. Rates fell in 2023, so this still overstates slightly; correcting for that needs a rate series the build does not carry, and inventing one would be a worse error than the one being fixed. Revisit when AIES publishes 2024 — if 481212 stays at the new level it is a re-based series, and the fix becomes re-anchoring rather than indexing off 2022.

The frozen coverage ratio is now recorded as a weak choice

Kept for now, but no longer reading as settled. For water and air mode_coverage_ratio is not a coverage correction — it stands in for a domestic/international split that genuinely moves, and it survives on bounded blast radius (3.8% of TRANS) rather than evidence. This defect was a symptom of the same gap: a frozen ratio passes any revenue error straight through to the margin. Replacing it needs an observed domestic leg, not a different frozen number.

---

SUB needs nothing

Checked rather than assumed: it reconciles to NIPA T31300 exactly in all eight years, is computed live so cannot go stale, and its only commodity-output dependency (the 2020–21 PPP split) does not overlap the years Step 4a changed.

Resulting bridge coverage

year     T007   MCIF   MADJ   T013  TRADE  TRANS   T014   MDTY    TOP    SUB   T015   T016        T016 $M
2017       ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     37,128,652
2018       ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     39,304,294
2019       ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     40,442,617
2020       ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     38,697,894
2021       ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     44,256,413
2022       ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     49,670,535
2023       ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     ok     51,543,283
2024       ok     ok     ok     ok      -      -      -     ok     ok     ok     ok      -    unevaluable

2024 blocks on TRADE and TRANS only, both on the same cause: AIES has not published data year 2024. That is a release date, not an engineering gap.

Testing

Four gates repo-wide: black ✅, ruff ✅, mypy ✅ (4 pre-existing Windows-only resource errors in settings.py, untouched here, green on ubuntu), pytest ✅ 753 passed, 2 skipped, 1 xfailed — up from 745 on nowcast.

Two tests were rewritten because their contract changed by design (TRANS is no longer NaN at 2023; the TOP divergence from the naive default grows from 3.7%→6.3% of the 2020 column and 5.8%→7.7% of 2024, because the other 70% now moves — a fall there would be the bug).

Eight added:

• test_aies_continues_the_sas_taxonomy_across_the_2023_seam — fails if Census renames a truck group, which would otherwise silently drop that group's revenue on a missed crosswalk join
• test_pipeline_items_still_partition_naics_486_under_aies — the partition check either side of the seam
• test_air_revenue_does_not_break_against_its_own_volume_at_the_seam
• test_water_keeps_its_published_revenue_across_the_seam
• test_air_volume_index_refuses_years_with_a_published_revenue
• test_purchaser_base_reaches_what_domestic_output_cannot — the S00402 case
• test_residual_does_not_drift_against_its_control — 2017–2023
• test_2024_holds_2023_shares_rather_than_reverting_to_2017

GCS extract input — staged ✅

Census_AIES_MiscSector_2023.csv is uploaded to gs://cornerstone-default/extract/input-data/Census_AIES_MiscSector/2023/ and verified byte-identical on round-trip, so CI's keyless path resolves. AIES requires an API key and CI has none.

⚠️ Census_AIES FBA must be regenerated for the widened _TYPE_OF_OPERATION — editing the parse does not invalidate the parquet.

🤖 Generated with Claude Code