fix(trade): Census residuals, direction-specific IEA financials, and explore S00300 imports

[bl-young]

cc:
Closes:

What changed? Why?

Three Crosswalk fixes on the 2017 Trade FBS path, each tied to a specific sector cluster scored against published Use F04000 / Supply MCIF. Census goods keep one shared Crosswalk; IEA services split into direction-specific files because BEA books financial trade and noncomparable imports asymmetrically between exports and imports.

Aerospace and other Census residual NAICS (33641*, 1121*, 3113*, 3118*)

Problem. Census USA Trade publishes suppressed-detail residual codes (33641X, 11211X, 1123XX, 31131X, 31135X, 31181X) alongside digit-6 NAICS. The extract dropped \d{5}X / \d{4}XX rows, so Crosswalk targets for those residuals were unreachable. Aerospace exports (336411–336413) were MISS by ~$100 B combined; bakery (311810), sugar/confectionery (311300), and livestock residuals were zeroed.

Approach. census_usatrade_parse retains residual NAICS patterns. Sector_Crosswalk_Census_USATrade.csv maps each residual 1:m onto the Detail commodities of that family, split by frozen 2017 Use F04000 weights (same rule as other 1:m Census rows). 33641X (~$121 B export mass) fans out to 336411/336412/336413/336414/33641A.

Scorecard (baseline pin). 336411/336412/336413 move from ~0.05–0.25× ref toward ~1.1–1.9× ref (PARTIAL, not MISS). 311810, 1121A0, 112300, 311300 fill from their *X residuals.

Financial services imports (52A000 family)

Problem. A single IEA Crosswalk mapped parent Financial (~ $38 B imports) 1:m onto all four financial Detail codes by Supply MCIF weights. Published 2017 MCIF is almost entirely FISIM on 52A000 (~$6.6 B); explicit fee subtypes (FinSecBrokAndMM, FinCredCard*, …) have no matching MCIF cells. The unified map put ~$37.6 B on 52A000 (5.7× published).

Approach. Replace Sector_Crosswalk_BEA_IEA.csv with direction-specific files:

Direction	File	Financial rule
Exports	Sector_Crosswalk_BEA_IEA_exports.csv	Parent Financial 1:m → 52A000, 522A00, 523A00, 523900; split by 2017 Use F04000
Imports	Sector_Crosswalk_BEA_IEA_imports.csv	FinFisim 1:1 → 52A000 only; parent Financial and explicit Fin* leaves unmapped

Trade_Exports_2017.yaml selects BEA_IEA_exports; Trade_Imports_2017.yaml selects BEA_IEA_imports. Non-financial rows match across both files except the #606 block below.

Scorecard. Import 52A000 moves from +466% (EXTRA-class) to ~−0.7% vs ref ($6.6 B). Import 523A00 EXTRA clears (mass no longer misallocated from parent Financial).

Noncomparable imports — #606 tranche 1 (S00300, 533000, 483000, 425000)

Problem. Five IEA service types land on Detail commodities with zero published Supply MCIF. The top three (533000 licensing ~$33 B, 483000 sea freight ~$28 B, 425000 trade-related wholesale ~$2 B) are textbook noncomparable imports per #606: services produced and consumed abroad, and payments for IP rights. Exports correctly book these on named Detail commodities (F04000).

Approach. On BEA_IEA_imports only, route:

IEA TypeOfService	Import target	Export target (unchanged)
TransportSeaFreight	S00300	483000
CipLicensesOutcomesResearchAndDev	S00300	533000
CipLicensesFranchiseFees	S00300	533000
CipLicensesTrademarks	S00300	533000
TradeRelated	S00300	425000
TransportAirPort / TransportSeaPort	S00300	unmapped (no 488000; 48A000 F040 ≪ IEA port mass)

Scorecard. Import EXTRA clears on 533000, 483000, 425000 (~ $64 B reallocated). Import S00300 moves from MISS ($0) to PARTIAL (~$79 B vs $260 B ref — ~30% of the published bucket). Remaining S00300 mass is hold-from-Supply / residual allocation, not Trade FBS.

Couriers

Both IEA Crosswalks: TransportAirFreight 1:m → 481000/492000 (native proportional weights). Export 492000 ~$2.4 B PARTIAL (was MISS); fixed-share splits deferred. Export ports remain unmapped.

Crosswalk review tooling

• score_2017_trade_fbs.py — score mapped Trade FBS parquet without rebuilding the supply bridge.
• review_census_crosswalk.py — probe Census NAICS coverage vs Crosswalk membership and residual reachability.

Documented in bedrock/transform/trade/README.md (direction-specific IEA table, residual rules, rejected experiments).

Checked and rejected (not in diff)

Prototypes were scored on 2017 Trade FBS; neither ships in this PR. Rationale is recorded under Crosswalk experiments (rejected) in bedrock/transform/trade/README.md.

Census goods → S00300 on imports (339116, 33211A). Direction-specific Census Crosswalk routed dental-lab and forging NAICS (~$668 M) to S00300 when Detail MCIF is zero. That cleared import EXTRA on those codes but #606 covers services/IP licensing, not Census merchandise. intermediate_estimation_plan.md treats 339116 / 33211A as concordance noise; 339116 has nonzero export F040. Family-level fit did not meaningfully improve. Track under #658 / #670.

TravelHealth 1:m across healthcare Detail (621100–621900, 622000, 623A00, 623B00). Split ~$1.1 B export / ~$640 M import TravelHealth by frozen SUT F040/MCIF weights. Healthcare family L1 error vs published totals unchanged ($2.7 B exports / $3.9 B imports). Ambulatory/residential MISS became PARTIAL by shifting mass off 622000 without improving fit. IEA TravelHealth covers ~29% of published healthcare export F040; the remainder has no IntlServTrade or Census source — TTSA or other Travel* types are the path if pursued.

Live scorecard after these commits (pinned with --update-baseline):

direction	National % vs SUT	Pearson all / non-S00*	Top-20 Jaccard all / non-S00*	n_miss
F040 exports	+12.20%	0.96 / 0.94	0.67 / 0.67	39
MCIF imports	+1.14%	0.77 / 0.85	0.74 / 0.82	18

Export national % rises mainly from Census 33641X aerospace recovery (~ $121 B). Import national % and non-S00* Pearson hold; S00300 is PARTIAL (~$64 B mapped), not a ≥$1 B MISS hole.

Testing

Regenerate Trade FBS and pin baseline per commodity:

uv run python -m bedrock.transform.trade.generate Trade_Exports_2017 Trade_Imports_2017
uv run python -m bedrock.analysis.nowcasting.trade_data.score_2017_trade_detail --update-baseline

Fast sector checks without full bridge rebuild:

uv run python -m bedrock.analysis.nowcasting.trade_data.score_2017_trade_fbs
uv run python -m bedrock.analysis.nowcasting.trade_data.review_census_crosswalk

Spot-check baseline pins for:

• Export 336411/336412/336413 — PARTIAL, no longer MISS
• Import 52A000 — ~$6.6 B (was ~$37.6 B)
• Import 533000/483000/425000 — zero EXTRA
• Import S00300 — ~$64 B mapped (PARTIAL vs $260 B ref)

[WesIngwersen]

Approach. census_usatrade_parse retains residual NAICS patterns. Sector_Crosswalk_Census_USATrade.csv maps each residual 1:m onto the Detail commodities of that family, split by frozen 2017 Use F04000 weights (same rule as other 1:m Census rows). 33641X (~$121 B export mass) fans out to 336411/336412/336413/336414/33641A.

Instead of using frozen 2017 weights for export ratios, use commodity output which is a better annual signal.

[bl-young]

Instead of using frozen 2017 weights for export ratios, use commodity output which is a better annual signal.

Ok both trade methods are set up to use their respective vectors as the allocation - are you suggesting that we make that change across the board (for both imports and exports, for just exports, or just for these *X sectors?)

[WesIngwersen]

Instead of using frozen 2017 weights for export ratios, use commodity output which is a better annual signal.

Ok both trade methods are set up to use their respective vectors as the allocation - are you suggesting that we make that change across the board (for both imports and exports, for just exports, or just for these *X sectors?)

For exports it should be commodity output.

For imports it should be scaled on total uses. We do not yet have total uses until we get a full Use block (For FD its change in inventories (me) and exports (you) then we are complete, and for intermediate uses its in the big stack of Step 3 related PRs (not clearly marked at step 3; most in draft). I would suggest handling this for exports in this PR and we create an issue to do this for Imports as a Step 5 precursor.

[WesIngwersen]

As described in comments - tie export allocaton to commodity output and not frozen 2017 shares

[WesIngwersen]

For imports it should be scaled on total uses. We do not yet have total uses until we get a full Use block (For FD its change in inventories (me) and exports (you) then we are complete, and for intermediate uses its in the big stack of Step 3 related PRs (not clearly marked at step 3; most in draft). I would suggest handling this for exports in this PR and we create an issue to do this for Imports as a Step 5 precursor.

Slight revision to previous recommendation. Use allocation factors based on ratios of total uses - commodity output to be more precise

[WesIngwersen]

Clarification on export approach:
For exports it should be commodity output applied to all the cases where we are allocating the Census lines across BEA commodities. The ratios can be applied as scalars to the 2017 mixes you identify. You can test whether this falls miserably or is passable, but its a theoretically more logical approach.

[bl-young]

ok tell you what I'm working on the time series stacked on top of this. why don't i do a stack next to focus just on that shift of allocation source for imports and exports to keep that delta clean.