Extract the NIPA tables the value-added block is built from (#536)

[WesIngwersen]

Closes #536, and reshapes the Step 2 plan ahead of #538.

Draft — the plan revisions want review before the FBS methods get built on them.

Scope: three rows, and one of them is a conversion question

Step 2 builds three rows. SUT_VALUE_ADDED_CODES and the use_va_detail_sut diagnostic already target V00100, T00OTOP, V00300 — the VABAS components.

T00TOP/T00SUB are not a sourcing question. They are already built by commodity in Step 4d (#690), so the industry row is a transformation of money that exists, not a second estimate. What was open is whether the transformation works — now measured for 2017 rather than assumed (tax_axis_conversion.py).

Using the benchmark market-share matrix from the Supply table, D[c,i] = V[c,i] / T007[c]:

row	correlation	abs. error
T00TOP (TOP + MDTY)	0.202	114.6% of the row
T00TOP (TOP only)	0.211	109.5%
T00SUB	0.676	79.8%

⚠️ The reason is structural, not noise. 55.7% of published T00TOP sits in wholesale and retail industries, because a tax on a product is remitted by whoever sells it while market shares place it with whoever makes it:

industry	estimate	published
424700	13	88,362	petroleum wholesalers
324110	92,893	397	petroleum refineries
441000	2,301	45,947	motor vehicle dealers
336111+336112	24,141	26	automobile manufacturing

The whole petroleum tax moves from wholesalers to refineries; the whole vehicle tax from dealers to assemblers. The operator is not noisy, it is aimed at the wrong stage of the chain.

⚠️ Measured in the benchmark year, against the very table the mix comes from — so this is not the "2017 ratios drift" objection the 2026-08-17 decision was argued from. It is stronger: a conversion that fails in 2017 cannot be rescued by being applied nearer to 2017.

That decision therefore stands — the industry distribution stays free for Step 5 under economy-wide soft targets — but free is not unseeded:

• ✅ 4200ID takes MDTY exactly: 38,513 published against a Supply MDTY of 38,507. Customs duties are a lookup, 5.1% of the row, every year. Independently confirms the identity already listed in the Step 5 target set.
• T00SUB's residual is two named structures, not a smear — S00203 transit subsidies (19,471 vs 1,964) and S00102 (6,339 vs 102) are paid to an operator rather than attaching to a product, so no product-side operator can place them. T30800 already carries them.
• ⏳ The remaining split should ride the margin structure, not the Make matrix: a commodity's wholesale and retail margins say which trade industries handle it, which is the point-of-sale signal the tax follows. Those margins are built (nowcast_trade_margins, Build the trade margin column, wholesale and retail (#613) #684), so that operator is testable exactly as this one was. Open follow-up.

#536 — the tables

31 tables added to BEA_NIPA.yaml, all annual and complete for 2012–2024, at ~950 extra FBA rows per year: controls (1.10, 1.1.5, 1.3.5, 1.14), compensation (6.2D, 6.3D, 6.10D, 6.11D, 3.10.5, 7.18), gross operating surplus (6.12D–6.22D, 7.5, 7.7, 7.9, 3.8, 7.3.5, 7.4.5, 7.11), and the QCEW-gap set below.

test_bea_nipa_value_added_tables.py (17 tests) is the acceptance check: not "the tables extract" but "the numbers the block is built from come back correct through the FBA". Every expectation is a published 2017 figure, so a silent BEA vintage change fails here rather than inside Step 2.

Three findings that changed the plan:

• ⚠️ Take the paid line, not the table's root. 6.2D and 6.3D each state their total twice — line 1 received by residents, line 2 paid by domestic industries and government. On line 2 the identity closes to 0 (8,485,016 + 604,656 + 1,345,306 = 10,434,978) and lands on the SUT's V00100 within 3. Reading line 1 is what left the ~10,600 the compensation plan carried as an open item; it is the rest-of-world adjustment A4187C, stated in 6.2D's own lines 97–99.
• ⚠️ 6.11D is three panels under one code, and only the first is by industry: lines 1–20 industry, 22–36 type of fund, 37–45 benefits paid (2,370,770, a different concept). Its industry grain is 17, not the 36 the plan claimed, and selecting the whole table double-counts. T61600D, T71100 and T11400 also restate a code; select by line in all four.
• ✅ Table 3.8 gives the government-enterprise surplus an industry axis it was thought not to have, federal and state-and-local summing to 1.10's A108RC exactly.

The QCEW-to-NIPA wage gap

QCEW covers 93.8% of NIPA wages paid. The plan treated the other 6.2% as a residual to spread pro rata. NIPA states it.

7.18 closes it exactly: BLS 7,968,336 + misreporting 106,273 + not-UI-covered 399,801 + timing 0 = 8,474,410. And it is non-uniform before any industry table is opened — government's uncovered rate is 11.3% against private's 3.5%.

6.4D turns the assumption into a measurement. QCEW publishes employment on the same axis, so QCEW/NIPA is a coverage ratio per industry. 2017, private employment, thousands:

NIPA 6.4D private line	6.4D	QCEW	QCEW/NIPA
Farms	819	818	99.9%
Construction	7,127	6,919	97.1%
Manufacturing	12,440	12,407	99.7%
Retail trade	15,989	15,854	99.2%
Health care and social assistance	19,576	19,322	98.7%
Educational services	3,662	2,824	77.1%
Other services, except government	7,042	4,435	63.0%

Coverage is 97–100% across most of the economy and collapses in exactly two places — religious and grantmaking organisations are largely UI-exempt. So QCEW growth is trustworthy broadly and the exceptions are a named short list, not a flat 6.2% haircut.

⚠️ Construction at 97.1% is a warning, not a reassurance. The count agrees because both sources count the same workers; the trade-versus-structure-type disagreement that actually matters is invisible to a coverage ratio.

Three lookups located: RfHhInstComp W151RC = 18,684, the SUT's 814000 to the dollar — the sector QCEW covers worst is the one NIPA hands over directly. U32500 splits general government compensation into wages and supplements, which 3.10.5 only totals. T60600D wages-per-FTE is the plausibility check Phase 5 lacked: a share can be positive, sum to one, and still imply an implausible wage.

Two parser defects the new tables exposed

Scale and unit were assumed, not read. A flat * 1000000 with Class/Unit hardcoded to Money/USD is right only while every declared table is in millions of dollars. 6.4D/6.5D are thousands of persons and 6.6D is a ratio, so 6.4D's 150,654 thousand employees would have been published as 1.5e11 USD. Both now come from each series' MetricName/DefaultScale, with a map that raises on an unrecognised metric rather than defaulting to dollars. Every BEA dollar table has DefaultScale = -6, so all 1,812 dollar rows are bit-identical across the change. 1.14's three chained-dollar lines move to Class: Other, so a Class: Money selection cannot add real dollars to nominal.

Table ids were matched by substring. Asking for U70205 also returned U70205S's 44 physical-quantity and price-index series under their own id. Nothing downstream selected them so it never showed — the metric guard above is what surfaced it. Matched exactly now.

Plan reshape, ahead of #538

Step 2 is settled as three FBS methods, one per Use row — not one method, not plain Python modules. BEA_Detail_Use_SUT already melts the Use SUT through VAPRO, so the 2017 benchmark VA block by industry is an attribution source today; no new extractor. Three rather than one because V00100 carries a hard group-level constraint in Step 5 while V00300/T00OTOP are seeds only, and V00300's ~20 activity sets should not destabilise the one row Step 5 constrains.

⚠️ The one step with no FBS primitive is the anchor-and-move itself: multiplication does not preserve the group total and there is no renormalise step. Recorded as the design seam — Phases 1–2 build a cached FBS_outside_flowsa moved-share source, Phase 3 is a single proportional attribution against it, and the old Phase 3.4 exact rescale then holds by construction.

Also corrected in the plan: the #567/#568 crosswalk gate is closed (three CSV lines now, not a decision); and the construction claim was too strong — the crosswalk carries 236 NAICS 23* rows, so the pipeline will not fail, it will produce an even split and call it an answer.

QCEW cache

Generating the BLS_QCEW FBA is slow (~9M rows/year, county grain). The cached per-year parquets work with no code change: copy one into extract/output_data/ and getFlowByActivity finds it on the ordinary local path — esupy matches on name and ignores the v2.0.4 tag. Verified for 2017. Two things flagged rather than deferred: the cache is 2017–2023, so the nowcast's 2024 has no QCEW; and Employment_common.yaml cannot be reused — every _bls_selection_fields_* block hardcodes Class: Employment.

Verification

• NIPA_final_dom_uses_2017 rebuilds bit-identical — all 17 columns, same row count — across every change here.
• 2012/2019/2020/2024 extract cleanly.
• 17 tests in the new file; tax_axis_conversion --check green.
• black / ruff / pytest green. The 4 mypy errors in utils/config/settings.py are pre-existing and Windows-only (resource module); confirmed present on a clean tree.

🤖 Generated with Claude Code

[WesIngwersen]

Update: the QCEW-to-NIPA wage gap, and two parser defects it exposed

Pushed 64833a5c. Seven more tables, in answer to "is there anything that fills the 6.2% gap between QCEW and NIPA compensation?" — yes, and it changes the method rather than just documenting the shortfall.

7.18 closes the gap exactly

	code	2017, $M
BLS published wages	BA06RC	7,968,336
+ misreporting on employment tax returns	BA07RC	106,273
+ not, or not fully, UI-covered	W873RC	399,801
+ timing (accrual)	Y663RC	0
= NIPA wages, received	A034RC	8,474,410

And it is already known to be non-uniform before any industry table is opened: government's uncovered rate is 11.3% against private's 3.5%.

6.4D turns the assumption into a measurement

QCEW publishes employment on the same axis, so QCEW/NIPA is a coverage ratio per industry. 2017, private employment, thousands:

NIPA 6.4D private line	6.4D	QCEW	QCEW/NIPA
Farms	819	818	99.9%
Construction	7,127	6,919	97.1%
Manufacturing	12,440	12,407	99.7%
Retail trade	15,989	15,854	99.2%
Health care and social assistance	19,576	19,322	98.7%
Educational services	3,662	2,824	77.1%
Other services, except government	7,042	4,435	63.0%

Coverage is 97–100% across most of the economy and collapses in exactly two places. So QCEW growth can be trusted broadly and the exceptions are a named short list — a different method from a flat 6.2% haircut.

⚠️ Construction at 97.1% is a warning, not a reassurance. The count agrees because both sources count the same workers; the trade-versus-structure-type disagreement that actually matters is invisible to a coverage ratio. Phase 4's treatment stands.

Three lookups located

• RfHhInstComp W151RC = 18,684 — the SUT's 814000 to the dollar. The sector QCEW covers worst is the one NIPA hands over directly; the plan described this line without finding it.
• U32500 splits general government compensation into wages (1,233,594) and supplements (535,640), which 3.10.5 only totals.
• T60600D wages per FTE is the plausibility check Phase 5 lacked — a share can be positive, sum to one, and still imply an implausible wage.

Two parser defects the new tables exposed

Scale and unit were assumed, not read. A flat * 1000000 with Class/Unit hardcoded to Money/USD is right only while every declared table is in millions of dollars. 6.4D/6.5D are thousands of persons and 6.6D is a ratio, so 6.4D's 150,654 thousand employees would have been published as 1.5e11 USD. Both now come from each series' MetricName/DefaultScale, with a map that raises on an unrecognised metric rather than defaulting to dollars. Every dollar table BEA publishes has DefaultScale = -6, so all 1,812 dollar rows are bit-identical across the change. 1.14's three chained-dollar lines move to Class: Other, so a Class: Money selection cannot add real dollars to nominal.

Table ids were matched by substring. Asking for U70205 also returned U70205S's 44 physical-quantity and price-index series under their own id. Nothing downstream selected them so it never showed — the metric guard above is what surfaced it. Matched exactly now.

Verification

NIPA_final_dom_uses_2017 still rebuilds bit-identical; 2012/2019/2020/2024 extract cleanly; 17 tests in the new file; black/ruff/pytest green.

[WesIngwersen]

Do we need to differentiate trade industries within wholesale and within retail?

Pushed cb991b3b. Measured — and the answer is only within wholesale, and only for one code.

Testing the proposed margin operator turned up something better: Step 4c already computes what this needs. top_by_level splits TOP per commodity into producer-level (325,829) and trade-level (391,096) from an identity with nothing modelled in it — excise sits in Producers' Value, sales tax inside the margin columns. That is the producer-versus-seller distinction market shares get wrong, and its trade-level total lands within +2.2% of published wholesale-plus-retail T00TOP (391,096 vs 382,491).

operator	corr	abs. error
market share on all TOP + MDTY	0.204	114.6%
+ level split, trade-level by trade output	0.743	41.9%
+ motor fuel routed to 424700 by name	0.946	29.9%

By block

• Non-trade industries — no matrix at all. 44.3% of the row, and once the producer-level portion is separated, plain market shares give corr 0.987. Solved.
• Within retail — no. Output shares (for a trade industry, very nearly margin shares) give 0.744. Retail product tax is general sales tax: broad-based, HHI 0.137, an effective 7.3 of 9 industries carrying it.
• Within wholesale — yes, and output shares are worse than useless: corr −0.192. 424700 petroleum wholesalers takes 51.3% of wholesale product tax on 3.4% of wholesale output — a 15× concentration — because wholesale tax is motor fuel excise, not a broad-based tax. HHI 0.321, an effective 3.1 of 10.

Published T00TOP inside each group, share vs that industry's share of group output:

wholesale	T00TOP	share	output share
424700	88,362	51.3%	3.4%
424A00	35,487	20.6%	18.3%
423A00	16,113	9.4%	14.7%
424200	2,783	1.6%	11.5%

retail	T00TOP	share	output share
441000	45,947	21.8%	18.9%
452000	37,990	18.1%	12.0%
4B0000	24,926	11.9%	11.3%
445000	22,253	10.6%	13.9%

Retail tracks output; wholesale does not, and one code is why.

The useful part: wholesale needs a named routing, not a matrix

NAMED_TAX_LINES already carries motor fuel as 324110, and trade_level_share already says that commodity's tax is 99.8% trade-level. Sending that 98,842 to 424700 lands against a published 88,362 and takes wholesale from −0.192 to 0.973, the whole row to 0.946. With petroleum pulled out by name, the other nine wholesale industries score 0.825 on output shares — they behave like retail.

So the general commodity × trade-industry margin matrix that the PRO:PUR producer-price work will eventually need is not required here. The tax conversion is served by the level split, which exists, plus a handful of named-line routings, which are already enumerated.

Still a seed and not a target — 29.9% absolute error. tax_axis_conversion --check now asserts the seed's argument too, including that 424700 still dominates wholesale, so a later vintage that spreads the fuel tax differently fails loudly rather than quietly degrading the seed.

[bl-young]

I recommend avoiding comments that refer to what the code used to do