Step 2: the value-added block for 2017-2024, all five rows (#538) - #735
Open
WesIngwersen wants to merge 42 commits into
Open
Step 2: the value-added block for 2017-2024, all five rows (#538)#735WesIngwersen wants to merge 42 commits into
WesIngwersen wants to merge 42 commits into
Conversation
Adds 24 tables to BEA_NIPA.yaml, all annual and complete for 2012-2024, at roughly 950 extra FBA rows per year: the controls (1.10, 1.1.5, 1.3.5, 1.14), compensation (6.2D, 6.3D, 6.10D, 6.11D, 3.10.5, 7.18) and gross operating surplus (6.12D-6.22D, 7.5, 7.7, 7.9, 3.8, 7.3.5, 7.4.5, 7.11). T00TOP and T00SUB are deliberately absent. Their commodity-axis counterparts shipped in Step 4d (#690), and their industry split is an output of Step 5's balance rather than an input to it, so sourcing them here would estimate the same money twice on two axes. Three findings, all pinned by the new test rather than left in prose: - Take the *paid* line, not the table's root. 6.2D and 6.3D each state their total twice - line 1 received by residents, line 2 paid by domestic industries and government - and value added wants line 2. On it, wages plus both supplements close to 0 against compensation and land on the Use SUT's V00100 within 3. Reading line 1 is what left the ~10,600 the compensation plan carried as an open item; it is the rest-of-world adjustment A4187C, stated in 6.2D's own lines 97-99. - 6.11D is three panels under one code, and only the first is by industry: lines 1-20 industry, 22-36 type of fund, 37-45 benefits paid - a different concept at 2,370,770. Its industry grain is 17, not the 36 the plan claimed, and selecting the whole table double-counts. T61600D, T71100 and T11400 also restate a code; select by line in all four. - Table 3.8 gives the government-enterprise surplus an industry axis it was thought not to have, federal and state-and-local summing to 1.10's A108RC exactly. The config half of the test needs no data and guards the substring hazard in bea_nipa_parse, which selects a table's series with str.contains: two declared ids where one contains the other would contaminate silently. NIPA_final_dom_uses_2017 rebuilds bit-identical - all 17 columns, same row count - so the added rows reach no existing method. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ache The plan's Phases 1-5 were written as a pure-Python pipeline. With Step 2 settled as three FBS methods, most of it survives as configuration rather than code: BEA_Detail_Use_SUT already melts the Use SUT through VAPRO, so the 2017 benchmark V00100 by industry is an attribution source today, and Phase 1 is a selection_fields clause rather than an extraction. The one step with no FBS primitive is the anchor-and-move itself - multiplication does not preserve the group total and there is no renormalise step, which is what the old Phase 3.4 needed. Recorded as the design seam: Phases 1-2 build a cached FBS_outside_flowsa source holding the moved shares, Phase 3 is a single proportional attribution against it, and the exact rescale then holds by construction instead of being a follow-up step. QCEW is cached locally as per-year FBA parquets because generating it is slow (~9M rows a year, county grain). Dropping one into extract/output_data lets getFlowByActivity find it through the ordinary local path with no code change - esupy matches on name and ignores the v2.0.4 tag. Verified for 2017. Four things measured off that cache rather than assumed: - QCEW national NAICS-6 payroll is 93.8% of NIPA wages paid. That number is what settles "allocator, never control", and the missing 6.2% concentrates in the sectors Phase 4 already carves out. - 1,027 of the 1,048 NAICS-6 codes in the BEA detail crosswalk are present, so the gap is a bounded list of 21 rather than a structural hole. - Ownership is on the flow, not a separate axis. It still does not separate government enterprises from general government, which is the distinction the SUT needs - so government stays on NIPA, and the plan's claim is narrowed to what is actually true. - The cache is 2017-2023, so the nowcast's 2024 has no QCEW and needs a stated fallback. Better found here than in Phase 3. Employment_common.yaml cannot be reused as-is: every _bls_selection_fields_* block hardcodes Class: Employment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
QCEW covers 93.8% of NIPA wages paid. The plan treated the other 6.2% as an unexplained residual to spread pro rata. NIPA states it. 7.18 closes it exactly: BLS published 7,968,336 + misreporting 106,273 + not UI-covered 399,801 + timing 0 = 8,474,410, wages received. And the gap is already non-uniform before any industry table is opened - government's uncovered rate is 11.3% of government wages against private's 3.5%. 6.4D is the table that turns the assumption into a measurement: QCEW publishes employment on the same axis, so QCEW/NIPA is a coverage ratio per industry. Measured on 2017 private employment it is 97-100% almost everywhere and collapses in exactly two places - educational services 77.1%, other services except government 63.0%, where religious and grantmaking organisations are largely UI-exempt. So QCEW growth is trustworthy broadly and the exceptions are a named short list, which is a better method than a flat haircut. Construction at 97.1% is a warning rather than a reassurance: the count agrees because both sources count the same workers, and the trade-versus-structure-type disagreement that actually matters is invisible to a coverage ratio. Three lookups located: RfHhInstComp W151RC is 18,684, the SUT's 814000 to the dollar - the sector QCEW covers worst is the one NIPA hands over directly, and the plan had described this line without finding it. U32500 splits general government compensation into wages and supplements, which 3.10.5 only totals. 6.6D wages per FTE is the plausibility check Phase 5 lacked: a share can be positive, sum to one, and still imply an implausible wage. Adding those tables exposed two defects in bea_nipa_parse: - Scale and unit were assumed, not read. A flat `* 1000000` with Class and Unit hardcoded to Money/USD is right only while every declared table is in millions of dollars. 6.4D/6.5D are thousands of persons and 6.6D is a ratio, so 6.4D's 150,654 thousand employees would have been published as 1.5e11 USD. Both now come from each series' own MetricName/DefaultScale, with an explicit map that raises on an unrecognised metric rather than defaulting to dollars. Every dollar table BEA publishes has DefaultScale -6, so all 1,812 dollar rows are bit-identical across the change. 1.14's three chained-dollar lines move to Class Other, so a Class: Money selection cannot add real dollars to nominal. - Table ids were matched by substring. Asking for U70205 also returned U70205S's 44 physical-quantity and price-index series under their own id. Nothing downstream selected them so it never showed, and it was the metric guard above that surfaced it. Matched exactly now. NIPA_final_dom_uses_2017 still rebuilds bit-identical, and 2012/2019/2020/2024 extract cleanly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Corrects an overstatement in the previous commit. T00TOP/T00SUB were described as "not sourced here ... building them would estimate the same money twice". That was wrong. They are already built by commodity in Step 4d, so the industry row is a transformation of money that exists, not a second estimate - and the plan's own coordination note said exactly that: "build once, derive the other through the Make/Supply structure". So the open question was never whether to derive it, but whether the derivation works. Now measured for 2017, where the published answer exists. It does not. Using the benchmark market-share matrix from the Supply table, D[c,i] = V[c,i] / T007[c]: T00TOP (TOP + MDTY) correlation 0.202 |error| 114.6% of the row T00TOP (TOP only) correlation 0.211 |error| 109.5% T00SUB correlation 0.676 |error| 79.8% The reason is structural rather than noise. 55.7% of the published T00TOP row sits in wholesale and retail industries, because a tax on a product is remitted by whoever sells it while market shares place it with whoever makes it. The pairs are stark: petroleum wholesalers 424700 are 88,362 published against 13 estimated while refineries 324110 are 397 against 92,893; motor vehicle dealers 441000 are 45,947 against 2,301 while assemblers 336111+336112 are 26 against 24,141. The whole tax moves one stage up the chain. This is measured in the benchmark year, against the table the mix comes from, so it is not the "2017 ratios drift" objection the 2026-08-17 decision was argued from - it is stronger. A conversion that fails in 2017 cannot be rescued by being applied nearer to 2017. T00SUB fails differently and more usefully: its residual is two named structures, not a smear. S00203 other state and local enterprises is 19,471 published against 1,964 (transit operating subsidies) and S00102 is 6,339 against 102 - subsidies paid to an operator rather than attaching to a product, which no product-side operator can place. T30800 already carries them. One piece converts exactly: published T00TOP on 4200ID is 38,513 against a Supply MDTY of 38,507, so customs duties are a lookup rather than an allocation - 5.1% of the row, free, every year. The decision stands: the industry distribution stays free for Step 5 under economy-wide soft targets. But free is not unseeded, and the better operator is the margin structure rather than the Make matrix - a commodity's wholesale and retail margins say which trade industries handle it, which is the point-of-sale signal the tax follows. Those margins are built, so that operator is testable the same way. Left as the open follow-up. Findings carried as a --check flag rather than unit tests, per the convention for analysis modules. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ADME Three passages still read as though T00TOP/T00SUB were simply out of Step 2's scope, which was the overstatement the previous commit corrected in the Step 2 section but not elsewhere: - the framework status table said "Not sourced", where the truth is that they are built by commodity and the industry row is a conversion Step 5 solves; - the board-provenance note said they "left Step 2's scope entirely"; - the data-gap section already said "Step 2 still supplies a seed" and was right all along - it now also carries what the seed can and cannot contain, since the market-share operator was measured and fails at r=0.20. Adds tax_axis_conversion.py to the analysis README index. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ersion Only within wholesale, and only for one code. The previous commit left "ride the margin structure instead of the Make matrix" as an untested proposal. Testing it turned up something better: Step 4c already computes the operator this needs. top_by_level splits TOP per commodity into producer-level (325,829) and trade-level (391,096) from an identity with nothing modelled in it - excise sits in Producers' Value, sales tax inside the margin columns. That is exactly the producer-versus-seller distinction market shares get wrong, and its trade-level total lands within +2.2% of published wholesale-plus- retail T00TOP. market share on all TOP + MDTY corr 0.204 |err| 114.6% + level split, trade-level by trade output corr 0.743 |err| 41.9% + motor fuel routed to 424700 by name corr 0.946 |err| 29.9% On the question itself, the answer differs by block: - Non-trade industries are 44.3% of the row and need no matrix at all. Once the producer-level portion is separated, plain market shares give corr 0.987. - Within retail, no. Output shares - which for a trade industry are very nearly margin shares - give 0.744. Retail product tax is general sales tax: broad-based, HHI 0.137, an effective 7.3 of 9 industries carrying it. - Within wholesale, yes, and output shares are worse than useless: corr -0.192. 424700 petroleum wholesalers takes 51.3% of wholesale product tax on 3.4% of wholesale output, because wholesale tax is motor fuel excise rather than a broad-based tax. HHI 0.321, an effective 3.1 of 10. But wholesale does not need a commodity-by-trade-industry matrix either. It needs one named routing: NAMED_TAX_LINES already carries motor fuel as 324110 and trade_level_share already says that tax is 99.8% trade-level, so sending it to 424700 takes wholesale from -0.192 to 0.973 and the whole row to 0.946. With petroleum pulled out by name the other nine wholesale industries score 0.825 on output shares - they behave like retail. So the general commodity-by-trade-industry margin matrix that the PRO:PUR producer-price work will eventually need is not required here. Still a seed and not a target: 29.9% absolute error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Construction was the right last block to check and the wrong one to worry about. It is block-shaped like the trade industries and could have been a second petroleum; it is the opposite. Commodity TOP is 1,907 against a published T00TOP of 1,857 and plain market shares give corr 1.000, |error| 31 = 1.7%. The reason is the Make matrix. BEA defines the construction industries by type of structure, so the block is 94.5% in-block and 100.0% diagonal - there is no producer-versus-seller distinction when whoever builds the structure sells it. top_by_level agrees: 100% of construction TOP is producer-level, so the routings that rescue wholesale are inert here and all three operators give one number. MDTY and SUB are zero on both axes, so there is no duties or subsidy question in the block at all. Tax sits on 3 of 12 codes, none a NAMED_TAX_LINES entry. The residual 1.7% is a leak, and market shares get its direction right and its size about half: 5.5% of construction output is own-account or secondary work (531HST 20,279, state and local government 19,991), and published moves 50 off the block where market shares move 30.4. That answers the larger question of whether the remaining sectors each need this treatment. They do not - the error is 20 industries, not 402: top 5 industries carry 35.1% of the error, top 20 carry 80.3% 17 of those 20 are wholesale or retail wholesale 22.8% of the row 41.9% of the error retail 27.8% 33.7% non-trade 44.3% 24.4% 4200ID 5.1% 0.0% What is left is the within-trade allocation already characterised plus five named non-trade structures: 721000 accommodation (-6,592), 517210 wireless (-4,310), 221100 electric power (-3,934), and government enterprises handed tax they do not carry (S00202 +3,439 against a published zero, GSLGE +1,698, both of which belong to the Step 7 reallocation). So build against the seed and repair the named twenty later - Step 5 balances these cells under soft targets anyway, and seed accuracy below the block level is not what the build is waiting on. Adds construction_codes, construction_scores and error_concentration, wires both into the report, and adds five assertions to --check: construction stays usable, its error stays under 10%, its Make block stays diagonal, it stays MDTY/SUB-free, and the top-20 error share stays above 60% - that last one fails if the residual ever spreads and a broader sweep becomes necessary after all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BEA books zero taxes on production to all ten government industry codes - T00OTOP and T00TOP are both zero on every one of them in 2017, the single exception being 538 of T00TOP on S00203. The columns are real (V00100 and VABAS are populated), so the zero is an accounting rule and not a gap: a tax levied by government and remitted by a government producer nets out. The market-share leg was violating that rule by 10,513 against a published 538, because government genuinely does produce taxed commodities and market shares hand it a share of the tax on them: S00202 S&L electric utilities 3,439 against a published 0 S00203 other S&L enterprises 2,428 against 538 GSLGE S&L educational services 1,698 against 0 GSLGO S&L other services 940 against 0 S00101 federal electric utilities 854 against 0 market_share_matrix gains an exclusion that drops those columns and renormalises each commodity over the producers that remain, so the tax stays with its own commodity rather than being deleted or smeared economy-wide. That is the fourth operator in the progression: + motor fuel routed to 424700 by name corr 0.946 |err| 29.9% + government columns zeroed, renormalised corr 0.948 |err| 27.9% It moves 15,692 of error, and only two thirds of that is the excess itself. The rest lands where it belongs: 221100 electric power |err| 3,934 -> 443 721000 accommodation |err| 6,592 -> 5,088 622000 hospitals |err| 763 -> 296 Read 221100 against S00202: the previous operator was under-attributing private electric power by almost exactly the amount it had misplaced onto the government utility, so the money was misplaced in a recoverable direction. The non-trade block goes from corr 0.987 / 24.4% of the error to 0.992 / 18.8%, and no government code appears in the top-20 error ranking any more. Only the producer-level leg needs the exclusion - the trade-level leg lands on wholesale and retail, where no government code sits, and duties land on 4200ID. No commodity is stranded: no commodity produced entirely by government carries any product tax in 2017, so the fallback that keeps such a row unrenormalised never fires, and the seed total is unchanged to the dollar. This one was worth fixing before the build rather than after. The other residuals are misallocations Step 5's balance can pull back; these columns are redistributed into private industries by the Step 7 government-enterprise reallocation, so a wrong seed here propagates into work that would have to be unpicked. Adds government_industries and government_scores, and four assertions to --check: the published rule still holds, the seed puts nothing there, the exclusion still reduces the row error, and renormalisation still moves money between industries without creating or destroying it. _frames is now cached - the scoring functions call it a dozen times over and --check had grown to over two minutes; it is 9s now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Step 2's third row was scoped as "no NIPA table has an industry axis for this, accept a cruder method". The first half is true and the second does not follow: measured, T00OTOP is the best-behaved of the three rows. T30500 puts 88.1% of it in recurrent taxes on immovable property, so it is a property tax and corporate income tax is not in it at all - that lands in V00300. Which means the intuitive allocator is the wrong one. Scored against the published 2017 detail row, industry output gives correlation 0.590 and an absolute error of 92.3% of the row, missing 531HSO alone by 150,567; the effective rate runs 0.49% at p10 to 15.20% on 531HST. What works instead is that the row is concentrated - three real-estate codes carry 46.3% - and BEA publishes the big cells. T70405 B1031C is the 531HSO+531HST pair to the dollar and T70305 B1017C the ten farm codes within 3, and neither is a 2017 coincidence: against the summary SUT they hold in six and seven of the eight years 2017-2024. The remainder rides frozen 2017 shares, graded out of sample at 1.9% composition drift against a 40.5% level move. The summary SUT is evidence here and never an input, which is the role Step 5's Decision 3 reserves it for. Also lands the two pieces of Step 2 plumbing every one of the three methods needs: the three BEA_2017_Code identity crosswalk rows (Phase 0), and assign_use_row_from_clean_parameter - the transpose of the final-demand helper, since value added is a Use row rather than a Use column. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NIPA_VA_othertax_2017 is the first of Step 2's three methods and the one that proves the seam the other two need: the BEA_2017_Code identity crosswalk rows, Sector_Crosswalk_BEA_NIPA_VA.csv, the melted BEA_Detail_Use_SUT as an attribution source, and the Use-row transpose. One activity set. The T30500 control (LA000365 state and local other taxes on production, plus LA000237, the federal remainder after taxes on product) attributed proportionally across the 392 non-government industries on the 2017 benchmark T00OTOP row, then transposed onto SectorProducedBy. Replaying 2017 gives correlation 1.0000 against the published row, 389 industries of 389, zero on government, and the control's own 9 of rounding. Five tests pin it; NIPA_final_dom_uses_2017 rebuilds bit-identical across the crosswalk change. The housing and farm lookups are NOT in it, and that is the finding rather than an omission. They cannot be their own activity sets: NIPA states no "other taxes on production excluding housing and farm" line, so a third set's control would still be the whole 608,533 and the three would sum to 872,044. Folding them into the weight vector instead needs an FBS_outside_flowsa attribution source, and that path does not work - get_flowby_from_config builds a FlowBySector for that data_format while attribute_flows_to_sectors then calls map_to_sectors, which only FlowByActivity defines. All four existing uses of the hatch in the repo are top-level sources. That matters well beyond this row: the plan commits V00100's anchor-and-move build to the same hatch, so it has to be cleared before Step 2's largest row. Finding it on the 1.8% row rather than the 30.9% one is the argument for having built T00OTOP first. Here the cost of going without is bounded and measured - 1.92% against 1.68% of row error in 2024, less than the control's own 2.9% vintage error in 2021 - and lookup_improvement() reproduces it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All three of Step 2's value-added rows now generate, and the use_va_detail_sut diagnostic is switched on: every cell of the 3 x 402 block matches, worst 2.6 million on a 178 billion cell, which is BEA's own rounding. derive_initial_value_added stacks the three methods. V00100 runs on the 69 T60200D industry groups as separate controls rather than one national total, because those 69 leaves partition the 71 BEA summary industries exactly - no gaps, no overlaps - and 63 of them equal a summary industry's published compensation to the dollar. Verified by value rather than by name. So the frozen 2017 shares only have to hold within a summary industry, which is a far smaller claim than a national spread. Correlation 1.000000 against the published row, 400 industries of 400. That build reverses the plan's headline decision, and on measurement. Splitting wages from supplements was meant to avoid imposing one industry's supplement rate on another. It cannot: T60300D matches T60200D at 69 groups but T61000D and T61100D publish only 16, so the split imposes a coarse group's rate on industries whose own rates differ and misplaces 99,025 - 0.95% of the row - against a T60200D-only method that is exact. NIPA publishing both halves by industry does not help when it publishes them at coarser grain than the total. V00300 is eight controls across five tables on one industry distribution, and stays plain deliberately. Four of its components have industry tables on mutually incompatible partitions (20 / 21 / 12-plus-financial / 63 groups) and three have no axis at all, so a component build imposes four coarse partitions at once - the same mistake, four times over. It keeps S00201 at its published -36,919; nothing here may assume positivity. The best next source for Step 2 is a value added by industry extractor. V00300 = VABAS - V00100 - T00OTOP, and two of those three are now built at detail, so BEA's GDP-by-Industry accounts would give V00300 by 71 industry groups as a residual - better than assembling eight incompatible component tables, and ahead of Fixed Assets because it serves the 41.6% row rather than one 40% component of it. The summary Use SUT carries the same thing and Decision 3 holds it in the test set, so it cannot stand in. 2018-2024 are not written: every method holds its within-group shares at 2017, which is complete for 2017 and not for later years. Compensation needs the moved-share source, still blocked on the FBS_outside_flowsa gap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BEA_GDPbyIndustry reads table TVA113, Components of Value Added by Industry, out of BEA's GDP-by-Industry release archive: value added at producer prices plus compensation, taxes on production and imports less subsidies, and gross operating surplus, for 100 industries, annually. No API key - BEA's GDPbyIndustry API dataset needs a UserID and this archive does not, so it follows BEA_NIPA's pattern of caching one archive under extract/input_data with a GCS fallback. 2012-2024 declared, 1997-2024 available. The root identity holds exactly in every year checked: compensation + taxes + surplus = value added. This was asked for because V00300 has no NIPA table of its own and its eight component tables publish on mutually incompatible industry partitions. TVA113 does state gross operating surplus by industry directly - so it looked like the one source that would give V00300 a real industry axis. It is not, and the extractor is what proves it. All 71 BEA summary industries' V003 match a TVA113 surplus row to the dollar, and all 71 V001 match compensation the same way. BEA's industry accounts and the SUTs are the same estimates published twice, so this is the summary Use SUT by another door - and Step 5's Decision 3 holds the summary SUT out of the target set precisely so it can grade the build. Two tests pin that, so the finding cannot quietly stop being true. NIPA_VA_surplus is therefore left unchanged. Wiring it in is a two-line swap of the attribution source; what it buys is a near-exact V00300 seed at summary grain, and what it costs is summary V003's ability to grade that seed. That is a testing-strategy judgement rather than a data-availability one, so it is documented and left open rather than taken. The extractor is worth having regardless: it makes the series reachable as a versioned FBA rather than a private workbook loader, and it is the natural loader for the test set itself. It also corrects the recommendation the previous commit made - "the highest-value missing source" was the wrong framing, since it was never missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Step 2's section had a target and no picture. It has both now: use_va_detail_sut_2017.png, rendered in the same style as the Step 1 and Step 4 images, plus the counts, coverage and accuracy tables the other two sections carry. Every section in the diagnostic now has a candidate; none carries candidate=None. The block is 100% coverage and 100% accuracy, and the report says at both the summary table and the section that this is NOT comparable to the other two rows. All three methods take their within-group distribution from the 2017 benchmark, which is the reference, so a 2017 run tests the plumbing and not the estimate - solid green is the floor, not an achievement. What it does certify is that the orientation transpose, the identity crosswalk rows, the 69-way compensation control set and the eight-line V00300 assembly carry mass end to end without losing or misrouting it: the grand total is off by 0.000016%, three million on eighteen trillion. The picture earned its place on the first run. Reading the 17 absent cells showed T00OTOP is absent on eleven government codes, not the canonical ten: the prefix rule (S00, G) misses the US Postal Service, 491000, a federal government enterprise whose BEA code is shaped like an industry's. It was landing on zero only because the benchmark weight happened to be zero, not because the rule excluded it. Now named in write_value_added_crosswalk.py. No 2017 number moves - which is why a total would never have found it. Also shortens the section note so the plot header stays one line, matching the other two images. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BEA_GDPbyIndustry and BEA_Detail_GrossOutput_IO read sibling zips from the same BEA release page, and GrossOutput.xlsx is duplicated between them - gdp.py gets it from GCS as a manually extracted workbook pinned to 2025Q2, BEA_GDPbyIndustry fetches the summary archive straight from BEA. Filed as #694; this just leaves a pointer at both ends so the next reader does not re-derive the relationship. Records the two things that make it more than a file move: the 2025Q2 pin is deliberate and the archives are currently different vintages (June 2026 against September 2025), so dropping it would silently move gross output under electricity disaggregation and derived_gross_industry_output. Also records the negative result, since it is the reason V00300 stays at summary grain: UGdpByInd's ValueAdded.xlsx has no counterpart to TVA113. UVA201-UVA210 are indexes, real value added and UVA205 value added totals - there is no components table, so compensation, taxes and gross operating surplus are published at summary grain only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pytest names a test module by walking up from it until it reaches a directory without an __init__.py. Both bedrock/transform/nipa/__tests__ and bedrock/utils/taxonomy/mappings/__tests__ carried an __init__.py while their parent directories did not, so both claimed the top-level module name __tests__. Whichever pytest collected first won, and the other failed to import: ModuleNotFoundError: No module named '__tests__.test_bea_ceda_v7__cornerstone_commodity' Removing the empty NIPA __init__.py puts those tests on the convention the majority of __tests__ directories here already follow, giving them the bare module names test_nipa_va_othertax and test_nipa_va_rows, both unique across the repo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Step 3 freezes every industry's input structure at 2017 and carries it on a price index. For manufacturing the part of that column the annual surveys cannot refresh is the materials bill: AIES publishes all materials, parts and supplies as one cell, 82.5% of the column, so only 8.3% of manufacturing's intermediate is commodity-mappable annually (#564). The commodity breakout is quinquennial Economic Census, and 2022 is a second observation of it sitting close to the middle of the 2018-2025 span. #564 called that a consolation prize. Measured, it is the main prize. Census_EC_MatFuel pulls ecnmatfuel for both vintages -- 4,624 rows in 2017 and 4,399 in 2022, every industry at NAICS-6 and every material an 8-digit code. The orientation is the opposite of Census_EC_PxI on purpose: PxI asks what an industry sells, this asks what it buys, so the industry goes in ActivityConsumedBy and the material in ActivityProducedBy, which is the Use table's own orientation. materials_structure.py answers the two questions that decide whether the source is worth having. Coverage: 66.2% of the 2017 materials bill and 69.1% of 2022 is placeable on a BEA commodity, against 8.3% annually -- 52.9% and 54.0% resolving 1:1 by NAICS prefix, the rest onto a BEA group that needs a within-group split on 2017 Use shares. A third is residual buckets Census could not place, and that is the ceiling. Movement: the mix moved 0.153 between the two censuses against 0.173 for the whole Use column over 2012-2017, with 264 of 345 industries moving more than 10 points. So the largest and least-observed part of the manufacturing column moves as fast as the rest of it, and freezing 2017 out to 2025 discards a reallocation this source can see. Two traps are documented rather than worked around. 00772000 "Total Materials" is the industry total and the named codes sum to it exactly, so summing the FBA unfiltered doubles the table; it is kept because it is the control a suppression recovery subtracts published children from, the same role NAICS 00 plays for PxI. And the vintages sit on different NAICS bases, sharing 345 industries and 291 materials carrying 90% of each year's cost -- 336411 aircraft reallocating 59% of its materials bill is almost certainly a code reassignment, not economics, and is flagged as suspect. Not built yet, and named in the plan: suppression recovery against the 00772000 control, the group-tier within-group split, the vintage code diff, and the interpolation itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ted (#698) Census withholds 412 of 4,624 cells in 2017 and 330 of 4,399 in 2022. They are not zero: they sit inside each industry's published 00772000 total, and leaving them there biases a materials mix toward whatever happens to be publishable, which is systematically the large materials. The control turns out to be exact, and that is measured rather than assumed -- for every industry with nothing withheld the named materials sum to 00772000 to within 0.1%, 238 of 238 in 2017 and 247 of 247 in 2022, with fuels carrying their own exact control in 00772002. After recovery all 406 and 386 industry-by-kind controls close to within 0.1% and no negative cell is created; the one negative in the output is published Census data. The prior is chosen by holdout rather than by argument. An economy-wide prior was tried first and is visibly wrong: it hands an idiosyncratic industry the economy's shopping list, and put $8.6bn of motor vehicle seating into aircraft manufacturing while cutting its aircraft engines from $15.5bn to $0.5bn. Masking published cells and recovering them scores every peer-prefix length; NAICS-3 wins on average, at WAPE 0.602 and 0.718 against economy-wide's 0.640 and 1.033. Cross-vintage priors are deliberately excluded even though 2017 is the best predictor of a withheld 2022 cell, because filling 2022 from 2017 biases the movement measurement toward zero -- a recovery must not manufacture the answer the analysis is testing. That WAPE is the finding that matters, and it corrects the last commit. The mass a recovery places is exact, so all 0.6-0.7 of that error is allocation across materials within the column -- which is exactly what a mix score measures. Restricting to the 193 industries with nothing withheld in either year, the materials mix moved 0.1330, not the 0.153 reported before, and the count of columns over 0.25 collapses from 66 to 17. Most of the extremes were the fill, not the economy. 336411 aircraft at 0.592 was the loudest of them and chasing it is what found the defect: its 2022 column is mostly withheld. The argument survives, weaker and better supported: materials mix moves 0.133 over five years against 0.173 for the entire Use column over 2012-2017, with 133 of 193 clean industries moving more than 10 points. So the largest and least-observed part of the manufacturing column moves substantially -- somewhat less than the column as a whole, not more, which is what the previous commit claimed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…lation (#698) The last three #698 items are built. Two dissolved a problem the plan expected to fight; the third overturned the interpolation form the plan had already chosen. materials_structure.py becomes inputs_structure.py, because what these sources reach is no longer just materials. The group-tier split is settled on evidence rather than on which prior sounds more principled. A group cell is divided over the BEA commodities its NAICS could be, on the purchasing industry's own 2017 Use row; scored by demoting every direct cell one prefix and comparing against the commodity Census actually named, the column prior puts 72.0% and 72.9% of the money on the right commodity against 46.9% and 49.5% for an economy-wide one. Accuracy falls off with group breadth -- 79.8% at 2-4 commodities down to 51.9% at 10-29 -- but 73% of group-tier dollars sit in groups of nine or fewer. The bare 33 prefix, 136 commodities, should be read as barely better than residual. The split lifts the placeable bill from $2,097B to $2,681B and the commodities reached from 137 to 204. The vintage code diff turns out not to be a diff. The plan expected to lose 10% of each year's cost to the 2017-2022 revision and it loses none: the material axis shares 289 of 289 and 290 MATFUEL codes, and every off-frame dollar on the industry axis is NAICS 2022 merging pairs of 2017 codes. Connected components of the year concordance put 100% of both vintages on one 365-industry basis with no split assumption. This does not rescue 336411 -- aircraft was on the shared frame all along, so its score is the suppression fill and the plan's guess about why it mattered was wrong. Linear interpolation was called "the obvious first form". It is obvious and it is wrong, and seeing that meant stopping treating 2018-2025 as unobserved. Manufacturing's materials bill is published every year the census misses, so Census_ASM_Expenses and Census_AIES_Expenses now pull it -- ASM through 2021, AIES for 2023. A straight line overstates 2020 by 28.8% because it cannot bend around a pandemic, and past 2022 it gets the sign wrong: the bill fell 6.8% into 2023 and the line says it rose 5.0%. That is the span the nowcast leans on hardest. Scope has to be matched or a definition reads as a growth rate -- against CSTMTOT the census-to-ASM step is a median 1.181, on materials-plus- fuels 1.063, a year of inflation. Those extractors also buy more than a control. ASM and AIES publish electricity, contract work, resales and 9-12 named purchased services at NAICS-6, 11.7% of the manufacturing column, each mapping onto a BEA service commodity -- 91.0% of the column reachable in total, and a separate cheaper task with no suppression recovery or group split to do. AIES's expense block is manufacturing-only, though: sectors 21, 22, 23 and 51-81 publish nothing at any NAICS level, so it confirms #564 on the service drifters rather than overturning it, and it does not cover the mining that the census does. The headline moves down again, and this time because the frame was wrong. 0.133 is a MATFUEL-code score and the 0.173 it was compared against is a BEA detail commodity one. On the same frame the clean subsample gives 0.0941, so the materials block moves roughly half the column's rate rather than "somewhat less". Aggregating 289 materials onto ~200 commodities nets off within-commodity substitution and the split holds 2017 structure fixed inside each group; both are properties of the seed, not corrections to 0.133. Finally, BEA's 2022 and 2023 tables are still annual-survey updates carried over the 2017 benchmark, so differencing a census-seeded block against them is not a check. That cuts both ways, and the second way is the argument for this work: it is information BEA has not yet incorporated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…, #664) S3b was scoped as the cheap half of Step 3 -- ten named cells, no suppression recovery, no group split. It was cheap, but not for the stated reason: the work turned out to be a scope measurement rather than a mapping exercise, and two of the claims it rested on were wrong. The first is the coverage. 11.7% and 91.0% were survey-side dollars, and two of the largest entries are not purchases of a commodity at all -- resales are goods bought and sold on untransformed, which the Use table handles through trade margins, and contract work is manufacturing services whose commodity is the buyer's own industry rather than any fixed row. With the survey's own residual they are reached as expense but cannot be placed. The honest figures are 6.4% seedable and 85.8% reachable, and the seedable block is $228.6B over ten cells. The second is that these cells could be read as levels. They cannot. Census_EC_ Expenses is new and pulls ecnbasic for 2017 and 2022, which publishes the cells under the same variable names ASM uses -- so census and survey form one panel with no crosswalk, and the splice is continuous: 2017 electricity $47.5B against ASM's $51.0B in 2018, repair $52.7B against $55.2B. That 2017 observation is the year the benchmark Use table is built on, and against it the survey and BEA disagree about what the same cells contain by factors of 0.40 to 8.01. The disagreements are structural, not noise: expensed software and computers are operating expense to Census and mostly investment to BEA, repair is one Census question against four BEA rows carrying parts BEA books elsewhere, and professional services runs the other way, one question against BEA's legal, accounting, engineering, consulting and R&D rows together. So the seed moves BEA's cell rather than replacing it -- Use2017 times survey(t)/survey(2017) -- which cancels every one of those because a constant scope factor divides out, and preserves BEA's own level and its own split across the commodities of a multi-row kind. 2017 reproduces the benchmark exactly, which is the check that the form is right. The block carries the same pandemic signature the materials bill does, -1.7% in 2020, and a frozen 2017 understates it by 23.8% by 2023. AIES publishes no telephony and no expensed software -- both variables exist in the 2023 table and both are zero in all 883 rows -- so those two are held at the 2022 census and marked rather than seeded as a collapse. 2024 and 2025 raise rather than quietly extrapolating. Chasing scrap through the same sources then found a defect in the materials placement. Every MATFUEL scrap code begins 33, which is not a NAICS that maps to any single commodity, so the prefix walk was filing purchased metal scrap into the bare 33 group of 136 commodities and smearing it across most of manufacturing. That group was the weak end of the group tier, flagged in the last commit as barely better than residual -- and it turns out to have been entirely scrap. BEA carries S00401 for exactly this concept, $49.1B into manufacturing in 2017, and Census's "excluding home scrap" is the same thing: bought in rather than generated on site. The five codes now map straight onto S00401 ahead of the prefix walk, the 30+ band collapses from $32.1B/$54.9B to $2.5B/$0.6B, and the clean commodity mix score moves 0.0941 to 0.0949. The direct+group frame reaches 200 commodities rather than 204, and the four lost were reached only through the smear. Scrap is metal and only metal in this source -- no wastepaper, no cullet, no plastic regrind, no textile rags. Census reproduces BEA's scrap concentration independently and more finely, separating secondary aluminium at 0.628 from secondary nonferrous at 0.529, and iron and steel mills move 0.328 to 0.440 between the vintages. The fuller picture, including the output side where ecnpxi does carry paper and plastics as wholesale recyclable sales, is written up in cornerstone-data/methods#59. Census_EC_Inventories is extraction only, for #664. ecnbasic carries all three stages of fabrication plus the totals, beginning and end of year, at NAICS-6 for both vintages -- which is the industry x stage cell BEA does not publish anywhere, and the level Hill's rules operate at. Verified additive: the stages sum to the published total at a median ratio of exactly 1.0000. These are stock levels and differencing them imports the holding gains CIPI excludes, so they carry the stage shares and U50705BU1 keeps the level. The ASM annual equivalent is deliberately not pulled yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan carried 2024 and 2025 as years the estimate had to reach and could not source, which framed an unobservable extrapolation as a risk this work owned. It does not. AIES 2024 still returns 204 No Content, ASM ends at 2021 and the census is quinquennial, so the observed panel runs 2017-2023 and the estimate stops there. Extending it to 2024 is #707, Phase 2 work tied to producing a 2024 table, and 2025 is out of scope entirely. So the open question in S3 is now only which interpolation form to fit, scored on observed years, rather than which form to fit plus how far to extrapolate it past the data. S3b needs no extrapolation at all: its span is fully covered, and nonmaterial_seed() already raises for later years rather than inventing one. unobserved_years() keeps reporting 2024 and 2025 because that is still the fact a caller wants to assert on -- what changed is that they are outside the span rather than gaps inside it. Left alone deliberately: the references to BEA's published summary panel running to 2024, gross output extracted for 2017-2024, the theta fitted on 2022-2024, and FIWS covering to 2025. Those are statements about which data exists, not about how far this estimate reaches, and the column-scaling argument in particular is quantified on a 2024 seed and would lose its point without them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#705 asked, per drifting column, what could source it. Every one of them already had a source - BEA named one for each at the 2017 benchmark - so the question was whether a later vintage exists on the same basis and beats holding BEA's answer. Four no, one marginally yes. Census_SAS_Expenses splices SAS Table 5 across the two vintages that carry it, sas-17 (2013-2017) and sas-22 (2020-2022). #564 recorded Table 5 as "2020-2022 only"; that is the latest workbook's display window, not the series. 63 industries at 2- to 4-digit NAICS, ~19 mappable items. The two vintages sit on different Economic Census benchmarks, so every row carries the benchmark it was built on in Description. service_expense_seed.py indexes BEA's 2017 531ORE column on it and scores against BEA's current summary Use: +4.4 / +3.8 / +4.5% at 2020-2022, positive at every endpoint on a test biased against the seed, but at the inflation carry's bar rather than over it. --reachable says why: 18.07pp of ORE's movement sits on rows no survey item names against 13.51pp that a seed can touch, and the largest single mover has no counterpart question at all. Two corrections to earlier work in this branch: - An exploratory cut of this score reported +18.9 to +24.7% by applying each item's index to whole summary rows. Temporary staff maps to 561300, $8.6B of the column; at summary it multiplied all of 561, $97.8B, mostly 561700 services to buildings. A coarse commodity mapping inflates a result rather than blurring it. Build at detail, aggregate to score. - _load_usa_summary_sut pins the workbook by year, so --drift's series changes basis at 2023. The same 2022 read from both vintages differs by a dollar-weighted 0.0557 against a measured drift of 0.0986, and by 0.0976 for ORE alone. Near-misses elsewhere on the page are inside that noise. The seed scores on one vintage; fixing the shared diagnostic is follow-on work. Also records the negative results: the trade Business Expenses Supplement exists for 2017 and 2022 on one benchmark but loses every item of 4A0 to suppression; construction's ecnbasic pair is clean but 51% of the column is one undifferentiated materials cell; and ecnpurmode, ecnpurelec and ecnpurgas publish concepts BEA has no cell for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ives (#705) intermediate_structure_drift read summary Use through io_2017._load_usa_summary_sut, which pins the workbook by year: 2017-2022 from the 2017-2022 release, 2023-2024 from the 1997-2024 one. That is right for FBA consumers, whose published values must not move under BEA's revisions, and wrong for a module that differences years against each other -- it put a vintage seam between 2022 and 2023 in the middle of --drift's series. Read every summary year from the current workbook instead, via this module's own summary_use / summary_intermediate(year, workbook). io_2017's year-pinning is left alone. service_expense_seed.summary_intermediate_current, which had made a local copy of exactly this fix, is deleted in favour of the shared function; its scores are unchanged, confirming the two were equivalent. Add --revision, the same year read from both vintages. The revision table in intermediate_estimation_plan.md was not reproducible by any committed code, which broke that page's norm; the flag reproduces it exactly. The question this was for: does #705's top-drifter ranking survive one basis? It does, bit-for-bit. The seam sits between 2022 and 2023, but 2017 and 2018 are identical across the two vintages (one cell, $3M) and 2024 was only ever read from the current workbook, so the 2017-against-2024 comparison never crossed it. ORE is still 0.141 and the candidate list -- ORE, GSLG/GFGD, 42, 5412OP, 81 -- needs no revisiting. The feared shift, that GFGD and 521CI revise by more than ORE does, could not bite: the revision only starts at 2019. What did move is the middle of the series, understated by 0.005-0.011: 2019 0.0452 -> 0.0504, 2020 0.0743 -> 0.0833, 2021 0.0711 -> 0.0823, 2022 0.0838 -> 0.0859. 2018, 2023 and 2024 are unchanged. The 2022 ranking does not survive -- four columns in, four out of the top ten, ORE 0.084 -> 0.158 -- but nothing on the page ranks at 2022. Two numbers corrected rather than restated. The plan doc and the seed docstring both quoted a 2022 drift of 0.0986 that no code reproduces on either basis; replaced with the measured 0.0859. The seed docstring said the gain was 4.4% at 2022 where the doc and the code both say 4.5%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Margins.2 specified the margin rate as (TRADE + TRANS) / T013 -- margins over *basic* value. BEA gross output is valued at producers' prices (confirmed), so the price ratio #497 carries already contains the product-tax layer and the rate must be taken over producer value: T014 / (T013 + T015). One factor, not two. Also record the valuation chain the section had been using implicitly. T014 is the margins alone, not a running subtotal: T016 = T013 + T014 + T015, verified to $1M at detail, and independently stated in margins_estimation_plan.md. Restated on the right denominator. The level error from using basic is a median 3.3% and lands hardest on the commodities this section quotes: 315AL apparel 1.372 not 1.793 (+31%), 324 petroleum 0.246 not 0.289, 313TT textiles 0.779 not 0.858, 311FT 0.489 not 0.533. The panel is 26 commodities with a rate above 1%, not 36; median absolute change 3.6pp and p90 12.7pp, not 2.8 and 12.1. Most of the level error divides out of a ratio, so the correction to the carry factor itself is second-order -- median 0.35pp across the 26 receiving commodities above $20B, p90 1.4pp, at most 2.1pp among named ones. Worth having, not decisive; the section now says so rather than implying the fix is large. New guard, which the section needed and did not have. For margin *suppliers* T014 is large and negative -- the margin is allocated away from the trade or transport commodity onto the goods it carries, which is why the columns net to zero -- so mu is -0.94 for 42 wholesale, -0.99 for 486 pipeline, -0.88 for 482 rail. 1 + mu is then 0.06, 0.01 and 0.12 and the factor is a ratio of two near-zero numbers. Set it to 1 wherever mu <= 0. Nearly free: those rows carry almost no dollars in the purchaser-priced intermediate block, for the same reason their mu is negative. Finally, name the experiment. A missing deflator term and substitution under relative-price dispersion produce the same symptom -- a low theta on the summary panel against 1.00 on the detail one -- so theta must be fitted with and without this factor. If adding it pulls summary theta toward 1 the gap was the deflator; if not, the substitution reading stands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…egative Two results, and the second was not the one being looked for. Add the margin leg of the purchaser deflator -- summary_supply, summary_margin_rate and summary_margin_factor -- on the denominator fixed in 4fd1122: mu_c = T014_c / (T013_c + T015_c), margins over producer value, because BEA gross output is at producers' prices. Margin suppliers are held at 1.0, since their T014 is large and negative (42 is -0.94, 486 is -0.99) and 1 + mu is then a near-zero denominator. Add --theta, which fits the exponent with and without that leg. It was a genuine question: Margins.2 hypothesised that the summary panel's low theta was a missing-deflator artefact rather than substitution, and the two readings predict the same symptom. The hypothesis is wrong. Theta is unmoved in six years of seven and moves away from 1.00 in the seventh; the score differs by under 0.001 either way. Not a null test -- 26 of 73 commodities move, up to 10%, on the largest goods rows (325 chemicals x1.049 on a $705B row, 3361MV x1.047). It fails *because* of that: 22 of the 26 have a factor above 1, 17 of those lost intermediate share, and the touched set lost 3.26pp of the block. Both legs inflate nominal goods shares during real substitution away from goods. Keep the term as the correct deflator, worth a median 0.35pp on the carry factor; drop the claim that it explains theta. Then the larger result. THETA_GRID started at 0.0, which censored the panel: 2023 and 2024 both pinned to the floor and were read as "the carry contributes nothing". They fit -0.25 and -0.50. The frozen structure scores better when commodity shares are moved *against* their own price movement, so #497's theta = 1 is not merely too strong for the target years, it is the wrong sign. The grid now runs -1.0 to 1.5 and carries a warning about the floor. Detail 2012->2017 still fits 1.00 -- its optimum was interior, so nothing measured there moves -- and the +1.00 at 2020 to -0.50 at 2024 span is monotone in the price regime. Consequence for S1: theta must be a parameter, and its default for the recent span is negative rather than 0 or 1. A build that hardcodes 1 applies a correction pointing away from the answer on the block's largest rows. Guard added with the negative exponents: a zero ratio would now raise a fit to infinity rather than harmlessly to zero. No year has one today. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…704) S1 and S0b of intermediate_estimation_plan.md. `nowcast_intermediate.py` is #497 as scoped: seed the published 2017 detail Use SUT interior (402x402, purchaser, before redefinitions), carry each column's shares on the commodity price ratio at theta, renormalise, and scale the column to `GO_producer - VAPRO_seed`. `nowcast.derive_initial_U_intermediate` is the entry point. Runs 2017-2024, bounded by `BEA_Detail_GrossOutput_IO_<year>` rather than by the price index. S0b came with it rather than ahead of it: `use_intermediate_detail_sut` went into sections.py runnable, not `candidate=None`. The 2017 section run passes on every cell - 44,281 populated cells, 366 row totals and 400 column totals inside tolerance, 100% coverage and accuracy. That is the plumbing, not the movement: at 2017 every carry factor is 1.0. The 2017 rescale is not the identity, and the residual is BEA's own rounding. Published T005 is one rounded number; the interior sums 402 separately rounded cells to a different one - $350M on $14.9T, at most $13M on a column. A small column wears that as a large fraction, so `atol` carries those cells rather than `rtol`: 334610 is $482M of intermediates and is rescaled 1.05%, the largest relative error, against a largest absolute error of $6.0M on a $19.2B cell. `reproduction_check` reports both, because either alone reads as the wrong kind of error. The seven published negative cells survive in all eight years and are not clipped. The two structurally empty columns - 4200ID and 814000 - stay empty, and a control that puts real dollars on one raises rather than dropping them. So does a seed column whose nonzero cells cancel: an empty column has no structure to normalise, a cancelling one has structure that cannot be written as shares of its own total, and collapsing both to all-zero would lose that. theta is an argument, defaulting to #497's 1.0. It fits negative at 2023 and 2024; choosing it stays #699. The column control is well levelled and badly allocated. Step 2 is unbuilt, so `vapro_seed` freezes 2017's VA share of gross output, which collapses the control to `GO(t) x T005(2017)/GO(2017)` - gross-output movement and nothing else. Scored against the published summary T005 (`--control`): within 2.3% economy-wide in every year 2018-2024, but weighted MAE by industry runs 2.5% to 8.0% between 2018 and 2022, with GSLG 18.3% low at 2022. Two corrections to claims this branch had been repeating. GSLG topping that list is the column *level* and is not #578. #578 is the commodity *mix* inside the G* columns, sourced from govslocalfin's function x object split and sequenced at S5 behind a go/no-go. Separately, NIPA T31005 matches the published government T005 at $0/$0/$2M in 2017 and is referenced in no source file - so the worst cell of the control table is the one with an exact annual source already identified and never wired. And "Step 5 imposes both margins hard, so Step 3 estimates a shape not levels" holds only once Step 2 exists. Checked in nowcast_targets.py: T1 is hard and real for 2017-2024 but pins the column's sum, not the split. T4 and T6 are both soft and still PLACEHOLDER, and `va_row_targets` does `del year` and reads its values off `published_2017_panel`. T5 - T00OTOP and V00300 - is deliberately not imposed, entering as seed only so the income side stays out-of-sample evidence; V00300 is $7.873T. The balance cannot re-derive VABAS, and there is no VA seed for any year but 2017, so it cannot run on 2024 at all until Step 2 lands. The shape is insulated from this - it is renormalised before the control is applied, so a wrong control rescales a column without moving a share inside it - but the level is not, and the module and plan doc now say so. Tests are structural and run on a toy panel: the arithmetic (`carry_shares`, `apply_column_control`) is split from the data wiring, so they need neither GCS nor the gross-output parquet. The year-by-year numbers are CLI flags on the drift diagnostic (`--seed`, `--control`) rather than tests, so everything quoted above is reproducible from committed code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…control
BEA publishes intermediate inputs (UII205-A) and value added (UVA205-A)
annually 1997-2024 on its 191-row "underlying" industry frame, in two
workbooks sitting beside the GrossOutput.xlsx this repo already reads.
Nothing read either one. This extracts both, allocates them to the 402
detail industries, and replaces Step 3's frozen-2017 VAPRO seed.
The 191->402 mapping is derived, not hand-written. UGO205-A and UGO305-A
order industries the same way, so the 138 leaves of the 205-A hierarchy
partition the 414 rows of 305-A into contiguous runs; the runs are closed
by matching gross output in all 28 years. All 138 leaves match, exactly
414 of 414 detail rows are consumed, and the 402 codes cover the model
schema once each. derive_underlying_line_mapping reproduces the
checked-in constant exactly, and --mapping on the new diagnostic is how
to re-check it when the BEA vintage moves.
Value added is allocated and intermediate inputs are taken as the
residual GO - VAPRO. That is what makes GO = T005 + VAPRO hold per
industry, which is the form T1 imposes; allocating both independently
broke it by up to $15.2B a cell at 2024. It costs nothing against BEA:
- VAPRO summed back to the 138 lines reproduces UVA205-A exactly
(0.0 on 3,864 cells); T005 reproduces UII205-A to $9M, which is
BEA's own GO=II+VA rounding.
- At 2017 the derived columns reproduce the published detail Use SUT
margins to $0.89M (VAPRO) and $1.48M (T005) per industry.
- Economy-wide VA matches published GDP to at most $9M on $29T in
every year 1997-2024.
- GO - T005 - VAPRO at detail is 2.9e-11.
It also removes the suppression problem. BEA suppresses intermediate
inputs in every year on lines 83 (Customs duties) and 176 (Private
households); both are single-industry lines whose published 2017 T005 is
zero, and the residual recovers that zero rather than needing a special
case.
Step 3's column control is now GO - VAPRO with both sides observed.
Aggregated to summary and scored against the published summary T005 it
is within 0.00007% economy-wide and 0.00023% weighted MAE by industry in
every year 2018-2024, worst summary industry 0.003%. The superseded
frozen-ratio seed scored 0.2-2.3% and 2.5-8.0%, with GSLG 18.3% low at
2022. That is a consistency check and not an independent validation --
UII205-A and the summary Use SUT's T005 are the same BEA estimate
published two ways -- so what it establishes is that the allocation adds
back correctly and the control now *is* BEA's T005. vapro_seed becomes
vapro; the now-dead _row helper and published_gross_output import go.
Signs are carried, not clipped. S00201 has a published 2017 VAPRO of
-$10,069M and stays negative in all 28 years. The residual T005 goes
negative in 11 cells, all in 5191A0 and all in 2002-2015, so the
2017-2024 nowcast span is clear; the other residual direction was worse
(13 spurious negatives across three industries, reaching -$13,117M).
This is not Step 2. VAPRO is the column total; Step 2 owes the split
across the five value-added rows, and T4/T6 are still soft placeholders.
What changes is that there is now a VA level for every year rather than
for 2017 alone, so the balance can run on 2024. It also arrives off the
P1-gated path -- a straight Excel read on the shape of the working
UGO305-A loader, so map_fbs_sectors_to_model_schema never enters it.
Repo-wide: black, ruff, mypy clean (bar the four known Windows-only
settings.py errors); 779 passed, 2 skipped, 1 xfailed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
allocate_underlying_to_detail is public and takes an arbitrary mapping. A code appearing under two lines would be indexed twice by .loc[children] and written twice into the output, so the group totals would silently stop adding up rather than raising. Same for a duplicated line in group_values, where .loc[line] returns a frame instead of a row. The checked-in mapping has neither -- 402 codes, all distinct -- so this guards the function, not the current data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ation The plan for the value-added time series was 2017 detail shares carried on QCEW wage growth, renormalised inside T60200D's 69 groups and rescaled to a NIPA control. Two sources landed after it was written and they change the estimand, not just an input: UVA205-A gives VAPRO as a column total per detail industry annually (#712), and TVA113 splits that same VAPRO three ways at 71 summary industries annually (#538). value_added_timeseries.py measures the four things that decision turns on. They nest. The yaml warns the two release archives are different vintages, so this was checked: rolled to summary, the detail VAPRO panel and TVA113 agree to $9M in the worst of 923 industry-years. Both margins of a 3 x n block are therefore observed, and Step 2 estimates a cross-structure rather than a level -- the same reframing Step 3 got from #497. Rolling up needs the INDUSTRY summary map; the commodity one drops 331314, S00101, S00201 and S00202 and leaves state and local government enterprises 15% short in every year. Most of it is already determined. 20 of the 71 summary industries have one detail child (19.4% of VA, no allocator at all), and 74 of the 138 underlying leaf lines ARE a single detail industry (63.9% of VA, BEA's own annual number). Only 26.9% sits in the 17 groups where a within-group allocator earns its keep. The column control is worth 0.45% of VA in 2018 rising to 2.32% -- $678bn -- in 2024, graded on BEA's own leaf lines so no allocation model enters the measurement; 3.75% of the $18.1T where the movement is measurable at all, 81% of it in ten groups led by other retail at 11.7%. A lower bound, and it matters past its size because Step 5 imposes GO = T005 + VAPRO hard, so value-added error lands in the intermediate block. T00TOP's industry axis turns out to be published. #536's rejection of the market-share conversion (r=0.202) stands, but its conclusion does not: TVA113's V00200 is T00OTOP + T00TOP - T00SUB by industry, max $1M across all 71, so with T00OTOP built the product-tax row is an annual summary residual with no operator. QCEW is still needed for the ~80% of compensation in multi-child groups, but it no longer carries the level, and it can now be graded on the 2012->2017 detail holdout instead of BEA's own later years -- which needs QCEW 2012, undeclared until #728. Left open for review: whether TVA113 becomes an input, spending summary V001/V003 as a test. Two arguments that the trade is smaller than #538 thought are recorded in the plan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both sides declared a 402 x 402 interior section since the branch point -- step3_core the Use interior (Step 3), nowcast the Supply interior (Step 4a) -- so sections.py carried the whole conflict. Both are kept, and the docstring moves from "the one 402 x 402 section" to two of them: the sparsity caveat belongs to Supply alone, and the Use interior gets the note that it has no candidate until Step 5's RAS runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Carries nowcast's Step 4a work through the stack. No conflicts of its own -- sections.py and plan.md merged cleanly once step3_core had taken the five-section resolution. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TVA113 is NOT consumed as an input. The row controls stay NIPA's, so summary V001/V002/V003 keep their ability to grade the build. That withdraws two claims the first pass made: T00TOP does NOT get a build route from TVA113's V00200 identity (#536's conclusion stands unchanged, the split stays free for Step 5), and T60200D's 69 groups are NOT superseded. What the identity buys instead is a grader -- it says where the product-tax money should have landed, so Step 5's answer can be scored rather than trusted. The VAPRO column control is a separate decision and it is kept. UVA205-A is the sibling of UGO305-A, which is already T1. The slack goes to V00300, never to T005. #710 checked that T1 pins the column's sum and not the split, and that T5 is deliberately unimposed -- so today income-side error lands in the intermediate column total, which is the scale of a column of A, and propagates through L into every N. Pinning VAPRO makes T005 = GO - VAPRO, both observed, and moves the slack into value added. residual_headroom() prices that, and states the counterargument rather than burying it: in percentage terms T005 is the MORE forgiving absorber, because it is the bigger number -- a 1% compensation error is a median 0.50% of an industry's T005 against 1.55% of its V00300. It loses because V00300 is terminal. Nothing reads gross operating surplus: not A, not L, not any emission factor. A 7% error in a number nothing reads beats a 1.4% error in a column scale that multiplies through the Leontief inverse. 22 industries cannot absorb much and need a sign guard rather than trust: a 1% compensation error moves their surplus more than 10%, worst 336414 at 121% on an $81M surplus under $9.8B of compensation. Only one industry has a published negative V00300 (S00201, -36,919), so a residual manufacturing new negatives is visibly wrong and cheap to detect. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e carve-out The test the compensation plan could never run. Two things unblocked it: SUPPLY-USE_2026-08-24.zip carries V00100 at BEA detail for 2007/2012/2017 on one 2017 code basis (#704), and QCEW 2012 now exists (#728). Scored the way #704 requires -- on the observed benchmark span, never against BEA's carried-forward 2018-2024 -- as the share of a group's compensation dollars on the wrong detail industry, with the group total given to every candidate so this measures shape and not level. misplaced $M % scored vs frozen frozen 487,348 5.84 -- qcew 517,328 6.20 +6.2% qcew_covered 462,044 5.54 -5.2% qcew_resolvable 438,534 5.26 -10.0% Applied everywhere QCEW makes the block WORSE. Applied where the concordance can resolve it, it is a clear go. The damage is essentially one group: GSLG goes from 4,748 misplaced to 71,694, on its own more than twice the net degradation. The carve-out is derived, not fitted. The crosswalk puts 47 NAICS codes under more than one BEA detail industry -- 23 construction, 24 government -- reaching 18 detail industries in exactly five summary groups: 23, GFE, GFGN, GSLE, GSLG. Decidable before any score is computed, and it is the carve-out Phase 4 argued for structurally: BEA splits construction by type of structure and NAICS by trade, so a plumbing contractor's payroll belongs to no single BEA construction industry. The coverage ratio is REJECTED as the selector. Trusting QCEW where payroll covers >=75% of compensation scores -5.2%, but the floor is non-monotonic: -9.7% at 0.70, -5.2% at 0.75, -3.5% at 0.80, +0.0% at 0.90. Between 0.70 and 0.75 it drops the five groups where QCEW helps most. Coverage and predictive value are different quantities. Two vintage traps had to be cleared and each faked a verdict. QCEW 2012 is on NAICS 2012 against a NAICS 2017 crosswalk -- 28 codes in 2012 not 2017, 20 the other way -- which put 541700 at 20.9x growth and made raw QCEW score +92% before the filter. And the ambiguous concordance would not have raised; it would have produced an even split and called it an answer, exactly as #536 warned. Neither is visible in a total. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
T1 pins the Use column's sum; nothing pinned the split. T4 is soft and aggregated to summary groups, T6 is two economy-wide scalars, and T5 is unimposed by design -- so every dollar of value-added estimation error landed in the intermediate column total, which is the scale of a column of the technology matrix and multiplies through L into every downstream result. T18 pins VAPRO from UVA205-A, the sibling of the UGO305-A already behind T1, allocated to 402 detail. With both, T005 is determined per industry and the slack moves inside value added, where V00300 is free -- gross operating surplus, which BEA largely computes as a residual and which appears in no A, no L and no emission factor. Measured, the error it takes off the intermediate block is 2.75% of T005 in 2018 rising to 9.24% ($1.98tn) in 2024, worst industry GSLGE at 23.5% of its own T005. 2017 reads 31, which is what says the measurement is not scoring its own anchor. T1 and T18 cannot disagree: T1 reads the extracted UGO305-A parquet and T18's series is derived against load_go_detail(), and the two gross output vectors are identical to the dollar for every industry in every year 2017-2024. A column margin restricted to five rows partitions the column, and gras_balance takes one column vector -- so T18 is imposed as a closer after each Use pass, the way T4 is: +d on V00300 and -d across that column's free sign-flexible commodity cells, leaving the column sum and therefore T1 untouched. All of it on V00300 deliberately; a proportional scale would move V00100 and undo the T4 closer that just ran. On the 2017 replay T18 closes to 0.005 and T1 holds at 0.15. T5's absence is now load-bearing rather than tolerated: imposing it would push the slack back into T005 and undo this. Said so where someone would go to add it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan's design was "2017 detail share x QCEW growth, renormalised in-parent, then the NIPA control", blocked on FBS_outside_flowsa not working as an attribution source. It is bypassed rather than cleared: a weight vector that is a rescaling of an FBA already in the method needs no new source, so a clean_fba socket on the existing BEA_Detail_Use_SUT attribution source scales the 2017 benchmark weights by QCEW growth before sector mapping. The blocker is diagnosed to two lines on #731, where T00OTOP's housing and farm lookups still want it. All eight years build. 2017 still reproduces the published benchmark row to $1M, which is the check that wiring the movement in did not disturb the identity case; 206 industries move in 2018 and $66.8bn changes hands against frozen shares, 0.61% of the row. Pointing proportional straight at the QCEW FBA would have been the trivial build, and it is rejected on measurement: QCEW level shares score +21.9% worse than frozen at their best and +270.9% raw, against -10.0% for the movement form. QCEW's share of an industry's compensation is not that industry's share of compensation, but its change carries signal. Four corrections, each of which changes the answer, and none of which is visible in a total: - The concordance cannot resolve construction or government. 47 NAICS codes sit under more than one BEA detail industry and reach exactly five summary groups. Applied everywhere QCEW is +6.2%; carved out it is -10.0%. Derived from the crosswalk, not fitted. - QCEW changes NAICS vintage at data year 2022, not 2023. Unbridged it costs 31 detail industries their coverage and doubles unmapped payroll from 10.4% to 20.7%. The vintage is now detected from the concordance and the two years are paired through it, so composition is identical across the ratio by construction. - 482000 rail is outside QCEW's universe -- railroad employees fall under the Railroad Retirement Board, not state UI -- so QCEW sees 0.11% of its compensation and rail "grows" 4.45x by 2024. A 1% coverage guard catches it. The holdout is neutral on that guard (-10.016%, identical), because the 2012->2017 span does not contain the failure; it is kept on an argument about the source rather than the score, and tuning it upward makes the block worse. - A published zero is a suppression. QCEW reports exactly 0.0 for both NAICS under 334610 in 2021 with normal payroll either side; read as an observation it zeroes the weight and deletes the industry. One occurrence in seven years, fatal each time, so --check sweeps all of them. Also generates the BEA_NIPA FBAs for 2018-2023, which existed without T60200D and so produced no activity sets at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…kups V00100 shipped on QCEW movement last commit; T00OTOP and V00300 were still 2017-only, so derive_initial_value_added raised for every other year. Both now have per-year files and the guard is lifted: the block builds for 2017-2024, VABAS 18.92tn to 28.29tn. The three rows are three different claims and the docstring now says so rather than calling the block "value added, nowcast": - T00OTOP is a level plus two lookups. The T30500 control is read per year (+40.5%), and 43.3% of the row is no longer a frozen share. - V00300 is a seed and only a seed. Level from the eight-line assembly per year (+57.4%), shares frozen at 2017 where drift reaches 12.51% by 2022 - six times T00OTOP's. That is acceptable only because T18 makes it the residual the balance overwrites, and TVA113 must not "fix" it: that is the grader. T00OTOP's housing and farm lookups are in, and they should have been in the first time. T70405 B1031C is the 531HSO+531HST pair and T70305 B1017C is the ten farm codes, both exact, both published annually, together 43.3% of the row. They were left out because folding a computed weight vector in was read as needing an FBS_outside_flowsa attribution source, which does not work. That hatch is still broken and was never on the path: a weight vector that is a rescaling of an FBA already in the method is a clean_fba socket, which is how V00100 carries QCEW. Both rows were blocked on a hatch neither needed. Worth 1.92% to 1.68% of row error in 2024, 0.12-0.39pp across the span - real, bounded, and smaller than the control's own 2.9% vintage error in 2021. Two things measured on the way that are not about these rows: - The summary SUT is a stale grader for exactly 2019-2022. Its own VAPRO total sits 0.09-1.21% below current-vintage UVA205-A in those four years and matches to the dollar in 2017, 2018, 2023 and 2024. So V00300's apparent 2.64% assembly error in 2022 is the workbook being behind, not the assembly being wrong - and V00300 shows it at twice VAPRO's rate because it is the row a NIPA revision lands in. - Selecting NIPA controls by line number is only safe because the lines do not move. Verified for all eight years and now asserted under --check: T30500 line 37 is LA000365 and line 17 is LA000237 in every one, and the eight V00300 lines are equally stable. A restructuring would point these at different series and no total would look wrong. Also withdraws stale text in five places saying the T00TOP/T00SUB industry split is "an output of Step 5's balance, not an input to it". tax_axis_conversion.py takes that apart: rejecting the Make matrix (r = 0.202) is not the same as having no operator, and Step 4c's producer/trade level split plus a few named routings reaches r = 0.948. The split has no industry target and is still seeded, so it is an input to the balance as well as an output. The seed itself is not built here - it reads commodity output and margins, both mid-update. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…l, and the NAICS-2022 goods Crosswalk # Conflicts: # bedrock/analysis/nowcasting/README.md # bedrock/analysis/nowcasting/progress_report.md # bedrock/analysis/nowcasting/sections.py # bedrock/transform/eeio/nowcast.py # bedrock/transform/iot/nowcast_intermediate.py
…ded block The last two value-added rows, and the only two that are not estimated: they are the same money as the Supply table's TOP/MDTY/SUB columns, carried on the industry axis. So `nowcast_va_taxes` converts rather than re-estimates, and its whole content is the operator. Both rows are stacked into `derive_initial_value_added`, which now returns all five rows for 2017-2024. The two rows needed two different operators, and only one of them is a seed. T00TOP - the level split, at r = 0.947 and 27.9% error. Market shares fail this badly (r = 0.202) and not by being noisy: a tax on a product is remitted by whoever sells it, so they send the whole petroleum tax from wholesalers to refineries. `top_by_level` already draws the producer-versus-seller line Step 4c needed for its own reasons. Two of the three legs are exact rather than estimated - duties are a lookup onto 4200ID, and the ten government columns are zero by an accounting rule, which the market-share leg was violating by 10,513. Still a seed; the residual is 20 named trade industries Step 5 moves. T00SUB - not a seed. A subsidy is paid to an operator, so it stays on its own code, and code identity's entire residual is two pairs of cells rather than a smear: S00203 public housing authorities and S00102 federal insurance enterprises, both government enterprises that produce a subsidised commodity. Routing those two by name closes 2017 to shape agreement of 7e-17. What is frozen is one number and it is named - S00203's 54.4% of the housing line, which NIPA T31300 does not split. Three things found while building it, all now asserted: - The routings must not fire in 2020-21, where `sub_decomposition` replaces the `other` type with PPP. Routing 5241XX then would put pandemic support on a federal enterprise. PPP is already industry-shaped, so identity is at its best in those years rather than its worst. - S00102 is over-subsidised from 2022 (36.6bn against 6.3bn in 2017). The cause is upstream: NIPA's `other` line still carries pandemic-era programmes, and the same 36.6bn sits on commodity 5241XX with or without this conversion. Documented as the `other` line's residue, not as a measurement. - The T00SUB level carries a 1 $M gap - NIPA's 59,875 against the workbook's 59,876 - spread proportionally across all twelve subsidised cells. `check` tests shape and level separately, because scoring raw levels reports BEA's own rounding as error and could hide a real one behind it. `use_va_detail_sut` now grades five rows instead of three, with the published T00SUB flipped to the balance's sign convention. The 48 partial column totals are all T00TOP inside the trade block, which is the documented state of the seed. Also updates two constants in test_nowcast_product_taxes that moved when the Trade FBSs were rebuilt on the NAICS-2022 goods Crosswalk (#734): TOP's residual now rides a purchaser-price base carrying MCIF, so 2024's tobacco gap went -26,153 to -26,282 and its share 7.7% to 7.3%. 2020's two numbers did not move at all, which is the check that the shift is the Crosswalk and not the method. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
WesIngwersen
added a commit
that referenced
this pull request
Aug 27, 2026
Docs only. The plan and progress report predated #733/#734 and were describing a Supply bridge that no longer exists. Supply column table, corrected against a live coverage run rather than from memory: eight of the twelve columns now run 2017-2024, and only TRADE, TRANS and therefore T014/T016 stop at 2023. MCIF, MADJ and MDTY were still listed as "2017 candidate" and have been annual since #730. TRANS was "2017-2022" and now reaches 2023 from AIES. TOP's residual is no longer frozen, which the plan asserted in two places. The purchaser-price base did not have to wait for Step 5 as the plan predicted - it is assembled from the bridge's own components, and is not circular because TOP sits in T015 and is excluded from its own base. Recorded with what it buys (S00402's 174,312 base, the 0.939-1.008 rescale) and what it costs: TOP now depends on margin coverage, so 2024 holds 2023's shares. Step 1: F04000 was described as all-zero for 2018-2024 and has been annual since #730 (2,211bn to 3,139bn). F03000 really is still 2017-only, so the two are now stated separately instead of together. Numbers re-measured rather than edited by hand. Steps 1 and 4 re-run, and the embedded images regenerated with them: supply bridge coverage 99.0% to 99.2%, accuracy 61.8% to 62.1%, grand total error 0.092% to 0.048%. That improvement is not Step 4a - it is the Trade FBSs rebuilt on the NAICS-2022 goods Crosswalk, which moved MCIF (misses 27 to 22) and carried through T013 and T016. Per-column table updated with it. Step 3 is unchanged and was not re-run.⚠️ The 2017 trade scorecard moved in two directions at once and the README now says so rather than reporting the half that flatters it: the export national error roughly halved, +12.20% to +6.16%, while Pearson fell 0.96 to 0.92, top-20 Jaccard 0.67 to 0.60 and n_miss rose 39 to 43. The extra mass lands closer to the published total but spread over more Detail codes. The prior row is kept for comparison.⚠️ The stored per-commodity trade baseline was deliberately NOT refreshed. score_2017_trade_detail flags a handful of expected status changes against it; --update-baseline is a judgement call and has not been made. Step 2's plan entries are not touched here - they belong to #735. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 2 in full: the Use table's five value-added rows, built for 2017-2024.
Supersedes #691 and #693, both of which are ancestors of this branch.
The five rows are five different claims
Reading the block as "value added, nowcast" overstates most of it. What each row is worth:
V00100T00OTOPT30500per year; housing rescaled toT70405 B1031C, farm toT70305 B1017C. 43.3% of the row observed, the rest keeping 2017's within-block shapeV00300T00TOPT00SUBThe first three are
VABASat basic prices and are estimated from NIPA. The last two are the wedge toVAPROat producer prices and are converted from money that already exists in the Supply table'sTOP/MDTY/SUBcolumns — so their levels carry no modelling content at all and only the industry split is estimated.The tax rows: the obvious operator is wrong, and it was measured first
tax_axis_conversionscored the benchmark market-share matrix against the published 2017 rows before anything was built: r = 0.202 at 114.6% error onT00TOP. It is not noisy, it is pointed at the wrong stage of the chain — a tax on a product is remitted by whoever sells it, so market shares send the whole petroleum tax from wholesalers to refineries and the whole motor-vehicle tax from dealers to assemblers. 55.7% of the published row sits in wholesale and retail.T00TOPuses the producer/trade level splittop_by_levelalready computes for Step 4c:TOP + MDTY424700by nameTwo of the three legs are exact rather than estimated: duties are a lookup onto
4200ID, and the ten government columns are zero by an accounting rule the market-share leg was violating by 10,513. Still a seed — the residual is 20 named trade industries Step 5 moves under soft targets.T00SUBturned out not to be a seed at all. A subsidy is paid to an operator, so it stays on its own code. Market shares score 0.676; code identity alone scores 0.569 — worse, and that is the finding, because identity's entire residual is two pairs of cells rather than a smear:S00203531HSTS001025241XXRouting those two by name closes 2017 to shape agreement of 7e-17. What is frozen is one number and it is named:
S00203's 54.4% of the housing line, which NIPAT31300does not split.Traps found and now asserted
sub_decompositionreplaces theothertype with BEA's PPP allocation. Routing5241XXthen would put pandemic support on a federal enterprise. Conversely PPP is already industry-shaped — BEA publishes it by industry — so identity is at its best in those years, not its worst.S00102is over-subsidised from 2022 (36.6bn against 6.3bn in 2017). The cause is upstream: NIPA'sotherline still carries pandemic-era programmes, and the same 36.6bn sits on commodity5241XXwith or without this conversion. Documented as theotherline's residue, not as a measurement.T00SUB's 1 $M level gap is BEA's rounding — NIPA's 59,875 against the workbook's 59,876 — spread proportionally across all twelve subsidised cells.checktests shape and level separately, because scoring raw levels reports that rounding as error and could hide a real one behind it.VAPROsits 0.09-1.21% below current-vintageUVA205-Ain those four years and matches to the dollar in 2017/2018/2023/2024. SoV00300's apparent 2.64% error in 2022 is the workbook, not the assembly.Line: [37, 17]is a property of the table, not the series.other_taxes_allocation.check()now re-derives it fromSeriesRegister, which selects by code, so it grades the selection.S00201state and local passenger transit carries aV00300of -36,919 in 2017 and is negative in every year; it is the only industry that is.Diagnostics
use_va_detail_sutnow grades five rows instead of three, with the publishedT00SUBflipped to the balance's sign convention (BEA publishes it positive and subtracts it; comparing unflipped reports every subsidised industry as a 200% error). 2017:T00TOPinside the trade block — the documented state of the seed, not a regressionAlso in here
Two constants in
test_nowcast_product_taxesmoved when the Trade FBSs were rebuilt on the NAICS-2022 goods Crosswalk (#734). TOP's residual now rides a purchaser-price base carryingMCIF, so 2024's tobacco gap went -26,153 → -26,282 and its share 7.7% → 7.3%. 2020's two numbers did not move at all, which is the check that the shift is the Crosswalk and not the method.Gates
ruff check .clean;mypy bedrockclean apart from the four pre-existing Windows-resourceerrors insettings.py.bedrock/transform/iot+bedrock/transform/eeio— 472 passed, 1 skipped, 1 xfailed; the newtest_nowcast_va_taxesis 70 of those.Still open
TVA113stays held out — it is the grader, not an input. Row controls stay NIPA's.336414at 121%).🤖 Generated with Claude Code