Step 2: build the value-added block for 2017 (#538) - #693
Open
WesIngwersen wants to merge 18 commits into
Open
Conversation
Adds 24 tables to BEA_NIPA.yaml, all annual and complete for 2012-2024, at roughly 950 extra FBA rows per year: the controls (1.10, 1.1.5, 1.3.5, 1.14), compensation (6.2D, 6.3D, 6.10D, 6.11D, 3.10.5, 7.18) and gross operating surplus (6.12D-6.22D, 7.5, 7.7, 7.9, 3.8, 7.3.5, 7.4.5, 7.11). T00TOP and T00SUB are deliberately absent. Their commodity-axis counterparts shipped in Step 4d (#690), and their industry split is an output of Step 5's balance rather than an input to it, so sourcing them here would estimate the same money twice on two axes. Three findings, all pinned by the new test rather than left in prose: - Take the *paid* line, not the table's root. 6.2D and 6.3D each state their total twice - line 1 received by residents, line 2 paid by domestic industries and government - and value added wants line 2. On it, wages plus both supplements close to 0 against compensation and land on the Use SUT's V00100 within 3. Reading line 1 is what left the ~10,600 the compensation plan carried as an open item; it is the rest-of-world adjustment A4187C, stated in 6.2D's own lines 97-99. - 6.11D is three panels under one code, and only the first is by industry: lines 1-20 industry, 22-36 type of fund, 37-45 benefits paid - a different concept at 2,370,770. Its industry grain is 17, not the 36 the plan claimed, and selecting the whole table double-counts. T61600D, T71100 and T11400 also restate a code; select by line in all four. - Table 3.8 gives the government-enterprise surplus an industry axis it was thought not to have, federal and state-and-local summing to 1.10's A108RC exactly. The config half of the test needs no data and guards the substring hazard in bea_nipa_parse, which selects a table's series with str.contains: two declared ids where one contains the other would contaminate silently. NIPA_final_dom_uses_2017 rebuilds bit-identical - all 17 columns, same row count - so the added rows reach no existing method. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ache The plan's Phases 1-5 were written as a pure-Python pipeline. With Step 2 settled as three FBS methods, most of it survives as configuration rather than code: BEA_Detail_Use_SUT already melts the Use SUT through VAPRO, so the 2017 benchmark V00100 by industry is an attribution source today, and Phase 1 is a selection_fields clause rather than an extraction. The one step with no FBS primitive is the anchor-and-move itself - multiplication does not preserve the group total and there is no renormalise step, which is what the old Phase 3.4 needed. Recorded as the design seam: Phases 1-2 build a cached FBS_outside_flowsa source holding the moved shares, Phase 3 is a single proportional attribution against it, and the exact rescale then holds by construction instead of being a follow-up step. QCEW is cached locally as per-year FBA parquets because generating it is slow (~9M rows a year, county grain). Dropping one into extract/output_data lets getFlowByActivity find it through the ordinary local path with no code change - esupy matches on name and ignores the v2.0.4 tag. Verified for 2017. Four things measured off that cache rather than assumed: - QCEW national NAICS-6 payroll is 93.8% of NIPA wages paid. That number is what settles "allocator, never control", and the missing 6.2% concentrates in the sectors Phase 4 already carves out. - 1,027 of the 1,048 NAICS-6 codes in the BEA detail crosswalk are present, so the gap is a bounded list of 21 rather than a structural hole. - Ownership is on the flow, not a separate axis. It still does not separate government enterprises from general government, which is the distinction the SUT needs - so government stays on NIPA, and the plan's claim is narrowed to what is actually true. - The cache is 2017-2023, so the nowcast's 2024 has no QCEW and needs a stated fallback. Better found here than in Phase 3. Employment_common.yaml cannot be reused as-is: every _bls_selection_fields_* block hardcodes Class: Employment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
QCEW covers 93.8% of NIPA wages paid. The plan treated the other 6.2% as an unexplained residual to spread pro rata. NIPA states it. 7.18 closes it exactly: BLS published 7,968,336 + misreporting 106,273 + not UI-covered 399,801 + timing 0 = 8,474,410, wages received. And the gap is already non-uniform before any industry table is opened - government's uncovered rate is 11.3% of government wages against private's 3.5%. 6.4D is the table that turns the assumption into a measurement: QCEW publishes employment on the same axis, so QCEW/NIPA is a coverage ratio per industry. Measured on 2017 private employment it is 97-100% almost everywhere and collapses in exactly two places - educational services 77.1%, other services except government 63.0%, where religious and grantmaking organisations are largely UI-exempt. So QCEW growth is trustworthy broadly and the exceptions are a named short list, which is a better method than a flat haircut. Construction at 97.1% is a warning rather than a reassurance: the count agrees because both sources count the same workers, and the trade-versus-structure-type disagreement that actually matters is invisible to a coverage ratio. Three lookups located: RfHhInstComp W151RC is 18,684, the SUT's 814000 to the dollar - the sector QCEW covers worst is the one NIPA hands over directly, and the plan had described this line without finding it. U32500 splits general government compensation into wages and supplements, which 3.10.5 only totals. 6.6D wages per FTE is the plausibility check Phase 5 lacked: a share can be positive, sum to one, and still imply an implausible wage. Adding those tables exposed two defects in bea_nipa_parse: - Scale and unit were assumed, not read. A flat `* 1000000` with Class and Unit hardcoded to Money/USD is right only while every declared table is in millions of dollars. 6.4D/6.5D are thousands of persons and 6.6D is a ratio, so 6.4D's 150,654 thousand employees would have been published as 1.5e11 USD. Both now come from each series' own MetricName/DefaultScale, with an explicit map that raises on an unrecognised metric rather than defaulting to dollars. Every dollar table BEA publishes has DefaultScale -6, so all 1,812 dollar rows are bit-identical across the change. 1.14's three chained-dollar lines move to Class Other, so a Class: Money selection cannot add real dollars to nominal. - Table ids were matched by substring. Asking for U70205 also returned U70205S's 44 physical-quantity and price-index series under their own id. Nothing downstream selected them so it never showed, and it was the metric guard above that surfaced it. Matched exactly now. NIPA_final_dom_uses_2017 still rebuilds bit-identical, and 2012/2019/2020/2024 extract cleanly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Corrects an overstatement in the previous commit. T00TOP/T00SUB were described as "not sourced here ... building them would estimate the same money twice". That was wrong. They are already built by commodity in Step 4d, so the industry row is a transformation of money that exists, not a second estimate - and the plan's own coordination note said exactly that: "build once, derive the other through the Make/Supply structure". So the open question was never whether to derive it, but whether the derivation works. Now measured for 2017, where the published answer exists. It does not. Using the benchmark market-share matrix from the Supply table, D[c,i] = V[c,i] / T007[c]: T00TOP (TOP + MDTY) correlation 0.202 |error| 114.6% of the row T00TOP (TOP only) correlation 0.211 |error| 109.5% T00SUB correlation 0.676 |error| 79.8% The reason is structural rather than noise. 55.7% of the published T00TOP row sits in wholesale and retail industries, because a tax on a product is remitted by whoever sells it while market shares place it with whoever makes it. The pairs are stark: petroleum wholesalers 424700 are 88,362 published against 13 estimated while refineries 324110 are 397 against 92,893; motor vehicle dealers 441000 are 45,947 against 2,301 while assemblers 336111+336112 are 26 against 24,141. The whole tax moves one stage up the chain. This is measured in the benchmark year, against the table the mix comes from, so it is not the "2017 ratios drift" objection the 2026-08-17 decision was argued from - it is stronger. A conversion that fails in 2017 cannot be rescued by being applied nearer to 2017. T00SUB fails differently and more usefully: its residual is two named structures, not a smear. S00203 other state and local enterprises is 19,471 published against 1,964 (transit operating subsidies) and S00102 is 6,339 against 102 - subsidies paid to an operator rather than attaching to a product, which no product-side operator can place. T30800 already carries them. One piece converts exactly: published T00TOP on 4200ID is 38,513 against a Supply MDTY of 38,507, so customs duties are a lookup rather than an allocation - 5.1% of the row, free, every year. The decision stands: the industry distribution stays free for Step 5 under economy-wide soft targets. But free is not unseeded, and the better operator is the margin structure rather than the Make matrix - a commodity's wholesale and retail margins say which trade industries handle it, which is the point-of-sale signal the tax follows. Those margins are built, so that operator is testable the same way. Left as the open follow-up. Findings carried as a --check flag rather than unit tests, per the convention for analysis modules. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ADME Three passages still read as though T00TOP/T00SUB were simply out of Step 2's scope, which was the overstatement the previous commit corrected in the Step 2 section but not elsewhere: - the framework status table said "Not sourced", where the truth is that they are built by commodity and the industry row is a conversion Step 5 solves; - the board-provenance note said they "left Step 2's scope entirely"; - the data-gap section already said "Step 2 still supplies a seed" and was right all along - it now also carries what the seed can and cannot contain, since the market-share operator was measured and fails at r=0.20. Adds tax_axis_conversion.py to the analysis README index. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ersion Only within wholesale, and only for one code. The previous commit left "ride the margin structure instead of the Make matrix" as an untested proposal. Testing it turned up something better: Step 4c already computes the operator this needs. top_by_level splits TOP per commodity into producer-level (325,829) and trade-level (391,096) from an identity with nothing modelled in it - excise sits in Producers' Value, sales tax inside the margin columns. That is exactly the producer-versus-seller distinction market shares get wrong, and its trade-level total lands within +2.2% of published wholesale-plus- retail T00TOP. market share on all TOP + MDTY corr 0.204 |err| 114.6% + level split, trade-level by trade output corr 0.743 |err| 41.9% + motor fuel routed to 424700 by name corr 0.946 |err| 29.9% On the question itself, the answer differs by block: - Non-trade industries are 44.3% of the row and need no matrix at all. Once the producer-level portion is separated, plain market shares give corr 0.987. - Within retail, no. Output shares - which for a trade industry are very nearly margin shares - give 0.744. Retail product tax is general sales tax: broad-based, HHI 0.137, an effective 7.3 of 9 industries carrying it. - Within wholesale, yes, and output shares are worse than useless: corr -0.192. 424700 petroleum wholesalers takes 51.3% of wholesale product tax on 3.4% of wholesale output, because wholesale tax is motor fuel excise rather than a broad-based tax. HHI 0.321, an effective 3.1 of 10. But wholesale does not need a commodity-by-trade-industry matrix either. It needs one named routing: NAMED_TAX_LINES already carries motor fuel as 324110 and trade_level_share already says that tax is 99.8% trade-level, so sending it to 424700 takes wholesale from -0.192 to 0.973 and the whole row to 0.946. With petroleum pulled out by name the other nine wholesale industries score 0.825 on output shares - they behave like retail. So the general commodity-by-trade-industry margin matrix that the PRO:PUR producer-price work will eventually need is not required here. Still a seed and not a target: 29.9% absolute error. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Construction was the right last block to check and the wrong one to worry about. It is block-shaped like the trade industries and could have been a second petroleum; it is the opposite. Commodity TOP is 1,907 against a published T00TOP of 1,857 and plain market shares give corr 1.000, |error| 31 = 1.7%. The reason is the Make matrix. BEA defines the construction industries by type of structure, so the block is 94.5% in-block and 100.0% diagonal - there is no producer-versus-seller distinction when whoever builds the structure sells it. top_by_level agrees: 100% of construction TOP is producer-level, so the routings that rescue wholesale are inert here and all three operators give one number. MDTY and SUB are zero on both axes, so there is no duties or subsidy question in the block at all. Tax sits on 3 of 12 codes, none a NAMED_TAX_LINES entry. The residual 1.7% is a leak, and market shares get its direction right and its size about half: 5.5% of construction output is own-account or secondary work (531HST 20,279, state and local government 19,991), and published moves 50 off the block where market shares move 30.4. That answers the larger question of whether the remaining sectors each need this treatment. They do not - the error is 20 industries, not 402: top 5 industries carry 35.1% of the error, top 20 carry 80.3% 17 of those 20 are wholesale or retail wholesale 22.8% of the row 41.9% of the error retail 27.8% 33.7% non-trade 44.3% 24.4% 4200ID 5.1% 0.0% What is left is the within-trade allocation already characterised plus five named non-trade structures: 721000 accommodation (-6,592), 517210 wireless (-4,310), 221100 electric power (-3,934), and government enterprises handed tax they do not carry (S00202 +3,439 against a published zero, GSLGE +1,698, both of which belong to the Step 7 reallocation). So build against the seed and repair the named twenty later - Step 5 balances these cells under soft targets anyway, and seed accuracy below the block level is not what the build is waiting on. Adds construction_codes, construction_scores and error_concentration, wires both into the report, and adds five assertions to --check: construction stays usable, its error stays under 10%, its Make block stays diagonal, it stays MDTY/SUB-free, and the top-20 error share stays above 60% - that last one fails if the residual ever spreads and a broader sweep becomes necessary after all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BEA books zero taxes on production to all ten government industry codes - T00OTOP and T00TOP are both zero on every one of them in 2017, the single exception being 538 of T00TOP on S00203. The columns are real (V00100 and VABAS are populated), so the zero is an accounting rule and not a gap: a tax levied by government and remitted by a government producer nets out. The market-share leg was violating that rule by 10,513 against a published 538, because government genuinely does produce taxed commodities and market shares hand it a share of the tax on them: S00202 S&L electric utilities 3,439 against a published 0 S00203 other S&L enterprises 2,428 against 538 GSLGE S&L educational services 1,698 against 0 GSLGO S&L other services 940 against 0 S00101 federal electric utilities 854 against 0 market_share_matrix gains an exclusion that drops those columns and renormalises each commodity over the producers that remain, so the tax stays with its own commodity rather than being deleted or smeared economy-wide. That is the fourth operator in the progression: + motor fuel routed to 424700 by name corr 0.946 |err| 29.9% + government columns zeroed, renormalised corr 0.948 |err| 27.9% It moves 15,692 of error, and only two thirds of that is the excess itself. The rest lands where it belongs: 221100 electric power |err| 3,934 -> 443 721000 accommodation |err| 6,592 -> 5,088 622000 hospitals |err| 763 -> 296 Read 221100 against S00202: the previous operator was under-attributing private electric power by almost exactly the amount it had misplaced onto the government utility, so the money was misplaced in a recoverable direction. The non-trade block goes from corr 0.987 / 24.4% of the error to 0.992 / 18.8%, and no government code appears in the top-20 error ranking any more. Only the producer-level leg needs the exclusion - the trade-level leg lands on wholesale and retail, where no government code sits, and duties land on 4200ID. No commodity is stranded: no commodity produced entirely by government carries any product tax in 2017, so the fallback that keeps such a row unrenormalised never fires, and the seed total is unchanged to the dollar. This one was worth fixing before the build rather than after. The other residuals are misallocations Step 5's balance can pull back; these columns are redistributed into private industries by the Step 7 government-enterprise reallocation, so a wrong seed here propagates into work that would have to be unpicked. Adds government_industries and government_scores, and four assertions to --check: the published rule still holds, the seed puts nothing there, the exclusion still reduces the row error, and renormalisation still moves money between industries without creating or destroying it. _frames is now cached - the scoring functions call it a dozen times over and --check had grown to over two minutes; it is 9s now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Step 2's third row was scoped as "no NIPA table has an industry axis for this, accept a cruder method". The first half is true and the second does not follow: measured, T00OTOP is the best-behaved of the three rows. T30500 puts 88.1% of it in recurrent taxes on immovable property, so it is a property tax and corporate income tax is not in it at all - that lands in V00300. Which means the intuitive allocator is the wrong one. Scored against the published 2017 detail row, industry output gives correlation 0.590 and an absolute error of 92.3% of the row, missing 531HSO alone by 150,567; the effective rate runs 0.49% at p10 to 15.20% on 531HST. What works instead is that the row is concentrated - three real-estate codes carry 46.3% - and BEA publishes the big cells. T70405 B1031C is the 531HSO+531HST pair to the dollar and T70305 B1017C the ten farm codes within 3, and neither is a 2017 coincidence: against the summary SUT they hold in six and seven of the eight years 2017-2024. The remainder rides frozen 2017 shares, graded out of sample at 1.9% composition drift against a 40.5% level move. The summary SUT is evidence here and never an input, which is the role Step 5's Decision 3 reserves it for. Also lands the two pieces of Step 2 plumbing every one of the three methods needs: the three BEA_2017_Code identity crosswalk rows (Phase 0), and assign_use_row_from_clean_parameter - the transpose of the final-demand helper, since value added is a Use row rather than a Use column. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
NIPA_VA_othertax_2017 is the first of Step 2's three methods and the one that proves the seam the other two need: the BEA_2017_Code identity crosswalk rows, Sector_Crosswalk_BEA_NIPA_VA.csv, the melted BEA_Detail_Use_SUT as an attribution source, and the Use-row transpose. One activity set. The T30500 control (LA000365 state and local other taxes on production, plus LA000237, the federal remainder after taxes on product) attributed proportionally across the 392 non-government industries on the 2017 benchmark T00OTOP row, then transposed onto SectorProducedBy. Replaying 2017 gives correlation 1.0000 against the published row, 389 industries of 389, zero on government, and the control's own 9 of rounding. Five tests pin it; NIPA_final_dom_uses_2017 rebuilds bit-identical across the crosswalk change. The housing and farm lookups are NOT in it, and that is the finding rather than an omission. They cannot be their own activity sets: NIPA states no "other taxes on production excluding housing and farm" line, so a third set's control would still be the whole 608,533 and the three would sum to 872,044. Folding them into the weight vector instead needs an FBS_outside_flowsa attribution source, and that path does not work - get_flowby_from_config builds a FlowBySector for that data_format while attribute_flows_to_sectors then calls map_to_sectors, which only FlowByActivity defines. All four existing uses of the hatch in the repo are top-level sources. That matters well beyond this row: the plan commits V00100's anchor-and-move build to the same hatch, so it has to be cleared before Step 2's largest row. Finding it on the 1.8% row rather than the 30.9% one is the argument for having built T00OTOP first. Here the cost of going without is bounded and measured - 1.92% against 1.68% of row error in 2024, less than the control's own 2.9% vintage error in 2021 - and lookup_improvement() reproduces it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All three of Step 2's value-added rows now generate, and the use_va_detail_sut diagnostic is switched on: every cell of the 3 x 402 block matches, worst 2.6 million on a 178 billion cell, which is BEA's own rounding. derive_initial_value_added stacks the three methods. V00100 runs on the 69 T60200D industry groups as separate controls rather than one national total, because those 69 leaves partition the 71 BEA summary industries exactly - no gaps, no overlaps - and 63 of them equal a summary industry's published compensation to the dollar. Verified by value rather than by name. So the frozen 2017 shares only have to hold within a summary industry, which is a far smaller claim than a national spread. Correlation 1.000000 against the published row, 400 industries of 400. That build reverses the plan's headline decision, and on measurement. Splitting wages from supplements was meant to avoid imposing one industry's supplement rate on another. It cannot: T60300D matches T60200D at 69 groups but T61000D and T61100D publish only 16, so the split imposes a coarse group's rate on industries whose own rates differ and misplaces 99,025 - 0.95% of the row - against a T60200D-only method that is exact. NIPA publishing both halves by industry does not help when it publishes them at coarser grain than the total. V00300 is eight controls across five tables on one industry distribution, and stays plain deliberately. Four of its components have industry tables on mutually incompatible partitions (20 / 21 / 12-plus-financial / 63 groups) and three have no axis at all, so a component build imposes four coarse partitions at once - the same mistake, four times over. It keeps S00201 at its published -36,919; nothing here may assume positivity. The best next source for Step 2 is a value added by industry extractor. V00300 = VABAS - V00100 - T00OTOP, and two of those three are now built at detail, so BEA's GDP-by-Industry accounts would give V00300 by 71 industry groups as a residual - better than assembling eight incompatible component tables, and ahead of Fixed Assets because it serves the 41.6% row rather than one 40% component of it. The summary Use SUT carries the same thing and Decision 3 holds it in the test set, so it cannot stand in. 2018-2024 are not written: every method holds its within-group shares at 2017, which is complete for 2017 and not for later years. Compensation needs the moved-share source, still blocked on the FBS_outside_flowsa gap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BEA_GDPbyIndustry reads table TVA113, Components of Value Added by Industry, out of BEA's GDP-by-Industry release archive: value added at producer prices plus compensation, taxes on production and imports less subsidies, and gross operating surplus, for 100 industries, annually. No API key - BEA's GDPbyIndustry API dataset needs a UserID and this archive does not, so it follows BEA_NIPA's pattern of caching one archive under extract/input_data with a GCS fallback. 2012-2024 declared, 1997-2024 available. The root identity holds exactly in every year checked: compensation + taxes + surplus = value added. This was asked for because V00300 has no NIPA table of its own and its eight component tables publish on mutually incompatible industry partitions. TVA113 does state gross operating surplus by industry directly - so it looked like the one source that would give V00300 a real industry axis. It is not, and the extractor is what proves it. All 71 BEA summary industries' V003 match a TVA113 surplus row to the dollar, and all 71 V001 match compensation the same way. BEA's industry accounts and the SUTs are the same estimates published twice, so this is the summary Use SUT by another door - and Step 5's Decision 3 holds the summary SUT out of the target set precisely so it can grade the build. Two tests pin that, so the finding cannot quietly stop being true. NIPA_VA_surplus is therefore left unchanged. Wiring it in is a two-line swap of the attribution source; what it buys is a near-exact V00300 seed at summary grain, and what it costs is summary V003's ability to grade that seed. That is a testing-strategy judgement rather than a data-availability one, so it is documented and left open rather than taken. The extractor is worth having regardless: it makes the series reachable as a versioned FBA rather than a private workbook loader, and it is the natural loader for the test set itself. It also corrects the recommendation the previous commit made - "the highest-value missing source" was the wrong framing, since it was never missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Step 2's section had a target and no picture. It has both now: use_va_detail_sut_2017.png, rendered in the same style as the Step 1 and Step 4 images, plus the counts, coverage and accuracy tables the other two sections carry. Every section in the diagnostic now has a candidate; none carries candidate=None. The block is 100% coverage and 100% accuracy, and the report says at both the summary table and the section that this is NOT comparable to the other two rows. All three methods take their within-group distribution from the 2017 benchmark, which is the reference, so a 2017 run tests the plumbing and not the estimate - solid green is the floor, not an achievement. What it does certify is that the orientation transpose, the identity crosswalk rows, the 69-way compensation control set and the eight-line V00300 assembly carry mass end to end without losing or misrouting it: the grand total is off by 0.000016%, three million on eighteen trillion. The picture earned its place on the first run. Reading the 17 absent cells showed T00OTOP is absent on eleven government codes, not the canonical ten: the prefix rule (S00, G) misses the US Postal Service, 491000, a federal government enterprise whose BEA code is shaped like an industry's. It was landing on zero only because the benchmark weight happened to be zero, not because the rule excluded it. Now named in write_value_added_crosswalk.py. No 2017 number moves - which is why a total would never have found it. Also shortens the section note so the plot header stays one line, matching the other two images. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BEA_GDPbyIndustry and BEA_Detail_GrossOutput_IO read sibling zips from the same BEA release page, and GrossOutput.xlsx is duplicated between them - gdp.py gets it from GCS as a manually extracted workbook pinned to 2025Q2, BEA_GDPbyIndustry fetches the summary archive straight from BEA. Filed as #694; this just leaves a pointer at both ends so the next reader does not re-derive the relationship. Records the two things that make it more than a file move: the 2025Q2 pin is deliberate and the archives are currently different vintages (June 2026 against September 2025), so dropping it would silently move gross output under electricity disaggregation and derived_gross_industry_output. Also records the negative result, since it is the reason V00300 stays at summary grain: UGdpByInd's ValueAdded.xlsx has no counterpart to TVA113. UVA201-UVA210 are indexes, real value added and UVA205 value added totals - there is no components table, so compensation, taxes and gross operating surplus are published at summary grain only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pytest names a test module by walking up from it until it reaches a directory without an __init__.py. Both bedrock/transform/nipa/__tests__ and bedrock/utils/taxonomy/mappings/__tests__ carried an __init__.py while their parent directories did not, so both claimed the top-level module name __tests__. Whichever pytest collected first won, and the other failed to import: ModuleNotFoundError: No module named '__tests__.test_bea_ceda_v7__cornerstone_commodity' Removing the empty NIPA __init__.py puts those tests on the convention the majority of __tests__ directories here already follow, giving them the bare module names test_nipa_va_othertax and test_nipa_va_rows, both unique across the repo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
WesIngwersen
marked this pull request as ready for review
August 24, 2026 12:26
Member
|
I'm not sure I understand the issue with |
Member
|
There is a lot going on in this stack so a bit hard to evaluate just by scanning the code. But it looks like its wired correctly. |
# Conflicts: # bedrock/analysis/nowcasting/plan.md
# Conflicts: # bedrock/analysis/nowcasting/README.md # bedrock/analysis/nowcasting/progress_report.md # bedrock/analysis/nowcasting/sections.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds all three of Step 2's value-added rows for 2017, closing #538. Stacked on
nipa_va_tables_536(#691), so review that first.All three rows generate,
derive_initial_value_addedstacks them, and theuse_va_detail_sutdiagnostic is switched on — every section in the #587 diagnostic now has a candidate and none carriescandidate=None.V00100compensationT60200Dindustry groupsT00OTOPother taxesT30500V00300gross surplusThree plan decisions changed, each on measurement
1.
T00OTOPis a property tax, and the intuitive allocator is the worst one. The plan said "no NIPA table has an industry axis for this, accept a cruder method". The first half is true; the second does not follow.T30500puts 88.1% of the row in recurrent taxes on immovable property — corporate income tax is not in taxes on production at all, it lands inV00300. So scored against the published detail row:T018VABASV00100Output-proportional misses
531HSOalone by −150,567 on a published 178,599. What works is that the row is concentrated — three real-estate codes carry 46.3% — and frozen 2017 shares drift only 1.9% while the level grows 40.5% across 2017–2024, graded out of sample on the held-out summary SUT.2. ❌ Splitting wages from supplements is reversed.
compensation_disaggregation_plan.md's headline decision was to disaggregate wages and supplements separately. The reasoning is sound and the data is not there:T60300DmatchesT60200Dat 69 industry groups, butT61000D/T61100Dpublish only 16. Splitting imposes a coarse group's supplement rate on industries whose own rates differ — the same error the decision was written to avoid, one level up — and misplaces 99,025, 0.95% of the row, against aT60200D-only method that is exact. NIPA publishing both halves by industry does not help when it publishes them at coarser grain than the total.3. ✅
T60200D's axis is the BEA summary axis. Not "about summary granularity" — its 69 leaves partition all 71 summary industries exactly, no gaps or overlaps, and 63 equal a summary industry's published compensation to the dollar. Verified by value, not by name. That is what lets each be its own control.V00300stays deliberately plain — eight controls, one distribution — because its four component tables publish on mutually incompatible partitions (20 / 21 / 12+financial / 63 groups) and three components have no axis at all. It keepsS00201at its published −36,919; nothing here may assume positivity.New extractor:
BEA_GDPbyIndustry— and it is the test setTable
TVA113, Components of Value Added by Industry, from BEA's release archive. No API key (the API dataset needs aUserID, the zip does not), cached underextract/input_datawith a GCS fallback, 2012–2024 declared. Root identity exact in every year: 10,434,978 + 1,304,097 + 7,873,027 = 19,612,102.It was built because it looked like the one source that would give
V00300a real industry axis. It is not, and the extractor is what proves it:V003match aTVA113surplus row to the dollarV001match compensation the same wayBEA's industry accounts and the SUTs are the same estimates published twice — so this is the summary Use SUT by another door, and Step 5's Decision 3 holds that out of the target set precisely so it can grade the build.
NIPA_VA_surplusis therefore left unchanged. Wiring it in is a two-line swap; what it buys is a near-exactV00300seed at summary grain, what it costs is summaryV003's ability to grade that seed. That trade is a testing-strategy judgement, not a data-availability one, so it is documented and left open for review.🚧 Blocker that lands on the later years
FBS_outside_flowsadoes not work as an attribution source:get_flowby_from_configbuilds aFlowBySectorfor thatdata_formatwhileattribute_flows_to_sectorsthen callsmap_to_sectors, which onlyFlowByActivitydefines. All four existing uses in the repo are top-level sources; this branch's was the first attribution-source use.That is the exact hatch the plan commits
V00100's anchor-and-move to, so it has to be cleared before the 2018–2024 files. Finding it on the 1.8% row rather than the 30.9% one is the argument for having builtT00OTOPfirst. ForT00OTOPthe cost of going without is bounded and measured: 1.92% against 1.68% of row error in 2024, less than the control's own 2.9% vintage error in 2021.2018–2024 are not written.
derive_initial_value_addedraises for any year but 2017 rather than silently returning a frozen block.What the diagnostic caught on its first run
Reading the 17 absent cells showed
T00OTOPis absent on eleven government codes, not the canonical ten: the prefix rule (S00,G) misses the US Postal Service,491000, a federal government enterprise whose BEA code is shaped like an industry's. It was landing on zero only because the benchmark weight happened to be zero, not because the rule excluded it. Now named explicitly. No 2017 number moves — which is why a total would never have found it.Verification
analysis/nowcastingNIPA_final_dom_uses_2017rebuilds bit-identical across the crosswalk changeother_taxes_allocation --checkandcompensation_allocation --checkgreen; both reproduce every figure quoted aboveGDPbyInd.zip(2.4MB) is cached locally and gitignored, same asFlatFiles.ZIP. It should be uploaded togs://cornerstone-default/extract/input-data/BEA_GDPbyIndustry/so others do not re-download — not done here, since that writes to shared storage.🤖 Generated with Claude Code