Step 3: the intermediate-uses block, reconciled onto nowcast (#497, #564, #698, #699, #700) - #742
Open
WesIngwersen wants to merge 42 commits into
Open
Step 3: the intermediate-uses block, reconciled onto nowcast (#497, #564, #698, #699, #700)#742WesIngwersen wants to merge 42 commits into
WesIngwersen wants to merge 42 commits into
Conversation
Step 3 freezes every industry's input structure at 2017 and carries it on a price index. For manufacturing the part of that column the annual surveys cannot refresh is the materials bill: AIES publishes all materials, parts and supplies as one cell, 82.5% of the column, so only 8.3% of manufacturing's intermediate is commodity-mappable annually (#564). The commodity breakout is quinquennial Economic Census, and 2022 is a second observation of it sitting close to the middle of the 2018-2025 span. #564 called that a consolation prize. Measured, it is the main prize. Census_EC_MatFuel pulls ecnmatfuel for both vintages -- 4,624 rows in 2017 and 4,399 in 2022, every industry at NAICS-6 and every material an 8-digit code. The orientation is the opposite of Census_EC_PxI on purpose: PxI asks what an industry sells, this asks what it buys, so the industry goes in ActivityConsumedBy and the material in ActivityProducedBy, which is the Use table's own orientation. materials_structure.py answers the two questions that decide whether the source is worth having. Coverage: 66.2% of the 2017 materials bill and 69.1% of 2022 is placeable on a BEA commodity, against 8.3% annually -- 52.9% and 54.0% resolving 1:1 by NAICS prefix, the rest onto a BEA group that needs a within-group split on 2017 Use shares. A third is residual buckets Census could not place, and that is the ceiling. Movement: the mix moved 0.153 between the two censuses against 0.173 for the whole Use column over 2012-2017, with 264 of 345 industries moving more than 10 points. So the largest and least-observed part of the manufacturing column moves as fast as the rest of it, and freezing 2017 out to 2025 discards a reallocation this source can see. Two traps are documented rather than worked around. 00772000 "Total Materials" is the industry total and the named codes sum to it exactly, so summing the FBA unfiltered doubles the table; it is kept because it is the control a suppression recovery subtracts published children from, the same role NAICS 00 plays for PxI. And the vintages sit on different NAICS bases, sharing 345 industries and 291 materials carrying 90% of each year's cost -- 336411 aircraft reallocating 59% of its materials bill is almost certainly a code reassignment, not economics, and is flagged as suspect. Not built yet, and named in the plan: suppression recovery against the 00772000 control, the group-tier within-group split, the vintage code diff, and the interpolation itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ted (#698) Census withholds 412 of 4,624 cells in 2017 and 330 of 4,399 in 2022. They are not zero: they sit inside each industry's published 00772000 total, and leaving them there biases a materials mix toward whatever happens to be publishable, which is systematically the large materials. The control turns out to be exact, and that is measured rather than assumed -- for every industry with nothing withheld the named materials sum to 00772000 to within 0.1%, 238 of 238 in 2017 and 247 of 247 in 2022, with fuels carrying their own exact control in 00772002. After recovery all 406 and 386 industry-by-kind controls close to within 0.1% and no negative cell is created; the one negative in the output is published Census data. The prior is chosen by holdout rather than by argument. An economy-wide prior was tried first and is visibly wrong: it hands an idiosyncratic industry the economy's shopping list, and put $8.6bn of motor vehicle seating into aircraft manufacturing while cutting its aircraft engines from $15.5bn to $0.5bn. Masking published cells and recovering them scores every peer-prefix length; NAICS-3 wins on average, at WAPE 0.602 and 0.718 against economy-wide's 0.640 and 1.033. Cross-vintage priors are deliberately excluded even though 2017 is the best predictor of a withheld 2022 cell, because filling 2022 from 2017 biases the movement measurement toward zero -- a recovery must not manufacture the answer the analysis is testing. That WAPE is the finding that matters, and it corrects the last commit. The mass a recovery places is exact, so all 0.6-0.7 of that error is allocation across materials within the column -- which is exactly what a mix score measures. Restricting to the 193 industries with nothing withheld in either year, the materials mix moved 0.1330, not the 0.153 reported before, and the count of columns over 0.25 collapses from 66 to 17. Most of the extremes were the fill, not the economy. 336411 aircraft at 0.592 was the loudest of them and chasing it is what found the defect: its 2022 column is mostly withheld. The argument survives, weaker and better supported: materials mix moves 0.133 over five years against 0.173 for the entire Use column over 2012-2017, with 133 of 193 clean industries moving more than 10 points. So the largest and least-observed part of the manufacturing column moves substantially -- somewhat less than the column as a whole, not more, which is what the previous commit claimed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…lation (#698) The last three #698 items are built. Two dissolved a problem the plan expected to fight; the third overturned the interpolation form the plan had already chosen. materials_structure.py becomes inputs_structure.py, because what these sources reach is no longer just materials. The group-tier split is settled on evidence rather than on which prior sounds more principled. A group cell is divided over the BEA commodities its NAICS could be, on the purchasing industry's own 2017 Use row; scored by demoting every direct cell one prefix and comparing against the commodity Census actually named, the column prior puts 72.0% and 72.9% of the money on the right commodity against 46.9% and 49.5% for an economy-wide one. Accuracy falls off with group breadth -- 79.8% at 2-4 commodities down to 51.9% at 10-29 -- but 73% of group-tier dollars sit in groups of nine or fewer. The bare 33 prefix, 136 commodities, should be read as barely better than residual. The split lifts the placeable bill from $2,097B to $2,681B and the commodities reached from 137 to 204. The vintage code diff turns out not to be a diff. The plan expected to lose 10% of each year's cost to the 2017-2022 revision and it loses none: the material axis shares 289 of 289 and 290 MATFUEL codes, and every off-frame dollar on the industry axis is NAICS 2022 merging pairs of 2017 codes. Connected components of the year concordance put 100% of both vintages on one 365-industry basis with no split assumption. This does not rescue 336411 -- aircraft was on the shared frame all along, so its score is the suppression fill and the plan's guess about why it mattered was wrong. Linear interpolation was called "the obvious first form". It is obvious and it is wrong, and seeing that meant stopping treating 2018-2025 as unobserved. Manufacturing's materials bill is published every year the census misses, so Census_ASM_Expenses and Census_AIES_Expenses now pull it -- ASM through 2021, AIES for 2023. A straight line overstates 2020 by 28.8% because it cannot bend around a pandemic, and past 2022 it gets the sign wrong: the bill fell 6.8% into 2023 and the line says it rose 5.0%. That is the span the nowcast leans on hardest. Scope has to be matched or a definition reads as a growth rate -- against CSTMTOT the census-to-ASM step is a median 1.181, on materials-plus- fuels 1.063, a year of inflation. Those extractors also buy more than a control. ASM and AIES publish electricity, contract work, resales and 9-12 named purchased services at NAICS-6, 11.7% of the manufacturing column, each mapping onto a BEA service commodity -- 91.0% of the column reachable in total, and a separate cheaper task with no suppression recovery or group split to do. AIES's expense block is manufacturing-only, though: sectors 21, 22, 23 and 51-81 publish nothing at any NAICS level, so it confirms #564 on the service drifters rather than overturning it, and it does not cover the mining that the census does. The headline moves down again, and this time because the frame was wrong. 0.133 is a MATFUEL-code score and the 0.173 it was compared against is a BEA detail commodity one. On the same frame the clean subsample gives 0.0941, so the materials block moves roughly half the column's rate rather than "somewhat less". Aggregating 289 materials onto ~200 commodities nets off within-commodity substitution and the split holds 2017 structure fixed inside each group; both are properties of the seed, not corrections to 0.133. Finally, BEA's 2022 and 2023 tables are still annual-survey updates carried over the 2017 benchmark, so differencing a census-seeded block against them is not a check. That cuts both ways, and the second way is the argument for this work: it is information BEA has not yet incorporated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…, #664) S3b was scoped as the cheap half of Step 3 -- ten named cells, no suppression recovery, no group split. It was cheap, but not for the stated reason: the work turned out to be a scope measurement rather than a mapping exercise, and two of the claims it rested on were wrong. The first is the coverage. 11.7% and 91.0% were survey-side dollars, and two of the largest entries are not purchases of a commodity at all -- resales are goods bought and sold on untransformed, which the Use table handles through trade margins, and contract work is manufacturing services whose commodity is the buyer's own industry rather than any fixed row. With the survey's own residual they are reached as expense but cannot be placed. The honest figures are 6.4% seedable and 85.8% reachable, and the seedable block is $228.6B over ten cells. The second is that these cells could be read as levels. They cannot. Census_EC_ Expenses is new and pulls ecnbasic for 2017 and 2022, which publishes the cells under the same variable names ASM uses -- so census and survey form one panel with no crosswalk, and the splice is continuous: 2017 electricity $47.5B against ASM's $51.0B in 2018, repair $52.7B against $55.2B. That 2017 observation is the year the benchmark Use table is built on, and against it the survey and BEA disagree about what the same cells contain by factors of 0.40 to 8.01. The disagreements are structural, not noise: expensed software and computers are operating expense to Census and mostly investment to BEA, repair is one Census question against four BEA rows carrying parts BEA books elsewhere, and professional services runs the other way, one question against BEA's legal, accounting, engineering, consulting and R&D rows together. So the seed moves BEA's cell rather than replacing it -- Use2017 times survey(t)/survey(2017) -- which cancels every one of those because a constant scope factor divides out, and preserves BEA's own level and its own split across the commodities of a multi-row kind. 2017 reproduces the benchmark exactly, which is the check that the form is right. The block carries the same pandemic signature the materials bill does, -1.7% in 2020, and a frozen 2017 understates it by 23.8% by 2023. AIES publishes no telephony and no expensed software -- both variables exist in the 2023 table and both are zero in all 883 rows -- so those two are held at the 2022 census and marked rather than seeded as a collapse. 2024 and 2025 raise rather than quietly extrapolating. Chasing scrap through the same sources then found a defect in the materials placement. Every MATFUEL scrap code begins 33, which is not a NAICS that maps to any single commodity, so the prefix walk was filing purchased metal scrap into the bare 33 group of 136 commodities and smearing it across most of manufacturing. That group was the weak end of the group tier, flagged in the last commit as barely better than residual -- and it turns out to have been entirely scrap. BEA carries S00401 for exactly this concept, $49.1B into manufacturing in 2017, and Census's "excluding home scrap" is the same thing: bought in rather than generated on site. The five codes now map straight onto S00401 ahead of the prefix walk, the 30+ band collapses from $32.1B/$54.9B to $2.5B/$0.6B, and the clean commodity mix score moves 0.0941 to 0.0949. The direct+group frame reaches 200 commodities rather than 204, and the four lost were reached only through the smear. Scrap is metal and only metal in this source -- no wastepaper, no cullet, no plastic regrind, no textile rags. Census reproduces BEA's scrap concentration independently and more finely, separating secondary aluminium at 0.628 from secondary nonferrous at 0.529, and iron and steel mills move 0.328 to 0.440 between the vintages. The fuller picture, including the output side where ecnpxi does carry paper and plastics as wholesale recyclable sales, is written up in cornerstone-data/methods#59. Census_EC_Inventories is extraction only, for #664. ecnbasic carries all three stages of fabrication plus the totals, beginning and end of year, at NAICS-6 for both vintages -- which is the industry x stage cell BEA does not publish anywhere, and the level Hill's rules operate at. Verified additive: the stages sum to the published total at a median ratio of exactly 1.0000. These are stock levels and differencing them imports the holding gains CIPI excludes, so they carry the stage shares and U50705BU1 keeps the level. The ASM annual equivalent is deliberately not pulled yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan carried 2024 and 2025 as years the estimate had to reach and could not source, which framed an unobservable extrapolation as a risk this work owned. It does not. AIES 2024 still returns 204 No Content, ASM ends at 2021 and the census is quinquennial, so the observed panel runs 2017-2023 and the estimate stops there. Extending it to 2024 is #707, Phase 2 work tied to producing a 2024 table, and 2025 is out of scope entirely. So the open question in S3 is now only which interpolation form to fit, scored on observed years, rather than which form to fit plus how far to extrapolate it past the data. S3b needs no extrapolation at all: its span is fully covered, and nonmaterial_seed() already raises for later years rather than inventing one. unobserved_years() keeps reporting 2024 and 2025 because that is still the fact a caller wants to assert on -- what changed is that they are outside the span rather than gaps inside it. Left alone deliberately: the references to BEA's published summary panel running to 2024, gross output extracted for 2017-2024, the theta fitted on 2022-2024, and FIWS covering to 2025. Those are statements about which data exists, not about how far this estimate reaches, and the column-scaling argument in particular is quantified on a 2024 seed and would lose its point without them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#705 asked, per drifting column, what could source it. Every one of them already had a source - BEA named one for each at the 2017 benchmark - so the question was whether a later vintage exists on the same basis and beats holding BEA's answer. Four no, one marginally yes. Census_SAS_Expenses splices SAS Table 5 across the two vintages that carry it, sas-17 (2013-2017) and sas-22 (2020-2022). #564 recorded Table 5 as "2020-2022 only"; that is the latest workbook's display window, not the series. 63 industries at 2- to 4-digit NAICS, ~19 mappable items. The two vintages sit on different Economic Census benchmarks, so every row carries the benchmark it was built on in Description. service_expense_seed.py indexes BEA's 2017 531ORE column on it and scores against BEA's current summary Use: +4.4 / +3.8 / +4.5% at 2020-2022, positive at every endpoint on a test biased against the seed, but at the inflation carry's bar rather than over it. --reachable says why: 18.07pp of ORE's movement sits on rows no survey item names against 13.51pp that a seed can touch, and the largest single mover has no counterpart question at all. Two corrections to earlier work in this branch: - An exploratory cut of this score reported +18.9 to +24.7% by applying each item's index to whole summary rows. Temporary staff maps to 561300, $8.6B of the column; at summary it multiplied all of 561, $97.8B, mostly 561700 services to buildings. A coarse commodity mapping inflates a result rather than blurring it. Build at detail, aggregate to score. - _load_usa_summary_sut pins the workbook by year, so --drift's series changes basis at 2023. The same 2022 read from both vintages differs by a dollar-weighted 0.0557 against a measured drift of 0.0986, and by 0.0976 for ORE alone. Near-misses elsewhere on the page are inside that noise. The seed scores on one vintage; fixing the shared diagnostic is follow-on work. Also records the negative results: the trade Business Expenses Supplement exists for 2017 and 2022 on one benchmark but loses every item of 4A0 to suppression; construction's ecnbasic pair is clean but 51% of the column is one undifferentiated materials cell; and ecnpurmode, ecnpurelec and ecnpurgas publish concepts BEA has no cell for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ives (#705) intermediate_structure_drift read summary Use through io_2017._load_usa_summary_sut, which pins the workbook by year: 2017-2022 from the 2017-2022 release, 2023-2024 from the 1997-2024 one. That is right for FBA consumers, whose published values must not move under BEA's revisions, and wrong for a module that differences years against each other -- it put a vintage seam between 2022 and 2023 in the middle of --drift's series. Read every summary year from the current workbook instead, via this module's own summary_use / summary_intermediate(year, workbook). io_2017's year-pinning is left alone. service_expense_seed.summary_intermediate_current, which had made a local copy of exactly this fix, is deleted in favour of the shared function; its scores are unchanged, confirming the two were equivalent. Add --revision, the same year read from both vintages. The revision table in intermediate_estimation_plan.md was not reproducible by any committed code, which broke that page's norm; the flag reproduces it exactly. The question this was for: does #705's top-drifter ranking survive one basis? It does, bit-for-bit. The seam sits between 2022 and 2023, but 2017 and 2018 are identical across the two vintages (one cell, $3M) and 2024 was only ever read from the current workbook, so the 2017-against-2024 comparison never crossed it. ORE is still 0.141 and the candidate list -- ORE, GSLG/GFGD, 42, 5412OP, 81 -- needs no revisiting. The feared shift, that GFGD and 521CI revise by more than ORE does, could not bite: the revision only starts at 2019. What did move is the middle of the series, understated by 0.005-0.011: 2019 0.0452 -> 0.0504, 2020 0.0743 -> 0.0833, 2021 0.0711 -> 0.0823, 2022 0.0838 -> 0.0859. 2018, 2023 and 2024 are unchanged. The 2022 ranking does not survive -- four columns in, four out of the top ten, ORE 0.084 -> 0.158 -- but nothing on the page ranks at 2022. Two numbers corrected rather than restated. The plan doc and the seed docstring both quoted a 2022 drift of 0.0986 that no code reproduces on either basis; replaced with the measured 0.0859. The seed docstring said the gain was 4.4% at 2022 where the doc and the code both say 4.5%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Margins.2 specified the margin rate as (TRADE + TRANS) / T013 -- margins over *basic* value. BEA gross output is valued at producers' prices (confirmed), so the price ratio #497 carries already contains the product-tax layer and the rate must be taken over producer value: T014 / (T013 + T015). One factor, not two. Also record the valuation chain the section had been using implicitly. T014 is the margins alone, not a running subtotal: T016 = T013 + T014 + T015, verified to $1M at detail, and independently stated in margins_estimation_plan.md. Restated on the right denominator. The level error from using basic is a median 3.3% and lands hardest on the commodities this section quotes: 315AL apparel 1.372 not 1.793 (+31%), 324 petroleum 0.246 not 0.289, 313TT textiles 0.779 not 0.858, 311FT 0.489 not 0.533. The panel is 26 commodities with a rate above 1%, not 36; median absolute change 3.6pp and p90 12.7pp, not 2.8 and 12.1. Most of the level error divides out of a ratio, so the correction to the carry factor itself is second-order -- median 0.35pp across the 26 receiving commodities above $20B, p90 1.4pp, at most 2.1pp among named ones. Worth having, not decisive; the section now says so rather than implying the fix is large. New guard, which the section needed and did not have. For margin *suppliers* T014 is large and negative -- the margin is allocated away from the trade or transport commodity onto the goods it carries, which is why the columns net to zero -- so mu is -0.94 for 42 wholesale, -0.99 for 486 pipeline, -0.88 for 482 rail. 1 + mu is then 0.06, 0.01 and 0.12 and the factor is a ratio of two near-zero numbers. Set it to 1 wherever mu <= 0. Nearly free: those rows carry almost no dollars in the purchaser-priced intermediate block, for the same reason their mu is negative. Finally, name the experiment. A missing deflator term and substitution under relative-price dispersion produce the same symptom -- a low theta on the summary panel against 1.00 on the detail one -- so theta must be fitted with and without this factor. If adding it pulls summary theta toward 1 the gap was the deflator; if not, the substitution reading stands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…egative Two results, and the second was not the one being looked for. Add the margin leg of the purchaser deflator -- summary_supply, summary_margin_rate and summary_margin_factor -- on the denominator fixed in 4fd1122: mu_c = T014_c / (T013_c + T015_c), margins over producer value, because BEA gross output is at producers' prices. Margin suppliers are held at 1.0, since their T014 is large and negative (42 is -0.94, 486 is -0.99) and 1 + mu is then a near-zero denominator. Add --theta, which fits the exponent with and without that leg. It was a genuine question: Margins.2 hypothesised that the summary panel's low theta was a missing-deflator artefact rather than substitution, and the two readings predict the same symptom. The hypothesis is wrong. Theta is unmoved in six years of seven and moves away from 1.00 in the seventh; the score differs by under 0.001 either way. Not a null test -- 26 of 73 commodities move, up to 10%, on the largest goods rows (325 chemicals x1.049 on a $705B row, 3361MV x1.047). It fails *because* of that: 22 of the 26 have a factor above 1, 17 of those lost intermediate share, and the touched set lost 3.26pp of the block. Both legs inflate nominal goods shares during real substitution away from goods. Keep the term as the correct deflator, worth a median 0.35pp on the carry factor; drop the claim that it explains theta. Then the larger result. THETA_GRID started at 0.0, which censored the panel: 2023 and 2024 both pinned to the floor and were read as "the carry contributes nothing". They fit -0.25 and -0.50. The frozen structure scores better when commodity shares are moved *against* their own price movement, so #497's theta = 1 is not merely too strong for the target years, it is the wrong sign. The grid now runs -1.0 to 1.5 and carries a warning about the floor. Detail 2012->2017 still fits 1.00 -- its optimum was interior, so nothing measured there moves -- and the +1.00 at 2020 to -0.50 at 2024 span is monotone in the price regime. Consequence for S1: theta must be a parameter, and its default for the recent span is negative rather than 0 or 1. A build that hardcodes 1 applies a correction pointing away from the answer on the block's largest rows. Guard added with the negative exponents: a zero ratio would now raise a fit to infinity rather than harmlessly to zero. No year has one today. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…704) S1 and S0b of intermediate_estimation_plan.md. `nowcast_intermediate.py` is #497 as scoped: seed the published 2017 detail Use SUT interior (402x402, purchaser, before redefinitions), carry each column's shares on the commodity price ratio at theta, renormalise, and scale the column to `GO_producer - VAPRO_seed`. `nowcast.derive_initial_U_intermediate` is the entry point. Runs 2017-2024, bounded by `BEA_Detail_GrossOutput_IO_<year>` rather than by the price index. S0b came with it rather than ahead of it: `use_intermediate_detail_sut` went into sections.py runnable, not `candidate=None`. The 2017 section run passes on every cell - 44,281 populated cells, 366 row totals and 400 column totals inside tolerance, 100% coverage and accuracy. That is the plumbing, not the movement: at 2017 every carry factor is 1.0. The 2017 rescale is not the identity, and the residual is BEA's own rounding. Published T005 is one rounded number; the interior sums 402 separately rounded cells to a different one - $350M on $14.9T, at most $13M on a column. A small column wears that as a large fraction, so `atol` carries those cells rather than `rtol`: 334610 is $482M of intermediates and is rescaled 1.05%, the largest relative error, against a largest absolute error of $6.0M on a $19.2B cell. `reproduction_check` reports both, because either alone reads as the wrong kind of error. The seven published negative cells survive in all eight years and are not clipped. The two structurally empty columns - 4200ID and 814000 - stay empty, and a control that puts real dollars on one raises rather than dropping them. So does a seed column whose nonzero cells cancel: an empty column has no structure to normalise, a cancelling one has structure that cannot be written as shares of its own total, and collapsing both to all-zero would lose that. theta is an argument, defaulting to #497's 1.0. It fits negative at 2023 and 2024; choosing it stays #699. The column control is well levelled and badly allocated. Step 2 is unbuilt, so `vapro_seed` freezes 2017's VA share of gross output, which collapses the control to `GO(t) x T005(2017)/GO(2017)` - gross-output movement and nothing else. Scored against the published summary T005 (`--control`): within 2.3% economy-wide in every year 2018-2024, but weighted MAE by industry runs 2.5% to 8.0% between 2018 and 2022, with GSLG 18.3% low at 2022. Two corrections to claims this branch had been repeating. GSLG topping that list is the column *level* and is not #578. #578 is the commodity *mix* inside the G* columns, sourced from govslocalfin's function x object split and sequenced at S5 behind a go/no-go. Separately, NIPA T31005 matches the published government T005 at $0/$0/$2M in 2017 and is referenced in no source file - so the worst cell of the control table is the one with an exact annual source already identified and never wired. And "Step 5 imposes both margins hard, so Step 3 estimates a shape not levels" holds only once Step 2 exists. Checked in nowcast_targets.py: T1 is hard and real for 2017-2024 but pins the column's sum, not the split. T4 and T6 are both soft and still PLACEHOLDER, and `va_row_targets` does `del year` and reads its values off `published_2017_panel`. T5 - T00OTOP and V00300 - is deliberately not imposed, entering as seed only so the income side stays out-of-sample evidence; V00300 is $7.873T. The balance cannot re-derive VABAS, and there is no VA seed for any year but 2017, so it cannot run on 2024 at all until Step 2 lands. The shape is insulated from this - it is renormalised before the control is applied, so a wrong control rescales a column without moving a share inside it - but the level is not, and the module and plan doc now say so. Tests are structural and run on a toy panel: the arithmetic (`carry_shares`, `apply_column_control`) is split from the data wiring, so they need neither GCS nor the gross-output parquet. The year-by-year numbers are CLI flags on the drift diagnostic (`--seed`, `--control`) rather than tests, so everything quoted above is reproducible from committed code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…control
BEA publishes intermediate inputs (UII205-A) and value added (UVA205-A)
annually 1997-2024 on its 191-row "underlying" industry frame, in two
workbooks sitting beside the GrossOutput.xlsx this repo already reads.
Nothing read either one. This extracts both, allocates them to the 402
detail industries, and replaces Step 3's frozen-2017 VAPRO seed.
The 191->402 mapping is derived, not hand-written. UGO205-A and UGO305-A
order industries the same way, so the 138 leaves of the 205-A hierarchy
partition the 414 rows of 305-A into contiguous runs; the runs are closed
by matching gross output in all 28 years. All 138 leaves match, exactly
414 of 414 detail rows are consumed, and the 402 codes cover the model
schema once each. derive_underlying_line_mapping reproduces the
checked-in constant exactly, and --mapping on the new diagnostic is how
to re-check it when the BEA vintage moves.
Value added is allocated and intermediate inputs are taken as the
residual GO - VAPRO. That is what makes GO = T005 + VAPRO hold per
industry, which is the form T1 imposes; allocating both independently
broke it by up to $15.2B a cell at 2024. It costs nothing against BEA:
- VAPRO summed back to the 138 lines reproduces UVA205-A exactly
(0.0 on 3,864 cells); T005 reproduces UII205-A to $9M, which is
BEA's own GO=II+VA rounding.
- At 2017 the derived columns reproduce the published detail Use SUT
margins to $0.89M (VAPRO) and $1.48M (T005) per industry.
- Economy-wide VA matches published GDP to at most $9M on $29T in
every year 1997-2024.
- GO - T005 - VAPRO at detail is 2.9e-11.
It also removes the suppression problem. BEA suppresses intermediate
inputs in every year on lines 83 (Customs duties) and 176 (Private
households); both are single-industry lines whose published 2017 T005 is
zero, and the residual recovers that zero rather than needing a special
case.
Step 3's column control is now GO - VAPRO with both sides observed.
Aggregated to summary and scored against the published summary T005 it
is within 0.00007% economy-wide and 0.00023% weighted MAE by industry in
every year 2018-2024, worst summary industry 0.003%. The superseded
frozen-ratio seed scored 0.2-2.3% and 2.5-8.0%, with GSLG 18.3% low at
2022. That is a consistency check and not an independent validation --
UII205-A and the summary Use SUT's T005 are the same BEA estimate
published two ways -- so what it establishes is that the allocation adds
back correctly and the control now *is* BEA's T005. vapro_seed becomes
vapro; the now-dead _row helper and published_gross_output import go.
Signs are carried, not clipped. S00201 has a published 2017 VAPRO of
-$10,069M and stays negative in all 28 years. The residual T005 goes
negative in 11 cells, all in 5191A0 and all in 2002-2015, so the
2017-2024 nowcast span is clear; the other residual direction was worse
(13 spurious negatives across three industries, reaching -$13,117M).
This is not Step 2. VAPRO is the column total; Step 2 owes the split
across the five value-added rows, and T4/T6 are still soft placeholders.
What changes is that there is now a VA level for every year rather than
for 2017 alone, so the balance can run on 2024. It also arrives off the
P1-gated path -- a straight Excel read on the shape of the working
UGO305-A loader, so map_fbs_sectors_to_model_schema never enters it.
Repo-wide: black, ruff, mypy clean (bar the four known Windows-only
settings.py errors); 779 passed, 2 skipped, 1 xfailed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
allocate_underlying_to_detail is public and takes an arbitrary mapping. A code appearing under two lines would be indexed twice by .loc[children] and written twice into the output, so the group totals would silently stop adding up rather than raising. Same for a duplicated line in group_values, where .loc[line] returns a frame instead of a row. The checked-in mapping has neither -- 402 codes, all distinct -- so this guards the function, not the current data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
BEA publishes 2007, 2012 and 2017 detail Supply and Use SUT as one zip of per-table workbooks with a sheet per year, all three already on the 2017 code basis in one frame. It was a local drop that only the Step 3 drift diagnostic read, through an ad-hoc zip reader. io_2017 now carries `_load_benchmark_detail_supply_use_usa(matrix, year)`, GCS-backed like every other table, with `load_benchmark_detail_U_intermediate_usa` and `load_benchmark_detail_supply_usa` as the typed 402 x 402 accessors. The panel is a second and third observation of every structural question in the build -- Step 3's input mix, Step 4a's commodity mix, the margin rates, the FD splits -- not just this one. The panel's 2017 sheets are the single-year workbooks cell for cell, on both matrices: 0 differing cells on 413 x 424 and 405 x 415. `assert_benchmark_panel_matches_2017()` is the check. The two loaders are kept separate anyway, because `_load_2017_detail_supply_use_usa` is what `bea_parse` emits as the BEA_Detail_Use_SUT / BEA_Detail_Supply FBAs and those stay pinned to their published workbook; the FBAs are not extended to 2007 and 2012 here. BEA's subsidy sign convention holds on all three years, so `_assert_bea_subsidy_signs` runs on every year the panel loader returns. `--holdout` reproduces unchanged, including the 2012->2017 best theta of 1.00. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…699) S2's experiment, and the answer is not the one the plan expected. Every theta on the plan's table starts at 2017, so elapsed years, cumulative inflation, price dispersion and structural drift all move with the calendar and none can be told from the others. The summary Use SUT publishes 1997-2024 and the price index reaches 2012, so 78 non-nested spans are free -- different bases, different lengths, different inflation. On those (`--regime`): crosses the 2021-22 surge R2 0.613 cumulative price level R2 0.525 elapsed years R2 0.142 relative-price dispersion R2 0.014 So the dispersion candidate the plan named is dead -- 1.4% of the variance, and the coefficient points the wrong way -- and so is elapsed time. Holding span length fixed, spans that cross the surge fit theta 0.0-0.5 and spans that do not fit 0.7-0.9, at every length from one to nine years. `default_theta` ships that: 0.75 off the surge, 0.0 across it (fitted 0.755 and 0.141). Rounding up to zero rather than to the target spans' own -0.25/-0.50 is deliberate -- a negative theta says nominal shares move against their own price, and it buys 0.6%. The median gain of the best theta over a frozen A is 5.44% of the score off the surge and 0.59% across it, so in the regime this build targets the carry is worth well under one percent however theta is set. What #497's theta = 1 cost was the 12.6% it gave away by pointing the wrong way. The margin-rate leg is built too, so the deflator is the purchaser one. The non-obvious part is reaching 402 rows annually: detail Supply is published only for benchmark years, so the rate's level is detail-observed at 2017 and only its movement is borrowed from the summary parent. S0a made that testable -- against the observed 2012 detail factor, weighted by 2017 intermediate dollars, the shipped rule is 0.756pp off, the parent's factor taken down unchanged is 1.010pp, and no factor at all is 1.818pp. A year with no published Supply table is refused rather than carried: MARGIN_YEARS (1997-2024, BEA's vintage) is a separate constraint from INTERMEDIATE_YEARS (gross output). They agree at 2024 today, so nothing is blocked; a 2025 build would reach a year with one and not the other, and a silent factor of 1.0 there would read as "margins did not move". The margin leg is inert in exactly the years the build targets -- at theta = 0 every factor is raised to the zero power. It moves 0.21% of the 2019 block and 0.54% of the 2021 block and 0.000% of the 2024 one. Kept because it is the correct deflator for a purchaser-valued cell, not because it changes the current answer. Also restates the plan's Step 3 sections that PR #712 left describing `vapro_seed`: the level table (the block total is theta-independent, so it is a check on the control), the column control table, and the GSLG/T31005 paragraphs. 2017 reproduction is unchanged -- $6.02M max absolute, 1.05% max relative, seven negatives, and the section still scores 100% on 44,281 cells. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`intermediate_estimation_plan.md` covers the theta findings but never states the mechanics, and the name invites reading it as a price ratio or a valuation bridge. It is neither: it is a scalar exponent, one per span, applied identically to all 402 commodity rows and all 402 industry columns. `About_the_price_carry.md` is the reference for that: the carry formula, the two legs of the commodity deflator and their sources, the BAS -> PRO -> PUR chain the margin leg sits in (and why its denominator is producer and not basic value), the theta = 1 - sigma CES reading, what is commodity-specific and what is not, the shipped values, and the approximations the carry rests on. Linked from the nowcasting README, from the plan's Inflation section where a reader first meets theta, and from the module docstring. The README's intermediate_structure_drift entry also gains the --regime flag added in 7761934. Every claim in it re-checked against the code: grid -1.0 to 1.5 in 0.25 steps, theta 0.75 for 2018-2021 and 0.0 for 2022-2024, MARGIN_YEARS 1997-2024 against INTERMEDIATE_YEARS 2017-2024, the four unpriced commodities, and mu of -0.944 for `42` and -0.989 for `486`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan sequenced S5's government leg behind a go/no-go on whether Census `govslocalfin`'s function x object split bridges to a commodity mix. It has been run and the answer is no. The model that premise implies has an exact ceiling: with within-function mixes held fixed, commodity-mix movement is bounded by the function-mix movement itself, since each within-function mix sums to 1 over commodities. So the question is answerable without building the bridge. - the function mix moves 0.0464 over five years, against a 0.201 commodity drift in the same columns; - government functions overlap by 36% (mean pairwise dissimilarity 0.639 on BEA's own government columns), so the realistic ceiling is 0.030; - realising the bound on BEA's three general state-and-local columns -- a real function split with observed weights and observed within-function mixes -- buys +2.4%. 97.6% of the drift is *within* function. A finer function list does not help: BEA's 3-function split moves 0.0453 where govslocalfin's 33 functions move 0.0464. Two independent defects would also have sunk it. `Current Operations` by function is published only for 2022-2024, so there is no 2017 anchor at the seed year -- the 137->232 row jump the source note flagged is exactly this. And the only continuous function series, `Total Expenditure`, is a 0.062 proxy for it, larger than the 0.0385 of movement it would carry. `govslocalfin` is also state-and-local only, and federal holds 43.1% of the block's misplaced dollars including S00500, the worst column in the table. This closes govslocalfin as the route to the `G*` mix. It does not close #578: the columns still drift 230-258 $B and the drift is within-function. Corrects two claims in annual_survey_expense_sources.md that are now known wrong -- that the `G*` columns need a total rather than a mix, and that Current Operations by function runs for every year in the span. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The form left open by S3 is settled, and the candidate the plan named is not the answer. Separate the two things called "interpolation" first. The materials LEVEL is observed every year (ASM to 2021, AIES from 2023), so it never needed interpolating and section 4 rules out doing so. The commodity MIX is observed only at 2017 and 2022, and nothing observes its interior -- the fuels share of the census universe stays between 0.98% and 1.14% -- so the open question was only ever about the mix. The price-carried path is rejected. Carrying the 2017 census mix to 2022 on the commodity price ratio fits theta = 0.00 on the unsuppressed frame and -0.25 (+0.4%) on the full one. The level moves with price; the mix does not. The published summary panel cannot arbitrate the form and its answer must not be used: BEA carries the last benchmark forward, so every interior year of a summary span is itself an interpolation, and since BEA has not taken up the 2022 Economic Census its 2022-2024 tables are still 2017-benchmark carries -- so spans ending in the nowcast horizon are contaminated at the endpoint too. The benchmark detail panel can arbitrate it, because 2007, 2012 and 2017 are three independent Economic-Census-anchored observations. Interpolating 2007 -> 2017 and scoring at the observed 2012, manufacturing: frozen 0.0889, linear 0.0764, geometric 0.0710, endpoint 0.1216. So interpolating beats freezing by 20.1%, geometric beats linear by 7.1%, and adopting the newer observation early is worse than freezing. But do not extend the trend past the last observation: reaching 2017 from 2007 and 2012, holding the 2012 mix scores 0.1232 against 0.1569 linear and 0.1513 geometric -- 27.4% worse on manufacturing, 41.7% on the whole table. S3 therefore ships an asymmetry: geometric interpolation between the two censuses, and the 2022 mix held flat for 2023 and 2024. materials_seed() is that seed, an index on the share with the column total held, moving the block 0.0979 off frozen 2017 by 2022 -- alongside the independently computed 0.0949 in the mix score section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…564) Section S4 rejected SAS Table 5 for services and generalised that into "#564's negative result generalises". It does not survive, for four measurable reasons. The weighting was wrong. Every ranking in the plan -- which columns drift, which sources are worth building, what "a small prize" means -- is dollar-weighted. Cornerstone is an EEIO model, so the quantity that matters is kg CO2e, read here from the shipped v0.3 B_USA_non_finetuned snapshot characterised to CO2e. On the 2017 detail block: manufacturing 31-33 $4,699B 31.6% of dollars 28.0% of impact utilities 22 $350B 2.4% 26.1% agriculture 111/112 $413B 2.8% 23.3% mining 21 $510B 3.4% 11.5% all other (services) $8,260B 55.6% 7.2% Utilities and agriculture are 5.2% of the dollars and 49.4% of the impact. "All other" -- every column S4 ranked -- is 7.2%. The top 10 commodity rows carry 65.3% of direct impact and 221100 electricity alone carries 25.2%. Two consequences: agriculture (#577) is not "a small prize", and purchased electricity in the service columns is the highest-value cell in the step. The AIES finding was an artefact of the wrong endpoint. Census_AIES_Expenses queries timeseries/aies/basic, where every service row is a well-formed zero. timeseries/aies/exp02 publishes all 41 expense variables for 13 service sectors in 2023. Both source notes are corrected. The resource is 97 BEA detail industries holding $6,269B (42.2% of the block), not the one column the built seed uses. Dollar-weighted reach is 41.4%; impact-weighted reach is 63.8%, and per column the gap runs to 65pp -- 481000 reaches 53.8% of dollars and 95.6% of impact. The benchmark seam is not the blocker. On matched three-year spans it adds 3.1% to level movement over an equal span inside a vintage, while also containing 2020; on energy items the seam excess on shares is -7.4%. Shares are 30-54% quieter than levels within a vintage, so a share-based index is right on its own merits and cancels a proportional rebenchmark for free. Also records that wholesale and retail are absent from exp02 entirely and that ecnbasic carries expenses for sectors 21, 23 and 31-33 only, in both 2017 and 2022 -- so the trade counterpart remains the AWTS/ARTS supplement, whose suppression rejection is untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Builds the source re-evaluation in the parent commit into an extractor and a seed over 103 BEA detail columns, against the one column ORE covered. New source Census_AIES_Service_Expenses reads timeseries/aies/exp02, which carries the 13 service sectors that Census_AIES_Expenses' basic endpoint returns as well-formed zeros. It is deliberately a separate source rather than a repoint: basic uniquely publishes EXPS_CSTMTOT_DVAL and RCPT_TOT_VAL, the controls the materials additivity check uses, and exp02 does not have them. Verified exp02 is a strict superset of basic on manufacturing -- 648 rows, 0 differing cells. Census_AIES.py's parse filters on a hardcoded AIES_EXPENSE_FLOWS whitelist, so seven sector-specific cells were silently dropped until added there: transport fuel, transport repair, freight, medical supplies, insurance, professional liability and printing. service_seed(year) moves 103 columns holding $6,939B. relative_index now accepts an injected panel so a year can be read from a different survey than its base, and SEEDED_ELSEWHERE keeps the seed off the sectors inputs_structure already covers. Scored against a frozen 2017 on BEA's published summary, the seed wins on the SAS years and wins three to four times bigger on impact than on dollars: 2020 +1.1% dollar +3.6% impact 2021 +1.2% dollar +3.8% impact 2022 +0.4% dollar +4.2% impact 2023 -5.5% dollar -5.8% impact That is the re-evaluation's thesis holding up -- the survey names purchased electricity and fuels for every industry, and those are the rows the model weights. The dollar gains are inside the noise S4 rejected other columns on, so the impact column is the case for this seed and should be quoted as such. 2023 is refused. It is AIES read against a SAS 2017 base, so it crosses the survey change on top of the sas-17/sas-22 rebenchmark; the splice step is a median |log| of 0.203 against 0.118 within-instrument. When AIES publishes a second year, 2024 against 2023 is within-instrument and this becomes usable. 2018 and 2019 get no seed and are not interpolated -- they sit on the seam. Also fixes a units error the work surfaced: the SAS FBA is USD and the AIES FBA is Thousand USD, so the two panels need different divisors to meet in $M. It is invisible in relative_index, which divides it out, and wrong for anything reading a level. The 2023 score failure is not caused by it and is unchanged after the fix. Gates: ruff check and mypy clean; pytest NOT run on this commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The materials interpolation commit d722010 left three constructs black 24.10 rewrites: a parenthesised return annotation on _census_mix_frames, a split %-format key in interpolation_forms' assign, and a nested conditional in the census-mix treatment label. Formatting only -- no behaviour changes. This is the format gate failing on PR #715 and, inherited, on #717 and the service seed branch stacked above it. Fixed here so one change clears all three rather than each branch carrying its own. Gates: black, ruff, mypy clean; pytest 795 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… sectors Three corrections to the seed the parent commit built, all from checking our sourcing against what BEA says it used (Table C2, now in the repo as bea_2017_benchmark_sources.md). 1. The denominator was wrong. relative_index divides each item's ratio by the industry's own growth, and that growth was computed over the MAPPED items only. BEA names fourteen SAS items for these industries; SAS_ITEM_TO_BEA covers thirteen, and the two it cannot -- expensed purchases of other materials, parts and supplies ($327B) and all other operating expenses ($2,005B) -- are $2.33T of the panel's $13.1T. Using the mapped subset silently asserted those two grew at the mapped average. industry_growth() now runs over intermediate_items(), the whole input bill. Worth about +0.8pp on impact, and it costs the 2020 dollar figure: the denominator is right now, not tuned. 2. A scope guard was built and is withdrawn. It refused industries whose own input bill contradicted their published column, named six of them, and appeared to be worth 5x on impact. Scored on the full bill instead of the mapped subset, only 486 fails -- and 486 is one of the seed's better columns (+54.7% at 2020), so a level disagreement does not predict a bad shape. The six-industry list was measuring the item map's incompleteness and reporting it as a fact about the survey. published_agreement() and contradicting_industries() stay as diagnostics so the negative result stays reproducible; nothing is excluded. 3. Utilities is held at the benchmark instead (NOT_SEEDED), on the mechanism rather than on the score. BEA did use SAS for utilities, so "different universe" is not available. What is left is that utilities' column moves on a price-driven reweighting -- 211 goes 7.7% -> 23.3% of it as the column rises 160.4B -> 258.4B -- while the survey's purchased-fuels line falls 102.5B -> 79.6B. A relative share index divides out common movement, which is the signal here. It was the largest single drag: -119% of the aggregate gain at 2020. Its drift is now unaddressed, not fixed; the EIA route (861/861M, form 176) is filed separately. This is only utilities' OWN input column, 3 detail columns and $160B. The purchased-electricity and purchased-fuels ROWS in the other 100 columns are still survey-seeded, which is the high-impact part of the seed. Scored against a frozen 2017 on BEA's published summary: 2020 +0.5% dollar +22.6% impact 2021 +1.4% dollar +25.1% impact 2022 +2.3% dollar +22.6% impact Part of the rise from +4.5-5.0% is a smaller denominator: holding 22 back removes a column carrying half the block's frozen dissimilarity. The seeded error falls too, 0.0318 -> 0.0128, so it is not only rescaling -- but this should not be quoted as "5x better" than the figures with 22 in. Renamed service_* to services_transport_* throughout, and the module with it. SAS is the Service Annual Survey but it also publishes utilities and transportation, and naming the block after the survey blurred exactly the distinction that decided NOT_SEEDED. sector_coverage() states the scope on BEA's own sector taxonomy: seven sectors, FIRE PROF 6 51 7 48TW 81, all services or transportation. New CLI flags --sectors, --by-column, --agreement. Also adds services_transport_score_by_column, which is what re-scores S4's per-column no-goes on impact: 622, 722 and 486 win consistently, 5412OP does not. The aggregate is carried by weight, not count -- ORE alone is 71-78%. Gates: black, ruff, mypy clean; targeted Step 3 tests pass (28). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit renamed the seed module and left service_expense_resource.py importing the old name, which broke it at import time. My blast-radius grep had excluded the resource file from its own results, so the breakage did not show up until something imported it. Renames the resource module to match and repoints every reference, in Census_AIES.py's source note as well. Gates: black, ruff, mypy clean; both modules import. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wes, 2026-08-25: rank by total kg CO2e per dollar -- direct plus indirect --
not by the direct slice. A Use cell is an entry in A, so an error in it
propagates through the whole Leontief inverse; what a row is worth getting
right is its total embodied emissions. Some sectors have much higher N than D
because their indirect is large, and their inputs are still very important.
total_impact_intensity() computes N = C B L with L = (I - (Adom + Aimp))^-1,
matching the inverse cornerstone_disagg_pipeline builds N from. impact_intensity
now takes kind= and defaults to 'total'; direct_impact_intensity keeps D so the
contrast stays legible.
The ranking moves a long way. On the 2017 detail block, by commodity row:
dollars D N
manufacturing 31-33 31.6% 28.0% 44.7%
agriculture 111/112 2.8% 23.7% 16.4%
all other (services) 55.6% 7.2% 14.3%
utilities 22 2.4% 26.1% 13.7%
mining 21 3.4% 11.5% 8.0%
Manufacturing nearly doubles, because a manufactured good carries a long
upstream chain. Utilities halves, because electricity's emissions are almost
entirely direct and D therefore flatters it. Services double. Individual rows
move harder still: 31161A meat processing goes 0.10% -> 2.61%, a 26x change
that is all upstream livestock, and 531ORE goes 1.48% -> 3.97%. The block is
less concentrated on N -- top 10 rows are 48.5% against 65.3% on D.
The seed's score, re-run on N:
2020 +0.5% dollar +10.2% impact (22/33 columns win)
2021 +1.4% dollar +10.8% impact (21/33)
2022 +2.3% dollar +9.2% impact (17/33)
On D these read +22.6/+25.1/+22.6%, so D was overstating the seed by more than
double -- it over-weights electricity, which the survey names well.
It also flips a verdict. Re-scoring S4's per-column no-goes on N: 622
(+11.4/+12.3/+27.5%) and 722 (+7.5/+11.2/+10.6%) overturn, 5412OP still loses
(-9.0/-7.7/-2.0%), and 81 does NOT overturn (-0.8/+2.9/+0.6%) although it
looked like a consistent win on D. More columns win on N than on D, because N
spreads weight off the few electricity-heavy rows.
Every ranking recorded before this is a D ranking, including the table PR #717
re-sorted the plan on; the resource module's docstring is restated accordingly.
Gates: black, ruff, mypy clean; targeted Step 3 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…577) C2 says BEA estimates agriculture's inputs from "BEA NIPA estimates based on USDA ERS statistics for farm income, EIA data, and 2017 Economic Census data", so ERS Farm Income and Wealth Statistics is the source BEA itself used. The extractor was already in bedrock. Its parse filtered to Cash receipts alone and dropped intermediate expenses entirely -- the same class of bug as Census_AIES's AIES_EXPENSE_FLOWS whitelist, and it made a concept ERS has published since 1910 look absent. Three separate silent filters, all fixed and all now tested: 1. KEPT_CONCEPTS replaces the hardcoded Cash receipts test, as prefixes. Adding a concept there is the only way to surface it, which is now said in one place. 2. The concept prefix is 'Cash receipt', singular. ERS writes the concept as 'Cash receipts value' everywhere except bell peppers, which are 'Cash receipt value' -- 129 rows over 2008-2023. The old contains('Cash receipts') test dropped them: $626M of $1,470,607M in 2017, 0.04%. A source typo, not a real distinction, so this changes the cash receipts total by that amount. 3. Operator-dwelling variants are dropped. Several expense categories publish both an 'excl.' and an 'incl. operator dwellings' series, identical in VariableDescriptionPart2, so they collide in ActivityProducedBy and double any groupby. Farm dwellings are not an intermediate input to farming, and the 'excl.' series is the one that reconciles. 4. Expense activities take the full published description. Four series share Part2 = 'Miscellaneous' -- the group total, insurance premiums, federal insurance premiums and irrigation -- so the Part1+Part2 name is not unique for expenses and a groupby would add a group to its own members. Cash receipts keep the Part1+Part2 name, which is what Sector_Crosswalk_USDA_ERS_FIWS.csv is keyed on. Verified against the real February 2025 file. 2017 intermediate product expenses, excl. operator dwellings, is $226,611M, and farm origin $104,467M + manufactured inputs $56,316M + other intermediate $65,828M reconciles to it exactly. 2025 is $298,474M, matching #577's table. The declared years ran to 2023 while the file already carried 2024 and 2025; extended to 2025. 2025 is an ERS forecast, not a realized estimate, and is flagged as such. Irrigation is discontinued after 2023 -- 18 categories in 2024-25 against 19 before -- so a consumer must not read its absence as zero. Widened rather than forked: no FBS method or Python module consumed this FBA, and the inventories plan needs a variable the same filter was dropping. Gates: black, ruff, mypy clean; 7 new tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan still described a state three findings out of date: services rejected, agriculture a small prize, ORE marginal, every ranking on dollars. New section: What a column is worth. Sets out N against D against dollars, and why N is the ordering to source on -- a Use cell is an entry in A, so an error propagates through the whole Leontief inverse. Carries the row-group table and the warning that any ranking dated before 2026-08-25 is a D ranking at best. Also states the rows-versus-columns trap explicitly, since agriculture rows are 16.4% of impact and agriculture columns are 6.9%, and they are different questions. New section: What is still held at the 2017 benchmark. The backlog as an inventory rather than a feeling -- 50 columns, $3,577B, 24.1% of the block, broken out by BEA sector with the reason and issue for each. It also answers the question of how to get them back, and the answer is not uniform: the geometric-interpolation form is settled, but the Economic Census expense breakout covers sectors 21, 23 and 31-33 only, so "rebenchmark at 2022 and interpolate" is the right shape for trade alone. Agriculture and utilities have annual sources and should be seeded directly. For trade the work is a suppression recovery, not an extractor. S4's verdict table is no longer the answer. Re-scored on N, 622 (+11.4/+12.3/ +27.5%) and 722 (+7.5/+11.2/+10.6%) overturn; 5412OP still loses and 81 is flat -- and 81 is the cautionary case, because it wins on D. What stands from S4 is 42, 4A0 and 23, whose problem was suppression and reach rather than weighting. S5 rewritten for the ERS extractor, which is built. It reconciles exactly at 2017 and agrees with BEA's published 111CA within 0.988-1.047 over seven years. But #577's rationale -- "the value is in the levels" -- does not survive: the level is already observed through GO - VAPRO, so the value has to be the mix, which is what #577 says not to expect. Recorded with the mix actually measured. New S6 for utilities from EIA (#719), which exists because the services seed had to refuse that column, and ORE re-titled: on N it gains +25.1/+32.6/+37.2% and carries 59-83% of the block's gain, so "marginal" was a dollar verdict. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The extractor in the parent commit made the seed buildable. Built, measured on BEA's published 111CA for 2018-2024, and it does not earn its place. year dollar gain impact gain 2018 -6.7% +19.1% 2019 -17.6% -42.9% 2020 +10.5% +4.6% 2021 +0.4% -18.8% 2022 -5.8% -10.5% 2023 +3.0% +10.7% 2024 +7.3% +33.5% Four of seven years positive on each weighting, swinging from -42.9% to +33.5%. That is noise around zero, not a seed that tracks. leave_one_out finds no culprit: feed contributes +8.0pp at 2022 and -5.9pp at 2024, livestock purchases -5.1pp then +46.9pp. A category that helps one year and hurts the next is not a mapping problem. The mechanism is absent, and that is the finding. Comparing each category's share movement against its BEA commodity row's share movement, six cleanest pairs over seven years: electricity +0.61, pesticide +0.38, petroleum +0.27, livestock +0.16, feed +0.13, fertilizer -0.38, pooled +0.18 over 42 observations. ERS category shares do not track BEA's commodity shares. The explanation is consistent with everything else measured here. BEA uses ERS for the farm income LEVELS -- which is why published_agreement runs 0.988 to 1.047 across seven years, the closest agreement of any source in this step -- and distributes across commodities by its own means. A category-share index cannot reproduce a commodity mix that is not built from category shares. #577's instinct was right for a reason it did not give. It says not to expect a shifting mix because the mix is stable; the stronger reason is that where it does move, it does not move with BEA's. Also records what this does not establish: the test is at 111CA summary because that is the only place a later year is published, so detail-level mix is unobservable; fertilizer and pesticide both collapse to 325 at summary; and the span contains the 2022 rebenchmark. The module is kept as the record of a tested no-go rather than deleted, in the same way the government function bridge is. What is worth keeping outright is the extractor: it surfaces a concept it was filtering away, three silent bugs are fixed, and it reaches 2025 for Step 2 and inventories. Gates: black, ruff, mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
S5 said "source extracted, seed not built". It is now built and rejected: 4 of 7 years positive, swinging -42.9% to +33.5%, and ERS category shares correlate 0.18 with BEA commodity shares over 42 observations. BEA uses ERS for farm income levels and distributes across commodities by its own means. The extractor stands and is the section's real yield. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…son (#564) S4 rejected the trade Business Expenses Supplement on suppression and left the door open: "no-go now, reopen only behind a suppression recovery". Retested with three of that verdict's objections removed. The answer is a firmer no, and the suppression recovery it pointed at should not be built. What was wrong with the first test: 1. It was run at BEA summary and Step 3 estimates BEA detail. "4A0 loses every item because it spans nine three-digit NAICS" is a summary artefact -- at detail 4A0 is six columns and five of the six are a single three-digit NAICS, needing no aggregation at all. 2. It scored items BEA did not use. Its account of 452 rests on contract labour x4.45, and contract labour is not one of BEA's thirteen trade items (Table C2). BEA_ITEMS is BEA's own list. 3. It was dollar-weighted. Re-scored at BEA detail, on BEA's thirteen items, at 2022, both weightings: 42 -3.3% dollar -2.5% N 441 -24.4% -43.6% 445 -24.0% +5.4% 452 -132.7% -183.6% 4A0 -3.3% -31.8% Every column loses on dollars and four of five on impact. 441 is the decisive one: 13 of 13 items published in both years, no aggregation, 61.5% dollar and 75.7% impact reach -- and -43.6%. Neither suppression nor reach is the binding constraint, so recovering suppressed cells buys nothing. That is a build saved. The mechanism is absent, and it is the same one that sank ERS agriculture. Each BES item's movement against its BEA commodities' published share movement correlates +0.06 pooled over 54 pairs, against agriculture's +0.18. A survey can measure an industry's expenses well and still not describe how BEA moved the commodity mix, because BEA does not build the mix from item shares. Some movements are not credible anyway: 452 reports building rent falling 9,037M -> 5,577M, a 38% nominal fall in five years, with professional services x2.83 and communication x0.38. What stands from the original verdict is recorded rather than dropped: suppression is real (retail 1 of 9 columns with all 13 items, 3 with none; wholesale 0 of 9 and 4 with none, and 2022 is far more suppressed than 2017); the percent-of-total column is suppressed on exactly the same cells, so 0 of 471 are recoverable by arithmetic; 425000 has no AWTS coverage at all because AWTS surveys merchant wholesalers and 425 is agents and brokers. No extractor is built. Nothing continues the BES after 2022 -- AIES publishes no expense cell for 42 or 44-45 -- so a Census_BES source would carry two observations and no future. The module downloads the four workbooks directly so the finding stays reproducible, and uses the REVISED 2017 wholesale file: the original is benchmarked to the 2012 Economic Census. Caveat recorded: 2017 to 2022 is the only span the BES offers, so the correlations rest on cross-sectional spread inside one interval rather than on repeated observation. Gates: black, ruff, mypy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…no-go Wes, 2026-08-25: the answer key every score in this step used is wrong, for two reasons. BEA has not incorporated the 2022 Economic Census, so its published 2018-2024 tables are the 2017 benchmark carried forward on BEA's own annual methods -- grading a seed against them measures agreement with an extrapolation, and a seed that caught real structural change would lose. And they are scored at summary, which sums the ten farm columns into 111CA and gives several survey items the same target, so most of what a seed moves cancels before it is graded. benchmark_holdout.py is the key that has neither problem. BEA publishes three detail benchmarks, 2007/2012/2017, and io_2017 reads all of them onto the same 2017 code axis. So: seed the observed 2012 block with a source's 2012->2017 movement and score against the observed 2017 block, at detail. Out of sample, against an observation rather than an extrapolation, and nothing in the test is derived from what is being tested. Only the mix is on trial -- every column is renormalised to the observed 2017 total, because Step 3 observes the level. Economy-wide 2012->2017 mix drift is 0.0606 impact-weighted, which is the bar. Regraded on it, the agriculture seed reverses from rejected to validated: dollar frozen 0.2269 -> seeded 0.2074 +8.6% 8 of 10 columns impact frozen 0.3286 -> seeded 0.2698 +17.9% 9 of 10 columns against the 4-of-7-years swinging -42.9% to +33.5% the old key reported. No leakage: the only 2017 information entering the seed is the ERS index itself, and the renormalisation divides out because frozen and seeded are compared as shares. The prize is also much larger than the plan said. Farm columns are among the worst-drifting in the whole table at detail on N -- 1111A0 0.474, 1121A0 0.424, 1111B0 0.311 -- so "agriculture is not in this list at all" was a dollar judgement taken at summary and is wrong twice over. 111400 greenhouse and nursery is the one column the seed worsens. The +0.18 correlation that carried the rejection is withdrawn: several ERS categories map to the same summary parent, which gives different category movements identical targets and pushes the correlation toward zero by construction. agriculture_score is kept, reframed as evidence about the key. Two other verdicts are flagged rather than changed, because they rest on the same key and I have not regraded them: the trade BES retest (it has 2012 and 2017 vintages, so it CAN be regraded, and the no-go must not be acted on until it is), and the services and transportation block, which cannot be regraded here at all because Census_SAS_Expenses starts at 2013. Gates: black, ruff, mypy clean; targeted tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ason (#564) The trade rejection rested on the same wrong answer key that reversed the agriculture one, so it had to be regraded before it could be acted on. The blocker was the 2012 vintage: it exists, under names the 2017 and 2022 files do not predict -- arts/tables/2012/bes.xls (final census, not the preliminary 2012_arts_detailed_operating_expenses.xls) and the 2012r wholesale revision, which is the file benchmarked to the 2012 Economic Census. On benchmark_holdout the seed is a wash: +0.3% on dollars and -5.6% on impact, 9 and 8 of 18 columns winning. The no-go stands, but the -43.6% / -183.6% and the "+0.06, movements do not track" finding are both withdrawn -- at detail against the observed 2017 block the same comparison is +0.62 over 164 pairs. What decides it is the weighting: the BES tracks BEA on items that carry no impact (rent of machinery +0.89, water and sewer +0.81) and fails on the three that carry 60% of it (electricity -0.19 at 38% of N, building rent +0.21 at 13%, transport +0.01 at 9%). Mean correlation 0.469 unweighted, 0.281 on dollars, 0.028 on N. Shrinking the index toward 1 does not rescue it, so it is not a calibration problem either. The one conclusion that survives intact is the actionable one: do not build the suppression recovery. 441000 has 13 of 13 items in both holdout years, 75.7% impact reach, and still loses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…719) The utilities columns were held at the 2017 benchmark because the services survey had to refuse them: what the column does is a price-driven reweighting of its fuel bill, and a relative share index divides common movement out by construction. EIA measures receipts times delivered price, which is exactly a reweighting. On benchmark_holdout the seed gains +18.9% on dollars and +16.0% on impact, with all three electric columns winning on both weightings -- 221100 +17.9%, S00101 +10.0%, S00202 +13.5%. These are among the worst-drifting columns in the table (S00101 0.340, 221100 0.194 against an economy-wide 0.0606), so this is the first live route into the held block. The fuel mix is doing the work, not a general "fuel got cheaper" signal: one uniform index across the three fuels scores +7.1% against the per-fuel +16.0%. The gain survives the index raised to any power from 0.25 to 2.0, and the index is deliberately not divided by the industry's expense growth, which leaves it independent of everything observed at the target year. Three things to hold against it, all in the module docstring: coal carries the gain and coal is the leg with the least future (51.7% of the fuel bill in 2012, 16.6% in 2022); only gas matches on level, and BEA's 324110 row for the electric columns is not oil-fired generation, so the petroleum leg works through prices rather than through the concept; and it is three columns on one span. 221200 gas distribution and 221300 stay held, because BEA books gas-for-resale net and there is no cell for what EIA measures. Nothing is wired into the pipeline. The source is PUDL's public S3 mirror of EIA 923 and 861 -- no credentials, and only complete years are used, because PUDL publishes the current year month by month. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…719) The first pass left 221200 held on a concept argument alone -- BEA books gas-for-resale net, so ecnpurgas's $51.4B faces a $1M own-row cell. That was the same argument I had just overruled for the petroleum leg, which works through prices despite covering 3% of its cell, so 176 deserved measuring rather than reasoning about. Measured, it fails, and the way it fails is worth keeping. The only index that wins is citygate receipts for sales customers, +29.0% on N -- and its 45% fall across the span is EIA splitting that line in 2014, when the transportation-customers line first appears. Receipts over merchant sales volume runs 1.2-1.6 through 2013 and 0.8 in every one of the eleven years since: a step, not a trend. The merchant function itself is flat, 6,959 -> 7,237 bcf and $57.0B -> $62.0B, and every concept continuous across the break loses by 2 to 5%. Respondent counts are smooth across 2014, which rules out a universe change. Second, and independent of the break: form 176 is not a survey of the utilities sector. Its ~2,000 respondents include interstate and intrastate pipelines, storage and LNG operators and direct-delivery producers, each filing one report per state. 78-81% of the volume is pipeline movement (486000) against 17% delivered to consumers, ~30 respondents a year report producer lease use (211000), and total disposition is about 5.7x US consumption because the same molecule is counted at each step of the chain. The general lesson is the mirror of the trade regrade: there a wrong answer key made a sound source look useless, here a vintage break in the source makes a useless index look excellent. A holdout gain is only as good as the source's continuity across the span. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#497) Grading the S3 materials seed on benchmark_holdout needs a 2012 vintage of ecnmatfuel. It exists -- api.census.gov/data/2012/ecnmatfuel, under a group named on the older pattern (EC1231SM1, not EC__31MATFUEL) -- and covers 393 industries and 1,272 materials across NAICS 21 and 31-33. Three things it needed. The parse assumed columns 2012 does not publish: no label columns and no MATFUELCOST_F, so a withheld cell arrives as a bare 0 (11.9% of rows against 9.1% flagged in 2017) and Suppressed is left empty for that vintage rather than guessed at. The total codes differ -- 2012 uses 00000001 materials and 00000002 fuels where 2017 and 2022 use 00772000 and 00772002 -- and missing them leaves a $6.4T total row in as if it were a material. All four codes are disjoint across vintages, so one tuple serves. The bug: census_EC_URL_helper has a 2012 branch that appends both &for=us:* and &for=state:* so Census_EC.yaml, which deliberately leaves the geography blank, gets its national and state calls. Census_EC_MatFuel.yaml sets for: us:* itself because 2022 returns HTTP 400 without one, so the branch appended a second for clause the API ignores: two URLs, identical answers, every row twice. Shares survive that; levels and coverage do not. The branch now fires only when the config has not already set a geography, which leaves Census_EC unchanged. The grade itself is not settled and is not claimed here: 2012 publishes 1,272 material codes against 2017's 289 and only 132 are shared, so differencing the two vintages unrestricted measures a code-scheme change (-134.7% on N, index cells up to 16,000x). Restricted to the shared codes the index is sane -- median 0.723, p95 1.9 -- and scores -2.8% on N over 390 live cells, which is neither a pass nor enough coverage to be a verdict. Grading the materials seed needs a 2012 -> 2017 MATFUEL concordance first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two decisions from Wes and one measurement. The materials seed ships ungraded: the census mix is the best observation of manufacturing's structure that exists, and the alternative is not a better source but the frozen 2017 mix, which asserts nothing changed in eight years -- a stronger claim than the data makes. The gate decides where a decision hangs on it and there is an alternative to switch to; it cannot decide anything where the source is the only observation of the movement. Two checks that need no second benchmark year stand in when it cannot run: continuity inside the source, and level/coverage sanity against the BEA cell. Both caught a real error today. And the durable output of a holdout run is which source-to-Use-table correspondences it confirms, not the gain, because a confirmed correspondence replicates to 2022 and a gain number does not. The per-row tables are recorded as such: EIA gas to 211000 confirmed, coal partial at 2.1x the cell, oil not confirmed at 3% and carried as a price proxy, and the 132 shared MatFuel codes confirmed with the rest unverified rather than wrong. The measurement: services and transportation, regraded on the holdout with a 2013 base -- the earliest SAS publishes -- passes at +11.5% on N with 58 of 99 scored columns winning, against +2.0% on dollars. 531ORE, the heaviest column at $747.6B, gains +15.5%. The base is 2013 against a 2012 frozen comparator, so the seed is not credited with the 2012->2013 movement, which biases the test against it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…sal (#497) Two claims in this repo about Census_SAS_Expenses were wrong, both found by Wes pushing back on "2018/2019 are missing". The years are published. The extractor docstring said both vintages carry the sheet for 2018 and 2019 with only a handful of items populated. Neither half is true: sas-17 carries 2013-2017 and sas-22 carries 2020-2022, so those years are in neither, and they live in sas-18.xlsx and sas-19.xlsx, which this source does not fetch. There they are fully populated -- 8 of 8 cells for every NAICS -- on a cut item list, 8-12 items per NAICS against 24-28 in sas-17. What Census cut for those two collection years is exactly the seed's input list: electricity, communication, fuels, professional and technical services, advertising, repairs, water and sewer, rent of buildings and machinery, data processing. The total Expenses line survives and is a denominator rather than a mix, usable for industry_growth and not fetched today. So the mix gap is real and the stated reason for it was not. The 2023 refusal rested half on a -5.0%/-4.1% score against BEA's carry-forward summary, which is the key this branch discredited this morning. AIES does continue the collection and AIES_TO_SAS_ITEM aligns the names, so the correspondence is not what fails. What fails is the dispersion the index reads: a relative index divides out whatever is common to the industry, so only each item's deviation from its industry's own step survives. That residual is 0.035 in a quiet year, 0.043-0.046 across the energy surge, and 0.164 across the survey change -- an eventful year is worth about +0.01 and the instrument change about +0.12. The refusal stands on evidence that does not depend on the answer key, and the score figure is withdrawn. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wes: 2023 data reset from the 2022 benchmark, so do not discard it for jumping. That is right, and the refusal I re-grounded an hour ago used the wrong control. The seed's index divides out whatever is common to an industry, so only each item's deviation from its industry's own step reaches it. I compared that residual against within-vintage pairs -- 0.035 quiet, 0.043 across the energy surge -- and read 0.164 as instrument noise. The right control is a pair that crosses a benchmark in the same instrument: SAS 2017 (2012 EC) to SAS 2020 (2017 EC) scores 0.146. So 0.164 is what crossing a benchmark looks like, not what a broken instrument looks like; the survey change itself adds about 0.018. A four-year within-vintage span scores 0.108, so span length does not explain a one-year step either. And the rebenchmark is information rather than contamination. AIES replaced SAS outright at data year 2023, and a survey launched off the 2022 Economic Census resets levels and composition toward what that census revealed -- which is the observation BEA has not incorporated. Refusing 2023 holds the mix on a 2017-benchmarked structure, which is the frozen-benchmark problem this step exists to fix. Census documentation has not been read to confirm the 2022-EC benchmark explicitly; the timing and the dispersion signature are consistent. services_transport_seed now builds AIES years by default. What is genuinely lost is the ~0.018 that is instrument rather than economics, and coverage: AIES carries 466 item-industry pairs against 974 in the last SAS year. A 2023 movement is weaker evidence than a 2020-2022 one, not unusable evidence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…the CI split # Conflicts: # bedrock/analysis/nowcasting/README.md # bedrock/analysis/nowcasting/inputs_structure.py # bedrock/analysis/nowcasting/intermediate_estimation_plan.md # bedrock/analysis/nowcasting/intermediate_structure_drift.py # bedrock/analysis/nowcasting/progress_report.md # bedrock/analysis/nowcasting/sections.py # bedrock/extract/census/Census_AIES.py # bedrock/extract/census/Census_AIES_Expenses.yaml # bedrock/extract/census/Census_EC.py # bedrock/extract/census/Census_EC_MatFuel.yaml # bedrock/extract/census/Census_SAS_Expenses.py # bedrock/transform/eeio/nowcast.py # bedrock/transform/iot/__tests__/test_nowcast_intermediate.py # bedrock/transform/iot/nowcast_intermediate.py
T1 pins the Use column's sum; nothing pinned the split. T4 is soft and aggregated to summary groups, T6 is two economy-wide scalars, and T5 is unimposed by design -- so every dollar of value-added estimation error landed in the intermediate column total, which is the scale of a column of the technology matrix and multiplies through L into every downstream result. T18 pins VAPRO from UVA205-A, the sibling of the UGO305-A already behind T1, allocated to 402 detail. With both, T005 is determined per industry and the slack moves inside value added, where V00300 is free -- gross operating surplus, which BEA largely computes as a residual and which appears in no A, no L and no emission factor. Measured, the error it takes off the intermediate block is 2.75% of T005 in 2018 rising to 9.24% ($1.98tn) in 2024, worst industry GSLGE at 23.5% of its own T005. 2017 reads 31, which is what says the measurement is not scoring its own anchor. T1 and T18 cannot disagree: T1 reads the extracted UGO305-A parquet and T18's series is derived against load_go_detail(), and the two gross output vectors are identical to the dollar for every industry in every year 2017-2024. A column margin restricted to five rows partitions the column, and gras_balance takes one column vector -- so T18 is imposed as a closer after each Use pass, the way T4 is: +d on V00300 and -d across that column's free sign-flexible commodity cells, leaving the column sum and therefore T1 untouched. All of it on V00300 deliberately; a proportional scale would move V00100 and undo the T4 closer that just ran. On the 2017 replay T18 closes to 0.005 and T1 holds at 0.15. T5's absence is now load-bearing rather than tolerated: imposing it would push the slack back into T005 and undo this. Said so where someone would go to add it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 461563e)
…or calls The reconciliation took the Step 3 stack's Census_AIES.py wholesale, which dropped four functions nowcast's newer work depends on - census_aies_miscsector_call and its filename/load/parse siblings - and broke 27 tests in test_nowcast_va_taxes with AttributeError. The two sides were not alternatives. nowcast has four functions the stack lacks; the stack has none nowcast lacks, and contributes content rather than code: the corrected finding that timeseries/aies/exp02 publishes all 41 expense variables for 13 service sectors in 2023, and the seven sector-specific entries in AIES_EXPENSE_FLOWS that go with it. Taking nowcast's file as the base and applying the stack's two additions keeps both. The superseded claim that sectors 21, 22, 23 and 51-81 "publish nothing at all" is gone, which was the point of preferring the stack there. test_nowcast_va_taxes: 70 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Collapses the #713 → #714 → #715 → #717 → #718 stack into one PR, reconciled onto
nowcastnow that Step 2 (#740) and Step 4a have landed.39 files / +9,179 — not the 308 the raw branch diff suggests, because most of the stack was already in
nowcastvia the #710 and #712 squashes.Why one PR
Per Wes: the five PRs do not need separate review. Collapsing also dissolves #717's
CONFLICTINGstate for free — it conflicted against its own parent, andstep3_service_seedalready contains both sides of that.How the 14 conflicts were resolved
9 were not real. For each,
nowcast's version of the file is byte-identical to a commit inside this stack — #710 and #712 are squashes of the stack's own early commits, so git could not see the shared history:inputs_structure.pyaf22c987nowcast_intermediate.py2168d6c9intermediate_structure_drift.py2168d6c9intermediate_estimation_plan.md4e3ffad5test_nowcast_intermediate.py4e3ffad5Census_EC.py2e9c6eb3Census_EC_MatFuel.yamled1a96fbCensus_AIES_Expenses.yamlaf22c987Census_SAS_Expenses.pyece48d30The stack continues past each, so it wins with nothing lost.
5 were real, and split by whose work each hunk is rather than by branch:
sections.py—nowcastfor the section registry and docstring table (five sections now, including Step 4a'sSUPPLY_OUTPUT_DETAIL_SUT, and Step 2's live candidate); the stack for θ as a fitted regime rule and the purchaser-deflator carry.nowcast.py—nowcastfor the trade extension toTRADE_OVERLAY_YEARS; the stack for θ (DEFAULT_THETAis gone; θ is per-span, 0.75 off the 2021-22 surge and 0.0 across it, not Step 3: Prepare initial intermediate estimates using commodity inflation #497's 1.0).Census_AIES.py— the stack, which correctsnowcast.nowcaststates sectors 21/22/23 and 51-81 "publish nothing at all"; the stack showstimeseries/aies/exp02publishes all 41 expense variables for 13 service sectors in 2023. Primary data that had been wrongly written off.README.md/progress_report.md— newer run per block.Two things the merge surfaced
A duplicate definition.
use_sut_intermediate_referenceended up defined twice insections.py— both sides added it, at different positions. The copies are byte-identical; removed the second.A real integration break. Step 2's
compensation_movement_holdout.pyimportsBENCHMARK_SUT_ARCHIVEfromintermediate_structure_drift, but Step 3 moved that constant to its canonical home,matrix_mappings.USA_BENCHMARK_DETAIL_SUT_ARCHIVE(same value). Repointed the import — this is the genuine seam between the two lines of work, and neither branch could have caught it alone.461563e0replayed"Pin VAPRO per industry as T18, so income-side error stops reaching A" was deliberately left out of #740 and cherry-picked here instead. It touches
nowcast_intermediate.py,nowcast_sut_gras.pyandnowcast_targets.py, and it applied with no conflicts — it was written against this stack's version of those files, which is why it did not belong in the Step 2 PR.That commit existed only on
origin/step2_va_timeseries. It is now here, so #735 can be closed.Gates
black,ruff,mypyclean (bar the 4 Windows-onlysettings.pyerrors that do not occur on CI);test-fast136 passed.