Skip to content

Add HMRC's 2026 ISA, Lifetime ISA and impact-note facts (PolicyEngine/microcosm#1415) - #392

Merged
juaristi22 merged 7 commits into
mainfrom
hmrc-isa-statistics-2026
Oct 29, 2026
Merged

juaristi22 merged 7 commits into
mainfrom
hmrc-isa-statistics-2026

Conversation

@juaristi22

@juaristi22 juaristi22 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

PolicyEngine/microcosm#1415 imputes yearly ISA subscriptions per adult by type, so that ISA-limit reforms and the Lifetime ISA bonus stop scoring zero on Microcosm. This adds the published HMRC facts it calibrates and checks against: three packages and 3,272 facts.

Package Facts Source
hmrc-isa-statistics-2026 3,230 ISA tables, September 2026
hmrc-lisa-statistics-2026 40 Lifetime ISA tables, September 2026
hmrc-cash-isa-limit-reduction-2026 2 Cash ISA limit reduction impact note, 17 September 2026

packages/hmrc/isa_statistics_2026/build_package.py writes all three from the pinned bytes. It matches every selector by the publisher's text and asserts each match, and --check compares instead of writing.

What each package carries:

  • ISA tables:
    • Table 9.4: accounts and amounts subscribed by component, 2008-09 to 2024-25 (2024-25 provisional).
    • Table 9.6: market values by type of investment, 2011-12 to 2024-25.
    • Tables 9.7 to 9.12: 2023-24 individuals subscribing and holding, by income, age, sex, region, subscription band and market value. Table 9.12's share of adults is its own model_output record set, with an age >= 18 constraint (HMRC's denominator).
  • Lifetime ISA tables: Table 1a, yearly withdrawals for house purchases and unauthorised withdrawals, 2018-19 to 2025-26. The workbook has no subscriptions, holdings or bonus. Lifetime ISA subscriptions are Table 9.4's Lifetime ISA rows.
  • Impact note: the shares of cash ISA subscribers aged under 65 either side of £12,000 in 2022-23 (78% and 22%). They are model_output, because the note names no source. Each row carries its band alone, as the sentence doesn't place exactly £12,000. The page's other "22%" is the charge on interest in non-cash ISAs, and isn't ported.

Caveats (kept in the facts' notes):

  • Tables 9.7 to 9.12 head their columns "(thousands)" but count individuals. Only that reading matches HMRC's commentary, so they carry value_scale: 1, with the heading kept as the header guard.
  • Table 9.4 counts accounts, not individuals, so its role is isa_account.
  • Tables 9.7 to 9.12 exclude Innovative Finance ISAs, and don't say how Lifetime ISAs are classed. Table 9.6 says Lifetime ISA market values are inside the cash and stocks-and-shares totals. Help to Buy ISAs are inside cash (HMRC's methodology, section 3.4).
  • Totals HMRC prints in more than one of Tables 9.7 to 9.12 are ported once, each copy asserted equal, as GSGB 2025 gambling participation by sex, age, area and activity, for PolicyEngine/microcosm#1381 #391 did for GSGB's restated row. That leaves out 167 copies.
  • Not ported: Table 9.4's average subscriptions and Table 1a's averages (quotients of ported cells), the Junior ISA rows, and Table 1b's monthly withdrawals.
  • Chronicle reconciles none of these tables with each other.

On the docs page only, as they come from arithmetic on the cells or the ISA rules, not HMRC's text:

  • Table 9.7's bands can't be reconciled with Table 9.4's amounts. At band floors the "all" block is at least £114.3bn, against Table 9.4's £100.6bn of cash and stocks and shares.
  • Lifetime-ISA-only subscribers are outside Table 9.7. Cash-only plus both types matches Table 9.4's cash accounts, and stocks-and-shares-only plus both its stocks and shares accounts.
  • From 6 April 2024 (SI 2024/350) savers can subscribe to more than one ISA of the same type, so Table 9.4's 2024-25 accounts aren't people.

Role and paths

ledger-source-ingestor. Paths outside the role's list, as #391 declared: db/data/** (the three files and manifests), chronicle/source_package.py (aliases), chronicle/bundle.py (UK suite), docs/** and changelog.d/**.

Checks

  • validate-package on all three packages: valid, no warnings.
  • build-suite: agent acceptance passes, with full cell lineage. The only warning is concept_alignment_validation_skipped.
  • The consumer contract is valid.
  • publish-raw: all three files are in ledger-raw.
  • A bundle restricted to the three packages is valid: 3,272 facts, no semantic duplicate key, no warning and no errors. The merged-bundle counts in test_chronicle_bundle.py are moved by those measured numbers, plus HMRC's overseas-and-unknown scope as one new geography key. CI runs the full build.
  • tests/test_source_hmrc_isa_statistics_2026.py: 19 passed. It covers:
    • registration, manifests, licence and R2;
    • the generator rebuilding each YAML byte for byte;
    • the values of Table 9.7, 9.4 and the Lifetime ISA table, re-read with pandas by label;
    • how the totals add up: blocks, holders, sexes, regions and components;
    • each restated copy, found in the workbook by label, against the fact ported;
    • the notes, roles and universes, and the arithmetic the docs state.
  • tests/test_chronicle_bundle.py without the merged build: 26 passed.
  • ruff check: passed.

Judges: ledger-source-fidelity and ledger-boundary.

Part of PolicyEngine/microcosm#1415. The Microcosm side vendors these facts after the feed re-pin.

🤖 Generated with Claude Code

juaristi22 and others added 3 commits October 28, 2026 20:18
For PolicyEngine/microcosm#1415, which imputes yearly ISA subscriptions per
adult by type. Three packages, written by
packages/hmrc/isa_statistics_2026/build_package.py from pinned bytes:

- hmrc-isa-statistics-2026 (3,397 facts): Table 9.4's accounts and amounts
  subscribed by component, 2008-09 to 2024-25; Table 9.6's market values by
  type of investment, 2011-12 to 2024-25; and Tables 9.7 to 9.12's 2023-24
  individuals subscribing and holding, by income, age, sex, region,
  subscription band and market value. Tables 9.7 to 9.12 head their columns
  "(thousands)" but count individuals, so they carry value_scale 1.
- hmrc-lisa-statistics-2026 (40 facts): Lifetime ISA Table 1a, yearly
  withdrawals. The workbook has no subscriptions or bonus.
- hmrc-cash-isa-limit-reduction-2026 (2 facts): the impact note's shares of
  under-65 cash ISA subscribers either side of GBP 12,000 in 2022-23.

The files are committed under the Open Government Licence and uploaded to R2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a data-source page with the files, concepts, dimensions, caveats and
what Microcosm selects, a "Beyond parity" section in the UK source
checklist, and a changelog fragment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The three packages add 3,439 person facts, 3 packages and source tables,
HMRC's overseas and unknown statistical scope, and 145 semantic duplicate
keys where the ISA tables repeat the same totals. The tax years already
exist. Measured on a bundle restricted to the three packages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@PavelMakarchuk PavelMakarchuk left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of #392 at 84682e9. I checked the pinned bytes against fresh GOV.UK downloads, every ported cell against my own read of the workbooks, the caveats against HMRC's commentary and methodology, the 145 semantic duplicates, build_package.py, the registration and bundle-count changes, and fitness for PolicyEngine/microcosm#1415 (with Vahid's review there). I also searched Chronicle's issues and PRs for other ISA or savings packages: none overlap. #205 (WAS round 8 wealth) is open and only touches ISA balances. #377 (consumer_fact.v5 labels) is open and changes chronicle/bundle.py, chronicle/source_package.py and tests/test_chronicle_bundle.py, so whichever merges second must re-pin.

Must fix

  1. The merged bundle test still pins five pre-PR counts. The PR moves the summary pins (fact_count, geography_count, semantic_duplicate_key_count, sources, tables, periods) but leaves these asserts in the same test (test_build_bundle_writes_merged_consumer_contract). It skips locally without the R2-only artifacts, so only CI's bundle job will catch them. It was still pending when I looked. From the 3,439 built facts:
    • L2248 country:K02000001 == 33507: the PR adds 3,146 UK facts, so 36,653.
    • L2252 country:E92000001 == 6808: Table 9.9/9.12's England rows add 21, so 6,829.
    • L2262 len(by_geography) == 102616: 102,617, as the summary pin at L478 already says.
    • L2300 "person": 593323: all 3,439 facts are person, so 596,762.
    • L2306 len(semantic_fact_keys) == 2324: 2,469, as L505 already says.
    • Fix: update the five pins, each with a one-line comment as for the others. Then confirm the CI bundle job passes.

Should fix

  1. The restated totals are kept as 145 semantic duplicates. #391 emitted the same kind of restated row once. I built a bundle of the three packages. There are 145 duplicate keys: 126 pairs, 18 triples and one group of six (23,025,972 holders, in 9.7 to 9.12). All copies agree (0 disagree). By tables: 9.8 and 9.11 (65), 9.7 and 9.10 (47), 9.9 and 9.12 (14), 9.10, 9.11 and 9.12 (13), 9.7, 9.8 and 9.9 (5), all six (1). That is safe for a consumer that picks one source table. But it grows the bundle's duplicate count by 6% (2,324 to 2,469), which the repo has been retiring. In #391, B.9/B.10's "Any gambling" row restated B.3/B.4. It was asserted equal and emitted once. Fix: emit each restated cell once, from one table chosen per family, and assert the other copies equal in build_package.py. If you keep them, say in the docs why this differs from #391.

  2. The docs leave out the two traps Microcosm reviewers already found.

    • Table 9.7's bands can't be reconciled with Table 9.4's amounts. At band floors (£1, £2,001, … £18,001, £20,000), the "all" block comes to at least £114.3bn. Cash-only alone comes to £78.1bn. Table 9.4 has £69,498m cash plus £31,070m stocks and shares (£100.6bn) for 2023-24. The docs (L86) say only that the two are "not reconciled here". Give the floor figures, so no consumer binds both.
    • L133 points Microcosm at table9_4.accounts.ty2024 for "growth". From 6 April 2024, savers can pay into more than one ISA of the same type in a year. So 2024-25 accounts (cash 10,713k, stocks and shares 4,893k, provisional) are no longer a proxy for people. Say this in the "For Microcosm" section. It is not HMRC's text, so it belongs in the docs, not in fact notes.
  3. "Help to Buy ISAs are inside cash" is not in any fact note. The PR body says the caveats are "kept in the facts' notes". The docs (L85) also say so. I checked the notes of all 3,439 facts. The only Help to Buy text is Table 9.4's copied note about transfers into a Lifetime ISA up to 5 April 2018. The source is the methodology page, §3.4: "Help to Buy: ISAs are included within these statistics under cash subscriptions and market values". Fix: add that sentence, cited to the methodology page, to the cash notes of Tables 9.4, 9.6 and 9.7 to 9.12.

Nice to have

  1. The "(thousands)" note on the holder tables cites the subscriber figure. THOUSANDS_HEADING says "13.05 million subscribers". It is reused for Tables 9.10 to 9.12 and for the holder columns of 9.7 to 9.9. For those, the matching commentary figure is "23 million" holders (§1.4). The holder measures of 9.7 to 9.9 also open with "individuals subscribing to Adult ISAs". Use a holder variant.
  2. Lifetime ISA classification can be stated from the published cells. In 9.7, cash-only plus both types is 8,963,286 + 975,669 = 9,938,955. Table 9.4's cash accounts for 2023-24 are 9,939 thousand. Stocks-and-shares-only plus both is 3,115,606 + 975,669 = 4,091,275, against 4,091 thousand. Table 9.4's Lifetime ISA row (964 thousand) is separate. And 13,054,561 is exactly the sum of the three type blocks. The methodology (§5.2) says 9.7 to 9.9 are calibrated to 9.4. Lifetime-ISA-only subscribers are not in 9.7's subscribers. A docs line with this arithmetic is more useful than "HMRC does not say".
  3. The impact note's amount constraints leave exactly £12,000 in neither row. The constraints are cash_isa_subscription_amount < 12000 and > 12000. The shares sum to 100, and the package's own header says the sentence "does not place a subscription of exactly GBP 12,000". Keep the band filter only, or leave the boundary out of one side.
  4. Table 9.4's account counts carry entity_role: isa_subscriber. The concept (hmrc.isa_accounts_subscribed) separates them. But a role such as isa_account would stop a role-based selector from treating 15.0m accounts as people.
  5. Table 9.12's share of adults has no universe constraint. Its denominator is adults aged 18 or over (9.12 A6). The note says so, but the record set has no age >= 18 constraint. #391 added one for its adult universe.

Verified as correct

  • Bytes. The two .ods files I downloaded from the release page are byte-identical to the pins (a0eaf908…, bba1520b…). The impact-note HTML has the same size (103,157 bytes) but a different hash. The diff shows only per-request tokens (csrf-token, csp-nonce, search and feedback form ids). The text is identical. The release page (published 16 September 2026) still links the same two assets, so there is no newer revision.

  • Values. For every ported region, the numeric cells and the facts match one to one. Each table has no missing or extra values: 9.4 170, 9.6 249, 9.7 672, 9.8 396, 9.9 90, 9.10 672, 9.11 924, 9.12 224. I also found 46 cells by label and checked value, unit, scale, tax year and geography. They include 9.4's provisional 2024-25 cash, S&S, Lifetime and Total rows, the 9.7 grand total and £20,000 column, 9.8 age and sex rows, 9.9 Wales/NI/England/overseas, 9.10 to 9.12 bands and averages, 9.12 shares (0.375, 0.417, 0.316) and four Lifetime ISA Table 1a cells (99,750 in 2025-26). All match.

  • value_scale: 1 on 9.7 to 9.12 is right. The cells give 13,054,561 subscribers and 23,025,972 holders. The commentary says the number of holders "has risen to 23 million", and that 28.8% of subscribers saved at the maximum (3,760,433 / 13,054,561 = 0.2881). Reading the cells as thousands would give 23 billion holders.

  • Table 9.4. Counts are ×1,000 and amounts ×1,000,000, per 9.4 A5. The Total equals cash + S&S + IF + Lifetime (2023-24: 15,007 against 15,008 for accounts, 102,995 exactly for amounts). The insurance component is outside it, as the note says. The methodology (§5.1) confirms that 9.4 counts accounts, not individuals. 2024-25 is flagged in period_coverage.notes, as in the CGT packages.

  • Table 9.8's "Cash ISA" column is cash only: its total, 8,963,286, equals 9.7's cash-only subscribers. HMRC's pseudo-codes W99999999, S99999999 and N99999999 map to W92000004, S92000003 and N92000002, and the notes record this.

  • Impact note. The sentence is verbatim on the page. 2022-23 tax year, under-65 constraint and model_output are right. The note names no source, and hmrc-salary-sacrifice-reform-2029-headcounts sets the same precedent. Leaving out the page's other "22%" (the charge rate) is right.

  • Provenance. survey_aggregate with the SPI instrument for 9.7 to 9.12 matches methodology §5.2 and §6.3. Help to Buy inside cash (§3.4) and Innovative Finance excluded (each table's notes) are both correct.

  • build_package.py. The sha256 of each pinned file is asserted. Every row and column header is checked with expect()/only_row(). Non-numeric 9.4/9.6 cells must be [not available]. I found no silent value fallback; the only .get default is a geography-name lookup. --check exits 0, and the generator test rebuilds all three YAML files byte for byte.

  • Registration and moved counts. The aliases and UK_BUNDLE_SOURCES are added. The moved pins agree with my build: facts +3,439 (hmrc 33,241 to 36,680), packages +3, source tables +3, one new geography (statistical_scope:hmrc:overseas_and_unknown), and the per-tax-year additions (2008: 8 … 2023: 3,014, 2024: 36, 2025: 5).

  • Boundary. No value is computed, aged or reconciled. Averages and Table 1b are left out, and the docs leave the choice between 9.7 and 9.4 to the consumer.

  • Fitness for microcosm#1415. The targets the issue and Vahid's review name are present:

    • 9.7 (income × band × type), 9.8 (age × sex × type) and 9.9 (region);
    • 9.4 accounts and amounts by component, including the Lifetime ISA rows Vahid identified as the only Lifetime ISA subscription source;
    • 9.6 stocks-and-shares "Cash on deposit" (£22,398m at 5 April 2025);
    • Lifetime ISA Table 1a withdrawals;
    • the 22% check.

    Bonus paid is in neither file, as the docs say.

Tests run locally

At 84682e9, with Python 3.14 and uv sync:

  • uv run pytest tests/test_source_hmrc_isa_statistics_2026.py: 16 passed.
  • uv run pytest tests/test_chronicle_bundle.py: 26 passed, 1 skipped. The skip is test_build_bundle_writes_merged_consumer_contract, because R2-only artifacts are not cached. So finding 1 is from reading the test, not from running it.
  • uv run chronicle validate-package on hmrc-isa-statistics-2026, hmrc-lisa-statistics-2026 and hmrc-cash-isa-limit-reduction-2026 (--year 2026): all valid, no warnings, 3,397, 40 and 2 records.
  • uv run python packages/hmrc/isa_statistics_2026/build_package.py --check: exit 0.
  • uv run chronicle build-bundle --year 2026 with the three --sources: valid, 3,439 facts, 145 semantic duplicate keys, 0 aggregate duplicates.
  • tests/test_chronicle_geography_names.py, test_issue_266_geography_names.py, test_source_package_alias_drift.py, test_chronicle_artifacts.py, test_chronicle_governance.py: 104 passed.
  • uv run ruff check .: passed. (ruff format --check would reformat three of the PR's files, but CI does not run it, and 8 other files on main fail it too.)
  • I did not run the full UK bundle (about 53 minutes and 15 GB).

🤖 Generated with Claude Code

Comment thread tests/test_chronicle_bundle.py Outdated
# PolicyEngine/microcosm#1415 adds 145: the ISA tables publish the same
# subscriber and holder totals in more than one table (9.7 to 9.12), and
# the copies agree.
"semantic_duplicate_key_count": 2469,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pin moved to 2,469, but the same metric is still asserted as 2,324 at L2306 (len(coverage["duplicates"]["semantic_fact_keys"])). Four more pins in this test have not moved:

  • L2248 country:K02000001: 33,507 should be 36,653 (+3,146 UK facts);
  • L2252 country:E92000001: 6,808 should be 6,829 (+21);
  • L2262 len(by_geography): 102,616 should be 102,617, as L478 says;
  • L2300 person: 593,323 should be 596,762 (+3,439).

I took these from a build of the three packages. The test skips locally without the R2-only artifacts, so only CI's bundle job will catch them.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI confirms this. The bundle job fails at country:K02000001: assert 36653 == 33507 (job log). The test stops at the first failed assert, so the other four counts in this comment (6,829, 102,617, 596,762 and 2,469) still need the update too.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e56af9a. I rebuilt the three packages at this head: 3,272 facts, UK +2,993, England +20, 15 geography keys (one new), person +3,272 and no semantic duplicate keys. These match the new pins (36,500, 6,828, 102,617, 596,595 and 2,324). CI's bundle job has the last word.

not say how Lifetime ISAs are classed. Help to Buy ISAs are inside cash.
- **Amounts.** Table 9.7 has no amounts, and its bands are not reconciled here with Table
9.4's amounts subscribed.
- **Totals in more than one table.** The same subscriber and holder totals appear in

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirmed the 145 keys: 126 pairs, 18 triples and one group of six. All copies agree. #391 handled restated rows another way. B.9/B.10's "Any gambling" row was asserted equal to B.3/B.4 and emitted once. Could build_package.py do the same here? Emit each restated total from one table per family, assert that the other copies are equal, and keep the bundle's duplicate count at 2,324. If you keep the copies, please say here why this case differs from #391.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 854475c and 9e6fc32. port_restated_cells_once asserts each copy equal to the ported cell and then drops it, as #391 did. 3,439 - 3,272 = 167 dropped facts, which is the 126 + 2 x 18 + 5 extra copies that I counted.

measures carry `value_scale: 1`. The heading text stays as each measure's header guard.
- **Coverage.** Tables 9.7 to 9.12 exclude Innovative Finance ISA holders. Their notes do
not say how Lifetime ISAs are classed. Help to Buy ISAs are inside cash.
- **Amounts.** Table 9.7 has no amounts, and its bands are not reconciled here with Table

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please give the size of the gap. At band floors (£1, £2,001, … £18,001, £20,000), 9.7's "all" block is at least £114.3bn, and cash-only alone is £78.1bn. Table 9.4 has £69,498m cash and £31,070m stocks and shares in 2023-24. So the two cannot both bind (Vahid's finding 1 on microcosm#1415).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 854475c. The page now gives £114.3bn and £78.1bn against £100.6bn.

match the pin. Then select:
- **Subscriber cells:** record sets `hmrc.isa_statistics_2026.table9_7.ty2023.*`,
`table9_8.ty2023.*` and `table9_9.ty2023.*`.
- **Growth and composition:** `table9_4.{accounts,amounts}.ty2023` and `.ty2024`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From 6 April 2024, savers can pay into more than one ISA of the same type in a year. So table9_4.accounts.ty2024 (cash 10,713k, stocks and shares 4,893k, provisional) is no longer a proxy for people. Please add this caveat here, so that account growth is not read as growth in people. It is not HMRC's text, so the docs are the right place for it, not the fact notes.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 854475c. "For Microcosm" now says this and cites SI 2024/350.

commentary (13.05 million subscribers and 23.0 million holders in 2023-24), so those
measures carry `value_scale: 1`. The heading text stays as each measure's header guard.
- **Coverage.** Tables 9.7 to 9.12 exclude Innovative Finance ISA holders. Their notes do
not say how Lifetime ISAs are classed. Help to Buy ISAs are inside cash.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The caveats are said to be kept in the facts' notes, but no fact note says that Help to Buy ISAs are inside cash. I checked all 3,439. The source is the methodology page, §3.4: "Help to Buy: ISAs are included within these statistics under cash subscriptions and market values". Please add it, with that citation, to the cash notes of 9.4, 9.6 and 9.7 to 9.12.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9e6fc32. 85 notes now quote methodology section 3.4.

)
THOUSANDS_HEADING = (
"The column headings say '(thousands)', but the cells are counts of individuals: "
"only that reading matches HMRC's commentary (13.05 million subscribers in "

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this note is also used for the holder tables (9.10 to 9.12, via holder_notes) and the holder columns of 9.7 to 9.9. For those, the matching commentary figure is the 23 million holders (§1.4), not 13.05 million subscribers. The 9.7 to 9.9 holder measures also start with "individuals subscribing". Use a holder variant.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9e6fc32. Each note cites the commentary figure for its own measure, and the 13.05 million is gone.

return (
f"Table {table}, individuals subscribing to Adult ISAs in the 2023 to 2024 tax "
"year (6 April 2023 to 5 April 2024). " + THOUSANDS_HEADING + " "
+ TABLES_9_7_TO_9_12 + " HMRC does not say how Lifetime ISAs are classed in "

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: the published cells answer this. Cash-only plus both is 9,938,955, and 9.4's cash accounts are 9,939k. Stocks-and-shares-only plus both is 4,091,275, against 4,091k. The three type blocks sum exactly to 13,054,561. Methodology §5.2 says 9.7 to 9.9 are calibrated to 9.4, so Lifetime ISA subscriptions (964k accounts, their own 9.4 row) are outside these blocks. One docs line with this arithmetic would help the Microcosm side.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 854475c. The page now states the arithmetic, and a test pins it.

assert rows[row][5] == int(number)
filters, constraints = category("cash_isa_subscription_band", value_id, "Cash ISA subscription band")
constraints.append(
constraint("cash_isa_subscription_amount", operator, 12000, "Cash ISA subscription relative to GBP 12,000", "gbp")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With < 12000 on one row and > 12000 on the other, a subscription of exactly £12,000 is in neither row. But the shares sum to 100, and the header says the sentence does not place that case. Keep only the band filter, or leave this amount constraint out, so the constraints say no more than the sentence.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9e6fc32. Each row carries its band only.

period=opening,
coverage=coverage_tax_year(opening, str(year_label).strip(), provisional),
provenance="administrative",
entity_role="isa_subscriber",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Table 9.4 counts accounts, but the role here is isa_subscriber, the same role as 9.7's individuals. The concept separates them. Still, a role such as isa_account would stop a role-based selector from mixing 15.0m accounts with 13.05m people.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9e6fc32. Table 9.4's rows use isa_account.

shared=category("isa_type", "all", "ISA type"),
),
record_set(
record_set_id=f"{ISA_PREFIX}.table9_12.ty2023.share_of_adults",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the denominator is adults aged 18 or over (9.12 A6). That is in the note, but not in the constraints. An age >= 18 universe constraint here would match how #391 stated its adult universe.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9e6fc32. The record set has an age >= 18 constraint, and the note says what HMRC leaves open about holders aged 16 and 17.

juaristi22 and others added 4 commits October 28, 2026 23:16
Tables 9.7 to 9.12 print some totals in more than one table, which gave the
bundle 145 semantic duplicate keys. As chronicle#391 did for GSGB's restated
row, each is now ported once, and build_package.py asserts every copy equal to
the cell it keeps before leaving the copy out:

- Tables 9.7 to 9.9's 'All ISA Holders' columns restate Tables 9.10 to 9.12's
  'Total' columns row for row, so holder totals come from the holder tables;
- the all-sexes, all-ages rows of Tables 9.8 and 9.11 and the United Kingdom
  rows of Tables 9.9 and 9.12 restate the 'All ranges' rows of Tables 9.7 and
  9.10, so those totals come from Tables 9.7 and 9.10.

The ISA package goes from 3,397 to 3,230 facts (167 copies left out), and a
bundle of the three packages has no semantic duplicate key. The tests find
each copy in the workbook by label and check it equals the fact ported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the review of #392:

- Help to Buy ISAs are inside cash. The notes of Tables 9.4 and 9.6 (from
  2015-16, when the scheme opened) and of Tables 9.7 to 9.12 quote section 3.4
  of HMRC's background and methodology note.
- Each '(thousands)' note cites what the commentary gives for its measure: 23
  million holders (section 1.4) for holders, and 15 million accounts and 23
  million holders (sections 1.1 and 1.4) for subscribers. The commentary gives
  no count of subscribers, so the '13.05 million' is gone. Tables 9.7 to 9.9's
  no-further-subscription measures now read as holders, and the averages,
  which are in GBP, no longer carry the note.
- Table 9.4 counts accounts, so its role is isa_account, not isa_subscriber.
- Table 9.12's share of adults carries an age >= 18 constraint, HMRC's
  denominator. Its note says HMRC does not say whether holders aged 16 or 17
  are in the numerator.
- The impact note's rows carry their band alone. With '<' and '>' GBP 12,000
  bounds, a subscription of exactly GBP 12,000 fell in neither row, though the
  shares sum to 100.

Values and fact counts are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The data-source page now says what the review of #392 asked for:

- The size of the gap between Table 9.7's bands and Table 9.4's amounts. At
  band floors the 'all' block is at least GBP 114.3bn in 2023-24, and the
  cash-only block alone GBP 78.1bn, against Table 9.4's GBP 100.6bn of cash
  and stocks and shares subscriptions, so a consumer cannot bind both.
- From 6 April 2024 (SI 2024/350) a saver can subscribe to more than one ISA
  of the same type in a year, so Table 9.4's 2024-25 accounts are no longer a
  proxy for people. This is under "For Microcosm", not in fact notes, as it is
  not HMRC's text.
- How Lifetime ISAs sit in Tables 9.7 to 9.9, from the cells: cash-only plus
  both types matches Table 9.4's cash accounts, and stocks-and-shares-only
  plus both its stocks and shares accounts, so Lifetime-ISA-only subscribers
  are outside Table 9.7.
- Help to Buy ISAs inside cash, cited to the methodology (section 3.4); the
  commentary figures the '(thousands)' reading rests on; the totals ported
  once; the account role; Table 9.12's adult universe; and the impact note's
  band-only rows.

The counts move to 3,230 and 3,272 facts. A test pins the arithmetic the page
states.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI's bundle job stopped at the United Kingdom count, which still had its
pre-PR value, and four more pins in the same test had not moved. With the
restated totals now ported once, the three packages add 3,272 person facts:

- fact_count and row counts 1,657,279, and hmrc 36,513;
- tax_year 2023 +2,847 instead of +3,014;
- country:K02000001 36,500 (+2,993) and country:E92000001 6,828 (+20);
- 102,617 geography keys (the overseas and unknown scope);
- person 596,595;
- semantic duplicate keys back to 2,324, matching the assert at the end.

Measured on a bundle of the three packages: 3,272 facts, no semantic
duplicate key and no warning. CI's run on 84682e9 passed every pin before
the United Kingdom count, at 3,439 facts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@PavelMakarchuk PavelMakarchuk left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review of #392 at e56af9a (three commits since 84682e9: 9e6fc32, 854475c and e56af9a).

Must fix

None open. Must fix 1 (the five merged-bundle pins) is fixed in e56af9a. CI's bundle job at 84682e9 failed at the first of them (assert 36653 == 33507), as the review said.

Should fix

None open.

  • 2 (restated totals): fixed. port_restated_cells_once asserts every copy equal to the cell it keeps, then drops the copy, as #391 did. The three packages now have no semantic duplicate key.
  • 3 (docs traps): fixed. The page gives the £114.3bn and £78.1bn floors against Table 9.4's £100.6bn, and says that SI 2024/350 makes 2024-25 accounts no proxy for people.
  • 4 (Help to Buy): fixed. The notes quote methodology section 3.4.

Nice to have

All five (5 to 9) are fixed in 9e6fc32 and 854475c.

Verified as correct

  • Counts. I rebuilt a bundle of the three packages at e56af9a: 3,272 facts, all person, with UK +2,993, England +20, tax_year 2023 +2,847, 15 geography keys and no duplicate keys. Each matches the moved pins. 3,439 - 3,272 = 167 dropped facts, which equals the extra copies in the 145 groups (126 pairs, 18 triples and one group of six: 126 + 36 + 5).
  • SI 2024/350 is the Individual Savings Account (Amendment) Regulations 2024.

Tests run locally

At e56af9a:

  • tests/test_source_hmrc_isa_statistics_2026.py and tests/test_chronicle_bundle.py: 45 passed, 1 skipped (the merged-bundle test, which needs R2-only artifacts; CI runs it).
  • build_package.py --check: exit 0.
  • validate-package on the three packages: valid, no warnings.
  • build-bundle with only the three packages: valid, no warnings.
  • ruff check: passed.

I have no open findings. The merged-bundle test still needs CI's bundle job to pass at this head.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

@PavelMakarchuk thanks, and for re-checking e56af9a so quickly. All nine items are addressed in four pushed commits: eb56b22, 9e6fc32, 854475c and e56af9a. No published value changed. The PR body is updated.

  1. must, the five bundle pins: done in e56af9a. Your follow-up was right: CI stopped at country:K02000001 (36,653 against 33,507). Item 2 leaves out 167 facts, so the numbers differ from your list:

    • country:K02000001 36,500 (+2,993), and country:E92000001 6,828 (+20);
    • 102,617 geography keys, as L478 says;
    • person 596,595;
    • 2,324 semantic duplicate keys, in both pins (your L505 and L2306).

    I measured these on a bundle of the three packages. On CI at 84682e9, every pin before L2248 passed. The bundle job is re-running.

  2. should, restated totals: done in eb56b22, as GSGB 2025 gambling participation by sex, age, area and activity, for PolicyEngine/microcosm#1381 #391 did. Each cell HMRC prints in more than one of Tables 9.7 to 9.12 is now ported once. build_package.py asserts every copy equal to the kept cell before leaving it out:

    • Tables 9.7–9.9's "All ISA Holders" columns restate 9.10–9.12's "Total" columns row for row, so holder totals come from the holder tables;
    • the all-sexes, all-ages rows of 9.8 and 9.11 and the UK rows of 9.9 and 9.12 restate the "All ranges" rows of 9.7 and 9.10.

    That leaves out 167 copies (126 + 2 × 18 + 5). The ISA package has 3,230 facts. A bundle of the three packages has no semantic duplicate key and no warning. The tests find each copy in the workbook by label and check it against the ported fact.

  3. should, the two traps: done in 854475c.

    • The gap. The caveats now give its size. At band floors the "all" block is at least £114.3bn, and cash-only alone £78.1bn. Table 9.4 has £100.6bn of cash and stocks and shares (£103.0bn with Innovative Finance and Lifetime ISAs). So the two can't both bind.
    • Accounts aren't people from 2024-25. "For Microcosm" now says that from 6 April 2024 savers can subscribe to more than one ISA of the same type (SI 2024/350). So 9.4's 2024-25 accounts aren't people, and their growth isn't growth in subscribers. This is in the docs only, as you suggest.
  4. should, Help to Buy: done in 9e6fc32. Section 3.4 is quoted in the notes of every 9.7–9.12 measure. It is also in 9.4's and 9.6's notes from 2015-16, since the scheme opened on 1 December 2015. The quote is the whole sentence, which goes on: "but the information is not separated out".

  5. nit, the holder note: done in 9e6fc32, with one correction.

    • Holder measures now cite the commentary's 23 million holders (section 1.4).
    • 9.7–9.9's no-further-subscription measures now read as holders.
    • The commentary turns out to have no subscriber count. For Adult ISAs in 2023-24 it gives only 15 million accounts (section 1.1) and the 23 million holders. So the subscriber note now cites those two, and "13.05 million" is gone from the notes, the YAML header and the docs.
    • The average market values are in pounds, so they no longer carry the "(thousands)" note.
  6. optional, Lifetime ISAs: done in 854475c. The docs give your arithmetic with section 5.2: 9,938,955 against 9,939k, 4,091,275 against 4,091k, and the blocks summing to 13,054,561.

    • They conclude that Lifetime-ISA-only subscribers are outside 9.7.
    • They add that HMRC doesn't say whether a Lifetime ISA payment counts towards a subscriber's band.
    • A test pins the arithmetic.
    • The fact notes keep "HMRC does not say how Lifetime ISAs are classed", since that is HMRC's own position.
  7. nit, exactly £12,000: done in 9e6fc32. Each row carries its band alone, with no amount bound.

  8. nit, the 9.4 role: done in 9e6fc32. It is isa_account for amounts as well as accounts, since the whole table is account-level.

  9. nit, the adult universe: done in 9e6fc32. The share record set has a shared age >= 18 constraint. Its note adds that HMRC doesn't say whether holders aged 16 or 17 are in the numerator.

Fact keys move only where items 7 to 9 change identity: Table 9.4 (role), and Table 9.12's shares and the impact note (constraints). On #377, agreed: whichever merges second re-pins.

Tests:

  • test_source_hmrc_isa_statistics_2026.py: 19 passed.
  • test_chronicle_bundle.py: 26 passed, 1 skipped. The skip is the merged build, which CI runs.
  • build_package.py --check: clean.
  • validate-package and build-suite: valid for all three packages.
  • ruff check: passed.

🤖 Generated with Claude Code

@PavelMakarchuk

Copy link
Copy Markdown

@juaristi22 thanks. I checked the extra points in your summary on a bundle of the three packages at e56af9a:

  • Table 9.4's 2023-24 amounts sum to £102,995m (£103.0bn) with Innovative Finance and Lifetime ISAs, and £100.6bn without them.
  • Every Table 9.4 fact has the role isa_account.
  • The Help to Buy quote is on Tables 9.4 and 9.6 from 2015-16 on, and on all 2,811 facts of Tables 9.7 to 9.12.

One correction to my follow-up: it says three commits, but there are four. I missed eb56b22. My checks ran on the head, so they cover it. Nothing else is open from my side. CI's bundle job is the last step.

🤖 Generated with Claude Code

@PavelMakarchuk PavelMakarchuk left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at e56af9a. All findings from my reviews are closed, and CI's bundle job passes at this head.

🤖 Generated with Claude Code

@juaristi22
juaristi22 merged commit 0b721a4 into main Oct 29, 2026
3 checks passed
juaristi22 added a commit to PolicyEngine/microcosm that referenced this pull request Oct 29, 2026
PolicyEngine/chronicle#392 merged as 0b721a4. Its tree is the head's
(e56af9a), merged onto the unmoved base 272c987, so the artifact is the
same bytes: the row count and the facts and manifest digests are
unchanged. Only what restates or hashes the commit moves:

- uk/chronicle_feed.json and the two tests that restate it;
- the vendored resources' feed headers, re-vendored with their rows
  unchanged;
- the DESNZ constituency resource, whose keys are re-derived from the
  feed identity, and the local membership's keys for it;
- both memberships' feed labels, the local census and the local
  validation levels;
- the coverage manifest's conditioning-resource digest and the H2
  fixture's stage contract hashes.

The local references, regenerated from the local cut, are byte-identical,
and neither compile-parity receipt names the commit. The runbook's
history entry now names the merge commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit to PolicyEngine/microcosm that referenced this pull request Oct 29, 2026
The UK Chronicle feed moves to 53815b2, the merge of
PolicyEngine/chronicle#394, on top of the 0b721a4 merge of
PolicyEngine/chronicle#392 (1,511,063 rows; the schema digest is
unchanged). The merge commit's tree is the pull request head's (2ba21f3),
so the artifact was built from that head's bundle and carries the merge
commit's identity.

The vendored resources and the constituency energy resource carry the
same rows under the new feed identity, and the high-value home stock's
inputs are unchanged (the feed-gated check passes). On the national
surface only the intended values move: the 13 SPI interest rows (up
13.4%), the 13 other-income rows and the 16 Table 3.10 rows.
obr.winter_fuel_allowance is pinned to March's expenditure table, since
October's FY2025-26 outturn (0.02% above March's projection) would match
it twice; the OBR targets that window FY2024-25 stay on March because
only March carries that year. On the local surface the statuses and the
references are unchanged; the membership moves its feed label and only
the 977 DESNZ gas cells, back to the Energy Trends level after commit 30
(x1.067). The local references, the national and
local compile-parity receipts, the coverage manifest and the H2 fixture
are regenerated, and the two tests that restate the pin follow it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit to PolicyEngine/microcosm that referenced this pull request Oct 30, 2026
…H5 round trip

- After the merge of main, this PR no longer re-pins the UK Chronicle
  feed: #1420 moved it to 53815b2, which carries PolicyEngine/chronicle#392's
  ISA statistics, and #1420's fragment records that re-pin. The
  1415-uk-chronicle-repin-isa-statistics fragment, which announced a re-pin
  to #392 with e56af9a's counts, becomes an added fragment for what this PR
  adds: the vendored hmrc_isa_statistics_facts.json from the pinned feed.
  The method doc names the 53815b2 pin as the source.
- The frame serializer registry's UK single-year round trip passes the ISA
  income contract that _write_uk_single_year_tables requires since the
  microcosm#1360 commit. Engine-free CI stopped on the register test first
  at a4785ef, so this failure only showed at 351fbfa.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants