Zarr I/O PR 2: CFDataset, and the CF variable classes rewritten against it

[trexfeathers]

Part of #6977. Second of seven pull requests into brownfield; see
docs/superpowers/specs/2026-09-21-zarr-io-design.md §5 for the programme
and §12.1 for its state. The implementation plan is in
docs/superpowers/plans/2026-09-24-zarr-io-pr2.md, and ships with it.

What this does

Introduces iris.fileformats.cf.dataset: a pair of narrow abstract classes,
CFDataset and CFDatasetVariable, describing what Iris needs from an array
store. iris.fileformats.netcdf._dataset implements them over netCDF4, and
everything that used to reach through CFVariable into a netCDF4.Variable
now goes through the interface instead — the CF reader, the load rules, the
netCDF loader, and the netCDF saver.

The change that will be most visible to a reader is CFVariable.__getattr__.
It resolved an attribute name against the file and against the backing
netCDF4 object, and cached the answer with setattr. It now resolves against
self.attributes only. A CF attribute called shape, dtype or name used
to be shadowed by the netCDF4 member of that name and silently lost; it
reaches the cube now.

Behaviour changes

Ten, all deliberate. Items 1 to 7 and items 9 and 10 change what Iris
does. Item 8 changes what Iris requires of its inputs, which is still
a behaviour change from the caller's side — a save that used to work now
raises — so it is in the list rather than in a compatibility footnote.

This list has been declared complete three times and grown three times —
two, then eight, then nine, now ten — each time because a review enumerated
something the previous pass had walked past, not because the code changed
underneath it; read "ten" as the number three reviews have reached, not as a
number anyone has proved.

Each item names the test that pins it. Every one of those tests was checked
by reverting the behaviour in the source and confirming the test fails; where
that check found no coverage, the item says so plainly instead of implying
there is a pin.

1. A CF attribute whose name collides with a member name is no longer
   lost. One headline, two unrelated mechanisms, so they are listed
   separately.

   • On save (finding F13). The "don't clobber" guard in
     Saver._set_cf_var_attributes was hasattr(cf_var, name) and is now
     name not in cf_var.attributes. An attribute named shape, size,
     name, dtype, dimensions or mask is written to the file, where the
     guard used to see the netCDF4 member of that name and drop the attribute
     silently.
     Pinned by lib/iris/tests/integration/netcdf/test_attributes.py::TestAttributesNamedLikeNetcdf4Members::test_attributes_named_after_netcdf4_members_are_saved.
     Restoring the hasattr form fails it.
   • On load, for names that are not declared CFVariable properties.
     __getattr__ resolves against self.attributes instead of falling
     through to the netCDF4 object, so the file's value wins and is recorded
     as used rather than being consumed and discarded.
     Pinned by lib/iris/tests/unit/fileformats/cf/test_CFVariable.py::TestShadowedAttributeNames,
     parametrised over SHADOWED_NAMES = ["filename", "cf_name", "spans", "attributes", "cf_data"]. Filtering those names out of the attribute
     mapping fails 15 of the class's 17 tests.

2. The five declared-property names behave differently again — a different
   mechanism with the same headline. dimensions, shape, ndim, dtype
   and size are declared properties on CFVariable and still resolve to the
   storage object: cf_var.shape returns the array shape, not a file
   attribute called shape. What changed is that the read is no longer
   recorded. The old __getattr__ marked such a name as read before
   forwarding to the netCDF4 variable, so _add_unused_attributes dropped it
   and a legitimate user attribute vanished from the loaded cube — even though
   the value returned was never the file's. A declared property short-circuits
   __getattr__ entirely, so nothing is recorded and the attribute survives
   onto the cube. This is emphatically not "the shadowing was removed": the
   member still wins.
   Pinned by lib/iris/tests/unit/fileformats/cf/test_CFVariable.py::TestTypedProperties::test_a_file_attribute_colliding_with_a_typed_property,
   parametrised over all five. Making each property record its own name fails
   all five parametrisations.

3. CFVariable.attributes is a snapshot taken at construction, so nothing
   on the load path can write to the file it is reading.
   Pinned by lib/iris/tests/unit/fileformats/cf/test_CFVariable.py::TestAttributeAccess::test_attributes_are_a_snapshot_taken_at_construction,
   with lib/iris/tests/unit/fileformats/cf/dataset/test_TrackedAttributes.py::TestMutation
   underneath it for the write-through semantics that make the snapshot
   necessary. Wrapping the backend mapping live instead of copying it fails
   the first; making TrackedAttributes.__setitem__ not write through fails
   the second.

4. A malformed file that used to raise now loads. Where ncattrs() lists
   a name that getncattr() then refuses — which the netCDF4 library does for
   some malformed files — _NetCDFAttributes substitutes "" and the load
   succeeds. cf/_reader.py's old _getncattr already tolerated it with the
   same default; the leniency now lives one layer down, so the same malformed
   file gets one answer rather than two different ones depending on which
   layer read it.
   Pinned by lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py::TestAttributes::test_unreadable_attribute_becomes_empty_string.
   Removing the except AttributeError fails it.

5. The exception type on a missing CF attribute moved AttributeError →
   KeyError. Call sites that read a CF attribute by attribute access
   (cf_var.long_name) now subscript the mapping
   (cf_var.attributes["long_name"]), and a miss raises KeyError where it
   used to raise AttributeError. Minor, but downstream code catching
   AttributeError around a CF read will see a different exception. There
   are six such sites — four on the load path, and two on the save path,
   which is worth stating separately because a reader of this list would not
   think to look there:

   • Load. cf/_reader.py:493-494 reads standard_name and then
     long_name; with FUTURE.derived_bounds off — the default — both
     subscripts are unguarded, and with it on, standard_name is guarded by
     an in test three lines above while long_name is not.
     _nc_load_rules/helpers.py:1158 reads grid_mapping_name.
     netcdf/ugrid_load.py:380 reads node_coordinates, and :406 reads
     the connectivity's cf_role, inside an assert.
   • Save. netcdf/saver.py:1159 reads standard_name and :1187 reads
     coordinates, both while rewriting a dimensionless vertical
     coordinate's formula_terms.

   A seventh converted read, _nc_load_rules/helpers.py:567, is deliberately
   not in that list: its contextlib.suppress was changed from
   AttributeError to KeyError along with the read, which is exactly the
   handling the six above did not get.

   Pinned by lib/iris/tests/unit/fileformats/cf/test_CFReader.py::Test_translate__formula_terms_derived_bounds::test_raises_when_standard_name_empty_and_long_name_absent
   — for the cf/_reader.py site only, and there for its long_name
   subscript. The other five have no test that fails if reverted. That is
   measured, not assumed, and measured site by site rather than extrapolated:
   softening each subscript to .get() and running the whole of
   lib/iris/tests/unit/fileformats/ and lib/iris/tests/integration/ left
   the suite green every time — same collected total as the unmutated
   control, no new failure at any of the five.

   All five are malformed-input paths — a UGRID mesh variable with no
   node_coordinates or no connectivity attribute, a grid-mapping variable
   with no grid_mapping_name, a formula_terms variable with no
   standard_name, a cube variable with no coordinates — and none is
   reachable from a well-formed file. They ship flagged rather than
   fixed. Closing them means writing new tests for five malformed-input
   paths into the final commit of a branch whose whole claim is that it
   changed nothing; disclosing them here is the better trade. A follow-up
   issue covering all six sites — settling once whether a missing CF
   attribute should raise, and which exception — would sit naturally with the
   merge-back changelog; this pull request proposes one rather than opening
   it.

6. IRIS_RAW differs on the error-recovery path. CFReader now writes
   its synthesised and invalidated bounds links into the attributes mapping
   rather than onto the CFVariable object, and build_raw_cube reads that
   mapping unfiltered. A formula-term or derived-bounds variable that falls
   back to build_raw_cube gains a bounds key it did not have — and where
   the file carried its own bounds, the invalidation writes None over it,
   so IRIS_RAW shows None where it used to show the file's value. Narrow,
   but real.

   The write is pinned; the behaviour change is not. Deleting the
   invalidating write fails
   lib/iris/tests/integration/netcdf/derived_bounds/test_bounds_files.py::test_load_legacy_hh[with_db]
   and lib/iris/tests/unit/fileformats/cf/test_CFReader.py::Test_translate__formula_terms_derived_bounds::test_promotes_non_formula_root_bounds_to_data.
   But replacing it with the pre-PR form — root_var.bounds = None, a plain
   Python attribute that never reaches the mapping and so never reaches
   IRIS_RAW — leaves every test green, including
   test_CFReader.py::TestSynthesisedBoundsLink, whose own docstring is
   explicit that it characterises TrackedAttributes rather than exercising
   _reader.py. So this item joins item 5 as declared-but-unpinned, and for
   the same reason it is flagged rather than fixed here.

7. A borrowed bare netCDF4.Dataset now decodes character data.
   CFReader and iris.load both accept an already-open dataset. Where that
   object is not already one of Iris's wrappers — what the ncdata /
   Xarray bridge passes — NetCDFDataset.from_existing now wraps it in an
   EncodedDataset, where CFReader previously stored it untouched. Checked
   against a real pre-branch checkout: cf_group["labels"][:] goes
   |S1 (3, 4) → <U4 (3,), and iris.load(bare_dataset) on a file whose
   coordinates names a char(x, strlen) variable goes aux_coords=[] →
   [('labels', '<U4')]. So a char auxiliary coordinate that used to be
   dropped now loads, and a bare borrow behaves like every other input. Two
   qualifications: the wrapping is unconditional and does not consult
   DECODE_TO_STRINGS_ON_READ (only __init__ does), so the read-time
   opt-out cannot be exercised on a borrowed dataset; and NetCDFDataset.mode
   reports the stipulated "r+" for a read-only borrow, which is cosmetic —
   nothing in the library reads it but __repr__.
   Pinned by lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py::TestABorrowedBareDataset::test_char_data_is_decoded.
   Removing the wrap fails it.

8. Saving into an emulated netCDF4 dataset now requires three members it
   did not before. This lands on emulators — ncdata and the Xarray bridge
   (Xarray bridge #4994) — not on real netCDF4 datasets. The module docstring of the test
   that covers it says it best:

   Saving through a CFDataset asks an emulating object for three members
   that the saver never used to reach for. All three are public netCDF4 API,
   and all three are now required of an emulator:

   • Dimension.size - the CF layer reads dimension lengths through it.
     __len__ is not an alternative, as _EmulatedDimension explains.
   • Dataset.ncattrs() - read once, as the dataset is wrapped.
   • Variable.ncattrs() - read once per variable, as each is wrapped.

   They are marked below where the emulators define them. This is an
   accepted consequence of the change, not an oversight: there is
   deliberately no fallback for an emulator that lacks them.

   And, at the _EmulatedDimension definition site, on why len() will not
   serve:

   netCDF4.Dimension.size, which is how the length is read back. len() is
   not an alternative: the emulator is put inside a _thread_safe_nc wrapper,
   whose getattr forwards named members but is never consulted for the
   len() protocol.

   The absence of a shim is deliberate: a speculative
   getattr(dim, "size", None) fallback was declined as defensive wrapping
   for an unconfirmed problem, which lib/iris/AGENTS.md bans.
   Pinned by the emulator doubles and the saves through them in
   lib/iris/tests/unit/fileformats/netcdf/saver/test_Saver__user_dataset.py.
   Deleting any one of the three members makes all five emulator saves error
   — confirmed for each of the three separately. This is the only one of the
   ten whose pin is a test double rather than a behavioural assertion,
   which is appropriate: the thing being pinned is an interface demand on
   third-party code, and the double is Iris's only standing description of
   what that code must provide. It is not a live ncdata test.

   A question for a maintainer before this lands: none of this is
   verified against real ncdata. ncdata is not installed in the development
   environment used here, and installing it needs your say-so. Is an ncdata
   round-trip check wanted first? If ncdata turns out to lack any of the
   three members, the fix is one getattr in the dataset layer, not a
   redesign.

9. Reaching a netCDF4 member through a CFVariable now warns. The old
   __getattr__ answered any name the backing netCDF4 object carried —
   value = getattr(self.cf_data, name), silently. The route survives for
   one deprecation cycle, but it now goes through
   NetCDFDatasetVariable.deprecated_netcdf_member, which issues an
   IrisDeprecation naming the member and pointing at
   cf_var.cf_data.variable. CFVariable is exported in
   iris.fileformats.cf.__all__, so this reaches callers outside Iris:
   third-party code that read cf_var.getncattr or cf_var.group gets a
   warning where it got silence, and under -W error a read that worked
   before now raises. A hasattr() probe that comes back True warns too,
   because the member was fetched; one that comes back False does not,
   because nothing was used. Nothing in Iris itself takes the route — see
   the reach-through run below, which measures that at zero.
   Pinned by lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDatasetVariable.py::TestDeprecatedNetcdfMember::test_a_reach_through_warns
   and …::TestDeprecatedNetcdfMember::test_the_message_names_the_replacement,
   both pytest.warns(IrisDeprecation), with
   …::TestDeprecatedNetcdfMember::test_a_missing_name_raises_and_does_not_warn
   holding the probe boundary. Deleting the warn_deprecated call fails the
   first two and leaves the third passing.

10. A dataset the caller hands to iris.save is now mutated.
    NetCDFDataset.from_existing calls set_auto_chartostring(False) on the
    object it borrows, so saving into a caller's already-open dataset turns
    auto-chartostring off on it, permanently, as a side effect of the save.
    The merge base's save path never called it. Two qualifications, both
    narrowing: the load path already did exactly this before this branch
    (cf/_reader.py applied the same call to datasets it borrowed), so the
    change is to the save path alone; and it is inert for anything
    from_existing has to wrap — a bare netCDF4.Dataset, or an emulator —
    because the wrapping EncodedDataset does its own decoding and swallows
    the call rather than forwarding it. What is left is a save into an
    object that is already an Iris thread-safe wrapper, where the call
    reaches the real netCDF4 dataset underneath. Iris's saver reads no
    character data, so nothing here does anything with the setting; it is
    listed because a permanent change to an object the caller still owns is
    caller-visible whether or not it matters.
    Pinned by lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py::TestAutoChartostring::test_turned_off_on_a_borrowed_dataset,
    which spies on the class rather than the instance because
    DatasetWrapper.__setattr__ forwards instance-level patching to the
    contained object. Removing the call fails it.

An eleventh that is not a change: spec §5 allowed one behaviour fix, to
CFVariable.spans. Reading the code showed the gap is unreachable —
_NCZARR_SCALAR_DIMENSION only ever appears as a variable's sole dimension —
so this pull request characterises the current behaviour rather than altering
it, and spends the allowance on (1) instead.

Five things a reviewer will want stated

cf_patch still receives netCDF4 objects. The public
iris.site_configuration["cf_patch"] hook has been documented since Iris 1.3
as being handed netCDF4 objects. Both arguments — the dataset and the
variable — are unwrapped before the call, so a CFVariable never reaches it.
An interim version of this branch passed a NetCDFDatasetVariable, which
defines no __setattr__ and so would have swallowed variable.name = value
silently; that was caught and fixed, and is now pinned by
lib/iris/tests/integration/netcdf/test_attributes.py::TestCfPatch::test_cf_patch_receives_a_netcdf4_variable.

.hooks/check_netcdf4_imports.py gained two entries. The repository
forbids import netCDF4 outside _thread_safe_nc;
lib/iris/tests/unit/fileformats/netcdf/dataset/test_NetCDFDataset.py and
lib/iris/tests/unit/fileformats/cf/test_CFReader__dataset.py are added to
its _PERMITTED_SUFFIXES because they must construct a real, bare
netCDF4.Dataset to test the wrapping path that exists for exactly such an
object. This widens a lint allow-list for two test modules; it is not a
behaviour change.

Where a file attribute and an undeclared netCDF4 member share a name, the
file wins now. That is the load half of behaviour change 1, stated the
other way round, and it is worth saying explicitly because it inverts a
precedence. The stronger statement is that nothing inside Iris depends on
that precedence any more, in either direction: after this branch no reader in
the library reaches an undeclared netCDF4 member through a CFVariable at
all. That is measured rather than asserted — the reach-through run below
promotes this branch's own deprecation warning to an error and gets zero
occurrences. (ugrid_load.py's two error messages did read cf_var.name,
which would have been exactly such a reach; this branch moved them to
cf_var.cf_name, Iris's own name for the variable.) So the inversion is
visible only to a caller outside Iris, reading a colliding name from a
CFVariable — and only for a file perverse enough to carry a CF attribute
named after a netCDF4 member.

Six assertions in
lib/iris/tests/unit/fileformats/netcdf/loader/test__translate_constraints_to_var_callback.py
changed, and that is not a loosening. They were only ever reachable through
MagicMock auto-vivification — the double returned a MagicMock for any
attribute, so the assertions described the mock, not a variable. Re-measured
against a real backing variable across 36 cases, the old and new forms
diverge in none of them; the new form describes what a real CFVariable
does.

Eight grid-mapping parameter assignments gained coverage they never had —
Mercator's scale_factor_at_projection_origin and the seven in the
PolarStereographic branch, in
lib/iris/tests/integration/netcdf/test_attributes.py::TestGridMappingAttributes.
These are tests added, not behaviour changed: a round-trip against the
pre-branch tree confirmed the values were always reaching the file.

How the negative claim was checked

• Full suite against the merge base. Baseline (recorded on 2cc0b01d4
  before any of this work): 11724 passed, 66 skipped, 6482 warnings, with
  zero FAILED and zero ERROR lines. This branch: 11992 passed, 66 skipped, 6483 warnings, also zero and zero. The normalised diff of the
  FAILED/ERROR lines is empty on both sides. Read nothing into the
  warning totals. Five runs of one unchanging tree gave 6483, 6485, 6484,
  6483, 6483 — the summary names the same 13 warning sites every time, but
  import-time module deprecations are counted once per -n auto worker that
  imports the module, so the total moves with how the tests happened to be
  distributed. An earlier draft of this section explained the 6482 → 6483
  step as one extra iris.save; that was a cause invented for a difference
  inside run-to-run noise, and it is withdrawn. The pass and skip counts are
  stable across those runs and are the ones carrying the claim. The skip
  count is read deliberately and is unchanged — a test that silently stopped
  running is neither a pass nor a failure, so a failure diff cannot see it.
• +268 passed, accounted for exactly. 200 tests in new modules, plus 67
  added in modified modules (counting parametrisations), plus 6 from two
  inherited mixin methods reaching three modules, minus 5 removed. No
  parametrize decorator was removed and no existing parametrisation was
  widened, so the arithmetic is not hiding a substitution.
• The boundary. cf/dataset.py imports only abc, collections.abc,
  types, typing and numpy; a token-level scan excluding comments and
  docstrings finds exactly one backend name in executable code,
  deprecated_netcdf_member, which spec §4.2 mandates. saver.py contains
  no import netCDF4 and no _setncattr, and names netCDF4 in exactly 68
  places: three self._dataset.dataset escape hatches (two file_format
  reads and the cf_patch hand-off), two cf_var_grid. and sixty-three
  grid_variable. grid-mapping assignments.
• The deprecating reach-through fires nowhere in Iris's own paths.
  Running lib/iris/tests/unit/fileformats/ and
  lib/iris/tests/integration/netcdf/ with the reach-through warning
  promoted to an error produces zero occurrences.

Not here

No changelog fragment: feature-branch pull requests defer those to the
merge-back, along with the closing keywords. See §5 of the spec. Behaviour
change 9 — the deprecation on the netCDF4 reach-through — is the kind of
change a fragment would normally accompany, and a follow-up issue for the
six KeyError sites in behaviour change 5 is proposed above; both belong
with the merge-back, where the fragment is written.

The sixty-three grid-mapping parameter assignments in saver.py keep a
netCDF4 handle rather than move to .attributes, because they bypass the
ASCII-to-bytes coercion and moving them would change the CDL — crs_wkt
would stop being NC_STRING. Recorded as Q8 in the spec.

One unrelated defect was found and is not fixed here, and is not a flake.
Running lib/iris/tests/unit/fileformats/netcdf/loader/test_load_cubes.py
before lib/iris/tests/integration/netcdf/test_coord_systems.py in the same
process errors all ten tests in the latter. test_coord_systems.py imports
the unit module as tlc, and its module-scoped autouse fixture only reaches
its yield inside if not hasattr(tlc, "TMP_DIR"), while the unit module
sets that global at import. Unit module first ⇒ the fixture returns without
yielding ⇒ every test in the module errors. It is deterministic on module
ordering, not on scheduling. All three files are byte-identical to the merge
base, so it predates this branch — but it is a latent CI failure under an
unlucky ordering and is worth an issue of its own.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 98.11321% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.79%. Comparing base (6d61c5d) to head (a2bca4c).

Files with missing lines	Patch %	Lines
lib/iris/fileformats/netcdf/_dataset.py	96.77%	7 Missing ⚠️
lib/iris/fileformats/netcdf/ugrid_load.py	81.25%	0 Missing and 3 partials ⚠️
lib/iris/fileformats/_nc_load_rules/helpers.py	98.91%	1 Missing ⚠️
lib/iris/fileformats/cf/dataset.py	99.20%	1 Missing ⚠️
lib/iris/fileformats/netcdf/saver.py	99.21%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@              Coverage Diff               @@
##           brownfield    #7303      +/-   ##
==============================================
+ Coverage       90.61%   90.79%   +0.17%     
==============================================
  Files              96       98       +2     
  Lines           25859    26204     +345     
  Branches         4802     4819      +17     
==============================================
+ Hits            23433    23791     +358     
+ Misses           1668     1657      -11     
+ Partials          758      756       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[trexfeathers]

Two regressions were reproduced against the merge base: silent corruption of deferred string writes to borrowed datasets, and failure to load NetCDF files with Dask's process scheduler. Details are attached inline.

Created by Open AI Codex 🤖

[trexfeathers]

Both findings fixed in 0c5ac2e30; ed0d0f766 corrects the body.

Neither becomes an eleventh documented behaviour change — they're regressions fixed before merge, not behaviour this branch means to introduce, the same call made earlier for the boundsterm_varname regression. The list stays at ten.

Two things these findings taught me, both now reflected in the body:

• P1 sits on exactly the path documented behaviour change 10 describes. Disclosing a side effect is not the same as auditing the path it runs on.
• Re-deriving the suite numbers turned up something else: 6483 warnings is not reproducible. Five runs of an unchanging tree gave 6483, 6485, 6484, 6483, 6483 — import-time deprecations are counted once per xdist worker. The body had explained the baseline's 6482 → 6483 step as one extra iris.save; that was a cause invented for noise, and it's withdrawn. Pass and skip counts are stable and are what the claim rests on.

Suite now 11992 passed, 66 skipped (was 11989/66) — the +3 are the three added tests.

🤖 Generated with Claude Code

[trexfeathers]

Follow-up review by Codex, verified at commit ed0d0f766306749ff20a8ab07ecd735125752bb1.

Both original concerns are satisfactorily resolved:

• P1 — borrowed dataset string writes: Write handles now always use EncodedNetCDFWriteProxy, preserving Unicode strings when saving lazy data into a borrowed, unencoded DatasetWrapper.
• P2 — process-scheduler loading: Reading variables no longer creates saver locks. Variables use a factory to obtain the shared dataset lock only when a write handle is requested.

Both regression tests reproduce the original failures on the pre-fix commit and pass at the verified commit. All 118 dataset tests and 31 scheduler/deferred-save integration tests also pass, with no skips.

A broader run reported 401 passing tests before a segmentation fault during interpreter shutdown. The corresponding pre-fix run also crashed during shutdown after reporting 398 passes, so that crash predates these fixes. The broader run should therefore not be treated as a clean test-process exit.

No remaining concerns with these two fixes.