0.3: grids and blocks

CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## 0.3.0 (2026-10-22)
+
+Grids and blocks, built to spec draft 3 (Behaviours 17 to 19). Nothing on the read or
+write side changes.
+
+- `blocks(shape, block_shape, context=0, box=None, clip=True)`: the array (or a voxel
+  box, which may reach past the array) cut into write boxes that tile it with no overlap
+  and no gap, each with a read box grown by `context` (one int, one per axis, or
+  `(before, after)` pairs) and clipped to the array when `clip`; C order, `len`,
+  `blocks[id]` in O(1), `index_of(begin)`; a `Block` carries `index`, `id`, `begin`,
+  `end`, `read_begin`, `read_end`, `at_start` and `at_end` per axis and `inner`, the write
+  box's slices inside the read box. Plain picklable records; nothing is scheduled.
+  `vol.blocks()` defaults the block to `write_shape`, so default blocks are aligned writes.
+  `vol.write_chunk(index, data)` writes one whole unit.
+- `Grid(shape, voxel_size, translation, axes, units, chunks, shards)`: a `WorldArray`
+  without a store, with `box`, `corner`, `index_box`, `chunks_covering`, `write_shape`,
+  `blocks` and `meta(dtype)`; `Grid.from_meta(meta, voxel_size=)` is an existing array's
+  grid or the resample target grid over it. `create(path, grid=g, dtype=)` creates from it.
+- `level_factors(voxel_size, ratio=1.5, ...)`: the anisotropy-aware pyramid schedule
+  mesh-n-bone and cellmap-analyze share, per level.
+- `units.abbreviation`, and `store.join`, `store.parent`, `store.basename` as public
+  URL-aware path helpers.
+
 ## 0.2.0 (2026-10-22)
 
 Metadata and placed creation, built to spec draft 3 (Behaviours 10 to 16). Nothing on the

README.md

@@ -97,6 +97,11 @@ s1 = wa.open_pyramid("out.zarr", mode="r+").add_level("s1", factors=2)   # place
   and only two answers to "it exists": raise, or adopt an identical array after checking;
   whole-unit writes from many processes with no coordination; `add_level` as metadata plus
   an empty array; chunk keys for every format. Nothing is ever deleted or overwritten.
+- **Plans outputs and blocks** (0.3): a store-less `Grid` with the same geometry as an
+  array, so an output is planned, cut into blocks and only then created; `blocks` tiling an
+  array or a box into write boxes with context-grown read boxes, in C order, as plain
+  picklable records a cluster job takes by id; the anisotropy-aware pyramid schedule.
+  Nothing is scheduled or run.
 
 ## The placement rule
 
@@ -143,6 +148,14 @@ wa.ome_multiscales(axes, units, levels, version="0.5", type=None) ; wa.level_pla
 wa.level_shape(shape, factors, origin=0) ; wa.chunk_key(index, format, ...) ; wa.parse_chunk_key(key, format, ...)
 wa.ensure_group_entry(group_path, name, voxel_size, translation, axes, units="nanometer")   # list an array you made yourself
 wa.units.ome_name("nm") -> "nanometer" ; wa.write.codecs_of(path) -> (codecs, shards)
+
+# grids and blocks (0.3)
+g = wa.Grid(shape, voxel_size, translation, axes=None, units="nanometer", chunks=None, shards=None)   # or Grid.from_meta(meta, voxel_size=)
+wa.create(path, grid=g, dtype="uint64")
+for b in wa.blocks(shape, block_shape, context=0, box=None, clip=True):   # also vol.blocks() and g.blocks()
+    b.begin, b.end, b.read_begin, b.read_end, b.inner, b.at_end, b.id    # plain records; blocks[id] is O(1)
+wa.level_factors(voxel_size, ratio=1.5, max_levels=None, stop_shape=None, shape=None)
+wa.units.abbreviation("nanometer") ; wa.store.join(path, "s0") ; wa.store.parent(path) ; wa.store.basename(path)
 ```
 
 `WorldArray`: `ts`, `meta`, `grid`, `shape`, `dtype`, `axes`, `voxel_size`, `translation`,

docs/spec.md

@@ -765,24 +765,38 @@ Behaviours 10 to 16 are 0.2; 17 to 19 are 0.3.
 17. **Blocks tile, context grows, nothing runs.** `blocks(shape, block_shape, context, box,
     clip)` yields, in C order, blocks whose write boxes `[begin, end)` partition `[0,
     shape)`, or `box` (a voxel box, the grid anchored at its begin) when given, with no
-    overlap and no gap, the last block on an axis clipped. A block's read box is its write
-    box grown by `context` on each side, where `context` is one int, one per axis, or a
-    `(before, after)` pair per axis, clipped to the array when `clip=True` (the default;
-    cellmap-analyze's `create_block` edge rule) and left unclipped when `clip=False` for a
-    padded `read_index`. `len(blocks)`, `blocks[b.id]`, `blocks.index_of(b.begin)` and
+    overlap and no gap, the last block on an axis clipped to the box. `box` may reach past
+    the array on any side (a ROI rounded outward to a factor multiple, mesh-n-bone's halo
+    mode): the write boxes tile it exactly, outside the array too. A block's read box is
+    its write box grown by `context` on each side, where `context` is one int, one per
+    axis, or a `(before, after)` pair per axis, clipped to the **array** (never to `box`,
+    never a write box) when `clip=True` (the default; cellmap-analyze's `create_block` edge
+    rule) and left unclipped when `clip=False` for a padded `read_index` (its
+    `neg_with_edge_pos` mode, which reads past the array for fill). A block also carries
+    `at_start` and `at_end` per axis, whether it is the first or last block on that axis,
+    so a caller adds context at the array's far faces itself, and `inner`, the slices of
+    the write box inside the read box, correct also when `clip=False` lets `read_begin`
+    go below 0. `len(blocks)`, `blocks[b.id]` (O(1)), `blocks.index_of(b.begin)` and
     `b.index` agree. `vol.blocks()` defaults `block_shape` to `write_shape`, so default
     blocks are aligned writes. Nothing is scheduled: the iterable and its blocks are plain,
-    picklable records. `write_chunk(index, data)` writes the unit at `index` whole;
-    `data.shape` must equal that unit's clipped shape.
+    picklable records, and the iterable pickles without its blocks, so a worker takes an
+    id. One grid per call: a consumer that writes several levels from one block derives
+    the per-level boxes itself (mesh-n-bone's direct mode, two lines). `write_chunk(index,
+    data)` writes the unit at `index` whole; `data.shape` must equal that unit's clipped
+    shape.
 
 18. **A `Grid` is a `WorldArray` without a store.** `Grid(shape, voxel_size, translation,
     axes, units, chunks, shards)` answers `box`, `corner`, `index_box`, `chunks_covering`,
     `write_shape` and `blocks` exactly as a `WorldArray` of those facts does, and
-    `g.meta(dtype)` equals `wa.create(path, grid=g, dtype=dtype).meta`, so a pipeline plans
+    `g.meta(dtype)` agrees with `wa.create(path, grid=g, dtype=dtype).meta` in every
+    field a grid has: shape, dtype, chunk and shard shape, axes, units, voxel size and
+    translation (a grid has no path, format, documents or fill value), so a pipeline plans
     its output before anything exists. `Grid.from_meta(meta, voxel_size=t)` is Behaviour
     4's target grid over that array: anchored at its corner with `ceil(n * s / t)` voxels,
-    so a `read(voxel_size=t)` of the source fills it exactly; this is the output grid
-    cellmap-analyze's commands compute today in the integer frame.
+    so a `read(voxel_size=t)` of the source fills it exactly, and its `box` is the target
+    grid's extent, which may reach past the source's own `box` by less than one target
+    voxel; this is the output grid cellmap-analyze's commands compute today in the integer
+    frame.
 
 19. **Schedules and helpers.** `level_factors(voxel_size, ratio, max_levels, stop_shape,
     shape)` returns per level the per-axis factors, 2 on every axis whose current voxel size
@@ -1017,11 +1031,12 @@ over funlib geometry, by this spec's own assignment.
    attributes? If so, should the package offer `funlib_attrs(voxel_size, translation, axes,
    units)`, writing `offset` as the corner, funlib's meaning, for `create(attrs=)`, or do
    the repos keep writing their own?
-6. **The block iterator's shape** (Behaviour 17) is written from cellmap-analyze's
-   `create_block` (directional context, the edge rule, a fourth axis) and mesh-n-bone's
-   super-chunk grids (factor-aligned strides per level) without either session confirming
-   it. Confirm with both before 0.3 is built. mesh-n-bone's direct mode, where one task
-   writes several levels at once, may need per-level grids that `blocks` does not give.
+6. **The block iterator's shape** (Behaviour 17). Resolved 2026-10-22: both sessions
+   confirmed it. mesh-n-bone asked that `box` may reach past the array and that `clip`
+   clips read boxes only, and derives its per-level boxes from one grid itself;
+   cellmap-analyze asked for `at_start`/`at_end` per axis and `inner`, and confirmed that
+   `(before, after)` pairs cover its context modes and that `clip=False` with a padded
+   `read_index` is its edge-reading mode. All are in Behaviour 17.
 7. **Sidecar attributes.** Resolved 2026-10-22: registration-station keeps `fs` and
    `fix_origin_um` on the group through `group_attrs` and the part attributes (`y0`, `y1`,
    `y0_core`, `y1_core`, `feather_vox`) on the array through `attrs`, all known at creation;
@@ -1056,8 +1071,8 @@ over funlib geometry, by this spec's own assignment.
 13. **World boxes on blocks.** cellmap-flow reads its input at one voxel size by a world
    box grown by a context in nanometres and writes at another; `blocks` (0.3) gives index
    boxes on one grid. `Grid.blocks` could carry world boxes too (`b.box`, `b.read_box` as
-   `Box`, a context in world units). Two lines on cellmap-flow's side today; a candidate
-   for 0.4.
+   `Box`, a context in world units). Two lines on cellmap-flow's side today. Not planned
+   (2026-10-22): daisy does cellmap-flow's tiling and stays; reopen if cellmap-flow drops it.
 
 ## 8. Constraints for whoever builds on this
 

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "worldarray"
-version = "0.2.0"
+version = "0.3.0"
 description = "Read zarr, N5 and Neuroglancer precomputed volumes in world coordinates through tensorstore: correct voxel placement, pyramid levels, padded region reads, resampling. numpy + tensorstore only."
 readme = "README.md"
 license = { file = "LICENSE" }

src/worldarray/__init__.py

@@ -2,16 +2,19 @@
 
 Open zarr v2, zarr v3, N5 and Neuroglancer precomputed arrays and pyramids by path, know
 where each sits (voxel size, the centre of voxel 0, units, axes), and read voxel boxes or
-world boxes as numpy arrays, padded where the box leaves the data. ``docs/spec.md`` is the
-contract; the README is its first five sections.
+world boxes as numpy arrays, padded where the box leaves the data; create placed OME-NGFF
+levels and write them from any number of jobs; plan an output grid and cut it into blocks
+before anything exists. ``docs/spec.md`` is the contract; the README is its first five
+sections.
 """
 
 from __future__ import annotations
 
-from . import store, units, write
+from . import grid, store, units, write
 from .array import Pyramid, WorldArray, create, metadata, open, open_pyramid, open_tensorstore
 from .context import capped_context, capped_context_spec, shared_context
 from .geometry import Box, index_range
+from .grid import Block, Grid, blocks, level_factors
 from .levels import select_level
 from .placement import ArrayMeta, corner_to_centre, default_axes
 from .retry import Retry
@@ -25,19 +28,24 @@
     parse_chunk_key,
 )
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"
 
 __all__ = [
     "ArrayMeta",
+    "Block",
     "Box",
+    "Grid",
     "Node",
+    "blocks",
     "chunk_key",
     "corner_to_centre",
     "create",
     "default_axes",
     "detect",
     "ensure_group_entry",
+    "grid",
     "index_range",
+    "level_factors",
     "level_placement",
     "level_shape",
     "list_chunk_keys",

src/worldarray/array.py

@@ -551,6 +551,34 @@ def write(self, box, data: np.ndarray, partial: bool = False) -> None:
                 )
         self.write_index(begin, end, data, partial=partial)
 
+    def blocks(self, block_shape=None, context=0, box=None, clip: bool = True):
+        """The array cut into blocks (Behaviour 17): write boxes tiling the stored grid (or a
+        voxel ``box``) with ``block_shape`` (default ``write_shape``, so the default blocks are
+        aligned writes), read boxes grown by ``context`` and clipped to the array when
+        ``clip``. Plain records, nothing scheduled."""
+        from .grid import blocks as _blocks
+
+        return _blocks(
+            self._grid.shape,
+            self.write_shape if block_shape is None else block_shape,
+            context=context,
+            box=box,
+            clip=clip,
+        )
+
+    def write_chunk(self, index: Sequence[int], data: np.ndarray) -> None:
+        """Write the whole write unit at ``index`` on the storage grid (the shard of a sharded
+        array); ``data`` has that unit's shape, clipped at the array's end."""
+        unit, shape = self.write_shape, self._grid.shape
+        index = tuple(int(i) for i in index)
+        if len(index) != len(shape):
+            raise ValueError(f"{self.path}: index needs one entry per axis of {shape}")
+        begin = tuple(i * u for i, u in zip(index, unit))
+        if any(i < 0 or b >= n for i, b, n in zip(index, begin, shape)):
+            raise ValueError(f"{self.path}: no write unit at {index}")
+        end = tuple(min(b + u, n) for b, u, n in zip(begin, unit, shape))
+        self.write_index(begin, end, data)
+
     def read_index(
         self,
         begin: Sequence[int],
@@ -1051,11 +1079,12 @@ def __len__(self) -> int:
 
 def create(
     path: str,
-    shape: Sequence[int],
-    dtype,
-    voxel_size,
-    translation,
+    shape: Sequence[int] | None = None,
+    dtype=None,
+    voxel_size=None,
+    translation=None,
     *,
+    grid=None,
     axes: Sequence[str] | None = None,
     units: Units = "nanometer",
     chunks: Sequence[int] | None = None,
@@ -1079,6 +1108,19 @@ def create(
     raises ``FileExistsError``; with ``"adopt"`` an identical array (shape, dtype, chunks,
     shards, placement) is opened for writing and a different one raises ``ValueError``
     naming every difference. Returns a writable ``WorldArray``."""
+    if grid is not None:
+        # a Grid planned the output (Behaviour 18): its facts are the array's, chunks and
+        # shards overridable; the placement arguments must not also be given
+        if any(v is not None for v in (shape, voxel_size, translation, axes)):
+            raise ValueError("pass grid= or shape, voxel_size, translation and axes, not both")
+        shape, voxel_size, translation = grid.shape, grid.voxel_size, grid.translation
+        axes, units = grid.axes, grid.units
+        chunks = grid.chunks if chunks is None else chunks
+        shards = grid.shards if shards is None else shards
+    if shape is None or dtype is None or voxel_size is None or translation is None:
+        raise ValueError(
+            "create needs shape, dtype, voxel_size and translation, or grid= and dtype"
+        )
     if chunks is None:
         raise ValueError("chunks is required: the package never picks a chunk shape")
     ndim = len(shape)