Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 69 additions & 51 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,12 @@ Notable changes per release. Versions follow the rules in
[contributing](docs/contributing.md#plugin-api-and-versioning): the plugin API is stable,
and a release that breaks it says so here.

## 0.1.0a1 (unreleased)
## 0.1.0a2 (2026-10-21)

The first release, an alpha: the plugin API can still change before 0.1.0.
The second alpha. The plugin API only grew (`overlap`, `computed_from`,
`demand.cancelled()`, source matchers, parameter models), but an `ArrayInfo` given no
translation now has voxel 0's corner at the origin, not its centre (under Sources): a
plugin's source that meant its centre there passes `translation=(0.0,) * ndim`.

### Core and contrib

Expand All @@ -22,14 +25,8 @@ The first release, an alpha: the plugin API can still change before 0.1.0.

### Serving

- Pipelines served as N5, Zarr v2 (OME-NGFF 0.4), Zarr v3 (OME-NGFF 0.5) and Neuroglancer
precomputed at once, and meshes made when fetched.
- A per-stage chunk cache keyed by the pipeline's hash, and cache-busting digests in links.
- Live edits through the REST API, Server-Sent Events, a control page and a
python-neuroglancer viewer that follows them.
- Work bounded, shared between requests and ordered finest level first, then by when
each request arrived; work no client waits for any more is dropped, and a long op that
asks `demand.cancelled()` between its steps gives up once none does.
- A long op that asks `demand.cancelled()` between its steps gives up once no client
waits for its work any more.
- BLAS kept to one thread per call in the server (`threadpoolctl`), and affines applied
without it: its threads fought the server's, and a three-field `warp://` read took ten
times as long.
Expand All @@ -43,18 +40,6 @@ The first release, an alpha: the plugin API can still change before 0.1.0.
A100, 13.5 s on 24 CPU threads); values within a grey level of the CPU's, cache keys the
same. A sampled field is read in its own precision, not copied
to float64.
- `--token` for the control API, `--https` with a self-signed certificate.
- zarr v2 and v3 chunks compressed with blosc (zstd, byte shuffle) by default, 10 to 30
times quicker to encode than gzip; `--compressor` chooses another. `--cache-gb` takes effect (an empty cache
given to the registry used to be replaced by the default 2 GiB one), and `0` turns the
cache off.
- `serve --port 0` and `--ready-file` for launchers; the port is bound before it is
announced.
- Datasets resolved by name on first request (`resolver`), routes from other packages
(`extra_routes`, the `chunkmirage.routes` entry point), and the app mounted under a prefix
in another app.
- `chunkmirage.serve` and `Server`: an app served from Python on a port bound first, with a
callback once it accepts connections, in a background thread or this one.
- `chunkmirage materialize` and `chunkmirage.materialize`: a dataset, or a box of it,
computed and written into a store (a directory, `gs://`, `s3://`) as a frontend serves it,
zarr v3, zarr v2 or N5, so it reads back as a stored dataset of its own; the metadata
Expand All @@ -78,17 +63,6 @@ The first release, an alpha: the plugin API can still change before 0.1.0.
whether it is computed and whether the full-resolution result under it is (a preview
told from a result while browsing), read afresh at every request;
`GET /api/datasets/{name}/coverage` gives the counts per level.

- zarr v2/v3, N5, precomputed and HDF5 on file, S3, GCS and HTTP; xarray arrays with their
coordinates and CF packing; cloud-optimized GeoTIFFs.
- Computed sources: `synthetic://`, `scene://` (OME-Zarr 0.6 transformations), `warp://`,
`register://` (deformable registration on a GPU), `stitch://` (BigStitcher tiles),
`stack://` and `flip://`.
- S3 read anonymously before credentials, GCS through its public URL when credentials
fail, metadata reads that give up after seconds, zarr compressors with members
tensorstore rejects.
- Voxels placed as Neuroglancer places them: precomputed `voxel_offset` and funlib `offset`
are corners.
- Stored arrays are opened and placed by worldarray (`worldarray>=0.1,<1`), the reader
chunkmirage shares with cellmap-flow, cellmap-analyze and mesh-n-bone; chunkmirage keeps
xarray coordinates, CF packing, the overrides and `kind`
Expand All @@ -109,8 +83,6 @@ The first release, an alpha: the plugin API can still change before 0.1.0.
as precomputed their `voxel_offset` is 0 rather than a rounded half voxel. Explicit
translations, and arrays that take their input's, are unchanged
([formats](docs/concepts/formats.md#stored-sources)).
- Stored integers of 32 bits or more read as labels, booleans as masks, unless the spec's
`kind` says otherwise.

### Ops and pipelines

Expand All @@ -124,28 +96,12 @@ The first release, an alpha: the plugin API can still change before 0.1.0.
cached as a stage of their own, and a served chunk is the weighted mean of every block
reaching it, smooth where blocks computed on their own disagree and the same whatever
order they are computed in; `register://`'s refined blocks blend with the same weights.

- Pointwise ops, filters, morphology, connected components, spots, contacts, differences,
gradients, downsampling, normalized differences, slope and hillshade, with halos handled
by the pipeline.
- Ops that return their block shaved by the halo or add leading axes; ops at one
resolution with the coarser levels downsampled from their output, or the nearest level
read as it is (`input_level`, `level_rtol`).
- `cache_token()` for ops whose output depends on state outside their parameters, and
`DatasetRegistry.refresh` to read it again.
- `slots` to bound how many calls of an op run at once.
- `Pipeline.plan(level, box)`: what a read would compute, computing nothing: per stage, the
chunks it is asked for, those held and those to compute, following `dataset://` into
the datasets it reads and stopping at other sources.
- Zero padding past the volume's edge (`padding`).
- Channel axes read whole, so an op may change their length: select or combine channels.
- A level shrunk by a whole factor for an op at one voxel size is the mean of each block,
not linear samples.

### Browser engine

- Registration on WebGPU and pipelines of the package's own ops in Pyodide, in a gallery of
demos.
- The engine as a base for a page of another project
([design](docs/design.md#another-page-on-the-engine)): Python loads only when a view
needs it, a view with no ops served from the reader alone; `unserve()` forgets a view and
Expand Down Expand Up @@ -204,3 +160,65 @@ The first release, an alpha: the plugin API can still change before 0.1.0.
behind every chunk a client waits for, in all the queue's slots but one, and never dropped
as abandoned (`submit`, `claim` and `release` take `background`;
[caching](docs/concepts/caching.md#order-of-work-and-requests-given-up-on)).

## 0.1.0a1 (2026-10-05)

The first release, an alpha: the plugin API can still change before 0.1.0.

### Serving

- Pipelines served as N5, Zarr v2 (OME-NGFF 0.4), Zarr v3 (OME-NGFF 0.5) and Neuroglancer
precomputed at once, and meshes made when fetched.
- A per-stage chunk cache keyed by the pipeline's hash, and cache-busting digests in links.
- Live edits through the REST API, Server-Sent Events, a control page and a
python-neuroglancer viewer that follows them.
- Work bounded, shared between requests and ordered finest level first, then by when
each request arrived; work no client waits for any more is dropped.
- `--token` for the control API, `--https` with a self-signed certificate.
- zarr v2 and v3 chunks compressed with blosc (zstd, byte shuffle) by default, 10 to 30
times quicker to encode than gzip; `--compressor` chooses another. `--cache-gb` takes effect (an empty cache
given to the registry used to be replaced by the default 2 GiB one), and `0` turns the
cache off.
- `serve --port 0` and `--ready-file` for launchers; the port is bound before it is
announced.
- Datasets resolved by name on first request (`resolver`), routes from other packages
(`extra_routes`, the `chunkmirage.routes` entry point), and the app mounted under a prefix
in another app.
- `chunkmirage.serve` and `Server`: an app served from Python on a port bound first, with a
callback once it accepts connections, in a background thread or this one.

### Sources

- zarr v2/v3, N5, precomputed and HDF5 on file, S3, GCS and HTTP; xarray arrays with their
coordinates and CF packing; cloud-optimized GeoTIFFs.
- Computed sources: `synthetic://`, `scene://` (OME-Zarr 0.6 transformations), `warp://`,
`register://` (deformable registration on a GPU), `stitch://` (BigStitcher tiles),
`stack://` and `flip://`.
- S3 read anonymously before credentials, GCS through its public URL when credentials
fail, metadata reads that give up after seconds, zarr compressors with members
tensorstore rejects.
- Voxels placed as Neuroglancer places them: precomputed `voxel_offset` and funlib `offset`
are corners.
- Stored integers of 32 bits or more read as labels, booleans as masks, unless the spec's
`kind` says otherwise.

### Ops and pipelines

- Pointwise ops, filters, morphology, connected components, spots, contacts, differences,
gradients, downsampling, normalized differences, slope and hillshade, with halos handled
by the pipeline.
- Ops that return their block shaved by the halo or add leading axes; ops at one
resolution with the coarser levels downsampled from their output, or the nearest level
read as it is (`input_level`, `level_rtol`).
- `cache_token()` for ops whose output depends on state outside their parameters, and
`DatasetRegistry.refresh` to read it again.
- `slots` to bound how many calls of an op run at once.
- Zero padding past the volume's edge (`padding`).
- Channel axes read whole, so an op may change their length: select or combine channels.
- A level shrunk by a whole factor for an op at one voxel size is the mean of each block,
not linear samples.

### Browser engine

- Registration on WebGPU and pipelines of the package's own ops in Pyodide, in a gallery of
demos.
4 changes: 3 additions & 1 deletion docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,4 +135,6 @@ to the stable API waits for a major release.
`pyproject.toml` and `chunkmirage/__init__.py` (`tests/test_docs.py` checks the three
agree).
2. Merge to `main` with CI green.
3. Publish to PyPI only when the owners say so: `uv build && uv publish`.
3. Publish to PyPI only when the owners say so: a GitHub release tagged `v<version>` runs
`.github/workflows/publish.yml`, which builds the wheel and uploads it (a manual run of
the workflow uploads to TestPyPI instead).
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "chunkmirage"
version = "0.1.0a1"
version = "0.1.0a2"
description = "Spoof chunked array formats (zarr, n5, neuroglancer precomputed) over HTTP with on-the-fly processing, caching, and pluggable ops."
readme = "README.md"
license = { file = "LICENSE" }
Expand Down
2 changes: 1 addition & 1 deletion src/chunkmirage/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@
from chunkmirage.sources import open_source

__all__ = ["ArrayInfo", "Box", "Pipeline", "PipelineSpec", "Server", "create_app", "materialize", "open_source", "serve"]
__version__ = "0.1.0a1"
__version__ = "0.1.0a2"
2 changes: 1 addition & 1 deletion uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading