Add support for partial Zarr download and upload

[yarikoptic]

Summary

Design and implementation for partial zarr download and upload support, addressing #1462, #1474, and related archive issues.

The PR covers five areas:

1. --zarr TYPE:PATTERN filtering for dandi download — glob, path, and regex filters for selecting entries within zarr assets, with a metadata alias for common zarr metadata files
2. URL parsing with zarr boundary detection — AssetZarrEntryURL to handle URLs like dandi://dandi/000108/.../file.ome.zarr/0/0/0
3. --zarr-mode {full, patch} for dandi upload — patch mode uploads changed files without deleting remote files absent locally
4. Checksums and manifests — documents that per-directory checksums are computed hierarchically by the zarr_checksum library but are NOT persisted (only the root digest is stored in the DB); legacy .checksum files exist on S3 at zarr-checksums/ for ~72% of older zarrs but are orphaned since Dec 2022
5. dandi ls for zarr contents — listing files within a zarr asset when URL points into a zarr

Key findings from investigation

• The zarr_checksum algorithm IS hierarchical (Merkle tree, bottom-up via ZarrChecksumTree)
• The archive's ingest_zarr_archive task computes checksums entirely in memory and stores only the root digest
• Per-directory .checksum files on S3 (zarr-checksums/ prefix) were written by ZarrChecksumFileUpdater, removed in dandi-archive PRs Testing started to fail due to an error in parsing (?) + new deprecationwarning #1390/[FEAT] dandi fsspec filesystem #1395 (Dec 2022). Legacy files remain for older zarrs but no API exposes them
• Subtree checksum verification is not possible today without recomputation from file ETags

Implementation

New files

• dandi/zarr_filter.py — ZarrFilter dataclass with glob/path/regex matching, parse_zarr_filter(), make_zarr_entry_filter(), ZARR_FILTER_ALIASES (includes metadata alias)
• dandi/tests/test_zarr_filter.py — 52 unit tests covering all filter types, parsing, aliases, edge cases, invalid regex validation

Modified files

• dandi/dandiarchive.py — split_zarr_location(), AssetZarrEntryURL class, updated parse_dandi_url() to detect zarr boundaries
• dandi/download.py — zarr_filters parameter on download(), filter threading through Downloader and _download_zarr(), skip deletion/checksum when filter active
• dandi/files/zarr.py — zarr_mode: Literal["full", "patch"] on iter_upload(), patch mode skips remote file deletion and client-side checksum verification
• dandi/upload.py — zarr_mode parameter on upload(), conditional pass-through for ZarrAsset
• dandi/cli/cmd_download.py — --zarr click option (multiple, OR logic)
• dandi/cli/cmd_upload.py — --zarr-mode click option
• dandi/cli/cmd_ls.py — list zarr entries when URL is AssetZarrEntryURL
• dandi/cli/tests/test_download.py — updated mock expectations for new zarr_filters parameter

Integration tests

• dandi/tests/test_download.py — 6 tests (glob filter, metadata alias, no-delete, path filter, nonexistent filter, sync conflict)
• dandi/tests/test_upload.py — 3 tests (patch no-delete, full delete, patch updates)
• dandi/tests/test_dandiarchive.py — 3 URL parsing cases + 8 split_zarr_location cases

Review checklist

Please review the design at doc/design/partial-zarr.md and comment on:

• --zarr TYPE:PATTERN syntax — is the filter approach right? Are glob/path/regex the right types?
• metadata alias expansion — does glob:**/.z* + glob:**/zarr.json + glob:**/.zmetadata cover all cases?
• --zarr-mode patch semantics — is "upload without deleting" the right default for patch? Should subtree cleanup happen?
• URL parsing — is AssetZarrEntryURL with zarr boundary detection the right approach?
• Checksum strategy — relying on per-file ETags for partial ops, deferring subtree checksums to future manifests
• Open questions in the doc (OR composition chosen, --sync conflict raises error, server-side glob deferred)
• Should legacy zarr-checksums/ files on S3 be cleaned up as part of this or separately?

TODO

• Implement dandi/zarr_filter.py — filter parsing and matching
• Implement AssetZarrEntryURL and split_zarr_location() in dandi/dandiarchive.py
• Add --zarr option to dandi download CLI
• Modify _download_zarr() for partial download support
• Add --zarr-mode option to dandi upload CLI
• Implement patch mode in iter_upload() (dandi/files/zarr.py)
• Thread zarr_mode through dandi/upload.py
• dandi ls zarr contents support
• Unit tests for zarr_filter.py (52 tests)
• Unit tests for URL parsing and split_zarr_location (11 test cases)
• Integration tests for download filtering (6 tests)
• Integration tests for upload patch mode (3 tests)
• Fix existing CLI mock test expectations
• Code review fixes (regex pre-compilation, Literal typing, ** collapse, redundant override removal)
• Run integration tests against docker-compose archive instance
• Coordinate with dandi-archive on manifest design (#2702) for future subtree checksum support

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 91.96217% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.29%. Comparing base (3f2b6d1) to head (4c24ad0).

Files with missing lines	Patch %	Lines
dandi/dandiarchive.py	64.28%	10 Missing ⚠️
dandi/download.py	85.07%	10 Missing ⚠️
dandi/upload.py	37.50%	5 Missing ⚠️
dandi/zarr_filter.py	95.31%	3 Missing ⚠️
dandi/cli/cmd_ls.py	60.00%	2 Missing ⚠️
dandi/tests/test_download.py	96.15%	2 Missing ⚠️
dandi/files/zarr.py	94.73%	1 Missing ⚠️
dandi/tests/test_upload.py	98.63%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1816      +/-   ##
==========================================
+ Coverage   76.88%   77.29%   +0.40%     
==========================================
  Files          88       90       +2     
  Lines       12901    13234     +333     
==========================================
+ Hits         9919    10229     +310     
- Misses       2982     3005      +23     

Flag	Coverage Δ
unittests	77.29% <91.96%> (+0.40%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kabilar]

Thanks @yarikoptic. An additional use case would be adding shards to an existing Zarr. For example, we have 355+ TB from a subject that were are stitching together into a single Zarr. We don't have enough space locally so will need to upload a shard, delete the local copy, and repeat the process.

cc @satra @ayendiki

[yarikoptic]

do you see currently proposed/implemented interface be sufficient for this use case?

[kabilar]

Yes, thanks. Should we add this as a use case in the design doc?

[yarikoptic]

we could -- just push it to this branch or send a PR against it if would like a review

[kabilar]

Suggested change

	## Primary Use Case
	From @kabilar (#1462):
	```bash
	# 1. Download just the metadata files from a zarr
	dandi download --zarr glob:'**/.z*' --zarr glob:'**/zarr.json' \
	dandi://dandi/001289/rawdata/.../PC.ome.zarr
	# 2. Edit .zattrs locally
	vim PC.ome.zarr/.zattrs
	# 3. Upload changes without deleting remote data chunks
	dandi upload --zarr-mode patch
	```
	## Use Cases
	### Use Case 1 - Update metadata
	1. Download just the metadata files from a Zarr.
	`dandi download --zarr glob:'**/.z*' --zarr glob:'**/zarr.json' \
	dandi://dandi/001289/rawdata/.../PC.ome.zarr`
	2. Edit metadata locally.
	`vim PC.ome.zarr/zarr.json`
	3. Upload changes without deleting remote data chunks.
	`dandi upload --zarr-mode patch`
	### Use Case 2 - Incrementally add shards to an existing Zarr
	We are generating a very large dataset for which there is not enough space to house the entire dataset locally.
	1. Add shard(s) locally to a Zarr that exists on DANDI.
	2. Upload shard(s) without deleting remote Zarr objects (that don't exist locally).
	3. Remove local shard(s) and repeat the process.

[kabilar]

Hi @yarikoptic, thanks for this pull request. I was able to add objects to an existing zarr, delete objects, and modify metadata (i.e. zarr.json).

I just have some minor feedback on the terminal output.

When I run dandi upload for a dandiset with a zarr that is the same between the local storage and dandi, the STATUS is done instead of skipped and the MESSAGE is exists - reuploading instead of file exists:

PATH                                              SIZE      ERRORS PROGRESS STATUS                         MESSAGE                  
dandiset.yaml                                     1.8 kB                    skipped                        should be edited online  
.bidsignore                                       13 Bytes    0             skipped                        file exists              
dataset_description.json                          404 Bytes   0             skipped                        file exists              
...atives/dandi-cli-1816-test/HG9_Z1_Y49.nii.zarr 5.9 GB      0             done                           exists - reuploading     
Summary:                                          5.9 GB                    3 skipped                      1 should be edited online
                                                                            1 done                         2 file exists            
                                                                                                           1 exists - reuploading 

And then when I modify the zarr locally and run dandi upload, the STATUS and MESSAGE are the same:

PATH                                             SIZE      ERRORS   PROGRESS STATUS                        MESSAGE                  
dandiset.yaml                                    1.8 kB                      skipped                       should be edited online  
.bidsignore                                      13 Bytes    0               skipped                       file exists              
dataset_description.json                         404 Bytes   0               skipped                       file exists              
...tives/dandi-cli-1816-test/HG9_Z1_Y49.nii.zarr 6.6 GB      0          100% done                          exists - reuploading     
Summary:                                         6.6 GB             3.3 MB/s 3 skipped                     1 should be edited online
                                                                             1 done                        2 file exists            
                                                                                                           1 exists - reuploading 

So I would propose the following changes to the terminal output:

1. If the zarr exists on DANDI and is the same as the local version, the STATUS should be skipped and MESSAGE should be file exists.
2. If the zarr exists on DANDI and is different than the local version, the MESSAGE should be more specific to what is happening to the zarr. For example:
   • file exists - uploading additional objects (2 GB)
   • file exists - deleting objects (2 GB)
   • file exists - modifying objects (2 GB)
   • Note that I have also added (2 GB) in the example to denote how many of the objects in the zarr are being uploaded/deleted/modified, since the SIZE is that of the entire Zarr tree.

[kabilar]

Just explicitly noting that dandi download --zarr metadata <path> and dandi upload --zarr-mode patch both work as intended.

For dandi download --zarr metadata dandi://dandi/001769@draft/derivatives/dandi-cli-1816-test/HG9_Z1_Y49.nii.zarr, this downloaded all the zarr.json files in the Zarr tree.

For dandi upload --zarr-mode patch, only modified local files were uploaded and remote files (that are absent locally) were not deleted.