Skip to content

Add docs for downloading and uploading a partial Zarr #240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kabilar wants to merge 1 commit into
base: master
Choose a base branch
from
+31 −0
Open

Add docs for downloading and uploading a partial Zarr #240

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/user-guide-sharing/uploading-data.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,19 @@ For command-line users or those with larger datasets, the DANDI CLI provides a p
dandi upload -i dandi-sandbox
```

#### Uploading part of a Zarr

By default, `dandi upload` synchronizes a [Zarr](https://zarr.dev/) asset in full, whereby files that exist on the server but not locally are deleted so the remote copy matches your local copy exactly.
When you only want to add or update a subset of a Zarr's files, use `--zarr-mode patch`:

```bash
dandi upload --zarr-mode patch
```

In patch mode, only modified or new local files are uploaded, and remote files that are absent locally are **not** deleted.
This is useful for editing metadata or adding to a large Zarr without needing a complete local copy.
The default, `--zarr-mode full`, performs the usual full synchronization (including deletions).

## Storing Access Credentials

There are two options for storing your DANDI access credentials:
Expand Down
18 changes: 18 additions & 0 deletions docs/user-guide-using/accessing-data/downloading.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,24 @@ For example:

dandi download --preserve-tree dandi://dandi/000026@draft/sub-I58/ses-Hip-CT/micr/sub-I58_sample-01_chunk-01_hipCT.json

### Download part of a Zarr
[Zarr](https://zarr.dev/) assets are directory trees that can contain many files, and you may need only a subset of them.
The `--zarr` option lets you filter which entries within a Zarr asset are downloaded, so you can avoid transferring the full tree.

The most common use is the `--zarr metadata` alias, which downloads only the Zarr metadata files (e.g. `zarr.json`, `.zarray`, `.zgroup`, `.zattrs`, `.zmetadata`) without any of the chunk data:

dandi download --zarr metadata dandi://dandi/001278@draft/sourcedata/raw/sub-Hb1/micr/sub-Hb1_sample-lefthemi_acq-overview_XPCT.ome.zarr

You can also filter using an explicit `--zarr TYPE:PATTERN` expression, where `TYPE` is one of `glob`, `path`, or `regex`:

| Type | Syntax | Description |
|------|--------|-------------|
| `glob:PATTERN` | `--zarr glob:'**/.z*'` | Download entries whose zarr-internal path matches the glob pattern, using `fnmatch` semantics per path component. `*` matches within a component, `**` matches across directory levels. |
| `path:PREFIX` | `--zarr path:0/1/2` | Download entries whose zarr-internal path starts with the prefix. |
| `regex:PATTERN` | `--zarr regex:'\.z(array|group|attrs)$'` | Download entries whose zarr-internal path matches the regular expression. |

The `--zarr` option may be given multiple times, in which case an entry is downloaded if it matches **any** of the filters.

## Using DataLad

All dandisets are regularly mirrored to DataLad datasets which are made available at the GitHub organization https://github.com/dandisets.
Expand Down
Loading