Skip to content

Set uploaded asset metadata schemaKey to "Asset"#1886

Open
candleindark wants to merge 3 commits into
dandi:masterfrom
candleindark:set-asset-schemakey-on-upload
Open

Set uploaded asset metadata schemaKey to "Asset"#1886
candleindark wants to merge 3 commits into
dandi:masterfrom
candleindark:set-asset-schemakey-on-upload

Conversation

@candleindark

@candleindark candleindark commented Jul 7, 2026

Copy link
Copy Markdown
Member

Summary

Pin schemaKey to "Asset" on every metadata dict written to an asset endpoint. This is the lockstep client change for dandi/dandi-schema#419, which changes BareAsset.schemaKey from "Asset" to "BareAsset".

Whatever the metadata was gathered as (e.g. a BareAsset), what is created on the server is an Asset. The archive stores schemaKey verbatim and does not normalize it, so once BareAsset.schemaKey becomes "BareAsset", BareAsset-derived metadata would be stored with the wrong schemaKey. Setting it at the point of upload keeps this correct regardless of how the metadata was gathered.

It is a no-op against the current dandi-schema release (where BareAsset.schemaKey is still "Asset"), so it can merge and release ahead of the dandi-schema change.

What changed

schemaKey is set to "Asset" at every sink that writes to an asset create/update endpoint:

  • LocalFileAsset.iter_upload and ZarrAsset.iter_upload (POST create / PUT replace), next to the existing metadata.setdefault("path", ...) step. Covers the normal CLI upload, the deprecated upload_raw_asset / iter_upload_raw_asset, and direct LocalAsset.upload() / iter_upload() API use.
  • RemoteBlobAsset.set_raw_metadata and RemoteZarrAsset.set_raw_metadata (PUT update). The reextract service command reaches these with BareAsset-derived metadata (nwb2asset returns a BareAsset) without going through iter_upload.

RemoteDandisetVersion.set_raw_metadata is left untouched: it writes Dandiset metadata, not an asset.

Test adaptations for the BareAsset.schemaKey change

Two metadata tests hard-coded the top-level schemaKey of a BareAsset as "Asset", which becomes "BareAsset" under dandi/dandi-schema#419. Both now track the default of the schemaKey field in BareAsset rather than a literal, so they pass against the current dandischema and #419 alike:

  • test_nwb2asset / test_nwb2asset_remote_asset: drop the redundant explicit schemaKey="Asset", which merely restated the field default.
  • test_prepare_metadata: set the expected schemaKey to the default of the schemaKey field in BareAsset before the comparison (the four metadata2asset*.json files now store a placeholder for it), and promote the object to an Asset before the validate() call that checks it against the Asset class.

Test plan

🤖 Generated with Claude Code

@candleindark candleindark added patch Increment the patch version when merged schema Issues relating to metadata schema labels Jul 7, 2026
@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.89%. Comparing base (3f2b6d1) to head (399da87).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1886      +/-   ##
==========================================
+ Coverage   76.88%   76.89%   +0.01%     
==========================================
  Files          88       88              
  Lines       12901    12909       +8     
==========================================
+ Hits         9919     9927       +8     
  Misses       2982     2982              
Flag Coverage Δ
unittests 76.89% <100.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Whatever the metadata was gathered as (e.g. a BareAsset), what is
created on the server is an Asset. The server stores schemaKey verbatim
and does not normalize it, so set it at the point of upload rather than
relying on the caller.

This matters because dandi-schema is changing BareAsset.schemaKey from
"Asset" to "BareAsset". Metadata gathered via a BareAsset then
serializes with schemaKey="BareAsset", so an asset created or updated
from it would be stored with the wrong schemaKey.

Pin it at every asset-endpoint write: iter_upload (LocalFileAsset,
ZarrAsset) and set_raw_metadata (RemoteBlobAsset, RemoteZarrAsset); the
latter is reached by the reextract command without going through
iter_upload.

A no-op against the current dandi-schema release (where
BareAsset.schemaKey is still "Asset"), so it can ship first.

Co-Authored-By: Claude Code 2.1.202 / Claude Opus 4.8 claude-opus-4-8 <noreply@anthropic.com>
`test_nwb2asset` and `test_nwb2asset_remote_asset` compared the
`nwb2asset()` result against a `BareAsset` built with an explicit
`schemaKey="Asset"`, which just restates the field default.
`model_construct()` fills defaults in, so dropping it is a no-op against
the current dandischema and ensures the equalities continue to hold once
dandi-schema#419 lands, where `BareAsset.schemaKey` becomes "BareAsset".
… value

`prepare_metadata()` returns a `BareAsset`, whose top-level `schemaKey` is
`"Asset"` with the current dandischema but `"BareAsset"` under
dandi/dandi-schema#419 (which pins `BareAsset.schemaKey` to its class
name). To keep this test passing against either dandischema version, the
expected `schemaKey` is set to the default of the `schemaKey` field in
`BareAsset` before the comparison, and the four `metadata2asset*.json`
files now store a placeholder for the field.

The later `validate()` call checks the data against the `Asset` class, so
`data_as_dict` is promoted to an Asset (`schemaKey = "Asset"`) beforehand.
@candleindark candleindark requested a review from yarikoptic July 8, 2026 21:45
@candleindark

Copy link
Copy Markdown
Member Author

@yarikoptic Please take a look at this PR. We will need this merged and released in order to have the integration tests in dandi/dandi-schema#419 to become green.

@candleindark candleindark marked this pull request as ready for review July 8, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

patch Increment the patch version when merged schema Issues relating to metadata schema

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant