Skip to content

Dataloader output: dict instead of 4-tuple - #21

Merged
urancon merged 2 commits into
developfrom
refactor/dict-batch-api
Jun 2, 2026
Merged

Dataloader output: dict instead of 4-tuple#21
urancon merged 2 commits into
developfrom
refactor/dict-batch-api

Conversation

@urancon

@urancon urancon commented Jun 2, 2026

Copy link
Copy Markdown
Owner

What

NeuralDataset.__getitem__ and neural_collate now return a dict
({'stims', 'responses', 'valid_mask', 'stim_meta'}) instead of a positional
4-tuple.

Why

So future per-trial variables (behaviour, pupil size, running speed — as in the
Sinz-lab neuralpredictors / Sensorium datasets) can be added as extra keys
without changing the unpacking contract. neural_collate passes any unknown
per-item key straight through as a length-B list, so a dataset only has to add
the key to its __getitem__ dict:

def __getitem__(self, i):
    return {'stims': ..., 'responses': ..., 'valid_mask': ..., 'stim_meta': ...,
            'behav': self.behavior[i]}   # -> batch['behav'] for free

This is a hard break to positional unpacking, done pre-0.1.0 while there are no
published users.

Changed

  • Core: __getitem__ + neural_collate (library); Fitter (both loops).
  • Tests: all consumers + the shared _ToyDataset.__getitem__.
  • Docs: README quickstart, fitter.md, data_paradigm.md,
    metrics_paradigm.md; full rewrite of the stale README_datasets.md
    conventions page (it documented the long-gone (spectrogram, responses, ccmax, ttrc) tuple).
  • Notebooks: the 4 that unpack a batch manually (crcns_aa_tutorial,
    dataset_concatenation, explore_nat4, alice_eeg_tutorial). Source updated;
    a re-exec pass is folded into the pre-release notebook smoke test.

Verification

  • pytest -m "not slow" -> 440 passed (~17 s)
  • ns1 / downer / wingert slow integration files -> 87 passed
  • full suite -> 575 passed, 18 skipped (identical to the pre-refactor
    baseline — zero regressions)

🤖 Generated with Claude Code

urancon and others added 2 commits June 2, 2026 13:57
__getitem__ and neural_collate now return a dict with keys 'stims',
'responses', 'valid_mask', 'stim_meta' instead of a positional 4-tuple. This
lets datasets add extra per-trial keys later (e.g. 'behav' for behavioural
covariates, à la Sinz-lab neuralpredictors / Sensorium) without changing the
unpacking contract — neural_collate passes any unknown per-item key through
as a length-B list. Breaking change to positional unpacking; done pre-0.1.0
while there are no published users.

Updated the Fitter (both train/eval loops) + all test consumers. Fast loop
(pytest -m 'not slow') 440 passed; edited slow integration files (ns1/downer/
wingert) 87 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Updated all dataloader-unpacking examples to the dict API (batch['stims'],
batch['responses'], …): README quickstart, fitter.md, data_paradigm.md,
metrics_paradigm.md, and the 4 notebooks that unpack a batch manually
(crcns_aa_tutorial, dataset_concatenation, explore_nat4, alice_eeg_tutorial).
Also rewrote the stale README_datasets.md 'Conventions' page — it documented
the long-gone (spectrogram, responses, ccmax, ttrc) 4-tuple and select_neuron;
now describes the dict batch, (B,N,R,T) shapes, NaN sentinels, and the current
selection API, pointing to data_paradigm.md for the full contract.

Notebooks: source cells updated; outputs are data-identical (access pattern
only) but a re-exec pass is folded into the pre-release notebook smoke test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@urancon
urancon merged commit fa85719 into develop Jun 2, 2026
3 checks passed
@urancon
urancon deleted the refactor/dict-batch-api branch June 2, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant