dj-photon-codecs

DataJoint codec for photon-limited movies with Anscombe variance stabilization and compressed Zarr storage.

Overview

This codec enables efficient storage of photon-limited imaging data (calcium imaging, fluorescence microscopy) by:

1. Variance stabilization - Anscombe transformation converts Poisson noise to constant variance
2. High compression - Blosc/Zstd achieves 3-5x compression on stabilized data
3. Efficient access - Zarr format with temporal chunking for frame-by-frame processing

Quick Start

Installation

pip install dj-photon-codecs

Basic Usage

import datajoint as dj
import numpy as np

# Configure object storage
dj.config['stores'] = {
    'imaging': {
        'protocol': 's3',
        'bucket': 'my-data',
    }
}

# Define table
schema = dj.Schema('calcium_imaging')

@schema
class Recording(dj.Manual):
    definition = """
    recording_id : int32
    ---
    movie : <photon@imaging>  # Photon-limited movie
    """

# Insert raw photon counts
movie = np.random.poisson(lam=10, size=(1000, 512, 512))
Recording.insert1({'recording_id': 1, 'movie': movie})

# Fetch returns Zarr array (Anscombe-transformed)
zarr_array = (Recording & {'recording_id': 1}).fetch1('movie')
frame = zarr_array[100]  # Efficient frame access

Apply Inverse Transform

from anscombe import generalized_inverse_anscombe

# Recover original photon counts
original = generalized_inverse_anscombe(zarr_array[:])

Features

• Automatic variance stabilization using Anscombe transformation
• 3-5x compression with Blosc/Zstd on variance-stabilized data
• Temporal chunking optimized for sequential frame access
• Schema-addressed paths mirroring database structure
• Lazy loading - access frames without loading entire movie
• Invertible transformation - mathematically lossless

When to Use

✅ Use for:

• Photon-limited imaging (calcium imaging, fluorescence microscopy)
• Data where Poisson shot noise dominates
• Large movies requiring efficient storage
• Sequential frame processing workflows

❌ Don't use for:

• Preprocessed/normalized data (ΔF/F, z-scored)
• Data with negative values
• Non-Poisson noise dominates

Documentation

📚 Full Documentation

• Installation Guide
• Quick Start Tutorial
• User Guide
• API Reference
• Examples

Contributing

Contributions are welcome! See our Contributing Guide.

# Development setup
git clone https://github.com/datajoint/dj-photon-codecs.git
cd dj-photon-codecs
pip install -e ".[dev]"

# Run tests
pytest

# Build docs
mkdocs serve

Support

• 📖 Documentation
• 💬 GitHub Discussions - Questions and community
• 🐛 GitHub Issues - Bug reports and feature requests

Related Projects

• DataJoint - Scientific data pipeline framework
• anscombe-transform - Variance stabilization library
• dj-zarr-codecs - General Zarr array codec
• Zarr - Chunked, compressed array storage

License

Apache License 2.0. Copyright (c) 2026 DataJoint Inc.