diff --git a/CHANGELOG.md b/CHANGELOG.md index 3f9652f2a..5aa25d66f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # PyNWB Changelog -## PyNWB 4.0.0 (Upcoming) +## PyNWB 4.0.0 (June 29, 2026) ### Removed - Removed functionality that was deprecated with a "will be removed in PyNWB 4.0" notice. @rly [#2210](https://github.com/NeurodataWithoutBorders/pynwb/issues/2210) @@ -12,14 +12,15 @@ - The `paths` argument of `pynwb.validate`. Use `path` and call `validate` once per file instead. - Made `NWBFile.icephys_filtering` read-only. Use `IntracellularElectrode.filtering` instead. The legacy `/general/intracellular_ephys/filtering` value is still read from older files. @rly [#2210](https://github.com/NeurodataWithoutBorders/pynwb/issues/2210) -### Documentation and tutorial enhancements -- Added a tutorial on using HERD to annotate an NWB file with external resources and store it at `/general/external_resources`, plus a companion example showing how to annotate multiple NWB files streamed from a DANDI dandiset with a single HERD. @rly, @mavaylon1 [#2200](https://github.com/NeurodataWithoutBorders/pynwb/pull/2200) -- Added `pandas.ExtensionArray` to `nitpick_ignore` so the Sphinx build does not fail on the unresolved cross-reference that HDMF's `array_data` docval macro renders for every type that accepts array data. @rly [#2209](https://github.com/NeurodataWithoutBorders/pynwb/pull/2209) -- Added `app.readthedocs.org/projects/pynwb/*` to `linkcheck_ignore` to stop the Sphinx linkcheck CI job from intermittently failing when GitHub Actions runners get throttled by readthedocs. @h-mayorquin [#2191](https://github.com/NeurodataWithoutBorders/pynwb/pull/2191) -- Added documentation for `ExternalImage` to the images tutorial. @h-mayorquin [#2159](https://github.com/NeurodataWithoutBorders/pynwb/pull/2159) -- Fixed broken and redirecting links in documentation. @bendichter [#2165](https://github.com/NeurodataWithoutBorders/pynwb/pull/2165) -- Added `EventsTable` examples to the NWB file basics and behavior tutorials. @rly [#2156](https://github.com/NeurodataWithoutBorders/pynwb/pull/2156) -- Added example of setting `Units.resolution` in the ecephys tutorial. @h-mayorquin [#2174](https://github.com/NeurodataWithoutBorders/pynwb/pull/2174) +### Changed +- Consolidated dependency declarations into `pyproject.toml` and removed the `requirements.txt`, `requirements-dev.txt`, `requirements-opt.txt`, `requirements-doc.txt`, and `requirements-min.txt` files. Added user-installable `zarr` and `termset` optional-dependency extras (e.g. `pip install pynwb[zarr]`), and declared development dependencies as PEP 735 `[dependency-groups]` (`test`, `stream`, `docs`). `tox` now installs dependencies via extras and dependency groups, with minimum-version testing using `uv pip install --resolution lowest-direct`. Install development dependencies with `pip install --group test --group docs -e ".[zarr,termset]"`. @rly [#2205](https://github.com/NeurodataWithoutBorders/pynwb/pull/2205) +- Deprecated `NWBGroupSpec.add_group` and `NWBGroupSpec.add_dataset`. Use `NWBGroupSpec.set_group`, `NWBGroupSpec.set_dataset`, or pass the group or dataset to the `NWBGroupSpec` constructor. @rly [#2138](https://github.com/NeurodataWithoutBorders/pynwb/issues/2138) +- Fixed `TimeSeries.get_timestamps()` to handle numpy array timestamps when they are set. @pauladkisson [#2181](https://github.com/NeurodataWithoutBorders/pynwb/pull/2181) +- Fixed `Units.waveform_rate` and `Units.waveform_unit` to also map to the `sampling_rate` and `unit` attributes of the `waveforms` column on write and read, so waveform sampling metadata round-trips for `Units` tables that contain only `waveforms` (without `waveform_mean` or `waveform_sd`). @ehennestad [#2183](https://github.com/NeurodataWithoutBorders/pynwb/pull/2183) +- Added Python 3.14 support. @bendichter, @rly [#2168](https://github.com/NeurodataWithoutBorders/pynwb/pull/2168) +- Bumped the minimum HDMF dependency to >=6.1.0 for pandas 3.0 compatibility. See the [HDMF changelog](https://hdmf.readthedocs.io/en/stable/CHANGELOG.html) for the full list of changes in HDMF 6.1.0. @rly [#2171](https://github.com/NeurodataWithoutBorders/pynwb/issues/2171), [#2208](https://github.com/NeurodataWithoutBorders/pynwb/pull/2208) +- Deprecated Python 3.9 support. (EOL was Oct 31, 2025) @bendichter [#2141](https://github.com/NeurodataWithoutBorders/pynwb/pull/2141) +- Deprecated `BehavioralEvents` and `AnnotationSeries` in favor of using an `EventsTable` in `NWBFile.events`. Creating a new instance of either type now emits a `UserWarning`; reading existing files containing these types continues to work without warnings. @rly [#2156](https://github.com/NeurodataWithoutBorders/pynwb/pull/2156) ### Added - Added optional `source_description` attribute to `EventsTable` for a short free-text label of where events originated (e.g., `"Acquisition system"`, `"Manual video review"`). Added `NWBFile.merge_events_tables()` to merge a list of `EventsTable` objects into a single DataFrame sorted by timestamp with a `source_events_table` column. Added `NWBFile.get_all_events()` to merge all tables in `NWBFile.events`. @rly [#2192](https://github.com/NeurodataWithoutBorders/pynwb/pull/2192) @@ -32,18 +33,19 @@ - Added pandas 3.0 compatibility. `NWBFile.add_scratch` now accepts a `pandas.Series` or a pandas `ExtensionArray` (e.g., `StringArray` and `ArrowStringArray`) as `data`, which is normalized to numpy when constructing `ScratchData`. This relies on the coercion added in [hdmf-dev/hdmf#1469](https://github.com/hdmf-dev/hdmf/pull/1469) (HDMF 6.1.0), which also makes the inherited PyArrow-backed string columns from pandas 3 DataFrames work across `TimeSeries` subclasses, `add_unit`, `add_electrode`, and `DynamicTable.from_dataframe`. The `pandas<3` cap has been lifted. @rly [#2208](https://github.com/NeurodataWithoutBorders/pynwb/pull/2208) ### Fixed +- Fixed ROS3 streaming and added an `aws_region` argument to `validate` so the AWS region can be passed through when validating a file opened with the `ros3` driver. HDF5 2.1.0 (h5py 3.16.0) requires the AWS region to be specified when opening an S3 URL with the `ros3` driver, so the ROS3 tests and the streaming tutorial now pass `aws_region="us-east-2"` (the region of the DANDI Archive S3 bucket). @rly [#2201](https://github.com/NeurodataWithoutBorders/pynwb/pull/2201) - Fixed reading legacy files where `Device.model` is a string containing `/` or `:` (e.g., `"MFC_200/250-0.66_40mm"`), which previously raised a `ValueError`. The string is now remapped to a read-only `DeviceModel` that preserves the original name, with a warning explaining that the file cannot be written or exported until a `DeviceModel` with a valid name is created. Writing or exporting such a `DeviceModel` raises a clear error instead of silently corrupting the file. @rly [#2186](https://github.com/NeurodataWithoutBorders/pynwb/pull/2186) - Fixed invalid CSS properties in documentation assistant toggle that prevented proper positioning on displays ≥1400px wide. @rly [#2151](https://github.com/NeurodataWithoutBorders/pynwb/pull/2151) -## Changed -- Consolidated dependency declarations into `pyproject.toml` and removed the `requirements.txt`, `requirements-dev.txt`, `requirements-opt.txt`, `requirements-doc.txt`, and `requirements-min.txt` files. Added user-installable `zarr` and `termset` optional-dependency extras (e.g. `pip install pynwb[zarr]`), and declared development dependencies as PEP 735 `[dependency-groups]` (`test`, `stream`, `docs`). `tox` now installs dependencies via extras and dependency groups, with minimum-version testing using `uv pip install --resolution lowest-direct`. Install development dependencies with `pip install --group test --group docs -e ".[zarr,termset]"`. @rly [#2205](https://github.com/NeurodataWithoutBorders/pynwb/pull/2205) -- Deprecated `NWBGroupSpec.add_group` and `NWBGroupSpec.add_dataset`. Use `NWBGroupSpec.set_group`, `NWBGroupSpec.set_dataset`, or pass the group or dataset to the `NWBGroupSpec` constructor. @rly [#2138](https://github.com/NeurodataWithoutBorders/pynwb/issues/2138) -- Fixed `TimeSeries.get_timestamps()` to handle numpy array timestamps when they are set. @pauladkisson [#2181](https://github.com/NeurodataWithoutBorders/pynwb/pull/2181) -- Fixed `Units.waveform_rate` and `Units.waveform_unit` to also map to the `sampling_rate` and `unit` attributes of the `waveforms` column on write and read, so waveform sampling metadata round-trips for `Units` tables that contain only `waveforms` (without `waveform_mean` or `waveform_sd`). @ehennestad [#2183](https://github.com/NeurodataWithoutBorders/pynwb/pull/2183) -- Added Python 3.14 support. @bendichter, @rly [#2168](https://github.com/NeurodataWithoutBorders/pynwb/pull/2168) -- Bumped the minimum HDMF dependency to >=6.1.0 for pandas 3.0 compatibility. See the [HDMF changelog](https://hdmf.readthedocs.io/en/stable/CHANGELOG.html) for the full list of changes in HDMF 6.1.0. @rly [#2171](https://github.com/NeurodataWithoutBorders/pynwb/issues/2171), [#2208](https://github.com/NeurodataWithoutBorders/pynwb/pull/2208) -- Deprecated Python 3.9 support. (EOL was Oct 31, 2025) @bendichter [#2141](https://github.com/NeurodataWithoutBorders/pynwb/pull/2141) -- Deprecated `BehavioralEvents` and `AnnotationSeries` in favor of using an `EventsTable` in `NWBFile.events`. Creating a new instance of either type now emits a `UserWarning`; reading existing files containing these types continues to work without warnings. @rly [#2156](https://github.com/NeurodataWithoutBorders/pynwb/pull/2156) +### Documentation and tutorial enhancements +- Added a tutorial on using HERD to annotate an NWB file with external resources and store it at `/general/external_resources`, plus a companion example showing how to annotate multiple NWB files streamed from a DANDI dandiset with a single HERD. @rly, @mavaylon1 [#2200](https://github.com/NeurodataWithoutBorders/pynwb/pull/2200) +- Added `pandas.ExtensionArray` to `nitpick_ignore` so the Sphinx build does not fail on the unresolved cross-reference that HDMF's `array_data` docval macro renders for every type that accepts array data. @rly [#2209](https://github.com/NeurodataWithoutBorders/pynwb/pull/2209) +- Added `app.readthedocs.org/projects/pynwb/*` to `linkcheck_ignore` to stop the Sphinx linkcheck CI job from intermittently failing when GitHub Actions runners get throttled by readthedocs. @h-mayorquin [#2191](https://github.com/NeurodataWithoutBorders/pynwb/pull/2191) +- Added documentation for `ExternalImage` to the images tutorial. @h-mayorquin [#2159](https://github.com/NeurodataWithoutBorders/pynwb/pull/2159) +- Fixed broken and redirecting links in documentation. @bendichter [#2165](https://github.com/NeurodataWithoutBorders/pynwb/pull/2165) +- Added `EventsTable` examples to the NWB file basics and behavior tutorials. @rly [#2156](https://github.com/NeurodataWithoutBorders/pynwb/pull/2156) +- Added example of setting `Units.resolution` in the ecephys tutorial. @h-mayorquin [#2174](https://github.com/NeurodataWithoutBorders/pynwb/pull/2174) + ## PyNWB 3.1.3 (December 9, 2025) diff --git a/Legal.txt b/Legal.txt index c16773cbb..fdcfe2d57 100644 --- a/Legal.txt +++ b/Legal.txt @@ -1,4 +1,4 @@ -“pynwb” Copyright (c) 2017-2025, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. +“pynwb” Copyright (c) 2017-2026, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Innovation & Partnerships Office at IPO@lbl.gov. diff --git a/README.rst b/README.rst index d23d0a08b..cd3f304b3 100644 --- a/README.rst +++ b/README.rst @@ -99,7 +99,7 @@ Citing NWB LICENSE ======= -"pynwb" Copyright (c) 2017-2025, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. +"pynwb" Copyright (c) 2017-2026, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: (1) Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. @@ -115,7 +115,7 @@ You are under no obligation whatsoever to provide any bug fixes, patches, or upg COPYRIGHT ========= -"pynwb" Copyright (c) 2017-2025, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. +"pynwb" Copyright (c) 2017-2026, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Innovation & Partnerships Office at IPO@lbl.gov. NOTICE. This Software was developed under funding from the U.S. Department of Energy and the U.S. Government consequently retains certain rights. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, distribute copies to the public, prepare derivative works, and perform publicly and display publicly, and to permit other to do so. diff --git a/docs/gallery/advanced_io/streaming.py b/docs/gallery/advanced_io/streaming.py index 428ebe300..47d1dc940 100644 --- a/docs/gallery/advanced_io/streaming.py +++ b/docs/gallery/advanced_io/streaming.py @@ -173,19 +173,20 @@ # ------------------------ # ROS3 stands for "read only S3" and is a driver created by the HDF5 Group that allows HDF5 to read HDF5 files stored # remotely in s3 buckets. Using this method requires that your HDF5 library is installed with the ROS3 driver enabled. -# With ROS3 support enabled in h5py, we can instantiate a :py:class:`~pynwb.NWBHDF5IO` object with the S3 URL and -# specify the driver as "ros3". Like the other methods, you can use a context manager to open the file and close it, +# With ROS3 support enabled in h5py, we can instantiate a :py:class:`~pynwb.NWBHDF5IO` object with the S3 URL, +# specify the driver as "ros3", and pass the AWS region of the S3 bucket as ``aws_region`` (DANDI data are stored +# in ``us-east-2``). Like the other methods, you can use a context manager to open the file and close it, # or open the file and close it manually. from pynwb import NWBHDF5IO # open with context manager -with NWBHDF5IO(s3_url, mode='r', driver='ros3') as io: +with NWBHDF5IO(s3_url, mode='r', driver='ros3', aws_region="us-east-2") as io: nwbfile = io.read() streamed_data = nwbfile.acquisition['lick_times'].time_series['lick_left_times'].data[:] # open and close manually -io = NWBHDF5IO(s3_url, mode='r', driver='ros3') +io = NWBHDF5IO(s3_url, mode='r', driver='ros3', aws_region="us-east-2") nwbfile = io.read() streamed_data = nwbfile.acquisition['lick_times'].time_series['lick_left_times'].data[:] io.close() diff --git a/docs/gallery/general/plot_external_resources.py b/docs/gallery/general/plot_external_resources.py index b1a659f6d..819c8369c 100644 --- a/docs/gallery/general/plot_external_resources.py +++ b/docs/gallery/general/plot_external_resources.py @@ -51,9 +51,9 @@ # ------------------- # Use :py:meth:`~pynwb.file.NWBFile.get_external_resources` to get the file's # :py:class:`~pynwb.resources.HERD`. A file has at most one HERD, so this returns the existing HERD if -# the file already has one (for example, when the file was read from disk) and otherwise, it creates -# and attaches a new empty HERD. The :py:attr:`~pynwb.file.NWBFile.external_resources` attribute returns the -# HERD without creating one, returning ``None`` when the file has no external resources. +# the file already has one (for example, when the file was read from disk) and creates and attaches a +# new empty HERD otherwise. The :py:attr:`~pynwb.file.NWBFile.external_resources` attribute returns +# the HERD without creating one, returning ``None`` when the file has no external resources. herd = nwbfile.get_external_resources() diff --git a/docs/gallery/general/resources_streaming.py b/docs/gallery/general/resources_streaming.py index 17ba33b07..76262932f 100644 --- a/docs/gallery/general/resources_streaming.py +++ b/docs/gallery/general/resources_streaming.py @@ -22,11 +22,11 @@ .. note:: This example reads data over the network and is not run when the documentation is built. To run - it yourself, install the ``dandi`` and ``fsspec`` packages: + it yourself, install the streaming dependencies: .. code-block:: bash - pip install dandi fsspec aiohttp requests + pip install dandi fsspec tqdm aiohttp requests """ # sphinx_gallery_thumbnail_path = 'figures/gallery_thumbnails_streaming_external_resources.png' diff --git a/docs/source/conf.py b/docs/source/conf.py index 190ce5a91..1b07418e3 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -212,7 +212,7 @@ def __call__(self, filename): # General information about the project. project = u'PyNWB' -copyright = u'2017-2025, Neurodata Without Borders' +copyright = u'2017-2026, Neurodata Without Borders' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the diff --git a/environment-ros3.yml b/environment-ros3.yml index c10ad9bd7..348d6d25b 100644 --- a/environment-ros3.yml +++ b/environment-ros3.yml @@ -5,18 +5,18 @@ channels: - defaults dependencies: - python==3.14 - - h5py==3.15.1 + - h5py==3.16.0 - hdmf==6.0.2 - - matplotlib==3.10.8 - - numpy==2.4.2 + - matplotlib==3.11.0 + - numpy==2.4.3 - pandas==2.3.3 - - python-dateutil==2.9.0 + - python-dateutil==2.9.0.post0 - setuptools - - pytest==9.0.2 - - fsspec==2026.1.0 - - requests==2.32.5 - - aiohttp==3.13.3 + - pytest==9.0.3 + - fsspec==2026.6.0 + - requests==2.34.2 + - aiohttp==3.14.1 - pip - pip: - remfile==0.1.13 - - dandi==0.74.3 # NOTE: dandi is not available on conda for osx-arm64 + - dandi==0.76.4 # NOTE: dandi is not available on conda for osx-arm64 diff --git a/license.txt b/license.txt index c6d771edb..2d3b5b239 100644 --- a/license.txt +++ b/license.txt @@ -1,4 +1,4 @@ -“pynwb” Copyright (c) 2017-2025, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. +“pynwb” Copyright (c) 2017-2026, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: diff --git a/src/pynwb/file.py b/src/pynwb/file.py index e04ad3e19..a2aa9d4ed 100644 --- a/src/pynwb/file.py +++ b/src/pynwb/file.py @@ -1101,10 +1101,12 @@ def get_scratch(self, **kwargs): def get_external_resources(self, **kwargs): """Get the HERD external resources for this NWBFile. - Create an empty HERD and attach it to the file if the file does not have one yet. There is at - most one HERD per file, so this returns the existing HERD when the file already has one, for - example when the file was read from disk. The NWBFile.external_resources attribute returns the - HERD without creating one, returning None when the file has no external resources. + With the default ``linked=False``, return the HERD stored on this file, first creating and + attaching a new empty HERD if the file does not have one yet (there is at most one HERD per + file, so an existing HERD is returned as-is, e.g. when the file was read from disk). With + ``linked=True``, return the linked HERD set via link_resources, or None if none is set, + without creating one. The NWBFile.external_resources attribute returns the stored HERD without + creating one, returning None when the file has no external resources. """ linked = getargs('linked', kwargs) if linked: diff --git a/src/pynwb/validation.py b/src/pynwb/validation.py index 1400b0a81..4c177623a 100644 --- a/src/pynwb/validation.py +++ b/src/pynwb/validation.py @@ -125,7 +125,13 @@ def get_cached_namespaces_to_validate(path: Optional[str] = None, "type": str, "doc": "Driver for h5py to use when opening the HDF5 file.", "default": None, - }, + }, + { + "name": "aws_region", + "type": str, + "doc": "AWS region to use when opening the HDF5 file with the ros3 driver.", + "default": None, + }, returns="Validation errors in the file.", rtype=list, is_method=False, @@ -139,19 +145,20 @@ def validate(**kwargs): compliance with the schema and compliance of data with NWB best practices. """ - io, path, use_cached_namespaces, namespace, verbose, driver = getargs( - "io", "path", "use_cached_namespaces", "namespace", "verbose", "driver", kwargs + io, path, use_cached_namespaces, namespace, verbose, driver, aws_region = getargs( + "io", "path", "use_cached_namespaces", "namespace", "verbose", "driver", "aws_region", kwargs ) assert io != path, "Both 'io' and 'path' were specified! Please choose only one." path = str(path) if isinstance(path, Path) else path # get namespaces to validate namespace_message = "PyNWB namespace information" - io_kwargs = dict(path=path, mode="r", driver=driver) - + io_kwargs = dict(path=path, mode="r", driver=driver, aws_region=aws_region) + if use_cached_namespaces: - cached_namespaces, manager, namespace_dependencies = get_cached_namespaces_to_validate(path=path, - driver=driver, + cached_namespaces, manager, namespace_dependencies = get_cached_namespaces_to_validate(path=path, + driver=driver, + aws_region=aws_region, io=io) io_kwargs.update(manager=manager) diff --git a/tests/integration/ros3/test_ros3.py b/tests/integration/ros3/test_ros3.py index 666647ddf..7703be02c 100644 --- a/tests/integration/ros3/test_ros3.py +++ b/tests/integration/ros3/test_ros3.py @@ -30,14 +30,14 @@ def setUp(self): def test_read(self): s3_path = 'https://dandiarchive.s3.amazonaws.com/ros3test.nwb' with warnings.catch_warnings(): - with NWBHDF5IO(s3_path, mode='r', driver='ros3') as io: + with NWBHDF5IO(s3_path, mode='r', driver='ros3', aws_region="us-east-2") as io: nwbfile = io.read() test_data = nwbfile.acquisition['ts_name'].data[:] self.assertEqual(len(test_data), 3) def test_dandi_read(self): with warnings.catch_warnings(): - with NWBHDF5IO(path=self.s3_test_path, mode='r', driver='ros3') as io: + with NWBHDF5IO(path=self.s3_test_path, mode='r', driver='ros3', aws_region="us-east-2") as io: nwbfile = io.read() test_data = nwbfile.acquisition['TestData'].data[:] self.assertEqual(len(test_data), 3) @@ -76,13 +76,13 @@ def test_dandi_get_cached_namespaces(self): } } found_namespaces, _, found_namespace_dependencies = get_cached_namespaces_to_validate( - path=self.s3_test_path, driver="ros3" + path=self.s3_test_path, driver="ros3", aws_region="us-east-2" ) self.assertCountEqual(first=found_namespaces, second=expected_namespaces) self.assertDictEqual(d1=found_namespace_dependencies, d2=expected_namespace_dependencies) def test_dandi_validate(self): - result = validate(path=self.s3_test_path, driver="ros3") + result = validate(path=self.s3_test_path, driver="ros3", aws_region="us-east-2") assert result == []