Feature/hdf5 integration#20
Conversation
- added HDF5ImageDataset class to schedule, I/O, and process during batch inference - updated package dependency to include `h5py` pkg
- added scripts to convert webdataset into HDF5 file storage - added `webdataset` as the optional dependency
|
@thompsonmj, a variation on what you have for parquet to HDF5, but instead pulling from webdataset format. Should be merged before #16, which has pulled updates from here. |
|
@thompsonmj converting this to draft for a while as I'll be working on making HDF5 file available in the batch embed modules. |
- integrated HDF5 as image data source input option for batch embedding scripts - added SLURM job scripts + configs template for batch embed HDF5 tasks - modified documentation index page to reflect the HDF5 integration update
… refs - Add HDF5ImageDataset to top-level package exports for consistent API - Add conversion feature check to print_installation_guide() - Remove references to nonexistent HDF5Writer class in config and SLURM templates - Replace emoji symbols with plain text in installation guide output Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The conversion tool is a one-time data prep task, not an inference concern. Removing it keeps the package focused on its core mission of GPU saturation during inference. HDF5ImageDataset (the inference input format) is unaffected. Removed: - src/hpc_inference/utils/wds_to_hdf5.py - scripts/conversion/wds_to_hdf5.slurm - [conversion] optional dependency group (webdataset) - Conversion feature check from print_installation_guide() - Conversion references from config/SLURM templates Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
thompsonmj
left a comment
thompsonmj
left a comment
There was a problem hiding this comment.
Couple of quick-inline comments; probably ready for approval afterwards.
| try: | ||
| # Decode from bytes | ||
| img = Image.open(io.BytesIO(image_bytes)) | ||
|
|
||
| # Convert to desired color mode | ||
| if img.mode != self.color_mode: | ||
| img = img.convert(self.color_mode) | ||
|
|
||
| # Validate if requested |
There was a problem hiding this comment.
Probably good to put this before conversion so when validation is requested it happens before working on the file.
There was a problem hiding this comment.
Good catch, I'll move img.verify() before conversion
|
|
||
| class HDF5ImageDataset(IterableDataset): | ||
| """ | ||
| Loads images from HDF5 files created by HDF5Writer in a streaming fashion. |
There was a problem hiding this comment.
This has been addressed in the recent modifications. No HDF5Writer
There was a problem hiding this comment.
@thompsonmj was this comment from previous unsubmitted review? It says it is from today, so I was not sure. But it should be addressed in
a6b4079
There was a problem hiding this comment.
Right, so I think the docstrings just need updated to reflect that it is no longer with us 🪦
| "${OUTPUT_DIR}" \ | ||
| --input_type hdf5 \ | ||
| --config "${CONFIG_FILE}" \ | ||
| --file_list "${FILE_LIST}" # [OPTIONAL] Use this if you have a specific list of files to process |
There was a problem hiding this comment.
I think if this is optional, it could be commented out by default (along with FILE_LIST definition above). As-is, I think it will require a user to comment out manually or else "/path/to/your/file_list.txt" will be passed to open_clip_embed.py causing an error.
Relatedly, looks like this is in open_clip_embed_parquet_template.slurm as well here.
| help="Type of input data: 'images' for image directory, 'parquet' for Parquet files, 'hdf5' for HDF5 files") | ||
| parser.add_argument("--config", type=str, default=None, help="Path to YAML config file (optional)") | ||
| parser.add_argument("--file_list", type=str, default=None, | ||
| help="File containing list of Parquet files to process (only for --input_type parquet)") |
There was a problem hiding this comment.
Update file types similarly to the above help strings.
egrace479
left a comment
egrace479
left a comment
There was a problem hiding this comment.
One suggestion for documentation
- Move image validation before color conversion in HDF5ImageDataset so corrupted data is caught early (thompsonmj) - Comment out optional --file_list in HDF5 and Parquet SLURM templates to prevent placeholder path errors (thompsonmj) - Update --file_list help string to include HDF5 file types (thompsonmj) - Make templates a linkable header in docs/index.md (egrace479) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3 participants
HDF5 dataset support for batch inference
HDF5ImageDatasetclass inhdf5_dataset.py, which enables distributed processing of images stored in HDF5 files; supports rank-based and worker-based file partitioning similar toImageFolderDatasetandParquetImageDatasetHDF5ImageDatasetin thedatasetspackage's__init__.pyand included it in the module exports.Dataset conversion workflow
Added script
wds_to_hdf5to automate conversion ofWebDatasetshards to HDF5 formatDependency updates
h5pyas a required dependency for HDF5 file support and introduced an optionalconversiondependency group withwebdatasetfor dataset conversion.allfeature set to include both inference and conversion dependencies.