You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Parallelize file metadata inspection in 2-stage reader #74
During the initialization of reader_2stage, metadata from all input files are currently inspected sequentially by rank 0. For datasets containing a large number of files, this can result in significant initialization time.
I propose distributing the metadata inspection across the available reader ranks, gathering the resulting metadata on rank 0, and restoring the original file ordering before constructing _file_time, _shape, and related attributes.
This does not modify the data-reading algorithm itself, only the initialization step.
Activity