Skip to content

Fix stateful dataloader checkpointing across processes - #4165

Open
jrsmartin wants to merge 2 commits into
huggingface:mainfrom
jrsmartin:per-rank-dataloader-state
Open

Fix stateful dataloader checkpointing across processes#4165
jrsmartin wants to merge 2 commits into
huggingface:mainfrom
jrsmartin:per-rank-dataloader-state

Conversation

@jrsmartin

@jrsmartin jrsmartin commented Aug 14, 2026

Copy link
Copy Markdown

What does this PR do?

This saves stateful dataloader checkpoints per process during distributed training. Stateful dataloader can hold process-specific cursor, dataset, worker, and RNG state. Currently, every process currently writes its state to the same dl_state_dict.bin filename in a shared checkpoint directory, so the final file therefore contains whichever process wrote last, and other processes can restore the wrong data position.

Distributed checkpoints now use rank-qualified filenames such as dl_state_dict_rank0.bin. Loading prefers the current process's file and falls back to the legacy unqualified filename, preserving compatibility with existing checkpoints. Single-process checkpoint filenames remain unchanged.

Related context

I didn't find an existing issue or forum thread for this exact shared-filename collision but #3080 discusses a separate multi-worker/prefetch issue and a comment there describes manually saving dataloader state on every rank. Please note that this PR doesn't address the multi-worker issue in #3080.

Tests

  • Added a two-process regression using distinct per-rank dataset streams and a shared checkpoint directory. It verifies exact batch replay after restore and asserts that both rank-qualified state files exist.
  • Added a CPU/Gloo entry point so the distributed regression runs in the normal pull-request test suite as well as the existing multi-device suite. It skips when the optional torchdata dependency is unavailable.
  • Added unit coverage for single-process naming, multiple dataloaders, process-specific load precedence, and legacy fallback.
  • Verified the regression failed before the implementation and passes afterward.
  • make quality
  • pytest -q tests/test_state_checkpointing.py -k "not map_location" — 18 passed
  • pytest -q tests/test_accelerator.py -k stateful_dataloader — 17 passed
  • CPU/Gloo two-process regression — passed
  • Two-GPU distributed data-loop suite — passed

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed.

@jrsmartin
jrsmartin marked this pull request as ready for review August 14, 2026 20:59
@jrsmartin

Copy link
Copy Markdown
Author

@SunMarc would you mind reviewing this PR when you have a chance - thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant