Skip to content

Leave out a stream that cannot fill one batch instead of failing the epoch - #58

Merged
avgvi merged 1 commit into
mainfrom
fix/skip-short-streams
Oct 23, 2026
Merged

avgvi merged 1 commit into
mainfrom
fix/skip-short-streams

Conversation

@avgvi

@avgvi avgvi commented Oct 23, 2026

Copy link
Copy Markdown
Member

Under per_stream and device_per_stream batching, a stream that cannot fill a single batch raised ValueError, although the message itself said "Training will occur without any samples from this stream". The whole epoch failed because of one small stream.

Since #57, a stream with fewer samples than num_canonical_nodes no longer fails in the shuffle, but it then reached this check. With the default num_canonical_nodes (the number of nodes), such a stream puts one sample on each node. That fills a device batch only when the batch size is 1, so on more than one rank or with larger batches the epoch still failed. For example, a 1-sample stream next to two large ones, on 2 nodes × 2 ranks with batch size 4, or 2 nodes × 8 ranks with batch size 16.

This change:

  • per_stream: a stream that cannot fill one complete global batch is left out of the epoch, with a warning.
  • device_per_stream: a stream that cannot fill one device batch on a node is left out on that node, with a warning. Nodes with different batch counts are already padded to the same count, as before.
  • Neither method: if no stream can fill a batch (on a node, for device_per_stream), the epoch fails with an error that says so.

Tests:

  • test_stream_too_small_for_a_batch_is_skipped runs for both methods on 2 nodes × 2 ranks with batch size 4, with streams of 400, 300 and 1 samples. It checks for the warning and that the 1-sample stream does not appear. It also checks that the other two streams do appear, and that every device batch still comes from a single stream.
  • test_no_stream_fills_a_batch checks the new error.
  • All four cases fail on main. The existing batching tests pass.

…vice_per_stream batching, with a warning, instead of failing the epoch.
@avgvi
avgvi merged commit d448518 into main Oct 23, 2026
15 checks passed
@avgvi
avgvi deleted the fix/skip-short-streams branch October 23, 2026 15:02
@avgvi avgvi mentioned this pull request Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant