Enhancement - Fix preprocessing crash when num_workers=0 in Megatron GPT3 dataset generation#800

Open

polarG wants to merge 9 commits into

mainfrom

dev/hongtaozhang/fix-num_workers-usage

Commits on Apr 1, 2026

Fix the num_workers usage.
Hongtao Zhang
committed

Commits on Apr 27, 2026

Address PR review: derive --output-prefix from data_prefix, add tests
Hongtao Zhang
committed
Tighten test substring matches with trailing space
Hongtao Zhang
committed

Commits on May 2, 2026

Address PR review: use normalize_command and valid code_base in dataset generate test
Hongtao Zhang
committed

Commits on May 3, 2026

Address review feedback for num_workers / data_prefix handling
Hongtao Zhang
committed

Commits on May 15, 2026

Merge branch 'main' into dev/hongtaozhang/fix-num_workers-usage
polarG
authored
fix(megatron-gpt): address CI failures and Copilot review
Hongtao Zhang
committed
style: apply yapf 0.31.0 formatting (CI lint fix)
Hongtao Zhang
committed

Commits on May 27, 2026

Merge branch 'main' into dev/hongtaozhang/fix-num_workers-usage
polarG
authored