Skip to content

[Fix] Forward packed dataset mode to GPT-OSS preprocessors - #1276

Open
primorLee wants to merge 1 commit into
OpenGVLab:mainfrom
primorLee:fix/1166-packed-preprocessing
Open

[Fix] Forward packed dataset mode to GPT-OSS preprocessors#1276
primorLee wants to merge 1 commit into
OpenGVLab:mainfrom
primorLee:fix/1166-packed-preprocessing

Conversation

@primorLee

Copy link
Copy Markdown

Summary

  • forward use_packed_ds from LazySupervisedDataset to the selected preprocessor
  • cover single-image, multi-image, video, and pure-text supervised samples
  • add a regression test that checks every modality forwards the dataset's packed mode

Problem

For Qwen3-style SFT, get_preprocess_function() selects preprocess_internvl2_5. That preprocessor only disables per-sample max-length padding when group_by_length or use_packed_ds is true. Packed training intentionally does not rely on group_by_length, but the dataset did not forward self.use_packed_ds, so each item could be padded to tokenizer.model_max_length before reaching PackedDataset. This defeats variable-length packing and changes the effective token composition.

Validation

  • python -m pytest tests/test_gpt_oss_packed_preprocessing.py -q (1 test, 4 subtests passed)
  • python -m compileall -q ... on Windows Python 3.12
  • the same regression test and compile check in a clean Linux Python 3.10 container
  • the new test fails against the current main source because all four call sites omit the flag
  • Ruff passes for the new test; the modified legacy script retains the same seven pre-existing findings as main

Closes #1166

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] 微调internvl3.5_1B模型时,开启use_packed_ds但没有效果,num_samples均固定,且每条样本都达到最长token数量

1 participant