Skip to content

[Fix] Keep grouped token length estimates consistent - #1277

Open
primorLee wants to merge 1 commit into
OpenGVLab:mainfrom
primorLee:fix/1205-consistent-token-length
Open

[Fix] Keep grouped token length estimates consistent#1277
primorLee wants to merge 1 commit into
OpenGVLab:mainfrom
primorLee:fix/1205-consistent-token-length

Conversation

@primorLee

Copy link
Copy Markdown

Motivation

Fixes #1205.

When a conversation length was first encountered, the dataset appended the tokenizer-only length to self.length, but cached a larger value that also included the visual-token estimate. A later sample with the same string length therefore received a different estimate, making group_by_length order-dependent.

What changed

  • add the existing visual-token estimate to token_length before both appending and caching it
  • apply the same fix to the standard SFT, MPO, and pretraining loaders
  • keep the GPT-OSS SFT and MPO copies aligned
  • preserve the current approximate cache key and visual-token formula

Validation

  • python -m py_compile on all five changed training entrypoints
  • targeted pre-commit flake8 hook on all five files
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LazySupervisedDataset的token_length计算

1 participant