Skip to content

Fix eval loss normalization in BPB evaluation - #18

Open
art-test-stack wants to merge 2 commits into
masterfrom
fix-eval-loss-normalization
Open

Fix eval loss normalization in BPB evaluation#18
art-test-stack wants to merge 2 commits into
masterfrom
fix-eval-loss-normalization

Conversation

@art-test-stack

Copy link
Copy Markdown
Owner

Summary

  • remove the extra batch-size division from the debug loss returned by compute_bpb
  • add a regression test using multiple batches and a batch size greater than one
  • keep the BPB calculation unchanged

Cause

loss2d.mean() already averages the per-token loss over both batch and sequence dimensions. The accumulated batch means were subsequently divided by steps * batch_size, so eval/loss was smaller than the actual mean cross-entropy by exactly the per-rank batch size.

The correct denominator for the existing accumulator is world_size * steps.

Test

The regression test evaluates a constant loss of 2.5 with batch size 4 and two evaluation steps. The old implementation reports 0.625; the corrected implementation reports 2.5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant