Skip to content

Forward reduce_batch_size_fn in find_executable_batch_size decorator factory - #4166

Open
icn5381 wants to merge 1 commit into
huggingface:mainfrom
icn5381:fix-batch-size-reducer-forwarding
Open

Forward reduce_batch_size_fn in find_executable_batch_size decorator factory#4166
icn5381 wants to merge 1 commit into
huggingface:mainfrom
icn5381:fix-batch-size-reducer-forwarding

Conversation

@icn5381

@icn5381 icn5381 commented Aug 16, 2026

Copy link
Copy Markdown

Bug

find_executable_batch_size's parenthesized decorator form drops reduce_batch_size_fn when building its functools.partial, so a custom batch-size reducer is silently ignored and the default multiply-by-0.9 behavior always runs:

@find_executable_batch_size(starting_batch_size=128, reduce_batch_size_fn=my_reducer)
def train(batch_size):
    if batch_size > 10:
        raise RuntimeError("CUDA out of memory.")
    return batch_size

train()          # → 9, after iterating *0.9 twenty-odd times
# my_reducer was never called

Since the first positional parameter is the function itself, the factory form is the only way to pass a custom reducer — the parameter added in #3071 has been unusable, and it is documented in the docstring.

Fix

The functools.partial now forwards reduce_batch_size_fn alongside starting_batch_size.

Verification

  • New test_custom_reduce_batch_size_fn asserts the custom reducer is called and its return value is used: fails on main, passes with the fix.
  • tests/test_memory_utils.py: 8 passed.

…factory

The parenthesized decorator form dropped the reduce_batch_size_fn
argument when building its functools.partial, so a custom batch-size
reducer was silently ignored and the default multiply-by-0.9 behavior
was always used. The custom reducer is only reachable through this
factory form, which made the parameter unusable despite being
documented.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant