Skip to content

feat: warn on pathologically slow custom loss functions - #1301

Open
MilesCranmerBot wants to merge 2 commits into
astroautomata:masterfrom
MilesCranmerBot:bot/loss-speed-warning
Open

feat: warn on pathologically slow custom loss functions#1301
MilesCranmerBot wants to merge 2 commits into
astroautomata:masterfrom
MilesCranmerBot:bot/loss-speed-warning

Conversation

@MilesCranmerBot

Copy link
Copy Markdown
Contributor

Summary

When a user passes a custom loss (elementwise_loss, loss_function, or loss_function_expression), PySR now benchmarks it at the start of fit and warns if it is pathologically slow, using the built-in default squared-error loss as a yardstick:

  • The benchmark runs inside the user's own Julia process on their real data (same arrays/options the search will use): one warm-up call to trigger JIT, GC.gc(), then the median of 5 @elapsed passes. The default squared-error loss is timed identically in the same process, so the ratio (custom/default) is hardware/OS independent.
  • A warning is emitted — at most once per fit — only if the custom loss is both ≥50x slower than the default and takes ≥1 ms per pass over the data. The absolute floor means a fast loss that is merely "50x slower than nothing" never fires.
  • The warning is user-actionable: it prints both measured times, the likely causes (excessive allocations, Python callbacks via PythonCall, type instability — see @code_warntype), and the opt-out flag.
  • Opt out with check_loss_speed=False (on by default only when a custom loss is provided; there is zero overhead and no side effects when the default loss is used).
  • If the benchmark itself fails for any reason, it degrades silently — it can never break a user's fit.
  • v1 scope: losses only; custom operators are not benchmarked.

Full-objective losses (loss_function / loss_function_expression) are benchmarked with a constant tree on a real Dataset built from the user's data, and the baseline is eval_loss under default options (the user's options would just re-time their own loss). Elementwise losses are benchmarked with the same element-by-element call pattern the search uses.

Example warning:

UserWarning: Your custom loss (`elementwise_loss`) took 0.00654 s per pass over
your dataset, which is ~217965x slower than the default squared-error loss
(3e-08 s per pass), measured on the same data in the same Julia process. A loss
this slow will likely dominate the total search time. Common causes include:
excessive memory allocation inside the loss; calls back into Python from Julia
(e.g., via PythonCall); and type instability (you can check this with
`@code_warntype` on your loss). If this slowdown is expected, you can disable
this check with `PySRRegressor(..., check_loss_speed=False)`.

Testing

  • TestLossSpeedWarning (new): warning fires on an allocation-heavy elementwise_loss (plain and weighted) and on a slow loss_function; no warning with the default loss, with a fast custom loss, or with check_loss_speed=False; a failing benchmark degrades silently. All fits use 1 trivial iteration so the tests stay fast.
  • Full local runs: test_main.py (114 passed, 1 skipped), test_cli.py (6 passed), loss-related test_startup.py tests (notebook test needs the nbval plugin, skipped locally).
  • black, isort, and mypy (CI invocation) are clean.

MilesCranmerBot and others added 2 commits August 19, 2026 14:20
When a user passes a custom loss (`elementwise_loss`, `loss_function`, or
`loss_function_expression`), benchmark it inside their own Julia process on
their real data against the default squared-error loss, timed identically in
the same process so the ratio is hardware-independent (one warm-up call for
JIT, GC, then the median of 5 timed passes). Warn — at most once per fit —
only if the custom loss is both >=50x slower than the default and takes
>=1 ms per pass over the data, so a fast loss that is merely '50x slower
than nothing' never fires. The warning prints both measured times, likely
causes (excessive allocations, Python callbacks via PythonCall, type
instability — see `@code_warntype`), and the opt-out flag. The check is
opted out with `check_loss_speed=False`, degrades silently if the
benchmark itself fails, and adds zero overhead when no custom loss is used.
Custom operators are out of scope for this check.

Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
Custom losses are Julia strings evaluated in-process; Python callbacks
are not a reachable failure mode.

Co-authored-by: Miles Cranmer <miles.cranmer@gmail.com>
@codecov

codecov Bot commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.67442% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pysr/sr.py 97.67% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant