perf: audit recomputed its sort key on every comparison, not once per method - #31
Merged
sotashimozono merged 2 commits intoSep 9, 2026
Merged
Conversation
… method
Asked whether `@experimental` is really this expensive. It is not — the measurement that suggested
it was measuring something else.
The mark itself is free at call time. 10M calls of a numeric body, best of five, 36 threads:
plain 9.7 ms
marked 9.6 ms ratio 0.988, per-call overhead -0.01 ns
(The first attempt at this reported a ratio of 143808 because the result never escaped and the
optimiser deleted the whole loop. It needs a sink.)
What cost 0.74s was one line of `audit`, and it had nothing to do with marks:
sort!(out; by = mm -> (string(mm.name), string(mm.sig)))
`sort!(…; by = f)` calls `f` on **both sides of every comparison**. For this package's own 301
methods that is roughly five thousand `string(mm.sig)` calls instead of three hundred. Measured:
| | |
|---|---|
| iterate every `methods(f)` behind 1916 candidates — 11026 methods | 0.033s |
| build all 301 sort keys once | 0.039s |
| **the sort** | **0.601s** |
Computing the keys once and taking `sortperm` gives `audit(ExperimentalAPI)` **0.740s → 0.082s**,
with the same 301 own / 22 contributed / 22 unaccounted and the same order. `sortperm` is stable
where `sort!`'s default is not, and the keys are unique per method, so nothing depends on ties.
The same shape is in `record`'s two `sort!(hits; by = h -> (string(h.mod), string(h.name)))`. Left
alone, measured rather than assumed: at 100 entered marks the sort is 0.051 ms of a 1.17 ms
`record`, which is 4% of something already cheap.
Second, from the same investigation: `test_dogfood.jl` called `audit(ExperimentalAPI)` inside a
24-iteration loop. Hoisted.
test_dogfood.jl 18.8s → 3.2s (hoist) → 0.9s (both)
Suite wall time 110.6s → ~90s; the rest of the per-file movement is run-to-run noise on a shared
machine and should not be read as signal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
📚 Docs preview: https://codes.sota-shimozono.com/ExperimentalAPI.jl/previews/PR31/ (updates on each push to this PR) |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Follow-up measurement to the sort fix. What remains of `audit`'s cost is the search itself: Julia
indexes methods by generic, not by module, so "the methods `m` defines" means scanning every public
callable of every loaded module — 1916 candidates and 11026 methods behind them here.
That answer cannot change without a method being defined or deleted, and both bump the world
counter. So the result is cached under `(world, module)`, the same key `_public_generics` already
uses two functions above.
The key is argued from measurement rather than assumed, because the obvious version of the argument
is false: a `const` binding does **not** bump the world counter on 1.11.9 (it does on 1.12.2 and
1.14.0-DEV). It cannot change this answer either — a new `const` either aliases something whose
methods belong to another module, or creating it defined a method and bumped the counter — but
"world age covers every input" would have been the wrong reason.
audit(ExperimentalAPI), repeated in one world: 0.082s -> 0.0025s
Copy-on-write rather than `cache[m] = out`: the suite runs on four threads and a `Dict` is not safe
under concurrent `setindex!`. Swapping a freshly merged table in means the worst a race can cost is
a recomputation. Hammered with 200 concurrent calls across two modules: no corruption. The vector
is also copied out, so a caller that filters or empties it cannot poison the next one.
Honest about the size of it: across a full suite run the cache takes 31 hits to 36 misses, which is
about 2.5s of ~93s — below the run-to-run variance of this machine, and not visible in a wall-clock
total. The 32x is real for the pattern a user actually has, which is `audit` then `test_surface`
then `contributed_methods` on the same module.
One trap on the way in: putting the cache `const` between the docstring and `function own_methods`
detached the docstring onto the `const`, and `test_surface(ExperimentalAPI)` caught it — "every
public name has a docstring" failed on `own_methods`. The `const` now sits above the docstring.
1236 assertions, green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sotashimozono
deleted the
perf/audit-recomputed-its-sort-key-per-comparison
branch
September 9, 2026 06:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Asked whether
@experimentalis really this expensive. It is not — the measurement thatsuggested it was measuring something else.
The mark is free at call time
10M calls of a numeric body, best of five, 36 threads:
(My first attempt reported a ratio of 143808, because the result never escaped and the optimiser
deleted the loop. It needs a sink. Reporting that number would have been worse than useless.)
What actually cost 0.74s
One line of
audit, with nothing to do with marks:sort!(…; by = f)callsfon both sides of every comparison. For this package's own 301methods that is ~5000
string(mm.sig)calls instead of 301.methods(f)behind 1916 candidates — 11026 methodsComputing the keys once and taking
sortperm:audit(ExperimentalAPI): 0.740s → 0.082s (9×) — same 301 own / 22 contributed / 22unaccounted, same order.
sortpermis stable wheresort!'s default is not, and the keys areunique per method, so nothing depends on ties.
The same shape elsewhere, left alone — measured, not assumed
recordhas twosort!(hits; by = h -> (string(h.mod), string(h.name))). At 100 entered marks thesort is 0.051 ms of a 1.17 ms
record— 4% of something already cheap. Not worth the churn.And one test that called it in a loop
test_dogfood.jlcalledaudit(ExperimentalAPI)inside a 24-iteration loop.Suite wall time 110.6s → ~90s. The rest of the per-file movement is run-to-run noise on a shared
machine and should not be read as signal.
🤖 Generated with Claude Code