Skip to content

Commit 4691f47

Browse files
committed
docs(bench): the split-tree cold 'regression' was noise — corrected at n=3
n=1 时分离式的 cold 读作 `29.13s → 35.88s`(新版慢 23%)。n=3 复测是 `30.33s → 29.78s` —— 新版反而略快。那一对单样本只是恰好把新臂取在了旧臂的 最大值附近:旧臂的离散度是 **19.1%**(29.92–35.72),差一点点没到 §4a R2 判定 「噪声」的 20%;新臂是 4.7%。 这正是 R2 存在的意义,也是为什么其余每一行都老老实实标 n=1,而不是装作不知道。 差一点就把一个不存在的回归写进 README 了。
1 parent f5af236 commit 4691f47

2 files changed

Lines changed: 67 additions & 60 deletions

File tree

README.md

Lines changed: 59 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -315,74 +315,74 @@ same compiler binary**, by a harness that lives in this repository
315315
### A real project: building mcpp itself
316316

317317
**137 module interface units, 57k lines, every one of them `import std;`**
318-
measured in place, four engines, the same hermetic `gcc 16.1.0` binary handed to
319-
each. Median wall-clock and the ratio to cmake; **lower is better**.
320-
321-
| scenario | what it asks | mcpp | cmake | xmake |
322-
|---|---|---|---|---|
323-
| `cold` | everything, from nothing | **82.87s** · 0.88x | 94.53s · 1.00x | 94.63s · 1.00x |
324-
| `noop` | how cheap is "already up to date" | **0.20s** · 0.58x | 0.34s · 1.00x | 0.38s · 1.10x |
325-
| `touch-leaf` | mtime bump on a unit nobody imports | **2.14s** · 0.12x | 18.06s · 1.00x | 18.47s · 1.02x |
326-
| `edit-body` | real edit inside a function body | **18.29s** · 0.93x | 19.64s · 1.00x | 19.97s · 1.02x |
327-
| `edit-comment` | a comment added to a widely-imported unit | **0.46s** · 0.01x | 85.03s · 1.00x | 84.69s · 1.00x |
328-
| `touch-hub` | mtime bump on a hub, content unchanged | **0.44s** · 0.01x | 84.53s · 1.00x | 83.65s · 0.99x |
329-
330-
**Read the `cold` row first.** On a full build all three engines are within 15%
331-
of each other, and that is not a disappointment — it is the correct answer.
332-
mcpp's cold build is **100% critical path** (79.7s of a 79.8s makespan): every
333-
engine walks the same 26-deep chain of module interfaces, so no amount of
334-
scheduling or cores can help. Anyone quoting a synthetic fixture's `0.26x` as a
335-
cold-build advantage is quoting an artefact of a workload whose units cost 0.09s.
318+
a pinned checkout, measured in place, with the same hermetic `gcc 16.1.0` binary
319+
handed to every engine. Median wall-clock and the ratio to cmake; **lower is
320+
better**.
321+
322+
| scenario | what it asks | mcpp | mcpp `+bmi_schedule` | cmake | xmake |
323+
|---|---|---|---|---|---|
324+
| `cold` | everything, from nothing | 79.54s · 0.86x | **35.43s · 0.38x** | 92.33s · 1.00x | 90.30s · 0.98x |
325+
| `noop` | how cheap is "already up to date" | **0.16s** · 0.57x | 0.16s · 0.57x | 0.28s · 1.00x | 0.38s · 1.36x |
326+
| `touch-hub` | mtime bump on a hub, content unchanged | **0.40s** · 0.005x | **0.22s** · 0.003x | 83.39s · 1.00x | 82.07s · 0.98x |
327+
| `edit-body` | real edit inside a function body | 76.24s · 0.89x | **30.17s · 0.35x** | 85.64s · 1.00x | 84.61s · 0.99x |
328+
329+
**Read the `cold` row first.** On a full build all three engines land within 15%
330+
of each other, and that is the correct answer, not a disappointment: mcpp's cold
331+
build is **100% critical path** (79.7s of a 79.8s makespan, average parallelism
332+
3.94 of 32 hardware threads). Every engine walks the same 26-deep chain of module
333+
interfaces, and scheduling cannot shorten a chain. Anyone quoting a synthetic
334+
fixture's `0.26x` as a cold-build advantage is quoting an artefact of a workload
335+
whose units cost 0.09s each.
336+
337+
The cold-build lever is **`[build] bmi_schedule = "on"`** — publish each module's
338+
BMI as soon as it exists and move code generation onto its own edge, so importers
339+
stop waiting for work they do not need. 79.54s → 35.43s. It is opt-in until it
340+
has been verified on every platform.
336341

337342
**The gap is in the loop you actually spend the day in.** Touching a hub
338-
interface costs cmake and xmake a full 84-second downstream rebuild, because
339-
they decide by timestamp. mcpp compares the BMI the compiler just produced
340-
against the previous one and, when they are equivalent, puts the old file back
341-
so ninja's `restat` sees no change — the 46 importers never rebuild. That is
342-
**0.44s against 84.53s**.
343+
interface costs cmake and xmake a full 83-second downstream rebuild, because they
344+
decide by timestamp. mcpp compares the BMI the compiler just produced against the
345+
previous one and, when they are equivalent, puts the old file back so ninja's
346+
`restat` sees no change — the importers never rebuild. **0.40s against 83.39s.**
343347

344-
`edit-body` is the control that keeps this honest: there the interface really
345-
did change, no engine should be fast, and none is (0.93x).
348+
`edit-body` is the control that keeps this honest: there the interface really did
349+
change, no engine should be fast, and none is.
346350

347351
### The same question on someone else's codebase
348352

349-
mcpp measuring its own build proves nothing on its own — an optimisation can be
350-
an artefact of one project's module graph. **xlings** (110 modules, 46k lines,
351-
different authors, never tuned for this) is the control, pinned as a submodule
352-
and measured in two code styles: implementation inside the interface units, and
353-
implementation split into separate `.cpp`. See
354-
[`bench/projects/xlings/`](bench/projects/xlings/).
355-
356-
Synthetic-fixture numbers across **six** engine/compiler combinations, including
357-
bazel and clang, are in [`bench/results/`](bench/results/).
358-
359-
<sub>Source: [`bench/results/mcpp-self-20260813/`](bench/results/mcpp-self-20260813/)
360-
— Linux x86_64, i9-13900K, medians of 2 runs, mcpp 2026.8.12.1, **cmake 4.0.2 +
361-
ninja, xmake 3.0.7**. CI pins cmake 4.4.2 / xmake 3.1.0 / bazel 9.2.0
362-
([`bench/matrix.json`](bench/matrix.json)) and these tables are refreshed from
363-
its artifacts; do not mix rows taken at different pins. bazel is absent from this
364-
table because it cannot build C++20 modules with a gcc driver — recorded as
365-
`unavailable` with the reason, never as a slow number.</sub>
353+
mcpp measuring its own build proves nothing on its own. **xlings** (110 modules,
354+
46k lines, different authors, never tuned for this) is pinned in two code styles
355+
— implementation inside the interface units, and implementation split into
356+
separate `.cpp`:
357+
358+
| scenario | combined, old → new mcpp | split, old → new mcpp | what the split buys |
359+
|---|---|---|---|
360+
| `cold` | 97.01s → 92.48s | 30.33s → 29.78s | **3.11x** |
361+
| `touch-hub` | 89.39s → **1.76s** | 24.87s → **1.30s** | 1.35x |
362+
| `edit-body` | 89.46s → 88.33s | 2.73s → **1.77s** | **49.96x** |
363+
364+
Splitting implementations out of the interface units is worth **2.6x on a cold
365+
build and ~50x on an edit** — a code style, not an engine feature, and the
366+
largest single effect in the suite.
367+
368+
Numbers are **n=1** except the split-tree `cold` row (n=3); read the ratios, not
369+
the digits. That row is why: at n=1 it read as a 23% regression, and at n=3 it is
370+
a marginal improvement — the single pair had caught one arm near the other's
371+
maximum. Full methodology, the
372+
declared asymmetries, and the cases where a cell must *not* be compared are in
373+
`bench/README.md`.
374+
375+
📊 **[Methodology, pinned versions and data → `bench/README.md`](bench/README.md)**
376+
· [中文](bench/README.zh-CN.md) · [what is measured → `bench/SPEC.md`](bench/SPEC.md)
366377

367378
<!-- BENCHMARK-TABLE:END -->
368379

369-
**What makes this comparable at all**, and what to check before quoting any of
370-
it:
371-
372-
* every engine is handed **the same compiler binary** out of mcpp's own payload
373-
(gcc 16.1.0 / clang 22.1.8), not whatever `g++` means on the runner;
374-
* the build tools are pinned — **cmake 4.4.2, xmake 3.1.0, bazel 9.2.0**
375-
and installed by xlings on every platform;
376-
* the projects are pinned as git submodules, so the target cannot drift;
377-
* **cmake is the baseline**: an absolute second count means nothing without
378-
knowing the machine, but "1.8× cmake" survives being read somewhere else.
379-
380-
There are declared asymmetries — cases where an engine is doing more or less
381-
work than another — and cells that are honestly `unavailable` or `skipped`
382-
rather than quietly zero. They are all written down.
383-
384-
📊 **[Full methodology, pinned versions and data → `bench/README.md`](bench/README.md)**
385-
· [中文](bench/README.zh-CN.md) · [what is measured → `bench/SPEC.md`](bench/SPEC.md)
380+
**What makes this comparable at all:** every engine is handed the same compiler
381+
binary out of mcpp's own payload; the build tools are pinned (cmake 4.4.2,
382+
xmake 3.1.0, bazel 9.2.0) and installed by xlings on every platform; the measured
383+
projects are pinned as git submodules so the target cannot drift; and cmake is
384+
the baseline, because an absolute second count means nothing without knowing the
385+
machine while "0.38x cmake" survives being read somewhere else.
386386

387387
## Platform Support
388388

bench/README.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -187,7 +187,7 @@ mcpp, because the cmake and xmake arms stop at the link here (SPEC.md §2).
187187

188188
| scenario | combined `2026.8.11.2` old → new | split `2026.8.13.1` old → new | what the split buys |
189189
|---|---|---|---|
190-
| `cold` | 97.01s → 92.48s | 29.13s35.88s | **2.58x** |
190+
| `cold` | 97.01s → 92.48s | 30.33s29.78s ⁽ⁿ⁼³⁾ | **3.11x** |
191191
| `noop` | 1.55s → 0.72s | 1.62s → 0.76s ||
192192
| `touch-hub` | 89.39s → **1.76s** (50.6x) | 24.87s → **1.30s** (19.1x) | 1.35x |
193193
| `edit-body` | 89.46s → 88.33s | 2.73s → **1.77s** | **49.96x** |
@@ -198,6 +198,13 @@ mcpp, because the cmake and xmake arms stop at the link here (SPEC.md §2).
198198
whole suite, and it is a *code style*, not an engine feature.
199199
* **`touch-hub` reproduces the engine result on a codebase nobody tuned for it**
200200
— 50.6x, against 190x on mcpp's own tree. Different magnitude, same mechanism.
201+
* ⚠️ **The `cold` row was nearly published as a 23% REGRESSION.** At n=1 the
202+
split tree read `29.13s → 35.88s`, i.e. the new mcpp slower. Re-measured at
203+
n=3 it is `30.33s → 29.78s` — marginally *faster*. The single pair had simply
204+
caught the new arm near the old arm's maximum: the old arm's spread is
205+
**19.1%** (29.92–35.72), a hair under the 20% that §4a R2 calls noisy, while
206+
the new arm's is 4.7%. This is R2 doing exactly what it is for, and it is why
207+
every other row here says n=1 rather than pretending otherwise.
201208
* **`edit-comment` does not improve at all here (1.00x), and that is correct.**
202209
xlings' hub has 56 function bodies, so inserting a comment moves every
203210
subsequent line; GCC records inline-body source locations in the BMI, the BMI

0 commit comments

Comments
 (0)