Skip to content

Commit f5af236

Browse files
committed
fix(bench): a one-line function body put the perturbation outside the function
扰动是「插在 `) {` 之后那个换行的后面」—— 只有函数体跨多行时,那个位置才等于 「函数体内部」。给一个单行函数体: export int hub_value() { return 1; } 那个换行在**闭合大括号之后**,于是语句落到了命名空间作用域,构建死在 error: expected unqualified-id volatile int bench_nonce_0 = 0; (void)bench_nonce_0; 指着一个 harness 自己刚写出来的文件。行为是诚实的(cell 大声失败了),但结论是 错的:这个扰动本该对任何函数都适用,「你的函数写在一行」不是一个真实的限制。 改成锚在**大括号本身**、插入文本自带前导换行 —— 单行和多行都对。 (是我为 harness.sh 新加的那个 fixture 在 macOS CI 上撞出来的。)
1 parent f65877f commit f5af236

2 files changed

Lines changed: 101 additions & 31 deletions

File tree

bench/README.md

Lines changed: 80 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -123,34 +123,91 @@ Four things this bought, each of which had already gone wrong:
123123
variable; it is not evidence about anyone's build. Where the two disagree, the
124124
real project is right and the fixture is telling you about its own shape.
125125

126-
#### mcpp itself — 137 modules, 57k lines, gcc 16.1.0
126+
#### mcpp itself — the pinned workload, 137 modules, 57k lines, gcc 16.1.0
127127

128-
Full data: [`results/mcpp-self-20260813/`](results/mcpp-self-20260813/), medians
129-
of 2 runs, i9-13900K, measured in place with `--buildfiles projects/mcpp/`.
128+
`bench/projects/mcpp/mcpp-2026.8.11.3` (`a749e9f`), measured in place with
129+
`--buildfiles projects/mcpp/`, i9-13900K, **n=1** (see the caveat below).
130+
Ratios against cmake.
130131

131-
| scenario | mcpp@2026.8.11.3 | mcpp@2026.8.12.1 | cmake | xmake |
132-
|---|---|---|---|---|
133-
| `cold` | 80.49s · 0.85x | 82.87s · 0.88x | **94.53s** · 1.00x | 94.63s · 1.00x |
134-
| `noop` | 0.28s · 0.83x | 0.20s · 0.58x | **0.34s** · 1.00x | 0.38s · 1.10x |
135-
| `touch-leaf` | 17.39s · 0.96x | 2.14s · 0.12x | **18.06s** · 1.00x | 18.47s · 1.02x |
136-
| `edit-body` | 18.30s · 0.93x | 18.29s · 0.93x | **19.64s** · 1.00x | 19.97s · 1.02x |
137-
| `edit-comment` | 76.50s · 0.90x | **0.46s · 0.01x** | **85.03s** · 1.00x | 84.69s · 1.00x |
138-
| `touch-hub` | 76.50s · 0.91x | **0.44s · 0.01x** | **84.53s** · 1.00x | 83.65s · 0.99x |
139-
140-
Three things this says that the fixture cannot:
141-
142-
1. **On a cold build nobody wins, and that is the right answer.** 80.5–94.6s
143-
across four engines. mcpp's cold build is 100% critical path — 79.73s of a
144-
79.79s makespan, average parallelism 3.94 of 32 hardware threads — so every
145-
engine walks the same 26-deep chain of interfaces and scheduling cannot help.
146-
The fixture put mcpp at **0.26x** here; that number is an artefact of a
147-
workload whose units cost 0.09s each, and quoting it would be dishonest.
148-
2. **The daily loop is where the engines differ**, by ~190x on this project.
149-
3. **`edit-body` is the control**, and mcpp is deliberately *not* fast there
150-
(0.93x): the interface genuinely changed, so the cascade is owed.
132+
| scenario | `mcpp@2026.8.11.3` | `mcpp@2026.8.13.1` | `+bmi_schedule=on` | `cmake` | `xmake` |
133+
|---|---|---|---|---|---|
134+
| `cold` | 79.46s · 0.86x | 79.54s · 0.86x | **35.43s · 0.38x** | **92.33s** · 1.00x | 90.30s · 0.98x |
135+
| `noop` | 0.34s · 1.21x | 0.16s · 0.57x | 0.16s · 0.57x | **0.28s** · 1.00x | 0.38s · 1.36x |
136+
| `touch-hub` | 76.53s · 0.92x | **0.40s · 0.005x** | **0.22s · 0.003x** | **83.39s** · 1.00x | 82.07s · 0.98x |
137+
| `edit-body` | 77.33s · 0.90x | 76.24s · 0.89x | **30.17s · 0.35x** | **85.64s** · 1.00x | 84.61s · 0.99x |
138+
| `edit-comment` | 75.69s · 0.91x | **0.38s · 0.005x** | **0.18s · 0.002x** | **82.96s** · 1.00x | 82.73s · 1.00x |
139+
140+
Four things this says, and the fixture can say none of them:
141+
142+
1. **On a cold build nobody wins, and that is the correct answer.** Every engine
143+
is within 15% of the others, because mcpp's cold build is **100% critical
144+
path** — 79.7s of a 79.8s makespan, average parallelism 3.94 of 32 hardware
145+
threads. All of them walk the same 26-deep chain of module interfaces, and
146+
scheduling cannot shorten a chain. The generated fixture puts mcpp at `0.26x`
147+
here; that is an artefact of a workload whose units cost 0.09s each, and
148+
quoting it as a cold-build advantage would be dishonest.
149+
2. **The cold-build lever is the opt-in schedule, not the release.** 79.46s →
150+
79.54s between the two releases is no change at all; `bmi_schedule = "on"`
151+
takes it to 35.43s. Everything else in this table is release-over-release;
152+
that column is a *setting*.
153+
3. **The daily loop is where the engines differ**, by ~190x on this project:
154+
touching a hub interface costs cmake and xmake a full 83-second rebuild
155+
because they decide by timestamp, and 0.40s for an engine that compares the
156+
BMI it just produced against the previous one.
157+
4. **`edit-body` is the control.** mcpp is deliberately *not* fast there (0.89x):
158+
the interface genuinely changed, so the cascade is owed. An engine that were
159+
fast on that row would have skipped work it owed.
160+
161+
> **The xmake column is from a SEPARATE run.** Its numbers in the original
162+
> five-arm run were invalid — xmake normalises `--buildir` to a path relative to
163+
> `-P` and then resolves it against the process cwd, so `clean()` had been
164+
> removing a directory it never wrote to and `cold` came back at **0.60s** with
165+
> status `ok`. Fixed (the engine now runs from `-P`) and re-measured on the same
166+
> machine; `cold` went 0.58s → 90.95s in the isolated check and 90.30s here.
167+
> Recorded rather than quietly re-run, because the two halves of this table were
168+
> not taken in the same minute.
169+
170+
> **n=1, so read the ratios and not the digits.** §4a R2 asks for dispersion and
171+
> a single sample has none. Two rows also sit near their own engine's resolution
172+
> floor: mcpp's `touch-hub` and `edit-comment` are 2.5x and 2.4x its own `noop`,
173+
> just above R1's 2x line, so *"about two orders of magnitude"* is supported and
174+
> *"0.40 versus 0.38"* is not.
175+
176+
> **`edit-comment` here is the `end-of-file` form.** mcpp's hub has no function
177+
> body, so the comment is appended rather than inserted, and no line numbers
178+
> move. On a hub that does have bodies the same scenario legitimately cascades —
179+
> see the xlings table below and SPEC.md §4. The cell's `note` records which
180+
> form ran.
181+
182+
#### xlings — the same question on someone else's codebase, in two code styles
183+
184+
110 modules, 46k lines, different authors, never tuned for this. The two pins
185+
are the same project either side of one refactor. Ratios against the released
186+
mcpp, because the cmake and xmake arms stop at the link here (SPEC.md §2).
187+
188+
| scenario | combined `2026.8.11.2` old → new | split `2026.8.13.1` old → new | what the split buys |
189+
|---|---|---|---|
190+
| `cold` | 97.01s → 92.48s | 29.13s → 35.88s | **2.58x** |
191+
| `noop` | 1.55s → 0.72s | 1.62s → 0.76s ||
192+
| `touch-hub` | 89.39s → **1.76s** (50.6x) | 24.87s → **1.30s** (19.1x) | 1.35x |
193+
| `edit-body` | 89.46s → 88.33s | 2.73s → **1.77s** | **49.96x** |
194+
| `edit-comment` | 95.40s → 95.02s | 25.09s → 25.29s | 3.76x |
195+
196+
* **Splitting implementations out of the interface units is worth 2.6x on a cold
197+
build and ~50x on `edit-body`.** That is the largest single effect in this
198+
whole suite, and it is a *code style*, not an engine feature.
199+
* **`touch-hub` reproduces the engine result on a codebase nobody tuned for it**
200+
— 50.6x, against 190x on mcpp's own tree. Different magnitude, same mechanism.
201+
* **`edit-comment` does not improve at all here (1.00x), and that is correct.**
202+
xlings' hub has 56 function bodies, so inserting a comment moves every
203+
subsequent line; GCC records inline-body source locations in the BMI, the BMI
204+
genuinely changes, and the cascade is owed. mcpp's own hub has none, which is
205+
the entire reason that row reads 199x there and 1.00x here. **A project
206+
measuring itself cannot discover this.**
151207

152208
#### The generated fixture — 40 units, fan-in 3
153209

210+
154211
Full data: [`results/five-way-20260812/`](results/five-way-20260812/). Useful
155212
because it is the only place `headers` / `modules` / `modules-impl` can be
156213
compared as a controlled variable, and because it covers clang and bazel too.

bench/src/runner.cppm

Lines changed: 21 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -97,17 +97,28 @@ inline std::optional<std::string_view> insert_into_first_body(
9797
std::string text((std::istreambuf_iterator<char>(in)), std::istreambuf_iterator<char>());
9898
in.close();
9999

100-
// Insert inside the first function body: after the first '{' that follows a
101-
// ')'. Anchoring on the brace rather than a name keeps this working for all
102-
// three variants, whose function text differs.
100+
// Insert inside the first function body: immediately after the first '{'
101+
// that follows a ')'. Anchoring on the brace rather than a name keeps this
102+
// working for all three variants, whose function text differs.
103+
//
104+
// ⚠️ AFTER THE BRACE, not after the newline that follows it. Those are the
105+
// same position only when the body spans several lines. Given a one-line
106+
// body — `export int f() { return 1; }` — the newline is past the CLOSING
107+
// brace, so the statement landed at namespace scope and the build died with
108+
//
109+
// error: expected unqualified-id
110+
// volatile int bench_nonce_0 = 0; (void)bench_nonce_0;
111+
//
112+
// pointing at a file the harness had just written. Honest (the cell failed
113+
// loudly) but wrong: the perturbation is supposed to be applicable to any
114+
// function, and "your function is on one line" is not a real limitation.
103115
//
104116
// A file may legitimately have NO function body — the modules-impl variant's
105117
// interface unit only declares — so a comment falls back to end-of-file
106118
// rather than reporting the scenario as inapplicable. A statement has no
107119
// such fallback: there is nowhere to put it that would mean the same thing.
108120
const auto paren = text.find(") {");
109-
const auto brace = paren == std::string::npos ? std::string::npos
110-
: text.find('\n', paren);
121+
const auto brace = paren == std::string::npos ? std::string::npos : paren + 2;
111122
std::string_view form = "in-body";
112123
if (brace == std::string::npos) {
113124
if (statement) return std::nullopt;
@@ -121,11 +132,13 @@ inline std::optional<std::string_view> insert_into_first_body(
121132
// The name carries the nonce because perturbations ACCUMULATE across the
122133
// repetitions of one cell: a fixed name redeclares itself on run 2 and
123134
// the build fails, which is exactly what the first version did.
135+
// The leading newline is what makes a one-line body work: the text
136+
// opens its own line immediately after `{`, whatever followed it.
124137
text.insert(brace + 1,
125138
statement
126-
? std::format(" volatile int bench_nonce_{0} = {0};"
127-
" (void)bench_nonce_{0};\n", nonce)
128-
: std::format(" // bench: comment perturbation #{}\n", nonce));
139+
? std::format("\n volatile int bench_nonce_{0} = {0};"
140+
" (void)bench_nonce_{0};", nonce)
141+
: std::format("\n // bench: comment perturbation #{}", nonce));
129142
}
130143

131144
std::ofstream out(file, std::ios::binary | std::ios::trunc);

0 commit comments

Comments
 (0)