@@ -123,34 +123,91 @@ Four things this bought, each of which had already gone wrong:
123123variable; it is not evidence about anyone's build. Where the two disagree, the
124124real project is right and the fixture is telling you about its own shape.
125125
126- #### mcpp itself — 137 modules, 57k lines, gcc 16.1.0
126+ #### mcpp itself — the pinned workload, 137 modules, 57k lines, gcc 16.1.0
127127
128- Full data: [ ` results/mcpp-self-20260813/ ` ] ( results/mcpp-self-20260813/ ) , medians
129- of 2 runs, i9-13900K, measured in place with ` --buildfiles projects/mcpp/ ` .
128+ ` bench/projects/mcpp/mcpp-2026.8.11.3 ` (` a749e9f ` ), measured in place with
129+ ` --buildfiles projects/mcpp/ ` , i9-13900K, ** n=1** (see the caveat below).
130+ Ratios against cmake.
130131
131- | scenario | mcpp@2026.8.11.3 | mcpp@2026.8.12.1 | cmake | xmake |
132- | ---| ---| ---| ---| ---|
133- | ` cold ` | 80.49s · 0.85x | 82.87s · 0.88x | ** 94.53s** · 1.00x | 94.63s · 1.00x |
134- | ` noop ` | 0.28s · 0.83x | 0.20s · 0.58x | ** 0.34s** · 1.00x | 0.38s · 1.10x |
135- | ` touch-leaf ` | 17.39s · 0.96x | 2.14s · 0.12x | ** 18.06s** · 1.00x | 18.47s · 1.02x |
136- | ` edit-body ` | 18.30s · 0.93x | 18.29s · 0.93x | ** 19.64s** · 1.00x | 19.97s · 1.02x |
137- | ` edit-comment ` | 76.50s · 0.90x | ** 0.46s · 0.01x** | ** 85.03s** · 1.00x | 84.69s · 1.00x |
138- | ` touch-hub ` | 76.50s · 0.91x | ** 0.44s · 0.01x** | ** 84.53s** · 1.00x | 83.65s · 0.99x |
139-
140- Three things this says that the fixture cannot:
141-
142- 1 . ** On a cold build nobody wins, and that is the right answer.** 80.5–94.6s
143- across four engines. mcpp's cold build is 100% critical path — 79.73s of a
144- 79.79s makespan, average parallelism 3.94 of 32 hardware threads — so every
145- engine walks the same 26-deep chain of interfaces and scheduling cannot help.
146- The fixture put mcpp at ** 0.26x** here; that number is an artefact of a
147- workload whose units cost 0.09s each, and quoting it would be dishonest.
148- 2 . ** The daily loop is where the engines differ** , by ~ 190x on this project.
149- 3 . ** ` edit-body ` is the control** , and mcpp is deliberately * not* fast there
150- (0.93x): the interface genuinely changed, so the cascade is owed.
132+ | scenario | ` mcpp@2026.8.11.3 ` | ` mcpp@2026.8.13.1 ` | ` +bmi_schedule=on ` | ` cmake ` | ` xmake ` |
133+ | ---| ---| ---| ---| ---| ---|
134+ | ` cold ` | 79.46s · 0.86x | 79.54s · 0.86x | ** 35.43s · 0.38x** | ** 92.33s** · 1.00x | 90.30s · 0.98x |
135+ | ` noop ` | 0.34s · 1.21x | 0.16s · 0.57x | 0.16s · 0.57x | ** 0.28s** · 1.00x | 0.38s · 1.36x |
136+ | ` touch-hub ` | 76.53s · 0.92x | ** 0.40s · 0.005x** | ** 0.22s · 0.003x** | ** 83.39s** · 1.00x | 82.07s · 0.98x |
137+ | ` edit-body ` | 77.33s · 0.90x | 76.24s · 0.89x | ** 30.17s · 0.35x** | ** 85.64s** · 1.00x | 84.61s · 0.99x |
138+ | ` edit-comment ` | 75.69s · 0.91x | ** 0.38s · 0.005x** | ** 0.18s · 0.002x** | ** 82.96s** · 1.00x | 82.73s · 1.00x |
139+
140+ Four things this says, and the fixture can say none of them:
141+
142+ 1 . ** On a cold build nobody wins, and that is the correct answer.** Every engine
143+ is within 15% of the others, because mcpp's cold build is ** 100% critical
144+ path** — 79.7s of a 79.8s makespan, average parallelism 3.94 of 32 hardware
145+ threads. All of them walk the same 26-deep chain of module interfaces, and
146+ scheduling cannot shorten a chain. The generated fixture puts mcpp at ` 0.26x `
147+ here; that is an artefact of a workload whose units cost 0.09s each, and
148+ quoting it as a cold-build advantage would be dishonest.
149+ 2 . ** The cold-build lever is the opt-in schedule, not the release.** 79.46s →
150+ 79.54s between the two releases is no change at all; ` bmi_schedule = "on" `
151+ takes it to 35.43s. Everything else in this table is release-over-release;
152+ that column is a * setting* .
153+ 3 . ** The daily loop is where the engines differ** , by ~ 190x on this project:
154+ touching a hub interface costs cmake and xmake a full 83-second rebuild
155+ because they decide by timestamp, and 0.40s for an engine that compares the
156+ BMI it just produced against the previous one.
157+ 4 . ** ` edit-body ` is the control.** mcpp is deliberately * not* fast there (0.89x):
158+ the interface genuinely changed, so the cascade is owed. An engine that were
159+ fast on that row would have skipped work it owed.
160+
161+ > ** The xmake column is from a SEPARATE run.** Its numbers in the original
162+ > five-arm run were invalid — xmake normalises ` --buildir ` to a path relative to
163+ > ` -P ` and then resolves it against the process cwd, so ` clean() ` had been
164+ > removing a directory it never wrote to and ` cold ` came back at ** 0.60s** with
165+ > status ` ok ` . Fixed (the engine now runs from ` -P ` ) and re-measured on the same
166+ > machine; ` cold ` went 0.58s → 90.95s in the isolated check and 90.30s here.
167+ > Recorded rather than quietly re-run, because the two halves of this table were
168+ > not taken in the same minute.
169+
170+ > ** n=1, so read the ratios and not the digits.** §4a R2 asks for dispersion and
171+ > a single sample has none. Two rows also sit near their own engine's resolution
172+ > floor: mcpp's ` touch-hub ` and ` edit-comment ` are 2.5x and 2.4x its own ` noop ` ,
173+ > just above R1's 2x line, so * "about two orders of magnitude"* is supported and
174+ > * "0.40 versus 0.38"* is not.
175+
176+ > ** ` edit-comment ` here is the ` end-of-file ` form.** mcpp's hub has no function
177+ > body, so the comment is appended rather than inserted, and no line numbers
178+ > move. On a hub that does have bodies the same scenario legitimately cascades —
179+ > see the xlings table below and SPEC.md §4. The cell's ` note ` records which
180+ > form ran.
181+
182+ #### xlings — the same question on someone else's codebase, in two code styles
183+
184+ 110 modules, 46k lines, different authors, never tuned for this. The two pins
185+ are the same project either side of one refactor. Ratios against the released
186+ mcpp, because the cmake and xmake arms stop at the link here (SPEC.md §2).
187+
188+ | scenario | combined ` 2026.8.11.2 ` old → new | split ` 2026.8.13.1 ` old → new | what the split buys |
189+ | ---| ---| ---| ---|
190+ | ` cold ` | 97.01s → 92.48s | 29.13s → 35.88s | ** 2.58x** |
191+ | ` noop ` | 1.55s → 0.72s | 1.62s → 0.76s | — |
192+ | ` touch-hub ` | 89.39s → ** 1.76s** (50.6x) | 24.87s → ** 1.30s** (19.1x) | 1.35x |
193+ | ` edit-body ` | 89.46s → 88.33s | 2.73s → ** 1.77s** | ** 49.96x** |
194+ | ` edit-comment ` | 95.40s → 95.02s | 25.09s → 25.29s | 3.76x |
195+
196+ * ** Splitting implementations out of the interface units is worth 2.6x on a cold
197+ build and ~ 50x on ` edit-body ` .** That is the largest single effect in this
198+ whole suite, and it is a * code style* , not an engine feature.
199+ * ** ` touch-hub ` reproduces the engine result on a codebase nobody tuned for it**
200+ — 50.6x, against 190x on mcpp's own tree. Different magnitude, same mechanism.
201+ * ** ` edit-comment ` does not improve at all here (1.00x), and that is correct.**
202+ xlings' hub has 56 function bodies, so inserting a comment moves every
203+ subsequent line; GCC records inline-body source locations in the BMI, the BMI
204+ genuinely changes, and the cascade is owed. mcpp's own hub has none, which is
205+ the entire reason that row reads 199x there and 1.00x here. ** A project
206+ measuring itself cannot discover this.**
151207
152208#### The generated fixture — 40 units, fan-in 3
153209
210+
154211Full data: [ ` results/five-way-20260812/ ` ] ( results/five-way-20260812/ ) . Useful
155212because it is the only place ` headers ` / ` modules ` / ` modules-impl ` can be
156213compared as a controlled variable, and because it covers clang and bazel too.
0 commit comments