feat(bench): measure bazel's C++20 modules, and split "edit" into what it means

—— 四处让这份 benchmark 从「看起来对」变成「说得清」

**bazel 其实支持模块,写死的 `supports=false` 抹掉了一整列真实数据。**
实测 bazel 9.2.0 + rules_cc 0.2.22:`module_interfaces` 属性存在,
配 `--experimental_cpp_modules` + `--features=cpp_modules`(缺任一个报不同的错)
与 **clang** 能构建并运行模块程序;配 **gcc** 则死在它自己的扫描器:
`aggregate-ddi failed ... Invalid JSON string` —— 它解析不了 GCC 的 P1689 输出。
所以能力判断不是引擎的属性,而是引擎×编译器的属性,`supports()` 因此收下 compiler。
meson 1.10.2 的理由也改成实测原文(`module 'fx.a' not found`),不再是断言。

`--force_pic` 是模块单元能跑起来的前提:cc_binary 为 PIC 与非 PIC 两套目标文件
各注册一次 ddi 聚合动作,却共用 `<target>.CXXModules.json` 这一个输出名,
分析阶段就崩(`unit_0.pic.ddi` vs `unit_0.ddi`, `Outputs: are equal`)。
选 PIC 而不是 `-supports_pic`,因为它产出 PIE —— 和其他引擎的默认产物一致。

**`edit-body` 插的是注释,于是每一个「改代码快 N 倍」的数字其实在说注释。**
拆成两个场景:`edit-body` 插入带 nonce 的 `volatile` 语句(真改 codegen,
nonce 在标识符里 —— 固定名字会在第 2 轮重复声明把构建打挂),
`edit-comment` 往被广泛导入的接口单元插注释(字节变、接口没变)。
顺带记下一个反直觉的实测结论:GCC 16.1 **不把导出非模板函数的函数体写进 BMI**,
所以改函数体不重编导入者是**对的**。判据必须带对照组 —— 同一份源码编译两遍,
差的是同样两个偏移,落在 `buildtime:`/`localtime:` 的秒位上。

**相对路径的引擎二进制一直是不可用的。** 每条被测命令的 cwd 都是被测工程,
所以 `--engines mcpp=./mcpp-old` 解析到了 fixture 目录,整个矩阵报 `exited -1`
而日志是空的。规格转引擎的那一处统一锚定成绝对路径;裸名仍走 PATH。
同时把「起不来」和「跑了但失败」在措辞上分开——前者不再指向一个从未写入的日志。

**`touch-leaf` 定义了、文档写了、`--help` 也列了,却从未跑过**:它不在默认场景表里。
默认表改成全部六个,CI 的 `scenarios` 默认值同步。

引擎版本现在由引擎自己写进结果文件(cmake 4.0.2 / xmake v3.0.7+HEAD / bazel 9.2.0),
之前只记了 "cmake + ninja",数据自己说不清是哪个 cmake 产的;xmake 的彩色 banner
要剥 CSI,而按 `@`-`~` 直接扫会停在 `[` 上、留下每个 reset 的 "0m"。

结果:`bench/results/five-way-20260812.md` 两张完整矩阵(gcc / clang × 六引擎 ×
三变体 × 六场景,cmake 为基准)。gcc 模块增量 mcpp 0.29s vs cmake 10.29s(35×),
vs 上一版 mcpp 3.65s(12.5×);clang 下 cmake 冷构建自身快 3.3×,bazel 3.19s 参赛,
xmake 每一个模块增量都是 ~12.6s。

测试:e2e 230 增两项 —— 相对引擎路径必须解析,引擎 note 不得含 ANSI 转义。
本地 82/82 单测通过,e2e 230 通过。

Author: speak-agent

.agents/docs/2026-08-12-bench-suite-architecture-and-plan.md

@@ -112,7 +112,11 @@ export struct Engine {
 
 **加一个引擎 = 新增一个 `engines/<name>.cppm` + 在 `registry.cppm` 注册一行。** 不动 runner、不动协议、不动 CI。
 
-`supports(Variant)` 是必要的:并非所有引擎都支持 C++20 模块(bazel 的模块支持仍很有限),此时应报 `unavailable` 并说明,而不是硬跑出一个误导性的数字。
+`supports(Variant, compiler)` 是必要的,而且**编译器是这个问题的一部分** —— 实测:bazel 9.2 + rules_cc 0.2.22
+配 clang 能构建 C++20 模块,配 gcc 则死在它自己的扫描器里(`aggregate-ddi: Invalid JSON string`,
+它解析不了 GCC 的 P1689 输出);meson 1.10.2 两个编译器都不行(`module 'fx.a' not found`)。
+所以"bazel 支不支持模块"没有脱离具体运行的答案。不支持时报 `unavailable` **并附上得出该结论的那次测量**,
+而不是硬跑出一个误导性的数字。
 
 ---
 
@@ -202,4 +206,7 @@ fixtures/synth-<N>x<D>/
 - **不把基准挂进 PR CI**。噪声会淹没信号。
 - **不设性能回归阈值**。宿主差异(异构 CPU、云厂商邻居噪声)远大于多数真实回归。
 - **不重新实现计时统计学**。中位数 + min/max 足够;不做置信区间,因为样本量本来就小。
-- **不追求引擎功能对等**。bazel 不支持模块就报 unavailable —— 强行凑一个数字比没有数字更糟。
+- **不追求引擎功能对等**。引擎跑不了某个变体就报 unavailable —— 强行凑一个数字比没有数字更糟。
+  但"跑不了"必须是**测出来的**,不是假设的:最初这里写死了 `bazel supports(modules) = false`,
+  而实际上加上 `module_interfaces` + `--experimental_cpp_modules --features=cpp_modules` 之后,
+  bazel 配 clang 是能构建并运行模块程序的。写死的能力判断会把一整列真实数据变成空白。

.github/workflows/bench.yml

@@ -25,9 +25,12 @@ on:
         required: false
         default: 'headers,modules,modules-impl'
       scenarios:
-        description: 'comma-separated: cold,noop,touch-hub,edit-body,touch-leaf'
+        description: 'comma-separated: cold,noop,touch-hub,touch-leaf,edit-body,edit-comment'
         required: false
-        default: 'cold,noop,touch-hub,edit-body'
+        # All of them. A scenario left out of the default is a scenario nobody
+        # ever runs — `touch-leaf` was defined, documented and advertised, and
+        # had never appeared in a single result file.
+        default: 'cold,noop,touch-hub,touch-leaf,edit-body,edit-comment'
       units:
         description: 'fixture translation units'
         required: false

bench/README.md

@@ -113,7 +113,8 @@ when the build fails, which is precisely when a leftover edit would be missed.
 | `cold` | `clean()`, then time **configure + build** | full graph construction + every compile |
 | `noop` | nothing | the up-to-date check / fast path |
 | `touch-hub` | mtime bump on the most-depended-on unit, **content unchanged** | can the engine prove the interface did not change and stop the cascade? |
-| `edit-body` | insert a **numbered** marker inside a function body, interface untouched | the everyday developer loop |
+| `edit-comment` | insert a **comment** into the most-depended-on unit — bytes change, interface does not | mtime is no longer enough; only comparing the produced BMI avoids the cascade |
+| `edit-body` | insert a **numbered `volatile` statement** into a function body | the everyday developer loop: real codegen change, interface untouched |
 | `touch-leaf` | mtime bump on a unit nobody depends on | recompile 1 + link |
 
 Two details that are easy to get wrong and change the answer:
@@ -123,9 +124,26 @@ Two details that are easy to get wrong and change the answer:
   fails. Timing configure separately would also be wrong: the user waits for both,
   and engines that fold configure into the build (mcpp, bazel) would get a
   discount for it.
-* **`edit-body` uses a counter.** An idempotent edit is a real edit on run 1 and a
-  bare `touch` on runs 2..N — a different, much cheaper scenario, silently
-  dragging the median toward it.
+* **`edit-body` uses a counter**, and the counter is in the *identifier*. An
+  idempotent edit is a real edit on run 1 and a bare `touch` on runs 2..N — a
+  different, much cheaper scenario, silently dragging the median toward it. The
+  inserted statement is `volatile`, so no optimiser can delete it and hand back
+  the previous object file, and its name carries the nonce, because
+  perturbations ACCUMULATE within a cell and a fixed name redeclares itself on
+  run 2.
+* **`edit-body` and `edit-comment` are separate on purpose.** They were one
+  scenario, named `edit-body`, that inserted a comment — so every "N times
+  faster on edits" number it produced was really a statement about comments.
+  Splitting them costs one extra column and makes each number mean its name.
+
+  On GCC 16.1 both happen to be cheap for the same underlying reason, and it is
+  worth stating because it is easy to misread as a bug: **GCC does not encode
+  the body of an exported non-template function into the BMI.** Editing such a
+  body changes the object file and leaves the BMI byte-identical apart from its
+  embedded `buildtime:`/`localtime:` stamps, so skipping the importers is
+  correct, not a missed rebuild. Establishing that requires a control — compile
+  the *same* source twice and diff: the differing bytes land at the same offsets,
+  inside the timestamps.
 
 ---
 
@@ -164,9 +182,36 @@ These cannot be removed, so they are stated rather than hidden.
   cache outside the workspace. `clean` here is deliberately *not* `--expunge`,
   which would also discard the toolchain and turn the measurement into
   provisioning. Every bazel cell says so in its note.
-* **meson and bazel are headers-only.** Their C++20 named-module support is not
-  comparable to cmake's or xmake's; they report `unavailable` with a reason
-  rather than producing a number that does not mean what it looks like.
+* **Module support is a property of the engine *and* the compiler.** `supports()`
+  therefore takes both, and a `false` becomes `unavailable` **with the
+  measurement that produced it** — never a slow number. As measured here:
+
+  | engine | modules with clang | modules with gcc |
+  |---|---|---|
+  | mcpp | yes | yes |
+  | cmake ≥ 3.28 | yes | yes |
+  | xmake 3.x | yes | yes |
+  | bazel 9.2 + rules_cc 0.2.22 | **yes** | no — `aggregate-ddi failed … Invalid JSON string`, i.e. its ddi aggregator cannot parse GCC's P1689 output |
+  | meson 1.10.2 | no — `fatal error: module 'fx.a' not found`; no attribute declares an interface unit | no |
+
+  So a gcc run and a clang run legitimately have **different sets of populated
+  cells**, and a table must say which compiler it used before its `unavailable`
+  rows mean anything.
+* **bazel module builds are forced to one object flavour.** `cc_binary` registers
+  the ddi-aggregation action for both the PIC and the non-PIC object sets but
+  names the output `<target>.CXXModules.json` for both, so analysis aborts before
+  any compilation:
+
+  ```
+  Attempted action contains artifacts not in previous action: _objs/fx/unit_0.pic.ddi
+  Previous action contains artifacts not in attempted action:  _objs/fx/unit_0.ddi
+  Outputs: are equal
+  ```
+
+  The adapter passes `--force_pic` — to **every** variant, so bazel's own
+  headers-vs-modules rows stay comparable, and PIC rather than
+  `--features=-supports_pic` because it yields a PIE executable, which is what
+  the other engines produce by default.
 
 ---