Skip to content

Commit 44b08ed

Browse files
committed
fix(bench): every cell in the matrix was failing behind a green check
`bench (macos/clang/fixture)` reported success with **6 ok / 48 failed / 18 unavailable**, and all three xlings jobs reported success having measured nothing at all. 这批修复的是「为什么会这样」以及「为什么没人发现」。 五个独立的真因,每一个都单独足以让整份矩阵失效: 1. **每个引擎拿到的编译器不是同一个。** CI 用 `command -v g++` = runner 自带的 gcc 13.3.0,而 mcpp 一直悄悄用自己 registry 里的 gcc 16.1。cmake 配不出 C++23 modules,xmake 直接把 gcc 编崩(ICE)。套件自己的公平性规则 (`resolve_cxx`)写在注释里,但没有任何东西执行它。 → 新增 `bench.toolchain` + `--compiler payload:gcc|payload:clang`,解析到 **mcpp 自己载荷里的驱动**,并且 fixture 的 mcpp.toml 从同一处取版本。 2. **构建工具版本随 runner 漂移。** 镜像自带 cmake 3.31.6 —— 没有 CMake 4.0 的 `import std` 实验键,所以每个 module 格子都 configure 失败。 → cmake 4.4.2 / xmake 3.1.0 / bazel 9.2.0 全部由 xlings 按精确版本安装, pin 写在 matrix.json,job 打印实际解析到的版本并在不符时告警。 3. **被测工程是运行时从默认分支 clone 的。** `--hub src/xlings.cppm` 指的文件 几个月前就没了 —— 每个 xlings 格子报 `skipped`,每个 job 报成功。 → 两棵树改成 git 子模块钉住:2026.8.11.2 (`b1563fe`) 与 2026.8.13.1 (`f072075`)。 4. **`--hub`/`--body` 是按 harness 的 cwd 解析的**,不是按工程目录。只有在 「测你正站着的那棵树」时才对(mcpp 测自己),换任何工程都静默失效。 5. **harness 永远返回 0。** 时间不该设阈值(共享 runner),但「有没有测到东西」 可以。现在 `failed` 或「一个 ok 都没有」都返回非零;`unavailable`/`skipped` 是缺口,不影响退出码。真正的已知缺口写进格子的 `allow_failed`,且守卫强制 它必须带 `KNOWN GAP` 说明。 可观测性(用户报的「卡住而且没有进度」): * 进度实时打到 stderr 并逐行 flush —— 之前一个格子只在结束时才打印,所以卡在 第三个引擎和卡在第一个引擎看起来一模一样(两个 job 各卡了 25 分钟)。 * 每条 configure/build 有超时(默认 1800s),超时 kill 并明确报 `TIMED OUT after Ns and was killed`。POSIX 用 waitpid(WNOHANG) 轮询 + SIGKILL, Windows 用 WaitForSingleObject + TerminateProcess。 * 失败时直接打出子进程日志尾部 —— CI 上那个文件跟着 runner 一起销毁,只写 `see .../cmake-cold.log` 等于什么都没说。这一条当场找出了 xmake 的 `attempt to call a nil value (global 'bench_package_root')`。 顺带修掉的两个: * `find target -name bench | head -1` 在本机就挑中了一个两个半小时前的旧二进制 —— target/ 每个工具链指纹一个目录。收敛到 `.github/tools/newest_artifact.sh`。 * xlings 的 xmake arm 在 `on_load` 里调 include 进来的全局函数,xmake 的沙箱看 不到 —— 那条 arm 从来没跑起来过。改成在 description scope 解析成局部变量, 与 mcpp arm 已有的写法一致。 xlings 现在是一个**代码风格对比**:同一个工程、同一张模块图、46k 行, `f072075` 把实现从接口单元里拆了出来(110 .cppm + 2 .cpp → 110 .cppm + 92 .cpp)。 一份构建描述同时服务两棵树 —— 它 glob `src/**/*.{cppm,cpp}`,也就是 mcpp 自己 推导的那条规则,所以不需要写两份、不需要环境变量、不需要分支。 守卫(233 + bench/tests/harness.sh)新增:hub/body 必须在钉住的树里真实存在、 工具版本必须是精确版本、豁免必须命名真实引擎且带 KNOWN GAP、工程相对路径必须 从任意 cwd 解析、什么都没测到必须非零退出、超时必须真的开火。前两条分别用 「把 hub 改回 src/xlings.cppm」和「把 cmake 改成 latest」验证过会变红。
1 parent 8788c67 commit 44b08ed

29 files changed

Lines changed: 1522 additions & 215 deletions

.github/tools/newest_artifact.sh

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
#!/usr/bin/env bash
2+
# newest_artifact.sh <target-dir> <basename> — print the most recently built
3+
# copy of a binary under a target/ tree.
4+
#
5+
# WHY THIS IS NOT `find ... | head -1`. mcpp lays artifacts out under
6+
# target/<triple>/<toolchain-fingerprint>/bin/, and the fingerprint changes
7+
# whenever the toolchain, the standard or the flags do. A tree that has been
8+
# built more than once therefore holds SEVERAL binaries with the same name, and
9+
# `find | head -1` picks whichever the filesystem happens to list first.
10+
#
11+
# That is not hypothetical: it picked a two-and-a-half-hour-old bench binary on
12+
# the first machine it ran on, and the run that followed silently exercised code
13+
# that had already been replaced. In CI the same line would benchmark a stale
14+
# mcpp and report the numbers as the new one's — a wrong answer with no symptom,
15+
# which is the only kind this suite really has to defend against.
16+
#
17+
# `-printf` is GNU-only and macOS ships BSD find, so the mtime comes from a
18+
# per-file `stat` call whose flag differs by platform. Both spellings are here
19+
# because the alternative is a script that works on Linux and silently returns
20+
# the wrong file everywhere else.
21+
set -euo pipefail
22+
23+
dir="${1:?usage: newest_artifact.sh <target-dir> <basename>}"
24+
name="${2:?usage: newest_artifact.sh <target-dir> <basename>}"
25+
26+
[ -d "$dir" ] || { echo "newest_artifact: no such directory: $dir" >&2; exit 1; }
27+
28+
mtime() {
29+
# GNU coreutils first, then BSD/macOS. Windows runners use git-bash, which
30+
# ships GNU stat.
31+
stat -c %Y "$1" 2>/dev/null || stat -f %m "$1" 2>/dev/null || echo 0
32+
}
33+
34+
best=""
35+
best_t=-1
36+
# `bin/<name>` and `bin/<name>.exe` — anchored on the bin/ directory so a
37+
# same-named object or intermediate elsewhere in target/ cannot win.
38+
while IFS= read -r f; do
39+
[ -f "$f" ] || continue
40+
t=$(mtime "$f")
41+
if [ "$t" -gt "$best_t" ]; then best_t=$t; best=$f; fi
42+
done <<EOF
43+
$(find "$dir" -type f \( -path "*/bin/$name" -o -path "*/bin/$name.exe" \) 2>/dev/null)
44+
EOF
45+
46+
if [ -z "$best" ]; then
47+
echo "newest_artifact: no '$name' under $dir/*/*/bin/" >&2
48+
find "$dir" -maxdepth 4 -type d -name bin >&2 2>/dev/null || true
49+
exit 1
50+
fi
51+
52+
printf '%s\n' "$best"

.github/workflows/bench.yml

Lines changed: 168 additions & 61 deletions
Large diffs are not rendered by default.

.gitmodules

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# The benchmark's independent control target, pinned twice.
2+
#
3+
# WHY SUBMODULES RATHER THAN A CLONE IN CI. The workflow used to
4+
# `git clone --depth 1` xlings' default branch at run time, which means the
5+
# benchmark target moved with every upstream push. That is the drift these
6+
# descriptions warn about, and it had already happened: `--hub src/xlings.cppm`
7+
# named a file that no longer existed, so every xlings cell in every job
8+
# reported `skipped` and the jobs stayed green.
9+
#
10+
# A submodule is a PIN, not a vendored snapshot — the commit is in the diff, it
11+
# is reviewed like any other change, and `git submodule update --init` gives
12+
# everyone the tree CI measured. Updating it is deliberate, which is the whole
13+
# point of a benchmark target.
14+
#
15+
# WHY TWO OF THE SAME REPOSITORY. They are the two code styles being compared:
16+
#
17+
# tree-2026.8.11.2 (b1563fe) 110 .cppm + 2 .cpp — implementation lives
18+
# inside each interface unit
19+
# tree-2026.8.13.1 (f072075) 110 .cppm + 92 .cpp — implementation split out
20+
#
21+
# Same authors, same 46k lines, same module graph; the question is what the
22+
# split costs or saves on an incremental build. That is the `modules` vs
23+
# `modules-impl` axis the generated fixture has, on a real tree.
24+
#
25+
# ONE description serves both (bench/projects/xlings/{CMakeLists.txt,xmake.lua}):
26+
# it globs `src/**/*.{cppm,cpp}`, which is the same rule mcpp itself infers from,
27+
# so neither style needs its own file, an environment switch, or a branch.
28+
[submodule "bench/projects/xlings/xlings-2026.8.11.2"]
29+
path = bench/projects/xlings/xlings-2026.8.11.2
30+
url = https://github.com/openxlings/xlings
31+
[submodule "bench/projects/xlings/xlings-2026.8.13.1"]
32+
path = bench/projects/xlings/xlings-2026.8.13.1
33+
url = https://github.com/openxlings/xlings

README.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -304,6 +304,37 @@ import mcpplibs.cmdline;
304304

305305
</details>
306306

307+
## Benchmark
308+
309+
mcpp is measured against cmake, xmake and bazel on the **same sources with the
310+
same compiler binary**, by a harness that lives in this repository
311+
([`bench/`](bench/)) and runs in CI across Linux, macOS and Windows.
312+
313+
<!-- BENCHMARK-TABLE:START — see bench/README.md before quoting any of this -->
314+
315+
_Filled in from the CI matrix. Every number below is a median wall-clock, taken
316+
with the pins listed in [`bench/README.md` §0](bench/README.md)._
317+
318+
<!-- BENCHMARK-TABLE:END -->
319+
320+
**What makes this comparable at all**, and what to check before quoting any of
321+
it:
322+
323+
* every engine is handed **the same compiler binary** out of mcpp's own payload
324+
(gcc 16.1.0 / clang 22.1.8), not whatever `g++` means on the runner;
325+
* the build tools are pinned — **cmake 4.4.2, xmake 3.1.0, bazel 9.2.0**
326+
and installed by xlings on every platform;
327+
* the projects are pinned as git submodules, so the target cannot drift;
328+
* **cmake is the baseline**: an absolute second count means nothing without
329+
knowing the machine, but "1.8× cmake" survives being read somewhere else.
330+
331+
There are declared asymmetries — cases where an engine is doing more or less
332+
work than another — and cells that are honestly `unavailable` or `skipped`
333+
rather than quietly zero. They are all written down.
334+
335+
📊 **[Full methodology, pinned versions and data → `bench/README.md`](bench/README.md)**
336+
· [中文](bench/README.zh-CN.md) · [what is measured → `bench/SPEC.md`](bench/SPEC.md)
337+
307338
## Platform Support
308339

309340
mcpp's identity model has two orthogonal axes: a **toolchain** is

bench/README.md

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# `bench/` — build-engine benchmark suite
22

3+
**English** · [简体中文](README.zh-CN.md)
4+
35
A cross-platform harness for measuring **build engines** against each other on
46
the **same C++ sources**, and for measuring what C++20 named modules actually
57
cost compared to headers.
@@ -40,6 +42,70 @@ keep looking right. `tests/e2e/233_bench_matrix.sh` is what keeps it that way.
4042

4143
---
4244

45+
## 0. What is pinned, and why every one of these is pinned
46+
47+
A benchmark number is only worth the list of things that were held still while
48+
it was taken. Every row below was loose at some point in this suite's short
49+
life, and every one of them produced a table that was measuring something other
50+
than what it said.
51+
52+
| what | pinned to | declared in |
53+
|---|---|---|
54+
| cmake | **4.4.2** | `matrix.json``tools` |
55+
| xmake | **3.1.0** | `matrix.json``tools` |
56+
| bazel | **9.2.0** | `matrix.json``tools` |
57+
| gcc | **16.1.0** | `bench/src/toolchain.cppm` |
58+
| clang / libc++ | **22.1.8** (Windows: 20.1.7) | `bench/src/toolchain.cppm` |
59+
| reference mcpp | **2026.8.11.3** | `matrix.json``reference_mcpp` |
60+
| xlings (combined style) | **2026.8.11.2**`b1563fe` | submodule `projects/xlings/xlings-2026.8.11.2` |
61+
| xlings (split style) | **2026.8.13.1**`f072075` | submodule `projects/xlings/xlings-2026.8.13.1` |
62+
| mcpp under test | the checkout | built by CI, resolved by `newest_artifact.sh` |
63+
64+
**Everything is installed by xlings**, at those exact versions, on every runner.
65+
`xlings install cmake@4.4.2 xmake@3.1.0 bazel@9.2.0 mcpp@2026.8.11.3` is
66+
literally what CI runs, and the job prints the resolved version of each one and
67+
warns loudly if it is not the pinned one.
68+
69+
Four things this bought, each of which had already gone wrong:
70+
71+
* **cmake 3.31.6** is what the GitHub runner images ship. It does not have the
72+
CMake 4.0 experimental key for `import std`, so *every module cell failed to
73+
configure*. With 4.4.2 they pass.
74+
* **`command -v g++`** on those images is gcc 13.3.0. cmake cannot configure
75+
C++23 modules with it and xmake crashes it with an internal compiler error —
76+
while mcpp quietly used its own registry's gcc 16.1 regardless. The table read
77+
`48 failed / 6 ok` and was still called a comparison of build engines. The
78+
suite now hands **every** engine the driver out of mcpp's own payload
79+
(`--compiler payload:gcc`), which is its fairness rule finally enforced rather
80+
than merely written down.
81+
* **The projects were cloned from their default branch at run time**, so the
82+
benchmark target moved with every upstream push. `--hub src/xlings.cppm` had
83+
been naming a file that no longer existed for months; every xlings cell
84+
reported `skipped` and every xlings job reported success. They are git
85+
submodules now, and the guard checks that each `hub`/`body` exists in the
86+
pinned tree.
87+
* **Only one mcpp was measured.** A report that says how fast this branch is,
88+
without saying whether it got faster, is not what a benchmark on a pull
89+
request is for.
90+
91+
> **Not held still, and deliberately so:** the runner hardware. See §4a.
92+
93+
> **The one case where "never writes into the measured tree" does not hold.**
94+
> The editing scenarios save a file's exact bytes and restore them however the
95+
> function exits — including on a failed build — but that is a destructor, and a
96+
> destructor does not run when the process is `SIGKILL`ed. Interrupt a
97+
> `--project` run hard enough and the perturbation is still there, which for the
98+
> pinned submodules shows up as a dirty working tree. `git submodule foreach
99+
> 'git checkout -- .'` undoes it; the perturbations are named `bench_nonce_*`,
100+
> so they are also easy to recognise in a diff.
101+
102+
> **Not exercised by these numbers:** mcpp's split build schedule
103+
> (`[build] schedule = "on"`) is opt-in until it has been verified on every
104+
> platform, so both mcpp binaries run with it off. Its effect is measured
105+
> separately in `.agents/docs/2026-08-13-build-optimization-status.md`.
106+
107+
---
108+
43109
## 1. What is measured
44110

45111
**The build engine**, i.e. the graph it constructs and the order it schedules —

0 commit comments

Comments
 (0)