Commit 44b08ed
committed
fix(bench): every cell in the matrix was failing behind a green check
`bench (macos/clang/fixture)` reported success with **6 ok / 48 failed / 18
unavailable**, and all three xlings jobs reported success having measured
nothing at all. 这批修复的是「为什么会这样」以及「为什么没人发现」。
五个独立的真因,每一个都单独足以让整份矩阵失效:
1. **每个引擎拿到的编译器不是同一个。** CI 用 `command -v g++` = runner 自带的
gcc 13.3.0,而 mcpp 一直悄悄用自己 registry 里的 gcc 16.1。cmake 配不出
C++23 modules,xmake 直接把 gcc 编崩(ICE)。套件自己的公平性规则
(`resolve_cxx`)写在注释里,但没有任何东西执行它。
→ 新增 `bench.toolchain` + `--compiler payload:gcc|payload:clang`,解析到
**mcpp 自己载荷里的驱动**,并且 fixture 的 mcpp.toml 从同一处取版本。
2. **构建工具版本随 runner 漂移。** 镜像自带 cmake 3.31.6 —— 没有 CMake 4.0 的
`import std` 实验键,所以每个 module 格子都 configure 失败。
→ cmake 4.4.2 / xmake 3.1.0 / bazel 9.2.0 全部由 xlings 按精确版本安装,
pin 写在 matrix.json,job 打印实际解析到的版本并在不符时告警。
3. **被测工程是运行时从默认分支 clone 的。** `--hub src/xlings.cppm` 指的文件
几个月前就没了 —— 每个 xlings 格子报 `skipped`,每个 job 报成功。
→ 两棵树改成 git 子模块钉住:2026.8.11.2 (`b1563fe`) 与 2026.8.13.1
(`f072075`)。
4. **`--hub`/`--body` 是按 harness 的 cwd 解析的**,不是按工程目录。只有在
「测你正站着的那棵树」时才对(mcpp 测自己),换任何工程都静默失效。
5. **harness 永远返回 0。** 时间不该设阈值(共享 runner),但「有没有测到东西」
可以。现在 `failed` 或「一个 ok 都没有」都返回非零;`unavailable`/`skipped`
是缺口,不影响退出码。真正的已知缺口写进格子的 `allow_failed`,且守卫强制
它必须带 `KNOWN GAP` 说明。
可观测性(用户报的「卡住而且没有进度」):
* 进度实时打到 stderr 并逐行 flush —— 之前一个格子只在结束时才打印,所以卡在
第三个引擎和卡在第一个引擎看起来一模一样(两个 job 各卡了 25 分钟)。
* 每条 configure/build 有超时(默认 1800s),超时 kill 并明确报
`TIMED OUT after Ns and was killed`。POSIX 用 waitpid(WNOHANG) 轮询 + SIGKILL,
Windows 用 WaitForSingleObject + TerminateProcess。
* 失败时直接打出子进程日志尾部 —— CI 上那个文件跟着 runner 一起销毁,只写
`see .../cmake-cold.log` 等于什么都没说。这一条当场找出了 xmake 的
`attempt to call a nil value (global 'bench_package_root')`。
顺带修掉的两个:
* `find target -name bench | head -1` 在本机就挑中了一个两个半小时前的旧二进制
—— target/ 每个工具链指纹一个目录。收敛到 `.github/tools/newest_artifact.sh`。
* xlings 的 xmake arm 在 `on_load` 里调 include 进来的全局函数,xmake 的沙箱看
不到 —— 那条 arm 从来没跑起来过。改成在 description scope 解析成局部变量,
与 mcpp arm 已有的写法一致。
xlings 现在是一个**代码风格对比**:同一个工程、同一张模块图、46k 行,
`f072075` 把实现从接口单元里拆了出来(110 .cppm + 2 .cpp → 110 .cppm + 92 .cpp)。
一份构建描述同时服务两棵树 —— 它 glob `src/**/*.{cppm,cpp}`,也就是 mcpp 自己
推导的那条规则,所以不需要写两份、不需要环境变量、不需要分支。
守卫(233 + bench/tests/harness.sh)新增:hub/body 必须在钉住的树里真实存在、
工具版本必须是精确版本、豁免必须命名真实引擎且带 KNOWN GAP、工程相对路径必须
从任意 cwd 解析、什么都没测到必须非零退出、超时必须真的开火。前两条分别用
「把 hub 改回 src/xlings.cppm」和「把 cmake 改成 latest」验证过会变红。1 parent 8788c67 commit 44b08ed
29 files changed
Lines changed: 1522 additions & 215 deletions
File tree
- .github
- tools
- workflows
- bench
- projects
- mcpp
- xlings
- src
- engines
- fixture
- platform
- tests
- tests/e2e
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
304 | 304 | | |
305 | 305 | | |
306 | 306 | | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
307 | 338 | | |
308 | 339 | | |
309 | 340 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
| 4 | + | |
3 | 5 | | |
4 | 6 | | |
5 | 7 | | |
| |||
40 | 42 | | |
41 | 43 | | |
42 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
43 | 109 | | |
44 | 110 | | |
45 | 111 | | |
| |||
0 commit comments