Skip to content

Commit d2944dd

Browse files
speak-agentspeak-agent
committed
refactor(bench): 移除 meson,补齐 xlings 的 xmake/bazel,工具链逻辑收进 common/
三件事,一个方向:让「工具链」真的成为一个轴,而不是一个标签。 **1. 移除 meson。** meson 1.10.2 没有任何属性能声明一个 TU 是模块**接口**单元, 把 .cppm 当普通源码编,第一个导入者就 `fatal error: module 'x' not found`; `import std;` 也没有对应物。于是每个 module 格子都是同一条 `unavailable`, 一份报告里一行有用五行空。一个无法表达被测对象的引擎不是对照点。 引擎实现、构建描述、fixture 发射器一并删掉;为什么不测记在 SPEC.md 里, 哪天 meson 长出这个特性,diff 就是把 `engines/meson.cppm` 加回来 + registry 一行。 **2. bench/projects/xlings 补齐。** 之前只有 cmake。现在: | 引擎 | 状态 | |---|---| | cmake | 配置通过、110 个单元全编译,链接不过(依赖以源码形式来) | | xmake | 同样形状同样缺口 | | bazel | **不能** —— workspace 边界,而且这棵树根本不在仓库里(理由写在 MODULE.bazel) | **3. 工具链逻辑收进 `bench/projects/common/`。** 两个工程各有一份近乎相同的 payload 代码,而且**已经在分叉**(一份学会了 「CMake 自己生成 std 目标、directory-scope 选项够不着它」,另一份没有)。 两份工具链定义是最不该有副本的地方:差一个 flag,基准就把**两份描述的差别** 当成引擎结果报出来。 - `common/cmake/hermetic_payload.cmake` - `common/xmake/payload.lua` 两者都是**按编译器家族分支**,而不是一整块 flag: gcc 用 `-B<binutils>` + `--sysroot`;clang 用它自己的 include 链 (**不是** `--sysroot` —— 把 gcc 的 payload 递给 clang 会让两条臂用不同的 libc, 和 CMake 那侧记录的 `_IO_FILE::_unused2` 报错是同一类,而报错既不指 flag 也不指目标); msvc 什么都不加,因为 mcpp 用的也是系统 Visual Studio。 且编译器**不在 registry 里时一律不加** —— 那是调用方明确选择宿主世界, 和 mcpp 自己的 hermetic 检查同一条规则。 ⚠️ xmake 有两个作用域、两个不同的缺口,踩了一遍才写下来: - **description 作用域没有 `io`** ⇒ 顶层读 mcpp.toml 直接 `attempt to index a nil value (global 'io')`; - **`on_load` 的沙箱看不见本文件的全局** ⇒ `attempt to call a nil value`; - 而且**闭包带的是定义时的环境**,所以在外面定义一个 local 读取器,在 on_load 里 调用时 `io` 仍然是 nil。结论:读文件的代码只能写在 on_load 里面,一份文件里写两遍。 ⚠️ 新记一个缺口(matrix.json 里按 engine 作用域排除,不删格子): xmake+clang 找 libc++ 的 std 模块靠 `lib/libc++.modules.json`,而 mcpp 的 llvm 载荷 只有 `share/libc++/v1/std.cppm`。xmake 会警告 `std and std.compat modules not found` 然后 `build.c++.modules.std` **静默降级** —— 那条臂就变成「不用 import std 的工程」 去比「用 import std 的工程」。格子照跑,但报告里带着这条注解。 另外把 `embed_lua_stdlib.cmake` 从**抄来的 11 条列表**改成**规则** (`src/lua-stdlib` 下每个 .lua,变量名 = 文件名 + `_lua`,对 libxpkg 0.0.57 核过: 11 个文件 11 个嵌入、名字一致)。那份抄来的列表已经漂移过一次,漏了 `base64_lua`, 失败出现在三个文件之外的消费者里。规则不会和自己漂移,输出的副本会。
1 parent d34e9ed commit d2944dd

20 files changed

Lines changed: 922 additions & 410 deletions

‎.github/workflows/bench.yml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -173,7 +173,6 @@ jobs:
173173
set -uo pipefail
174174
xlings install bazel -y || echo "bazel unavailable on this runner"
175175
xlings install xmake -y || echo "xmake unavailable on this runner"
176-
python3 -m pip install --quiet meson || echo "meson unavailable on this runner"
177176
cmake --version || true
178177
ninja --version || true
179178

‎bench/README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ only run on Linux.
1010

1111
```bash
1212
# generated fixtures, across engines and source forms
13-
bench --engines mcpp,cmake,xmake,meson,bazel \
13+
bench --engines mcpp,cmake,xmake,bazel \
1414
--variants headers,modules,modules-impl \
1515
--scenarios cold,noop,touch-hub,edit-body \
1616
--compiler /path/to/g++ --jobs 32 --out report.json
@@ -117,7 +117,7 @@ so that "no flags" and `--preset standard` cannot mean different things.
117117

118118
| # | Invariant | How it is enforced |
119119
|---|---|---|
120-
| I1 | Identical compiler **binary** across engines | `--compiler <path>` is threaded into cmake (`-DCMAKE_CXX_COMPILER`), meson & xmake (`CXX`), bazel (`CC` + `--action_env`). mcpp uses its hermetic payload — a **declared asymmetry**, see §5. |
120+
| I1 | Identical compiler **binary** across engines | `--compiler <path>` is threaded into cmake (`-DCMAKE_CXX_COMPILER`), xmake (`CXX`), bazel (`CC` + `--action_env`). mcpp uses its hermetic payload — a **declared asymmetry**, see §5. |
121121
| I0 | Optimisations are measured, never emulated | Engines are parameterised by BINARY (`mcpp=<path>`). The harness contains no "what if we also set X" mode: emulating a change measures the harness's idea of it and silently stops tracking the implementation. |
122122
| I2 | Identical source set | All variants come from one generator; no engine globs its own inputs. |
123123
| I3 | Identical language level | C++23 everywhere; `import std;` is **absent from every fixture** (see §5). |
@@ -182,7 +182,7 @@ when the build fails, which is precisely when a leftover edit would be missed.
182182

183183
Two details that are easy to get wrong and change the answer:
184184

185-
* **`cold` includes configure.** cmake and meson keep configure output inside the
185+
* **`cold` includes configure.** cmake keeps its configure output inside the
186186
build directory that `clean` removes, so building without re-configuring simply
187187
fails. Timing configure separately would also be wrong: the user waits for both,
188188
and engines that fold configure into the build (mcpp, bazel) would get a
@@ -422,7 +422,8 @@ the original analysis:
422422

423423
| target | what it is for |
424424
|---|---|
425-
| [`mcpp/`](projects/mcpp/) | mcpp building itself, with cmake/xmake/meson/bazel descriptions beside it |
425+
| [`mcpp/`](projects/mcpp/) | mcpp building itself, with cmake/xmake/bazel descriptions beside it |
426+
| [`common/`](projects/common/) | the per-engine payload logic both projects share — one branch per compiler family |
426427
| [`xlings/`](projects/xlings/) | an **independent** codebase (110 modules / 46k lines, different authors) — the control that separates "a faster build engine" from "a faster benchmark target" |
427428

428429
⚠️ **An engine change that only helps the project it was developed on is not an

‎bench/SPEC.md‎

Lines changed: 32 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ which the report records in its run facts.
2222
|---|---|---|
2323
| **OS** | `linux` `macos` `windows` | one CI job each |
2424
| **Toolchain** | `gcc` `clang` `msvc` | one CI job each — `--compiler` |
25-
| **Build tool** | `mcpp` `cmake` `xmake` `meson` `bazel` | swept inside a job — `--engines` |
25+
| **Build tool** | `mcpp` `cmake` `xmake` `bazel` | swept inside a job — `--engines` |
2626
| **Project** | `fixture` `mcpp` `xlings` | one CI job each — `--project` |
2727
| **Variant** | `headers` `modules` `modules-impl` | swept inside a job — `--variants` |
2828
| **Scenario** | `cold` `noop` `touch-hub` `touch-leaf` `edit-body` `edit-comment` | swept inside a job — `--scenarios` |
@@ -55,7 +55,7 @@ of one graph shape:
5555
`--weight`). The only project where `headers` / `modules` / `modules-impl`
5656
all exist, so it is the only place the *variant* axis means anything.
5757
* **`mcpp`** — 138 modules / 57k lines, one source dependency, build
58-
descriptions for all five engines under `projects/mcpp/`.
58+
descriptions for every engine under `projects/mcpp/`.
5959
* **`xlings`** — 110 modules / 46k lines, **different authors**. This is the one
6060
that separates "a faster build engine" from "a faster benchmark target".
6161

@@ -86,6 +86,19 @@ that cannot be compared to anything.
8686

8787
---
8888

89+
### meson is not an engine here
90+
91+
meson 1.10.2 has no way to declare a translation unit to be a module
92+
**interface**. Listing `.cppm` files as ordinary sources compiles them as plain
93+
TUs and the first importer fails with `fatal error: module 'x' not found`, and
94+
there is no `import std;` equivalent either. So every module cell was an
95+
`unavailable` row with the same reason — one honest row and five empty ones per
96+
report, which is noise rather than a comparison. It was removed: engine,
97+
descriptions and fixture emitter.
98+
99+
The day meson grows the feature, the diff is adding
100+
`bench/src/engines/meson.cppm` back and one line in `registry.cppm`.
101+
89102
## 3. A cell may be undefined, and it must say why
90103

91104
Three outcomes are distinguishable in the result schema, and collapsing them is
@@ -104,10 +117,11 @@ seconds" must never render the same way, and neither must "not installed" and
104117

105118
The same rule applies one level up, to cells that CI does not run at all:
106119
[`matrix.json`](matrix.json) carries an `excluded` list where every entry has a
107-
`reason`. Two of those reasons currently say **KNOWN GAP** — `windows`+`gcc`,
108-
and `xlings` on Windows. Those are meaningful cells that are simply not wired
109-
up; they are written down so that "not measured" cannot quietly become "not
110-
applicable".
120+
`reason`, and the ones that say **KNOWN GAP** are meaningful cells that are
121+
simply not wired up rather than cells that make no sense — written down so that
122+
"not measured" cannot quietly become "not applicable". An exclusion may also
123+
name an `engine`, which scopes a caveat to one COLUMN instead of removing the
124+
job: the cell still runs, and its note says what to distrust.
111125

112126
---
113127

@@ -135,6 +149,18 @@ real tree does not.
135149

136150
---
137151

152+
### Shared build descriptions
153+
154+
`projects/common/` holds the parts every arm needs: `cmake/hermetic_payload.cmake`
155+
and `xmake/payload.lua`. Both answer one question — "make this engine drive the
156+
same process tree mcpp does" — and both are **one branch per compiler family**,
157+
because that is what makes the toolchain a real axis rather than a label.
158+
159+
They exist because the two projects had two copies of it, and the copies were
160+
already diverging. Two copies of a toolchain definition is the worst place for a
161+
copy: they drift by one flag and the benchmark reports the difference between the
162+
two *descriptions* as an engine result.
163+
138164
## 5. What CI runs
139165

140166
`.github/workflows/bench.yml` plans one job per entry in `matrix.json.cells`,

‎bench/matrix.json‎

Lines changed: 110 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -10,141 +10,204 @@
1010
"A cell is one CI job. Inside it the harness sweeps every engine x variant x",
1111
"scenario, so those axes are per-cell lists rather than more jobs: they share a",
1212
"checkout, a toolchain install and a fixture, and splitting them would multiply",
13-
"runner minutes without adding a single measurement."
13+
"runner minutes without adding a single measurement.",
14+
"meson is deliberately absent, not missing: meson 1.10.2 has no way to declare a translation unit to be a module INTERFACE, so every module cell was an `unavailable` row. An engine that cannot express the thing being measured is not a comparison point, and keeping it produced one honest row and five empty ones per report."
1415
],
15-
1616
"baseline": "cmake",
1717
"_baseline_note": "Every ratio in every report is against cmake. See SPEC.md S2.",
18-
1918
"axes": {
20-
"os": ["linux", "macos", "windows"],
21-
"toolchain": ["gcc", "clang", "msvc"],
22-
"engine": ["mcpp", "cmake", "xmake", "meson", "bazel"],
23-
"project": ["fixture", "mcpp", "xlings"],
24-
"variant": ["headers", "modules", "modules-impl"],
25-
"scenario": ["cold", "noop", "touch-hub", "touch-leaf", "edit-body", "edit-comment"]
19+
"os": [
20+
"linux",
21+
"macos",
22+
"windows"
23+
],
24+
"toolchain": [
25+
"gcc",
26+
"clang",
27+
"msvc"
28+
],
29+
"engine": [
30+
"mcpp",
31+
"cmake",
32+
"xmake",
33+
"bazel"
34+
],
35+
"project": [
36+
"fixture",
37+
"mcpp",
38+
"xlings"
39+
],
40+
"variant": [
41+
"headers",
42+
"modules",
43+
"modules-impl"
44+
],
45+
"scenario": [
46+
"cold",
47+
"noop",
48+
"touch-hub",
49+
"touch-leaf",
50+
"edit-body",
51+
"edit-comment"
52+
]
2653
},
27-
2854
"runners": {
29-
"linux": "ubuntu-24.04",
30-
"macos": "macos-14",
55+
"linux": "ubuntu-24.04",
56+
"macos": "macos-14",
3157
"windows": "windows-2022"
3258
},
33-
3459
"cells": [
3560
{
36-
"os": "linux", "toolchain": "gcc", "project": "fixture",
37-
"engines": "mcpp,cmake,xmake,meson",
61+
"os": "linux",
62+
"toolchain": "gcc",
63+
"project": "fixture",
64+
"engines": "mcpp,cmake,xmake",
3865
"variants": "headers,modules,modules-impl",
3966
"scenarios": "cold,noop,touch-hub,touch-leaf,edit-body,edit-comment",
4067
"preset": "standard",
4168
"note": "bazel omitted: its module support requires a clang driver (see the bazel engine's unsupported_reason)"
4269
},
4370
{
44-
"os": "linux", "toolchain": "clang", "project": "fixture",
45-
"engines": "mcpp,cmake,xmake,meson,bazel",
71+
"os": "linux",
72+
"toolchain": "clang",
73+
"project": "fixture",
74+
"engines": "mcpp,cmake,xmake,bazel",
4675
"variants": "headers,modules,modules-impl",
4776
"scenarios": "cold,noop,touch-hub,touch-leaf,edit-body,edit-comment",
4877
"preset": "standard"
4978
},
5079
{
51-
"os": "macos", "toolchain": "clang", "project": "fixture",
52-
"engines": "mcpp,cmake,xmake,meson,bazel",
80+
"os": "macos",
81+
"toolchain": "clang",
82+
"project": "fixture",
83+
"engines": "mcpp,cmake,xmake,bazel",
5384
"variants": "headers,modules,modules-impl",
5485
"scenarios": "cold,noop,touch-hub,touch-leaf,edit-body,edit-comment",
5586
"preset": "standard"
5687
},
5788
{
58-
"os": "windows", "toolchain": "clang", "project": "fixture",
59-
"engines": "mcpp,cmake,xmake,meson,bazel",
89+
"os": "windows",
90+
"toolchain": "clang",
91+
"project": "fixture",
92+
"engines": "mcpp,cmake,xmake,bazel",
6093
"variants": "headers,modules,modules-impl",
6194
"scenarios": "cold,noop,touch-hub,touch-leaf,edit-body,edit-comment",
6295
"preset": "standard"
6396
},
6497
{
65-
"os": "windows", "toolchain": "msvc", "project": "fixture",
66-
"engines": "mcpp,cmake,xmake,meson",
98+
"os": "windows",
99+
"toolchain": "msvc",
100+
"project": "fixture",
101+
"engines": "mcpp,cmake,xmake",
67102
"variants": "headers,modules,modules-impl",
68103
"scenarios": "cold,noop,touch-hub,touch-leaf,edit-body,edit-comment",
69104
"preset": "standard",
70105
"note": "bazel omitted: same clang-driver requirement as the gcc cell"
71106
},
72-
73107
{
74-
"os": "linux", "toolchain": "gcc", "project": "mcpp",
75-
"engines": "mcpp,cmake,xmake,meson",
108+
"os": "linux",
109+
"toolchain": "gcc",
110+
"project": "mcpp",
111+
"engines": "mcpp,cmake,xmake",
76112
"variants": "native",
77113
"scenarios": "cold,noop,touch-hub,edit-body,edit-comment",
78114
"note": "touch-leaf omitted: a real tree has no unit nobody imports that is also stable enough to name"
79115
},
80116
{
81-
"os": "linux", "toolchain": "clang", "project": "mcpp",
82-
"engines": "mcpp,cmake,xmake,meson,bazel",
117+
"os": "linux",
118+
"toolchain": "clang",
119+
"project": "mcpp",
120+
"engines": "mcpp,cmake,xmake,bazel",
83121
"variants": "native",
84122
"scenarios": "cold,noop,touch-hub,edit-body,edit-comment"
85123
},
86124
{
87-
"os": "macos", "toolchain": "clang", "project": "mcpp",
88-
"engines": "mcpp,cmake,xmake,meson,bazel",
125+
"os": "macos",
126+
"toolchain": "clang",
127+
"project": "mcpp",
128+
"engines": "mcpp,cmake,xmake,bazel",
89129
"variants": "native",
90130
"scenarios": "cold,noop,touch-hub,edit-body,edit-comment"
91131
},
92132
{
93-
"os": "windows", "toolchain": "clang", "project": "mcpp",
94-
"engines": "mcpp,cmake,xmake,meson,bazel",
133+
"os": "windows",
134+
"toolchain": "clang",
135+
"project": "mcpp",
136+
"engines": "mcpp,cmake,xmake,bazel",
95137
"variants": "native",
96138
"scenarios": "cold,noop,touch-hub,edit-body,edit-comment"
97139
},
98-
99140
{
100-
"os": "linux", "toolchain": "gcc", "project": "xlings",
101-
"engines": "mcpp,cmake",
141+
"os": "linux",
142+
"toolchain": "gcc",
143+
"project": "xlings",
144+
"engines": "mcpp,cmake,xmake",
102145
"variants": "native",
103146
"scenarios": "cold,noop,touch-hub,edit-body,edit-comment",
104147
"note": "mcpp-vs-mcpp is the point here (see projects/xlings/README.md); the cmake arm compiles all 110 units and is expected to stop at the link"
105148
},
106149
{
107-
"os": "linux", "toolchain": "clang", "project": "xlings",
108-
"engines": "mcpp,cmake",
150+
"os": "linux",
151+
"toolchain": "clang",
152+
"project": "xlings",
153+
"engines": "mcpp,cmake,xmake",
109154
"variants": "native",
110155
"scenarios": "cold,noop,touch-hub,edit-body,edit-comment"
111156
},
112157
{
113-
"os": "macos", "toolchain": "clang", "project": "xlings",
114-
"engines": "mcpp,cmake",
158+
"os": "macos",
159+
"toolchain": "clang",
160+
"project": "xlings",
161+
"engines": "mcpp,cmake,xmake",
115162
"variants": "native",
116163
"scenarios": "cold,noop,touch-hub,edit-body,edit-comment"
117164
}
118165
],
119-
120166
"excluded": [
121167
{
122-
"os": "macos", "toolchain": "gcc",
168+
"os": "macos",
169+
"toolchain": "gcc",
123170
"reason": "no gcc payload exists for macOS in mcpp's registry, and a Homebrew gcc would make the cell a comparison of distributions rather than of engines"
124171
},
125172
{
126-
"os": "windows", "toolchain": "gcc",
173+
"os": "windows",
174+
"toolchain": "gcc",
127175
"reason": "mcpp supports x86_64-windows-gnu, but the other four engines would each have to be pointed at an msys2 gcc that mcpp does not use. KNOWN GAP, not a decision: the cell is meaningful and is simply not wired up yet"
128176
},
129177
{
130-
"os": "linux", "toolchain": "msvc",
178+
"os": "linux",
179+
"toolchain": "msvc",
131180
"reason": "msvc is Windows-only"
132181
},
133182
{
134-
"os": "macos", "toolchain": "msvc",
183+
"os": "macos",
184+
"toolchain": "msvc",
135185
"reason": "msvc is Windows-only"
136186
},
137187
{
138-
"os": "windows", "toolchain": "msvc", "project": "mcpp",
188+
"os": "windows",
189+
"toolchain": "msvc",
190+
"project": "mcpp",
139191
"reason": "mcpp's own schedule policy reports msvc as unmeasured (src/build/schedule/policy.cppm), so the mcpp arm would be measuring a shape nobody has validated"
140192
},
141193
{
142-
"os": "windows", "toolchain": "msvc", "project": "xlings",
194+
"os": "windows",
195+
"toolchain": "msvc",
196+
"project": "xlings",
143197
"reason": "same as the mcpp project on msvc"
144198
},
145199
{
146-
"os": "windows", "toolchain": "clang", "project": "xlings",
200+
"os": "windows",
201+
"toolchain": "clang",
202+
"project": "xlings",
147203
"reason": "xlings has not been shown to build on Windows at all. KNOWN GAP: verify the plain build first, then add the cell — a bench cell that cannot build is not a measurement"
204+
},
205+
{
206+
"os": "*",
207+
"toolchain": "clang",
208+
"project": "*",
209+
"engine": "xmake",
210+
"reason": "KNOWN GAP, xmake+clang only: xmake locates libc++'s std module through lib/libc++.modules.json, which mcpp's llvm payload does not ship (it has share/libc++/v1/std.cppm). xmake warns 'std and std.compat modules not found' and build.c++.modules.std degrades SILENTLY — the arm would then measure a project without `import std;` against ones with it. The cell still runs; read its note before quoting the number"
148211
}
149212
]
150213
}

0 commit comments

Comments
 (0)