fix(bench): stop publishing a build that never happened, and fix the xmake arm's dependency

四个缺陷,都是同一个形状 —— **成功地什么也没做**。

**1. bazel 在 mcpp 工作负载上是个幽灵数字。**
`bench/projects/mcpp/BUILD.bazel` 里一条规则都没有(bazel 不能 glob 工作区外
的源码,`import std;` 也没有 bazel 写法),而 `bazel build //...` 对一个**没有
规则的包会退出 0**。于是矩阵发布了:

    bazel/clang/release/cold/mcpp-2026.8.11.3   0.43s   ok

旁边是 mcpp 的 12s 和 cmake 的 94s。每一层都「正确」:bazel 成功了,runner
计了时,报告打印了它。

- 矩阵里把 bazel 从 mcpp 格子移除,理由写进 `excluded`;
- 新增 `Engine::unbuildable_reason(Job)` —— `supports()` 只看得到 variant 和
  编译器,**看不到工程**,这正是缺口所在。bazel 用 `bazel query kind(rule,//...)`
  回答,无规则则报 `unavailable`(有理由)而不是 `ok`(有数字);
- e2e 233 加静态守卫:cell 若给 bazel 排了一个无规则的 BUILD.bazel 就红。
  **两个方向都钉过**:把 bazel 加回去,守卫确实变红。

**2. 那条「cold 必须 > 自己的 noop 两倍」的不变式抓不到它。** 它是**单引擎内部
的相对判据**,而一个什么都不编的引擎 cold 和 noop 都便宜、比值健康:0.43 对
0.22,以 0.01 秒之差擦过判据。新增**跨引擎**判据:同一 cell 里别的引擎编同一份
源码,快 20 倍就不是引擎快,是工作量小(mcpp 实测最好成绩是 3.1×,阈值远在其外)。

**3. xmake 的 mcpplibs.cmdline 版本是硬编码的,而那是「需求」不是「解析结果」。**
`mcpp.toml` 写 `= "0.0.1"` 是**版本要求**;我把它当成了解析结果写死。开发机上
恰好躺着 0.0.1 所以一直是绿的,干净 runner 上 `bench_package_root` 返回 nil,
`if` 静默地一个文件都没加,137 个单元之后死于

    missing mcpplibs.cmdline dependency for module mcpp.cli.cmd_cache

—— 指着消费者,不提版本也不提注册表。改为**读被测树自己的 mcpp.lock**(mcpp
在这棵树上实际做出的解析),两条臂由构造保证编同一份代码;读不到就 `raise`,
不再静默跳过。

⚠️ 读文件必须放进 `on_load`:描述作用域里 `io` 是 nil。这个坑
`common/xmake/payload.lua` 的注释里白纸黑字写着,我还是又踩了一次。

**4. bench.yml 的 `push:` 没限分支。** 仓库里另外 8 个 workflow 全都是
`branches: [main]`;不限的话,一个开着 PR 的分支会为**同一个 commit** 同时触发
push 和 pull_request 两个 run,而 `concurrency` 去不掉重 —— 两个事件的
`github.ref` 不同(`refs/heads/…` vs `refs/pull/N/merge`)。对一个两小时的矩阵
来说这是最贵的一种重复。

附带:`harness.sh` 的 `$MCPP` 未设置时死在 `line 22: : command not found`,
既不说变量名也不说怎么修。**不回落到 PATH** —— `command -v mcpp` 拿到的是
xlings shim,而 shim 按调用时的工作目录重新挑版本,bench 又刻意在被测树里跑
子进程,于是 cell 报 `mcpp@2026.8.11.2 | unknown command: build`(实测),
和今天早些时候 CI 上参照臂那个缺陷是同一个。改成必须显式给,并把相对路径在
`cd` **之前**转绝对;230 delegator 自己填默认值,这样手跑也成立。

Author: speak-agent

.github/workflows/bench.yml

@@ -37,6 +37,12 @@ on:
   # workflow's own artifacts land there, so including it would let a results
   # commit trigger the run that produces the next results commit.
   push:
+    # main only, matching every other workflow in this repo. Unscoped, a branch
+    # with an open PR fires BOTH this and the `pull_request` below for the same
+    # commit — two full matrices, and `concurrency` cannot dedupe them because
+    # github.ref differs between the events (refs/heads/... vs refs/pull/N/merge).
+    # For a two-hour job that is the most expensive kind of duplicate.
+    branches: [main]
     paths:
       - 'bench/**'
       - '!bench/**/*.md'

bench/matrix.json

@@ -145,7 +145,7 @@
       "os": "linux",
       "toolchain": "clang",
       "project": "mcpp-2026.8.11.3",
-      "engines": "mcpp,cmake,xmake,bazel",
+      "engines": "mcpp,cmake,xmake",
       "variants": "modules",
       "scenarios": "cold,noop,touch-hub,edit-body,edit-comment",
       "hub": "src/platform/platform.cppm",
@@ -156,7 +156,7 @@
       "os": "windows",
       "toolchain": "clang",
       "project": "mcpp-2026.8.11.3",
-      "engines": "mcpp,cmake,xmake,bazel",
+      "engines": "mcpp,cmake,xmake",
       "variants": "modules",
       "scenarios": "cold,noop,touch-hub,edit-body,edit-comment",
       "hub": "src/platform/platform.cppm",
@@ -277,6 +277,11 @@
       "toolchain": "clang",
       "project": "*",
       "reason": "KNOWN GAP, and an mcpp/xlings environment defect rather than a benchmark one. On macOS the registry's libc++ ends up on the dynamic loader's search path for EVERY child process, so Apple's own linker — which links against libc++ — resolves against the payload copy and aborts before linking: `dyld: Symbol not found: __ZdaPv, Referenced from: .../XcodeDefault.xctoolchain/usr/bin/ld, Expected in: .../registry/.../lib/libc++.1.0.dylib`. cmake, bazel and the reference mcpp all fail identically while the mcpp under test passes, which is what identifies it as environmental. Same symbol and mechanism as the build.mcpp helper failure in .agents/docs/2026-08-13-build-optimization-status.md S9a. Removing the payload libc++ flags on macOS did NOT fix it, so the contamination arrives through DYLD_* rather than through link flags. The cells return once that is understood — deliberately not guessed at from a machine that cannot reproduce it"
+    },
+    {
+      "project": "mcpp-*",
+      "engine": "bazel",
+      "reason": "bazel cannot build this workload: it will not glob sources from outside its workspace, and `import std;` has no bazel spelling. bench/projects/mcpp/BUILD.bazel therefore declares no rules — and `bazel build //...` over a package with no rules EXITS 0 having compiled nothing, which the matrix published as `bazel cold 0.43s` beside mcpp's 12s and cmake's 94s. Removed here, and the bazel adapter now reports `unavailable` for a ruleless package so the next one cannot be published as a measurement."
     }
   ],
   "_workload_note": [

bench/projects/mcpp/xmake.lua

@@ -64,11 +64,29 @@ end
 MCPP_ROOT = path.normalize(MCPP_ROOT)
 local MCPP_MANIFEST = path.join(MCPP_ROOT, "mcpp.toml")
 
--- mcpp.toml pins mcpplibs.cmdline = "0.0.1" exactly; newer versions may also be
--- unpacked in the registry, so pin rather than take the newest or the two builds
--- would not be compiling the same code.
-local CMDLINE_VER = "0.0.1"
-local CMDLINE_SRC = bench_package_root("mcpplibs-x-cmdline", CMDLINE_VER)
+-- Which version of mcpplibs.cmdline to compile is read from the measured tree's
+-- own mcpp.lock, NOT hardcoded and NOT "the newest unpacked".
+--
+--   * hardcoding it is what broke CI: this said "0.0.1" because mcpp.toml says
+--     `mcpplibs.cmdline = "0.0.1"`, but that is a REQUIREMENT, not a resolution.
+--     A developer box that had 0.0.1 unpacked from some earlier run worked; a
+--     fresh runner had only what mcpp resolved, `bench_package_root` returned
+--     nil, the `if` below quietly added no files, and the build died 137 units
+--     later with `missing mcpplibs.cmdline dependency for module mcpp.cli` —
+--     an error that names neither the version nor the registry.
+--   * "the newest unpacked" would silently compile different sources than mcpp
+--     did, which is the one thing a comparison arm may not do.
+--
+-- The lockfile is the resolution mcpp itself performed on this exact tree, so
+-- both arms compile the same code by construction.
+-- Read inside on_load, not here: `io` is nil in xmake's DESCRIPTION scope, so a
+-- reader written at this level dies with `attempt to index a nil value (global
+-- 'io')` — the same trap ../common/xmake/payload.lua documents for its manifest
+-- reader, walked into a second time. Both the lock path and the registry root
+-- are captured as upvalues for the same reason: on_load's sandbox cannot see
+-- this file's globals, so `bench_package_root` is not callable from in there.
+local MCPP_LOCK = path.join(MCPP_ROOT, "mcpp.lock")
+local XPKGS     = bench_xpkgs()
 
 option("pin_payload")
     set_default(true)
@@ -102,9 +120,39 @@ target("mcpp")
     -- has no such cache, so it compiles the 3 units from source. That is a ~1s
     -- handicap on xmake's cold build and is called out in the benchmark report
     -- rather than hidden.
-    if CMDLINE_SRC and os.isdir(path.join(CMDLINE_SRC, "src")) then
-        add_files(path.join(CMDLINE_SRC, "src", "*.cppm"))
-    end
+    --
+    -- Absence is FATAL rather than skipped. Skipping produced a build missing
+    -- three units out of 140 that announced itself only as
+    -- `missing mcpplibs.cmdline dependency for module mcpp.cli.cmd_cache` —
+    -- naming a consumer instead of the cause. A description that cannot name the
+    -- same sources mcpp compiled is not a comparison arm.
+    on_load(function (target)
+        local ver
+        if os.isfile(MCPP_LOCK) then
+            local in_section = false
+            for _, line in ipairs((io.readfile(MCPP_LOCK) or ""):split("\n", {plain = true})) do
+                local section = line:match("^%s*%[(.-)%]")
+                if section then in_section = (section == 'package."mcpplibs.cmdline"')
+                elseif in_section then
+                    ver = ver or line:match('^%s*version%s*=%s*"([^"]+)"')
+                end
+            end
+        end
+        if not ver then
+            raise("bench: cannot read mcpplibs.cmdline's resolved version from " .. MCPP_LOCK
+                  .. " — the measured tree must carry the lockfile mcpp resolved it with")
+        end
+        local base = path.join(XPKGS, "mcpplibs-x-cmdline", ver)
+        local dirs = os.isdir(base) and os.dirs(path.join(base, "*")) or {}
+        table.sort(dirs)
+        local src = dirs[1] and path.join(dirs[1], "src")
+        if not src or not os.isdir(src) then
+            raise("bench: mcpplibs.cmdline " .. ver .. " is not unpacked under " .. XPKGS
+                  .. " — build the tree with mcpp once first, so both arms compile the "
+                  .. "same dependency sources")
+        end
+        target:add("files", path.join(src, "*.cppm"))
+    end)
 
     set_policy("build.c++.modules", true)
     set_policy("build.c++.modules.std", true)

bench/src/engines/bazel.cppm

@@ -62,6 +62,34 @@ public:
         return {0.0, 0};   // MODULE.bazel/BUILD are the configuration
     }
 
+    // `bazel build //...` over a package with no rules is a SUCCESS that compiles
+    // nothing, in about 0.2s. Ask bazel itself what it is about to build rather
+    // than reading the BUILD file here — a hand-rolled rule detector would be one
+    // more parser to keep in step with the file it parses.
+    std::string unbuildable_reason(const Job& job) const override {
+        platform::RunResult r;
+        const auto out = platform::run_capture(
+            {"bazel", "query", "kind(rule, //...)", "--noshow_progress"},
+            job.buildfile_dir, &r);
+        // A query that ERRORS is not "no targets" — a broken query is a real
+        // failure, and it belongs in the build where the log gets reported.
+        if (!out || !r.ok()) return {};
+        // The capture is stdout+stderr combined, so presence is tested on the one
+        // token bazel never prints by accident: a target label at line start.
+        for (std::string_view rest = *out; !rest.empty();) {
+            const auto nl   = rest.find('\n');
+            const auto line = rest.substr(0, nl);
+            if (line.starts_with("//")) return {};
+            if (nl == std::string_view::npos) break;
+            rest.remove_prefix(nl + 1);
+        }
+        return std::format(
+            "{}/BUILD.bazel declares no rules, so `bazel build //...` would exit 0 "
+            "having compiled nothing and report a ~0.2s 'build'. bazel cannot glob "
+            "sources from outside its workspace and has no spelling for `import std;`",
+            job.buildfile_dir.filename().string());
+    }
+
     platform::RunResult build(const Job& job) const override {
         std::vector<std::string> argv{"bazel", "build", "//..."};
         if (job.jobs > 0) argv.push_back(std::format("--jobs={}", job.jobs));

bench/src/engines/engine.cppm

@@ -50,6 +50,22 @@ public:
     // explains itself without a reader consulting this source.
     virtual std::string unsupported_reason(Variant v, std::string_view compiler) const = 0;
 
+    // "This engine cannot build THIS PROJECT" — the question `supports()` cannot
+    // ask, because it only sees the variant and the compiler.
+    //
+    // The gap was not theoretical. bench/projects/mcpp/BUILD.bazel declares no
+    // targets at all (bazel will not glob sources from outside its workspace, and
+    // `import std;` has no bazel spelling), so `bazel build //...` succeeded
+    // having built nothing, and the cell was published as
+    //     bazel/clang/release/cold/mcpp-2026.8.11.3  0.43s
+    // next to mcpp's 12s and cmake's 94s. Every layer behaved correctly on its
+    // own: bazel exited 0, the runner timed it, the report printed it.
+    //
+    // Returning a non-empty reason marks the cell `unavailable` — a documented
+    // gap — instead of `ok` with a number that is off by two orders of magnitude.
+    // Empty (the default) means "nothing project-specific stops me".
+    virtual std::string unbuildable_reason(const Job&) const { return {}; }
+
     // Does `compiler` resolve to a clang driver? Several engines' module
     // support is clang-only today.
     static bool is_clang(std::string_view compiler) {

bench/src/engines/xmake.cppm

@@ -51,7 +51,7 @@ public:
     // ⚠️ EVERY COMMAND RUNS FROM `buildfile_dir`, i.e. the `-P` directory, and
     // that is load-bearing rather than tidiness.
     //
-    // xmake normalises `--buildir` (`-o`) to a path RELATIVE TO THE PROJECT
+    // xmake normalises `--builddir` (`-o`) to a path RELATIVE TO THE PROJECT
     // DIRECTORY, then resolves that relative path against the process's cwd when
     // it builds. Run it from anywhere other than `-P` and the two disagree. With
     // `-P bench/projects/mcpp` and `-o <workload>/build`, running from

bench/src/main.cpp

@@ -563,6 +563,41 @@ int main(int argc, char** argv) {
         }
     }
 
+    // --- cross-engine consistency: a cold build cannot be 20x cheaper than every
+    //     other engine building the same sources ---
+    //
+    // The check above is RELATIVE TO ONE ENGINE, so it is blind to the case that
+    // actually shipped: an engine that builds NOTHING has a cheap cold AND a
+    // cheap noop, and their ratio looks healthy. bazel on the pinned mcpp tree
+    // reported cold=0.43s / noop=0.22s — a ratio of 1.95, missing the 2x trip
+    // wire by one hundredth of a second — while compiling zero of 137 units.
+    //
+    // Peers are the honest yardstick here, and they are already in the report:
+    // engines in the same cell compile the same sources on the same machine, so
+    // a 20x gap is not a fast engine, it is a different workload. The factor is
+    // deliberately far past any real result (mcpp's best measured win over cmake
+    // is 3.1x) so this fires on phantoms and never on a good number.
+    for (const auto& c : report.cells) {
+        if (c.status != bench::Status::Ok || c.key.scenario != "cold") continue;
+        std::vector<double> peers;
+        for (const auto& p : report.cells)
+            if (p.status == bench::Status::Ok && p.key.scenario == "cold"
+                && p.key.fixture == c.key.fixture && p.key.variant == c.key.variant
+                && p.key.engine != c.key.engine && p.median_s() > 0.0)
+                peers.push_back(p.median_s());
+        if (peers.empty() || c.median_s() <= 0.0) continue;
+        std::ranges::sort(peers);
+        const double peer_median = peers[peers.size() / 2];
+        if (c.median_s() * 20.0 < peer_median) {
+            ++suspect;
+            std::println(std::cerr,
+                         "bench: {} reports cold={:.2f}s while other engines building the same "
+                         "sources take {:.2f}s — {:.0f}x is not a faster engine, it is a smaller "
+                         "workload; check that this engine's description actually names the sources.",
+                         c.key.str(), c.median_s(), peer_median, peer_median / c.median_s());
+        }
+    }
+
     std::size_t ok = 0, failed = 0, waived = 0;
     for (const auto& c : report.cells) {
         if (c.status == bench::Status::Ok) { ++ok; continue; }

bench/src/runner.cppm

@@ -288,6 +288,15 @@ public:
                                    job.log_path.filename().string(), tail));
         };
 
+        // Asked once the Job exists, because the answer depends on the PROJECT —
+        // which is why it cannot live beside the `supports()` check above.
+        if (auto why = engine.unbuildable_reason(job); !why.empty()) {
+            cell.status = Status::Unavailable;
+            cell.note   = std::move(why);
+            report(cell.note);
+            return cell;
+        }
+
         report("configure");
         if (const auto cfg = engine.configure(job); !cfg.ok()) {
             fail("configure", cfg);

bench/tests/harness.sh

@@ -18,7 +18,35 @@ TMP=$(mktemp -d)
 # directory is not a test result.
 trap "rm -rf $TMP || true" EXIT
 
+# mcpp's e2e runner exports MCPP as the binary under test; a standalone run has
+# nothing to inherit and used to die on `line 22: : command not found`, which
+# names neither the variable nor the fix.
+#
+# NOT defaulted to PATH. `command -v mcpp` returns the xlings SHIM, and a shim
+# re-resolves which mcpp to exec from the workspace it is invoked in — while
+# bench deliberately runs every engine with its cwd inside the tree under test.
+# The shim then picked a different mcpp than the one being tested and the cells
+# failed as `mcpp@2026.8.11.2 | unknown command: build`, naming a version nobody
+# asked for. This is the same defect as the reference-mcpp arm on CI; the rule
+# from bench/README §4 is absolute: an engine is named by BINARY, never by PATH.
+[ -n "${MCPP:-}" ] || {
+  echo "FAIL: MCPP is unset. Set it to a REAL mcpp binary, not a PATH name:"
+  echo "        MCPP=\$(bash .github/tools/newest_artifact.sh . mcpp) bash bench/tests/harness.sh"
+  echo "      (mcpp's e2e runner exports it; \`bash tests/e2e/230_bench_harness.sh\` works too.)"
+  exit 1
+}
+# Absolutised against the INVOKING cwd, which is why this runs before the `cd`
+# below: `MCPP=./target/.../mcpp` is what every natural way of producing the
+# path yields, and a relative one silently stops resolving the moment the script
+# changes directory — including the command this very message suggests.
+case "$MCPP" in
+  /*|?:[/\\]*) ;;
+  *) MCPP="$PWD/$MCPP" ;;
+esac
+[ -x "$MCPP" ] || { echo "FAIL: MCPP=$MCPP is not an executable file"; exit 1; }
+
 cd "$REPO/bench"
+echo "harness built with: $("$MCPP" --version 2>&1 | head -1) ($MCPP)"
 "$MCPP" build > /dev/null
 
 # NEWEST, not `find | head -1`: target/ holds one directory per toolchain

tests/e2e/230_bench_harness.sh

@@ -7,7 +7,19 @@
 # directory would make that separation a rename away from breaking.
 #
 # This delegator stays because deleting it would silently drop the harness from
-# every mcpp PR: `bench.yml` is workflow_dispatch-only, so nothing else runs it.
+# every mcpp PR that does not touch bench/: `bench.yml` is PATH-SCOPED to
+# `bench/**`, so a change elsewhere that breaks the suite runs nothing.
 set -e
 REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+
+# The e2e runner exports MCPP as the binary under test and that always wins.
+# Filling it in when unset is what makes this script runnable BY HAND, which the
+# harness cannot do for itself: it must not fall back to PATH (that resolves to
+# the xlings shim, which re-picks a version per working directory), but this
+# delegator lives in mcpp's own tree and can simply point at what was built.
+if [ -z "${MCPP:-}" ]; then
+  MCPP="$(bash "$REPO/.github/tools/newest_artifact.sh" "$REPO" mcpp 2>/dev/null || true)"
+  [ -n "$MCPP" ] || { echo "SKIP: no mcpp binary built yet — run \`mcpp build\` first"; exit 0; }
+  export MCPP
+fi
 exec bash "$REPO/bench/tests/harness.sh"