Skip to content

Commit 733279d

Browse files
speak-agentspeak-agent
committed
feat(bench): 矩阵规范 SPEC.md + matrix.json —— OS × 工具链 × 引擎 × 工程 × 场景
原来的 bench CI 只在 **OS** 一个轴上展开(linux/macos/windows),引擎/变体/场景是 harness 参数,而**工具链**和**被测工程**根本不是轴 —— 一直只测生成的 fixture、 只用各 runner 默认的编译器。 现在六个坐标都成为规范的一部分: | 轴 | 取值 | 在哪里选 | |---|---|---| | OS | linux / macos / windows | 一个 CI job | | 工具链 | gcc / clang / msvc | 一个 CI job(`--compiler`)| | 构建工具 | mcpp / cmake / xmake / meson / bazel | job 内扫(`--engines`)| | 工程 | fixture / mcpp / xlings | 一个 CI job(`--project`)| | 变体 | headers / modules / modules-impl | job 内扫 | | 场景 | cold / noop / touch-hub / touch-leaf / edit-body / edit-comment | job 内扫 | **一个 job = 一个 (OS, 工具链, 工程) 格**,共 **12 格**;另外 7 个不跑的格 **逐条写了原因**。后三个轴在 job 内扫,因为它们共用同一份 checkout、工具链安装和 fixture,拆成 job 只会成倍消耗 runner 分钟而不多出任何一次测量。 **工具链必须是轴**:光换编译器在 mcpp 自己的源码上就值 2.5×(gcc 81.8s → clang 32.6s), 而引擎优化叠在上面又是各自不同的倍数(gcc 2.30×、clang 1.78×)。钉死一个编译器的套件 会把其中一个数字当成答案报出来。且 `--compiler` 解析成**驱动路径**再传给每个引擎 —— 留一个引擎用宿主默认,比较就悄悄变成了编译器对编译器。 **cmake 作为基准**:`--baseline` 的默认值从空改成 `cmake`,不是随手选的 —— P1689 扫描 + ninja dyndep 本就是它的设计,其他引擎实现的是**它的**协议;它在每台跑这套 东西的机器上都在;而且绝对秒数换台机器就没意义,「1.8× cmake」才能被搬运。 忘了传 `--baseline` 会产出一张裸秒数的表 —— 这份数据唯一不能和任何东西比较的形态。 **单一真源**:格子清单只写在 `bench/matrix.json`,workflow 用 jq 读它来规划 job, `SPEC.md` 只解释轴、不复述清单。写两遍的矩阵是会自相矛盾的矩阵,而且矛盾是静默的 —— 两份都一直看着是对的。 `tests/e2e/233_bench_matrix.sh` 钉住这一点:每个坐标必须取自 `axes`;每个 axes 取值 必须是 **harness 真的接受**的(scenario 从 `scenario_from` 里读、engine 从 registry 里读, 而不是在测试里再抄一份);每个排除项必须有原因;workflow 不得再内联 runner 镜像。 四种坏法逐个验过先红。 ⚠️ `jq` 里 `select($plat | contains("," + .os + ","))` 是错的:管道之后 `.` 已经变成 `$plat`,`.os` 在索引一个**字符串**,而 jq 报的行号指向数据文件不是程序。需要 `. as $c`。 --- 同批修掉 macOS e2e 231 的红(clang ⇒ two-phase,第二次构建重建 1 个产物): `write_file` 无条件截断重写,而其中一个文件是**构建输入** —— `obj/mcpp_ios_init.c` (#336 的初始化顺序 TU,仅 macOS 静态 libc++)。于是它每次驱动 mtime 都动,ninja 每次 构建都重编它,包括 noop。改成内容相同就不写。 症状读起来像「拆分调度在 macOS 上不增量」,而在 Linux 上完全不存在(那里没有这个 shim)。 e2e 231 现在**列出**重建了哪些产物 —— 「rebuilt 1 artifact(s)」花了一个 CI 往返才变成 「是哪一个」,而答案从计数里猜不出来。
1 parent 37f31a5 commit 733279d

7 files changed

Lines changed: 627 additions & 59 deletions

File tree

‎.github/workflows/bench.yml‎

Lines changed: 132 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -42,22 +42,12 @@ on:
4242
- '!bench/results/**'
4343
- '.github/workflows/bench.yml'
4444
workflow_dispatch:
45+
# These FILTER the cell list in bench/matrix.json; they do not replace it.
46+
# Which engines / variants / scenarios a cell sweeps is a property of the
47+
# cell (a gcc cell cannot run bazel's module support, a real project has no
48+
# `headers` form), so those live in matrix.json next to the cell they
49+
# describe rather than as one global default applied to every platform.
4550
inputs:
46-
engines:
47-
description: 'comma-separated: mcpp,mcpp-opt,cmake,xmake,meson,bazel'
48-
required: false
49-
default: 'mcpp,mcpp-opt,cmake,xmake'
50-
variants:
51-
description: 'comma-separated: headers,modules,modules-impl'
52-
required: false
53-
default: 'headers,modules,modules-impl'
54-
scenarios:
55-
description: 'comma-separated: cold,noop,touch-hub,touch-leaf,edit-body,edit-comment'
56-
required: false
57-
# All of them. A scenario left out of the default is a scenario nobody
58-
# ever runs — `touch-leaf` was defined, documented and advertised, and
59-
# had never appeared in a single result file.
60-
default: 'cold,noop,touch-hub,touch-leaf,edit-body,edit-comment'
6151
preset:
6252
description: 'named fixture size: smoke | standard | large (overridden by units/fanin/weight below)'
6353
required: false
@@ -79,49 +69,82 @@ on:
7969
required: false
8070
default: 'release'
8171
platforms:
82-
description: 'comma-separated: linux,macos,windows'
72+
description: 'FILTER on bench/matrix.json cells: linux,macos,windows'
8373
required: false
8474
default: 'linux,macos,windows'
75+
toolchains:
76+
description: 'FILTER on bench/matrix.json cells: gcc,clang,msvc'
77+
required: false
78+
default: 'gcc,clang,msvc'
79+
projects:
80+
description: 'FILTER on bench/matrix.json cells: fixture,mcpp,xlings'
81+
required: false
82+
default: 'fixture,mcpp,xlings'
8583

8684
concurrency:
8785
group: bench-${{ github.ref }}
8886
cancel-in-progress: true
8987

9088
jobs:
91-
# The matrix is computed rather than written out, so `platforms: linux` runs
92-
# ONE job instead of three jobs where two are skipped — a skipped job still
93-
# queues a runner and still reports a check.
89+
# The matrix is READ, not written here. bench/matrix.json is the single source
90+
# of truth for which (OS, toolchain, project) cells exist and which engines /
91+
# variants / scenarios each one sweeps; bench/SPEC.md explains the axes and
92+
# deliberately does not repeat the list. A matrix written down twice is a
93+
# matrix that disagrees with itself, and the disagreement is silent — both
94+
# copies keep looking right.
95+
#
96+
# The dispatch inputs FILTER that list rather than replace it, so
97+
# `platforms: linux` runs the linux cells and nothing else — a skipped job
98+
# still queues a runner and still reports a check.
9499
plan:
95100
runs-on: ubuntu-latest
96101
outputs:
97102
matrix: ${{ steps.plan.outputs.matrix }}
98103
steps:
104+
- uses: actions/checkout@v4
99105
- id: plan
100106
shell: bash
101107
run: |
102108
set -euo pipefail
103-
# `inputs.*` is empty on a push/pull_request trigger, so every input needs
104-
# a fallback here — an empty `platforms` would otherwise plan an empty
105-
# matrix and the job would silently do nothing.
106-
want="${{ inputs.platforms || 'linux,macos,windows' }}"
107-
entries=()
108-
case ",$want," in *,linux,*) entries+=('{"os":"ubuntu-24.04","name":"linux"}');; esac
109-
case ",$want," in *,macos,*) entries+=('{"os":"macos-14","name":"macos"}');; esac
110-
case ",$want," in *,windows,*) entries+=('{"os":"windows-2022","name":"windows"}');; esac
111-
if [ ${#entries[@]} -eq 0 ]; then
112-
echo "no platform selected from '$want'" >&2
109+
# `inputs.*` is empty on push/pull_request, so every one needs a
110+
# fallback here — an empty filter would otherwise plan an empty matrix
111+
# and the job would silently do nothing.
112+
plat="${{ inputs.platforms || 'linux,macos,windows' }}"
113+
tool="${{ inputs.toolchains || 'gcc,clang,msvc' }}"
114+
proj="${{ inputs.projects || 'fixture,mcpp,xlings' }}"
115+
116+
# `. as $c` is load-bearing: inside `$plat | contains(...)` the `.` has
117+
# already become $plat, so a bare `.os` there indexes a STRING and jq
118+
# fails pointing at a line number in the data file rather than at the
119+
# program.
120+
include=$(jq -c \
121+
--arg plat ",$plat," --arg tool ",$tool," --arg proj ",$proj," \
122+
--argjson runners "$(jq -c .runners bench/matrix.json)" '
123+
[ .cells[]
124+
| . as $c
125+
| select($plat | contains("," + $c.os + ","))
126+
| select($tool | contains("," + $c.toolchain + ","))
127+
| select($proj | contains("," + $c.project + ","))
128+
| $c + { runs_on: $runners[$c.os] }
129+
]' bench/matrix.json)
130+
131+
count=$(printf '%s' "$include" | jq 'length')
132+
if [ "$count" -eq 0 ]; then
133+
echo "no cell in bench/matrix.json matches platforms='$plat' toolchains='$tool' projects='$proj'" >&2
113134
exit 1
114135
fi
115-
printf 'matrix={"include":[%s]}\n' "$(IFS=,; echo "${entries[*]}")" >> "$GITHUB_OUTPUT"
136+
echo "planning $count cell(s):"
137+
printf '%s' "$include" | jq -r '.[] | " \(.os)/\(.toolchain)/\(.project)"'
138+
printf 'matrix={"include":%s}\n' "$include" >> "$GITHUB_OUTPUT"
116139
117140
bench:
118141
needs: plan
119142
strategy:
120143
fail-fast: false # one platform's engine gap must not cancel the rest
121144
matrix: ${{ fromJSON(needs.plan.outputs.matrix) }}
122-
runs-on: ${{ matrix.os }}
145+
runs-on: ${{ matrix.runs_on }}
123146
timeout-minutes: 120
124-
name: bench (${{ matrix.name }})
147+
name: bench (${{ matrix.os }}/${{ matrix.toolchain }}/${{ matrix.project }})
125148

126149
steps:
127150
- uses: actions/checkout@v4
@@ -154,6 +177,44 @@ jobs:
154177
cmake --version || true
155178
ninja --version || true
156179
180+
# The compiler axis. Resolved to a DRIVER PATH here rather than passed as a
181+
# label, because `--compiler clang` means "whatever clang++ is on PATH" and
182+
# that is a different compiler on each runner — which is exactly the
183+
# comparison this suite is not making. msvc is the exception: cl.exe is
184+
# reached through the VS environment, not a path, so the label is passed
185+
# through and each engine's msvc handling applies.
186+
- name: Resolve the compiler for this cell
187+
shell: bash
188+
run: |
189+
set -euo pipefail
190+
case "${{ matrix.toolchain }}" in
191+
msvc) echo "BENCH_CXX=msvc" >> "$GITHUB_ENV" ;;
192+
gcc) echo "BENCH_CXX=$(command -v g++)" >> "$GITHUB_ENV" ;;
193+
clang) echo "BENCH_CXX=$(command -v clang++)" >> "$GITHUB_ENV" ;;
194+
esac
195+
echo "cell compiler: ${{ matrix.toolchain }}"
196+
197+
- uses: ilammy/msvc-dev-cmd@v1
198+
if: matrix.toolchain == 'msvc'
199+
200+
# The project axis. `fixture` needs nothing — the harness generates it.
201+
# `xlings` is cloned rather than vendored: a vendored snapshot rots, and a
202+
# benchmark whose target drifts from the real project measures the
203+
# snapshot (bench/projects/xlings/README.md).
204+
- name: Fetch the project under measurement
205+
if: matrix.project == 'xlings'
206+
shell: bash
207+
run: |
208+
set -euo pipefail
209+
git clone --depth 1 https://github.com/openxlings/xlings "$RUNNER_TEMP/xlings"
210+
echo "BENCH_PROJECT=$RUNNER_TEMP/xlings" >> "$GITHUB_ENV"
211+
git -C "$RUNNER_TEMP/xlings" rev-parse HEAD
212+
213+
- name: Locate the project under measurement
214+
if: matrix.project == 'mcpp'
215+
shell: bash
216+
run: echo "BENCH_PROJECT=$GITHUB_WORKSPACE" >> "$GITHUB_ENV"
217+
157218
- name: Report engine availability
158219
shell: bash
159220
run: |
@@ -163,29 +224,47 @@ jobs:
163224
shell: bash
164225
run: |
165226
set -euo pipefail
166-
# The preset names the size; units/fanin override it only when set to a
167-
# positive number. Passing raw numbers unconditionally would make every
168-
# run's size an accident of this file rather than a named, comparable
169-
# workload — and --preset must come first so the overrides still win.
170-
# Every `inputs.*` needs a fallback: on a push/pull_request trigger
171-
# they are all EMPTY, and an empty --engines would run nothing while
172-
# still reporting success.
173-
args=( --preset "${{ inputs.preset || 'smoke' }}" )
174-
[ "${{ inputs.units || 0 }}" -gt 0 ] 2>/dev/null && args+=( --units "${{ inputs.units }}" )
175-
[ "${{ inputs.fanin || 0 }}" -gt 0 ] 2>/dev/null && args+=( --fanin "${{ inputs.fanin }}" )
176-
"$BENCH" \
177-
--engines '${{ inputs.engines || 'mcpp,cmake,xmake,meson,bazel' }}' \
178-
--variants '${{ inputs.variants || 'headers,modules,modules-impl' }}' \
179-
--scenarios '${{ inputs.scenarios || 'cold,noop,touch-hub,touch-leaf,edit-body,edit-comment' }}' \
180-
--profile '${{ inputs.profile || 'release' }}' \
181-
"${args[@]}" \
182-
--runs '${{ inputs.runs || 0 }}' \
183-
--work "$RUNNER_TEMP/bench-work" \
184-
--out "bench-${{ matrix.name }}.json"
227+
args=( --engines '${{ matrix.engines }}'
228+
--variants '${{ matrix.variants }}'
229+
--scenarios '${{ matrix.scenarios }}'
230+
--baseline cmake
231+
--profile '${{ inputs.profile || 'release' }}'
232+
--runs '${{ inputs.runs || 0 }}'
233+
--work "$RUNNER_TEMP/bench-work"
234+
--out "bench-${{ matrix.os }}-${{ matrix.toolchain }}-${{ matrix.project }}.json" )
235+
236+
# `msvc` is a label, not a path — see the resolve step above.
237+
[ "$BENCH_CXX" != "msvc" ] && [ -n "$BENCH_CXX" ] && args+=( --compiler "$BENCH_CXX" )
238+
239+
if [ "${{ matrix.project }}" = "fixture" ]; then
240+
# The preset names the size; units/fanin override it only when set to
241+
# a positive number. Passing raw numbers unconditionally would make
242+
# every run's size an accident of this file rather than a named,
243+
# comparable workload — and --preset must come first so the
244+
# overrides still win.
245+
args+=( --preset "${{ inputs.preset || matrix.preset }}" )
246+
[ "${{ inputs.units || 0 }}" -gt 0 ] 2>/dev/null && args+=( --units "${{ inputs.units }}" )
247+
[ "${{ inputs.fanin || 0 }}" -gt 0 ] 2>/dev/null && args+=( --fanin "${{ inputs.fanin }}" )
248+
else
249+
# A real tree: measured in place, and the scenarios that perturb a
250+
# file must be TOLD which one. Without --hub/--leaf/--body they
251+
# report `skipped` with the reason rather than picking a file and
252+
# producing a number that looks valid.
253+
args+=( --project "$BENCH_PROJECT"
254+
--buildfiles "$GITHUB_WORKSPACE/bench/projects/${{ matrix.project }}" )
255+
case "${{ matrix.project }}" in
256+
mcpp) args+=( --hub "src/platform/platform.cppm"
257+
--body "src/version_req.cppm" ) ;;
258+
xlings) args+=( --hub "src/xlings.cppm"
259+
--body "src/xlings.cppm" ) ;;
260+
esac
261+
fi
262+
263+
"$BENCH" "${args[@]}"
185264
186265
- name: Upload report
187266
uses: actions/upload-artifact@v4
188267
with:
189-
name: bench-${{ matrix.name }}
190-
path: bench-${{ matrix.name }}.json
268+
name: bench-${{ matrix.os }}-${{ matrix.toolchain }}-${{ matrix.project }}
269+
path: bench-${{ matrix.os }}-${{ matrix.toolchain }}-${{ matrix.project }}.json
191270
if-no-files-found: error

0 commit comments

Comments
 (0)