From 0c15c66c90f682ee1f278ae503be45b9995213ef Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Thu, 6 Aug 2026 07:30:52 +0800 Subject: [PATCH] docs: describe the CI as it is after sharding, and make --shard mean one thing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CI section still described a three-platform matrix and a single `mcpp test --workspace` per platform. #155 replaced that with a `select` job that decides the whole plan once, per-platform shard counts taken from measured runner concurrency, measured-time bin packing, and a `timings` job that ranks members. None of it was written down. Also documented: the `cache` dispatch input, why the global package build cache came back (mcpp#344's reason was removed in 2026.8.3.4, and the bypass made the full linux run exceed its timeout), and the pre-test index refresh. `run_members.sh --shard` computed its own split by round-robin while CI used `plan_shards.lua`'s measured packing — two algorithms answering one question, in a script whose header claims local and CI measure the same thing. --shard now delegates to plan_shards.lua, so shard N locally holds the members shard N holds in CI; round-robin stays as the fallback where lua is absent and says so in its output. --platform selects which column of member-timings.tsv to read and defaults to the host. Both language versions updated. --- docs/repository-and-schema.md | 84 ++++++++++++++++++++++++++++---- docs/zh/repository-and-schema.md | 69 +++++++++++++++++++++++--- tests/run_members.sh | 59 +++++++++++++++++----- 3 files changed, 182 insertions(+), 30 deletions(-) diff --git a/docs/repository-and-schema.md b/docs/repository-and-schema.md index 7ddf60a4..13f33e6c 100644 --- a/docs/repository-and-schema.md +++ b/docs/repository-and-schema.md @@ -19,8 +19,15 @@ tests/examples// one test project per library (a workspace member; < tests/check_mirror_urls.lua lint: GLOBAL+CN table completeness, and that CN points at mcpp-res tests/check_package_name.lua lint: identity shape (name is a single atomic segment, hierarchy belongs to namespace) tests/list_cn_urls.lua extracts the CN urls for mirror-cn-reachable +tests/run_members.sh runs workspace members one at a time and times each. The entry point used both by CI + and locally; see "Running workspace members locally" below +tests/plan_shards.lua assigns members to shards from measured times. Called once by the `select` job, and by + run_members.sh --shard, so both produce the same split +tests/member-timings.tsv the measured per-member wall-clock plan_shards.lua reads. Refreshed deliberately from + the member-timings artifact rather than written back on every run README.md index overview and contribution entry point (README.zh-CN.md is the Chinese version) -.github/workflows/validate.yml CI: lint / mirror-cn-reachable / workspace (a 3-platform matrix) +.github/workflows/validate.yml CI: lint / mirror-cn-reachable / select / workspace (sharded per platform on a full + run) / timings .agents/docs/-*.md the design-document convention docs/ contributor reference documentation (this directory; docs/zh/ holds the Chinese version) tools/gtc the gitcode CLI, see cn-mirror.md @@ -150,7 +157,10 @@ locally with `mcpp xpkg parse pkgs//.lua`. ## CI behavior (validate.yml) - Triggers: a PR (touching `pkgs/**/*.lua`, `tests/**`, either README, `mcpp.toml`, `index.toml` or this workflow), - a push to main, the nightly cron, and manual dispatch. + a push to main, the nightly cron, and manual dispatch. Dispatch takes a `cache` input — `global`, the default, or + `local`. Under `local` every member rebuilds every dependency from scratch, which isolates a member's own cost from + what it inherited from the members that ran before it; that is the condition per-member times should be compared + under, and it is also far slower. - `env.MCPP_VERSION` is the mcpp version every job uses; local verification should match it. - `lint` (always runs): lua syntax via `loadfile(f,'t')`; `spec=`/`name=`/`xpm=` must be present; leading-v versions are rejected; runs `check_mirror_urls.lua`; runs `check_package_name.lua` (identity shape, see "Package identity" @@ -158,20 +168,54 @@ locally with `mcpp xpkg parse pkgs//.lua`. fails). `xpkg parse` in mcpp >= 0.0.106 enforces the identity shape itself, which makes the lua lint an earlier and cheaper redundant gate. - `mirror-cn-reachable` (always runs): `curl`s each CN url; all must return 200. -- `workspace (linux|macos|windows)`: the whole test surface is one mcpp workspace and the **only build/run channel** — - there is no shell-driven exception (the public module packages imgui/ffmpeg/opencv/tinyhttps are ordinary members - too, resolving from the checkout through a member-level `[indices] default = { path = "../../.." }`, - mcpp >= 0.0.97; members consuming `compat` inherit the root-level declaration, see "Index redirection" above). +- `select`: decides the entire plan once and emits it to the runners as data. Three questions are answered here + rather than on each runner — which members run, how many shards each platform gets, and which members land on + which shard. - Selective member testing: on a PR, `git diff` maps changed files to the affected members - (`pkgs//.lua` → members whose mcpp.toml references ``; `tests/examples//**` → member ``), and - only those run through `mcpp test -p `; global changes — the workflow itself, the non-member part of the - workspace manifest, `tools/` and so on — go to a full `mcpp test --workspace`. push/nightly/dispatch are always - full runs. + (`pkgs//.lua` → members whose mcpp.toml references ``; `tests/examples//**` → member ``). + A change that can affect everything selects the full workspace instead: a non-PR event, this workflow file, a + non-member edit to the workspace manifest, or a shared test script. Documentation-only and `tools/`-only changes + select nothing. + - Sharding applies to full runs only — a selective run is one job per platform. The shard count per platform is + that platform's **measured runner concurrency** (linux 3, macos 1, windows 2) rather than a round number. + Wall-clock is `ceil(shards / concurrency) × slowest-shard`, so shards beyond the concurrency remove no work and + each still pays its own checkout, mcpp download and cache restore. At concurrency 1, splitting macOS is strictly + slower than not splitting it. Re-measure with: + `gh api repos///actions/runs//jobs --paginate --jq '[.jobs[]|select(.status=="in_progress")]|length'` + - The assignment itself comes from `tests/plan_shards.lua`. It reads `tests/member-timings.tsv` and packs + longest-first onto the least-loaded shard; ties break toward the shard already holding members with overlapping + dependencies, because shards share no build cache and a dependency landing on two shards is built twice. A + member with no recorded time is charged the median, so a newly added member is assumed neither free nor huge. + Against round-robin on the real workspace the slowest linux shard falls from 4158s to 3706s and the spread from + 47% to 15%. One floor no split can beat remains: the single slowest member, `grpc-module` at 1701s. + - Planning runs here, on linux, because it needs lua: windows has no apt or brew, and Homebrew installs `lua` + rather than `lua5.4`. +- `workspace ( /)`: the whole test surface is one mcpp workspace and the **only build/run + channel** — there is no shell-driven exception (the public module packages imgui/ffmpeg/opencv/tinyhttps are + ordinary members too, resolving from the checkout through a member-level `[indices] default = { path = "../../.." }`, + mcpp >= 0.0.97; members consuming `compat` inherit the root-level declaration, see "Index redirection" above). The + shard suffix appears only where the platform is actually split. + - Members run through `tests/run_members.sh`, the same script used locally. A timing table that exists only in CI + cannot be consulted while deciding what to optimise, and a local harness that differs from CI measures something + else. + - The package build cache is global, which is mcpp's default. The step formerly set `MCPP_BUILD_CACHE: local` to + work around mcpp#344, in which one cache entry could hold two object layouts; mcpp 2026.8.3.4 keyed the cache per + package with the consumer-dependent layout included, so the reason no longer holds. Keeping the bypass was + expensive: under `local`, 59 members that largely share abseil, protobuf and opencv rebuilt each of them from + scratch, and a full linux run reached 2h30m — past the timeout, so it produced no result at all. - The `~/.mcpp/registry` cache carries the toolchains and the already-built compat packages, so a repeat run is incremental and fast. Its key is computed once, in a step of its own, from `git ls-files -s` over the tracked inputs — never with `hashFiles()`, which globs the working tree and would re-hash the multi-GB build output under `tests/examples/*/target` when actions/cache re-evaluates the key in its post (save) step (that blew past the runner's 120s template-evaluation cap on windows). + - The published index is refreshed before testing. Most members resolve everything from the checkout, but a member + redirecting a namespace other than `compat` takes the rest from the published index, whose snapshot is whatever + the pinned mcpp release vendored — older than main by construction, and never moved by anything else in the run. +- `timings`: merges the per-shard timing artifacts into one ranking per platform in the run summary, and publishes + the combined table as the `member-timings` artifact. Sharding otherwise hides where the time goes, since each + runner reports only its own slice. The table is not committed automatically: a number that rewrites itself on + every run makes every diff noisy and silently absorbs a one-off slow runner. Refresh `tests/member-timings.tsv` + from that artifact when the numbers have actually moved. ## Reproducing lint locally (equivalent to the CI lint job) @@ -187,6 +231,26 @@ done [ $fail -eq 0 ] && echo "ALL LINT PASS" ``` +## Running workspace members locally + +`tests/run_members.sh` is the entry point CI uses, so a local run measures the same thing under the same split: + +```bash +bash tests/run_members.sh --all # every member +bash tests/run_members.sh opencv-module protobuf # named members +bash tests/run_members.sh --all --shard 1/3 # exactly what CI's linux shard 1 runs +bash tests/run_members.sh --all --shard 0/2 --platform windows +bash tests/run_members.sh --all --cache local # bypass the package build cache +``` + +Shard indices are 0-based, and `--shard` delegates to `tests/plan_shards.lua` — the script the `select` job calls — +so shard N locally holds the members shard N holds in CI. Without lua on `PATH` it falls back to round-robin and +says so. `--platform` chooses which column of `tests/member-timings.tsv` to read and defaults to the host. + +`MCPP` selects the binary (default: `mcpp` on `PATH`); `MCPP_TIMINGS` names a file to append +`\t\t` rows to. The exit status is non-zero if any member failed, and the timing table +prints either way — a run worth diagnosing is exactly the one where the times matter. + ## After the merge `publish-artifact.yml` republishes the mcpp-index artifact and moves the pointer automatically once the change lands diff --git a/docs/zh/repository-and-schema.md b/docs/zh/repository-and-schema.md index ded14b3a..9e55db37 100644 --- a/docs/zh/repository-and-schema.md +++ b/docs/zh/repository-and-schema.md @@ -18,8 +18,13 @@ tests/examples// 每库测试工程(workspace 成员; 为包 tests/check_mirror_urls.lua lint:GLOBAL+CN 表完整性,以及 CN 指向 mcpp-res tests/check_package_name.lua lint:身份形态(name 为单一原子段,层级归 namespace) tests/list_cn_urls.lua 抽取 CN url,供 mirror-cn-reachable 使用 +tests/run_members.sh 逐个运行 workspace 成员并计时。CI 与本地共用的入口,见下文「本地运行 workspace 成员」 +tests/plan_shards.lua 依实测耗时将成员分配到各分片。由 `select` job 调用一次,`run_members.sh --shard` + 亦调用同一脚本,故两处得到同一划分 +tests/member-timings.tsv plan_shards.lua 读取的逐成员实测墙钟。由 member-timings artifact 择时手工刷新, + 而非每次运行自动回写 README.md 索引说明与贡献入口(英文;中文版为 README.zh-CN.md) -.github/workflows/validate.yml CI:lint / mirror-cn-reachable / workspace(3 平台矩阵) +.github/workflows/validate.yml CI:lint / mirror-cn-reachable / select / workspace(全量时按平台分片)/ timings .agents/docs/-*.md 设计文档惯例 docs/ 贡献者参考文档(英文);docs/zh/ 为中文版(本目录) tools/gtc gitcode CLI,见 cn-mirror.md @@ -115,25 +120,54 @@ mcpp 跑 `xpkg parse`(strict:未知键即失败),所以需要更新文法/键的 ## CI 行为(validate.yml) - 触发条件:PR(改动 `pkgs/**/*.lua`、`tests/**`、两份 README 之一、`mcpp.toml`、`index.toml` 或本 workflow)、 - push 至 main、nightly cron、手动触发。 + push 至 main、nightly cron、手动触发。手动触发接受 `cache` 输入 —— `global`(默认)或 `local`。`local` 下 + 每个成员均从零重建其全部依赖,从而把成员自身的开销与它从先前成员处继承到的部分隔离开;逐成员耗时应在此 + 条件下比较,代价是显著更慢。 - `env.MCPP_VERSION` 为全部 job 使用的 mcpp 版本,本地验证应与之对齐。 - `lint`(始终运行):lua 语法 `loadfile(f,'t')`;须含 `spec=`/`name=`/`xpm=`;禁止前导 v 版本;执行 `check_mirror_urls.lua`;执行 `check_package_name.lua`(身份形态,见上文「包身份」);再用 CI pin 的 mcpp 对每个描述符跑 `mcpp xpkg parse`(strict,未知键即失败)。mcpp ≥ 0.0.106 的 `xpkg parse` 自身 也强制身份形态,lua lint 因此是更早、更便宜的冗余闸门。 - `mirror-cn-reachable`(始终运行):逐个 `curl` CN url,均须返回 200。 -- `workspace (linux|macos|windows)`:整个测试面就是一个 mcpp workspace,**唯一的构建/运行通道**—— +- `select`:一次性决定整个计划,并以数据形式下发给各 runner。三个问题在此处而非各 runner 上回答 —— + 哪些成员要跑、每个平台分几片、哪些成员落在哪一片。 + - 选择性成员测试:PR 时由 `git diff` 将改动文件映射到受影响成员 + (`pkgs//.lua` → mcpp.toml 引用 `` 的成员;`tests/examples//**` → 成员 ``)。 + 可能影响全部成员的改动则选中整个 workspace:非 PR 事件、本 workflow 文件、workspace 清单的非成员 + 部分、共享测试脚本。仅文档与仅 `tools/` 的改动不选中任何成员。 + - 分片仅用于全量运行,选择性运行为每平台一个 job。每平台的分片数取该平台**实测的 runner 并发度** + (linux 3、macos 1、windows 2),而非取整数。墙钟为 `ceil(分片数 / 并发度) × 最慢分片`,故超出并发度 + 的分片不减少任何工作量,却各自仍要付出 checkout、mcpp 下载与缓存恢复的固定开销;并发度为 1 时, + 对 macOS 分片严格慢于不分片。重新测量: + `gh api repos///actions/runs//jobs --paginate --jq '[.jobs[]|select(.status=="in_progress")]|length'` + - 分配本身由 `tests/plan_shards.lua` 给出:读取 `tests/member-timings.tsv`,按耗时降序依次放入当前负载 + 最小的分片;负载接近时优先选择已含有共同依赖成员的分片 —— 分片之间不共享构建缓存,同一依赖落在两片 + 上就要构建两次。无实测记录的成员按中位数计价,故新增成员既不被假定为零成本,也不被假定为极重。 + 在真实 workspace 上与轮转法相比,linux 最慢分片由 4158s 降至 3706s,离散度由 47% 降至 15%。有一条 + 任何划分都无法突破的下界:最慢的单个成员,`grpc-module` 为 1701s。 + - 规划在此 linux job 内进行,因为它需要 lua:windows 上既无 apt 亦无 brew,而 Homebrew 安装的是 `lua` + 而非 `lua5.4`。 +- `workspace (<平台> <分片>/<总数>)`:整个测试面就是一个 mcpp workspace,**唯一的构建/运行通道**—— 没有任何 shell 驱动的例外(公开模块包 imgui/ffmpeg/opencv/tinyhttps 也是普通成员,经成员级 `[indices] default = { path = "../../.." }` 从 checkout 解析,mcpp ≥ 0.0.97;消费 `compat` 的 - 成员则继承根级声明,见上文「索引重定向」)。 - - 选择性成员测试:PR 时由 `git diff` 将改动文件映射到受影响成员 - (`pkgs//.lua` → mcpp.toml 引用 `` 的成员;`tests/examples//**` → 成员 ``), - 仅 `mcpp test -p ` 这些成员;workflow 本身、workspace 清单非成员部分、`tools/` 等 - 全局性改动 → `mcpp test --workspace` 全量。push/nightly/dispatch 恒为全量。 + 成员则继承根级声明,见上文「索引重定向」)。分片后缀仅在该平台确实被拆分时出现。 + - 成员经 `tests/run_members.sh` 运行,该脚本亦即本地入口。只存在于 CI 的耗时表无法在决定优化对象时被 + 参考,而与 CI 不同的本地测量装置度量的是另一回事。 + - 包构建缓存取全局,即 mcpp 的默认值。该步骤此前设 `MCPP_BUILD_CACHE: local` 以规避 mcpp#344 —— 同一 + 缓存条目可能持有两种对象布局;mcpp 2026.8.3.4 起缓存按包计键并纳入随消费方而变的布局,该理由已不成立。 + 保留该旁路的代价是实际的:`local` 之下,59 个大量共享 abseil、protobuf、opencv 的成员会各自从零重建 + 这些依赖,linux 全量运行达到 2h30m —— 超出超时上限,因而根本得不到结果。 - `~/.mcpp/registry` 缓存携带工具链与已构建的 compat 包,重复运行增量很快。其 key 由独立步骤经 `git ls-files -s` 就已跟踪的输入算出一次,而**不用** `hashFiles()`:后者按工作树 glob,而 actions/cache 会在 post(save)步骤重新求值 key,届时 `tests/examples/*/target` 下的数 GB 构建产物 也会被一并哈希 —— windows 上曾因此撞破 runner 的 120 秒模板求值上限。 + - 测试前刷新已发布索引。多数成员的全部依赖都从本 checkout 解析,但重定向了 `compat` 之外命名空间的成员 + 会从已发布索引取其余部分,而该快照是所 pin 的 mcpp 发行版随附的那一份 —— 按构造即早于 main,且运行中 + 没有任何其他环节会推进它。 +- `timings`:将各分片的耗时 artifact 合并为每平台一份排名写入 run summary,并以 `member-timings` artifact + 发布合并后的表。否则分片会掩盖时间的去向 —— 每个 runner 只报告自己那一片。该表不自动提交:每次运行都 + 改写自身的数字会让每份 diff 都充满噪声,并会悄悄吸收一次偶发的慢 runner。数字确实发生变化时,再据该 + artifact 刷新 `tests/member-timings.tsv`。 ## 本地 lint 复现(等价于 CI lint job) @@ -149,6 +183,25 @@ done [ $fail -eq 0 ] && echo "ALL LINT PASS" ``` +## 本地运行 workspace 成员 + +`tests/run_members.sh` 即 CI 使用的入口,故本地运行度量的是同一对象、同一划分: + +```bash +bash tests/run_members.sh --all # 全部成员 +bash tests/run_members.sh opencv-module protobuf # 指定成员 +bash tests/run_members.sh --all --shard 1/3 # 与 CI 的 linux 分片 1 完全一致 +bash tests/run_members.sh --all --shard 0/2 --platform windows +bash tests/run_members.sh --all --cache local # 绕过包构建缓存 +``` + +分片下标自 0 起。`--shard` 委托给 `tests/plan_shards.lua`,即 `select` job 所调用的同一脚本,因此本地的第 N +片与 CI 的第 N 片持有相同成员;`PATH` 上没有 lua 时退回轮转法,并在输出中说明。`--platform` 选择读取 +`tests/member-timings.tsv` 的哪一列,缺省为宿主平台。 + +`MCPP` 指定所用二进制(缺省为 `PATH` 上的 `mcpp`);`MCPP_TIMINGS` 指定追加 `<秒>\t<成员>\t` 行的 +文件。任一成员失败则退出码非零,而耗时表两种情况下均会打印 —— 值得诊断的运行恰恰就是耗时重要的那一次。 + ## 合并后 `publish-artifact.yml` 在合并至 `main` 后自动重新发布 mcpp-index artifact 并移动指针,无需发布新的 mcpp 版本。 diff --git a/tests/run_members.sh b/tests/run_members.sh index 31c1172d..d6d3a669 100755 --- a/tests/run_members.sh +++ b/tests/run_members.sh @@ -7,8 +7,15 @@ # # bash tests/run_members.sh --all # bash tests/run_members.sh opencv-module protobuf -# bash tests/run_members.sh --all --shard 3/8 -# bash tests/run_members.sh --all --cache local # bypass the package cache +# bash tests/run_members.sh --all --shard 1/3 # CI's linux shard 1 +# bash tests/run_members.sh --all --shard 0/2 --platform windows +# bash tests/run_members.sh --all --cache local # bypass the package cache +# +# --shard N/M reproduces the split CI runs: the assignment comes from +# tests/plan_shards.lua, the same script CI's `select` job calls, so shard N +# here holds the members shard N holds there. Shard indices are 0-based. +# --platform selects which column of tests/member-timings.tsv to read and +# defaults to the host. # # Env: # MCPP path to the mcpp binary (default: `mcpp` on PATH) @@ -25,13 +32,23 @@ shard="" members=() all=0 +# Which column of tests/member-timings.tsv --shard reads. Defaults to the host, +# because the host is the machine whose times are being reproduced. +case "$(uname -s)" in + Linux) platform=linux ;; + Darwin) platform=macos ;; + MINGW*|MSYS*|CYGWIN*) platform=windows ;; + *) platform=linux ;; +esac + while [ $# -gt 0 ]; do case "$1" in --all) all=1; shift ;; --shard) shard="$2"; shift 2 ;; + --platform) platform="$2"; shift 2 ;; --cache) cache="$2"; shift 2 ;; --timings) timings="$2"; shift 2 ;; - -h|--help) sed -n '2,20p' "$0"; exit 0 ;; + -h|--help) sed -n '2,25p' "$0"; exit 0 ;; -*) echo "unknown option: $1" >&2; exit 2 ;; *) members+=("$1"); shift ;; esac @@ -60,20 +77,38 @@ if [ "${#members[@]}" -eq 0 ]; then exit 2 fi -# --shard N/M keeps every M-th member starting at N. Round-robin by position, -# which is what separates adjacent expensive members (opencv-module, -# -dnn, -unifont) onto different runners. +# --shard N/M selects shard N of M. The assignment is NOT decided here: +# tests/plan_shards.lua owns it and CI's `select` job calls the same script. +# Two implementations of one question drift apart, and a local shard that +# splits differently from CI's measures a different split than the one being +# tuned — which defeats the reason this script is shared in the first place. +# +# Round-robin remains as the fallback for a machine with no lua. It is worse +# (it knows nothing about how long anything takes) but it is never wrong, and +# it keeps --shard usable where plan_shards.lua cannot run. if [ -n "$shard" ]; then idx=${shard%%/*} cnt=${shard##*/} - picked=() - i=0 - for m in "${members[@]}"; do - [ $((i % cnt)) -eq "$idx" ] && picked+=("$m") - i=$((i + 1)) + lua="" + for cand in lua5.4 lua; do + command -v "$cand" >/dev/null 2>&1 && { lua=$cand; break; } done + picked=() + if [ -n "$lua" ]; then + for m in $("$lua" tests/plan_shards.lua "$platform" "$idx" "$cnt" "${members[@]}"); do + picked+=("$m") + done + how="measured split for $platform" + else + i=0 + for m in "${members[@]}"; do + [ $((i % cnt)) -eq "$idx" ] && picked+=("$m") + i=$((i + 1)) + done + how="round-robin — no lua on PATH, so times were not consulted" + fi members=("${picked[@]+"${picked[@]}"}") - echo "shard $idx/$cnt -> ${#members[@]} member(s)" + echo "shard $idx/$cnt ($how) -> ${#members[@]} member(s)" fi [ -n "$cache" ] && export MCPP_BUILD_CACHE="$cache"