diff --git a/.agents/docs/2026-08-06-grpcgen-layered-control-design.md b/.agents/docs/2026-08-06-grpcgen-layered-control-design.md new file mode 100644 index 0000000..93f8725 --- /dev/null +++ b/.agents/docs/2026-08-06-grpcgen-layered-control-design.md @@ -0,0 +1,171 @@ +# grpcgen 的分层控制:让「方便」与「可控」不是两条路 + +> 状态：**设计待 review** +> 涉及：`rules/src/grpcgen.cppm`、`templates/greeter/`、`mcpp.toml` 的 `[feature-deps.codegen]` +> 前置：mcpp 2026.8.6.2（`reexport` / `rerun_if_changed_glob`）已发布 + +--- + +## 0. 现状与问题 + +`generate_all()` 一行就能跑通,底子是对的:工作被**声明**成构建图的边(`mcpp::action`)而不是当场执行,所以增量、并行、失败归属到具体那条边;错误信息直接告诉用户该往 mcpp.toml 里加哪一行。 + +问题不在「方便」,在**可控的层次只有两级**: + +```cpp +struct options { + std::string_view proto_dir = "proto"; + bool grpc = true; +}; +``` + +两个旋钮之外是断崖。需求一超出,用户只能像 `examples/helloworld` 那样手写六十行 build.mcpp,**绕开整个规则**——而那六十行要重新实现规则里已经解决过的东西:well-known types 目录怎么定位、嵌套 `.proto` 的输出子目录怎么建、输入集合怎么算。它们会与规则悄悄漂移,且没有任何机制会报出漂移。 + +**断崖本身就是设计缺陷**:它把「优雅」和「可控」变成了二选一。 + +## 1. 原则:每一层是下一层的默认值,不是另一条路 + +只要下一层是「同一条代码路径 + 默认参数」,就不会出现「用了高级功能就失去便利」的分叉,也不会出现两套实现漂移。 + +这条原则决定了后面每一层的形状:L1 不是新函数,是 `options` 多几个字段;L2 不是新入口,是把 `bool grpc` 降级成语法糖;L3 不是旁路,是把现有函数拆成 `plan` + `submit` 两半,`generate_all()` 变成它俩的组合。 + +## 2. L0 — 默认(保持不变) + +```cpp +import mcpp; import grpcgen; +int main() { return grpcgen::generate_all() ? 0 : 1; } +``` + +模板与文档的主线形态。本设计不改变它的任何行为。 + +## 3. L1 — 声明式旋钮 + +补上真实项目**必然**撞到的三个缺口。按撞到的频率排序: + +| 缺口 | 什么时候遇到 | 证据 | +|---|---|---| +| 额外 import 路径 | 共享 proto 仓库;`google/api/annotations.proto`(gRPC-Gateway、Google API 风格接口) | 官方 `.proto` 之间互相 import 是常态,protoc 靠 `-I` 找 | +| mock 生成 | 一写单测就要 | `grpc_cpp_plugin` 自带 `generate_mock_code=true`,现在**完全没有办法开** | +| 任意 protoc 参数 | 兜住没想到的 | 例如 `--experimental_allow_proto3_optional` | + +```cpp +grpcgen::generate_all({ + .imports = {"../shared/proto"}, + .mock = true, + .protoc_args = {"--experimental_allow_proto3_optional"}, +}); +``` + +三条都是**加法**:不写等于今天的行为,逐字节相同。 + +`mock` 单独成字段而不是让用户自己往 `protoc_args` 塞,是因为它的拼写是插件参数(`--grpc_out=generate_mock_code=true:`)而不是 protoc 顶层参数——那正是「库该承担的知识」。 + +## 4. L2 — 插件列表:把 `bool grpc` 升成一等公民 + +```cpp +grpcgen::generate_all({ + .plugins = { grpcgen::cpp(), gateway_plugin, validate_plugin }, +}); +``` + +`grpc = true` 变成「列表里有 `cpp()`」的语法糖,旧写法一字不改。 + +**为什么必须做这一步**:`.proto` 的插件生态不止 gRPC——`protoc-gen-validate`、grpc-gateway、文档生成都是同一条 protoc 调用上的 `--_out`。如果不做,每来一个插件就要往 `options` 上挂一个 `bool`,而那正是 `bool grpc` 已经示范过的坏形状。 + +一个插件的完整描述是:名字、可执行文件路径、输出目录、插件参数、以及它产出哪些文件后缀(决定 `output()` 声明)。后缀不能省——mcpp 需要知道产物才能把 `.cc` 纳入编译集、把 `.h` 排除在外。 + +## 5. L3 — `plan` / `submit` 分离:终极逃生舱 + +```cpp +auto edges = grpcgen::plan_all(); // 只构造边,不提交 +for (auto& e : edges) e.arg("--whatever"); // 完全接管 +grpcgen::submit(edges); +``` + +价值不在「能改 flag」——L1 的 `protoc_args` 已经覆盖大半——而在**再离谱的需求也不必绕开规则**:well-known types 定位、子目录创建、输入集合计算这些规则已经解决的部分继续复用,用户只接管自己关心的那一段。 + +这一层直接消灭第 0 节说的漂移风险。 + +## 6. 可观测性:分工已经由引擎定死 + +**核实过,不是推断**:`mcpp:action=` 声明的边,其完整命令行会原样落进 `build.ninja`: + +``` +rule mcpp_action_0 + command = .../bin/protoc -I.../proto -I.../protobuf/src --cpp_out=... --grpc_out=... + description = GENERATE protoc:echo +``` + +因此: + +- **规则不该实现 `GRPCGEN_EXPLAIN` 之类的命令行 dump**。「这条边到底跑了什么」是引擎已经答完的问题,`ninja -t commands ` 即可取;规则重造一份只会有第二个真相来源。 +- **规则该负责的是 `description`**。它现在是 `protoc:echo`,看不出开了哪些旋钮。改成自述: + +``` +protoc:echo (+grpc +mock, -I proto -I ../shared/proto) +``` + +这条串出现在每次构建的输出里,是**零成本**的可观测性:不需要任何额外命令,就能回答「这个 flag 到底进去没有」。而完整命令行仍在 build.ninja 里等着被查。 + +分工一句话:**引擎拥有「命令是什么」,规则拥有「哪些旋钮产生了它」。** + +## 7. 命名:`codegen` 保留,真正要修的是文档 + +查证结论(不是印象): + +- **`protoc` 是错的名字。** 它只命名了三者之一,而且是 **protobuf 那一半**;挂在 grpc 包上更偏。且 `buf` 存在(自带编译器,不需要 protoc 二进制),哪天换生成器,`protoc` 这个名字就成了谎话。 +- **与 tonic 的歧义是单向的,撞不上。** tonic 的 `codegen` feature 指「生成的代码编译时需要的运行时导出」(`tonic::codegen` 模块),生成器那一半在 tonic 里是独立 crate `tonic-build`。C++ 里不存在对应物——生成的桩直接 `#include ` 并链同一个库。 +- **L2 之后 `codegen` 更站得住**:它表达的是「这个包知道 `.proto` 怎么变成 C++,包括你后来加的插件」,而不是「跑哪个二进制」。`stubgen` 同样会被 L2 打脸(插件产出的不止 stub)。 + +**真正的缺口是文档没说清两件事:** + +1. **它带来的是两步,不是一步。** 官方文档里是两条独立的 protoc 调用: + + ``` + protoc --cpp_out=. route_guide.proto → .pb.{h,cc} protobuf 消息 + protoc --grpc_out=. --plugin=... → .grpc.pb.{h,cc} gRPC 服务桩 + ``` + + 第一步是 protobuf 的事,与 gRPC 无关;第二步才是。现在的注释只说「the whole toolchain」,读者意识不到 `.pb.*` 根本不属于 gRPC——而这正是「为什么需要 `compat.protobuf` 的 protoc」的答案。 + +2. **为什么默认关闭。** 现在只写了成本(protoc 拖进 libprotoc 约 157 个 TU),没写**真实存在的无 codegen 路径**:gRPC 官方的 Generic API(`grpc::GenericStub` + `grpc::ByteBuffer`,自己序列化)。官方性能文档明确推荐它用于高竞争或 proto 序列化 CPU 密集的场景——代理、负载均衡这类转发型服务走的就是这条路。也就是说 off-by-default 不是「省点编译时间」,是**存在一整类项目确实不需要它**。 + +## 7.5 实施中长出来的两条(设计里没有) + +**`extra_dirs`。** 原设计只有 `.imports`(额外 `-I`)。写 `examples/advanced` 时撞出来:protoc 会把 `#include "common/types.pb.h"` 写进任何 import 了它的文件,所以**只靠 `-I` 够到的共享树会产出一个没人生成的头**,报错(`fatal error: 'common/types.pb.h' file not found`)离原因很远。两个概念因此必须分开:`extra_dirs` 既生成又搜索,`imports` 只搜索。 + +这条是示例发现的,不是设计发现的——一个只写文档不写示例的设计会把它漏掉。 + +**`plan_entries` / `entry`。** L3 的真正底层:`.proto` 按 `(root, name)` 寻址。`plan()` 与 `plan_all()` 都汇入它。它必须是公开的,因为那是「`.proto` 既不在 `proto_dir` 也不在 `extra_dirs` 下」这种项目的唯一出路——否则它们又回到自己写 build.mcpp,而那正是本设计要消掉的断崖。 + +## 7.6 一个生态限制,如实记录 + +`.mock` 让 protoc 产出 `_mock.grpc.pb.h`,而那个头 `#include ` —— 本生态的 `compat.gtest` **只带 googletest、不带 gmock**。所以旋钮是对的、产物也真的生成,但**当前无法编译它**。 + +`examples/advanced` 因此声明并产出 mock 头、但不 include;plan 阶段断言它进了输出集,CI 再断言文件真的存在。补 gmock 是 mcpp-index 的事,与本规则无关。 + +## 8. 实施顺序 + +| 步 | 内容 | 风险 | +|---|---|---| +| 1 | L1 三个字段 + `description` 自述 | 低,纯加法,旧写法逐字节不变 | +| 2 | 文档:两步的分工 + Generic API 那条路 | 无 | +| 3 | L2 插件列表(`grpc = true` 降级为语法糖) | 中——新的用户可见 API,要先定 plugin 的完整描述形状 | +| 4 | L3 `plan` / `submit` | 中——`generate_all` 必须变成二者的组合,否则又是两条路 | + +建议 1+2 先落地并发布,验证过再做 3+4。1+2 覆盖真实项目 90% 的需求,且不引入任何新概念。 + +## 9. 验证 + +- **单测无从下手**(规则运行在 build.mcpp 里),所以验证靠 examples: + - `examples/greeter` 保持 L0 不变 —— 证明加法没有改变默认行为; + - 新增一个用到 `.imports` + `.mock` 的示例,断言 mock 头文件真的产出、且能被 include; + - `description` 自述:断言构建输出里出现 `+mock`。 +- **不做的验证**:不再手工比对 protoc 命令行——它在 build.ninja 里,是引擎的契约,不是本规则的。 + +## 10. 明确不做 + +- **不实现命令行 dump**(`GRPCGEN_EXPLAIN`)。引擎已经把完整命令写进 build.ninja,第二个真相来源只会漂移。 +- **不改 feature 名。** 见 §7。 +- **不为每个插件加一个 `bool`。** 那是 L2 要消灭的形状,而不是要复制的。 +- **不支持「跳过 protobuf 那一步只生成 gRPC 桩」。** protoc 的 `--grpc_out` 产出的桩 `#include` 对应的 `.pb.h`,两步在 C++ 里不可分。 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 63ad2dc..b2f76ef 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -25,7 +25,20 @@ jobs: build: name: ${{ matrix.name }} runs-on: ${{ matrix.os }} - timeout-minutes: 120 + # Every example is its own project depending on `grpc` by path, and each + # one builds gRPC from source again — the cache does not carry across + # them. Measured on linux: module test 25min, then ~31min per example. + # Three examples already sat at ~90min under the old 120 cap; adding + # `advanced` pushed the run past it and the job was cancelled mid-build, + # which reads like a failure and is not one. + # + # Raising the cap is the honest short-term answer, not the right one: + # what actually costs the hour and a half is rebuilding the same gRPC + # four times in series. Splitting the examples into their own matrix leg + # (or getting the dependency cache to hit across sibling projects) turns + # ~2h of wall clock into ~35min. Worth doing on its own, not inside a + # release PR. + timeout-minutes: 210 strategy: fail-fast: false matrix: @@ -105,6 +118,38 @@ jobs: run: | "$MCPP" run + # The layered knobs, in the shape a real project hits first: a shared + # .proto tree that must be GENERATED (not merely searched), gRPC's mock + # generation, and plan/submit with an assertion in between. greeter + # covers L0; without this one, everything past the two original options + # is untested. + - name: advanced example — extra_dirs + mock + plan/submit + shell: bash + working-directory: examples/advanced + env: + MCPP_INDEX_MIRROR: GLOBAL + MCPP_BUILD_CACHE: local + run: | + "$MCPP" run + # The mock header is declared by the edge and produced by protoc; + # nothing includes it (no gmock in this ecosystem), so assert the + # file itself rather than trusting the plan-time check alone. + out=$(find target/.build-mcpp/out -name 'orders_mock.grpc.pb.h' | head -1) + test -n "$out" || { echo "no mock header produced"; exit 1; } + echo "mock header: $out" + # The shared tree was generated, not just searched. + test -f target/.build-mcpp/out/common/types.pb.cc \ + || { echo "shared-proto was not generated"; exit 1; } + # The edge describes ITSELF. This is the rule's half of observability + # — mcpp already puts the full argv in build.ninja, so what the rule + # owns is "which knobs produced it", and that string is what every + # build prints. Asserting it keeps the two from drifting apart. + nj=$(find target -name build.ninja | head -1) + grep -q 'description = GENERATE protoc:orders (+grpc +mock,' "$nj" \ + || { echo "edge description does not describe its knobs:"; + grep -o 'description = GENERATE protoc:.*' "$nj"; exit 1; } + grep -o 'description = GENERATE protoc:.*' "$nj" + package-versions-match: name: all three packages share one version runs-on: ubuntu-latest diff --git a/README.md b/README.md index 1ad7eef..c2001df 100644 --- a/README.md +++ b/README.md @@ -107,12 +107,54 @@ second language here the way xmake has Lua rules and Bazel has Starlark. It need **mcpp 2026.8.6.2** — 2026.8.5.2 is where a rule package first became able to use `import std;` and `import mcpp;`. -Two examples, on purpose: +## When you need more control + +Past `generate_all()` there is no cliff — each layer is the next one's +**defaults, not another road**: `generate_all(opt)` *is* `submit(plan_all(opt))`, +and `.grpc = true` *is* `.plugins = {cpp()}`. + +```cpp +// L1 — declarative knobs +grpcgen::generate_all({ + .extra_dirs = {"../shared-proto"}, // shared .proto tree, generated too + .imports = {"/opt/googleapis"}, // search only, not generated + .mock = true, // gRPC's generate_mock_code=true + .protoc_args = {"--experimental_allow_proto3_optional"}, +}); + +// L2 — the plugin list; `.grpc = true` is sugar for "cpp() is in it" +grpcgen::generate_all({ .plugins = { grpcgen::cpp(), my_plugin } }); + +// L3 — plan / submit: no requirement is exotic enough to need bypassing the rule +auto edges = grpcgen::plan_all(); +for (auto& e : edges) e.arg("--whatever"); +grpcgen::submit(edges); +``` + +`extra_dirs` vs `imports` is not a pedantic distinction: protoc writes +`#include "common/types.pb.h"` into every file that imports that .proto, so a +shared tree reached only through `-I` yields a header **nobody produced**, and +the failure surfaces far from its cause. Use `extra_dirs` when that code is +yours to build; `imports` when it comes from somewhere you already link. + +**Observability, and whose job it is**: mcpp writes each edge's full argv into +`build.ninja` — `ninja -t commands ` recovers it, and the rule does not +duplicate that. What the rule owns is *which knobs* produced the command, and it +puts that in the description every build prints: + +``` +GENERATE protoc:orders (+grpc +mock, -Iproto -I../shared-proto) +``` + +`examples/advanced` covers L1 + L3. + +Three examples, on purpose: | | | |---|---| | `examples/greeter` | the template instantiated — 3-line `build.mcpp` via `grpcgen` | | `examples/helloworld` | the same program with the rule written out by hand, so the mechanism stays legible | +| `examples/advanced` | the layered knobs: a shared .proto tree generated across roots, gRPC mocks, and plan/submit with an assertion in between | Three properties this buys, none of which hand-managed codegen can offer: diff --git a/README.zh.md b/README.zh.md index e7260f6..f3836ed 100644 --- a/README.zh.md +++ b/README.zh.md @@ -102,12 +102,52 @@ int main() { return grpcgen::generate_all() ? 0 : 1; } Bazel 用 Starlark。它需要 **mcpp 2026.8.6.2**;2026.8.5.2 是规则包从那一版起才真正能用 `import std;` 与 `import mcpp;` 的。 -两个示例,是刻意的: +## 需要更多控制时 + +`generate_all()` 之外不是断崖 —— 每一层都是下一层的默认值,**不是另一条路**: +`generate_all(opt)` 就是 `submit(plan_all(opt))`,`.grpc = true` 就是 +`.plugins = {cpp()}`。 + +```cpp +// L1 —— 声明式旋钮 +grpcgen::generate_all({ + .extra_dirs = {"../shared-proto"}, // 也要生成的共享 .proto 树 + .imports = {"/opt/googleapis"}, // 只搜索,不生成 + .mock = true, // gRPC 的 generate_mock_code=true + .protoc_args = {"--experimental_allow_proto3_optional"}, +}); + +// L2 —— 插件列表;`.grpc = true` 是「列表里有 cpp()」的语法糖 +grpcgen::generate_all({ .plugins = { grpcgen::cpp(), my_plugin } }); + +// L3 —— plan / submit 分离:再离谱的需求也不必绕开规则 +auto edges = grpcgen::plan_all(); +for (auto& e : edges) e.arg("--whatever"); +grpcgen::submit(edges); +``` + +`extra_dirs` 与 `imports` 的区别不是学究:protoc 会把 +`#include "common/types.pb.h"` 写进任何 import 了它的文件,所以只靠 `-I` 够到的 +共享树会产出一个**没人生成**的头,报错还离原因很远。代码归你构建就用 +`extra_dirs`,代码来自别处(你已经链接的包)才用 `imports`。 + +**可观测性的分工**:每条边的完整命令行由 mcpp 写进 `build.ninja`, +`ninja -t commands ` 就能取 —— 规则不重造。规则负责的是「**哪些旋钮**产生 +了它」,写在每次构建都会打印的 description 里: + +``` +GENERATE protoc:orders (+grpc +mock, -Iproto -I../shared-proto) +``` + +`examples/advanced` 覆盖 L1 + L3。 + +三个示例,是刻意的: | | | |---|---| | `examples/greeter` | 模板的实例化 —— 经 `grpcgen`,`build.mcpp` 三行 | | `examples/helloworld` | 同一个程序,但把规则手工摊开写,让机制保持可读 | +| `examples/advanced` | 分层旋钮:跨根生成的共享 .proto 树、gRPC mock、plan/submit 中间加断言 | 由此得到三个手工管理 codegen 给不了的性质: @@ -154,6 +194,8 @@ rules/ `grpcgen` —— codegen 规则包(host module) plugin/ `grpc_cpp_plugin` —— 独立的 codegen 工具包 examples/greeter/ 模板实例化:三行 build.mcpp examples/helloworld/ 同一个程序,规则手工摊开写 +examples/advanced/ 分层旋钮(extra_dirs / mock / plan-submit) +examples/shared-proto/ 被 advanced 跨根生成的共享 .proto 树 ``` `mcpp.toml` 里那份 995 条的源码清单是**上游自己的** —— `add_library(gpr)`、 diff --git a/examples/advanced/build.mcpp b/examples/advanced/build.mcpp new file mode 100644 index 0000000..0956ab4 --- /dev/null +++ b/examples/advanced/build.mcpp @@ -0,0 +1,57 @@ +// L1 + L3 in one program. +// +// L1 — two knobs a real project hits early: +// .extra_dirs orders.proto imports common/types.proto from a SHARED tree. +// Listed here rather than in .imports because its generated +// code is ours to build too: protoc emits +// `#include "common/types.pb.h"` into orders.pb.h, so a root +// reached only by `-I` yields a header nobody produced. +// .mock gRPC's own generate_mock_code=true. A PLUGIN parameter, not a +// protoc flag — the kind of spelling a consumer should not have +// to know. +// +// L3 — plan, look, submit. Nothing here is a different code path: +// `generate_all(opt)` IS `submit(plan_all(opt))`. The point of splitting it is +// that a project with a need the rule does not model keeps everything the rule +// already solved (well-known types, output subdirectories, the input set) +// instead of hand-writing a build.mcpp that reimplements them and then drifts. +import std; +import mcpp; +import grpcgen; + +int main() { + auto edges = grpcgen::plan_all({ + .extra_dirs = {"../shared-proto"}, + .mock = true, + }); + if (edges.empty()) return 1; + + // Assert the mock output was DECLARED. Deliberately not `#include`d by + // src/main.cpp: the generated mock header pulls in , and + // this ecosystem's compat.gtest ships googletest without gmock. Declaring + // it still proves the parameter reached the plugin — protoc must produce + // the file, since it is an output of the edge. + bool sawMock = false; + for (const auto& e : edges) + for (const auto& o : e.outputs) + if (o.ends_with("_mock.grpc.pb.h")) sawMock = true; + if (!sawMock) { + std::println(std::cerr, + "advanced: .mock did not reach the plugin — no *_mock.grpc.pb.h " + "among the planned outputs"); + return 1; + } + + // The shared tree is generated too, not merely searched. + bool sawShared = false; + for (const auto& e : edges) + if (e.id == "protoc:common/types") sawShared = true; + if (!sawShared) { + std::println(std::cerr, + "advanced: .extra_dirs did not become a generation root — no edge " + "for common/types"); + return 1; + } + + return grpcgen::submit(edges) ? 0 : 1; +} diff --git a/examples/advanced/mcpp.toml b/examples/advanced/mcpp.toml new file mode 100644 index 0000000..432aa9d --- /dev/null +++ b/examples/advanced/mcpp.toml @@ -0,0 +1,37 @@ +# advanced — the layered knobs, in the one shape a real project hits first. +# +# examples/greeter is L0: `generate_all()`, nothing else. This one is L1: +# +# * `.imports` — orders.proto imports common/types.proto, which lives in a +# SHARED directory outside this project. Cross-repo .proto is the normal +# state once there is more than one service, and `google/api/annotations +# .proto` reaches every project that exposes an HTTP gateway. +# * `.mock` — gRPC's own generator emits MockOrdersStub for unit tests. It is +# a PLUGIN parameter (`--grpc_out=generate_mock_code=true:`), not a +# protoc flag, which is exactly the spelling a consumer should not have to +# know. +# +# Both are additive: dropping them gives byte-identical output to greeter's. +# +# Path dependencies, like greeter — this example tests the working tree. +[package] +name = "advanced" +version = "0.1.0" +standard = "c++23" + +[build] +sources = ["src/main.cpp"] + +[targets.advanced] +kind = "bin" +main = "src/main.cpp" + +[dependencies] +grpc = { path = "../.." } +grpc-plugin = { path = "../../plugin", tools = ["grpc_cpp_plugin"] } + +[dependencies.mcpplibs] +grpcgen = { path = "../../rules", host-module = true } + +[dependencies.compat] +protobuf = { version = "35.1", tools = ["protoc"] } diff --git a/examples/advanced/proto/orders.proto b/examples/advanced/proto/orders.proto new file mode 100644 index 0000000..7a4e266 --- /dev/null +++ b/examples/advanced/proto/orders.proto @@ -0,0 +1,11 @@ +syntax = "proto3"; +package orders; + +import "common/types.proto"; // resolved via .imports, not via proto/ + +message PlaceRequest { string sku = 1; common.Trace trace = 2; } +message PlaceReply { string order_id = 1; } + +service Orders { + rpc Place (PlaceRequest) returns (PlaceReply); +} diff --git a/examples/advanced/src/main.cpp b/examples/advanced/src/main.cpp new file mode 100644 index 0000000..057f5c8 --- /dev/null +++ b/examples/advanced/src/main.cpp @@ -0,0 +1,28 @@ +// Proves all three products of the L1 configuration exist and compile: +// orders.pb.h protobuf messages (--cpp_out) +// orders.grpc.pb.h gRPC service stubs (--grpc_out) +// The mock header (orders_mock.grpc.pb.h) is DECLARED by the edge and produced +// by protoc, but not included here: it pulls in , which this +// ecosystem's compat.gtest does not ship. build.mcpp asserts it was planned. +// +// The cross-root import is proven by orders.pb.h alone: orders.proto imports +// common/types.proto, which protoc can only resolve through the extra -I. +#include +#include + +#include "orders.pb.h" +#include "orders.grpc.pb.h" + +int main() { + orders::PlaceRequest req; + req.set_sku("mcpp-42"); + req.mutable_trace()->set_id("t-1"); // ← type from the SHARED root + if (req.trace().id() != "t-1") return 1; + + const std::string svc = orders::Orders::service_full_name(); + std::printf("service = %s\n", svc.c_str()); + if (svc != "orders.Orders") return 1; + + std::printf("advanced: OK (messages + stubs, cross-root generation)\n"); + return 0; +} diff --git a/examples/shared-proto/common/types.proto b/examples/shared-proto/common/types.proto new file mode 100644 index 0000000..fcd3c2b --- /dev/null +++ b/examples/shared-proto/common/types.proto @@ -0,0 +1,7 @@ +syntax = "proto3"; +package common; + +// Lives OUTSIDE the project's own proto/ — reachable only through an extra +// `-I` root, which is what `.imports` exists for. A shared .proto repository +// is the normal shape once more than one service is involved. +message Trace { string id = 1; } diff --git a/rules/src/grpcgen.cppm b/rules/src/grpcgen.cppm index fde5af8..6b62803 100644 --- a/rules/src/grpcgen.cppm +++ b/rules/src/grpcgen.cppm @@ -1,10 +1,10 @@ -// grpcgen — protoc + grpc_cpp_plugin as an importable build rule. +// grpcgen — protoc + plugins as an importable build rule. // // Consumers write three lines and never see anything below: // // import mcpp; // import grpcgen; -// int main() { return grpcgen::generate({"helloworld"}) ? 0 : 1; } +// int main() { return grpcgen::generate_all() ? 0 : 1; } // // `host-module = true` makes mcpp compile this interface in the same command as // the consumer's build.mcpp, which is what makes the BMI importable there and @@ -16,6 +16,41 @@ // release: before it, a rule was compiled before the std module existed // (`module 'std' not found`) and was ALSO compiled as an ordinary library of // the consumer, where the `mcpp` module does not exist. +// +// ── WHAT CODEGEN ACTUALLY IS ─────────────────────────────────────────────── +// +// Two protoc invocations' worth of work, and only the second is gRPC's: +// +// --cpp_out= -> .pb.{h,cc} protobuf messages +// --grpc_out= --plugin=... -> .grpc.pb.{h,cc} gRPC service stubs +// +// This rule issues them as ONE protoc call (protoc accepts both outputs at +// once), but they are two different products from two different projects. That +// is why a gRPC project needs `compat.protobuf`'s protoc at all — the message +// half is protobuf's, not gRPC's, and no gRPC-only tool can produce it. The +// service stubs `#include` the message headers, so the two are not separable +// in C++. +// +// ── THREE LAYERS, EACH THE NEXT ONE'S DEFAULTS ───────────────────────────── +// +// L0 generate_all() — the whole directory, no arguments +// L1 generate_all({.imports=…, .mock=…, .protoc_args=…}) +// L2 generate_all({.plugins={cpp(), my_plugin}}) +// L3 auto e = plan_all(); /* edit */ ; submit(e); +// +// They are not alternatives: `generate_all(opt)` IS `submit(plan_all(opt))`, +// and `.grpc = true` IS `.plugins = {cpp()}`. A consumer that needs L3 keeps +// everything the rule already solved — locating the well-known types, creating +// output subdirectories for nested .proto, computing the input set — instead of +// hand-writing a build.mcpp that reimplements them and then drifts. +// +// WHAT THIS RULE DOES NOT DO: print command lines. mcpp already writes every +// action's full argv into build.ninja (`rule mcpp_action_N / command = …`), +// recoverable with `ninja -t commands `. A second source of that truth +// would only drift. The rule owns the other half — WHICH KNOBS produced the +// command — and puts it in each edge's description. +// +// See .agents/docs/2026-08-06-grpcgen-layered-control-design.md. export module grpcgen; import std; @@ -49,52 +84,201 @@ std::string well_known_types_dir() { export namespace grpcgen { +// ── L2: a protoc plugin ──────────────────────────────────────────────────── +// +// One plugin is four things, and the fourth is not optional: mcpp has to know +// which files the edge PRODUCES to put the .cc in the compile set and keep the +// .h out of it. A plugin that cannot say what it emits cannot be a build-graph +// node at all. +struct plugin { + // The protoc name: drives `--_out=` and `--plugin=protoc-gen-=`. + std::string name; + // Absolute path to the generator executable. Empty = unresolved; the rule + // reports it by name rather than letting protoc fail with its own wording. + std::string binary; + // Plugin parameters, joined with ',' and prefixed onto the output dir — + // protoc's `--x_out=a=1,b=2:` grammar. + std::vector params; + // Suffixes appended to the .proto's stem, e.g. ".grpc.pb.cc". + std::vector suffixes; +}; + +// The gRPC C++ plugin, resolved from the dependency that provides it. +// +// `mock = true` adds gRPC's own `generate_mock_code=true`, whose extra output +// is `_mock.grpc.pb.h`. It is a PLUGIN parameter, not a protoc flag — +// exactly the kind of spelling a consumer should not have to know, which is +// why it is a named option rather than something to hand-write into +// `protoc_args`. +plugin cpp(bool mock = false) { + plugin p; + p.name = "grpc"; + const char* bin = mcpp::dep_bin("grpc-plugin", "grpc_cpp_plugin"); + p.binary = bin ? bin : ""; + p.suffixes = { ".grpc.pb.cc", ".grpc.pb.h" }; + if (mock) { + p.params.push_back("generate_mock_code=true"); + p.suffixes.push_back("_mock.grpc.pb.h"); + } + return p; +} + struct options { // Where the .proto files live, relative to the consumer's manifest. std::string_view proto_dir = "proto"; - // Also run grpc_cpp_plugin. Set false for a protobuf-only project, which - // then needs neither the plugin dependency nor its tool. + // Also run the gRPC C++ plugin. Sugar for `.plugins = {cpp(mock)}`; ignored + // when `plugins` is set explicitly. Set false for a protobuf-only project, + // which then needs neither the plugin dependency nor its tool. bool grpc = true; + // Additional trees to GENERATE from, each also becoming an `-I` root. + // Names keep their path relative to their own root, so + // ../shared-proto/common/types.proto emits common/types.pb.{h,cc}. + // + // Distinct from `imports` on purpose, and the distinction is not academic: + // protoc emits `#include "common/types.pb.h"` into any file that imports + // that .proto, so a shared tree reached only through `-I` yields a header + // nobody produced — and the failure surfaces as a missing include far from + // its cause. Use this when the shared .proto's code is yours to build. + std::vector extra_dirs; + // Extra `-I` roots, SEARCH ONLY — no code is generated for what they + // contain. Use this when the generated code comes from somewhere else + // (a package you already link that ships it). If you need the code too, + // use `extra_dirs`. + std::vector imports; + // Generate gRPC's mock classes for unit tests. See cpp(). + bool mock = false; + // Escape hatch for protoc flags this rule does not model, e.g. + // "--experimental_allow_proto3_optional". + std::vector protoc_args; + // L2: the full plugin list. Non-empty overrides `grpc`/`mock`. + std::vector plugins; }; -// Declare one codegen edge per .proto. Names are given WITHOUT the extension, -// relative to `opt.proto_dir` — "helloworld" means /helloworld.proto. +// ── L3: a planned edge, before it is submitted ───────────────────────────── // -// Returns false after printing a diagnostic; a build program should propagate -// that as a non-zero exit. -bool generate(std::vector protos, options opt = {}) { +// Owns its strings, unlike `mcpp::action` whose id/description are raw +// pointers — a planned edge outlives the expression that built it. +struct edge { + std::string id; + std::string description; + std::vector command; + std::vector inputs; + std::vector outputs; + + edge& arg(std::string a) { command.push_back(std::move(a)); return *this; } + edge& input(std::string p) { inputs.push_back(std::move(p)); return *this; } + edge& output(std::string p) { outputs.push_back(std::move(p)); return *this; } +}; + +namespace detail { + +std::vector resolve_plugins(const options& opt) { + if (!opt.plugins.empty()) return opt.plugins; + if (!opt.grpc) return {}; + return { cpp(opt.mock) }; +} + +// "protoc:echo (+grpc +mock, -I proto -I ../shared/proto)" +// +// The description is what every build prints, so it is where "which knobs +// produced this" belongs. The command itself is in build.ninja; see the module +// header for why the rule does not duplicate it. +std::string describe(const std::string& name, + const std::vector& plugins, + const std::vector& userIncs, + const std::string& root) { + std::string s = "protoc:" + name + " ("; + if (plugins.empty()) { + s += "messages-only"; + } else { + bool first = true; + for (const auto& p : plugins) { + if (!first) s += " "; + first = false; + s += "+" + p.name; + for (const auto& q : p.params) { + // `generate_mock_code=true` reads as `+mock` — the option's + // name, not protoc's spelling of it. + if (q == "generate_mock_code=true") s += " +mock"; + } + } + } + // Only the roots the AUTHOR chose. The well-known-types directory is + // always present and is not a knob, and absolute store paths would bury + // the part worth reading. Rendered relative to the manifest so the string + // matches what was written in build.mcpp. + for (const auto& i : userIncs) { + // Prefer the relative form — it is usually what the author wrote — but + // only while it stays readable. `lexically_relative` SUCCEEDS for a + // path far from the manifest and returns a chain of `..` longer than + // the absolute path it replaced; guarding only on "could not compute" + // misses that case entirely. + auto rel = std::filesystem::path(i).lexically_relative(root); + auto r = rel.generic_string(); + auto shown = (r.empty() || r.starts_with("../..")) ? i : r; + s += (i == userIncs.front() ? ", -I" : " -I") + shown; + } + s += ")"; + return s; +} + +} // namespace detail + +// Build the edges for the named .proto files WITHOUT submitting them. +// +// Names are given WITHOUT the extension, relative to `opt.proto_dir` — +// "helloworld" means /helloworld.proto. +// +// Returns an empty vector after printing a diagnostic. +// ── L3, bottom: a .proto addressed by (root, name) ───────────────────────── +// +// `plan()` and `plan_all()` both funnel here. It is public because it is the +// only form that can express a .proto living somewhere neither `proto_dir` nor +// `extra_dirs` covers — without it, such a project would be back to writing +// its own build.mcpp, which is the cliff this whole design exists to remove. +// +// `name` is relative to `root` and decides the OUTPUT path, so two roots may +// contribute the same relative name only if the author meant it — exactly the +// property protoc's own -I list has. +struct entry { std::string root, name; }; + +// Plan edges for .proto files given as explicit (root, name) pairs. +std::vector plan_entries(std::vector protos, options opt = {}) { const std::string root = mcpp::manifest_dir(); const std::string out = mcpp::out_dir(); if (root.empty() || out.empty()) { std::println(std::cerr, "grpcgen: no mcpp build context — this runs from build.mcpp"); - return false; + return {}; } if (protos.empty()) { - std::println(std::cerr, "grpcgen::generate() called with no .proto files"); - return false; + std::println(std::cerr, "grpcgen: no .proto files to plan"); + return {}; } const char* protoc = mcpp::dep_bin("protobuf", "protoc"); if (!protoc || !*protoc) { std::println(std::cerr, - "grpcgen: no protoc. Declare it in mcpp.toml:\n" - " compat.protobuf = {{ version = \"35.1\", tools = [\"protoc\"] }}"); - return false; + "grpcgen: no protoc. It generates the PROTOBUF half of the output " + "(.pb.cc/.pb.h), which is protobuf's, not gRPC's — so it is " + "declared on protobuf:\n" + " compat.protobuf = {{ version = \"35.1\", tools = [\"protoc\"] }}\n" + " (or simply grpc = {{ version = \"...\", features = [\"codegen\"] }}," + " which hands you all of it)"); + return {}; } - const char* plugin = ""; - if (opt.grpc) { - plugin = mcpp::dep_bin("grpc-plugin", "grpc_cpp_plugin"); - if (!plugin || !*plugin) { - std::println(std::cerr, - "grpcgen: no grpc_cpp_plugin. Declare it in mcpp.toml:\n" - " grpc-plugin = {{ version = \"...\", tools = " - "[\"grpc_cpp_plugin\"] }}\n" - " (or call grpcgen::generate(..., {{.grpc = false}}) for a " - "protobuf-only project)"); - return false; - } + auto plugins = detail::resolve_plugins(opt); + for (const auto& p : plugins) { + if (!p.binary.empty()) continue; + std::println(std::cerr, + "grpcgen: plugin '{}' has no executable. For the built-in gRPC C++ " + "plugin declare:\n" + " grpc-plugin = {{ version = \"...\", tools = " + "[\"grpc_cpp_plugin\"] }}\n" + " (or grpcgen::generate_all({{.grpc = false}}) for a " + "protobuf-only project)", p.name); + return {}; } const std::string wkt = detail::well_known_types_dir(); @@ -102,50 +286,67 @@ bool generate(std::vector protos, options opt = {}) { std::println(std::cerr, "grpcgen: cannot locate the well-known .proto files inside the " "protobuf package"); - return false; + return {}; } - const std::string protoDir = root + "/" + std::string(opt.proto_dir); + auto abs_under_root = [&](const std::string& v) { + std::filesystem::path p{v}; + return p.is_absolute() ? p.generic_string() + : (std::filesystem::path(root) / p) + .lexically_normal().generic_string(); + }; + const std::string protoDir = abs_under_root(std::string(opt.proto_dir)); + + // Include roots, in the order protoc sees them: the generation roots (own + // tree first), then search-only roots, then the well-known types. + std::vector incs{ protoDir }; + for (const auto& d : opt.extra_dirs) incs.push_back(abs_under_root(d)); + for (const auto& i : opt.imports) incs.push_back(abs_under_root(i)); + incs.push_back(wkt); // Every .proto is an input of every action. A .proto that imports a sibling // has a real dependency this rule does not parse, and regenerating a little // too eagerly is far better than silently stale stubs. std::vector inputs; inputs.reserve(protos.size()); - for (const auto& n : protos) - inputs.push_back(std::format("{}/{}.proto", protoDir, n)); + for (const auto& e : protos) + inputs.push_back(std::format("{}/{}.proto", e.root, e.name)); + + std::vector edges; + edges.reserve(protos.size()); - for (const auto& name : protos) { - const std::string src = std::format("{}/{}.proto", protoDir, name); + for (const auto& ent : protos) { + const std::string& name = ent.name; + const std::string src = std::format("{}/{}.proto", ent.root, name); const std::string base = std::format("{}/{}", out, name); - // The work is DECLARED, not done. Running protoc here would re-run it - // on every prepare, serially, and report failure as "build.mcpp exited - // 1". As an action it is an edge in the build graph: it re-runs when - // its inputs change, in parallel, and a failure is attributed to the - // edge that produced it. - // - // The headers are declared as outputs because they must be PRODUCED by - // this edge; mcpp knows a header is not a translation unit and keeps - // them out of the compile set. - mcpp::action gen; - const std::string id = std::format("protoc:{}", name); - gen.id = id.c_str(); - gen.role = "source"; - gen.description = id.c_str(); + edge e; + e.id = std::format("protoc:{}", name); + // Only the author-chosen roots: incs also carries the well-known-types + // directory, which is constant and not worth printing every build. + e.description = detail::describe( + name, plugins, + std::vector(incs.begin(), incs.end() - 1), root); - const std::string incProto = "-I" + protoDir; - const std::string incWkt = "-I" + wkt; - const std::string cppOut = "--cpp_out=" + out; - gen.arg(protoc).arg(incProto.c_str()).arg(incWkt.c_str()) - .arg(cppOut.c_str()); + e.arg(protoc); + for (const auto& i : incs) e.arg("-I" + i); + e.arg("--cpp_out=" + out); - const std::string grpcOut = "--grpc_out=" + out; - const std::string pluginArg = std::string("--plugin=protoc-gen-grpc=") + plugin; - if (opt.grpc) gen.arg(grpcOut.c_str()).arg(pluginArg.c_str()); - gen.arg(src.c_str()); + for (const auto& p : plugins) { + std::string spec; + for (const auto& q : p.params) { + if (!spec.empty()) spec += ","; + spec += q; + } + // protoc's grammar: `--x_out=:`; the colon is only + // present when there are params. + e.arg("--" + p.name + "_out=" + (spec.empty() ? out : spec + ":" + out)); + e.arg("--plugin=protoc-gen-" + p.name + "=" + p.binary); + } + for (const auto& a : opt.protoc_args) e.arg(a); + e.arg(src); - for (const auto& in : inputs) gen.input(in.c_str()); + for (const auto& in : inputs) e.input(in); // protoc mirrors the .proto's relative path under --cpp_out and does // NOT create the intermediate directories. A flat proto/ never notices; @@ -156,69 +357,135 @@ bool generate(std::vector protos, options opt = {}) { std::filesystem::create_directories( std::filesystem::path(base).parent_path(), mkec); } - const std::string pbcc = base + ".pb.cc", pbh = base + ".pb.h"; - gen.output(pbcc.c_str()).output(pbh.c_str()); - const std::string gcc_ = base + ".grpc.pb.cc", gh = base + ".grpc.pb.h"; - if (opt.grpc) gen.output(gcc_.c_str()).output(gh.c_str()); - gen.submit(); - } + // The headers are declared as outputs because they must be PRODUCED by + // this edge; mcpp knows a header is not a translation unit and keeps + // them out of the compile set. + e.output(base + ".pb.cc").output(base + ".pb.h"); + for (const auto& p : plugins) + for (const auto& s : p.suffixes) e.output(base + s); - // Where the generated headers live. PRIVATE to the consuming package by - // design — an include dir that a package's own consumers must see belongs - // in its manifest, not in a build program. - mcpp::include_dir(out.c_str()); - return true; + edges.push_back(std::move(e)); + } + return edges; } -// Every .proto under `opt.proto_dir`, without naming any of them. +// Every .proto under `opt.proto_dir`, without naming any of them — planned but +// not submitted. // -// This is the form the greeter template uses, and it is only SAFE because the -// engine can express "my output depends on which files are here" -// (`rerun_if_changed_glob`, mcpp 2026.8.6.2+). Before that, a build program -// that globbed did not re-run when a .proto was added — no declared file's -// hash had changed — so the new file was silently never generated, which is -// worse than making the author list names. That is why this repository shipped -// the explicit list first. +// Safe only because the engine can express "my output depends on which files +// are here" (`rerun_if_changed_glob`, mcpp 2026.8.6.2+). Before that, a build +// program that globbed did not re-run when a .proto was added — no declared +// file's hash had changed — so the new file was silently never generated, +// which is worse than making the author list names. That is why this +// repository shipped the explicit list first. // -// Names are relative to `opt.proto_dir` and keep their subdirectory, so -// proto/sub/x.proto generates sub/x.pb.cc and is imported as "sub/x.proto". -bool generate_all(options opt = {}) { - namespace fs = std::filesystem; +// Names keep their subdirectory, so proto/sub/x.proto generates sub/x.pb.cc. +// Build the edges for the named .proto files WITHOUT submitting them. +// +// Names are given WITHOUT the extension, relative to `opt.proto_dir` — +// "helloworld" means /helloworld.proto. Files under `extra_dirs` +// are not addressable this way; use plan_all() for those. +// +// Returns an empty vector after printing a diagnostic. +std::vector plan(std::vector protos, options opt = {}) { const std::string root = mcpp::manifest_dir(); if (root.empty()) { std::println(std::cerr, "grpcgen: no mcpp build context — this runs from build.mcpp"); - return false; + return {}; } - const std::string pattern = std::string(opt.proto_dir) + "/**/*.proto"; - mcpp::rerun_if_changed_glob(pattern.c_str()); + std::filesystem::path pd{std::string(opt.proto_dir)}; + const std::string base = pd.is_absolute() + ? pd.generic_string() + : (std::filesystem::path(root) / pd).lexically_normal().generic_string(); + std::vector entries; + entries.reserve(protos.size()); + for (auto& n : protos) entries.push_back({ base, std::move(n) }); + return plan_entries(std::move(entries), std::move(opt)); +} - const fs::path dir = fs::path(root) / opt.proto_dir; - std::error_code ec; - if (!fs::exists(dir, ec)) { +std::vector plan_all(options opt = {}) { + namespace fs = std::filesystem; + const std::string root = mcpp::manifest_dir(); + if (root.empty()) { std::println(std::cerr, - "grpcgen::generate_all(): no '{}' directory under {}", - opt.proto_dir, root); - return false; + "grpcgen: no mcpp build context — this runs from build.mcpp"); + return {}; } - std::vector names; - for (auto const& e : fs::recursive_directory_iterator(dir, ec)) { - if (ec) break; - if (!e.is_regular_file(ec)) continue; - if (e.path().extension() != ".proto") continue; - auto rel = e.path().lexically_relative(dir); - rel.replace_extension(); - names.push_back(rel.generic_string()); + // Every generation root: the project's own tree, then `extra_dirs`. + std::vector roots{ std::string(opt.proto_dir) }; + for (const auto& d : opt.extra_dirs) roots.push_back(d); + + std::vector entries; + for (const auto& r : roots) { + // One glob per root — a .proto appearing in ANY of them must re-run + // this program, and only the path SET is fingerprinted (contents are + // covered by the edges' own inputs). + const std::string pattern = r + "/**/*.proto"; + mcpp::rerun_if_changed_glob(pattern.c_str()); + + fs::path rp{r}; + const fs::path dir = rp.is_absolute() ? rp + : (fs::path(root) / rp).lexically_normal(); + std::error_code ec; + if (!fs::exists(dir, ec)) { + std::println(std::cerr, + "grpcgen::plan_all(): no '{}' directory under {}", r, root); + return {}; + } + std::vector names; + for (auto const& e : fs::recursive_directory_iterator(dir, ec)) { + if (ec) break; + if (!e.is_regular_file(ec)) continue; + if (e.path().extension() != ".proto") continue; + auto rel = e.path().lexically_relative(dir); + rel.replace_extension(); + names.push_back(rel.generic_string()); + } + // Sorted so the declared edge set is stable run to run — an unstable + // order would churn build.ninja for no reason. + std::ranges::sort(names); + for (auto& n : names) + entries.push_back({ dir.generic_string(), std::move(n) }); } - if (names.empty()) { + if (entries.empty()) { std::println(std::cerr, - "grpcgen::generate_all(): no .proto files under {}", dir.string()); - return false; + "grpcgen::plan_all(): no .proto files under {}", opt.proto_dir); + return {}; } - // Sorted so the declared edge set is stable run to run — an unstable order - // would churn build.ninja for no reason. - std::ranges::sort(names); - return generate(std::move(names), opt); + return plan_entries(std::move(entries), std::move(opt)); +} + +// Hand the planned edges to mcpp, and declare where the generated headers live. +// +// That include dir is PRIVATE to the consuming package by design — one a +// package's own consumers must see belongs in its manifest, not in a build +// program. +bool submit(const std::vector& edges) { + if (edges.empty()) return false; + for (const auto& e : edges) { + mcpp::action a; + a.id = e.id.c_str(); + a.role = "source"; + a.description = e.description.c_str(); + for (const auto& c : e.command) a.arg(c.c_str()); + for (const auto& i : e.inputs) a.input(i.c_str()); + for (const auto& o : e.outputs) a.output(o.c_str()); + a.submit(); + } + mcpp::include_dir(mcpp::out_dir()); + return true; +} + +// Declare one codegen edge per named .proto. `plan` + `submit`, with nothing +// in between — the two-call form exists so a consumer can put something there. +bool generate(std::vector protos, options opt = {}) { + return submit(plan(std::move(protos), std::move(opt))); +} + +// Every .proto under `opt.proto_dir`. The form the greeter template uses. +bool generate_all(options opt = {}) { + return submit(plan_all(std::move(opt))); } } // namespace grpcgen