diff --git a/.agents/docs/2026-08-04-add-re2-and-protobuf-upb-plan.md b/.agents/docs/2026-08-04-add-re2-and-protobuf-upb-plan.md new file mode 100644 index 00000000..af9d11f0 --- /dev/null +++ b/.agents/docs/2026-08-04-add-re2-and-protobuf-upb-plan.md @@ -0,0 +1,123 @@ +# 新增 compat.re2 2022-04-01 与 compat.protobuf 的 upb feature(2026-08-04) + +[gRPC 收录可行性分析](2026-08-04-grpc-feasibility-analysis.md) 的 **P1 索引侧**,承接 +[P0](2026-08-04-add-abseil-and-protobuf-plan.md)。目标是把 gRPC 剩下的索引侧依赖补齐,使后续 +`grpc-m` 独立仓只需 vendor gRPC 自己的源码。 + +产出: + +| 变更 | 规模 | 说明 | +|---|---|---| +| `compat.re2@2022-04-01` | 22 TU | 新包,Shape A | +| `compat.protobuf` 新增 `upb` feature | 64 TU | 源码全在已收录的 protobuf tarball 内,无新下载 | + +workspace 成员:`tests/examples/re2`、`tests/examples/protobuf-upb`。 + +## 0. 先做的事:确认 mcpp 真能编 gRPC + +在写任何描述符之前先打掉最大的未知数 —— 用 mcpp 的 gcc@16.1.0 直接编 gRPC 1.83.0 的代表性 TU, +只给 include 路径,不打补丁、不跑 configure、不做 codegen: + +``` +OK src/core/lib/slice/slice.cc +OK src/core/lib/surface/call.cc <- core 里最重的之一 +OK src/core/lib/promise/activity.cc +OK src/core/ext/upb-gen/google/protobuf/any.upb_minitable.c +OK src/cpp/common/channel_arguments.cc +OK src/cpp/client/channel_cc.cc <- C++ API 层 +``` + +以及 SSL 层对着 **compat.openssl 3.5.1 的头文件**(不是 boringssl)4/4 通过: + +``` +OK src/core/tsi/ssl_transport_security.cc +OK src/core/credentials/transport/ssl/ssl_credentials.cc +OK src/core/tsi/ssl/session_cache/ssl_session_openssl.cc +OK src/core/tsi/ssl_telemetry_utils.cc +``` + +需要的 include 路径只有:仓库根、`include`、`src/core/ext/upb-gen`、`src/core/ext/upbdefs-gen`, +外加 abseil / protobuf(含 upb)/ openssl。另核实 gRPC 源码树内**没有任何 `.h.in` 或 +`config.h.cmake`**,确认它不需要 configure 步骤。 + +结论:整源码直编路线成立,可以继续投入。 + +## 1. compat.re2 —— 为什么钉在 2022-04-01 + +`2022-04-01` 是 RE2 的上游 release tag,也是 **gRPC 1.83.0 的 pin**(其 `third_party/re2` submodule += commit `0c5616d` = 该 tag)。这个 pin 是有意义的:gRPC 的 xds matcher +(`src/core/util/matchers.h`、`src/core/xds/grpc/xds_route_config*`)是照着这一版 API 写的。 +RE2 在 2023 年的版本里把自己的字符串类型从 `re2::StringPiece` 换成了 `absl::string_view` +并引入了 Abseil 依赖,**新版不是 drop-in**。将来若有别的消费者需要新版,可按 `compat.catch2` +的先例在同一描述符里追加版本。 + +上游该 tag 没有 release asset,用 tag 归档(RE2 无 submodule,自包含),sha256 两次下载一致。 + +源码清单直接对应上游 `CMakeLists.txt` 的 `RE2_SOURCES`:`re2/*.cc` 全部(上游把测试放在 +低一级的 `re2/testing/`,glob 够不到)+ `util/rune.cc` + `util/strutil.cc`。`util/` 逐条列而不通配, +因为该目录另外两个 TU **都带 `main()`**(`util/benchmark.cc`、`util/test.cc`,连同 +`util/fuzz.cc` 与顶层 `testinstall.cc`),依赖的 `.o` 会全量入链,任何一个都会和消费者的 +`main()` 撞车;`util/pcre.cc` 还会额外拖进 libpcre。 + +## 2. upb feature —— 组成必须严格照抄上游 libupb + +upb 是 protobuf 自 v22 起吸收进来的小型 C 运行时,**源码就在已收录的 protobuf 35.1 tarball 里** +(根目录 `upb/`),所以不需要新包、不需要新下载。默认关闭:C++ 运行时一点都不用它,而它是 64 个额外 +C TU。gRPC 是需要它的那个消费者 —— `src/core/ext/upb-gen/**.c` 就是 upb 代码。 + +两处**不能想当然**的地方: + +1. **不能用 `upb/**/*.c` 通配。** 实测通配会多带 14 个文件,其中包含 **descriptor 表的两份替代构建** + (`upb/reflection/stage0/…` 与 `upb/reflection/cmake/…`)以及 `upb/message/promote.c`、 + `decode_fast/` 的若干变体。同时编进多份 descriptor 变体 = 重复符号链接失败。因此逐条转录 + `src/file_lists.cmake` 的 `libupb_srcs`(63 条)。 +2. **必须额外加上 `upb/reflection/cmake/google/protobuf/descriptor.upb_minitable.c`**,它**不在** + `libupb_srcs` 里。上游 `cmake/libupb.cmake` 正是把这一个文件作为 `bootstrap_sources` 叠加在清单之上, + 因为整个 reflection 层经 `upb/reflection/descriptor_bootstrap.h` 都要用 descriptor 表。 + 漏掉它能编译通过,**只在链接期炸**。 + +`include_dirs` 因此追加了两条(tarball 根、`upb/reflection/cmake`)。它们是**无条件**声明的: +`features` 能门控 sources/defines/deps,**不能门控 include dirs**,而 feature 自己的 TU 需要它们才能编译。 +两条都是纯追加、不构成遮蔽 —— 根目录只提供 `upb/…`(protobuf 的 C++ 头在 `src/` 下而非根), +bootstrap 目录提供的是 `google/protobuf/descriptor.upb*.h`,与隔壁 C++ 的 `descriptor.h` 文件名不同。 + +## 3. 验证结论 + +全部使用与 CI 一致的配置(mcpp **2026.8.3.3**、gcc@16.1.0、`MCPP_INDEX_MIRROR=GLOBAL`、 +`MCPP_BUILD_CACHE=local`),每个成员先删 `target/` 冷构建: + +``` +abseil test result ok. 1 passed; 0 failed (17.84s) +protobuf test result ok. 1 passed; 0 failed (40.88s) +protobuf-gzip test result ok. 1 passed; 0 failed (51.99s) +protobuf-upb test result ok. 1 passed; 0 failed (70.20s) +re2 test result ok. 1 passed; 0 failed ( 2.35s) +``` + +前三个是**回归**:本次改了 `compat.protobuf` 的 `include_dirs`,必须确认既有成员未受影响。 + +- **目标文件数已核对**:`re2` 23 个(22 源 + 1 测试);`protobuf-upb` 296 个 + = 151(abseil)+ 80(protobuf)+ 64(upb)+ 1(测试)。 +- **`upb` feature 做了负向验证**:把成员依赖改回裸 `protobuf = "35.1"` 后,目标文件数从 296 落到 + **232(正好少 64 个)**,链接给出 `undefined reference to upb_Arena_Init` / `upb_Decode` / + `upb_DefPool_Free`,以及 **`google__protobuf__FileDescriptorProto_msg_init`** —— 最后这一个正好 + 证明 §2 说的 bootstrap descriptor 表既是必需的、也确实由该 feature 带入。 + (注意关键词:这里工具链是 gcc,用 GNU ld,报的是 `undefined reference`;lld 才报 `undefined symbol`。) +- **测试打的是真符号**。re2 侧覆盖 FullMatch/PartialMatch(并断言 FullMatch **不**接受尾部残余)、 + 预编译 RE2 对象取捕获、GlobalReplace、`RE2::Set` 多模式匹配、非法模式必须被拒、UTF-8 按 rune 计数; + upb 侧让**两个运行时对同一段字节交叉验证** —— C++ 运行时构造并序列化 `FileDescriptorProto`, + 再由 upb 解析、灌进 `upb_DefPool`、回答反射查询,并断言一个从未声明过的名字**查不到**。 +- **CN 镜像已闭环**:`mcpp-res/re2@2022-04-01` 返回 `http=200` 且与 GLOBAL **字节一致**。 + +## 4. 下一步(grpc-m) + +索引侧到此备齐:`compat.abseil` / `compat.protobuf`(+`upb`)/ `compat.re2` / `compat.openssl`。 +`grpc-m` 只需 vendor gRPC 自己的源码。已知的两个待办: + +- **c-ares 暂不引入。** 上游有官方裁剪开关 `grpc_no_ares=true` → `GRPC_ARES=0` + (`BUILD:75`、`bazel/grpc_build_system.bzl:175`),且只有 10 个文件 include `ares.h`,全在 + `resolver/dns/c_ares/` 与 `event_engine/*ares*`,可 glob 排除。第一版走 gRPC 原生 DNS 解析器, + helloworld 连 `localhost` 用不到异步 DNS。将来以 `ares` feature + `compat.c-ares@1.34.5` 补上 + (grpc 已冻好 `third_party/cares/config_{linux,darwin,windows}/ares_config.h`,可作为配置快照)。 +- **descriptor 表二选一。** gRPC 自带 `src/core/ext/upb-gen/google/protobuf/descriptor.upb_minitable.c`, + 与本 feature 带入的 bootstrap 版本**定义同一批符号**。grpc-m 必须排除其中一份,否则重复符号。 diff --git a/README.md b/README.md index 4bf63e74..a1fc1a50 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ Two kinds of packages live here: |------|------| | Native module library (Form A) | [`mcpplibs.xpkg`](pkgs/x/xpkg.lua) · [`mcpplibs.tinyhttps`](pkgs/t/tinyhttps.lua) · [`tensorvia-cpu`](pkgs/t/tensorvia-cpu.lua) · [`ffmpeg`](pkgs/f/ffmpeg.lua) (module layer; sources compiled directly through `compat.ffmpeg`) · [`opencv`](pkgs/o/opencv.lua) (single repository: the module layer and the full OpenCV 5 source build both live in the package, and only this descriptor stays on the index side) | | C-source compat (with `features`) | [`compat.cjson`](pkgs/c/compat.cjson.lua) · [`compat.zlib`](pkgs/c/compat.zlib.lua) | -| C++-source compat, one depending on the other | [`compat.abseil`](pkgs/c/compat.abseil.lua) (151 TUs; a wildcard over `absl/**` trimmed by upstream's test/benchmark naming conventions) · [`compat.protobuf`](pkgs/c/compat.protobuf.lua) (the libprotobuf runtime, 79 TUs transcribed from upstream's own `src/file_lists.cmake`; declares `compat.abseil` as a dependency because protobuf's public headers include `absl/…`, and its `gzip` feature defines `HAVE_ZLIB` and pulls `compat.zlib`) | +| C++-source compat, one depending on the other | [`compat.abseil`](pkgs/c/compat.abseil.lua) (151 TUs; a wildcard over `absl/**` trimmed by upstream's test/benchmark naming conventions) · [`compat.protobuf`](pkgs/c/compat.protobuf.lua) (the libprotobuf runtime, 79 TUs transcribed from upstream's own `src/file_lists.cmake`; declares `compat.abseil` as a dependency because protobuf's public headers include `absl/…`, and its `gzip` feature defines `HAVE_ZLIB` and pulls `compat.zlib`, while `upb` adds protobuf's 64-TU C runtime out of the same tarball) · [`compat.re2`](pkgs/c/compat.re2.lua) (22 TUs, upstream's own `RE2_SOURCES`) | | header-only (with `features`) | [`compat.eigen`](pkgs/c/compat.eigen.lua) | | Runtime loader compat (pure sources, sidestepping upstream codegen/asm) | [`compat.vulkan`](pkgs/c/compat.vulkan.lua) (the Khronos loader: `loader/generated/` is checked in, and the assembly path degrades to plain C through `UNKNOWN_FUNCTIONS_SUPPORTED`, so no CMake/Python/assembler is needed; windows deferred) · [`compat.vulkan-headers`](pkgs/c/compat.vulkan-headers.lua) | | Whole-source direct build + generated config (only where a platform lacks one) | [`compat.curl`](pkgs/c/compat.curl.lua) (win32 uses upstream's checked-in config, unix generates one) · [`compat.sdl2`](pkgs/c/compat.sdl2.lua) (win/mac use upstream's checked-in config; linux generates one and enables X11 by hand) | diff --git a/README.zh-CN.md b/README.zh-CN.md index fcfb694a..be3805af 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -36,7 +36,7 @@ mcpp self config --mirror CN # 切换至国内镜像,默认使用 GLOBAL 上 |------|------| | 原生模块库(Form A) | [`mcpplibs.xpkg`](pkgs/x/xpkg.lua) · [`mcpplibs.tinyhttps`](pkgs/t/tinyhttps.lua) · [`tensorvia-cpu`](pkgs/t/tensorvia-cpu.lua) · [`ffmpeg`](pkgs/f/ffmpeg.lua)(模块层,源码经 `compat.ffmpeg` 直编) · [`opencv`](pkgs/o/opencv.lua)(单仓库:模块层与 OpenCV 5 全源码构建同在包内,索引侧只留本描述符) | | C 源码 compat(含 `features`) | [`compat.cjson`](pkgs/c/compat.cjson.lua) · [`compat.zlib`](pkgs/c/compat.zlib.lua) | -| C++ 源码 compat(彼此依赖) | [`compat.abseil`](pkgs/c/compat.abseil.lua)(151 TU;对 `absl/**` 取通配后,按上游自身的 test/benchmark 命名约定裁剪) · [`compat.protobuf`](pkgs/c/compat.protobuf.lua)(libprotobuf 运行时,79 TU 逐条转录自上游 `src/file_lists.cmake`;因 protobuf 公开头文件 include 了 `absl/…`,故显式依赖 `compat.abseil`;`gzip` feature 定义 `HAVE_ZLIB` 并拉入 `compat.zlib`) | +| C++ 源码 compat(彼此依赖) | [`compat.abseil`](pkgs/c/compat.abseil.lua)(151 TU;对 `absl/**` 取通配后,按上游自身的 test/benchmark 命名约定裁剪) · [`compat.protobuf`](pkgs/c/compat.protobuf.lua)(libprotobuf 运行时,79 TU 逐条转录自上游 `src/file_lists.cmake`;因 protobuf 公开头文件 include 了 `absl/…`,故显式依赖 `compat.abseil`;`gzip` feature 定义 `HAVE_ZLIB` 并拉入 `compat.zlib`,`upb` feature 则从同一个 tarball 里再编出 protobuf 的 64 TU C 运行时) · [`compat.re2`](pkgs/c/compat.re2.lua)(22 TU,取自上游自身的 `RE2_SOURCES`) | | header-only(含 `features`) | [`compat.eigen`](pkgs/c/compat.eigen.lua) | | 运行时 loader compat(纯源码,绕开上游 codegen/asm) | [`compat.vulkan`](pkgs/c/compat.vulkan.lua)(Khronos loader:`loader/generated/` 已签入,汇编路径经 `UNKNOWN_FUNCTIONS_SUPPORTED` 降级为纯 C,故无需 CMake/Python/汇编器;windows 延后)· [`compat.vulkan-headers`](pkgs/c/compat.vulkan-headers.lua) | | 全源码直编 + 生成 config(仅缺口平台) | [`compat.curl`](pkgs/c/compat.curl.lua)(win32 用上游签入 config,unix 生成) · [`compat.sdl2`](pkgs/c/compat.sdl2.lua)(win/mac 用上游签入 config,linux 生成 + 手工开 X11) | diff --git a/mcpp.toml b/mcpp.toml index 064ac629..e7f95882 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -43,10 +43,12 @@ members = [ "tests/examples/openblas", "tests/examples/openssl", "tests/examples/protobuf", + "tests/examples/protobuf-upb", "tests/examples/protobuf-gzip", "tests/examples/opencv-module", "tests/examples/opencv-module-dnn", "tests/examples/opencv-module-unifont", + "tests/examples/re2", "tests/examples/sdl2", "tests/examples/spdlog", "tests/examples/freetype", diff --git a/pkgs/c/compat.protobuf.lua b/pkgs/c/compat.protobuf.lua index 294ebe8c..fb5ab549 100644 --- a/pkgs/c/compat.protobuf.lua +++ b/pkgs/c/compat.protobuf.lua @@ -76,7 +76,20 @@ package = { -- `*/src` carries the public headers; -- `*/third_party/utf8_range` carries utf8_range.h + utf8_validity.h, -- which protobuf's own TUs include unqualified. - include_dirs = { "*/src", "*/third_party/utf8_range" }, + -- + -- The last two exist for the `upb` feature but are declared + -- unconditionally: `features` can gate sources/defines/deps, not + -- include dirs, and the feature's own TUs need them to compile. Both + -- are additive rather than shadowing — the tarball root supplies only + -- `upb/…` (protobuf's C++ headers live under src/, not at the root), + -- and the bootstrap dir supplies `google/protobuf/descriptor.upb*.h`, + -- a different file name from the C++ `descriptor.h` next door. + include_dirs = { + "*/src", + "*/third_party/utf8_range", + "*", -- upb/… headers + "*/upb/reflection/cmake", -- google/protobuf/descriptor.upb*.h + }, -- Transcribed from upstream's own authoritative list — the -- `libprotobuf_srcs` set in src/file_lists.cmake, which upstream @@ -199,6 +212,98 @@ package = { defines = { "HAVE_ZLIB=1" }, deps = { ["compat.zlib"] = "1.3.2" }, }, + + -- upb — protobuf's small C runtime, vendored in the SAME tarball + -- (upb/ at the root) since protobuf v22 absorbed it. Off by + -- default: nothing in the C++ runtime uses it, and it is 64 extra + -- C TUs. gRPC is the consumer that needs it — its generated + -- src/core/ext/upb-gen/**.c is upb code and links against this + -- runtime. + -- + -- The list is transcribed from upstream's `libupb_srcs` + -- (src/file_lists.cmake) rather than globbed, and that is not + -- fussiness: `upb/**/*.c` would pull in 14 more files, among them + -- TWO ALTERNATIVE BUILDS of the descriptor tables + -- (upb/reflection/stage0/… and upb/reflection/cmake/…) plus + -- upb/message/promote.c and the decode_fast/ variants. Compiling + -- more than one descriptor variant is a duplicate-symbol link + -- failure. + -- + -- descriptor.upb_minitable.c IS included even though it is not in + -- libupb_srcs: upstream's cmake/libupb.cmake adds exactly this + -- file as `bootstrap_sources` on top of the list, because + -- upb/reflection/descriptor_bootstrap.h — reached by the whole + -- reflection layer — needs the descriptor tables. Without it the + -- feature builds and only fails at link time. + ["upb"] = { + sources = { + "*/upb/base/status.c", + "*/upb/hash/common.c", + "*/upb/json/decode.c", + "*/upb/json/encode.c", + "*/upb/lex/atoi.c", + "*/upb/lex/round_trip.c", + "*/upb/lex/strtod.c", + "*/upb/lex/unicode.c", + "*/upb/mem/alloc.c", + "*/upb/mem/arena.c", + "*/upb/message/accessors.c", + "*/upb/message/array.c", + "*/upb/message/compare.c", + "*/upb/message/compat.c", + "*/upb/message/copy.c", + "*/upb/message/internal/compare_unknown.c", + "*/upb/message/internal/extension.c", + "*/upb/message/internal/iterator.c", + "*/upb/message/internal/message.c", + "*/upb/message/map.c", + "*/upb/message/map_sorter.c", + "*/upb/message/merge.c", + "*/upb/message/message.c", + "*/upb/mini_descriptor/build_enum.c", + "*/upb/mini_descriptor/decode.c", + "*/upb/mini_descriptor/internal/base92.c", + "*/upb/mini_descriptor/internal/encode.c", + "*/upb/mini_descriptor/link.c", + "*/upb/mini_table/compat.c", + "*/upb/mini_table/debug_string.c", + "*/upb/mini_table/extension_registry.c", + "*/upb/mini_table/generated_registry.c", + "*/upb/mini_table/internal/message.c", + "*/upb/mini_table/message.c", + "*/upb/reflection/def_pool.c", + "*/upb/reflection/def_type.c", + "*/upb/reflection/desc_state.c", + "*/upb/reflection/enum_def.c", + "*/upb/reflection/enum_reserved_range.c", + "*/upb/reflection/enum_value_def.c", + "*/upb/reflection/extension_range.c", + "*/upb/reflection/field_def.c", + "*/upb/reflection/file_def.c", + "*/upb/reflection/internal/def_builder.c", + "*/upb/reflection/internal/strdup2.c", + "*/upb/reflection/message.c", + "*/upb/reflection/message_def.c", + "*/upb/reflection/message_reserved_range.c", + "*/upb/reflection/method_def.c", + "*/upb/reflection/oneof_def.c", + "*/upb/reflection/service_def.c", + "*/upb/text/debug_string.c", + "*/upb/text/encode.c", + "*/upb/text/internal/encode.c", + "*/upb/util/def_to_proto.c", + "*/upb/util/required_fields.c", + "*/upb/wire/byte_size.c", + "*/upb/wire/decode.c", + "*/upb/wire/decode_fast/select.c", + "*/upb/wire/encode.c", + "*/upb/wire/eps_copy_input_stream.c", + "*/upb/wire/internal/decoder.c", + "*/upb/wire/reader.c", + -- upstream's bootstrap_sources (cmake/libupb.cmake) + "*/upb/reflection/cmake/google/protobuf/descriptor.upb_minitable.c", + }, + }, }, linux = { diff --git a/pkgs/c/compat.re2.lua b/pkgs/c/compat.re2.lua new file mode 100644 index 00000000..e7002bf5 --- /dev/null +++ b/pkgs/c/compat.re2.lua @@ -0,0 +1,102 @@ +-- compat.re2 — RE2, Google's linear-time regular expression engine. +-- +-- Shape A (C++ source compat): no configure step, no code generation, no +-- submodules, and platform handling is entirely in-source, so one source list +-- and one sha256 cover linux/macosx/windows. +-- +-- WHY THIS VERSION. `2022-04-01` is an upstream RE2 release tag, and it is the +-- one gRPC 1.83.0 pins (its third_party/re2 submodule is commit 0c5616d == +-- tag 2022-04-01). That pin is the point: gRPC's xds matchers +-- (src/core/util/matchers.h, src/core/xds/grpc/xds_route_config*) are written +-- against this API. RE2 switched its own string type from re2::StringPiece to +-- absl::string_view in the 2023 releases and took on an Abseil dependency +-- there, so a newer RE2 is not a drop-in for this consumer. Newer releases can +-- be added as ADDITIONAL versions of this same descriptor when something wants +-- them — the index carries multi-version packages already (compat.catch2). +-- +-- The GitHub tag archive is used rather than a release asset because upstream +-- publishes no assets for this tag; the archive is self-contained (RE2 has no +-- submodules) and its digest was verified stable across two downloads. +package = { + spec = "1", + namespace = "compat", + name = "re2", + description = "RE2 — fast, safe, thread-friendly regular expression engine (static)", + licenses = {"BSD-3-Clause"}, + repo = "https://github.com/google/re2", + type = "package", + + xpm = { + linux = { + ["2022-04-01"] = { + url = { + GLOBAL = "https://github.com/google/re2/archive/refs/tags/2022-04-01.tar.gz", + CN = "https://gitcode.com/mcpp-res/re2/releases/download/2022-04-01/re2-2022-04-01.tar.gz", + }, + sha256 = "1ae8ccfdb1066a731bba6ee0881baad5efd2cd661acd9569b689f2586e1a50e9", + }, + }, + macosx = { + ["2022-04-01"] = { + url = { + GLOBAL = "https://github.com/google/re2/archive/refs/tags/2022-04-01.tar.gz", + CN = "https://gitcode.com/mcpp-res/re2/releases/download/2022-04-01/re2-2022-04-01.tar.gz", + }, + sha256 = "1ae8ccfdb1066a731bba6ee0881baad5efd2cd661acd9569b689f2586e1a50e9", + }, + }, + windows = { + ["2022-04-01"] = { + url = { + GLOBAL = "https://github.com/google/re2/archive/refs/tags/2022-04-01.tar.gz", + CN = "https://gitcode.com/mcpp-res/re2/releases/download/2022-04-01/re2-2022-04-01.tar.gz", + }, + sha256 = "1ae8ccfdb1066a731bba6ee0881baad5efd2cd661acd9569b689f2586e1a50e9", + }, + }, + }, + + mcpp = { + language = "c++23", + import_std = false, + -- RE2 is included as , so the include root is the tarball + -- wrap dir. `*` absorbs `re2-2022-04-01/`. + include_dirs = { "*" }, + + -- Upstream's RE2_SOURCES (CMakeLists.txt) exactly: every .cc directly + -- under re2/, plus two of the four under util/. The re2/ glob is safe + -- because upstream keeps its tests one level down in re2/testing/, + -- which `re2/*.cc` does not reach. + -- + -- util/ is listed file-by-file on purpose — the other two TUs there + -- both define main() (util/benchmark.cc, util/test.cc, and likewise + -- util/fuzz.cc and the top-level testinstall.cc). A dependency's + -- objects all enter the consumer's link, so any of them would collide + -- with the consumer's own main(). util/pcre.cc is excluded for a + -- second reason: it is upstream's PCRE comparison harness and would + -- drag in libpcre. + sources = { + "*/re2/*.cc", + "*/util/rune.cc", + "*/util/strutil.cc", + }, + + targets = { ["re2"] = { kind = "lib" } }, + deps = { }, + + linux = { ldflags = { "-lpthread" } }, + -- macOS: libSystem carries pthread. + windows = { + -- RE2's util/mutex.h includes , whose min/max + -- function-like MACROS then eat RE2's own accessors: regexp.cc + -- calls `a->min()` / `a->max()` on Regexp, and the preprocessor + -- turns those into "too few arguments provided to function-like + -- macro invocation". Upstream's CMake never has to say this + -- because a CMake consumer normally sets NOMINMAX project-wide; + -- here the package owns its own flags. No extra import libs — + -- RE2 uses only the CRT and the Win32 CRITICAL_SECTION already in + -- kernel32. + cxxflags = { "-DNOMINMAX", "-DWIN32_LEAN_AND_MEAN" }, + }, + }, +} diff --git a/tests/examples/protobuf-upb/mcpp.toml b/tests/examples/protobuf-upb/mcpp.toml new file mode 100644 index 00000000..c19f1adf --- /dev/null +++ b/tests/examples/protobuf-upb/mcpp.toml @@ -0,0 +1,13 @@ +# protobuf `upb` feature member. The sibling tests/examples/protobuf covers the +# DEFAULT build, where none of upb's 64 C TUs are compiled at all. +# +# upb is protobuf's small C runtime, vendored in the same tarball since v22. +# gRPC is what needs it — its generated src/core/ext/upb-gen/**.c links against +# this runtime — so this member is the standalone proof that the feature builds +# and works before gRPC depends on it. +[package] +name = "protobuf-upb-tests" +version = "0.1.0" + +[dependencies.compat] +protobuf = { version = "35.1", features = ["upb"] } diff --git a/tests/examples/protobuf-upb/tests/upb_runtime.cpp b/tests/examples/protobuf-upb/tests/upb_runtime.cpp new file mode 100644 index 00000000..424ccee5 --- /dev/null +++ b/tests/examples/protobuf-upb/tests/upb_runtime.cpp @@ -0,0 +1,116 @@ +// Behavioral test for compat.protobuf's `upb` feature. +// +// Drives BOTH runtimes across the same bytes, which is the strongest thing +// this member can assert: the C++ runtime builds a FileDescriptorProto and +// serializes it, then upb parses those bytes and loads them into a upb_DefPool +// and answers reflection queries about them. +// +// What that reaches: +// upb_Arena / upb_DefPool -> upb/mem/arena.c, reflection/def_pool.c +// parse of a descriptor message -> upb/wire/decode.c, message/*.c, mini_table/*.c +// AddFile + name lookups -> upb/reflection/{file,message,field}_def.c +// the descriptor tables themselves -> upb/reflection/cmake/…/descriptor.upb_minitable.c +// +// That last one is the reason this test exists in this shape: those tables are +// NOT in upstream's libupb_srcs, they come from cmake/libupb.cmake's +// bootstrap_sources, and leaving them out fails only at LINK time. Anything +// that merely compiles upb would not notice. +// +// Returns non-zero on any mismatch. +#include +#include + +#include "google/protobuf/descriptor.pb.h" + +// NOT wrapped in extern "C": upb's headers manage their own linkage, and +// upb/message/accessors.h deliberately declares C++ OVERLOADS outside the +// extern "C" block. Forcing the whole tree into C linkage makes those overloads +// collide with their C counterparts ("conflicting declaration of C function"). +#include "google/protobuf/descriptor.upb.h" +#include "upb/base/status.h" +#include "upb/mem/arena.h" +#include "upb/reflection/def_pool.h" +#include "upb/reflection/field_def.h" +#include "upb/reflection/file_def.h" +#include "upb/reflection/message_def.h" + +namespace gpb = google::protobuf; + +namespace { + +// Build the schema with the C++ runtime and hand upb its wire bytes. +std::string make_descriptor_bytes() { + gpb::FileDescriptorProto file; + file.set_name("mcpp_upb_test.proto"); + file.set_package("mcpp.test"); + file.set_syntax("proto3"); + + gpb::DescriptorProto* msg = file.add_message_type(); + msg->set_name("Person"); + + gpb::FieldDescriptorProto* name = msg->add_field(); + name->set_name("name"); + name->set_number(1); + name->set_type(gpb::FieldDescriptorProto::TYPE_STRING); + name->set_label(gpb::FieldDescriptorProto::LABEL_OPTIONAL); + + gpb::FieldDescriptorProto* id = msg->add_field(); + id->set_name("id"); + id->set_number(2); + id->set_type(gpb::FieldDescriptorProto::TYPE_INT32); + id->set_label(gpb::FieldDescriptorProto::LABEL_OPTIONAL); + + return file.SerializeAsString(); +} + +} // namespace + +int main() { + const std::string wire = make_descriptor_bytes(); + if (wire.empty()) return 1; + + upb_Arena* arena = upb_Arena_New(); + if (arena == nullptr) return 1; + + // upb parses what the C++ runtime wrote. + google_protobuf_FileDescriptorProto* parsed = + google_protobuf_FileDescriptorProto_parse(wire.data(), wire.size(), arena); + if (parsed == nullptr) { + upb_Arena_Free(arena); + return 1; + } + + upb_DefPool* pool = upb_DefPool_New(); + if (pool == nullptr) { + upb_Arena_Free(arena); + return 1; + } + + upb_Status status; + upb_Status_Clear(&status); + const upb_FileDef* file_def = upb_DefPool_AddFile(pool, parsed, &status); + + int rc = 1; + if (file_def != nullptr) { + const upb_MessageDef* msg = upb_DefPool_FindMessageByName(pool, "mcpp.test.Person"); + if (msg != nullptr && upb_MessageDef_FieldCount(msg) == 2) { + const upb_FieldDef* f_name = upb_MessageDef_FindFieldByName(msg, "name"); + const upb_FieldDef* f_id = upb_MessageDef_FindFieldByName(msg, "id"); + const bool names_ok = + std::string(upb_FileDef_Name(file_def)) == "mcpp_upb_test.proto" && + std::string(upb_MessageDef_FullName(msg)) == "mcpp.test.Person"; + // A name that was never declared must NOT resolve — otherwise a + // stub lookup returning something would pass everything above. + const bool absent_ok = + upb_DefPool_FindMessageByName(pool, "mcpp.test.Missing") == nullptr; + if (f_name != nullptr && f_id != nullptr && names_ok && absent_ok && + upb_FieldDef_Number(f_name) == 1 && upb_FieldDef_Number(f_id) == 2) { + rc = 0; + } + } + } + + upb_DefPool_Free(pool); + upb_Arena_Free(arena); + return rc; +} diff --git a/tests/examples/re2/mcpp.toml b/tests/examples/re2/mcpp.toml new file mode 100644 index 00000000..48339233 --- /dev/null +++ b/tests/examples/re2/mcpp.toml @@ -0,0 +1,10 @@ +# re2 test project: consumes compat.re2 and asserts behavior under `mcpp test`. +# Part of the mcpp-index self-referential workspace — the workspace-root +# `[indices]` redirect points at this repo, so the dependency resolves to the +# checked-in recipe (pkgs/c/compat.re2.lua) rather than the published index. +[package] +name = "re2-tests" +version = "0.1.0" + +[dependencies.compat] +re2 = "2022-04-01" diff --git a/tests/examples/re2/tests/match.cpp b/tests/examples/re2/tests/match.cpp new file mode 100644 index 00000000..33643331 --- /dev/null +++ b/tests/examples/re2/tests/match.cpp @@ -0,0 +1,83 @@ +// Behavioral test for compat.re2. +// +// Exercises the parts of RE2 that gRPC's xds matchers actually use — full and +// partial match with captures, a pre-compiled RE2 object, replacement, and the +// RE2::Set multi-pattern API — so a package that compiled but linked nothing +// useful cannot pass: +// +// RE2 ctor / FullMatch / PartialMatch -> re2/re2.cc, parse.cc, compile.cc, prog.cc +// captures + rewriting -> re2/re2.cc, regexp.cc, simplify.cc +// RE2::Set -> re2/set.cc, dfa.cc, nfa.cc +// invalid pattern reporting -> re2/parse.cc, tostring.cc +// UTF-8 handling -> util/rune.cc, unicode_groups.cc +// +// Returns non-zero on any mismatch. +#include +#include + +#include "re2/re2.h" +#include "re2/set.h" + +namespace { + +bool matching_ok() { + if (!RE2::FullMatch("grpc-1.83.0", R"(grpc-\d+\.\d+\.\d+)")) return false; + // A full match must NOT accept trailing junk — proves this is FullMatch + // semantics and not an accidental substring search. + if (RE2::FullMatch("grpc-1.83.0-rc1", R"(grpc-\d+\.\d+\.\d+)")) return false; + return RE2::PartialMatch("grpc-1.83.0-rc1", R"(grpc-\d+\.\d+\.\d+)"); +} + +bool captures_ok() { + const RE2 re(R"((\w+)://([^/:]+):(\d+))"); // pre-compiled, the way a matcher holds it + if (!re.ok()) return false; + + std::string scheme, host; + int port = 0; + if (!RE2::FullMatch("dns://localhost:50051", re, &scheme, &host, &port)) return false; + return scheme == "dns" && host == "localhost" && port == 50051; +} + +bool rewrite_ok() { + std::string s = "service/Echo, service/Ping"; + const int n = RE2::GlobalReplace(&s, R"(service/(\w+))", R"(grpc.\1)"); + return n == 2 && s == "grpc.Echo, grpc.Ping"; +} + +bool set_ok() { + RE2::Set set(RE2::DefaultOptions, RE2::UNANCHORED); + const int a = set.Add(R"(^/helloworld\.Greeter/)", nullptr); + const int b = set.Add(R"(SayHello$)", nullptr); + if (a < 0 || b < 0 || !set.Compile()) return false; + + std::vector hits; + if (!set.Match("/helloworld.Greeter/SayHello", &hits)) return false; + if (hits.size() != 2) return false; + + std::vector none; + return !set.Match("/other.Service/Method", &none); +} + +bool invalid_pattern_reported() { + // An unbalanced group must be REJECTED with an error string, not silently + // accepted — otherwise a broken matcher config would look valid. + // log_errors(false) keeps RE2 from also printing the (expected) parse + // failure to stderr, so real problems stay the only noise in the log. + RE2::Options quiet; + quiet.set_log_errors(false); + const RE2 bad("(unclosed", quiet); + return !bad.ok() && !bad.error().empty(); +} + +bool utf8_ok() { + // '.' counts runes, not bytes, in UTF-8 mode (the default). + return RE2::FullMatch("héllo", "h.llo") && RE2::PartialMatch("世界", "界"); +} + +} // namespace + +int main() { + const bool ok = matching_ok() && captures_ok() && rewrite_ok() && set_ok() + && invalid_pattern_reported() && utf8_ok(); + return ok ? 0 : 1; +}