Skip to content

feat: add compat.re2 2022-04-01 + a upb feature on compat.protobuf #148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Sunrisepeak merged 2 commits into from
Aug 4, 2026
Merged

feat: add compat.re2 2022-04-01 + a upb feature on compat.protobuf #148

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9fff857
feat: add compat.re2 2022-04-01 + a `upb` feature on compat.protobuf
Sunrisepeak Aug 4, 2026
d3ecfd0
fix(re2): define NOMINMAX on windows — <windows.h> ate Regexp::min()/…
Sunrisepeak Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
123 changes: 123 additions & 0 deletions .agents/docs/2026-08-04-add-re2-and-protobuf-upb-plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
# 新增 compat.re2 2022-04-01 与 compat.protobuf 的 upb feature(2026-08-04)

[gRPC 收录可行性分析](2026-08-04-grpc-feasibility-analysis.md) 的 **P1 索引侧**,承接
[P0](2026-08-04-add-abseil-and-protobuf-plan.md)。目标是把 gRPC 剩下的索引侧依赖补齐,使后续
`grpc-m` 独立仓只需 vendor gRPC 自己的源码。

产出:

| 变更 | 规模 | 说明 |
|---|---|---|
| `compat.re2@2022-04-01` | 22 TU | 新包,Shape A |
| `compat.protobuf` 新增 `upb` feature | 64 TU | 源码全在已收录的 protobuf tarball 内,无新下载 |

workspace 成员:`tests/examples/re2`、`tests/examples/protobuf-upb`。

## 0. 先做的事:确认 mcpp 真能编 gRPC

在写任何描述符之前先打掉最大的未知数 —— 用 mcpp 的 gcc@16.1.0 直接编 gRPC 1.83.0 的代表性 TU,
只给 include 路径,不打补丁、不跑 configure、不做 codegen:

```
OK src/core/lib/slice/slice.cc
OK src/core/lib/surface/call.cc <- core 里最重的之一
OK src/core/lib/promise/activity.cc
OK src/core/ext/upb-gen/google/protobuf/any.upb_minitable.c
OK src/cpp/common/channel_arguments.cc
OK src/cpp/client/channel_cc.cc <- C++ API 层
```

以及 SSL 层对着 **compat.openssl 3.5.1 的头文件**(不是 boringssl)4/4 通过:

```
OK src/core/tsi/ssl_transport_security.cc
OK src/core/credentials/transport/ssl/ssl_credentials.cc
OK src/core/tsi/ssl/session_cache/ssl_session_openssl.cc
OK src/core/tsi/ssl_telemetry_utils.cc
```

需要的 include 路径只有:仓库根、`include`、`src/core/ext/upb-gen`、`src/core/ext/upbdefs-gen`,
外加 abseil / protobuf(含 upb)/ openssl。另核实 gRPC 源码树内**没有任何 `.h.in` 或
`config.h.cmake`**,确认它不需要 configure 步骤。

结论:整源码直编路线成立,可以继续投入。

## 1. compat.re2 —— 为什么钉在 2022-04-01

`2022-04-01` 是 RE2 的上游 release tag,也是 **gRPC 1.83.0 的 pin**(其 `third_party/re2` submodule
= commit `0c5616d` = 该 tag)。这个 pin 是有意义的:gRPC 的 xds matcher
(`src/core/util/matchers.h`、`src/core/xds/grpc/xds_route_config*`)是照着这一版 API 写的。
RE2 在 2023 年的版本里把自己的字符串类型从 `re2::StringPiece` 换成了 `absl::string_view`
并引入了 Abseil 依赖,**新版不是 drop-in**。将来若有别的消费者需要新版,可按 `compat.catch2`
的先例在同一描述符里追加版本。

上游该 tag 没有 release asset,用 tag 归档(RE2 无 submodule,自包含),sha256 两次下载一致。

源码清单直接对应上游 `CMakeLists.txt` 的 `RE2_SOURCES`:`re2/*.cc` 全部(上游把测试放在
低一级的 `re2/testing/`,glob 够不到)+ `util/rune.cc` + `util/strutil.cc`。`util/` 逐条列而不通配,
因为该目录另外两个 TU **都带 `main()`**(`util/benchmark.cc`、`util/test.cc`,连同
`util/fuzz.cc` 与顶层 `testinstall.cc`),依赖的 `.o` 会全量入链,任何一个都会和消费者的
`main()` 撞车;`util/pcre.cc` 还会额外拖进 libpcre。

## 2. upb feature —— 组成必须严格照抄上游 libupb

upb 是 protobuf 自 v22 起吸收进来的小型 C 运行时,**源码就在已收录的 protobuf 35.1 tarball 里**
(根目录 `upb/`),所以不需要新包、不需要新下载。默认关闭:C++ 运行时一点都不用它,而它是 64 个额外
C TU。gRPC 是需要它的那个消费者 —— `src/core/ext/upb-gen/**.c` 就是 upb 代码。

两处**不能想当然**的地方:

1. **不能用 `upb/**/*.c` 通配。** 实测通配会多带 14 个文件,其中包含 **descriptor 表的两份替代构建**
(`upb/reflection/stage0/…` 与 `upb/reflection/cmake/…`)以及 `upb/message/promote.c`、
`decode_fast/` 的若干变体。同时编进多份 descriptor 变体 = 重复符号链接失败。因此逐条转录
`src/file_lists.cmake` 的 `libupb_srcs`(63 条)。
2. **必须额外加上 `upb/reflection/cmake/google/protobuf/descriptor.upb_minitable.c`**,它**不在**
`libupb_srcs` 里。上游 `cmake/libupb.cmake` 正是把这一个文件作为 `bootstrap_sources` 叠加在清单之上,
因为整个 reflection 层经 `upb/reflection/descriptor_bootstrap.h` 都要用 descriptor 表。
漏掉它能编译通过,**只在链接期炸**。

`include_dirs` 因此追加了两条(tarball 根、`upb/reflection/cmake`)。它们是**无条件**声明的:
`features` 能门控 sources/defines/deps,**不能门控 include dirs**,而 feature 自己的 TU 需要它们才能编译。
两条都是纯追加、不构成遮蔽 —— 根目录只提供 `upb/…`(protobuf 的 C++ 头在 `src/` 下而非根),
bootstrap 目录提供的是 `google/protobuf/descriptor.upb*.h`,与隔壁 C++ 的 `descriptor.h` 文件名不同。

## 3. 验证结论

全部使用与 CI 一致的配置(mcpp **2026.8.3.3**、gcc@16.1.0、`MCPP_INDEX_MIRROR=GLOBAL`、
`MCPP_BUILD_CACHE=local`),每个成员先删 `target/` 冷构建:

```
abseil test result ok. 1 passed; 0 failed (17.84s)
protobuf test result ok. 1 passed; 0 failed (40.88s)
protobuf-gzip test result ok. 1 passed; 0 failed (51.99s)
protobuf-upb test result ok. 1 passed; 0 failed (70.20s)
re2 test result ok. 1 passed; 0 failed ( 2.35s)
```

前三个是**回归**:本次改了 `compat.protobuf` 的 `include_dirs`,必须确认既有成员未受影响。

- **目标文件数已核对**:`re2` 23 个(22 源 + 1 测试);`protobuf-upb` 296 个
= 151(abseil)+ 80(protobuf)+ 64(upb)+ 1(测试)。
- **`upb` feature 做了负向验证**:把成员依赖改回裸 `protobuf = "35.1"` 后,目标文件数从 296 落到
**232(正好少 64 个)**,链接给出 `undefined reference to upb_Arena_Init` / `upb_Decode` /
`upb_DefPool_Free`,以及 **`google__protobuf__FileDescriptorProto_msg_init`** —— 最后这一个正好
证明 §2 说的 bootstrap descriptor 表既是必需的、也确实由该 feature 带入。
(注意关键词:这里工具链是 gcc,用 GNU ld,报的是 `undefined reference`;lld 才报 `undefined symbol`。)
- **测试打的是真符号**。re2 侧覆盖 FullMatch/PartialMatch(并断言 FullMatch **不**接受尾部残余)、
预编译 RE2 对象取捕获、GlobalReplace、`RE2::Set` 多模式匹配、非法模式必须被拒、UTF-8 按 rune 计数;
upb 侧让**两个运行时对同一段字节交叉验证** —— C++ 运行时构造并序列化 `FileDescriptorProto`,
再由 upb 解析、灌进 `upb_DefPool`、回答反射查询,并断言一个从未声明过的名字**查不到**。
- **CN 镜像已闭环**:`mcpp-res/re2@2022-04-01` 返回 `http=200` 且与 GLOBAL **字节一致**。

## 4. 下一步(grpc-m)

索引侧到此备齐:`compat.abseil` / `compat.protobuf`(+`upb`)/ `compat.re2` / `compat.openssl`。
`grpc-m` 只需 vendor gRPC 自己的源码。已知的两个待办:

- **c-ares 暂不引入。** 上游有官方裁剪开关 `grpc_no_ares=true` → `GRPC_ARES=0`
(`BUILD:75`、`bazel/grpc_build_system.bzl:175`),且只有 10 个文件 include `ares.h`,全在
`resolver/dns/c_ares/` 与 `event_engine/*ares*`,可 glob 排除。第一版走 gRPC 原生 DNS 解析器,
helloworld 连 `localhost` 用不到异步 DNS。将来以 `ares` feature + `compat.c-ares@1.34.5` 补上
(grpc 已冻好 `third_party/cares/config_{linux,darwin,windows}/ares_config.h`,可作为配置快照)。
- **descriptor 表二选一。** gRPC 自带 `src/core/ext/upb-gen/google/protobuf/descriptor.upb_minitable.c`,
与本 feature 带入的 bootstrap 版本**定义同一批符号**。grpc-m 必须排除其中一份,否则重复符号。
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Two kinds of packages live here:
|------|------|
| Native module library (Form A) | [`mcpplibs.xpkg`](pkgs/x/xpkg.lua) · [`mcpplibs.tinyhttps`](pkgs/t/tinyhttps.lua) · [`tensorvia-cpu`](pkgs/t/tensorvia-cpu.lua) · [`ffmpeg`](pkgs/f/ffmpeg.lua) (module layer; sources compiled directly through `compat.ffmpeg`) · [`opencv`](pkgs/o/opencv.lua) (single repository: the module layer and the full OpenCV 5 source build both live in the package, and only this descriptor stays on the index side) |
| C-source compat (with `features`) | [`compat.cjson`](pkgs/c/compat.cjson.lua) · [`compat.zlib`](pkgs/c/compat.zlib.lua) |
| C++-source compat, one depending on the other | [`compat.abseil`](pkgs/c/compat.abseil.lua) (151 TUs; a wildcard over `absl/**` trimmed by upstream's test/benchmark naming conventions) · [`compat.protobuf`](pkgs/c/compat.protobuf.lua) (the libprotobuf runtime, 79 TUs transcribed from upstream's own `src/file_lists.cmake`; declares `compat.abseil` as a dependency because protobuf's public headers include `absl/…`, and its `gzip` feature defines `HAVE_ZLIB` and pulls `compat.zlib`) |
| C++-source compat, one depending on the other | [`compat.abseil`](pkgs/c/compat.abseil.lua) (151 TUs; a wildcard over `absl/**` trimmed by upstream's test/benchmark naming conventions) · [`compat.protobuf`](pkgs/c/compat.protobuf.lua) (the libprotobuf runtime, 79 TUs transcribed from upstream's own `src/file_lists.cmake`; declares `compat.abseil` as a dependency because protobuf's public headers include `absl/…`, and its `gzip` feature defines `HAVE_ZLIB` and pulls `compat.zlib`, while `upb` adds protobuf's 64-TU C runtime out of the same tarball) · [`compat.re2`](pkgs/c/compat.re2.lua) (22 TUs, upstream's own `RE2_SOURCES`) |
| header-only (with `features`) | [`compat.eigen`](pkgs/c/compat.eigen.lua) |
| Runtime loader compat (pure sources, sidestepping upstream codegen/asm) | [`compat.vulkan`](pkgs/c/compat.vulkan.lua) (the Khronos loader: `loader/generated/` is checked in, and the assembly path degrades to plain C through `UNKNOWN_FUNCTIONS_SUPPORTED`, so no CMake/Python/assembler is needed; windows deferred) · [`compat.vulkan-headers`](pkgs/c/compat.vulkan-headers.lua) |
| Whole-source direct build + generated config (only where a platform lacks one) | [`compat.curl`](pkgs/c/compat.curl.lua) (win32 uses upstream's checked-in config, unix generates one) · [`compat.sdl2`](pkgs/c/compat.sdl2.lua) (win/mac use upstream's checked-in config; linux generates one and enables X11 by hand) |
Expand Down
2 changes: 1 addition & 1 deletion README.zh-CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ mcpp self config --mirror CN # 切换至国内镜像,默认使用 GLOBAL 上
|------|------|
| 原生模块库(Form A) | [`mcpplibs.xpkg`](pkgs/x/xpkg.lua) · [`mcpplibs.tinyhttps`](pkgs/t/tinyhttps.lua) · [`tensorvia-cpu`](pkgs/t/tensorvia-cpu.lua) · [`ffmpeg`](pkgs/f/ffmpeg.lua)(模块层,源码经 `compat.ffmpeg` 直编) · [`opencv`](pkgs/o/opencv.lua)(单仓库:模块层与 OpenCV 5 全源码构建同在包内,索引侧只留本描述符) |
| C 源码 compat(含 `features`) | [`compat.cjson`](pkgs/c/compat.cjson.lua) · [`compat.zlib`](pkgs/c/compat.zlib.lua) |
| C++ 源码 compat(彼此依赖) | [`compat.abseil`](pkgs/c/compat.abseil.lua)(151 TU;对 `absl/**` 取通配后,按上游自身的 test/benchmark 命名约定裁剪) · [`compat.protobuf`](pkgs/c/compat.protobuf.lua)(libprotobuf 运行时,79 TU 逐条转录自上游 `src/file_lists.cmake`;因 protobuf 公开头文件 include 了 `absl/…`,故显式依赖 `compat.abseil`;`gzip` feature 定义 `HAVE_ZLIB` 并拉入 `compat.zlib`) |
| C++ 源码 compat(彼此依赖) | [`compat.abseil`](pkgs/c/compat.abseil.lua)(151 TU;对 `absl/**` 取通配后,按上游自身的 test/benchmark 命名约定裁剪) · [`compat.protobuf`](pkgs/c/compat.protobuf.lua)(libprotobuf 运行时,79 TU 逐条转录自上游 `src/file_lists.cmake`;因 protobuf 公开头文件 include 了 `absl/…`,故显式依赖 `compat.abseil`;`gzip` feature 定义 `HAVE_ZLIB` 并拉入 `compat.zlib`,`upb` feature 则从同一个 tarball 里再编出 protobuf 的 64 TU C 运行时) · [`compat.re2`](pkgs/c/compat.re2.lua)(22 TU,取自上游自身的 `RE2_SOURCES`) |
| header-only(含 `features`) | [`compat.eigen`](pkgs/c/compat.eigen.lua) |
| 运行时 loader compat(纯源码,绕开上游 codegen/asm) | [`compat.vulkan`](pkgs/c/compat.vulkan.lua)(Khronos loader:`loader/generated/` 已签入,汇编路径经 `UNKNOWN_FUNCTIONS_SUPPORTED` 降级为纯 C,故无需 CMake/Python/汇编器;windows 延后)· [`compat.vulkan-headers`](pkgs/c/compat.vulkan-headers.lua) |
| 全源码直编 + 生成 config(仅缺口平台) | [`compat.curl`](pkgs/c/compat.curl.lua)(win32 用上游签入 config,unix 生成) · [`compat.sdl2`](pkgs/c/compat.sdl2.lua)(win/mac 用上游签入 config,linux 生成 + 手工开 X11) |
Expand Down
2 changes: 2 additions & 0 deletions mcpp.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,12 @@ members = [
"tests/examples/openblas",
"tests/examples/openssl",
"tests/examples/protobuf",
"tests/examples/protobuf-upb",
"tests/examples/protobuf-gzip",
"tests/examples/opencv-module",
"tests/examples/opencv-module-dnn",
"tests/examples/opencv-module-unifont",
"tests/examples/re2",
"tests/examples/sdl2",
"tests/examples/spdlog",
"tests/examples/freetype",
Expand Down
107 changes: 106 additions & 1 deletion pkgs/c/compat.protobuf.lua
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,20 @@ package = {
-- `*/src` carries the public <google/protobuf/…> headers;
-- `*/third_party/utf8_range` carries utf8_range.h + utf8_validity.h,
-- which protobuf's own TUs include unqualified.
include_dirs = { "*/src", "*/third_party/utf8_range" },
--
-- The last two exist for the `upb` feature but are declared
-- unconditionally: `features` can gate sources/defines/deps, not
-- include dirs, and the feature's own TUs need them to compile. Both
-- are additive rather than shadowing — the tarball root supplies only
-- `upb/…` (protobuf's C++ headers live under src/, not at the root),
-- and the bootstrap dir supplies `google/protobuf/descriptor.upb*.h`,
-- a different file name from the C++ `descriptor.h` next door.
include_dirs = {
"*/src",
"*/third_party/utf8_range",
"*", -- upb/… headers
"*/upb/reflection/cmake", -- google/protobuf/descriptor.upb*.h
},

-- Transcribed from upstream's own authoritative list — the
-- `libprotobuf_srcs` set in src/file_lists.cmake, which upstream
Expand Down Expand Up @@ -199,6 +212,98 @@ package = {
defines = { "HAVE_ZLIB=1" },
deps = { ["compat.zlib"] = "1.3.2" },
},

-- upb — protobuf's small C runtime, vendored in the SAME tarball
-- (upb/ at the root) since protobuf v22 absorbed it. Off by
-- default: nothing in the C++ runtime uses it, and it is 64 extra
-- C TUs. gRPC is the consumer that needs it — its generated
-- src/core/ext/upb-gen/**.c is upb code and links against this
-- runtime.
--
-- The list is transcribed from upstream's `libupb_srcs`
-- (src/file_lists.cmake) rather than globbed, and that is not
-- fussiness: `upb/**/*.c` would pull in 14 more files, among them
-- TWO ALTERNATIVE BUILDS of the descriptor tables
-- (upb/reflection/stage0/… and upb/reflection/cmake/…) plus
-- upb/message/promote.c and the decode_fast/ variants. Compiling
-- more than one descriptor variant is a duplicate-symbol link
-- failure.
--
-- descriptor.upb_minitable.c IS included even though it is not in
-- libupb_srcs: upstream's cmake/libupb.cmake adds exactly this
-- file as `bootstrap_sources` on top of the list, because
-- upb/reflection/descriptor_bootstrap.h — reached by the whole
-- reflection layer — needs the descriptor tables. Without it the
-- feature builds and only fails at link time.
["upb"] = {
sources = {
"*/upb/base/status.c",
"*/upb/hash/common.c",
"*/upb/json/decode.c",
"*/upb/json/encode.c",
"*/upb/lex/atoi.c",
"*/upb/lex/round_trip.c",
"*/upb/lex/strtod.c",
"*/upb/lex/unicode.c",
"*/upb/mem/alloc.c",
"*/upb/mem/arena.c",
"*/upb/message/accessors.c",
"*/upb/message/array.c",
"*/upb/message/compare.c",
"*/upb/message/compat.c",
"*/upb/message/copy.c",
"*/upb/message/internal/compare_unknown.c",
"*/upb/message/internal/extension.c",
"*/upb/message/internal/iterator.c",
"*/upb/message/internal/message.c",
"*/upb/message/map.c",
"*/upb/message/map_sorter.c",
"*/upb/message/merge.c",
"*/upb/message/message.c",
"*/upb/mini_descriptor/build_enum.c",
"*/upb/mini_descriptor/decode.c",
"*/upb/mini_descriptor/internal/base92.c",
"*/upb/mini_descriptor/internal/encode.c",
"*/upb/mini_descriptor/link.c",
"*/upb/mini_table/compat.c",
"*/upb/mini_table/debug_string.c",
"*/upb/mini_table/extension_registry.c",
"*/upb/mini_table/generated_registry.c",
"*/upb/mini_table/internal/message.c",
"*/upb/mini_table/message.c",
"*/upb/reflection/def_pool.c",
"*/upb/reflection/def_type.c",
"*/upb/reflection/desc_state.c",
"*/upb/reflection/enum_def.c",
"*/upb/reflection/enum_reserved_range.c",
"*/upb/reflection/enum_value_def.c",
"*/upb/reflection/extension_range.c",
"*/upb/reflection/field_def.c",
"*/upb/reflection/file_def.c",
"*/upb/reflection/internal/def_builder.c",
"*/upb/reflection/internal/strdup2.c",
"*/upb/reflection/message.c",
"*/upb/reflection/message_def.c",
"*/upb/reflection/message_reserved_range.c",
"*/upb/reflection/method_def.c",
"*/upb/reflection/oneof_def.c",
"*/upb/reflection/service_def.c",
"*/upb/text/debug_string.c",
"*/upb/text/encode.c",
"*/upb/text/internal/encode.c",
"*/upb/util/def_to_proto.c",
"*/upb/util/required_fields.c",
"*/upb/wire/byte_size.c",
"*/upb/wire/decode.c",
"*/upb/wire/decode_fast/select.c",
"*/upb/wire/encode.c",
"*/upb/wire/eps_copy_input_stream.c",
"*/upb/wire/internal/decoder.c",
"*/upb/wire/reader.c",
-- upstream's bootstrap_sources (cmake/libupb.cmake)
"*/upb/reflection/cmake/google/protobuf/descriptor.upb_minitable.c",
},
},
},

linux = {
Expand Down
Loading
Loading