From 1fec0c503d9bf3176ed7e645311ea6e5ccec0537 Mon Sep 17 00:00:00 2001 From: Steve Downey Date: Mon, 19 Oct 2026 13:32:40 -0400 Subject: [PATCH 1/9] Plan the extraction of the negative compile test macro into a reusable infra module The add_fail_test macro in tests/beman/expected/CMakeLists.txt is used 59 times here and once, in weaker form, in beman.optional. Lay out how it becomes beman-xfail-compile.cmake in bemanproject/infra, consumed the same way as beman-install-library and BuildTelemetry: a project-local copy and call-site conversion first, a driver script that tells a successful compile from a wrongly-failed one, the EXPECT comment made authoritative after reconciling the 24 files where it has drifted from the CMake regex, then the infra PR and the pin bump. --- docs/plan/extract-xfail-compile.md | 373 +++++++++++++++++++++++++++++ 1 file changed, 373 insertions(+) create mode 100644 docs/plan/extract-xfail-compile.md diff --git a/docs/plan/extract-xfail-compile.md b/docs/plan/extract-xfail-compile.md new file mode 100644 index 0000000..39cd417 --- /dev/null +++ b/docs/plan/extract-xfail-compile.md @@ -0,0 +1,373 @@ +# Plan: extract-xfail-compile + +**Branch:** `extract-xfail-compile` (from `upstream/main` at `cb743bf`) +**Read first:** `docs/plan/tests-overview.md` §3 and §6 — the negative +compile test pattern and the coverage rule that makes it load-bearing. + +--- + +## Goal + +Turn the `add_fail_test()` macro in `tests/beman/expected/CMakeLists.txt` +into a reusable CMake module that lives in `bemanproject/infra` next to +`beman-install-library.cmake` and `BuildTelemetryConfig.cmake`, is consumed +the same way (`include(infra/cmake/.cmake)` then one function call per +test), and is documented in the infra README in the same shape as those two. + +`beman.expected` becomes the first consumer, with no change to which tests +pass, and `beman.optional` is the obvious second one. + +## What exists today + +### In `beman.expected` + +`tests/beman/expected/CMakeLists.txt` defines a three-argument macro and +calls it 59 times: + +```cmake +macro(add_fail_test name source reason) + add_library(beman.expected.tests.${name} OBJECT) + target_sources(beman.expected.tests.${name} PRIVATE ${source}) + target_link_libraries(beman.expected.tests.${name} PRIVATE beman::expected) + set_target_properties(beman.expected.tests.${name} + PROPERTIES EXCLUDE_FROM_ALL true EXCLUDE_FROM_DEFAULT_BUILD true) + add_test(NAME ${name} + COMMAND ${CMAKE_COMMAND} --build "${CMAKE_BINARY_DIR}" + --target beman.expected.tests.${name} --config $) + set_tests_properties(${name} PROPERTIES PASS_REGULAR_EXPRESSION "${reason}") +endmacro() +``` + +The moving parts, each of which the module has to keep: + +- **An OBJECT library per test**, excluded from `all`, so the ill-formed + translation unit is never built by an ordinary build and is never linked. + Linking to `beman::expected` is what gives it the real include paths, + compile features, and (under `BEMAN_EXPECTED_USE_MODULES`) the module BMI. +- **The test is a nested build.** ctest runs `cmake --build --target` on that + one object library, so the compiler invocation is exactly the one the + library's own tests get: same toolchain file, same flags, same launcher. +- **Pass means "the output matched the regex".** A compile that succeeds + produces no diagnostic and fails the test. A compile that fails for an + unrelated reason also fails the test, because the regex names the intended + `static_assert` text, deleted-function wording, or "no matching function". + The regexes carry alternations for gcc, clang, and MSVC phrasing. + +Every `*_fail.test.cpp` is also supposed to carry a header comment +(`// NEGATIVE: ...` and `// EXPECT: ""`) repeating the regex. An +audit on this branch's base shows that convention has drifted: + +| EXPECT comment vs CMake regex | Files | +|---|---| +| identical | 35 | +| different (CMake regex is a superset, usually adding MSVC/clang wording) | 15 | +| no EXPECT comment at all | 9 | + +So the CMake argument is the authority today and the comment is +documentation that has fallen behind. See +[expect-authority](#expect-authority). + +### In `beman.optional` + +One negative test (`constructor_fails`) built the same way, but checked with +`WILL_FAIL true` and the regex check commented out. That is the weaker form: +it passes when the file fails to compile for *any* reason, including a typo. +It is the same mechanism with the diagnostic check turned off, which is +exactly what a shared module makes cheap to turn back on. + +### In `infra` + +`infra/cmake/` holds two reusable modules and the shape to copy: + +- `beman-install-library.cmake`: `include_guard(GLOBAL)`, a long header + comment giving Usage / Arguments / Brief / Cache variables / Caveats, a + `cmake_parse_arguments` keyword interface, verbose `message()`s prefixed + with the function name, and a sibling data file (`Config.cmake.in`) found + via `CMAKE_CURRENT_FUNCTION_LIST_DIR`. +- `BuildTelemetryConfig.cmake` plus `telemetry.sh`: a module that ships a + helper script next to itself, callable via `include()` or via + `find_package()` with `CMAKE_PREFIX_PATH=infra/cmake`. +- Both documented in `infra/README.md` under "CMake Modules". +- Formatting is gersemi, enforced by infra's pre-commit. + +Nothing in infra does compile-failure tests, and infra has no tests of its +own modules; consumers are the test. + +`infra/` in this repo is subtree-vendored by the `beman-submodule` CLI +(pinned in `infra/.beman_submodule` to `749c558`), so edits made under +`infra/` here are overwritten by the next `beman-submodule update`. The +module has to go upstream through a fork of `bemanproject/infra` +(`~/src/beman-infra/main`, `origin` = `steve-downey/infra`; it needs an +`upstream` remote added for the PR). + +## Proposed interface + +```cmake +include(infra/cmake/beman-xfail-compile.cmake) + +beman_add_xfail_compile_test(unexpected_array_fail + SOURCE unexpected_array_fail.test.cpp + LINK_LIBRARIES beman::expected + EXPECT "E must not be an array type" +) +``` + +``` +beman_add_xfail_compile_test( + SOURCE + [EXPECT ] # omitted: read `// EXPECT: "..."` from SOURCE + [LINK_LIBRARIES ...] + [COMPILE_DEFINITIONS ...] + [TARGET_PREFIX ] # default "${PROJECT_NAME}.tests." + [LABELS