Repository navigation
Extract the negative compile test macro into a reusable beman_add_xfail_compile_test module - #108
Merged
steve-downey merged 9 commits intoOct 20, 2026
Conversation
…e infra module The add_fail_test macro in tests/beman/expected/CMakeLists.txt is used 59 times here and once, in weaker form, in beman.optional. Lay out how it becomes beman-xfail-compile.cmake in bemanproject/infra, consumed the same way as beman-install-library and BuildTelemetry: a project-local copy and call-site conversion first, a driver script that tells a successful compile from a wrongly-failed one, the EXPECT comment made authoritative after reconciling the 24 files where it has drifted from the CMake regex, then the infra PR and the pin bump.
…compile_test module cmake/beman-xfail-compile.cmake holds the function in the form intended for bemanproject/infra: include_guard, a documented keyword interface via cmake_parse_arguments, and a per-test OBJECT library excluded from the default build that ctest compiles in a nested build and checks against EXPECT. The 59 call sites move to the keyword form with their regexes unchanged; the per-test target names and ctest names are the same as before. Call sites are laid out as gersemi formats them when it knows the function definition, so they stay stable once a definitions path is configured. Verified: 1249/1249 tests pass on gcc 16 and on clang 23 with libstdc++, ctest -N lists the same 59 negative test names as before, and breaking one regex makes exactly that test fail.
… a wrong failure beman-xfail-compile-driver.cmake performs the nested build itself and exits nonzero with a distinct report when the source compiled, and when it failed without a line matching EXPECT; on success it prints the matched diagnostic. PASS_REGULAR_EXPRESSION alone could not tell the first two apart and would have passed a successful build whose output happened to contain the regex. Each test also gets the label xfail-compile, so the negative suite can be selected or skipped with -L/-LE, and the RESOURCE_LOCK beman-xfail-compile-build-tree, so ctest -j never runs two nested builds in one tree at once. The config reaches the driver as $<CONFIG>, in place of the deprecated $<CONFIGURATION>, and is omitted from the nested build when empty. EXPECT may not contain ';', which would split on the way to the driver. Verified on gcc 16: a source that compiles and a source with a syntax error each fail with their own message, the 59 real tests pass with -L xfail-compile -j 8, and the full suite passes on clang 23 with libstdc++.
…ment The regex that names a negative test's intended diagnostic now lives only in the test source, as one or more '// EXPECT: "<regex>"' lines that beman_add_xfail_compile_test reads at configure time and joins with '|'. The 59 call sites drop their EXPECT arguments; passing one still overrides the comment. Configure re-runs when a test source changes, and fails naming the test and file when a source has no EXPECT line. Of the 59 sources, 35 already carried the exact regex, 15 carried an older subset, and 9 had none. All now carry the regex that CMake used, which is the one proven on gcc, clang, and MSVC. The six files that described themselves with a prose comment or not at all get a NEGATIVE line like the rest. Alternations longer than clang-format's 119-column limit are split across EXPECT lines at an alternative boundary, because clang-format reflows a longer comment and merges it into the comment that follows. Verified: every regex ctest receives is byte-identical to the one the call site used to pass; 1249/1249 tests pass on gcc 16 and on clang 23 with libstdc++; removing an EXPECT line fails configure with a message naming the test and file; touching a test source re-runs configure. Section 3 of docs/plan/tests-overview.md describes the new rule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The 59 negative compile tests were registered through a three-argument
add_fail_testmacro local totests/beman/expected/CMakeLists.txt. This PR turns it into a reusable CMake module,cmake/beman-xfail-compile.cmake, written in the shape ofbeman-install-library.cmakeso it can move tobemanproject/infraunchanged. Which tests exist and what each one expects does not change: every regex ctest receives was checked byte-identical to the one the macro passed.Three commits do the work, each with its own verification; the plan and its decision log are in
docs/plan/extract-xfail-compile.md.beman_add_xfail_compile_testreplaces the macro. Keyword interface (SOURCE,LINK_LIBRARIES,COMPILE_DEFINITIONS,TARGET_PREFIX,LABELS,EXPECT), same per-test target names, same ctest names.PASS_REGULAR_EXPRESSION. It tells apart "the source compiled" from "it failed, but not with the expected diagnostic", neither of which the bare regex could report. Tests get the labelxfail-compileand aRESOURCE_LOCK, soctest -jis safe andctest -L xfail-compileruns just this suite.$<CONFIGURATION>becomes$<CONFIG>._fail.test.cppcarries// EXPECT: "<regex>"next to the ill-formed line; the module reads it at configure time, and a source without one fails configure naming the file. 35 files already matched, 15 carried an older subset, 9 had none; all 59 now carry the regex that was proven on gcc, clang, and MSVC. Long alternations are split across severalEXPECTlines (joined with|) because clang-format reflows comments past column 119.Demonstration
A real test under
ctest -Vnow says which diagnostic it matched:A probe file that compiles cleanly, registered with the same regex:
A probe file with a syntax error, which
WILL_FAILwould have accepted:Removing the
EXPECTline from a source:Verification
ctest -L xfail-compile -j 8, gcc 16Follow-up
The module is proposed to
bemanproject/infrawith identical content. Once it lands there, a small PR here bumps theinfra/pin, deletescmake/beman-xfail-compile*.cmake, and changes oneinclude()path.