Skip to content

Extract the negative compile test macro into a reusable beman_add_xfail_compile_test module - #108

Merged
steve-downey merged 9 commits into
bemanproject:mainfrom
steve-downey:extract-xfail-compile
Oct 20, 2026
Merged

steve-downey merged 9 commits into
bemanproject:mainfrom
steve-downey:extract-xfail-compile

Conversation

@steve-downey

Copy link
Copy Markdown
Member

Summary

The 59 negative compile tests were registered through a three-argument add_fail_test macro local to tests/beman/expected/CMakeLists.txt. This PR turns it into a reusable CMake module, cmake/beman-xfail-compile.cmake, written in the shape of beman-install-library.cmake so it can move to bemanproject/infra unchanged. Which tests exist and what each one expects does not change: every regex ctest receives was checked byte-identical to the one the macro passed.

Three commits do the work, each with its own verification; the plan and its decision log are in docs/plan/extract-xfail-compile.md.

  1. beman_add_xfail_compile_test replaces the macro. Keyword interface (SOURCE, LINK_LIBRARIES, COMPILE_DEFINITIONS, TARGET_PREFIX, LABELS, EXPECT), same per-test target names, same ctest names.
  2. A driver script runs the nested build instead of relying on PASS_REGULAR_EXPRESSION. It tells apart "the source compiled" from "it failed, but not with the expected diagnostic", neither of which the bare regex could report. Tests get the label xfail-compile and a RESOURCE_LOCK, so ctest -j is safe and ctest -L xfail-compile runs just this suite. $<CONFIGURATION> becomes $<CONFIG>.
  3. The regex lives in the test source. Each _fail.test.cpp carries // EXPECT: "<regex>" next to the ill-formed line; the module reads it at configure time, and a source without one fails configure naming the file. 35 files already matched, 15 carried an older subset, 9 had none; all 59 now carry the regex that was proven on gcc, clang, and MSVC. Long alternations are split across several EXPECT lines (joined with |) because clang-format reflows comments past column 119.

Demonstration

A real test under ctest -V now says which diagnostic it matched:

5: -- beman-xfail-compile-driver: beman.expected.tests.unexpected_array_fail failed as expected:
   .../include/beman/expected/unexpected.hpp:73:25: error: static assertion failed: unexpected<E>: E must not be an array type
1/1 Test #5: unexpected_array_fail ............   Passed    0.42 sec

A probe file that compiles cleanly, registered with the same regex:

1/1 Test #64: probe_compiles ...................***Failed    0.67 sec
  beman-xfail-compile-driver: beman.expected.tests.probe_compiles compiled,
  but was expected to fail with a diagnostic matching
      E must not be an array type

A probe file with a syntax error, which WILL_FAIL would have accepted:

1/1 Test #65: probe_syntax .....................***Failed    0.62 sec
  beman-xfail-compile-driver: beman.expected.tests.probe_syntax failed to
  compile, but not for the expected reason. No line matches
      E must not be an array type
  Build output: ...

Removing the EXPECT line from a source:

CMake Error at cmake/beman-xfail-compile.cmake:120 (message):
  beman_add_xfail_compile_test(unexpected_cvref_fail): no '// EXPECT: "<regex>"' line in
  .../tests/beman/expected/unexpected_cvref_fail.test.cpp; add one or pass EXPECT

Verification

Check Result
Regex ctest receives vs the macro's, all 59 byte-identical
gcc 16, full suite 1250/1250
clang 23 with libstdc++, full suite 1249/1249 (before the rebase onto #107)
ctest -L xfail-compile -j 8, gcc 16 59/59, about half the serial time
Touch a test source CMake re-runs before the build
pre-commit on changed files clang-format, gersemi, codespell pass

Follow-up

The module is proposed to bemanproject/infra with identical content. Once it lands there, a small PR here bumps the infra/ pin, deletes cmake/beman-xfail-compile*.cmake, and changes one include() path.

…e infra module

The add_fail_test macro in tests/beman/expected/CMakeLists.txt is used 59
times here and once, in weaker form, in beman.optional. Lay out how it
becomes beman-xfail-compile.cmake in bemanproject/infra, consumed the same
way as beman-install-library and BuildTelemetry: a project-local copy and
call-site conversion first, a driver script that tells a successful
compile from a wrongly-failed one, the EXPECT comment made authoritative
after reconciling the 24 files where it has drifted from the CMake regex,
then the infra PR and the pin bump.
…compile_test module

cmake/beman-xfail-compile.cmake holds the function in the form intended
for bemanproject/infra: include_guard, a documented keyword interface via
cmake_parse_arguments, and a per-test OBJECT library excluded from the
default build that ctest compiles in a nested build and checks against
EXPECT. The 59 call sites move to the keyword form with their regexes
unchanged; the per-test target names and ctest names are the same as
before. Call sites are laid out as gersemi formats them when it knows the
function definition, so they stay stable once a definitions path is
configured.

Verified: 1249/1249 tests pass on gcc 16 and on clang 23 with libstdc++,
ctest -N lists the same 59 negative test names as before, and breaking
one regex makes exactly that test fail.
… a wrong failure

beman-xfail-compile-driver.cmake performs the nested build itself and
exits nonzero with a distinct report when the source compiled, and when it
failed without a line matching EXPECT; on success it prints the matched
diagnostic. PASS_REGULAR_EXPRESSION alone could not tell the first two
apart and would have passed a successful build whose output happened to
contain the regex.

Each test also gets the label xfail-compile, so the negative suite can be
selected or skipped with -L/-LE, and the RESOURCE_LOCK
beman-xfail-compile-build-tree, so ctest -j never runs two nested builds
in one tree at once. The config reaches the driver as $<CONFIG>, in place
of the deprecated $<CONFIGURATION>, and is omitted from the nested build
when empty. EXPECT may not contain ';', which would split on the way to
the driver.

Verified on gcc 16: a source that compiles and a source with a syntax
error each fail with their own message, the 59 real tests pass with
-L xfail-compile -j 8, and the full suite passes on clang 23 with
libstdc++.
…ment

The regex that names a negative test's intended diagnostic now lives only
in the test source, as one or more '// EXPECT: "<regex>"' lines that
beman_add_xfail_compile_test reads at configure time and joins with '|'.
The 59 call sites drop their EXPECT arguments; passing one still
overrides the comment. Configure re-runs when a test source changes, and
fails naming the test and file when a source has no EXPECT line.

Of the 59 sources, 35 already carried the exact regex, 15 carried an
older subset, and 9 had none. All now carry the regex that CMake used,
which is the one proven on gcc, clang, and MSVC. The six files that
described themselves with a prose comment or not at all get a NEGATIVE
line like the rest. Alternations longer than clang-format's 119-column
limit are split across EXPECT lines at an alternative boundary, because
clang-format reflows a longer comment and merges it into the comment
that follows.

Verified: every regex ctest receives is byte-identical to the one the
call site used to pass; 1249/1249 tests pass on gcc 16 and on clang 23
with libstdc++; removing an EXPECT line fails configure with a message
naming the test and file; touching a test source re-runs configure.
Section 3 of docs/plan/tests-overview.md describes the new rule.
@steve-downey
steve-downey merged commit a174df2 into bemanproject:main Oct 20, 2026
79 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant