Skip to content

Add beman_add_xfail_compile_test for negative compile tests - #67

Merged
steve-downey merged 1 commit into
bemanproject:mainfrom
steve-downey:xfail-compile
Oct 22, 2026
Merged

steve-downey merged 1 commit into
bemanproject:mainfrom
steve-downey:xfail-compile

Conversation

@steve-downey

Copy link
Copy Markdown
Member

Work in progress. Opened early for naming and interface feedback; the module is in use and verified downstream, but the name, the argument shape, and whether infra wants a self-test for it are open.

What this adds

cmake/beman-xfail-compile.cmake provides beman_add_xfail_compile_test(), for negative compile tests: a translation unit that must fail to compile, and must fail with a specific diagnostic (a Mandates static_assert message, a = delete("...") message, or "no matching function" when a constraint removes every candidate). cmake/beman-xfail-compile-driver.cmake is the script CTest runs. A README section documents both next to beman_install_library.

include(infra/cmake/beman-xfail-compile.cmake)

beman_add_xfail_compile_test(unexpected_array_fail
    SOURCE unexpected_array_fail.test.cpp
    LINK_LIBRARIES beman::something
)
// NEGATIVE: unexpected<int[]> is ill-formed [expected.un.general]
// EXPECT: "E must not be an array type"
#include <beman/something/unexpected.hpp>
beman::something::unexpected<int[]> u(std::in_place); // must not compile

The source becomes an OBJECT library excluded from the default build; the test is a nested cmake --build of just that target, so it compiles with exactly the toolchain, flags, launcher, and usage requirements the rest of the project gets. The driver passes only when that build fails and the output matches the EXPECT regex, and reports "compiled, but was expected to fail" and "failed, but not for the expected reason" separately. The regex is read from the source at configure time (several EXPECT lines are OR-ed), or passed as EXPECT. Tests get the label xfail-compile and a RESOURCE_LOCK so ctest -j never overlaps two nested builds in one tree.

Why in infra

  • beman.expected has 59 such tests behind a local macro: Extract the negative compile test macro into a reusable beman_add_xfail_compile_test module expected#108 converts them to this module, with identical content, in its own cmake/ directory pending this PR.
  • beman.optional has one, checked with WILL_FAIL alone, which passes on any compile error at all, including a typo in the test. The module makes the diagnostic check the default.
  • Both repos already use the same nested-build mechanism; this is the shared, documented form of it.

Verification (downstream, in expected)

Check Result
59 tests, gcc 16 full suite 1250/1250
59 tests, clang 23 with libstdc++ pass
ctest -L xfail-compile -j 8 59/59
Probe that compiles / probe with a syntax error each fails with its own message
Source without an EXPECT line configure fails naming the test and file
This repo's pre-commit (gersemi 0.27.2) passes

Open questions for reviewers

  1. Name. beman-xfail-compile.cmake / beman_add_xfail_compile_test(), following beman-install-library.cmake / beman_install_library(). "xfail" is the DejaGnu/pytest/lit term for expected-to-fail; "compile" names the phase.
  2. Self-test. Infra has no tests of its modules today. A tiny smoke project under cmake/ would be a new kind of thing here; happy to add one if wanted.
  3. MSVC. The driver uses cmake --build --config, so it should work with Visual Studio generators, but it has only been run on Ninja Multi-Config so far.

A negative compile test is a translation unit that must fail to compile
with a specific diagnostic: a Mandates static_assert message, a deleted
overload's message, or the no-matching-function wording when a constraint
removes every candidate. beman.expected has 59 of them behind a local
macro and beman.optional has one checked only with WILL_FAIL, which
passes on any compile error at all.

beman-xfail-compile.cmake registers such a test: the source becomes an
OBJECT library excluded from the default build, and the CTest test runs
beman-xfail-compile-driver.cmake, which does a nested build of just that
target and passes only when the build fails and its output matches the
expected regex. The driver reports the two ways of failing separately,
so a constraint that stopped firing reads differently from a typo in the
test or a compiler whose wording is not yet covered. The regex lives in
the source as one or more '// EXPECT: "<regex>"' lines, read at
configure time, or is passed as EXPECT. Tests carry the label
xfail-compile and a RESOURCE_LOCK so ctest -j never runs two nested
builds in one tree.

The module is in use, with identical content, in beman.expected's
cmake/ directory pending this change; the README documents it alongside
beman_install_library.
@ednolan

ednolan commented Oct 19, 2026

Copy link
Copy Markdown
Member

This all looks good to me.

Name. beman-xfail-compile.cmake / beman_add_xfail_compile_test(), following beman-install-library.cmake / beman_install_library(). "xfail" is the DejaGnu/pytest/lit term for expected-to-fail; "compile" names the phase.

That seems like the right name.

Self-test. Infra has no tests of its modules today. A tiny smoke project under cmake/ would be a new kind of thing here; happy to add one if wanted.

I think the approach to testing of these infra components that we take, such as it is, is that they get used by downstream projects which then in turn need to pass CI. That's not perfect, but I think it's good enough here; this component has been tested to work when incorporated into beman.expected, so I think that covers it

MSVC. The driver uses cmake --build --config, so it should work with Visual Studio generators, but it has only been run on Ninja Multi-Config so far.

Not a blocker.

@steve-downey
steve-downey marked this pull request as ready for review October 20, 2026 17:46
@steve-downey steve-downey changed the title WIP: Add beman_add_xfail_compile_test for negative compile tests Add beman_add_xfail_compile_test for negative compile tests Oct 20, 2026
@steve-downey
steve-downey merged commit 0356b23 into bemanproject:main Oct 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants