Repository navigation
Add beman_add_xfail_compile_test for negative compile tests - #67
Conversation
A negative compile test is a translation unit that must fail to compile with a specific diagnostic: a Mandates static_assert message, a deleted overload's message, or the no-matching-function wording when a constraint removes every candidate. beman.expected has 59 of them behind a local macro and beman.optional has one checked only with WILL_FAIL, which passes on any compile error at all. beman-xfail-compile.cmake registers such a test: the source becomes an OBJECT library excluded from the default build, and the CTest test runs beman-xfail-compile-driver.cmake, which does a nested build of just that target and passes only when the build fails and its output matches the expected regex. The driver reports the two ways of failing separately, so a constraint that stopped firing reads differently from a typo in the test or a compiler whose wording is not yet covered. The regex lives in the source as one or more '// EXPECT: "<regex>"' lines, read at configure time, or is passed as EXPECT. Tests carry the label xfail-compile and a RESOURCE_LOCK so ctest -j never runs two nested builds in one tree. The module is in use, with identical content, in beman.expected's cmake/ directory pending this change; the README documents it alongside beman_install_library.
|
This all looks good to me.
That seems like the right name.
I think the approach to testing of these infra components that we take, such as it is, is that they get used by downstream projects which then in turn need to pass CI. That's not perfect, but I think it's good enough here; this component has been tested to work when incorporated into beman.expected, so I think that covers it
Not a blocker. |
Work in progress. Opened early for naming and interface feedback; the module is in use and verified downstream, but the name, the argument shape, and whether infra wants a self-test for it are open.
What this adds
cmake/beman-xfail-compile.cmakeprovidesbeman_add_xfail_compile_test(), for negative compile tests: a translation unit that must fail to compile, and must fail with a specific diagnostic (a Mandatesstatic_assertmessage, a= delete("...")message, or "no matching function" when a constraint removes every candidate).cmake/beman-xfail-compile-driver.cmakeis the script CTest runs. A README section documents both next tobeman_install_library.The source becomes an OBJECT library excluded from the default build; the test is a nested
cmake --buildof just that target, so it compiles with exactly the toolchain, flags, launcher, and usage requirements the rest of the project gets. The driver passes only when that build fails and the output matches theEXPECTregex, and reports "compiled, but was expected to fail" and "failed, but not for the expected reason" separately. The regex is read from the source at configure time (severalEXPECTlines are OR-ed), or passed asEXPECT. Tests get the labelxfail-compileand aRESOURCE_LOCKsoctest -jnever overlaps two nested builds in one tree.Why in infra
cmake/directory pending this PR.WILL_FAILalone, which passes on any compile error at all, including a typo in the test. The module makes the diagnostic check the default.Verification (downstream, in expected)
ctest -L xfail-compile -j 8EXPECTlineOpen questions for reviewers
beman-xfail-compile.cmake/beman_add_xfail_compile_test(), followingbeman-install-library.cmake/beman_install_library(). "xfail" is the DejaGnu/pytest/lit term for expected-to-fail; "compile" names the phase.cmake/would be a new kind of thing here; happy to add one if wanted.cmake --build --config, so it should work with Visual Studio generators, but it has only been run on Ninja Multi-Config so far.