[msan] Keep the stack when an async signal arrives during a report - #118
Open
groeneai wants to merge 1 commit into
Open
Conversation
MSan intermittently produces a report with no stack at all:
==117==WARNING: MemorySanitizer: use-of-uninitialized-value
MemorySanitizer: nested bug in the same thread, aborting.
LLVM ERROR: IO failure on output stream: Broken pipe
ScopedErrorReportLock::Lock records the reporting thread and clears it only in
Unlock. Printing a report is slow, because stack->Print and ReportErrorSummary
round-trip to the out-of-process llvm-symbolizer. If an asynchronous signal is
delivered to the reporting thread inside that window and its handler touches
uninitialized memory, the second report re-enters the lock, takes the
same-thread branch and calls internal__exit without printing anything. The
first report is lost, and the pipe to the abandoned symbolizer child produces
the trailing LLVM ERROR line.
This is easy to hit in practice: ClickHouse arms a per-thread profiling timer on
every query thread, so every MSan report races with a signal. Its CI has been
collecting these empty reports on both x86_64 and AArch64, roughly one a day
across all MSan job types, and none of them can be diagnosed.
Block asynchronous signals for the duration of the report in
PrintWarningWithOrigin, the single funnel every UMR report goes through. On
Linux BlockSignals keeps the synchronous signals (SIGSEGV, SIGBUS, SIGILL,
SIGTRAP, SIGABRT, SIGFPE, SIGPIPE), SIGSYS and SIGSETXID unblocked, so a genuine
crash while reporting still aborts as before; only the async signals that cause
the wipe are deferred. Those exemptions are themselves Linux-only, so the change
is gated on SANITIZER_LINUX: elsewhere BlockSignals would mask SIGSEGV too.
The mask covers only the report body. ScopedErrorReportLock releases inside
ReportUMR, so a signal arriving during the caller's Die() can no longer reach
the same-thread branch.
The new test raises the signal from __sanitizer_on_print, which the runtime
calls for every line it prints, so the signal is delivered while the report lock
is held rather than at a timer-dependent moment. Measured 50 runs each: without
the change 50/50 abort with "nested bug" and print zero frames; with it 0/50
abort and every run prints report_here's full symbolized stack, the origin chain
and the summary line.
This was referenced Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MSan intermittently produces a report with no stack at all:
ScopedErrorReportLock::Lockrecords the reporting thread and clears it only inUnlock. Printing a report is slow, becausestack->PrintandReportErrorSummaryround-trip to the out-of-processllvm-symbolizer. If an asynchronous signal is delivered to the reporting thread inside that window and its handler touches uninitialized memory, the second report re-enters the lock, takes the same-thread branch and callsinternal__exitwithout printing anything. The first report is lost, and the pipe to the abandoned symbolizer child produces the trailingLLVM ERRORline. Upstreamsanitizer_symbolizer_report.cppdocuments this case ("This is either asynch signal or nested error during error reporting") but does not prevent it.This is easy to hit in practice: ClickHouse arms a per-thread profiling timer on every query thread, so every MSan report races with a signal. Its CI has collected 23 of these empty reports in the last 30 days on both x86_64 and AArch64, across six different MSan job types, and none can be diagnosed.
Block asynchronous signals for the duration of the report in
PrintWarningWithOrigin, the single funnel every UMR report goes through, placed after the existingmsan_expect_umrearly-return so__msan_expect_umrtests are unaffected. On LinuxBlockSignalskeeps the synchronous signals (SIGSEGV,SIGBUS,SIGILL,SIGTRAP,SIGABRT,SIGFPE,SIGPIPE),SIGSYSandSIGSETXIDunblocked, so a genuine crash while reporting still aborts as before, and it never unblocks a signal the caller had deliberately blocked. Those exemptions are themselves inside#if SANITIZER_LINUX, so the change is gated onSANITIZER_LINUX: elsewhereBlockSignalswould maskSIGSEGVtoo.The mask covers only the report body.
ScopedErrorReportLockreleases insideReportUMR, so a signal arriving during the caller'sDie()can no longer reach the same-thread branch, and__msan_warning_noreturn's unconditionalDie()still runs outside the mask.ScopedBlockSignalsis already MSan's tool for this class of protection (MsanTSDDtor, thepthread_createinterceptor) and has its own unit test, so this adds no new state.Measured with the new test, 50 runs each, against a runtime built from this branch:
nested bugabortsSUMMARY)Removing only the added line and rebuilding returns it to 50/50 aborts with 0 frames. A synchronous
SIGSEGVorSIGABRTraised inside the report window still produces a deadly-signal report and still aborts. The synchronous interceptor report path (CHECK_UNPOISONED_0viamemcmp) is unchanged and unaffected.Not covered: a nested report from a different thread (correctly serialized by
ScopedErrorReportLock), the same defect in ASan/TSan/UBSan report funnels (not observed), and non-Linux platforms (unchanged, still affected).I intend to offer this to
llvm/llvm-projectas well; there is no upstream issue or PR for it today.The paired ClickHouse submodule bump is ClickHouse/ClickHouse#112008.