Skip to content

[msan] Keep the stack when an async signal arrives during a report - #118

Open
groeneai wants to merge 1 commit into
ClickHouse:ClickHouse/release-22.xfrom
groeneai:ClickHouse/fix-msan-nested-report-stack-wipe
Open

[msan] Keep the stack when an async signal arrives during a report#118
groeneai wants to merge 1 commit into
ClickHouse:ClickHouse/release-22.xfrom
groeneai:ClickHouse/fix-msan-nested-report-stack-wipe

Conversation

@groeneai

@groeneai groeneai commented Jul 27, 2026

Copy link
Copy Markdown

MSan intermittently produces a report with no stack at all:

==117==WARNING: MemorySanitizer: use-of-uninitialized-value
MemorySanitizer: nested bug in the same thread, aborting.
LLVM ERROR: IO failure on output stream: Broken pipe

ScopedErrorReportLock::Lock records the reporting thread and clears it only in Unlock. Printing a report is slow, because stack->Print and ReportErrorSummary round-trip to the out-of-process llvm-symbolizer. If an asynchronous signal is delivered to the reporting thread inside that window and its handler touches uninitialized memory, the second report re-enters the lock, takes the same-thread branch and calls internal__exit without printing anything. The first report is lost, and the pipe to the abandoned symbolizer child produces the trailing LLVM ERROR line. Upstream sanitizer_symbolizer_report.cpp documents this case ("This is either asynch signal or nested error during error reporting") but does not prevent it.

This is easy to hit in practice: ClickHouse arms a per-thread profiling timer on every query thread, so every MSan report races with a signal. Its CI has collected 23 of these empty reports in the last 30 days on both x86_64 and AArch64, across six different MSan job types, and none can be diagnosed.

Block asynchronous signals for the duration of the report in PrintWarningWithOrigin, the single funnel every UMR report goes through, placed after the existing msan_expect_umr early-return so __msan_expect_umr tests are unaffected. On Linux BlockSignals keeps the synchronous signals (SIGSEGV, SIGBUS, SIGILL, SIGTRAP, SIGABRT, SIGFPE, SIGPIPE), SIGSYS and SIGSETXID unblocked, so a genuine crash while reporting still aborts as before, and it never unblocks a signal the caller had deliberately blocked. Those exemptions are themselves inside #if SANITIZER_LINUX, so the change is gated on SANITIZER_LINUX: elsewhere BlockSignals would mask SIGSEGV too.

The mask covers only the report body. ScopedErrorReportLock releases inside ReportUMR, so a signal arriving during the caller's Die() can no longer reach the same-thread branch, and __msan_warning_noreturn's unconditional Die() still runs outside the mask.

ScopedBlockSignals is already MSan's tool for this class of protection (MsanTSDDtor, the pthread_create interceptor) and has its own unit test, so this adds no new state.

Measured with the new test, 50 runs each, against a runtime built from this branch:

runtime nested bug aborts stack frames
without the change 50/50 0
with the change 0/50 1600 (full stack + origin chain + SUMMARY)

Removing only the added line and rebuilding returns it to 50/50 aborts with 0 frames. A synchronous SIGSEGV or SIGABRT raised inside the report window still produces a deadly-signal report and still aborts. The synchronous interceptor report path (CHECK_UNPOISONED_0 via memcmp) is unchanged and unaffected.

Not covered: a nested report from a different thread (correctly serialized by ScopedErrorReportLock), the same defect in ASan/TSan/UBSan report funnels (not observed), and non-Linux platforms (unchanged, still affected).

I intend to offer this to llvm/llvm-project as well; there is no upstream issue or PR for it today.

The paired ClickHouse submodule bump is ClickHouse/ClickHouse#112008.

MSan intermittently produces a report with no stack at all:

    ==117==WARNING: MemorySanitizer: use-of-uninitialized-value
    MemorySanitizer: nested bug in the same thread, aborting.
    LLVM ERROR: IO failure on output stream: Broken pipe

ScopedErrorReportLock::Lock records the reporting thread and clears it only in
Unlock. Printing a report is slow, because stack->Print and ReportErrorSummary
round-trip to the out-of-process llvm-symbolizer. If an asynchronous signal is
delivered to the reporting thread inside that window and its handler touches
uninitialized memory, the second report re-enters the lock, takes the
same-thread branch and calls internal__exit without printing anything. The
first report is lost, and the pipe to the abandoned symbolizer child produces
the trailing LLVM ERROR line.

This is easy to hit in practice: ClickHouse arms a per-thread profiling timer on
every query thread, so every MSan report races with a signal. Its CI has been
collecting these empty reports on both x86_64 and AArch64, roughly one a day
across all MSan job types, and none of them can be diagnosed.

Block asynchronous signals for the duration of the report in
PrintWarningWithOrigin, the single funnel every UMR report goes through. On
Linux BlockSignals keeps the synchronous signals (SIGSEGV, SIGBUS, SIGILL,
SIGTRAP, SIGABRT, SIGFPE, SIGPIPE), SIGSYS and SIGSETXID unblocked, so a genuine
crash while reporting still aborts as before; only the async signals that cause
the wipe are deferred. Those exemptions are themselves Linux-only, so the change
is gated on SANITIZER_LINUX: elsewhere BlockSignals would mask SIGSEGV too.

The mask covers only the report body. ScopedErrorReportLock releases inside
ReportUMR, so a signal arriving during the caller's Die() can no longer reach
the same-thread branch.

The new test raises the signal from __sanitizer_on_print, which the runtime
calls for every line it prints, so the signal is delivered while the report lock
is held rather than at a timer-dependent moment. Measured 50 runs each: without
the change 50/50 abort with "nested bug" and print zero frames; with it 0/50
abort and every run prints report_here's full symbolized stack, the origin chain
and the summary line.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant