ci: record the native side of the Windows coverage crash

faulthandler stops exactly where the answer starts. The fault is inside an
extension module's initialiser, so the frames that matter are the ones it
cannot reach, and every report ends `<cannot get C stack on this system>`.

Every ordinary way to see past that is closed on this runner. cdb is not
installed: the SDK is present without the Debugging Tools feature, so
Windows Kits ships the debugger DLLs and none of the executables. procdump
would mean an outbound download on a runner whose agent exists to police
outbound requests. A postmortem debugger under AeDebug never fires, because
this image does not record user-process crashes with WER, which is why the
crash-dumps directory came back empty when that was tried. And there are no
PDBs in any case, since build_hooks.py refuses debuggable builds on Windows.

dbgcore.dll in System32 is always there though, and exports
MiniDumpWriteDump, so the process reports on itself. A vectored handler
writes the exception record, the faulting address resolved to module + RVA,
the native return chain resolved the same way, the module map, and a
minidump, then returns EXCEPTION_CONTINUE_SEARCH so the process still dies
with the exit code CI would have seen. Symbols are absent, but which binary
faulted is the open question and module + RVA answers it.

crash_probe.py drives four runs. A selftest faults on purpose, so a clean
report proves the handler works here and an empty one is never mistaken for
"nothing crashed". A run under coverage reproduces the nightly's own
conditions on the 8 MB stack the pytest wrapper uses. A run with a bare
trace function instead of coverage arms Cython's linetrace the same way, so
a fault there means any tracer will do rather than coverage specifically,
and the last line of its breadcrumb names the .pyx statement that was
executing. env_fingerprint.py records the environment around the import --
driver, CPU, and the loaded-module map -- which is what comparing against a
machine that does not fault requires.

This replaces the per-submodule probes, which measured nothing: importing
cuda.core._resource_handles executes cuda/core/__init__.py on the way in, so
all six walked the same path and all six failed identically.

Verified on two Windows runners that do not fault, where the selftest
produced a resolved address, a 44-frame backtrace and a 300 KB dump, and the
breadcrumb recorded .pyx line numbers through to the end of the import.

Author: rluo8

.github/workflows/coverage.yml

@@ -387,54 +387,88 @@ jobs:
             --cov-config="$GITHUB_WORKSPACE/.coveragerc" \
             "$GITHUB_WORKSPACE/cuda_bindings/tests"
 
-      # Import tracing put the fault in cuda.core._event, reached through
-      # _stream from _context.  These probes strip everything that is not
-      # needed to see it: no pytest, no conftest, no tests -- one fresh
-      # interpreter per module, importing exactly one name under coverage,
-      # since the linetrace wheel and the ctracer are the two conditions
-      # `test-wheel-windows.yml` lacks on this same runner.
+      # The whole crash reduces to one `import cuda.core` under coverage, with
+      # no pytest, no conftest and no tests: the linetrace wheel and the
+      # ctracer are the two conditions `test-wheel-windows.yml` lacks on this
+      # same runner.  Importing submodules one at a time was tried first and
+      # measured nothing -- every one of them executes cuda/core/__init__.py
+      # on the way in, so all six probes walked the same path and all six
+      # failed identically.
       #
-      # The order runs leaf-first, so the first line that fails to print
-      # `PROBE OK` is the smallest unit that reproduces.  Importing _event
-      # directly also enters the _context/_stream/_event cimport cycle from
-      # the opposite side, which says whether the direction matters.
+      # What is missing is the native half.  The fault happens inside an
+      # extension module's initialiser, and faulthandler's report ends at
+      # `<cannot get C stack on this system>`.  The usual ways to see past
+      # that are all closed here: cdb is not installed (the SDK ships without
+      # the Debugging Tools feature), procdump would need an outbound
+      # download, a postmortem debugger under AeDebug never fires because this
+      # image does not record user-process crashes with WER, and there are no
+      # PDBs because build_hooks.py refuses debuggable builds on Windows.
       #
-      # Placed here rather than before the other suites because this is the
-      # machine state the crash is known to occur in; a clean-machine
-      # placement is the follow-up if these come back clean.
+      # dbgcore.dll in System32 is always present though, so the process
+      # reports on itself.  ci/tools/native_crash_handler.py catches the fault
+      # first-chance and writes the faulting address resolved to module + RVA,
+      # the native return chain resolved the same way, the module map, and a
+      # minidump, then lets the process die exactly as it would have.  Which
+      # binary faults is the open question, and module + RVA answers it
+      # without symbols.
       #
-      # COVERAGE_FILE keeps the probes out of the real .coverage, which
+      # Four runs, each answering something the others cannot:
+      #
+      #   selftest     faults on purpose, so a clean report proves the handler
+      #                works here and an empty one is not mistaken for "no
+      #                crash"
+      #   coverage     the nightly's own conditions, on the 8 MB stack the
+      #                pytest wrapper uses
+      #   breadcrumb   the same import with a bare trace function instead of
+      #                coverage.  It arms Cython's linetrace the same way, so
+      #                a crash here means any tracer will do rather than
+      #                coverage specifically -- and the last line of its log
+      #                names the .pyx statement that was executing
+      #   fingerprint  the environment around the import: driver, CPU, and
+      #                the loaded-module map, which is what a comparison
+      #                against a machine that does not crash needs
+      #
+      # COVERAGE_FILE keeps all of this out of the real .coverage, which
       # `coverage run` would otherwise truncate.
-      - name: Probe which module import faults
+      - name: Record the native side of the fault
         if: always()
         continue-on-error: true
         env:
           COVERAGE_FILE: ${{ github.workspace }}/.coverage.probe
         run: |
-          cat > "$GITHUB_WORKSPACE/probe_import.py" <<'EOF'
-          import importlib
-          import sys
-
-          name = sys.argv[1]
-          importlib.import_module(name)
-          print(f"PROBE OK {name}", flush=True)
-          EOF
-
+          probe="$GITHUB_WORKSPACE/ci/tools/crash_probe.py"
+          python="$GITHUB_WORKSPACE/.venv/Scripts/python"
           cd "${{ steps.install-root.outputs.INSTALL_ROOT }}"
-          for mod in \
-            cuda.core._resource_handles \
-            cuda.core._device_resources \
-            cuda.core._event \
-            cuda.core._stream \
-            cuda.core._context \
-            cuda.core
-          do
-            rc=0
-            "$GITHUB_WORKSPACE/.venv/Scripts/python" -m coverage run \
-              --rcfile="$GITHUB_WORKSPACE/.coveragerc" \
-              "$GITHUB_WORKSPACE/probe_import.py" "$mod" || rc=$?
-            echo "PROBE EXIT $mod rc=$rc"
-          done
+
+          rc=0
+          "$python" "$probe" --selftest \
+            --native-log "$GITHUB_WORKSPACE/native-selftest.txt" \
+            --dump "$GITHUB_WORKSPACE/selftest.dmp" || rc=$?
+          echo "SELFTEST rc=$rc"
+          head -n 10 "$GITHUB_WORKSPACE/native-selftest.txt" || true
+
+          rc=0
+          "$python" -m coverage run --rcfile="$GITHUB_WORKSPACE/.coveragerc" \
+            "$probe" --stack-mb 8 \
+            --native-log "$GITHUB_WORKSPACE/native-coverage.txt" \
+            --dump "$GITHUB_WORKSPACE/crash-coverage.dmp" || rc=$?
+          echo "COVERAGE PROBE rc=$rc"
+          cat "$GITHUB_WORKSPACE/native-coverage.txt" || true
+
+          rc=0
+          "$python" "$probe" --stack-mb 8 \
+            --breadcrumb "$GITHUB_WORKSPACE/breadcrumb.txt" \
+            --native-log "$GITHUB_WORKSPACE/native-breadcrumb.txt" \
+            --dump "$GITHUB_WORKSPACE/crash-breadcrumb.dmp" || rc=$?
+          echo "BREADCRUMB PROBE rc=$rc"
+          echo "=== breadcrumb: $(wc -l < "$GITHUB_WORKSPACE/breadcrumb.txt") lines, last 25 ==="
+          tail -n 25 "$GITHUB_WORKSPACE/breadcrumb.txt" || true
+
+          rc=0
+          "$python" -m coverage run --rcfile="$GITHUB_WORKSPACE/.coveragerc" \
+            "$GITHUB_WORKSPACE/ci/tools/env_fingerprint.py" --stack-mb 8 \
+            "$GITHUB_WORKSPACE/fingerprint-windows.txt" || rc=$?
+          echo "FINGERPRINT rc=$rc"
 
       # Only this step crashes.  cuda.bindings runs the same wrapper, the same
       # 8 MB thread and the same coverage wheel, and finishes normally, so the
@@ -532,6 +566,14 @@ jobs:
             crashlog-core.txt
             faulthandler-core.txt
             verbose-core.txt
+            native-selftest.txt
+            native-coverage.txt
+            native-breadcrumb.txt
+            breadcrumb.txt
+            fingerprint-windows.txt
+            selftest.dmp
+            crash-coverage.dmp
+            crash-breadcrumb.dmp
           retention-days: 7
           if-no-files-found: warn
 

ci/tools/crash_probe.py

@@ -0,0 +1,125 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+"""Import cuda.core under full instrumentation and record where it dies.
+
+The crash reduces to a single import, on a thread the size of the one
+run_pytest_with_stack.py creates.  Two records are taken around it, and they
+answer different halves of the same question:
+
+    the breadcrumb   the last .pyx line that executed, i.e. which source
+                     statement was running -- written by our own trace
+                     function, which also arms Cython's linetrace hooks the
+                     way coverage does, so the fault still triggers
+    the native log   which module faulted and at what offset, from
+                     native_crash_handler
+
+A run under coverage answers "does it still crash", and a run with the
+breadcrumb instead of coverage answers "is coverage necessary, or does any
+tracer do".  Only one trace function can be installed at a time, so those are
+separate invocations rather than one.
+
+    crash_probe.py --stack-mb 8 --native-log native.txt --dump crash.dmp
+    crash_probe.py --stack-mb 8 --breadcrumb lines.txt --native-log native.txt
+    crash_probe.py --selftest --native-log native.txt
+
+--selftest writes to address zero on purpose.  It is how the whole chain gets
+verified somewhere that does not crash on its own: if the handler, the module
+map, the dump and the artifact upload all work there, then the one gated run
+on the runner that does crash is not spent discovering a typo.
+"""
+
+from __future__ import annotations
+
+import argparse
+import ctypes
+import importlib
+import os
+import sys
+import threading
+import time
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+import native_crash_handler
+
+
+def install_breadcrumb(path: str, needle: str) -> None:
+    """Log every executed line whose file matches `needle`, line buffered.
+
+    Cython emits __Pyx_TraceLine for module-level statements too, which is the
+    only reason this can see anything at all: the fault happens while a module
+    body is still executing, before any function in it has been called.
+    """
+    log = open(  # noqa: SIM115 - must outlive this function
+        os.path.abspath(path), "w", buffering=1, encoding="utf-8", errors="backslashreplace"
+    )
+
+    def tracer(frame, event, _arg):
+        if event == "line":
+            filename = frame.f_code.co_filename
+            if needle in filename:
+                log.write(f"{filename}:{frame.f_lineno}\n")
+        return tracer
+
+    # Only one trace function exists per thread, so arming this one under
+    # `coverage run` would quietly switch coverage off and leave a run that
+    # looks like it measured something.  Say so rather than let it pass.
+    displaced = sys.gettrace()
+    if displaced is not None:
+        message = f"# WARNING: replaced an existing trace function: {displaced!r}"
+        log.write(message + "\n")
+        print(message, flush=True)
+
+    log.write(f"# breadcrumb armed, filter={needle!r}\n")
+    sys.settrace(tracer)
+
+
+def selftest() -> None:
+    """Deliberately fault, to prove the handler reports what it should."""
+    print("selftest: writing to address 0", flush=True)
+    ctypes.memset(0, 0, 1)
+    print("selftest: still alive -- the write did NOT fault", flush=True)
+
+
+def body(args) -> None:
+    if args.breadcrumb:
+        install_breadcrumb(args.breadcrumb, args.breadcrumb_filter)
+    if args.selftest:
+        selftest()
+        return
+    started = time.perf_counter()
+    importlib.import_module(args.module)
+    print(f"PROBE OK {args.module} in {time.perf_counter() - started:.2f}s", flush=True)
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--module", default="cuda.core")
+    parser.add_argument("--stack-mb", type=float, default=0.0)
+    parser.add_argument("--native-log", default="crash-native.txt")
+    parser.add_argument("--dump")
+    parser.add_argument("--breadcrumb")
+    parser.add_argument("--breadcrumb-filter", default="cuda")
+    parser.add_argument("--selftest", action="store_true")
+    args = parser.parse_args()
+
+    # Process-wide, so arming it here covers the worker thread as well.
+    native_crash_handler.install(args.native_log, dump_path=args.dump)
+
+    if not args.stack_mb:
+        body(args)
+        return 0
+
+    # sys.settrace only affects the thread that calls it, so the breadcrumb is
+    # installed inside body(), on the thread that does the import.
+    threading.stack_size(int(args.stack_mb * 1024 * 1024))
+    worker = threading.Thread(target=body, args=(args,), name=f"stack-{args.stack_mb:g}mb")
+    worker.start()
+    worker.join()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())