shim: bind USDT probe arguments per architecture, and prove it on arm64 - #152
Merged
Merged
Conversation
usdt_probe.h bound its three arguments to rdi/rsi/rdx by name with no architecture guards, so the shim had never compiled on aarch64 -- which only surfaced when #151 first tried to build the image there. probe_args_test already knew, skipping with "not x86-64", but nothing built the shim on that arch to find out. The contract does not change: the probe still binds the first three integer-argument registers of the platform ABI, which is why bpf/gpu_usdt.bpf.c needs no port at all -- PT_REGS_PARM1..3 already read exactly those. Only the names differ, so only four defines branch and the .note.stapsdt generator stays in one copy. Two of them would drift, and the drift is invisible until a consumer reads a probe whose descriptor disagrees with where the arguments are. probe_args_test is the real deliverable. It patches its own probe site to a breakpoint and reads the argument registers out of the trapped context, so a wrong register name or descriptor FAILS rather than compiling. Three things differ on aarch64 and the last two are easy to miss: BRK is four bytes where int3 is one; BRK leaves PC ON the instruction, so the handler must advance it or re-trap forever; and writing an instruction through the data path needs an explicit i-cache flush, without which the trap simply never fires. The alarm() is there because the PC mistake hangs instead of failing, and a stuck CI job says far less than an assertion. A shim-tests job runs the suite on both native runners. It needs no CUDA, no CUPTI and no GPU -- just g++ -- which is what makes the aarch64 port verifiable without hardware nobody has. Verified on x86-64: descriptor still "8@%rdi 8@%rsi 8@%rdx" byte for byte, probe_args_test passes, full shim suite exits 0. aarch64 is unverified here by construction; the arm64 runner is the evidence. Claude-Session: https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im
The arm64 run got further than the port's risky parts and died somewhere I had not looked: mprotect, before any trap was involved. The test took pagesize*2 unconditionally. That works until the probe sits near the end of its text mapping and the second page is not mapped at all, when mprotect fails ENOMEM. Layout-dependent rather than arch-dependent -- aarch64 is just where this repo's layout hit it first, and x86-64 would hit it on a different day. Now it protects exactly the pages the instruction spans: one normally, two only when it straddles a boundary. The failure also cost a round trip to diagnose, because a bare assert on mprotect says only that memory protection went wrong. ENOMEM (range not mapped) and EACCES (kernel refuses W|X) want completely different fixes, so it now prints errno with the probe address, page size and instruction width. Claude-Session: https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im
dpsoft
added a commit
that referenced
this pull request
Sep 19, 2026
…rch shim image (#153) * build: one definition of the arch-sensitive shim flags, and a multi-arch shim The same compile command existed in four copies -- shim/Makefile's CXXFLAGS, its nvidia-portable target, and twice inline in ci.yml -- each spelling out -mtls-dialect and targets/x86_64-linux for itself. Adding aarch64 inline would have made six. Divergence between copies is invisible until a shim is built with the wrong TLS dialect and fails to dlopen, which is precisely what #121 was. TLS_DIALECT and CUDA_TARGET are now derived once from uname in shim/Makefile and used everywhere. A third architecture is two lines in one ifeq rather than four edits that can disagree. CI stops inlining the portable build and calls the Makefile target, with CONTAINER=docker because the runtime is the only thing that genuinely differs there; it also stops passing CUDA_INC, which is what let the job and the Makefile disagree about the target directory. elfgate.sh now reports the GLIBC_ABI_GNU2_TLS marker as n/a on non-x86-64 rather than "absent". That string is never emitted on aarch64, so reporting absence there would be a check that cannot fail dressed as a check that passed -- about the very property the gate exists to protect. The numeric glibc floor carries the weight on ARM and is genuinely arch-neutral. With the probes ported (#152) the shim image builds and publishes on both architectures, and both images get real manifest lists. The CUPTI integration is still unexercised on ARM for want of an arm64 GPU node, and the README says so rather than implying full support. Claude-Session: https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im * build: follow the portable shim to where the Makefile writes it Both arches failed the gate, which is the useful shape of failure: it says the break is mine rather than the architecture's. Calling make -C shim nvidia-portable instead of inlining the compile moved the artifact. The Makefile writes it next to the sources in shim/; the inline copy wrote it to the repo root, and the "Prove it loads" step still looked there -- so dlopen reported a missing file rather than an unloadable one. Running the make target locally did not catch it because the step that consumes the artifact is the next one along. Also fixes a collision the gate never reached: matrixing the job left both arches uploading an artifact under one name, which upload-artifact@v4 refuses. Per-arch now. Claude-Session: https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
shim/core/usdt_probe.hbound its three probe arguments tordi/rsi/rdxby name, with no architecture guards, so the shim had never compiled on aarch64. That only surfaced when #151 first tried to build the image there.probe_args_test.ccalready knew — it skipped with"not x86-64"— but nothing built the shim on that arch to find out.The contract does not change
The probe still binds the first three integer-argument registers of the platform ABI. That is why
bpf/gpu_usdt.bpf.cneeds no port at all —PT_REGS_PARM1..3already read exactly those, and its comment says so.Only the names differ, so only four defines branch and the
.note.stapsdtgenerator stays in one copy. Two generators would drift, and the drift is invisible until a consumer reads a probe whose descriptor disagrees with where the arguments actually are.probe_args_testis the real deliverableIt patches its own probe site to a breakpoint, fires the probe, and reads the argument registers out of the trapped context — so a wrong register name or a wrong descriptor fails rather than compiling.
Three things differ on aarch64, and the last two are easy to miss:
int3, 1 byteBRK #0, 4 bytespc += 4or re-trap forever__builtin___clear_cache, or the trap never firesThe
alarm(10)is deliberate: the PC mistake hangs rather than failing, and a stuck CI job says far less than an assertion does.What makes this verifiable at all
A
shim-testsjob runs the suite on both native runners. It needs no CUDA, no CUPTI and no GPU — justg++— which I confirmed by running it to completion in a bareubuntu:24.04container. That is what makes the aarch64 port provable without hardware nobody has.Verified here, and not
On x86-64: the argument descriptor is still
8@%rdi 8@%rsi 8@%rdxbyte for byte,probe_args_testpasses, the full shim suite exits 0,internal/usdt(which parses the notes) passes, 36 packages green.On aarch64: nothing, by construction. No ARM hardware here, and QEMU would not faithfully test signal and trap semantics. The arm64 runner is the evidence, and this PR failing there is the intended outcome if I got the trap semantics wrong.
Not in scope
Flipping the shim image to multi-arch. That needs the
shimgate job's own x86_64 hardcoding (CUDA keyring URL, include path) reworked — a packaging change that belongs in its own diff rather than riding along with an assembly port. The comment inci.ymlnow says exactly that instead of claiming the source cannot build.https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im