build: publish the agent (multi-arch) and shim (amd64) images on a tag (#139) - #151
Merged
Merged
Conversation
The images are built and gated in CI and nothing has ever pushed them, so both Kubernetes examples name images that do not exist. This adds the publish, and fixes the reason it could only ever have worked on one architecture. Dockerfile.shim hardcoded x86_64 twice. The include path becomes a glob over targets/*-linux, which needs no branching at all; the CUDA repo URL genuinely must branch, and picks sbsa on aarch64 -- the server-class ARM tree, not the Jetson one, because the target is a GH200 or Graviton node. An unknown machine now fails with a message rather than a 404 from wget. Verified on amd64 by building the image and running the repo's own gate against the shim it produced: exactly one export, no libstdc++, no RUNPATH, 14 USDT notes, glibc 2.34, "-> portable". The arm64 build cannot be verified here, which is why the CI container job is now matrixed over both native runners: the arm64 path is exercised on every PR instead of being discovered broken during a release. Both images carry the release tag, and that is load-bearing rather than tidy. The shim's USDT record layouts are frozen per version and the agent decodes them, so a shim from one release paired with an agent from another is a decode failure with nothing in the pod spec to reveal it. Stated in both the workflow and the README that this is a convention: nothing at runtime refuses a mismatched pair today. Nothing publishes until a v* tag is pushed. The example manifests now say so rather than implying the images are there. Claude-Session: https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im
The arm64 image build failed the first time it ever ran: g++: error: unrecognized argument in option '-mtls-dialect=gnu' g++: note: valid arguments to '-mtls-dialect=' are: desc trad Not where I expected. The sbsa CUDA repo resolved, the packages installed and the include glob found its directory -- the compile itself rejected an x86-only flag. That flag selects TRADITIONAL TLS over TLS-descriptors, which is what keeps GLIBC_ABI_GNU2_TLS out of the object (#121). Both architectures have the choice; only the spelling differs, and the values are not interchangeable: x86-64 calls them gnu/gnu2, aarch64 trad/desc. So the fix is per-arch selection rather than dropping the flag, which would have reintroduced the marker that made the shim unloadable in the first place. Verified with a --no-cache amd64 build: the repo's own elfgate reports GLIBC_ABI_GNU2_TLS absent, glibc 2.34, one export, no libstdc++, no runpath, 14 USDT notes, portable. An earlier "successful" rebuild here had silently reused a cached layer, which is why this one is uncached. The explanation sits above the RUN rather than inside its line continuation, where a # is a parser footgun. Claude-Session: https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im
The arm64 image build surfaced a source portability gap rather than a packaging one. shim/core/usdt_probe.h binds its probe arguments to rdi/rsi/rdx by name, with no architecture guards, so the shim does not compile on aarch64 at all. probe_args_test.cc already knew -- it skips with "not x86-64" -- but nothing built the shim there to find out. So the shim is built and published for amd64 only, and that is deliberate rather than a gap left open. CUDA injection fails open and silent, so an arm64 shim that compiled with wrong register bindings would yield an empty profile and no error anywhere. Not shipping one is safer than shipping one nobody can validate. Dockerfile.agent had its own hardcoding on the same theme: it downloaded go...linux-amd64.tar.gz unconditionally, so the agent image could only ever have been built on one architecture either. Now per-arch, and the agent publishes a real multi-arch manifest while the shim gets single-arch tags rather than a manifest list naming an image that was never built. Verified by rebuilding both images here: the shim passes elfgate (portable, GLIBC_ABI_GNU2_TLS absent, one export, 14 USDT notes) and the agent still refuses without capabilities, naming which, with file caps on the binary. The port itself is next, and is verifiable without a GPU: probe_args_test traps its own probe and reads the argument registers, so teaching it aarch64 proves the binding on a native arm64 runner. Claude-Session: https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im
dpsoft
changed the title
Owner
Author
Updated: the arm64 build found a source gap, not a packaging oneThe CI matrix added in this PR failed on its first run, twice, and both failures were real. First: Second, and the reason the shim is now amd64-only:
Why not ship an arm64 shim anywayCUDA injection fails open and silent. An arm64 shim that compiled with wrong register bindings produces an empty profile and no error anywhere — indistinguishable from a workload that ran nothing. Not shipping one is safer than shipping one nobody has validated. Also found
Verified locallyBoth images rebuilt on amd64: the shim passes NextThe port is a separate PR, and it is verifiable without a GPU: |
dpsoft
added a commit
that referenced
this pull request
Sep 17, 2026
…64 (#152) * shim: bind USDT probe arguments per architecture, and prove it on arm64 usdt_probe.h bound its three arguments to rdi/rsi/rdx by name with no architecture guards, so the shim had never compiled on aarch64 -- which only surfaced when #151 first tried to build the image there. probe_args_test already knew, skipping with "not x86-64", but nothing built the shim on that arch to find out. The contract does not change: the probe still binds the first three integer-argument registers of the platform ABI, which is why bpf/gpu_usdt.bpf.c needs no port at all -- PT_REGS_PARM1..3 already read exactly those. Only the names differ, so only four defines branch and the .note.stapsdt generator stays in one copy. Two of them would drift, and the drift is invisible until a consumer reads a probe whose descriptor disagrees with where the arguments are. probe_args_test is the real deliverable. It patches its own probe site to a breakpoint and reads the argument registers out of the trapped context, so a wrong register name or descriptor FAILS rather than compiling. Three things differ on aarch64 and the last two are easy to miss: BRK is four bytes where int3 is one; BRK leaves PC ON the instruction, so the handler must advance it or re-trap forever; and writing an instruction through the data path needs an explicit i-cache flush, without which the trap simply never fires. The alarm() is there because the PC mistake hangs instead of failing, and a stuck CI job says far less than an assertion. A shim-tests job runs the suite on both native runners. It needs no CUDA, no CUPTI and no GPU -- just g++ -- which is what makes the aarch64 port verifiable without hardware nobody has. Verified on x86-64: descriptor still "8@%rdi 8@%rsi 8@%rdx" byte for byte, probe_args_test passes, full shim suite exits 0. aarch64 is unverified here by construction; the arm64 runner is the evidence. Claude-Session: https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im * shim: protect exactly the pages the probe instruction spans The arm64 run got further than the port's risky parts and died somewhere I had not looked: mprotect, before any trap was involved. The test took pagesize*2 unconditionally. That works until the probe sits near the end of its text mapping and the second page is not mapped at all, when mprotect fails ENOMEM. Layout-dependent rather than arch-dependent -- aarch64 is just where this repo's layout hit it first, and x86-64 would hit it on a different day. Now it protects exactly the pages the instruction spans: one normally, two only when it straddles a boundary. The failure also cost a round trip to diagnose, because a bare assert on mprotect says only that memory protection went wrong. ENOMEM (range not mapped) and EACCES (kernel refuses W|X) want completely different fixes, so it now prints errno with the probe address, page size and instruction width. Claude-Session: https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI builds and gates both images and nothing has ever pushed them, so both Kubernetes examples name images that do not exist. This adds the publish, and fixes the reason it could only ever have worked on one architecture.
Decisions implemented
v*tag, alongside the binaries. Nothing publishes until a tag is cut.Dockerfile.shimhardcoded x86_64 twicetargets/*-linux, which needs no branching at all, plus a check so a missing directory fails with a message instead of an empty-I.sbsaon aarch64 — the server-class ARM tree, since the target is a GH200 or Graviton node, not Jetson'saarch64tree. An unknown machine now fails with a message rather than a 404 fromwget.What is verified, and what is not
Verified on amd64 — built the image and ran the repo's own
shim/elfgate.shagainst the shim it produced:Not verified: the arm64 shim build. No native ARM here, and QEMU-emulating a CUDA toolkit install is not a faithful test. That is exactly why the CI container job is now matrixed over both native runners — the arm64 path gets exercised on every PR rather than discovered broken during a release. Without that, this change would be untested until the moment it mattered.
The publish path itself cannot be verified without pushing a tag, which I have not done.
Why one tag for both images
Load-bearing, not tidiness: the shim's USDT record layouts are frozen per version and the agent decodes them, so a shim from one release paired with an agent from another is a decode failure with nothing in the pod spec to reveal it.
Stated as a convention, because that is what it is — nothing at runtime refuses a mismatched pair. The
v1in@perfagent-gpu-enroll.v1.<major>.<minor>.<inode>versions the rendezvous name format, not the build. A runtime check would make the pairing a guarantee; that is a separate change and is noted in the workflow.Docs
The example manifests and README said the images were unpublishable. They now say the mechanism exists and nothing is published until the first tag, so
kubectl applyfailing to pull before then is expected rather than a bug.36 packages pass; all three YAML files parse.
https://claude.ai/code/session_01P5889hA6CrX8ysnQkvv6im