[IR] Add the llvm.zeroize intrinsic - #2
Conversation
|
|
66b3ca1 to
a50c171
Compare
68b8fec to
0bd0b1b
Compare
claude
Bot
changed the title
0bd0b1b to
f01c5be
Compare
a50c171 to
a7281ea
Compare
Add an overloaded pointer-and-length intrinsic that writes zero over [dest, dest + len) and guarantees the clear is emitted without calling an external function and without establishing a call frame. That codegen guarantee is the reason for the intrinsic. Clearing a buffer by other means goes through memset, and a memset over a dynamic length or a large fixed one lowers to a libcall. Measured on x86-64 with six values live across the clear, the call forces all six into callee-saved registers and pushes them in the prologue, grows the frame from 8 to 56 bytes, and plants a callee frame below the stack pointer. A clear that spills the live state it is trying to erase, into the stack it is trying to erase, defeats the purpose. Expanded as a pseudo after register allocation the same clear is a rep stosb with no libcall at any size. Non-removability is part of the contract too, but it is not what makes the intrinsic necessary and the previous version of this change oversold it. A volatile memset is also not removable: DSEState::isRemovable refuses to remove volatile memory intrinsics unconditionally, so a volatile memset survives dead store elimination and a full pipeline in every shape the test covers. What a volatile memset cannot do is lower without the libcall. llvm.memset.inline does not close the gap either. It survives dead store elimination and lowers without a libcall, but SROA rewrites it back into a plain memset on non-escaping stack allocas, which is exactly the case this targets, and the no-libcall guarantee is gone before code generation. That rewrite is legal because LangRef defines the two as equivalent. llvm.memset.inline is also the precedent for adding an intrinsic whose optimizer semantics duplicate an existing one. Its Intrinsics.td entry carries byte-identical properties to int_memset and LangRef states its behavior is equivalent to llvm.memset, with a codegen guarantee as the whole of its justification. This is the same argument. The survival half of the contract comes out of the declared memory effects rather than out of changes to any pass. Claiming inaccessible memory in addition to argument memory is more pessimistic than the intrinsic really is, but a write that is not confined to argument pointees is not one dead store elimination can attribute to a single location, so it has nothing to remove, while the argument memory half keeps the write to the region itself visible to alias analysis. llvm.prefetch is pessimistic in the same way. IntrNoDuplicate keeps one clear from becoming several. The test covers the three situations dead store elimination handles with four intrinsics each: a plain memset, a volatile memset, a volatile llvm.memset.inline and llvm.zeroize. The plain memset is removed everywhere and llvm.zeroize survives everywhere; the volatile rows are carried so the file does not read as claiming a distinction it does not show, and the memset.inline rows pin the SROA rewrite. The lowering guarantee is not an IR-level property and is tested with the lowering change rather than here. Nothing lowers the intrinsic yet; expanding it for a target is left to a later change. The name is a recommendation still awaiting sign-off on trailofbits/vspells-ct-internal-notes#64.
f01c5be to
00269dd
Compare
| : DefaultAttrsIntrinsic<[], | ||
| [llvm_anyptr_ty, // Destination. | ||
| llvm_anyint_ty], // Number of bytes to clear. | ||
| [IntrWriteMem, IntrInaccessibleMemOrArgMemOnly, IntrNoDuplicate, |
There was a problem hiding this comment.
Why do we need IntrNoDuplicate here? Does it actually prevent it from inlining?
There was a problem hiding this comment.
You're right — it should be removed, and the inlining half is the reason.
noduplicate does affect inlining: InlineCost.cpp:2475 sets ContainsNoDuplicateCall for any noduplicate call in the callee and InlineCost.cpp:3037 then hard-fails the inline unless the call is the sole live use of a local function (InlineCost.cpp:1250), so any function containing an llvm.zeroize becomes uninlinable — LangRef.md:2384 and the in-tree note on llvm.codeview.annotation at Intrinsics.td:1737 ("This is expensive, as it disables inlining") both say so, and it reproduces: an internal clear wrapper with two call sites is refused with never inlined (cost=never): noduplicate, while the same wrapper around a volatile llvm.memset.inline inlines. That is backwards for this intrinsic in particular, whose stated purpose is that the clear establishes no call frame — noduplicate guarantees a frame around it whenever the clear sits in a helper.
As for why we need it: we don't. The justification I wrote (Intrinsics.td:1216, "IntrNoDuplicate keeps a single clear from becoming several") states the mechanism and never says why several clears would be harmful, and they aren't — zeroing is idempotent, each copy clears its own region, so duplication costs a store and never weakens the guarantee. Nothing else leans on it either: non-removability comes entirely from the declared memory effects, and DeadStoreElimination.cpp never consults cannotDuplicate. Every in-tree IntrNoDuplicate user has execute-exactly-once semantics (codeview.annotation emits a label, coro.suspend.async is a suspend point, the hardware-loop family sets a counter register); no store-like intrinsic carries it, volatile memset included.
So the IntrNoDuplicate property, the Intrinsics.td comment sentence about it, and the LangRef line "Calls also carry noduplicate, so one call may not be turned into several" should all come out. You've been pushing to this branch, so it's yours to take if you want it — say the word and I'll do it instead.
Generated by Claude Code
There was a problem hiding this comment.
Done — IntrNoDuplicate is out of the int_zeroize definition, along with the comment sentence that justified it and the matching noduplicate line in the LangRef entry; the memory effects that carry non-removability are untouched, so the declaration now reads nocallback nofree nosync nounwind willreturn memory(argmem: write, inaccessiblemem: write). Confirmed the behaviour actually changed: an internal clear wrapper with two call sites, which opt -passes=inline previously refused at both with cost=never: noduplicate, now inlines at both, and llvm/test/Transforms/DeadStoreElimination/zeroize.ll still passes on both of its RUN lines.
One housekeeping note: this went in as a second commit rather than an amend, so the branch now carries two and the first one's message still mentions the property — the pair wants squashing, or that message tidying, whenever this lands.
Generated by Claude Code
The property defeats the intrinsic it was meant to protect. InlineCost refuses a call site whose callee contains a noduplicate call unless that site is the sole call to a local function, where inlining deletes the original and duplicates nothing. Every other site fails outright, with cost=never and noduplicate as the reason. So a small internal helper that wraps a clear and is called from more than one place, which is the natural way to reach for this intrinsic, is refused at every one of its call sites, and the clear ends up pinning open any caller it appears in. An intrinsic whose purpose is to be dropped wherever a buffer needs erasing cannot also be a barrier to inlining the code that erases it. The justification did not hold either. Clearing a region is idempotent: performing it twice leaves the region in the state one pass would have left it, so there is nothing a duplicated clear can undo and no reason to keep the call sites distinct. The half of the contract that does matter, that the write survives even where the region is provably never read again, does not rest on this property and is unchanged. It comes from the declared memory effects, and the dead store elimination test continues to pin it.
|
Two problems in the LangRef entry, and one in the declared memory effects. The entry says the intrinsic's behavior "is equivalent to the behavior of Second, On memory effects: Detail, and the rest of the findings on this patch, in the write-up: https://claude.ai/code/artifact/accb9215-36b5-4022-ae0f-249ee54e99cb Generated by Claude Code |
| ##### Arguments: | ||
|
|
||
| The first argument is a pointer to the memory to clear, and the second is an | ||
| integer specifying the number of bytes to clear. | ||
|
|
||
| The {ref}`align <attr_align>` parameter attribute can be provided for the | ||
| first argument. | ||
|
|
||
| ##### Semantics: | ||
|
|
||
| The '`llvm.zeroize.*`' intrinsics set `<len>` bytes of memory starting at the | ||
| destination location to zero. | ||
|
|
||
| If `<len>` is 0, it is a no-op modulo the behavior of attributes attached to | ||
| the arguments. | ||
| If `<len>` is not a well-defined value, the behavior is undefined. | ||
| If `<len>` is not zero, `<dest>` should be well-defined, otherwise the | ||
| behavior is undefined. | ||
|
|
||
| The behavior of '`llvm.zeroize.*`' is equivalent to the behavior of | ||
| '`llvm.memset.*`' with a value of zero and `isvolatile` set to `true`, but the | ||
| generated code is guaranteed not to call any external functions and not to | ||
| establish a call frame in order to perform the clear. That guarantee is the | ||
| property the intrinsic exists for: a clear lowered through `memset` spills the | ||
| values live across it and enlarges the frame, leaving in the very stack and | ||
| registers being cleaned up the residue the clear was meant to remove. | ||
| '`llvm.memset.inline.*`' provides the no-external-call half of it, but only | ||
| until a transform rewrites it into a plain '`llvm.memset.*`', which is permitted | ||
| precisely because those two are equivalent. | ||
|
|
||
| Performing the write is itself the effect the call is there for, rather than a | ||
| means of setting up a value some later read is meant to observe, so the usual | ||
| grounds for discarding a store do not apply to it. A transform may not delete a | ||
| call to '`llvm.zeroize.*`', narrow the region it covers, or replace the bytes it | ||
| writes with any other value, and in particular may not do so because a later | ||
| store overwrites the same bytes, because the object's lifetime ends, or because | ||
| the object is unreachable from the rest of the program: the region being | ||
| provably dead afterwards is the expected case here, not a reason to drop the | ||
| write. | ||
|
|
||
| A volatile '`llvm.memset.*`' is not removable either, so this part of the | ||
| contract is not what separates the two; it is stated because a clearing | ||
| intrinsic that could be discarded would be useless, not because it is the | ||
| reason the intrinsic exists. | ||
|
|
||
| A transform may move a call to '`llvm.zeroize.*`' to a later point on the same | ||
| control-flow paths when nothing in between can read the region, and may combine | ||
| two calls covering adjacent or overlapping regions into one call covering their | ||
| union. | ||
|
|
||
| The clearing sequence a given target emits is specified separately; the | ||
| guarantee that it calls no external function and establishes no call frame holds | ||
| for every target. |
There was a problem hiding this comment.
seems a bit long. Should we make this more concise?
There was a problem hiding this comment.
Trimmed — the Semantics section is now about half its previous length: out go the rationale for why a clear shouldn't be lowered through memset, the comparison with llvm.memset.inline, the explanation of why the usual grounds for discarding a store don't apply here, and the aside noting that a volatile llvm.memset.* isn't removable either, with every normative sentence kept verbatim (it's a new commit on top, so the branch is three commits now and wants squashing when it lands). I also dropped the sentence claiming the no-external-call, no-call-frame guarantee "holds for every target" — no target lowers llvm.zeroize yet, so nothing backs it, and it stays wrong for AArch64, RISC-V, PowerPC and WebAssembly even once the stacked X86 expansion is in. One thing I deliberately left alone for you to settle: the entry still says the behaviour is equivalent to llvm.memset.* with isvolatile set to true, and further down still permits sinking a call to a later point and merging two calls over adjacent or overlapping regions — volatile forbids both, since the Volatile Memory Accesses section says optimizers must not change the number of volatile operations or their order relative to other volatile operations. Which should go, the volatile-memset equivalence or the sink/merge permission?
Generated by Claude Code
The entry argued for the intrinsic instead of specifying it. Drop the rationale for why a clear must not go through memset, the comparison with llvm.memset.inline, the explanation of why the usual grounds for discarding a store do not apply, and the aside noting that a volatile llvm.memset.* is not removable either. None of it constrains an implementation or a transform; the normative statements it surrounded are kept verbatim. Also drop the claim that the no-external-call, no-call-frame guarantee "holds for every target". No target lowers llvm.zeroize yet, so nothing backs the claim, and it stays wrong for the targets the stacked X86 expansion does not cover. Documentation only; Intrinsics.td and the tests are unchanged.
5 participants
Requested by Francesco Bertolaccini · Slack thread
Add
llvm.zeroize, an overloaded pointer-and-length intrinsic that writes zeroover
[dest, dest + len)and guarantees the clear reaches the machine withoutcalling an external function and without establishing a call frame.
Why an intrinsic
The guarantee that matters here is a codegen guarantee. Every other way of
clearing a buffer goes through
memset, and amemsetover a dynamic length —the shape that ships, since
__builtin_zeroize_memory(p, n)takes a dynamicn— or over any large fixed length lowers tocallq memset@PLT. Measured onx86-64 with six values live across the clear:
memsetllvm.zeroizecallq memset@PLTmovb $0, %al; rep;stosb%rspA clear that spills the live state it is trying to erase into the stack it is
trying to erase, and then opens a fresh callee frame underneath the pointer it
was handed, manufactures exactly the residue class this feature exists to
remove. The stacked lowering change expands the intrinsic as a pseudo after
register allocation, so there is no libcall at any size.
Why not a volatile memset
An earlier revision of this description led with non-removability. That was
wrong, and @kumarak was right to push on it: a volatile
llvm.memset(i1 true) is not removable either.DSEState::isRemovablerefuses to removevolatile memory intrinsics unconditionally
(
llvm/lib/Transforms/Scalar/DeadStoreElimination.cpp:1475-1477), so a volatilememset survives
-passes=dse, survivesdefault<O2>, and survives the backend,in all three shapes this PR's test covers.
Non-removability is still part of the intrinsic's contract — a clearing
intrinsic that could be discarded would be useless — but it is a consequence of
the contract rather than the reason for it, and this PR no longer argues
otherwise. What a volatile memset cannot do is avoid the libcall above.
Why not llvm.memset.inline
llvm.memset.inlinesurvives dead store elimination and lowers without alibcall, so it looks like it closes the gap. It does not: SROA rewrites it back
into a plain
llvm.memsetatllvm/lib/Transforms/Scalar/SROA.cpp:3686, onexactly the non-escaping sensitive stack buffers this feature targets, so the
no-libcall guarantee is lost before code generation sees it. That rewrite is
legal precisely because LangRef defines the two as equivalent. The test pins the
rewrite so the argument does not have to be taken on trust.
On adding a second intrinsic with an overlapping job
llvm.memset.inlineis also the governing precedent for the objection itself.llvm/include/llvm/IR/Intrinsics.td:1185gives it byte-identical properties toint_memset, and LangRef states its behavior is equivalent tollvm.memset,with a codegen guarantee as the whole of its justification. Upstream accepted a
second intrinsic with identical optimizer semantics on that basis alone.
llvm.zeroizeis the same argument with a stronger guarantee: no external calland no call frame, and not rewritable into the weaker form.
Implementation
The non-removability half of the contract comes out of the intrinsic's declared
memory effects rather than out of changes to any pass. Claiming inaccessible
memory in addition to argument memory is more pessimistic than the intrinsic
really is, but a write that is not confined to argument pointees is not one
MemoryLocation::getForDestcan reduce to a single location, so dead storeelimination has nothing to remove; the argument memory half keeps the write to
the region itself visible to alias analysis, so the intrinsic does not become a
general optimization barrier.
llvm.prefetchis deliberately pessimistic in thesame way and carries a comment saying so, and
llvm.sideeffectuses theinaccessible memory half alone and therefore writes nothing.
IntrNoDuplicatekeeps one clear from becoming several. Overloading on both the pointer and the
length gives mangled names like
llvm.zeroize.p0.i64.LangRef now leads with the clearing contract and the codegen guarantee, states
the equivalence to a zero-valued volatile
llvm.memsetin the same formllvm.memset.inlineuses, and derives non-removability from the contract ratherthan presenting it as the point. It also records what stays permitted — moving
the call to a later point on the same control-flow paths when nothing in between
can read the region, and merging calls over adjacent or overlapping regions — so
the guarantee is not read as a blanket barrier.
Test
llvm/test/Transforms/DeadStoreElimination/zeroize.llcovers the threesituations dead store elimination handles — dead at the end of a function, fully
overwritten by a later write, dead at the end of the object's lifetime — with
four intrinsics in each: a plain
memset, a volatilememset, a volatilellvm.memset.inline, andllvm.zeroize. The plain memset is removedeverywhere;
llvm.zeroizesurvives everywhere, under-passes=dsefor thenarrow claim and under
default<O2>for the full pipeline.The volatile rows are carried deliberately: they show that at the IR level
non-removability does not distinguish
llvm.zeroizefrom a volatile memoryintrinsic, and a comment in the file says so, so the test is not read as pinning
a distinction it does not make. The
memset_inlinerows pin the SROA rewrite onthe alloca cases while showing it does not fire on a plain pointer argument. The
lowering guarantee is not an IR-level property and is tested with the lowering
change rather than here.
Notes
Nothing lowers the intrinsic yet. Expansion in the backend is deliberately left
to trailofbits/vspells-ct-internal-notes#11, and LangRef says the target
sequence is specified separately; IR containing the intrinsic will fail to
select if it reaches code generation before then.
The name and the argument shape are recommendations still awaiting sign-off on
trailofbits/vspells-ct-internal-notes#64; the internal draft signature was
llvm.zeroize.clear.p0. A rename touches onedefplus the test and the docreferences.
Stacked on
zeroize-stack-attributeand targets that branch, notenforced_secrecy_main. Closes trailofbits/vspells-ct-internal-notes#10.AI tool use
This pull request contains AI-generated content. It was prepared with the assistance of Claude Code; the contributor has reviewed the generated code and text, is the author of the contribution, and is accountable for it, per the LLVM AI Tool Use Policy.
Generated by Claude Code