Add Mach-O support - #50
Conversation
377e742 to
05fa039
Compare
LibObjectFile has no Mach-O test inputs. Adds committed fixtures covering the variants an implementation has to handle: 32- and 64-bit, LC_UNIXTHREAD and LC_MAIN entry points, dyld info opcode streams and chained fixups, a dylib, an object file with relocations, and a universal binary. Both 32-bit fixtures are synthesized with yaml2obj because LLVM and current cctools have dropped 32-bit Mach-O linking, so neither the LC_UNIXTHREAD nor the LC_MAIN-with-dyld-info shape can come from a linker any more. The sources call into libc so the linked fixtures carry lazy-binding stubs and an indirect symbol table, and the fixtures keep the padding after the load commands that in-place injection consumes. - src/LibObjectFile.Tests/MachO/generate_files.sh: regenerates the fixtures via OSXCross cctools and LLVM, and links to where OSXCross comes from - src/LibObjectFile.Tests/MachO/unixthread_i386_rpath: the same input after install_name_tool adds a runpath, used as an encoding reference - src/LibObjectFile.Tests/LibObjectFile.Tests.csproj: copies the fixtures to the test output
Nothing in the library described the Mach-O format, so there was no vocabulary to write a reader against. Adds the load command, CPU, file type, header flag, segment, section, platform and relocation type enumerations, the packed version helpers, and the blittable structures matching the on-disk layout. The structures are hand-written rather than generated from Apple's headers, whose licence this project cannot bundle, which follows what the PE support already does. A test pins every structure size, because the fields are copied by value and a wrong size would shift every later field rather than fail outright. - src/LibObjectFile/MachO/MachOLoadCommandType.cs: keeps the LC_REQ_DYLD high bit as part of the stored value - src/LibObjectFile/MachO/MachORelocation.cs: the two forms of entry, told apart by the top bit of the first word - src/LibObjectFile/MachO/Internal: the on-disk structures, including the always big-endian universal binary header
The format constants had nothing to build on, so a Mach-O image could not be turned into anything inspectable. Adds the file, segment, section and load command model, and a reader that walks the command table and decodes every command an image of these architectures carries. Anything unrecognised is kept as raw bytes, so images from a newer linker still load. Everything in the file becomes an ordered content list: the header, the load command table, the padding after it, the bytes of each section, each table in __LINKEDIT, and the gaps between them. Every byte belongs to an element, which is what makes writing the list back reproduce the image and a layout a single walk over it. Content carrying an address is pinned, because a section's address is its segment's address plus its distance from the segment's file offset, so moving it in the file would move it in memory. Padding is kept as the bytes that were read rather than regenerated, since a linker pads executable sections with nop and zero-filling would leave a different instruction somewhere reachable. - src/LibObjectFile/MachO/MachOFile.Read.cs: turns the file into content, leaving nothing implicit - src/LibObjectFile/MachO/Content: the element types, and which of them a layout may move - src/LibObjectFile/MachO/MachOPathLoadCommand.cs: keeps the linker's own padding on the commands carrying a string
Reading an image was of no use without being able to write one back. Adds the write path, which lays the content out and then writes each element at its position. Only the header, the load command table and the padding after it are placed by that layout; everything else keeps the position recorded for it, because moving content in a Mach-O moves the addresses that refer to it. The padding is what absorbs a command table that has grown, so writing fails when the commands no longer fit rather than moving content and invalidating the image. - src/LibObjectFile/MachO/MachOFile.Write.cs: writes the content list - src/LibObjectFile.Tests/MachO/MachOSimpleTests.cs: byte-exact round-trip per fixture, and every recorded file offset being reachable
Every other backend implements Verify, twenty-four types between ELF and PE, and Mach-O implemented none of it, so nothing checked the invariant the format rests on: a section's address is its segment's address plus its distance from the segment's file offset. Break that and the loader maps a section somewhere other than where the code expects, which no round-trip test would notice because the bytes still match. Checks that relationship, that sections stay inside their segment, that a section header agrees with the content holding its bytes, that the content list covers the file with no gap or overlap, and that load command sizes are walkable by dyld. Seventy-five real images pass it. - src/LibObjectFile/MachO/MachOFile.Verify.cs: the checks
There was no way to add a dependency or a runpath to an existing Mach-O image, which is what makes a shipped binary load a library it was not linked against. Adds the operations install_name_tool offers, appending commands into the padding the linker left ahead of the first section so no content moves and every address in the image stays valid. A dependency is appended rather than inserted, because dyld identifies a library by its position among the load commands and the symbol table binds against that number. When the padding runs out the edit throws and names the shortfall, since the alternative is moving content and invalidating the image. Removing a dependency is deliberately not offered for the same numbering reason, matching what install_name_tool exposes. - src/LibObjectFile/MachO/MachOFile.Edit.cs: the add, change and remove operations and the space check - src/LibObjectFile.Tests/MachO/MachOEditingTests.cs: compares against the install_name_tool reference fixture, and asserts no section moves
Apple Silicon refuses to execute an unsigned image, and any edit invalidates an existing signature, so an arm64 binary could be modified by this library only to become unrunnable. Adds ad-hoc signing: a code directory of SHA-256 page digests plus an empty requirement set, appended to __LINKEDIT with the load command added when the image was previously unsigned. The signature covers exactly the bytes preceding it, so its size is derived from the identifier and the signed length before any digest is taken, and the image is laid out once and then hashed as it will finally be written. Signing has to be the last thing done, since editing afterwards leaves the digests covering bytes that are no longer there. The editing operations record that, and writing fails while it is set, so the library cannot hand back a file that looks signed and would be refused at execution. - src/LibObjectFile/MachO/CodeSign/MachOAdHocSignatureBuilder.cs: builds the superblob, which is big-endian throughout unlike the rest of the format - src/LibObjectFile/MachO/MachOFile.Sign.cs: appends the signature as content and grows __LINKEDIT to cover it - src/LibObjectFile.Tests/MachO/MachOSigningTests.cs: recomputes every page digest from the written image, which is the property the kernel checks
The symbol table, the indirect symbol table and the relocations of a section were bytes that nothing decoded, so there was no way to see what an image defines, imports or fixes up. Adds reading for all three, resolving symbol names against the string table and separating the four things a symbol's type byte packs together. An object file keeps its tables past the end of its only segment, so the lookup covers that as well as a linked image's __LINKEDIT. The result is a decoded snapshot rather than a live view, and the documentation says so, because resizing a table would move everything after it. Relocations come in two forms sharing eight bytes, told apart by the top bit of the first word. Encoding is implemented alongside decoding and tested to round-trip, because a field read from the wrong bit still reads back consistently on its own and would otherwise look correct. - src/LibObjectFile/MachO/MachOSymbol.cs: separates the debug, external and kind bits of the type byte - src/LibObjectFile/MachO/MachOFile.Symbols.cs: reads both symbol tables, from a segment or from the bytes past one - src/LibObjectFile/MachO/MachOFile.Relocations.cs: reads the entries a section points at
A universal binary could not be read at all, so an image shipping for both Intel and Apple Silicon was out of reach even though each slice inside it was already readable. Adds reading and writing of the container, exposing one image per architecture. The header and its slice table are big-endian whatever the architectures inside are, which is the one place the format departs from the image's own byte order, so they are read through explicit big-endian primitives rather than by copying a structure. Each slice is read through a view bounded to it, so a malformed slice cannot reach into its neighbours. The slices are laid out before writing, so one that changed size since it was read is placed and recorded correctly rather than overrunning the one after it. The space between them is left zeroed: every universal binary examined pads with zeros, and slices are page aligned rather than packed. - src/LibObjectFile/MachO/MachOFatFile.cs: the container, its slice table and the bounded reads - src/LibObjectFile/MachO/MachOFatSlice.cs: one architecture's placement and image
A decoded image could only be inspected through a debugger. Adds printing of the header and every load command, and snapshots the result for each fixture, so decoding a new command extends the snapshot rather than needing another test and a change to how anything is read shows up as a diff. Commands are named by their LC_ spelling rather than the enumeration's, so output can be read next to otool's. That mapping is written out rather than derived from the enumeration names, because several do not follow from them: LC_UNIXTHREAD is one word and LC_VERSION_MIN_MACOSX breaks in places the casing does not. The command sequence each fixture prints was checked against otool before the snapshots were taken. - src/LibObjectFile/MachO/MachOPrinter.cs: the printer - src/LibObjectFile.Tests/Verified: one snapshot per fixture
The readme listed Mach-O among the formats left for contributors, and the manual did not mention it. Adds it to both, following how the other formats are covered: a feature list in the readme and a section in the manual with an overview, reading, writing, editing and signing. The overview states the relationship a caller has to know about, that a section's address is its segment's address plus the section's distance from the segment's file offset, since that is what decides which parts of an image a layout may move and why adding a load command is bounded by the padding the linker left. - readme.md: Mach-O added to the supported formats and dropped from the longer term plan - doc/readme.md: the manual section
7866dbf to
1b9d7cf
Compare
The signing section said an ad-hoc signature is what lets an edited arm64 image run, which is true of the kernel and not of Gatekeeper. A reader could reasonably take it to mean a signed bundle will launch, and then find that a downloaded one is refused whatever this library did to it. Says what an ad-hoc signature does not buy: no Developer ID, no notarization, and no way back to notarized once it is replaced. Also that signing one image is not sealing a bundle, and which of the two is at stake depends on what was edited. A bundle's main executable is not listed in the seal, so re-signing it leaves the seal correct, while nested code is listed by hash and editing it does not. - doc/readme.md: Gatekeeper, and sealing a bundle, under code signing
0fba7a2 to
749245d
Compare
|
This is ready for review, I tried my best to keep tests and the API shape consistent with the ELF support that exists. I also re-wrote the history to hopefully make this easier to review commit by commit. This code should make it's debut in https://github.com/LaneDibello/Kotor-Patch-Manager when I have the chance to start working on integrating everything there 😄 Let me know if anything looks off to you. |
|
Thanks a lot for this, that's very cool! I'm going to push some comments made by my AI coding agent. |
There was a problem hiding this comment.
Focused architecture and correctness review; inline comments below.
Reviewed by GPT 5.6 Sol/High with the CodeAlta harness.
|
Thanks! Will take a look at patching up these holes. |
The header's sizeofcmds was ignored and each command was bounded only by the whole stream, so a command declaring a cmdsize too small for its own fields still read them, taking bytes that belong to whatever follows. A count-bearing command could do the same: a segment claiming more sections than fit, or a build version claiming more tools, read section headers and tool entries out of the commands after it. The table is now bounded by sizeofcmds and has to end exactly there, each command is checked against the size its fixed part needs before it is read, and a command that reads past its own cmdsize is reported rather than trusted. The counts inside segments and build versions are checked against the space the command declares. Also states what file offset remapping covers, since the contract claimed every recorded offset and did not include a section's relocations. Those are now mapped. Placement is deliberately excluded and says why: a segment's or a section's file offset is where something is mapped, not merely stored, so moving it moves the thing in memory. The remapping test read offsets back through the same walk it was testing, which made an omitted field invisible to it. It now reads them off the commands directly, and fails if the walk misses one. - src/LibObjectFile/MachO/MachOFile.Read.cs: the bounded walk - src/LibObjectFile/MachO/MachOLoadCommand.cs: the smallest cmdsize each kind can legally have - src/LibObjectFile/MachO/MachOSegment.cs: section counts bounded, section relocations remapped
Writing did not verify and did not flush, so a model that Verify rejects could still be emitted, and a buffered stream could be left holding the tail of the file. ELF and PE both run Verify, then the layout, then the write, then flush. TryWrite now does the same, and the universal binary container flushes once its slices are written. Wiring Verify in exposed a hole signing was leaving. Aligning the signature to sixteen bytes can leave a gap before it, and nothing occupied that gap, so the content list no longer covered every byte of the file. The gap is now content like any other rather than a hole the writer happened to skip over. - src/LibObjectFile/MachO/MachOFile.Write.cs: verify, lay out, write, flush - src/LibObjectFile/MachO/MachOFile.Sign.cs: the alignment gap before a signature is content - src/LibObjectFile/MachO/MachOFatFile.cs: flush after the slices
The edits assigned before they checked. ChangeDylibName and SetInstallName wrote the new name and only then asked whether it fits, so an edit that did not fit left the command renamed but not resized, describing a string longer than it has room for. AppendCommand marked the signature stale before finding out the command would not fit, so a failed edit left the image unable to be written until it was signed again. Each now works out what the change costs, checks there is room, and only then makes it, so a failure leaves the image exactly as it was. - src/LibObjectFile/MachO/MachOFile.Edit.cs: validate, then mutate - src/LibObjectFile.Tests/MachO/MachOEditingTests.cs: a failed rename leaves names, sizes and the written bytes untouched
The empty CMS wrapper looked like it might not belong, since the linker-produced fixture here carries only a code directory. A linker writes a lone code directory flagged LINKER_SIGNED, which is a different thing from what signing a finished image produces: codesign writes a code directory, a twelve byte empty requirement set and an eight byte empty CMS wrapper, which is what a shipping ad-hoc signed dylib contains and what this writes. Says so where the wrapper is written, and checks the sizes of all three blobs rather than only their count. - src/LibObjectFile/MachO/CodeSign/MachOAdHocSignatureBuilder.cs: why the slot is present and empty
|
Should be everything the agent brought up 😄 |
Reviewing the load command bounds turned up three more of the same kind elsewhere. A symbol, indirect symbol or relocation count was multiplied by an entry size in 32-bit arithmetic. A count large enough to wrap that product gave a small, plausible length that passed the coverage check, and the loop then read past what had been read. Those products are now computed in 64 bits, so the wrap cannot happen and an impossible length is reported instead. The symbol table count reached the caller as an ArgumentOutOfRangeException rather than a diagnostic. A universal binary header's slice count was used to size an allocation before anything checked the file was big enough to hold that many. It is now checked against the bytes actually there. Signing changed a good deal before it could fail: it removed the previous signature's content, moved the command, resized __LINKEDIT and cleared the stale flag, then wrote the image to take the digests, and that write can fail. What it changes is now put back if it does, so a failed signing leaves the image exactly as it was rather than half signed. - src/LibObjectFile/MachO/MachOFile.Symbols.cs: lengths computed in 64 bits - src/LibObjectFile/MachO/MachOFatFile.cs: the slice count checked before it sizes anything - src/LibObjectFile/MachO/MachOFile.Sign.cs: signing restores what it changed if it fails
4ea0f6e to
c612397
Compare
|
Just want to make sure that you have seen this comment |
The count checks added with the previous bounds could be walked around by the arithmetic in the checks themselves. A segment declaring 0x40000001 sections multiplies out to 124 in 32 bits, the size of the command being checked, so the check passed and the section headers were read out of the commands after it. A build version declaring 0x20000000 tools multiplies out to nothing at all. The largest count that fits is now derived by division, so there is no product to wrap. ComputeCommandSize computes wide and rejects a count that would not fit rather than returning a wrapped size, since it is public and a caller can reach it directly. The same widening was missing where the reader gathers the regions the commands point at: those sizes were counts times an entry size in 32-bit arithmetic, so a wrapped product would have named a small region and left the rest of the table looking like padding. - src/LibObjectFile/MachO/MachOSegment.cs: section count derived by division, command size computed wide - src/LibObjectFile/MachO/MachOBuildVersionCommand.cs: tool count derived by division - src/LibObjectFile/MachO/MachOFile.Read.cs: region sizes computed wide - src/LibObjectFile.Tests/MachO/MachOSimpleTests.cs: the two overflow counts, which fail against the multiplying checks
I missed that comment, it should be resolved now 😄 |
| MACHO_ERR_UnsupportedByteOrder = 5001, | ||
| MACHO_ERR_InvalidLoadCommandSize = 5002, | ||
| MACHO_ERR_TruncatedLoadCommand = 5003, | ||
| MACHO_ERR_InvalidSegmentFileRange = 5004, |
There was a problem hiding this comment.
This diagnostic id doesn't seem to be used.
|
Another agent review, distilled Bug:
|
Adds Mach-O support for i386, x86_64 and arm64: read/write with a byte-exact round-trip,
segments and sections, and every load command an image of these architectures carries.
Anything unmodelled round-trips as raw bytes, so images from a newer linker still survive
a read/write.
On top of that, the two things I actually wanted it for:
AddLoadDylib/AddRPath/ChangeDylibName/SetInstallName: basicallyinstall_name_toolas a .NET API.AdHocSign(identifier): SHA-256 code directory appended to__LINKEDIT. Apple Siliconwon't run unsigned code, so an edited arm64 binary needs re-signing to stay usable.
Since the last update the rest of the support I had planned is finished: symbol table and
relocation reading, universal binaries, verification, and an
otool -l-style printer with snapshots.The whole file is one ordered content list, but only the header, the command table
and the padding after it get placed by the layout. Everything else keeps its recorded
position, because a section's address is its segment's address plus its distance from the
segment's file offset, so moving a section in the file moves it in memory too.
__LINKEDITis the part nothing addresses that way, and that's the part a layout can actually touch.
Untouched images still come out byte-identical, which is how I checked the layout agrees
with what the linker did.
Notes
Raw structs are hand-written rather than run through
LibObjectFile.CodeGen, since the firstinput that comes to mind is Apple's headers and the licence isn't one this repo can bundle
(same approach the PE backend takes).
The fixtures are built by a committed script. The two 32-bit ones are synthesized with
yaml2objrather than linked, because LLVM and current cctools have both dropped 32-bitMach-O linking: neither the
LC_UNIXTHREADnor theLC_MAIN-with-dyld-info shape comes outof a linker any more, so producing them would take an older toolchain.