Skip to content

[Apple mobile] Stage the Helix R2R app payload outside the crossgen2 input directory - #132489

Draft
davidnguyen-tech wants to merge 10 commits into
dotnet:mainfrom
davidnguyen-tech:davidnguyen-tech-super-guide
Draft

[Apple mobile] Stage the Helix R2R app payload outside the crossgen2 input directory#132489
davidnguyen-tech wants to merge 10 commits into
dotnet:mainfrom
davidnguyen-tech:davidnguyen-tech-super-guide

Conversation

@davidnguyen-tech

@davidnguyen-tech davidnguyen-tech commented Aug 18, 2026

Copy link
Copy Markdown
Member

Fixes #131922

Problem

Apple mobile CoreCLR test apps are ReadyToRun-compiled on the Helix machine: the work item ships a publish/ directory of IL assemblies, and ProxyProjectForAOTOnHelix.proj runs crossgen2 over it (--composite --strip-il-bodies --obj-format:macho) before building the app bundle.

_AddR2RFilesToAppleBundle copied the crossgen2 output (obj/R2R/*.dll, obj/R2R/*.r2r.dylib) back over publish/ — which is also the crossgen2 input directory. That is fine as long as the AOT build runs once per work-item directory. It does not always run once: a Helix work item retry re-runs the whole work item command in the same $HELIX_WORKITEM_ROOT directory, so the second run feeds crossgen2 the images the first run produced. Compiling already-compiled, IL-stripped assemblies yields an image whose method bodies are gone, and the app aborts during startup:

Fatal error.
A method body required at runtime was stripped from the ReadyToRun image.
   at System.Collections.Generic.Dictionary`2[...]..ctor(Int32)
   at System.AppContext.Setup(Char**, Char**, Int32, System.Exception*)

This is a build-infrastructure defect — an MSBuild step that mutates its own retry inputs — not a crossgen2 compiler defect. Crossgen2 did exactly what it was asked to do; it was asked twice.

Not the Dictionary theory

The failure signature in #131922 points at Dictionary.ctor and AppContext.Setup, which reads as "crossgen2 stripped a method body it should have kept" — the theory the earlier mitigations followed (opting individual test projects out of PublishReadyToRunStripILBodies). Those are the first methods the runtime needs, so they are simply the first casualty; nothing is specific to Dictionary. The image is bad wholesale, because it was compiled twice.

Evidence from the reported failure itself

The Helix work item behind the latest hit of #131922 (build 1555007, job 22bb8241-6a3c-4036-b844-f0859313552d, work item System.Net.Security.Unit.Tests) ran twice, and both attempts uploaded an AOTBuild.binlog:

attempt 1 attempt 2
crossgen2 inputs publish/*.dll (pristine IL) publish/*.dllrewritten by attempt 1
bundled app size APP size: 42.9 Mb APP size: 19.3 Mb
result 128 test results reported; work item then failed on unrelated infra (gen-debug-dump-docs.py not found → exit 1 → retry) APP_CRASH (exit 80), stripped-IL fatal error

Attempt 1's binlog contains the copies that cause it:

Copying file from ".../obj/R2R/System.Collections.dll" to ".../publish/System.Collections.dll".
Copying file from ".../obj/R2R/System.Net.Security.Unit.Tests.r2r.dylib" to ".../publish/System.Net.Security.Unit.Tests.r2r.dylib".

and attempt 2's binlog shows crossgen2 invoked on those same publish/*.dll paths, plus the stale publish/*.r2r.dylib left behind by attempt 1.

Fix

Stop writing into the crossgen2 input directory; stage the app payload next to it.

  • _AddR2RFilesToAppleBundle now creates obj/r2r-app-payload/, copies the pristine publish/ tree into it, overlays the crossgen2 output there, and points AppleBuildDir and the AppleAssembliesToBundle / AppleNativeFilesToBundle item lists at the app-payload directory. publish/ keeps holding the original IL for every run in the work item.
  • The app-payload directory is removed and recreated on every run, so a retry cannot inherit a previous attempt's dylibs or bin-<app>-<arch> app-builder leftovers.
  • _PrepareR2RItemsOnHelix removes obj/R2R before compiling — _CreateR2RImages only tracks the composite image as an output and the per-assembly files written next to it are untracked, so an attempt killed mid-write would otherwise look up to date to the next one.
  • A warning is emitted if the crossgen2 output directory is empty, so a silently empty overlay is not mistaken for success.

Why redirecting the item lists alone is not enough

AppleAppBuilderTask receives AppDir="$(AppleBuildDir)" and enumerates it directly — Directory.GetFileSystemEntries(workspace, "", SearchOption.TopDirectoryOnly) in Xcode.cs turns every top-level entry of AppDir into a bundle resource, and Directory.GetFiles(workspace, ...)/runtimeconfig.bin lookups read from the same directory. So the staged payload has to be the AppDir; staging inside publish/ would additionally ship the staging directory inside the .app.

Moving AppDir also stops the app-builder's own writes from landing in the crossgen2 input directory: AppleBuild.targets copies libcoreclr_static.a and writes runtimeconfig.bin into $(AppleBuildDir), so before this change those files were added to publish/ on every attempt too.

Regression test

src/mono/msbuild/apple/tests/Apple.Build.Tests/AppleHelixR2RTests.cs (new) locks the behaviour in.

  • It drives the real src/mono/msbuild/apple/data/ProxyProjectForAOTOnHelix.proj with dotnet msbuild — the production file is copied into the test payload as content, so the target logic under test is the shipping one, not a copy of it.
  • It lays out a Helix-shaped work item (publish/, extraFiles/, obj/), stands in for crossgen2 with a stub props file that plants files where crossgen2 writes them, and invokes _PrepareForAppleBuildAppOnHelix;_PrepareR2RItemsOnHelix;_AddR2RFilesToAppleBundle twice in the same directory, planting leftovers from a killed attempt in obj/R2R and in the app-payload directory in between.
  • Assertions are exact SHA-256 manifests, not spot checks: publish/ must be byte-for-byte identical after both attempts; the app-payload directory must equal pristine publish + this attempt's crossgen2 output and nothing else (so the first attempt's FirstAttemptOnly.dll must be gone in attempt 2); AppleBuildDir and every Apple*ToBundle item must resolve inside the app-payload directory; nested, native, resource and extra files must land where the target claims.

Why Apple.Build.Tests

The regression is owned by the Apple MSBuild targets and has no WebAssembly dependency, so it lives in an area-owned host test project under src/mono/msbuild/apple/tests. The project shells out to dotnet msbuild and asserts on files; it requires no Apple SDK, device, browser, workload, or emscripten installation.

CI wiring

eng/Subsets.props adds the on-demand mono.applebuildtests subset. eng/pipelines/common/evaluate-default-paths.yml maps src/mono/msbuild/apple/* to applebuildtests, and runtime.yml runs that subset with -test on ordinary Linux x64 and Windows x64 hosts. Test results are published directly from the Arcade xUnit run.

The real path evaluator selects applebuildtests for nested Apple target and test changes and no longer selects wasmbuildtests for the proxy path.

Validation

Everything below was run on this branch, on macOS arm64.

  • NUGET_PACKAGES=$(pwd)/.packages ./build.sh mono+libsBuild succeeded. 0 Warning(s) 0 Error(s).
  • NUGET_PACKAGES=$(pwd)/.packages ./build.sh -s mono.applebuildtests -c Release -testBuild succeeded. 0 Warning(s) 0 Error(s); Total: 1, Errors: 0, Failed: 0, Skipped: 0.
  • Runtime build 1559636 ran the dedicated jobs on Linux x64 (test run 4293280) and Windows x64 (test run 42932898). Each published exactly one test result for R2RAppPayloadLeavesPublishPristineAndIsRebuiltOnRetry: 1 passed, 0 failed, 0 skipped.
  • The same test passes with a hostile temporary parent global.json, hostile working directory, and unset PATH, proving the child build is isolated from ambient SDK/build configuration.

Fail-before / pass-after, swapping only the proxy project the test drives:

proxy project under test result
this branch pass
pre-fix (main) failpublish/: content changed: AppleTestRunner.dll, content changed: Lib.dll, unexpected: FirstAttemptOnly.dll, unexpected: KilledAttempt.dll, unexpected: app.r2r.dylib
this branch minus RemoveDir $(_R2RAppPayloadDir) fail — app payload: unexpected: bin-previous/leftover.txt, unexpected: stale-marker.txt
this branch minus RemoveDir $(IntermediateOutputPath)R2R fail — app payload: unexpected: KilledAttempt.dll

The pre-fix failure is the real bug (attempt 1 rewrote the crossgen2 inputs), not a setup artifact, and each RemoveDir fails independently and distinctly.

Additional evidence gathered while developing the fix:

  • Real crossgen2 double-compilation, 11 assemblies from the local osx-arm64 build, --composite --strip-il-bodies: the pristine first-generation composite and the fixed second invocation are both 13,381,120 bytes and byte-identical (7c0280f83c96…), while compiling the previous output again (pre-fix behavior) collapses to 2,999,808 bytes (6f5c943cff7c…, 78% smaller) — the same shrink ratio seen between attempt 1 and attempt 2 in CI.

Note on payloads that are ReadyToRun-compiled on the build machine

One project (src/tests/FunctionalTests/iOS/Simulator/CoreCLR/iOS.CoreCLR.R2R.Test.csproj) sets PublishReadyToRun itself, so its payload arrives with a composite *.r2r.dylib already in publish/ and gets compiled again on Helix. That is harmless and unrelated to this bug: the in-tree crossgen2 targets used on the build machine never pass the composite strip argument, so the shipped assemblies still hold their IL. Measured on a real payload — build-machine publish/: 194 assemblies, 145,956 method bodies, 0 stripped; the Helix obj/R2R output of the same run: 93,063 stripped. The failure in this PR needs stripped inputs, which only a previous Helix run could produce, which is exactly what the staging change prevents.

Trade-offs, considered and declined

  • Hardlinking the pristine stage instead of copying: rejected. It re-introduces aliasing between the staged AppDir and the crossgen2 inputs — exactly the coupling this change removes — and would stay safe only as long as every writer that touches a staged file replaces it instead of truncating it in place. The cost of copying is one extra copy of publish/ per work item; the app bundle itself is already another copy.
  • A success sentinel instead of the unconditional RemoveDir obj/R2R: rejected. It would spare a clean retry a recompile, but retries are precisely the failure path this PR is about, and a sentinel is more state that can itself be stale after a kill. Recompiling from now-pristine inputs is always correct.
  • Dropping the AppleNativeFilesToBundle swap because that item has no Helix-side consumer: rejected. Leaving it pointing at publish/ while AppleBuildDir moves would make the target internally inconsistent, and the swap keeps parity with _PrepareForAppleBuildAppOnHelix.

End-to-end tvOS validation

Manual runtime-extra-platforms build 1558829 submitted Helix job 39e8abde-adc0-475a-86d2-2e43035fc57a. System.Net.Security.Unit.Tests ran three attempts in the same work-item directory. Every attempt rebuilt R2R in about 14 seconds, recreated obj/R2R and the app-payload directory (then named obj/r2r-publish), launched the app, and reached 128 tests. All three logs had zero stripped-body and APP_CRASH matches. The current obj/r2r-app-payload name is a naming-only follow-up to that validated implementation.

The work item stayed red because one certificate-context test consistently exposed a separate Apple-specific ECDsaAppleCrypto to ECDsaOpenSsl cast failure. The overall pipeline also had unrelated failures and its tvOS CoreCLR AzDO agent hit the four-hour limit after submitting the Helix job.

Limitations

  • The host regression test stubs crossgen2 output so it can run cheaply on Linux and Windows; it does not require a real Apple toolchain.

Related: #131922 (this addresses the mechanism behind the retry-only recurrences; it does not claim to close every stripped-IL report there).

Note

This PR description, the change in it, and the regression test were prepared with GitHub Copilot assistance, iterated through a multi-model review loop, and reviewed by the submitting developer.

The Helix ReadyToRun step copied the crossgen2 output over the publish
directory, but that directory is also the crossgen2 input. When the AOT
build runs twice in the same work item - a Helix work item retry re-runs
the whole command in the same directory - the second run compiles the
already compiled assemblies again, and --strip-il-bodies drops the IL of
an image that no longer has the original method bodies. The app then dies
at startup with "A method body required at runtime was stripped from the
ReadyToRun image".

Stage the app payload in obj/r2r-publish instead: copy the pristine
publish tree there, overlay the crossgen2 output on top of it, and point
AppleBuildDir and the Apple*ToBundle items at the staging directory.
AppleAppBuilder bundles every top level entry of its AppDir, so
redirecting the item lists alone is not enough, the AppDir itself has to
move out of the pristine input directory.

Also fail with a clear error if the publish directory already contains
R2R output, and remove obj/R2R before compiling so that a partially
written attempt cannot look up to date to the next one.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a Helix retry hazard in Apple mobile CoreCLR test app ReadyToRun (R2R) compilation by preventing crossgen2 outputs from being written back into the publish/ directory (which is also the crossgen2 input). Instead, it stages a pristine app payload into an intermediate directory and overlays R2R outputs there, ensuring retries recompile from original IL inputs.

Changes:

  • Introduces an obj/r2r-publish/ staging directory: copies the original publish/ payload into it, then overlays crossgen2 outputs, and builds the app bundle from the staged payload.
  • Ensures retry safety by recreating the staging directory on every run and deleting obj/R2R before recompilation.
  • Adds guardrails: warnings when R2R outputs are unexpectedly empty and a fail-fast error if stale *.r2r.dylib outputs are detected in publish/.

A payload can legitimately arrive with ReadyToRun output in it: a test
project that sets PublishReadyToRun itself keeps that setting on the build
machine, where AppleBuild.props only suppresses the default, so the
composite image is published and then shipped inside the work item. Failing
the build on the mere presence of a *.r2r.dylib would break the first
attempt of such a work item, and the guard is not what keeps the crossgen2
inputs pristine, the staging directory is. Keep the diagnostic, drop the
hard failure, and say what the consequence is.

Also make the comment on the obj/R2R cleanup match what the SDK actually
tracks: the composite image is an output of _CreateR2RImages, the per
assembly files written next to it are not.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 18, 2026 21:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Measuring the actual payload showed the check has no signal. The build
machine does not strip IL when it publishes ReadyToRun for these projects,
because the in tree crossgen2 targets never pass the composite strip
argument: an agent published payload has 145956 method bodies and zero
stripped ones, and only the Helix compilation produces stripped bodies.
So the only thing that can still put a .r2r.dylib in the publish directory
is a build machine that pre compiled the app, where compiling again is
fine, and the message claimed the opposite. The staging directory is what
keeps the crossgen2 inputs pristine.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 18, 2026 23:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

The Apple Helix ReadyToRun proxy build had no test surface, so the retry
bug it just got fixed for could come back unnoticed. Cover it from
Wasm.Build.Tests, the only in-tree suite that already runs MSBuild driven
build tests on a CI build machine.

The test ships the real ProxyProjectForAOTOnHelix.proj into the test
payload and drives it with `dotnet msbuild`, so the production target
logic is what is under test rather than a copy of it. It lays out a Helix
shaped work item, stands in for crossgen2 with a stub props file, and runs
the R2R targets twice in the same directory with leftovers from a killed
attempt planted in between. Exact manifests assert that the publish
directory stays byte for byte identical across both runs, that the staging
directory is rebuilt from scratch, and that AppleBuildDir and the
Apple*ToBundle item lists resolve inside the staging directory.

Wire src/mono/msbuild/apple/* into the wasmbuildtests path set so a change
to the proxy project actually runs the test on PR CI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 19, 2026 00:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/mono/wasm/Wasm.Build.Tests/AppleHelixR2RTests.cs:356

  • ReplaceDirectoryContent uses TryDeleteDirectory, which swallows IOException/UnauthorizedAccessException. Since this method is intended to ensure the crossgen output directory is clean between attempts, suppressing delete failures can leave stale files behind and make the test nondeterministic (or validate the wrong behavior). It’s better to delete the directory deterministically (or fail) and then recreate/populate it.
    private static void ReplaceDirectoryContent(string directory, (string RelativePath, string Content)[] files)
    {
        TryDeleteDirectory(directory);
        foreach ((string relativePath, string content) in files)
            WriteFile(Path.Combine(directory, ToNativePath(relativePath)), content);

The run script builds SDK_FOR_WORKLOAD_TESTING_PATH from $(dirname $0), so
it can be a relative path, and the child msbuild runs with a different
working directory. Resolve the host to a full path the way BuildEnvironment
already does, and point DOTNET_ROOT, DOTNET_INSTALL_DIR and PATH at it for
the same reason that file gives: the repo build environment sets them to its
own dotnet.

Narrow the path trigger to src/mono/msbuild/apple/data/*. The test only ever
loads the proxy project from data/, and the AppleBuild props and targets next
to it are shielded by the stub Directory.Build files, so triggering the whole
browser wasm test matrix on a change to those would buy no coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 19, 2026 01:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

The previous commit prepended the resolved host directory to PATH by
interpolating the parent value, which yields a trailing separator when the
parent has no PATH at all. An empty PATH entry means the working directory on
unix, and the child runs with the work item publish directory as its working
directory, so keep the separator out in that case.

Also say in the path trigger comment which files under apple/data the test
actually covers: it replaces the Directory.Build files with stubs, so a change
to those runs the test without exercising them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 19, 2026 02:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/mono/wasm/Wasm.Build.Tests/AppleHelixR2RTests.cs:378

  • ReplaceDirectoryContent deletes the directory via TryDeleteDirectory, which silently swallows IOException/UnauthorizedAccessException. In this method that behavior can hide real test setup failures and leave stale files behind (e.g., if deletion fails on Windows due to file locks), potentially making the test flaky or misleading. Prefer a deterministic delete that fails the test when the directory cannot be cleared; keep TryDeleteDirectory only for best-effort cleanup in the finally path.
    private static void ReplaceDirectoryContent(string directory, (string RelativePath, string Content)[] files)
    {
        TryDeleteDirectory(directory);
        foreach ((string relativePath, string content) in files)
            WriteFile(Path.Combine(directory, ToNativePath(relativePath)), content);

@davidnguyen-tech

Copy link
Copy Markdown
Member Author

/azp run runtime-extra-platforms

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

davidnguyen-tech and others added 3 commits August 19, 2026 19:16
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c120fa18-835a-418d-ae4c-d5888e699e7c
Directory.Build.props already reroutes the props half of the chain to the
repo defaults, but the targets half still resolved to
src/mono/Directory.Build.targets, so only half of the src/mono conventions
were skipped. Add the matching Directory.Build.targets, the way src/tools and
Wasm.Build.Tests pair theirs, and correct the comment: Wasm.Build.Tests skips
the src/mono conventions too, but redirects to the src/libraries ones.

Also finish the r2r-app-payload rename in the test: the remaining "stage"
nouns were left over from _R2RStagingDir.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c120fa18-835a-418d-ae4c-d5888e699e7c
The work item layout is created under the system temp directory and the SDK
resolver walks up from the working directory looking for a global.json, which
is a separate lookup from the Directory.Build one the layout already shields.
A global.json anywhere above the temp directory pins the child to an SDK
version the resolved host does not have, and the child then fails with exit
code 155 before MSBuild starts, which no amount of host resolution can fix.
Write an empty global.json into the work item root, the way Wasm.Build.Tests
shields its own child builds.

Also drop the ILAssembler.Tests reference from the Directory.Build.props
comment: it has no Directory.Build files of its own and simply has no
intermediate conventions to skip, so it is not the same mechanism.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c120fa18-835a-418d-ae4c-d5888e699e7c
Copilot AI review requested due to automatic review settings August 19, 2026 18:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 20, 2026 05:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[tvOS CoreCLR] System.Net.Security.Unit.Tests crashes because required Dictionary IL was stripped from R2R image

2 participants