[Sam Gammon (sgammon)]

Summary

This PR ports Shenandoah GC support for Native Image to the 25.5 release line (elide-25.5.x), for gvm-elide-25.5.5-elide.21. It supersedes #167 for this line. #167 stays open.

Selecting Shenandoah is opt-in with --gc=shenandoah. The default satb mode and the generational and passive modes are chosen at build time with -H:ShenandoahGCMode. The native GC libraries come from the svmgc component, which is built from labs-openjdk's substratevm-gcs suite. Images that use the serial or epsilon GC do not change.

Provenance

• Volker Simonis' Shenandoah implementation: simonis/GR-70306-upstream, pinned at 81cf4340316 and mirrored as elide/simonis/GR-70306-upstream. It sits on upstream master 7b7825ae92c and includes merges of graal-25.4.4.1.1. The non-merge commits are cherry-picked in order with -x, so the authorship of Volker and Christian Haeubl is preserved.
• svm: enable Shenandoah on macOS/AArch64 #167 (16e54ebd709): macOS/AArch64, libc++ linkage, the Linux-only physical memory provider, mx shenandoah-gc-stress, and the docs (substratevm/docs/shenandoah-{linux,macos}.md). Cherry-picked with -x.
• Native libraries: elide-dev/labs-openjdk feat/shenandoah-25.5 (f9e39e7add0), opened as Shenandoah native libraries for the GraalVM 25.5 line (jvmci-25.5-b24) labs-openjdk#3. It contains:
  • Volker's simonis/GR-77360-GenShen at d3b900812425;
  • Enable Shenandoah native libraries on macOS/AArch64 labs-openjdk#1 (8eddbda3939, Darwin/AArch64);
  • a merge of the LabsJDK tag jvmci-25.5-b24 (9a714e4a13b1), which is the JDK this line builds with. The only reduced source it touches is jvmci_globals.hpp.
• Header check: shenandoahGCStructs.h and sharedGCStructs.h match the labs-openjdk exports apart from comments and whitespace.

Commits

Picked unchanged or with mechanical conflict resolution:

• e68c3da2a25 Shenandoah-specific changes (Christian Haeubl)
• e971a1e0043 (its mx_sdk_vm_impl.py hunk is already in the fork)
• af4bcca4572, 5ee0b808f30, e05263802e2, 3d2b1049e71, b3e3c2352a8, d109836ac69, b38419bd75a, c227ea23a5c, d1dc0b6d41d, 79f6754f274, 5a9d37caa78, 81cf4340316
• 16e54ebd709 (svm: enable Shenandoah on macOS/AArch64 #167)

Skipped because the fork already has them by content:

• 5de209bcb57: the native GC infrastructure (upstream G1 work) is already in the fork.
• c0233ae84ba and 05ecda1dd5e: downports of 8b18247b and 3c242ec3, both ancestors of elide-25.5.x.
• 04fccd87b96 (GR-71991): in the fork as ccd9fb31c3e.
• ba1f5de24ab ("Restore mx_substratevm.py after merge"): net zero against upstream.
• bd90b5098b7 (auto-vectorization workaround): reverted later in 15af2e6295a. The fork already has "Query object vectorization support via the BarrierSet".
• The graal-25.4.4.1.1 merge parents (javadoc CI, field metadata types) and the merge commits themselves.

Adapted:

• 15af2e6295a ("cleanups and fixes after rebase"): the Shenandoah-only files take their state at 15af2e6295a. That state includes the API adaptations from the two merges (6663e98e6ed, bce2cb535f7), which can't be cherry-picked. The commit message records this.
• Conflict resolutions keep the fork's changes and add Shenandoah's:
  • SubstrateOptions: SpawnIsolates was removed upstream.
  • NativeGCOptions: Volker's final message.
  • SubstrateGCOptions: VerboseGCLevel moved to com.oracle.svm.guest.staging.
  • Substrate{AMD64,AArch64}Backend: kept the fork's PLT/GOT comment and the SubstrateUtil.HOSTED guard.
  • VMOperationControl: added next to the fork's isOkayToBlock().
  • ShenandoahBarrierSet: kept upstream's isInHeap(location) check instead of hasObjectBase, as Volker's final state does.
  • docs/README.md: the fork's README plus the Shenandoah section.
  • mx_substratevm.py: kept the Crema benchmark command and added shenandoah-gc-stress.
• Runtime string options: the runtime-option buffer also passes string values to the GC (for example -R:VerboseGCLevel=...). This was part of af4bcca4572, but a merge on Volker's branch dropped it.
• Not taken from the merges: a duplicated Javadoc paragraph and a comment in VMThreads, the self-imports in NativeGCOptions, and the deprecated UseG1GC API option that came in with the 25.4.4.1.1 merge.

New commits on top:

• svm: adapt the Shenandoah port to elide-25.5.x: these APIs moved on the fork after 7b7825ae92c:
  • RuntimeStateTrimConfig is now in nativeimage.impl;
  • ObjectLayoutCompilerSupport.getArrayBaseOffset;
  • the ObjectLayout constructor no longer takes the target;
  • JavaStackWalker.doWalkThread.
• svm: use the shared UnlockDiagnosticVMOptions for Shenandoah: the fork now defines UnlockDiagnosticVMOptions in SubstrateGuestOptions. The Shenandoah copy made every native-image run fail during option collection, whatever GC was selected. The shared field is now passed on to the GC library, which checks it for diagnostic modes such as passive.
• svm: keep Shenandoah LIR barriers out of images with other GCs: the LIR generators picked the Shenandoah barrier-set LIR generator from the barrier set at run time. Serial-GC images that include the runtime compiler (Truffle, Ristretto) therefore reached the Shenandoah LIR ops and ShenandoahHeap's thread locals. They then failed thread-local layout with Cannot place thread local ShenandoahHeap.barrierAndAllocationData within its maximum offset 71. The generators now check the folded SubstrateOptions.useShenandoahGC() first.
• svm: report a missing Shenandoah library before the image build: without svmgc, --gc=shenandoah used to build the whole image and then fail in the linker. It now fails before analysis, naming the library and the svmgc component.

git diff elide-25.5.x is Volker's net diff (7b7825ae92c..81cf4340316) plus #167, apart from the adaptations above. The 39 Shenandoah-only files are byte-identical to 16e54ebd709.

Testing

All runs are on Linux/AMD64 with JDK labsjdk-ce-latest-jvmci-25.5-b24 (25.0.4.1.1). #167 was tested on jvmci-25.4-b23; the only reduced-source difference is the jvmci_globals.hpp merge described above.

Default serial GC (regression):

• mx build --warning-as-error --force-javac (substratevm): passes.
• mx checkstyle (substratevm, compiler): passes.
• Ristretto regression set (SL, Ristretto deoptimization, VarHandle, MethodHandle, Dispatch, CallerClass, and HubExtension tests, built with -H:+RuntimeClassLoading -H:+GraalJITCompileAtRuntime): OK (50 tests), OK (2 tests).
• Release-style GraalVM home (--env ni-ce, /web-image, libgraal, native-image, and both agents): builds; libgraal loads (UseJVMCINativeLibrary = true); a serial-GC image builds and runs; the native-image agent writes its configuration.

Shenandoah (--dynamicimports substratevm-gcs --components=ni,nju,svmgc):

• mx shenandoah-gc-stress passes in generational, satb, and passive modes (compressed references).
• Native JUnit with -XX:-UsePerfData, in both generational and satb modes: FinishedThreadTest, FinishedDefaultThreadTest, ForkJoinTest, ProcessPropertiesTest, VirtualThreadMonitorInterruptTest, and RunMainInNewThreadTest give OK (7 tests) and OK (1 test).
• native-image --gc=shenandoah without svmgc now gives a clean error after about 6 seconds.
• Release-style home with substratevm-gcs and svmgc: builds and ships libshenandoahgc-{cr,ur,musl-cr,musl-ur}.a and libg1gc-* in lib/svm/clibraries/linux-amd64. Serial and generational Shenandoah images build and run from it.
• EE hybrid (ci/elide/build-ee.py on Oracle jdk-25i5-25.0.4.1.1-ea.05 with the svmgc CE home): builds and passes LinkCheck. Oracle's G1 libraries are kept and the Shenandoah libraries are added (this needs the build-ee.py change on the CI branch). smoke-ee.sh passes for PGO, G1, code compression, and the new Shenandoah smoke test.

Limitations

• One isolate per process: the native GC has process-wide state.
• No runtime class loading: the Shenandoah metaspace entry points are not implemented. Runtime-compiled (Truffle or Ristretto JIT) code under Shenandoah is not covered here.
• Platforms: Linux/AMD64, Linux/AArch64, and macOS/AArch64 only. There is no Windows or Intel macOS support.
• Performance data: the counters are not implemented. Images with perf data, including the native JUnit harness, need -XX:-UsePerfData.
• Uncompressed references: -H:-UseCompressedReferences crashes at startup in JNIJavaCallTrampolineHolder.valistJavaCallTrampoline. This is not Shenandoah-specific: the serial GC crashes the same way on unmodified elide-25.5.x (2964d0102a2). It needs a separate fix, so Shenandoah's -ur libraries are built but untested on Linux/AMD64.
• Release: the release workflow must build the svmgc component on Linux and macOS (labs-openjdk pinned by SHA) for the distributions to carry Shenandoah. The CI change is on the ci/shenandoah-svmgc branch (no PR).