From 9c3ba6e5a630a73860a51086e5a9765bb28427f9 Mon Sep 17 00:00:00 2001 From: BREW <81044362+b-re-w@users.noreply.github.com> Date: Sat, 3 Oct 2026 13:55:46 +0900 Subject: [PATCH 1/4] Docs: Specify the packaged desktop app's bundled stdlib (SPEC L-9, #60) --- README.md | 5 +- docs/SPEC.md | 18 +++ docs/locale/README_ko.md | 6 +- docs/platforms/desktop-packaged-app.md | 121 ++++++++++++++++++ docs/roadmap/ROADMAP.md | 13 +- .../src/desktopMain/README.md | 30 +++++ 6 files changed, 187 insertions(+), 6 deletions(-) create mode 100644 docs/platforms/desktop-packaged-app.md diff --git a/README.md b/README.md index 69e143704..b4a7106f8 100644 --- a/README.md +++ b/README.md @@ -131,7 +131,10 @@ plugins { `stagePythonHome` downloads the matching CPython build, verifies it against the release's `SHA256SUMS`, caches it machine-wide and sets `PYTHONHOME` on `run` and `test`. A `PYTHONHOME` you set yourself is never touched; disable staging with `pythonBindings { stagePythonHome.set(false) }`. -A packaged end-user app still has to ship a prefix and set `PYTHONHOME` itself. +A Compose Desktop app packaged with `createDistributable` (or a `package*` task) carries the staged +stdlib and `libpython` in its resources and starts without `PYTHONHOME` +(`pythonBindings { packagePythonHome.set(false) }` turns that off); any other packaging can name its +prefix with `-Dpython.multiplatform.home=...`. See `docs/platforms/desktop-packaged-app.md`.
diff --git a/docs/SPEC.md b/docs/SPEC.md index 107532c95..7e1f7dfef 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -71,6 +71,24 @@ fail, then implement. extracts it. `Status: partial` — exercised by the shared suite on the simulator; no iOS-specific lifecycle test beyond `PM/iosSimulatorArm64Test/.../AsyncioAvailabilityProbeTest.kt`. - **L-8** Initialisation sets `builtins.compiled` to an identity decorator (an existing one is kept), so `@compiled` needs no import on any platform (N-7, issue #42). `Status: implemented` — `PM/commonTest/.../ffi/BuiltinCompiledTest.kt`. +- **L-9** Desktop: a packaged application starts with no `PYTHONHOME` in its environment (issue #60, + ROADMAP §15.4). The Gradle plugin copies the prefix `stagePythonHome` stages into Compose Desktop's + application resources (every `prepare*AppResources` task, so `createDistributable` and the + `package*` tasks built on it) under `python-multiplatform-home/`: the standard library + (`lib/python/`, or `Lib/` and `DLLs/` on Windows) and the shared library, without + `__pycache__`, headers, `bin/` or Tcl/Tk. `pythonBindings { packagePythonHome.set(false) }` turns + this off. At run time, before `Py_Initialize`, desktop takes the first of: `PYTHONHOME` from the + environment (CPython reads it itself and nothing else happens); the system property + `python.multiplatform.home`; `/python-multiplatform-home` when + that directory exists. A prefix taken from a property goes through L-3's check (a catchable + `IllegalStateException` naming the path and where it came from), is handed to CPython with + `Py_SetPythonHome`, and is where `libpython` is loaded from when it contains one. + `Status: partial` — `GP/PackagedPythonHomeTest.kt` (what is copied, which tasks, the switch); + `PM/desktopTest/.../env/PackagedPythonHomeTest.kt` (resolution order, `wchar_t` encoding, library + lookup) and `PackagedPythonHomeLaunchTest.kt` (a child JVM with `PYTHONHOME` removed imports `json` + from the prefix a property names, or from a Compose resources directory). The + `createDistributable` run itself is a manual check — + `docs/platforms/desktop-packaged-app.md`. ## 2. Low-level C API (downcall surface) diff --git a/docs/locale/README_ko.md b/docs/locale/README_ko.md index 74f270bc9..dff48d279 100644 --- a/docs/locale/README_ko.md +++ b/docs/locale/README_ko.md @@ -129,8 +129,10 @@ plugins { `stagePythonHome` 이 맞는 CPython 빌드를 내려받아 릴리스의 `SHA256SUMS` 로 검증하고, 기기 전체가 쓰는 캐시에 풀고, `run` 과 `test` 에 `PYTHONHOME` 을 설정한다. 직접 설정한 `PYTHONHOME` 은 건드리지 않는다. -`pythonBindings { stagePythonHome.set(false) }` 로 끌 수 있다. 최종 사용자용으로 패키징한 앱은 여전히 -prefix 를 함께 배포하고 `PYTHONHOME` 을 직접 설정해야 한다. +`pythonBindings { stagePythonHome.set(false) }` 로 끌 수 있다. `createDistributable`(또는 `package*` +태스크)로 패키징한 Compose Desktop 앱은 스테이징된 stdlib 와 `libpython` 을 리소스에 담아 `PYTHONHOME` +없이 시작한다(`pythonBindings { packagePythonHome.set(false) }` 로 끈다). 그 밖의 패키징은 +`-Dpython.multiplatform.home=...` 으로 prefix 를 지정한다. `docs/platforms/desktop-packaged-app.md` 참고.
diff --git a/docs/platforms/desktop-packaged-app.md b/docs/platforms/desktop-packaged-app.md new file mode 100644 index 000000000..7ba3c6a41 --- /dev/null +++ b/docs/platforms/desktop-packaged-app.md @@ -0,0 +1,121 @@ +# Desktop: the standard library in a packaged application + +SPEC L-9, issue #60, ROADMAP §15.4. + +During development `stagePythonHome` downloads a python-build-standalone prefix once per machine and +the plugin sets `PYTHONHOME` on every `JavaExec` and `Test` task (SPEC L-4). A packaged application +is started by a launcher, not by Gradle, and nothing put `PYTHONHOME` in its environment, so +`Py_Initialize()` aborted with `Failed to import encodings module`. This page records how a packaged +app now gets its prefix, why it was done this way, and how to check it. + +## What happens + +**Build time** (`python-multiplatform-gradle-plugin`, `PythonBindingsPlugin.configurePackagedPythonHome`). +Every Compose Desktop `prepare*AppResources` task — the `Sync` that `createDistributable`, +`packageDmg`/`packageMsi`/`packageDeb` and their `Release` variants take their resources from — +depends on `stagePythonHome` and copies part of the staged prefix into `python-multiplatform-home/`: + +| Host | Copied (relative to the prefix) | +|---|---| +| macOS, Linux | `lib/python/**`, `lib/libpython.*` (`t` for a free-threaded build) | +| Windows | `Lib/**`, `DLLs/**`, `*.dll` | + +`**/__pycache__/**`, `include/`, `bin/`, `lib/pkgconfig` and Tcl/Tk are not copied. For 3.14.7 +macos-aarch64 that is 1,195 files and 39.6 MB (19.4 MB of it `libpython3.14.dylib`) of a 68.9 MB +prefix, measured on the staged prefix of this machine. + +In the app image the directory is `$APPDIR/resources/python-multiplatform-home`; on macOS +`.app/Contents/app/resources/python-multiplatform-home`, on Linux +`/lib/app/resources/python-multiplatform-home`. + +**Run time** (`python-multiplatform`). `PackagedPythonHome.resolve()` (`jvmMain`) takes the first of: + +1. `PYTHONHOME` in the environment — then nothing else happens; CPython reads it itself. +2. The system property `python.multiplatform.home`, used even if the path does not exist, so that a + wrong value is reported by name. +3. `/python-multiplatform-home`, only if that directory exists. + Compose sets `-Dcompose.application.resources.dir=$APPDIR/resources` on every launcher it builds. + +`Python3.initialize()` calls `applyPackagedPythonHome()` right after the `PYTHONHOME` pre-flight +check (SPEC L-3). On desktop it checks the resolved prefix with the same `PythonHomeCheck` +(throwing a catchable `IllegalStateException` that names the path and the property it came from) +and passes it to `Py_SetPythonHome` as a `wchar_t` string. Before that, when the bindings load, +`manager.loadLibPython` loads `libpython` from the same prefix instead of extracting the classpath +copy into the working directory. + +## Why this design + +**Into the app image, not a jar resource extracted at startup.** The consumer's `python/` payload is +a jar resource that `ClasspathPayload` extracts to a temp directory with a digest stamp. The same +mechanism for the stdlib was considered and not taken: + +- The stdlib is 1,195 files / 39.6 MB uncompressed. Extracted from a jar it is shipped twice (once + compressed in the app, once extracted per user), and the first launch pays for writing all of it. + Put into the app image, it is on disk at install time: no extraction ever, so there is no stamp to + check on each launch and no cold-start cost at all. +- The payload's cache lives under `java.io.tmpdir`, which operating systems clean up on their own + schedule. A stamp that survives a partial clean-up which removed part of `encodings/` would be an + abort at `Py_Initialize` rather than a re-extraction. (Not observed here; a reason not to take the + risk for the one tree whose absence is fatal.) +- `libpython` has to be a file for `System.load` anyway. Loading it from the classpath copy means + extracting 19.4 MB into the working directory and finding it through `java.library.path` — this + repo's tests pass `-Djava.library.path=.`, a launched app has no such flag and may have `/` as its + working directory. Loading it from the packaged prefix also pairs it with its own stdlib. +- jpackage builds an image for the host it runs on, so only one platform's prefix is ever packaged; + the cost `PythonHomeStaging`'s KDoc measured against shipping four platforms' stdlibs in the + *library* jar does not arise. + +**Into Compose's resources task, not `nativeDistributions.appResourcesRootDir`.** That property is a +single directory the consumer may already use for their own files; setting it from the plugin would +replace theirs. Adding a `from` to the `Sync` that assembles the resources composes with whatever +the consumer configured, and matching the task by name keeps the plugin free of a compile-time +dependency on the Compose Gradle plugin (as the wasm wiring matches `wasmJsProcessResources`). + +**`Py_SetPythonHome`, not an environment variable.** The launcher accepts `--java-options` and no +environment, and a JVM cannot set its own (`desktopMain/README.md`). `Py_SetPythonHome` is part of +the Stable ABI (deprecated since 3.11 in favour of `PyConfig.home`, whose struct layout the Stable +ABI does not promise) and writes CPython's path configuration directly, so `Py_Initialize()` uses it +whenever `PYTHONHOME` is unset. + +## Other packagers + +Anything that is not Compose's `createDistributable` (plain `jpackage`, Conveyor, an uber jar) gets +no copy from the plugin. Copy the same subset of the staged prefix (`stagePythonHome`'s output, +`~/.gradle/python-multiplatform/python-home/+//python`) next to the app +and pass `-Dpython.multiplatform.home=`; jpackage expands `$APPDIR` inside `--java-options`. + +## Limits + +- No `.pyc` is shipped. CPython compiles what it imports on first use and writes `__pycache__` into + the app image where it can; where it cannot (a read-only install), it compiles on every launch. +- `pythonBindings { packagePythonHome.set(false) }` turns the copy off. `stagePythonHome.set(false)` + does not: the packaged prefix must be the one the library was built against regardless of what a + developer's own `PYTHONHOME` points at, so the task is registered (lazily) either way. +- The library needs `--enable-preview` on JDK 21 (`java.lang.foreign` is a preview API there) and + `--enable-native-access=ALL-UNNAMED`. A packaged app gets JVM options only from + `compose.desktop.application { jvmArgs(...) }`; the sample's `build.gradle.kts` sets them for + `JavaExec` tasks only. + +## How to check it + +Run on the build host, from the repository root, with no `PYTHONHOME` in the launched app's +environment: + +```bash +./gradlew :sample:createDistributable --console=plain > .tmp/dist.log 2>&1; echo "EXIT=$?" + +# The prefix is in the image (macOS paths; Linux: sample/build/compose/binaries/main/app//lib/app/resources) +APP=sample/build/compose/binaries/main/app/org.thisisthepy.python.multiplatform.demo.app +ls "$APP/Contents/app/resources/python-multiplatform-home/lib" # libpython3.14.dylib python3.14 +test -f "$APP/Contents/app/resources/python-multiplatform-home/lib/python3.14/encodings/__init__.py" && echo STDLIB_OK +find "$APP/Contents/app/resources/python-multiplatform-home" -name __pycache__ | wc -l # 0 + +# Launch with PYTHONHOME removed; the sample prints its runtime line and runs asyncio before the window +env -u PYTHONHOME -u PYTHON_MULTIPLATFORM_LIBPYTHON \ + "$APP/Contents/MacOS/org.thisisthepy.python.multiplatform.demo" > .tmp/packaged-run.log 2>&1 & +``` + +Expected in `.tmp/packaged-run.log`: `runtime : 3.14.7 ...` and the `await` lines (which import +`asyncio`), with no `Fatal Python error`. The window stays open; close it or kill the process. +Running from `/` as the working directory (`cd / && env -u PYTHONHOME "$OLDPWD/$APP/..."`) also +checks that nothing is extracted into the working directory. diff --git a/docs/roadmap/ROADMAP.md b/docs/roadmap/ROADMAP.md index 136925281..2e727d374 100644 --- a/docs/roadmap/ROADMAP.md +++ b/docs/roadmap/ROADMAP.md @@ -318,9 +318,16 @@ Old §15a, §15b, §15d, §15f, §15g, §15h are closed; §15e item 4 is closed Windows-x86_64 are pinned as string assertions in `PythonHomeStagingTest`; Windows' `Lib/`+`DLLs/` layout and `linux-aarch64` (no `libpython` in `desktopJar`, relies on `manager.loadFromSidecar`) have never executed. -- **15.4 A packaged desktop application carries no CPython prefix — open.** `stagePythonHome` serves - `run`/`test` on a developer machine; nothing makes `jpackage`, Conveyor or an installer ship a - prefix and set `PYTHONHOME` for an end user. +- **15.4 A packaged desktop application carries no CPython prefix — implemented, end-to-end check + pending (issue #60, SPEC L-9).** The plugin copies the staged prefix's stdlib and `libpython` into + Compose Desktop's `prepare*AppResources` (so `createDistributable` and `package*`) as + `python-multiplatform-home/`; at run time the library resolves it through + `compose.application.resources.dir` (or `-Dpython.multiplatform.home`) and hands it to CPython with + `Py_SetPythonHome`. A child-JVM test with `PYTHONHOME` removed covers the run-time half + (`PM/desktopTest/.../env/PackagedPythonHomeLaunchTest.kt`); the packaged sample run is + `docs/platforms/desktop-packaged-app.md`. Left: Conveyor and non-Compose jpackage need the + property set by hand; no `.pyc` is shipped; only the build host's platform is packaged (jpackage + builds for its host anyway). - **15.5 The Android AAR duplicates the pure-Python stdlib per ABI — open (measured option, not taken).** 790 files / 11.91 MB are byte-identical between `arm64-v8a` and `x86_64`; de-duplicating saves 3.16 MB compressed but changes the published asset layout `/lib/python` that diff --git a/python-multiplatform/src/desktopMain/README.md b/python-multiplatform/src/desktopMain/README.md index c72b5f73f..5087fcf94 100644 --- a/python-multiplatform/src/desktopMain/README.md +++ b/python-multiplatform/src/desktopMain/README.md @@ -95,6 +95,36 @@ The staged prefix is shared per machine (under the Gradle user home), keyed by v release + platform, and stamped after the last extracted byte — so an interrupted extraction is never mistaken for a finished one. See ROADMAP §15h. +## A packaged app names its prefix in-process, with `Py_SetPythonHome` (SPEC L-9) + +The section above is about `PYTHONHOME` **the environment variable**. A packaged application +(`createDistributable`, an installer, `java -jar`) is started by a launcher that takes +`--java-options` and no environment, so it gets the prefix another way: + +- The plugin copies the staged prefix's stdlib and `libpython` (no `__pycache__`, headers, `bin/`, + Tcl/Tk) into Compose Desktop's application resources as `python-multiplatform-home/`, i.e. + `$APPDIR/resources/python-multiplatform-home` in the app image. +- At run time `PackagedPythonHome` (`jvmMain`) resolves: environment `PYTHONHOME` (wins; nothing + else happens) → `-Dpython.multiplatform.home` → `/python-multiplatform-home` + if it exists. `applyPackagedPythonHome` checks the result with `PythonHomeCheck` and hands it to + `Py_SetPythonHome` before `Py_Initialize()`. That writes CPython's path configuration directly; + it is not an environment variable, so there is nothing `PythonHomeCheck` could fail to see. +- `manager.loadLibPython` loads `libpython` from that prefix ahead of the classpath copy. The + classpath copy is extracted into the *working directory* and loaded via `java.library.path`, + which this repo's tests make work with `-Djava.library.path=.` and a launched app does not have. + +`Py_SetPythonHome` is deprecated since 3.11 in favour of `PyConfig.home`, but `PyConfig` is a +struct whose layout the Stable ABI does not promise, and the function is in the Stable ABI, so its +symbol stays exported. It takes `wchar_t *` — UTF-32 here, UTF-16 on Windows — encoded by hand in +`encodeWideString` rather than through `Py_DecodeLocale`, which before pre-initialisation decodes +with the C locale (the ANSI code page on Windows). The buffer is never freed (a few hundred bytes, +once per process): the API asks for static storage. + +Not shipped: bytecode. python-build-standalone's `install_only` prefix has no `.pyc`, and the +staged one's `__pycache__` reflects whatever ran against it, so neither is packaged; CPython writes +`__pycache__` into the app image on first use where it can. See +`docs/platforms/desktop-packaged-app.md`. + ## The consumer's own Python is a classpath resource, and `python/` collides with our own package `toolchain`'s `stagePythonBundleDesktop` puts a consumer's payload at the **root of the jar** as From 1946bd4ffbdec93442b3ce86fb9e1c286238cc21 Mon Sep 17 00:00:00 2001 From: BREW <81044362+b-re-w@users.noreply.github.com> Date: Sat, 3 Oct 2026 13:55:46 +0900 Subject: [PATCH 2/4] Test: A packaged desktop app finds its stdlib without PYTHONHOME (#60) --- .../gradle/PackagedPythonHomeTest.kt | 154 ++++++++++++++ .../env/PackagedPythonHomeLaunchTest.kt | 193 +++++++++++++++++ .../env/PackagedPythonHomeTest.kt | 197 ++++++++++++++++++ 3 files changed, 544 insertions(+) create mode 100644 python-multiplatform-gradle-plugin/src/test/kotlin/python/multiplatform/gradle/PackagedPythonHomeTest.kt create mode 100644 python-multiplatform/src/desktopTest/kotlin/python/multiplatform/env/PackagedPythonHomeLaunchTest.kt create mode 100644 python-multiplatform/src/desktopTest/kotlin/python/multiplatform/env/PackagedPythonHomeTest.kt diff --git a/python-multiplatform-gradle-plugin/src/test/kotlin/python/multiplatform/gradle/PackagedPythonHomeTest.kt b/python-multiplatform-gradle-plugin/src/test/kotlin/python/multiplatform/gradle/PackagedPythonHomeTest.kt new file mode 100644 index 000000000..833ea52d7 --- /dev/null +++ b/python-multiplatform-gradle-plugin/src/test/kotlin/python/multiplatform/gradle/PackagedPythonHomeTest.kt @@ -0,0 +1,154 @@ +package python.multiplatform.gradle + +import org.gradle.api.tasks.Sync +import org.gradle.testfixtures.ProjectBuilder +import java.io.File +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertFalse +import kotlin.test.assertNotNull +import kotlin.test.assertTrue + +/** + * SPEC L-9 (issue #60): what a packaged desktop application carries of the staged CPython prefix, + * and which tasks carry it. + * + * The run-time half -- the library finding that directory and handing it to CPython -- is + * `python-multiplatform/src/desktopTest/.../env/PackagedPythonHome*Test.kt`. Neither half can see + * the other: this file pins the directory name the library looks for as a literal for that reason. + */ +class PackagedPythonHomeTest { + + // ---- what is copied ---- + + @Test + fun theDirectoryNameIsTheOneTheLibraryLooksFor() { + // `PackagedPythonHome.DIRECTORY_NAME` in `python-multiplatform/src/jvmMain`. The two builds + // share no code, so a rename on one side only would package a prefix nothing finds -- and + // the app would still start whenever the developer's shell happened to have PYTHONHOME. + assertEquals("python-multiplatform-home", PACKAGED_HOME_DIRECTORY) + } + + @Test + fun aUnixPrefixShipsTheStdlibAndTheSharedLibraryOnly() { + val includes = packagedPrefixIncludes("3.14.7", freeThreaded = false, windows = false) + assertEquals(listOf("lib/python3.14/**", "lib/libpython3.14.*"), includes) + } + + @Test + fun aFreeThreadedPrefixUsesTheTaggedDirectoryAndLibrary() { + // A 3.14t install has `lib/python3.14t/` and `libpython3.14t.dylib` and nothing untagged, + // so the untagged patterns would package an empty tree. + val includes = packagedPrefixIncludes("3.14.7", freeThreaded = true, windows = false) + assertEquals(listOf("lib/python3.14t/**", "lib/libpython3.14t.*"), includes) + } + + @Test + fun aWindowsPrefixShipsLibDllsAndTheRootDlls() { + // python-build-standalone's Windows layout: `Lib/` (pure Python), `DLLs/` (extension + // modules), and `python3*.dll` plus the MSVC runtime at the prefix root, which is where + // `manager.libraryUnder` looks for the interpreter on that layout. + val includes = packagedPrefixIncludes("3.14.7", freeThreaded = false, windows = true) + assertEquals(listOf("Lib/**", "DLLs/**", "*.dll"), includes) + } + + @Test + fun bytecodeCachesAreNeverPackaged() { + // The staged prefix is shared by every project on the machine and accumulates + // `__pycache__` from whatever ran against it; copying that would make the app's contents + // depend on the developer's history. + assertTrue("**/__pycache__/**" in PACKAGED_PREFIX_EXCLUDES) + } + + // ---- which tasks ---- + + @Test + fun composeAppResourcesTasksAreRecognisedForEveryBuildType() { + assertTrue(isComposeAppResourcesTask("prepareAppResources")) + assertTrue(isComposeAppResourcesTask("prepareReleaseAppResources")) + assertFalse(isComposeAppResourcesTask("processResources")) + assertFalse(isComposeAppResourcesTask("desktopProcessResources")) + assertFalse(isComposeAppResourcesTask("prepareAppResourcesFoo")) + } + + // ---- wiring, against a fake staged prefix ---- + + @Test + fun theComposeResourcesTaskCopiesTheStagedPrefixSubsetAndDependsOnStaging() { + val fixture = Fixture() + val sync = fixture.syncTask() + + val dependencies = sync.taskDependencies.getDependencies(sync).map { it.name } + assertTrue("stagePythonHome" in dependencies, "prepareAppResources must run stagePythonHome first; got $dependencies") + + val copied = sync.source.files.map { it.relativeTo(fixture.prefix).invariantSeparatorsPath }.toSet() + assertTrue(fixture.stdlibFile in copied, "the stdlib must be packaged; got $copied") + assertTrue(fixture.libraryFile in copied, "the shared library must be packaged; got $copied") + assertFalse(fixture.headerFile in copied, "headers are build-time only; got $copied") + assertFalse(fixture.binaryFile in copied, "bin/ is not used by an embedding app; got $copied") + assertFalse(fixture.tclFile in copied, "Tcl/Tk is not packaged; got $copied") + assertFalse(fixture.bytecodeFile in copied, "__pycache__ is not packaged; got $copied") + } + + @Test + fun turningPackagingOffLeavesTheComposeTaskAlone() { + val fixture = Fixture() + fixture.extension.packagePythonHome.set(false) + val sync = fixture.syncTask() + + val dependencies = sync.taskDependencies.getDependencies(sync).map { it.name } + assertFalse("stagePythonHome" in dependencies, "disabled packaging must not stage; got $dependencies") + assertTrue(sync.source.files.none { it.startsWith(fixture.prefix) }, "disabled packaging must copy nothing") + } + + /** + * A project whose Gradle user home already holds a "staged" prefix, at exactly the path + * `configurePythonHomeStaging` computes, so the wiring can be read without a download. + */ + private class Fixture { + // Under this module's own build directory (the test task's working directory), not the + // system temporary directory: AGENTS.md rule 2. + val root: File = File(System.getProperty("user.dir"), "build/tmp/packagedPythonHomeTest/${java.util.UUID.randomUUID()}") + .apply { mkdirs() } + val gradleUserHome = File(root, "gradle-home") + val project = ProjectBuilder.builder() + .withProjectDir(File(root, "project").apply { mkdirs() }) + .withGradleUserHomeDir(gradleUserHome) + .build() + val extension: PythonBindingsExtension = + project.extensions.create("pythonBindings", PythonBindingsExtension::class.java) + + private val platform = assertNotNull( + hostDesktopPlatform(System.getProperty("os.name"), System.getProperty("os.arch")), + "this test needs a host python-build-standalone publishes for", + ) + private val windows = platform.startsWith("windows") + private val flavour = if (DEFAULT_PYTHON_FREE_THREADED) "-freethreaded" else "" + private val tag = DEFAULT_PYTHON_VERSION.split('.').take(2).joinToString(".") + + if (DEFAULT_PYTHON_FREE_THREADED) "t" else "" + + val prefix = File( + gradleUserHome, + "${PythonHomeStaging.CACHE_DIRECTORY}/$DEFAULT_PYTHON_VERSION+$DEFAULT_PBS_RELEASE$flavour/$platform/python", + ) + + val stdlibFile = if (windows) "Lib/os.py" else "lib/python$tag/os.py" + val libraryFile = if (windows) "python${tag.replace(".", "").removeSuffix("t")}.dll" else "lib/libpython$tag.dylib" + val headerFile = "include/python$tag/Python.h" + val binaryFile = if (windows) "python.exe" else "bin/python3" + val tclFile = if (windows) "tcl/tcl9.0/init.tcl" else "lib/libtcl9.0.dylib" + val bytecodeFile = if (windows) "Lib/__pycache__/os.cpython-314.pyc" else "lib/python$tag/__pycache__/os.cpython-314.pyc" + + init { + listOf(stdlibFile, libraryFile, headerFile, binaryFile, tclFile, bytecodeFile).forEach { + File(prefix, it).apply { parentFile.mkdirs(); writeText("x") } + } + project.tasks.register("prepareAppResources", Sync::class.java) { + into(project.layout.buildDirectory.dir("compose/tmp/prepareAppResources")) + } + PythonBindingsPlugin().configurePythonHomeStaging(project, extension) + } + + fun syncTask(): Sync = project.tasks.getByName("prepareAppResources") as Sync + } +} diff --git a/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/env/PackagedPythonHomeLaunchTest.kt b/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/env/PackagedPythonHomeLaunchTest.kt new file mode 100644 index 000000000..d9ddabdf5 --- /dev/null +++ b/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/env/PackagedPythonHomeLaunchTest.kt @@ -0,0 +1,193 @@ +package python.multiplatform.env + +import python.multiplatform.ffi.Python3 +import java.io.File +import java.nio.file.Files +import java.util.concurrent.TimeUnit +import kotlin.system.exitProcess +import kotlin.test.AfterTest +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertNotNull +import kotlin.test.assertTrue + +/** + * SPEC L-9 (issue #60), end to end: a JVM whose environment has **no** `PYTHONHOME` starts CPython + * and imports from the standard library, given only what a packaged application has -- a system + * property, or Compose's `compose.application.resources.dir`. + * + * It has to be a child process. This test JVM's interpreter was initialised from `PYTHONHOME` by an + * earlier test (or will be by a later one), `Py_Initialize` runs once per process, and a JVM cannot + * remove a variable from its own native environment. The child is launched the way a packaged app + * is: the same class path, no `-Djava.library.path=.` (which is what lets this build's own tests load + * the classpath copy of `libpython` out of the working directory), and `PYTHONHOME` removed. + * + * Before L-9 the first two tests fail with the child aborting in `Py_Initialize` + * (`Fatal Python error: ... Failed to import encodings module`, non-zero exit, no `PROBE_OK`), and + * the third with the same abort instead of a `PROBE_REFUSED` naming the path. The fourth is a guard, + * green before and after: an environment `PYTHONHOME` must keep winning. + */ +class PackagedPythonHomeLaunchTest { + + private val scratchDirs = mutableListOf() + + @AfterTest + fun cleanup() { + scratchDirs.forEach { it.deleteRecursively() } + scratchDirs.clear() + } + + /** Under `build/`, not the system temporary directory -- AGENTS.md rule 2. */ + private fun newScratchDir(): File = + File("build/tmp/packagedPythonHomeLaunchTest/${java.util.UUID.randomUUID()}").absoluteFile + .also { it.mkdirs(); scratchDirs += it } + + private fun stagedPrefix(): String = assertNotNull( + System.getenv("PYTHONHOME"), + "PYTHONHOME is unset for desktopTest; there is no real prefix to package", + ) + + private class Outcome(val exitCode: Int, val output: String) + + // Literals rather than `PackagedPythonHome`'s constants, so that this file compiles before + // L-9 exists and its red is a runtime abort, distinguishable from a compile error. + private val homeProperty = "python.multiplatform.home" + private val composeResourcesProperty = "compose.application.resources.dir" + private val homeDirectoryName = "python-multiplatform-home" + + /** + * @param dotOnLibraryPath adds `-Djava.library.path=.`, which is how this build's own test JVMs + * load the classpath copy of `libpython` (extracted into the working directory). Only the + * environment-`PYTHONHOME` guard uses it; a packaged app has no such flag. + */ + private fun launchProbe( + environmentHome: String?, + systemProperties: Map, + dotOnLibraryPath: Boolean = false, + ): Outcome { + val workDir = newScratchDir() + val log = File(workDir, "probe.log") + val java = File(System.getProperty("java.home"), "bin/java").path + val command = buildList { + add(java) + // java.lang.foreign is a preview API on JDK 21, which is what desktopTest runs on. + add("--enable-preview") + add("--enable-native-access=ALL-UNNAMED") + if (dotOnLibraryPath) add("-Djava.library.path=.") + systemProperties.forEach { (key, value) -> add("-D$key=$value") } + add("-cp") + add(System.getProperty("java.class.path")) + add(PackagedHomeProbe::class.java.name) + } + val builder = ProcessBuilder(command) + .directory(workDir) + .redirectErrorStream(true) + .redirectOutput(log) + builder.environment().remove("PYTHONHOME") + builder.environment().remove("PYTHON_MULTIPLATFORM_LIBPYTHON") + if (environmentHome != null) builder.environment()["PYTHONHOME"] = environmentHome + val process = builder.start() + if (!process.waitFor(120, TimeUnit.SECONDS)) { + process.destroyForcibly() + return Outcome(-1, "timed out after 120 s\n" + log.readText()) + } + return Outcome(process.exitValue(), log.readText()) + } + + @Test + fun aJvmWithoutPythonHomeImportsTheStdlibFromThePrefixAPropertyNames() { + val prefix = stagedPrefix() + val outcome = launchProbe( + environmentHome = null, + systemProperties = mapOf(homeProperty to prefix), + ) + assertEquals(0, outcome.exitCode, outcome.output) + val line = outcome.output.lineSequence().firstOrNull { it.startsWith("PROBE_OK ") } + assertNotNull(line, "the child never reported a successful import:\n${outcome.output}") + // sys.prefix is what CPython derived its stdlib path from; it must be ours, not a guess. + assertEquals(File(prefix).canonicalPath, File(line.substringAfter("PROBE_OK ")).canonicalPath, outcome.output) + } + + @Test + fun aJvmWithoutPythonHomeImportsTheStdlibFromComposeResources() { + // Exactly what `createDistributable` produces: `$APPDIR/resources/python-multiplatform-home` + // and Compose's own `-Dcompose.application.resources.dir=$APPDIR/resources`. The prefix is + // linked rather than copied to keep this test from writing 25 MB per run. + val resources = newScratchDir() + Files.createSymbolicLink( + File(resources, homeDirectoryName).toPath(), + File(stagedPrefix()).toPath(), + ) + val outcome = launchProbe( + environmentHome = null, + systemProperties = mapOf(composeResourcesProperty to resources.path), + ) + assertEquals(0, outcome.exitCode, outcome.output) + val line = outcome.output.lineSequence().firstOrNull { it.startsWith("PROBE_OK ") } + assertNotNull(line, "the child never reported a successful import:\n${outcome.output}") + assertTrue( + line.substringAfter("PROBE_OK ").trimEnd('/', '\\').endsWith(homeDirectoryName), + "sys.prefix is not the Compose resources home: $line", + ) + } + + @Test + fun anUnusablePropertyPrefixIsACatchableExceptionNamingThePath() { + val missing = File(newScratchDir(), "no-such-prefix").path + val outcome = launchProbe( + environmentHome = null, + systemProperties = mapOf(homeProperty to missing), + ) + assertEquals(PackagedHomeProbe.REFUSED_EXIT_CODE, outcome.exitCode, outcome.output) + val line = outcome.output.lineSequence().firstOrNull { it.startsWith("PROBE_REFUSED ") } + assertNotNull(line, "the child did not refuse with a Kotlin exception:\n${outcome.output}") + assertTrue(missing in line, "the refusal does not name the path: $line") + assertTrue(homeProperty in line, "the refusal does not say where the path came from: $line") + } + + @Test + fun anEnvironmentPythonHomeStillWinsOverTheProperty() { + // Guard (green before L-9 as well): a developer's or a test task's PYTHONHOME must not be + // overridden by a property baked into a launcher. + val prefix = stagedPrefix() + val outcome = launchProbe( + environmentHome = prefix, + systemProperties = mapOf(homeProperty to File(newScratchDir(), "ignored").path), + dotOnLibraryPath = true, + ) + assertEquals(0, outcome.exitCode, outcome.output) + val line = outcome.output.lineSequence().firstOrNull { it.startsWith("PROBE_OK ") } + assertNotNull(line, outcome.output) + assertEquals(File(prefix).canonicalPath, File(line.substringAfter("PROBE_OK ")).canonicalPath, outcome.output) + } +} + +/** + * The child's `main`. Prints `PROBE_OK ` after a real stdlib import, or + * `PROBE_REFUSED ` and exits with [REFUSED_EXIT_CODE] when `Python3.initialize()` throws. + * A `Py_FatalError` abort exits with neither line, which is the pre-L-9 failure. + */ +object PackagedHomeProbe { + const val REFUSED_EXIT_CODE = 3 + + @JvmStatic + fun main(args: Array) { + try { + Python3.initialize(silent = true) + } catch (e: IllegalStateException) { + println("PROBE_REFUSED ${e.message}") + System.out.flush() + exitProcess(REFUSED_EXIT_CODE) + } + // `json` pulls in `encodings`, `re` and a C extension (`_json`), so it exercises the pure + // Python tree and the extension-module directory alike. + Python3.exec( + """ + import json, sys + json.dumps({"a": [1, 2]}) + print("PROBE_OK " + sys.prefix, flush=True) + """.trimIndent() + ) + exitProcess(0) + } +} diff --git a/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/env/PackagedPythonHomeTest.kt b/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/env/PackagedPythonHomeTest.kt new file mode 100644 index 000000000..f8ca69a82 --- /dev/null +++ b/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/env/PackagedPythonHomeTest.kt @@ -0,0 +1,197 @@ +package python.multiplatform.env + +import python.native.ffi.manager +import java.io.File +import java.nio.ByteOrder +import kotlin.test.AfterTest +import kotlin.test.Test +import kotlin.test.assertContentEquals +import kotlin.test.assertEquals +import kotlin.test.assertNotNull +import kotlin.test.assertNull + +/** + * SPEC L-9 (issue #60), the decisions: which prefix a packaged desktop application uses, how it is + * spelled for `Py_SetPythonHome`, and where `libpython` is found in it. + * + * The end-to-end half -- a JVM with no `PYTHONHOME` actually starting CPython from such a prefix -- + * is [PackagedPythonHomeLaunchTest]. It needs a child process because this test JVM's interpreter + * was started from `PYTHONHOME` long before any test runs. + */ +class PackagedPythonHomeTest { + + private val scratchDirs = mutableListOf() + + @AfterTest + fun cleanup() { + scratchDirs.forEach { it.deleteRecursively() } + scratchDirs.clear() + } + + /** Under `build/`, not the system temporary directory -- AGENTS.md rule 2. */ + private fun newScratchDir(): File = + File("build/tmp/packagedPythonHomeTest/${java.util.UUID.randomUUID()}").absoluteFile + .also { it.mkdirs(); scratchDirs += it } + + private fun properties(vararg pairs: Pair): (String) -> String? { + val map = mapOf(*pairs) + return { map[it] } + } + + // ---- resolution order ---- + + @Test + fun anEnvironmentPythonHomeWinsAndNothingElseIsDone() { + // CPython reads PYTHONHOME itself, and L-3 checks it; a second source would only be a + // second opinion that could disagree with the one CPython acts on. + val resolved = PackagedPythonHome.resolve( + environmentHome = "/from/env", + property = properties(PackagedPythonHome.HOME_PROPERTY to "/from/property"), + isDirectory = { true }, + ) + assertNull(resolved) + } + + @Test + fun aBlankEnvironmentPythonHomeCountsAsUnset() { + // CPython ignores an empty PYTHONHOME too, so treating "" as "set" would start the + // interpreter with no home at all. + val resolved = PackagedPythonHome.resolve( + environmentHome = "", + property = properties(PackagedPythonHome.HOME_PROPERTY to "/from/property"), + isDirectory = { true }, + ) + assertEquals("/from/property", resolved?.path) + } + + @Test + fun theExplicitPropertyIsUsedEvenWhenItDoesNotExist() { + // So that a wrong `-Dpython.multiplatform.home` reaches PythonHomeCheck and is reported + // with its path, rather than silently falling through to "no home" and a Py_FatalError. + val resolved = PackagedPythonHome.resolve( + environmentHome = null, + property = properties(PackagedPythonHome.HOME_PROPERTY to "/does/not/exist"), + isDirectory = { false }, + ) + assertEquals("/does/not/exist", resolved?.path) + assertEquals("system property ${PackagedPythonHome.HOME_PROPERTY}", resolved?.source) + } + + @Test + fun theExplicitPropertyWinsOverComposeResources() { + val resolved = PackagedPythonHome.resolve( + environmentHome = null, + property = properties( + PackagedPythonHome.HOME_PROPERTY to "/explicit", + PackagedPythonHome.COMPOSE_RESOURCES_PROPERTY to "/app/resources", + ), + isDirectory = { true }, + ) + assertEquals("/explicit", resolved?.path) + } + + @Test + fun composeResourcesAreUsedWhenTheyHoldTheDirectory() { + val resources = newScratchDir() + val home = File(resources, PackagedPythonHome.DIRECTORY_NAME).apply { mkdirs() } + val resolved = PackagedPythonHome.resolve( + environmentHome = null, + property = properties(PackagedPythonHome.COMPOSE_RESOURCES_PROPERTY to resources.path), + isDirectory = { File(it).isDirectory }, + ) + assertEquals(home.path, resolved?.path) + } + + @Test + fun composeResourcesWithoutTheDirectoryAreNotAHome() { + // Every Compose Desktop app has `compose.application.resources.dir` set, including one + // whose build turned packaging off. That app must behave exactly as before: no home of + // ours, CPython's own default search, and L-3 unchanged. + val resources = newScratchDir() + val resolved = PackagedPythonHome.resolve( + environmentHome = null, + property = properties(PackagedPythonHome.COMPOSE_RESOURCES_PROPERTY to resources.path), + isDirectory = { File(it).isDirectory }, + ) + assertNull(resolved) + } + + @Test + fun nothingConfiguredIsNoHome() { + assertNull(PackagedPythonHome.resolve(environmentHome = null, property = { null }, isDirectory = { true })) + } + + @Test + fun theDirectoryNameIsTheOneThePluginPackages() { + // `PACKAGED_HOME_DIRECTORY` in python-multiplatform-gradle-plugin; the builds share no code. + assertEquals("python-multiplatform-home", PackagedPythonHome.DIRECTORY_NAME) + } + + // ---- wchar_t for Py_SetPythonHome ---- + + @Test + fun unixWideStringsAreUtf32WithAFourByteTerminator() { + // macOS and Linux have a 4-byte wchar_t. A non-BMP character is one code unit there, not + // a surrogate pair -- encoding through UTF-16 would hand CPython two invalid code points. + val encoded = encodeWideString("aé😀", windows = false, order = ByteOrder.LITTLE_ENDIAN) + assertContentEquals( + byteArrayOf( + 0x61, 0, 0, 0, + 0xE9.toByte(), 0, 0, 0, + 0x00, 0xF6.toByte(), 0x01, 0, + 0, 0, 0, 0, + ), + encoded, + ) + } + + @Test + fun windowsWideStringsAreUtf16WithATwoByteTerminator() { + val encoded = encodeWideString("a😀", windows = true, order = ByteOrder.LITTLE_ENDIAN) + assertContentEquals( + byteArrayOf(0x61, 0, 0x3D, 0xD8.toByte(), 0x00, 0xDE.toByte(), 0, 0), + encoded, + ) + } + + @Test + fun bigEndianOrderIsHonoured() { + assertContentEquals( + byteArrayOf(0, 0, 0, 0x61, 0, 0, 0, 0), + encodeWideString("a", windows = false, order = ByteOrder.BIG_ENDIAN), + ) + } + + // ---- libpython inside a packaged prefix ---- + + @Test + fun theSharedLibraryIsFoundUnderLibOrAtThePrefixRoot() { + // `lib/` on macOS and Linux, the prefix root on Windows -- the same two shapes + // `resolveSidecarLibrary` probes under PYTHONHOME. + val unixPrefix = newScratchDir() + val name = "python3.14" + val unixLibrary = File(unixPrefix, "lib/${System.mapLibraryName(name)}").apply { parentFile.mkdirs(); writeText("x") } + assertEquals(unixLibrary, manager.libraryUnder(unixPrefix.path, name)) + + val windowsPrefix = newScratchDir() + val windowsLibrary = File(windowsPrefix, System.mapLibraryName(name)).apply { writeText("x") } + assertEquals(windowsLibrary, manager.libraryUnder(windowsPrefix.path, name)) + + assertNull(manager.libraryUnder(newScratchDir().path, name)) + } + + @Test + fun theRealStagedPrefixHoldsTheLibraryThisBuildLoads() { + // The prefix the plugin packages is a subset of this one, so if `libraryUnder` cannot find + // the interpreter here, a packaged app falls back to the classpath copy -- which it extracts + // into the working directory, a directory a launched app does not own. + val home = assertNotNull(System.getenv("PYTHONHOME"), "PYTHONHOME is unset for desktopTest") + val tagged = python.multiplatform.Versions.currentVersion.taggedVersionString + val name = if (python.multiplatform.currentPlatform.os == python.multiplatform.OSType.Windows) { + "python" + tagged.replace(".", "") + } else { + "python$tagged" + } + assertNotNull(manager.libraryUnder(home, name), "no ${System.mapLibraryName(name)} under $home") + } +} From ff0e02b95e0cf0a87273b8ee5b44a5cf7bf39119 Mon Sep 17 00:00:00 2001 From: BREW <81044362+b-re-w@users.noreply.github.com> Date: Sat, 3 Oct 2026 13:55:47 +0900 Subject: [PATCH 3/4] Feat: Ship the stdlib in the packaged desktop app and point CPython at it (#60) --- .../gradle/PythonBindingsPlugin.kt | 98 +++++++++++++++---- .../multiplatform/gradle/PythonHomeStaging.kt | 57 +++++++++++ .../env/PackagedPythonHome.android.kt | 4 + .../multiplatform/env/PackagedPythonHome.kt | 16 +++ .../python/multiplatform/ffi/Python3.kt | 6 ++ .../env/PackagedPythonHome.desktop.kt | 73 ++++++++++++++ .../kotlin/python/native/ffi/Panama.kt | 40 +++++++- .../kotlin/python/native/ffi/bindings.kt | 12 +++ .../env/PackagedPythonHome.jvm.kt | 74 ++++++++++++++ .../kotlin/python/native/ffi/manager.kt | 26 ++++- .../env/PackagedPythonHome.native.kt | 4 + .../env/PackagedPythonHome.wasmJs.kt | 4 + 12 files changed, 389 insertions(+), 25 deletions(-) create mode 100644 python-multiplatform/src/androidMain/kotlin/python/multiplatform/env/PackagedPythonHome.android.kt create mode 100644 python-multiplatform/src/commonMain/kotlin/python/multiplatform/env/PackagedPythonHome.kt create mode 100644 python-multiplatform/src/desktopMain/kotlin/python/multiplatform/env/PackagedPythonHome.desktop.kt create mode 100644 python-multiplatform/src/jvmMain/kotlin/python/multiplatform/env/PackagedPythonHome.jvm.kt create mode 100644 python-multiplatform/src/nativeMain/kotlin/python/multiplatform/env/PackagedPythonHome.native.kt create mode 100644 python-multiplatform/src/wasmJsMain/kotlin/python/multiplatform/env/PackagedPythonHome.wasmJs.kt diff --git a/python-multiplatform-gradle-plugin/src/main/kotlin/python/multiplatform/gradle/PythonBindingsPlugin.kt b/python-multiplatform-gradle-plugin/src/main/kotlin/python/multiplatform/gradle/PythonBindingsPlugin.kt index bcc2bcad4..b51ae2379 100644 --- a/python-multiplatform-gradle-plugin/src/main/kotlin/python/multiplatform/gradle/PythonBindingsPlugin.kt +++ b/python-multiplatform-gradle-plugin/src/main/kotlin/python/multiplatform/gradle/PythonBindingsPlugin.kt @@ -128,6 +128,20 @@ interface PythonBindingsExtension { */ val stagePythonHome: Property + /** + * Whether a packaged desktop application carries the staged CPython prefix -- SPEC L-9, issue + * #60. Defaults to true. + * + * When on, every Compose Desktop `prepare*AppResources` task (what `createDistributable` and + * the `package*` tasks take their resources from) depends on `stagePythonHome` and copies the + * prefix's stdlib and shared library into `python-multiplatform-home/`; the library finds it + * there at run time through Compose's `compose.application.resources.dir`. Independent of + * [stagePythonHome]: an app must ship the prefix this library was built against even on a + * machine whose developer runs with their own `PYTHONHOME`. Turn it off for an app that + * provides a prefix some other way (and names it with `-Dpython.multiplatform.home=...`). + */ + val packagePythonHome: Property + /** CPython version to stage. Defaults to the one this library was built against. */ val pythonVersion: Property @@ -534,7 +548,8 @@ class PythonBindingsPlugin : Plugin { } /** - * Registers `stagePythonHome` and hands its result to every task that launches a JVM. + * Registers `stagePythonHome`, hands its result to every task that launches a JVM, and puts it + * into a packaged desktop application. * * The prefix path is computed here rather than carried as a `Provider`, and it can be: it is a * pure function of the version, the release, the host and the Gradle user home, none of which @@ -543,18 +558,15 @@ class PythonBindingsPlugin : Plugin { * only way CPython's `getenv(3)` and [python.multiplatform.env.PythonHomeCheck]'s * `System.getenv` can be guaranteed to read the same thing. * - * Skipped entirely when `PYTHONHOME` is already set, so a consumer with their own prefix keeps - * it, and when the host is not a platform this library ships for — the latter warns rather + * The `environment(...)` wiring is skipped when `PYTHONHOME` is already set, so a consumer with + * their own prefix keeps it. The task itself is registered regardless -- registration is lazy, + * and it runs only when something depends on it -- because packaging (SPEC L-9) must ship the + * prefix this library was built against whatever the developer's own shell points at. Nothing + * at all happens on a host that is not a platform this library ships for; that warns rather * than fails, because a project may well be building only its Android or iOS targets there. */ - private fun configurePythonHomeStaging(project: Project, extension: PythonBindingsExtension) { + internal fun configurePythonHomeStaging(project: Project, extension: PythonBindingsExtension) { val enabled = extension.stagePythonHome.getOrElse(true) - // `providers.environmentVariable`, not `System.getenv`: the latter reads the *daemon's* - // environment, and a daemon is reused across invocations that do not share one. A user - // running `PYTHONHOME=/their/prefix ./gradlew run` against a daemon that was started - // without it would otherwise be told, silently, that they had set nothing -- and have - // their prefix replaced by a staged one on the very task they were configuring. - if (!shouldStagePythonHome(project.providers.environmentVariable("PYTHONHOME").orNull, enabled)) return val platform = hostDesktopPlatform( System.getProperty("os.name").orEmpty(), @@ -591,15 +603,65 @@ class PythonBindingsPlugin : Plugin { downloadDir.set(java.io.File(cacheRoot, "archives")) } - val home = prefix.absolutePath - project.tasks.withType(org.gradle.api.tasks.JavaExec::class.java).configureEach { - dependsOn(stage) - environment("PYTHONHOME", home) - } - project.tasks.withType(org.gradle.api.tasks.testing.Test::class.java).configureEach { - dependsOn(stage) - environment("PYTHONHOME", home) + // `providers.environmentVariable`, not `System.getenv`: the latter reads the *daemon's* + // environment, and a daemon is reused across invocations that do not share one. A user + // running `PYTHONHOME=/their/prefix ./gradlew run` against a daemon that was started + // without it would otherwise be told, silently, that they had set nothing -- and have + // their prefix replaced by a staged one on the very task they were configuring. + if (shouldStagePythonHome(project.providers.environmentVariable("PYTHONHOME").orNull, enabled)) { + val home = prefix.absolutePath + project.tasks.withType(org.gradle.api.tasks.JavaExec::class.java).configureEach { + dependsOn(stage) + environment("PYTHONHOME", home) + } + project.tasks.withType(org.gradle.api.tasks.testing.Test::class.java).configureEach { + dependsOn(stage) + environment("PYTHONHOME", home) + } } + + configurePackagedPythonHome( + project = project, + extension = extension, + stage = stage, + includes = packagedPrefixIncludes(version, freeThreaded, windows = platform.startsWith("windows")), + ) + } + + /** + * SPEC L-9: copies the staged prefix's stdlib and shared library into every Compose Desktop + * `prepare*AppResources` task under [PACKAGED_HOME_DIRECTORY], so that `createDistributable` + * puts it at `$APPDIR/resources/python-multiplatform-home` -- where the library looks, through + * the `compose.application.resources.dir` property Compose's launcher always sets. + * + * Into Compose's own resources task rather than through `nativeDistributions.appResourcesRootDir`: + * that is a single directory property the consumer may already be using for their own files, + * and setting it here would replace theirs. Adding a `from` to the task that assembles it + * composes with whatever the consumer configured. One copy, too: the prefix goes straight from + * the Gradle-user-home cache into Compose's staging directory, and `Sync` is incremental. + * + * [PythonBindingsExtension.packagePythonHome] is read when the task is *executed* and its + * dependencies computed, not now -- this runs while the plugin is being applied, before the + * consumer's `pythonBindings { }` block has. + */ + private fun configurePackagedPythonHome( + project: Project, + extension: PythonBindingsExtension, + stage: org.gradle.api.tasks.TaskProvider, + includes: List, + ) { + val packaging = project.provider { extension.packagePythonHome.getOrElse(true) } + val stagedPrefix = stage.flatMap { it.destinationDir.dir("python") } + project.tasks.withType(org.gradle.api.tasks.Sync::class.java) + .matching { isComposeAppResourcesTask(it.name) } + .configureEach { + dependsOn(java.util.concurrent.Callable { if (packaging.get()) listOf(stage) else emptyList() }) + from(java.util.concurrent.Callable { if (packaging.get()) stagedPrefix else emptyList() }) { + include(includes) + exclude(PACKAGED_PREFIX_EXCLUDES) + into(PACKAGED_HOME_DIRECTORY) + } + } } /** diff --git a/python-multiplatform-gradle-plugin/src/main/kotlin/python/multiplatform/gradle/PythonHomeStaging.kt b/python-multiplatform-gradle-plugin/src/main/kotlin/python/multiplatform/gradle/PythonHomeStaging.kt index 71166d65d..f9f1e8dc4 100644 --- a/python-multiplatform-gradle-plugin/src/main/kotlin/python/multiplatform/gradle/PythonHomeStaging.kt +++ b/python-multiplatform-gradle-plugin/src/main/kotlin/python/multiplatform/gradle/PythonHomeStaging.kt @@ -60,6 +60,15 @@ import javax.inject.Inject * does, has neither problem: CPython and `PythonHomeCheck` read the same value from the same * place, exactly as they already do for this repository's own `desktopTest`. * + * ### A packaged application (SPEC L-9) + * + * Both points above are about the developer's machine. A packaged app has no Gradle to set its + * environment, so the same staged prefix is also copied into Compose Desktop's application + * resources ([packagedPrefixIncludes], `PythonBindingsPlugin.configurePackagedPythonHome`) and the + * library hands it to CPython in-process with `Py_SetPythonHome` -- which is not an environment + * variable, so the objection above does not apply to it. Only the host's platform is copied, into + * the *application*, never into the library artifact. See `docs/platforms/desktop-packaged-app.md`. + * * ### What it deliberately does not do * * It does not validate the prefix beyond [stdlibMarkerRelativePath], and that marker is the file @@ -195,6 +204,54 @@ internal fun stagingStamp( ): String = "python-multiplatform $pythonVersion+$pbsRelease $platform " + (if (freeThreaded) "freethreaded" else "default") +/** + * The directory a packaged desktop application carries its CPython prefix in, inside Compose + * Desktop's application resources (`$APPDIR/resources/` in a `createDistributable` image) -- + * SPEC L-9, issue #60. + * + * `PackagedPythonHome.DIRECTORY_NAME` in the library's `jvmMain` is the run-time half and must be + * the same string; the two builds share no code, so each side pins it in a test. + */ +internal const val PACKAGED_HOME_DIRECTORY = "python-multiplatform-home" + +/** + * What of a staged prefix a packaged application needs, as include patterns relative to the prefix. + * + * The standard library and the shared library `manager.loadLibPython` loads out of the same prefix. + * Measured for 3.14.7 macos-aarch64 on this machine, `__pycache__` excluded: 1,195 files and + * 39.6 MB (19.4 MB of it `libpython3.14.dylib`) out of the prefix's 68.9 MB. Left out: + * `include/` and `lib/pkgconfig` (build-time only), `bin/` (an embedding app has no `python3` + * executable to run), and Tcl/Tk (`libtcl*`, `tcl9.0/`, `tk9.0/`: `tkinter` is not something a + * Compose app reaches, and they are a further 4 MB of libraries alone). + * + * The Windows layout differs and is matched as the library's own lookups match it: `Lib/` and + * `DLLs/` (extension modules) under the prefix, and the interpreter plus the MSVC runtime as + * `*.dll` at its root, which is where `manager.libraryUnder` looks on that layout. + */ +internal fun packagedPrefixIncludes(pythonVersion: String, freeThreaded: Boolean, windows: Boolean): List { + if (windows) return listOf("Lib/**", "DLLs/**", "*.dll") + val parts = pythonVersion.split('.') + require(parts.size >= 2) { "python version '$pythonVersion' has no major.minor" } + val tag = "${parts[0]}.${parts[1]}" + if (freeThreaded) "t" else "" + return listOf("lib/python$tag/**", "lib/libpython$tag.*") +} + +/** + * Never packaged: the staged prefix is shared by every project on the machine, and CPython writes + * bytecode caches into it from every run that uses it, so its `__pycache__` directories reflect the + * developer's history rather than the build. + */ +internal val PACKAGED_PREFIX_EXCLUDES: List = listOf("**/__pycache__/**") + +/** + * Compose Desktop's `Sync` task that assembles an application's resources: `prepareAppResources` + * for the default build type, `prepareAppResources` for the others (`Release`). Matched + * by name, like the wasm wiring matches `wasmJsProcessResources`, so this plugin carries no + * compile-time dependency on the Compose Gradle plugin; `Sync` itself is Gradle's own type. + */ +internal fun isComposeAppResourcesTask(name: String): Boolean = + Regex("""prepare([A-Z]\w*)?AppResources""").matches(name) + /** * Whether the plugin should stage a prefix and set `PYTHONHOME`. * diff --git a/python-multiplatform/src/androidMain/kotlin/python/multiplatform/env/PackagedPythonHome.android.kt b/python-multiplatform/src/androidMain/kotlin/python/multiplatform/env/PackagedPythonHome.android.kt new file mode 100644 index 000000000..8c1f08419 --- /dev/null +++ b/python-multiplatform/src/androidMain/kotlin/python/multiplatform/env/PackagedPythonHome.android.kt @@ -0,0 +1,4 @@ +package python.multiplatform.env + +/** No-op: `PythonBootstrap` stages the stdlib and sets `PYTHONHOME` with `Os.setenv`. */ +internal actual fun applyPackagedPythonHome() = Unit diff --git a/python-multiplatform/src/commonMain/kotlin/python/multiplatform/env/PackagedPythonHome.kt b/python-multiplatform/src/commonMain/kotlin/python/multiplatform/env/PackagedPythonHome.kt new file mode 100644 index 000000000..086235884 --- /dev/null +++ b/python-multiplatform/src/commonMain/kotlin/python/multiplatform/env/PackagedPythonHome.kt @@ -0,0 +1,16 @@ +package python.multiplatform.env + +/** + * Points CPython at the prefix a packaged application ships, when there is one -- SPEC L-9. + * + * Called once by [python.multiplatform.ffi.Python3.initialize], after [PythonHomeCheck] and before + * `Py_Initialize()`, which is the only window in which a home can still be chosen. Only desktop + * does anything: a packaged desktop app is started by a launcher that cannot put `PYTHONHOME` in + * its environment (see `desktopMain/README.md`). Android sets `PYTHONHOME` with `Os.setenv` in + * `PythonBootstrap`, iOS and Android-native processes have a real environment, and wasm bakes its + * stdlib into the virtual filesystem, so each of those is a no-op. + * + * Must throw, rather than continue, when the packaged prefix is unusable: the next call is + * `Py_Initialize()`, which aborts the process instead of failing. + */ +internal expect fun applyPackagedPythonHome() diff --git a/python-multiplatform/src/commonMain/kotlin/python/multiplatform/ffi/Python3.kt b/python-multiplatform/src/commonMain/kotlin/python/multiplatform/ffi/Python3.kt index fbd3095e1..48aa9cc63 100644 --- a/python-multiplatform/src/commonMain/kotlin/python/multiplatform/ffi/Python3.kt +++ b/python-multiplatform/src/commonMain/kotlin/python/multiplatform/ffi/Python3.kt @@ -3,6 +3,7 @@ package python.multiplatform.ffi import python.multiplatform.BuildConfig import python.multiplatform.env.PythonHomeCheck import python.multiplatform.env.PythonPayload +import python.multiplatform.env.applyPackagedPythonHome import python.multiplatform.ffi.exceptions.PyException import python.multiplatform.ffi.types.modules.PyModule import python.native.ffi.* @@ -57,6 +58,11 @@ object Python3 { // PythonHomeCheck's doc comment). Cheap even so: one or two filesystem probes against a // handful of candidate paths, not a directory walk. PythonHomeCheck.verifyOrThrow() + // A packaged desktop app has no PYTHONHOME in its environment and hands its own prefix to + // CPython here instead (SPEC L-9); a no-op everywhere else, and whenever PYTHONHOME is set. + // Before Py_Initialize() because that is the last point a home can be chosen, and it + // throws rather than continuing because Py_Initialize() would abort instead. + applyPackagedPythonHome() memScoped { Py_Initialize() // There is nothing richer to report than this. `Py_Initialize()` returns void and, diff --git a/python-multiplatform/src/desktopMain/kotlin/python/multiplatform/env/PackagedPythonHome.desktop.kt b/python-multiplatform/src/desktopMain/kotlin/python/multiplatform/env/PackagedPythonHome.desktop.kt new file mode 100644 index 000000000..1621c2584 --- /dev/null +++ b/python-multiplatform/src/desktopMain/kotlin/python/multiplatform/env/PackagedPythonHome.desktop.kt @@ -0,0 +1,73 @@ +package python.multiplatform.env + +import python.multiplatform.OSType +import python.multiplatform.currentPlatform +import python.native.ffi.Panama +import python.native.ffi.bindings +import java.io.ByteArrayOutputStream +import java.nio.ByteOrder + +/** + * Desktop's half of SPEC L-9: hand a packaged prefix to CPython with `Py_SetPythonHome`. + * + * ### Why in-process, when `desktopMain/README.md` says `PYTHONHOME` is set as the process starts + * + * That rule is about `PYTHONHOME` the *environment variable*: a JVM cannot set one for itself, so + * the Gradle plugin sets it on `run`/`test`. A packaged application has no Gradle to do that, and + * its launcher (jpackage, and Compose Desktop's tasks on top of it) takes `--java-options`, not + * environment variables. `Py_SetPythonHome` is not an environment variable at all: it writes the + * home straight into CPython's path configuration, which `Py_Initialize()` reads ahead of `getenv` + * whenever the variable is unset. So nothing is set that `PythonHomeCheck` could fail to see -- + * the prefix is checked here, on the same value CPython is given, immediately before giving it. + * + * ### The buffer is never freed + * + * The C API asks for a string "in static storage whose contents will not change for the duration + * of the program's execution". Current CPython copies it, but that is an implementation detail the + * documentation does not promise, so the few hundred bytes stay allocated for the life of the + * process. This runs once per process (`Python3.initialize` reaches it only before the first + * `Py_Initialize`). + */ +internal actual fun applyPackagedPythonHome() { + val home = PackagedPythonHome.resolve() ?: return + try { + PythonHomeCheck.verifyOrThrow(home.path) + } catch (e: IllegalStateException) { + throw IllegalStateException( + "The Python home this application was packaged with (${home.source}) is unusable: ${e.message}", + e, + ) + } + val wide = encodeWideString(home.path, windows = currentPlatform.os == OSType.Windows) + bindings.Py_SetPythonHome(Panama.allocateBytesFreeable(wide)) +} + +/** + * [value] as a NUL-terminated C `wchar_t` string: UTF-16 (2-byte terminator) on Windows, UTF-32 + * (4-byte terminator) on macOS and Linux, in [order]. + * + * Encoded by hand rather than through `Py_DecodeLocale`, which before CPython's pre-initialisation + * decodes with the C locale's encoding -- the ANSI code page on Windows -- and would turn a + * non-ASCII install path (`C:\Users\José\...`) into a different one. UTF-32 is written code point + * by code point rather than through the `UTF-32` charset, which a GraalVM native image does not + * include by default. + */ +internal fun encodeWideString(value: String, windows: Boolean, order: ByteOrder = ByteOrder.nativeOrder()): ByteArray { + if (windows) { + val charset = if (order == ByteOrder.LITTLE_ENDIAN) Charsets.UTF_16LE else Charsets.UTF_16BE + return value.toByteArray(charset) + ByteArray(2) + } + val out = ByteArrayOutputStream((value.length + 1) * 4) + val writeUnit = { unit: Int -> + if (order == ByteOrder.LITTLE_ENDIAN) { + out.write(unit and 0xFF); out.write(unit ushr 8 and 0xFF) + out.write(unit ushr 16 and 0xFF); out.write(unit ushr 24 and 0xFF) + } else { + out.write(unit ushr 24 and 0xFF); out.write(unit ushr 16 and 0xFF) + out.write(unit ushr 8 and 0xFF); out.write(unit and 0xFF) + } + } + value.codePoints().forEach { writeUnit(it) } + writeUnit(0) + return out.toByteArray() +} diff --git a/python-multiplatform/src/desktopMain/kotlin/python/native/ffi/Panama.kt b/python-multiplatform/src/desktopMain/kotlin/python/native/ffi/Panama.kt index 1cca8976a..475b9c9b7 100644 --- a/python-multiplatform/src/desktopMain/kotlin/python/native/ffi/Panama.kt +++ b/python-multiplatform/src/desktopMain/kotlin/python/native/ffi/Panama.kt @@ -81,6 +81,14 @@ internal object Panama { /** Release a slot obtained from [allocatePointerSlot]. */ val freePointerSlot: (Long) -> Unit + /** Copy [ByteArray] verbatim into a new native `malloc` buffer and return its address. + * + * For C strings that are not UTF-8: `Py_SetPythonHome` takes a `wchar_t *`, which is UTF-32 + * on macOS and Linux and UTF-16 on Windows, so the caller encodes (terminator included) and + * this only places the bytes. Release with [freeUtf8Address] (both are `malloc`). Not + * available on the JDK 16-18 incubator backend, for the reason the pointer slots are not. */ + val allocateBytesFreeable: (ByteArray) -> Long + /** One `MethodHandle` per ABI shape, **unbound** to any specific target function: its * leading parameter is the callee's address (as a plain `long`, adapted from * `MemorySegment`/`MemoryAddress` so the handle's static Java type is exactly @@ -139,6 +147,7 @@ internal object Panama { allocatePointerSlot = data.allocatePointerSlot readPointerSlot = data.readPointerSlot freePointerSlot = data.freePointerSlot + allocateBytesFreeable = data.allocateBytesFreeable unboundDowncallHandle = data.unboundDowncallHandle createUpcallStubLongToLong = data.createUpcallStubLongToLong createUpcallStubII_L = data.createUpcallStubII_L @@ -157,6 +166,7 @@ internal object Panama { allocatePointerSlot = data.allocatePointerSlot readPointerSlot = data.readPointerSlot freePointerSlot = data.freePointerSlot + allocateBytesFreeable = data.allocateBytesFreeable unboundDowncallHandle = data.unboundDowncallHandle createUpcallStubLongToLong = data.createUpcallStubLongToLong createUpcallStubII_L = data.createUpcallStubII_L @@ -179,6 +189,7 @@ internal object Panama { val allocatePointerSlot: () -> Long, val readPointerSlot: (Long) -> Long, val freePointerSlot: (Long) -> Unit, + val allocateBytesFreeable: (ByteArray) -> Long, val unboundDowncallHandle: (Int, Int, ReturnKind) -> MethodHandle, val createUpcallStubLongToLong: (MethodHandle) -> Long, val createUpcallStubII_L: (MethodHandle) -> Long, @@ -475,6 +486,21 @@ internal object Panama { val freeSlot: (Long) -> Unit = freeAddr + // ---- Arbitrary bytes into a malloc buffer (non-UTF-8 C strings: wchar_t) ---- + // + // `allocFreeable` above, minus its UTF-8 encoding: the caller hands over the finished + // bytes, terminator included. Kept separate rather than having `allocFreeable` delegate to + // it, so the per-string path that every `withUtf8` call takes gains no extra indirection. + val allocBytes: (ByteArray) -> Long = { bytes -> + val len = bytes.size.toLong() + val addr = mallocHandle.invokeExact(len) as Long + if (addr == 0L) throw OutOfMemoryError("native malloc failed for a $len-byte buffer") + val srcSeg: Any = ofArrayExact.invokeExact(bytes) as Any + val dstSeg: Any = addressToSegmentExact.invokeExact(addr, len) as Any + copyExact.invokeExact(srcSeg, 0L, dstSeg, 0L, len) as Unit + addr + } + // ---- Shape vocabulary: unbound downcall handles, one per (intArgs, floatArgs, returnKind) ---- val downcallHandle2Method = linkerClass.getMethod("downcallHandle", functionDescriptorClass, optionArrayType) @@ -560,7 +586,7 @@ internal object Panama { segmentAddressExact.invokeExact(stub) as Long } - return ModernData(allocStr, readStr, findSym, findAddr, allocFreeable, freeAddr, allocSlot, readSlot, freeSlot, buildShape, buildUpcallStub, buildUpcallStubII_L, buildUpcallStubIII_I, buildUpcallStubI_I, buildUpcallStubI_V) + return ModernData(allocStr, readStr, findSym, findAddr, allocFreeable, freeAddr, allocSlot, readSlot, freeSlot, allocBytes, buildShape, buildUpcallStub, buildUpcallStubII_L, buildUpcallStubIII_I, buildUpcallStubI_I, buildUpcallStubI_V) } private fun adaptModernHandle( @@ -632,6 +658,7 @@ internal object Panama { val allocatePointerSlot: () -> Long, val readPointerSlot: (Long) -> Long, val freePointerSlot: (Long) -> Unit, + val allocateBytesFreeable: (ByteArray) -> Long, val unboundDowncallHandle: (Int, Int, ReturnKind) -> MethodHandle, val createUpcallStubLongToLong: (MethodHandle) -> Long, val createUpcallStubII_L: (MethodHandle) -> Long, @@ -818,6 +845,15 @@ internal object Panama { val readSlot: (Long) -> Long = { slotsUnsupported() } val freeSlot: (Long) -> Unit = { slotsUnsupported() } + // Same reason as the slots: this backend writes native memory only through `toCString`, + // which cannot place a wchar_t string. JDK 19+ resolves the modern backend. + val allocBytes: (ByteArray) -> Long = { + throw UnsupportedOperationException( + "Writing arbitrary bytes to native memory is not implemented on the " + + "jdk.incubator.foreign backend (JDK 16-18). Run on JDK 19 or newer." + ) + } + val freeAddr: (Long) -> Unit = { addr -> freeableScopes.remove(addr)?.let { scopeCloseMethod.invoke(it) } Unit @@ -910,7 +946,7 @@ internal object Panama { toRawLongMethod.invoke(segAddressMethod.invoke(stub)) as Long } - return IncubatorData(allocStr, readStr, findSym, findAddr, allocFreeable, freeAddr, allocSlot, readSlot, freeSlot, buildShape, buildUpcallStub, buildUpcallStubII_L, buildUpcallStubIII_I, buildUpcallStubI_I, buildUpcallStubI_V) + return IncubatorData(allocStr, readStr, findSym, findAddr, allocFreeable, freeAddr, allocSlot, readSlot, freeSlot, allocBytes, buildShape, buildUpcallStub, buildUpcallStubII_L, buildUpcallStubIII_I, buildUpcallStubI_I, buildUpcallStubI_V) } private fun adaptIncubatorHandle( diff --git a/python-multiplatform/src/desktopMain/kotlin/python/native/ffi/bindings.kt b/python-multiplatform/src/desktopMain/kotlin/python/native/ffi/bindings.kt index 36ed1d1b4..62030d26e 100644 --- a/python-multiplatform/src/desktopMain/kotlin/python/native/ffi/bindings.kt +++ b/python-multiplatform/src/desktopMain/kotlin/python/native/ffi/bindings.kt @@ -14,6 +14,17 @@ object bindings { inline fun Py_Initialize() = Py_InitializeHandle.invokeExact() as Unit val Py_InitializeExHandle: MethodHandle inline fun Py_InitializeEx(initsigs: Int) = Py_InitializeExHandle.invokeExact(initsigs) as Unit + /** + * `void Py_SetPythonHome(const wchar_t *home)` -- desktop only, not in `EmbedAPI.kt`. + * + * How a packaged desktop app, whose launcher cannot set `PYTHONHOME`, names its prefix before + * `Py_Initialize()` (SPEC L-9, `applyPackagedPythonHome`). Deprecated in 3.11 in favour of + * `PyConfig.home`, but `PyConfig` is a struct whose layout the Stable ABI does not promise (see + * `Python3.initialize`), and this function is part of the Stable ABI, so its symbol stays + * exported. Takes a `wchar_t *`, not UTF-8: see `encodeWideString`. + */ + val Py_SetPythonHomeHandle: MethodHandle + inline fun Py_SetPythonHome(home: Long) = Py_SetPythonHomeHandle.invokeExact(home) as Unit val Py_IsInitializedHandle: MethodHandle inline fun Py_IsInitialized(): Int = Py_IsInitializedHandle.invokeExact() as Int val Py_IsFinalizingHandle: MethodHandle @@ -930,6 +941,7 @@ inline fun PyDict_GetItemString(p: Long, key: String): Long { // Section 1 Py_InitializeHandle = find("Py_Initialize", Void.TYPE) Py_InitializeExHandle = find("Py_InitializeEx", Void.TYPE, Integer.TYPE) + Py_SetPythonHomeHandle = find("Py_SetPythonHome", Void.TYPE, P) Py_IsInitializedHandle = find("Py_IsInitialized", Integer.TYPE) Py_IsFinalizingHandle = find("Py_IsFinalizing", Integer.TYPE) Py_FinalizeExHandle = find("Py_FinalizeEx", Integer.TYPE) diff --git a/python-multiplatform/src/jvmMain/kotlin/python/multiplatform/env/PackagedPythonHome.jvm.kt b/python-multiplatform/src/jvmMain/kotlin/python/multiplatform/env/PackagedPythonHome.jvm.kt new file mode 100644 index 000000000..4cc21bb98 --- /dev/null +++ b/python-multiplatform/src/jvmMain/kotlin/python/multiplatform/env/PackagedPythonHome.jvm.kt @@ -0,0 +1,74 @@ +package python.multiplatform.env + +import java.io.File + +/** + * Where a **packaged** desktop application's CPython prefix is -- SPEC L-9, issue #60. + * + * During development the Gradle plugin's `stagePythonHome` sets `PYTHONHOME` on every `run` and + * `test` task, and CPython reads it with `getenv(3)`. A packaged application (`createDistributable`, + * an installer, a plain `java -jar`) is launched by something else, and nothing puts `PYTHONHOME` in + * its environment: jpackage launchers accept `--java-options`, not environment variables. What they + * *can* carry is a system property, and jpackage expands `$APPDIR` inside one. So a packaged prefix + * is named by a system property and handed to CPython in-process, with `Py_SetPythonHome`, before + * `Py_Initialize` (`applyPackagedPythonHome` in `desktopMain`). + * + * Lives in `jvmMain` rather than `desktopMain` only because `manager.loadLibPython` -- also + * `jvmMain` -- loads `libpython` out of the same prefix. It reads nothing Android sets, so on Android + * it always resolves to null. + */ +internal object PackagedPythonHome { + + /** An explicit prefix, e.g. `-Dpython.multiplatform.home=$APPDIR/python-multiplatform-home`. */ + const val HOME_PROPERTY = "python.multiplatform.home" + + /** + * Compose Desktop sets this on every launcher it builds (`$APPDIR/resources` in a packaged app) + * and on its own `run` task. The Gradle plugin copies the prefix into those resources under + * [DIRECTORY_NAME], so a Compose app needs no property of its own. + */ + const val COMPOSE_RESOURCES_PROPERTY = "compose.application.resources.dir" + + /** `PACKAGED_HOME_DIRECTORY` in `python-multiplatform-gradle-plugin`; the two must agree. */ + const val DIRECTORY_NAME = "python-multiplatform-home" + + /** A resolved prefix and a human-readable account of where it came from, for error messages. */ + data class Resolved(val path: String, val source: String) + + /** [resolve] against this process's environment and system properties. */ + fun resolve(): Resolved? = resolve( + environmentHome = System.getenv("PYTHONHOME"), + property = System::getProperty, + isDirectory = { File(it).isDirectory }, + ) + + /** + * The packaged prefix, or null when there is none to apply. + * + * 1. A non-blank `PYTHONHOME` in the environment wins, and the answer is null: CPython reads that + * variable itself and `PythonHomeCheck` already checks it. Overriding it would make a launcher + * property beat a developer's (or a test task's) explicit choice. + * 2. [HOME_PROPERTY], whether or not the path exists -- a wrong explicit path must reach + * `PythonHomeCheck` and be reported by name, not silently ignored into a `Py_FatalError`. + * 3. `<`[COMPOSE_RESOURCES_PROPERTY]`>/`[DIRECTORY_NAME], **only if that directory exists**: + * every Compose app has the property, including one whose build did not package a prefix, + * and that app must behave exactly as it did before this existed. + */ + fun resolve( + environmentHome: String?, + property: (String) -> String?, + isDirectory: (String) -> Boolean, + ): Resolved? { + if (!environmentHome.isNullOrBlank()) return null + property(HOME_PROPERTY)?.takeIf { it.isNotBlank() }?.let { + return Resolved(it, "system property $HOME_PROPERTY") + } + val resources = property(COMPOSE_RESOURCES_PROPERTY)?.takeIf { it.isNotBlank() } ?: return null + val candidate = File(resources, DIRECTORY_NAME).path + return if (isDirectory(candidate)) { + Resolved(candidate, "$DIRECTORY_NAME under system property $COMPOSE_RESOURCES_PROPERTY") + } else { + null + } + } +} diff --git a/python-multiplatform/src/jvmMain/kotlin/python/native/ffi/manager.kt b/python-multiplatform/src/jvmMain/kotlin/python/native/ffi/manager.kt index a04082fcb..242503b4b 100644 --- a/python-multiplatform/src/jvmMain/kotlin/python/native/ffi/manager.kt +++ b/python-multiplatform/src/jvmMain/kotlin/python/native/ffi/manager.kt @@ -3,6 +3,7 @@ package python.native.ffi import python.multiplatform.OSType import python.multiplatform.Versions import python.multiplatform.currentPlatform +import python.multiplatform.env.PackagedPythonHome import java.io.File import java.io.FileOutputStream import java.util.* @@ -64,9 +65,11 @@ internal object manager { * extraction path has: the stdlib comes from `PYTHONHOME` while the binary came from the jar, * so the two can disagree about the version. * - * Only [LIBPYTHON_OVERRIDE_ENV] takes precedence over the classpath copy. The `PYTHONHOME` - * probe runs *after* it, and only when the resource is absent, so a classpath deployment loads - * exactly the library it always did. + * Precedence: [LIBPYTHON_OVERRIDE_ENV]; then a packaged application's prefix + * ([PackagedPythonHome], SPEC L-9) when it holds the library; then the classpath copy. The + * `PYTHONHOME` probe runs last, and only when the resource is absent. [PackagedPythonHome] + * resolves to nothing while `PYTHONHOME` is set, so a development or test run loads exactly + * the library it always did. */ private fun loadFromSidecar(libraryName: String, prerequisites: List = emptyList()): Boolean { val override = System.getenv(LIBPYTHON_OVERRIDE_ENV) @@ -77,8 +80,16 @@ internal object manager { } } } else { - if (hasBundledLibrary(libraryName)) return false - resolveSidecarLibrary(libraryName) + // A packaged application's own prefix (SPEC L-9) comes before the classpath copy. That + // copy is extracted into the *working directory* and then loaded through + // `java.library.path` -- which works for this build's tests because they pass + // `-Djava.library.path=.`, and does not for an app a desktop launches with `/` as its + // working directory. Loading the library that sits beside the stdlib also keeps the + // two from coming from different builds. Null whenever PYTHONHOME is set, so a + // development run loads exactly what it always did. + val packaged = PackagedPythonHome.resolve()?.let { libraryUnder(it.path, libraryName) } + if (packaged == null && hasBundledLibrary(libraryName)) return false + packaged ?: resolveSidecarLibrary(libraryName) ?: throw UnsatisfiedLinkError( "CPython is neither bundled on the classpath nor present under PYTHONHOME" + (System.getenv("PYTHONHOME")?.let { " ($it)" } ?: " (PYTHONHOME is unset)") + @@ -102,6 +113,11 @@ internal object manager { */ internal fun resolveSidecarLibrary(libraryName: String): File? { val home = System.getenv("PYTHONHOME")?.takeIf { it.isNotBlank() } ?: return null + return libraryUnder(home, libraryName) + } + + /** [libraryName]'s shared library under the prefix [home], in either layout; null if absent. */ + internal fun libraryUnder(home: String, libraryName: String): File? { val fileName = System.mapLibraryName(libraryName) return sequenceOf( File(home, "lib/$fileName"), diff --git a/python-multiplatform/src/nativeMain/kotlin/python/multiplatform/env/PackagedPythonHome.native.kt b/python-multiplatform/src/nativeMain/kotlin/python/multiplatform/env/PackagedPythonHome.native.kt new file mode 100644 index 000000000..8b02c1726 --- /dev/null +++ b/python-multiplatform/src/nativeMain/kotlin/python/multiplatform/env/PackagedPythonHome.native.kt @@ -0,0 +1,4 @@ +package python.multiplatform.env + +/** No-op: iOS and Android-native processes take `PYTHONHOME` from a real process environment. */ +internal actual fun applyPackagedPythonHome() = Unit diff --git a/python-multiplatform/src/wasmJsMain/kotlin/python/multiplatform/env/PackagedPythonHome.wasmJs.kt b/python-multiplatform/src/wasmJsMain/kotlin/python/multiplatform/env/PackagedPythonHome.wasmJs.kt new file mode 100644 index 000000000..68917197d --- /dev/null +++ b/python-multiplatform/src/wasmJsMain/kotlin/python/multiplatform/env/PackagedPythonHome.wasmJs.kt @@ -0,0 +1,4 @@ +package python.multiplatform.env + +/** No-op: the wasm runtime's stdlib is baked into its virtual filesystem. */ +internal actual fun applyPackagedPythonHome() = Unit From b8cb969071a21f5ec13153ec60d5144c2272a5c2 Mon Sep 17 00:00:00 2001 From: BREW <81044362+b-re-w@users.noreply.github.com> Date: Sat, 3 Oct 2026 14:02:07 +0900 Subject: [PATCH 4/4] Fix: Clean the launch test's scratch dir without following its link to the real prefix (#60) deleteRecursively followed the symlink to the staged CPython prefix and emptied it, failing 159 desktop tests that ran after it. --- .../env/PackagedPythonHomeLaunchTest.kt | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/env/PackagedPythonHomeLaunchTest.kt b/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/env/PackagedPythonHomeLaunchTest.kt index d9ddabdf5..c0394292c 100644 --- a/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/env/PackagedPythonHomeLaunchTest.kt +++ b/python-multiplatform/src/desktopTest/kotlin/python/multiplatform/env/PackagedPythonHomeLaunchTest.kt @@ -33,10 +33,23 @@ class PackagedPythonHomeLaunchTest { @AfterTest fun cleanup() { - scratchDirs.forEach { it.deleteRecursively() } + scratchDirs.forEach { deleteWithoutFollowingLinks(it) } scratchDirs.clear() } + /** + * A scratch directory here holds a symbolic link to the *real* staged prefix. `File.deleteRecursively` + * follows it and empties the prefix every other test in this JVM (and the next run) depends on -- + * it did, and 159 desktop tests failed with "no readable standard library". `Files.walk` without + * `FOLLOW_LINKS` lists the link itself, so the link is deleted and its target is not. + */ + private fun deleteWithoutFollowingLinks(dir: File) { + if (!Files.exists(dir.toPath(), java.nio.file.LinkOption.NOFOLLOW_LINKS)) return + Files.walk(dir.toPath()).use { paths -> + paths.sorted(Comparator.reverseOrder()).forEach { Files.delete(it) } + } + } + /** Under `build/`, not the system temporary directory -- AGENTS.md rule 2. */ private fun newScratchDir(): File = File("build/tmp/packagedPythonHomeLaunchTest/${java.util.UUID.randomUUID()}").absoluteFile