diff --git a/docs/SPEC.md b/docs/SPEC.md index 181a76c95..ae73cbc9f 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -69,7 +69,8 @@ fail, then implement. (`PM/desktopTest/.../env/PythonPayloadTest.kt`) and Android (`PythonPayloadStagingTest.kt`). - **L-7** iOS: the framework carries no stdlib, so `PYTHONHOME` must point at one; the test build extracts it. `Status: partial` — exercised by the shared suite on the simulator; no iOS-specific - lifecycle test beyond `PM/iosSimulatorArm64Test/.../AsyncioAvailabilityProbeTest.kt`. + lifecycle test beyond `PM/iosSimulatorArm64Test/.../AsyncioAvailabilityProbeTest.kt`. An installed + app carries its own stdlib: L-11. - **L-8** Initialisation sets `builtins.compiled` to an identity decorator (an existing one is kept), so `@compiled` needs no import on any platform (N-7, issue #42). `Status: implemented` — `PM/commonTest/.../ffi/BuiltinCompiledTest.kt`. - **L-9** Desktop: a packaged application starts with no `PYTHONHOME` in its environment (issue #60, ROADMAP §15.4). The Gradle plugin copies the prefix `stagePythonHome` stages into Compose Desktop's @@ -111,6 +112,27 @@ fail, then implement. `--limit-modules java.base` installs the proxy base, and a dropped subclass instance releases its handle). The packaged sample's own log is a manual check — `docs/platforms/desktop-packaged-app.md`. +- **L-11** iOS: an installed app starts with no `PYTHONHOME` in its environment, and + `Python.xcframework` carries no standard library (issue #59, ROADMAP §13.1). The library build + stages one prefix per slice (`iphoneos-arm64`, `iphonesimulator-arm64`, `iphonesimulator-x86_64`): + the shared `Python.xcframework/lib/python/` merged with that slice's + `lib-/python/` (`lib-dynload/`, `_sysconfigdata`), without `__pycache__`, + `libpython*.dylib`, or the top-level packages `test`, `idlelib`, `tkinter`, `turtledemo` and + `ensurepip`; `stageIosPythonHomeForXcode` picks the slice from Xcode's `EFFECTIVE_PLATFORM_NAME` + and `ARCHS` and prints `PYTHON_HOME_DIR=` and `PYTHON_DYLIB_INFO_TEMPLATE=`. An Xcode Run Script + phase (`tools/xcode/install-python.sh`, after Copy Bundle Resources and before Embed Frameworks) + copies that prefix into `/python-multiplatform-home/` and the consumer's payload (L-6) into + `/python/`, and moves every `.so` under either into `Frameworks/.framework`, + leaving a `.fwork` placeholder, as CPython's `AppleFrameworkLoader` and Apple's rule against loose + binaries require. At run time, before `Py_Initialize`, iOS takes the first of: `PYTHONHOME` from + the environment (CPython reads it itself and nothing else happens); + `/python-multiplatform-home` when that directory exists. A + bundled prefix goes through L-3's check (a catchable `IllegalStateException` naming the path and + where it came from) and is handed to CPython with `Py_SetPythonHome`. With neither, nothing is set + and behaviour is as before. `Status: partial` — `GP/IosPythonHomeLayoutTest.kt` (slice choice, + source directories, file selection, a staged fake tree); `PM/iosSimulatorArm64Test/.../env/IosPythonHomeTest.kt` + (resolution order; the simulator test task's `PYTHONHOME` still wins). The installed-app run on the + simulator is a manual check — `docs/platforms/ios-app-bundle.md`. ## 2. Low-level C API (downcall surface) diff --git a/docs/platforms/ios-app-bundle.md b/docs/platforms/ios-app-bundle.md new file mode 100644 index 000000000..daf78fd55 --- /dev/null +++ b/docs/platforms/ios-app-bundle.md @@ -0,0 +1,181 @@ +# iOS: the standard library and the payload inside the app bundle + +SPEC L-11, issue #59, ROADMAP §13.1. + +`Python.xcframework` carries the interpreter and headers and no standard library, and an installed +app is launched with no `PYTHONHOME`. Until now only the simulator *test* binary had a stdlib +(`extractIosSimulatorStdlib` + `SIMCTL_CHILD_PYTHONHOME`), and that recipe does not carry over to an +app: a `PYTHONHOME` on this workspace's external volume parks the sandboxed app in `open()` at 0% CPU +(`python-multiplatform/src/iosMain/README.md`). This page records how an app now carries its own +prefix and payload, why, and how to check it. + +## What happens + +**Gradle** (`python-multiplatform/build.gradle.kts`, rules in the plugin's `IosPythonHomeLayout`). +One task per slice, `stageIosPythonHome__` (`iphoneos_arm64`, `iphonesimulator_arm64`, +`iphonesimulator_x86_64`), writes `python-multiplatform/build/python-ios-home/-/`: + +| Path | From `Python.xcframework/` | +|---|---| +| `home/lib/python/` | `lib/python/` (pure Python, every slice) merged with `/lib-/python/` (`lib-dynload/*.so`, `_sysconfigdata__ios_*`) | +| `dylib-Info-template.plist` | `build/iOS-dylib-Info-template.plist` (input to the Xcode phase; not shipped) | + +Not copied: `__pycache__`, `libpython*.dylib`, and the top-level packages `test` (36 MB and 1,641 of +the 2,407 files of the 3.14 stdlib), `idlelib`, `tkinter`, `turtledemo` (no `_tkinter` on iOS) and +`ensurepip` (pip cannot run in an app). `stageIosPythonHomeForXcode` reads `EFFECTIVE_PLATFORM_NAME` +and `ARCHS` from the environment Xcode gives a Run Script phase, stages only that slice and prints: + +``` +PYTHON_HOME_DIR=/python-multiplatform/build/python-ios-home/iphonesimulator-arm64/home +PYTHON_DYLIB_INFO_TEMPLATE=/python-multiplatform/build/python-ios-home/iphonesimulator-arm64/dylib-Info-template.plist +``` + +A universal `ARCHS` (`arm64 x86_64`) is refused with the fix (`ONLY_ACTIVE_ARCH=YES` or `ARCHS=arm64`): +one prefix holds one `lib-dynload`. Upstream's `utils.sh` has the same limit without saying so. + +**Xcode** (`tools/xcode/install-python.sh`, a Run Script phase after Copy Bundle Resources and before +Embed Frameworks; `ENABLE_USER_SCRIPT_SANDBOXING = NO`). + +1. Runs Gradle as above. The output is captured, then parsed, then checked to be a directory — a + failing Gradle stops the phase and never leaves `rsync` an empty source (issue #59's comment). +2. `rsync -a --delete` the prefix to `/python-multiplatform-home/`. +3. The payload, if configured, to `/python/` (`--delete`, without `__pycache__`): + `PYTHON_PAYLOAD_DIR=` (a directory whose contents are the payload root — the app's own Python + sources, or toolchain's `build/pythonStaging/ios/python`), or `PYTHON_PAYLOAD_TASK=` + that prints `PYTHON_PAYLOAD_DIR=` under `-q` (toolchain's `:app:stagePythonBundleIosForXcode`). + Neither set: no payload, and an old `/python/` is removed. +4. Every `.so` under both directories becomes `Frameworks/.framework/` + with an `Info.plist` from the template, a `.fwork` placeholder where the `.so` was (its + content is the framework binary's path relative to the app), and a `.origin` back-reference; + each framework is signed with `EXPANDED_CODE_SIGN_IDENTITY` when there is one. + +**Run time** (`python-multiplatform/src/iosMain/.../env/PackagedPythonHome.ios.kt`). +`Python3.initialize()` calls `applyPackagedPythonHome()` after the `PYTHONHOME` pre-flight check +(L-3) and before `Py_Initialize()`. `IosPythonHome.resolve()` takes the first of: + +1. `PYTHONHOME` in the environment — nothing else happens. The simulator test task and an Xcode + scheme variable keep working as before. +2. `/python-multiplatform-home`, only if that directory exists. + +A bundled prefix is checked with `PythonHomeCheck` (an `IllegalStateException` naming the path and +"the app bundle's resource directory"), converted with `Py_DecodeLocale` (UTF-8 on Apple platforms) +and passed to `Py_SetPythonHome`. `PythonPayload` then finds `/python` and puts it on +`sys.path`, as before. + +## Why this design + +**Upstream's framework rule, not loose `.so` files.** iOS does not load a binary that is not inside a +framework on a device, and App Store validation rejects one. CPython 3.13+ on iOS has +`importlib.machinery.AppleFrameworkLoader`: for `foo.cpython-314-iphoneos.fwork` it reads the file, +joins it to `dirname(sys.executable)` and loads that binary. The script is upstream's +`Python.xcframework/build/utils.sh` `install_dylib`, with two changes: the template comes from the +Gradle staging (so the phase needs no path into the archive), and signing is skipped when Xcode +exports no identity instead of failing. The simulator would load a loose `.so`; the device would not, +so the simulator run uses the device's layout. + +**`python-multiplatform-home/`, not upstream's `python/`.** Upstream's testbed sets +`config.home = /python`. Here `python/` is the consumer's payload root +(`PythonPayload.PAYLOAD_ROOT`, L-6), and the payload copy is `rsync --delete`: one directory for +both would let each copy delete the other. The name is the one desktop already uses (L-9). + +**Staged per slice by Gradle, copied by a thin script.** The stdlib merge (shared tree + one +`lib-` tree) and the exclusions are pure functions with unit tests +(`GP/IosPythonHomeLayoutTest.kt`); the script only copies, wraps and signs. The selection lives in the +Gradle plugin because the library build script already imports the plugin's public layout objects +(`CPythonIncludeLayout`) and a build script cannot be unit-tested. + +**`Py_SetPythonHome`, not `setenv`.** Same reasoning as desktop: it sets CPython's path +configuration for this process only, and it is checked by the same `PythonHomeCheck` immediately +before. `PyConfig.home` is the non-deprecated route, but the library carries no `PyConfig` layout. + +## Wiring an app + +The sample's project is `iosApp/iosApp.xcodeproj` (product `PythonDemo.app`, bundle id +`org.thisisthepy.python.multiplatform.demo`). Its phase: + +```sh +set -e +cd "$SRCROOT/.." +export PYTHON_PAYLOAD_DIR="$SRCROOT/../sample/python/src/main" +/bin/bash tools/xcode/install-python.sh +``` + +It replaced two phases: "Install Target Specific Python Standard Library", which rsynced the slice's +`lib/` — only `libpython3.14.dylib` — into `/lib/`, and "Prepare Python Binary Modules", which +did the framework wrapping from a template copied in as a resource. `iosApp/iosApp/dylib-Info-template.plist` +is no longer copied into the app; the file itself is still in the project. + +`sample/src/iosMain/app.xcodeproj` is not wired: it still references `../../dist/toolchain/...` and +a `ComposeApp` framework that this build does not produce, and has never been built here. + +A consumer outside this repository cannot call `:python-multiplatform:stageIosPythonHomeForXcode`: +the iOS archive download lives in this repository's build script, not in the published plugin. That +is the next step if consumers are to get this without copying the sample. + +## Checking it (completion criterion of #59) + +The sample app on the simulator imports a stdlib module, a stdlib extension module and a payload +module from the installed bundle, launched with **no** `SIMCTL_CHILD_*` variable. All commands from +the repository root; Gradle needs JDK 21 (`JAVA_HOME`), and Xcode's phases inherit it. + +```bash +export JAVA_HOME=/Users/ibrew/Library/Java/JavaVirtualMachines/jdk-21.0.12+8/Contents/Home +df -h / # the installed app (~60 MB) goes into CoreSimulator on the internal disk + +# 0. Stage once by hand and look at it +EFFECTIVE_PLATFORM_NAME=-iphonesimulator ARCHS=arm64 \ + ./gradlew -q :python-multiplatform:stageIosPythonHomeForXcode > .tmp/stage.out 2>&1; echo "EXIT=$?" +cat .tmp/stage.out # two lines: PYTHON_HOME_DIR=..., PYTHON_DYLIB_INFO_TEMPLATE=... +ls python-multiplatform/build/python-ios-home/iphonesimulator-arm64/home/lib/python3.14/lib-dynload | head +test ! -e python-multiplatform/build/python-ios-home/iphonesimulator-arm64/home/lib/python3.14/test && echo "test/ excluded" +# must fail, naming ONLY_ACTIVE_ARCH: +EFFECTIVE_PLATFORM_NAME=-iphonesimulator ARCHS="arm64 x86_64" \ + ./gradlew -q :python-multiplatform:stageIosPythonHomeForXcode > .tmp/stage-universal.out 2>&1; echo "EXIT=$?" + +# 1. Frameworks the Xcode project links against +./gradlew :sample:prepareIosFrameworks --console=plain > .tmp/prepare.log 2>&1; echo "EXIT=$?" + +# 2. Build the app (flags from the last verified run, ROADMAP history 2026-08-13: actool cannot +# compile the app icon against this machine's simulator runtimes, and -target avoids +# -destination's eligibility check) +(cd iosApp && xcodebuild -project iosApp.xcodeproj -target iosApp -configuration Debug \ + -sdk iphonesimulator ARCHS=arm64 ONLY_ACTIVE_ARCH=YES SYMROOT="$PWD/build" \ + ASSETCATALOG_COMPILER_APPICON_NAME="" \ + CODE_SIGN_IDENTITY="-" CODE_SIGN_STYLE=Manual DEVELOPMENT_TEAM="" build) \ + > .tmp/xcodebuild.log 2>&1; echo "EXIT=$?" +grep -E "Installing the Python|Wrapped|error:" .tmp/xcodebuild.log + +APP=iosApp/build/Debug-iphonesimulator/PythonDemo.app +ls "$APP/python-multiplatform-home/lib/python3.14/os.py" "$APP/python/example_py/__init__.py" +ls "$APP/python-multiplatform-home/lib/python3.14/lib-dynload" | grep -c '\.fwork$' # 67 for 3.14 +find "$APP/python-multiplatform-home" "$APP/python" -name '*.so' | wc -l # 0 +ls -d "$APP/Frameworks/_json.framework" "$APP/Frameworks/Python.framework" +codesign --verify --deep --strict "$APP" && echo SIGNED_OK + +# 3. Install and launch with no SIMCTL_CHILD_* variable +UDID=$(xcrun simctl list devices booted | grep -oE '[0-9A-F-]{36}' | head -n 1); echo "$UDID" +env | grep '^SIMCTL_CHILD_' && echo "unset these first" # must print nothing +xcrun simctl uninstall "$UDID" org.thisisthepy.python.multiplatform.demo +xcrun simctl install "$UDID" "$APP" +xcrun simctl launch --console-pty --terminate-running-process "$UDID" \ + org.thisisthepy.python.multiplatform.demo > .tmp/launch.log 2>&1 & +LAUNCH=$! +# wait for the DEMO block (Monitor/until-loop; no sleep in this shell), then: +kill "$LAUNCH" +grep '^DEMO' .tmp/launch.log +``` + +Pass: `DEMO 1 runtime` names `3.14.x · sys.platform=ios`, and `DEMO 8 bundle` is a `tuple:` whose +five entries are the payload's greeting as JSON (`"Hello from usage-example's Python payload 0.1.0"`), +`_json.__file__` ending `PythonDemo.app/python-multiplatform-home/lib/python3.14/lib-dynload/_json.cpython-314-iphonesimulator.fwork`, +`example_py.__file__` ending `PythonDemo.app/python/example_py/__init__.py`, `sys.prefix` ending +`PythonDemo.app/python-multiplatform-home`, and `sys.executable` ending `PythonDemo.app/PythonDemo`. + +Fail shapes and what they mean: + +| Seen | Meaning | +|---|---| +| No `DEMO` lines, 0% CPU (`sample ` parked in `open$NOCANCEL`) | the app read a path outside its container — check `env` for a stray `PYTHONHOME` | +| `Fatal Python error: Failed to import encodings module` | no `python-multiplatform-home/` in the installed app; the phase did not run | +| `DEMO 8 bundle | ModuleNotFoundError: No module named '_json'` | the `.fwork` did not resolve; compare `sys.executable`'s directory with the app directory | +| `DEMO 8 bundle | ModuleNotFoundError: No module named 'example_py'` | `/python/` missing: `PYTHON_PAYLOAD_DIR` not set in the phase | diff --git a/docs/roadmap/ROADMAP.md b/docs/roadmap/ROADMAP.md index a92bc25de..1e4a2a30b 100644 --- a/docs/roadmap/ROADMAP.md +++ b/docs/roadmap/ROADMAP.md @@ -262,17 +262,14 @@ plugin applies to Android modules (AGP 8.10.1); `ksp-fixtures/android` exists; t trampolines exist. GraalVM native image upcalls are verified by hand (`docs/platforms/graal-native-image-verification.md`, `PM/desktopTest/.../ReachabilityMetadataTest.kt`). -- **13.1 The iOS app bundle carries no Python standard library — open.** (Was old §9 item 4 and §14b - item 11.) The Xcode "Install Target Specific Python Standard Library" phase - (`iosApp/iosApp.xcodeproj/project.pbxproj`) rsyncs `…/ios-arm64_x86_64-simulator/lib/`, which holds - only `libpython3.14.dylib`, and nothing in the app path sets `PYTHONHOME` - (`sample/src/iosMain/.../MainViewController.kt` says so). The app starts only with a manual `rsync` - plus `SIMCTL_CHILD_PYTHONHOME` (recipe in git history, old §13). To decide first: stage by Gradle or - by a corrected Xcode phase; take `PYTHONHOME` from the environment or derive it from the bundle in - `iosMain` without breaking the test task. The "Prepare Python Binary Modules" phase rewrites - `lib-dynload/*.so` into `.fwork` placeholders, so a bundle-hosted stdlib needs BeeWare's importer - or that phase disabled. A `PYTHONHOME` on an external volume hangs the sandboxed app in `open()` - rather than failing. +- **13.1 The iOS app bundle carries no Python standard library — implemented, simulator proof + open.** SPEC L-11, issue #59, `docs/platforms/ios-app-bundle.md`: per-slice staging + (`stageIosPythonHome_*`, `stageIosPythonHomeForXcode`), `tools/xcode/install-python.sh` (stdlib to + `/python-multiplatform-home/`, payload to `/python/`, `.so` wrapped as frameworks with + `.fwork` placeholders) replacing `iosApp/`'s two broken phases, and `IosPythonHome` + + `Py_SetPythonHome` at run time. Left: the installed-app run on the simulator with no + `SIMCTL_CHILD_*` (the page's "Checking it"); a device run; and a way for consumers outside this + repository to stage the stdlib (the iOS archive download is in this build script, not the plugin). - **13.2 Native-image upcall verification as an automated test — open.** SPEC U-6/N-5: the procedure is manual (`:sample:nativeCompile`, Liberica NIK); only the reachability metadata is guarded automatically. diff --git a/iosApp/iosApp.xcodeproj/project.pbxproj b/iosApp/iosApp.xcodeproj/project.pbxproj index 796319f54..7f0dbc4af 100644 --- a/iosApp/iosApp.xcodeproj/project.pbxproj +++ b/iosApp/iosApp.xcodeproj/project.pbxproj @@ -12,7 +12,6 @@ 2152FB042600AC8F00CF470E /* iOSApp.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2152FB032600AC8F00CF470E /* iOSApp.swift */; }; 7555FF83242A565900829871 /* ContentView.swift in Sources */ = {isa = PBXBuildFile; fileRef = 7555FF82242A565900829871 /* ContentView.swift */; }; 8B3F1A512CC49F1D00B85330 /* Python.xcframework in Frameworks */ = {isa = PBXBuildFile; fileRef = 8B3F1A502CC49F1D00B85330 /* Python.xcframework */; }; - 8B3F1A5B2CC4A49700B85330 /* dylib-Info-template.plist in Resources */ = {isa = PBXBuildFile; fileRef = 8B3F1A5A2CC4A49700B85330 /* dylib-Info-template.plist */; }; 8B3F1A5C2CC574B200B85330 /* Python.xcframework in Embed Frameworks */ = {isa = PBXBuildFile; fileRef = 8B3F1A502CC49F1D00B85330 /* Python.xcframework */; settings = {ATTRIBUTES = (CodeSignOnCopy, RemoveHeadersOnCopy, ); }; }; /* End PBXBuildFile section */ @@ -120,8 +119,7 @@ 7555FF77242A565900829871 /* Sources */, B92378962B6B1156000C7307 /* Frameworks */, 7555FF79242A565900829871 /* Resources */, - 8B3F1A532CC49F9C00B85330 /* Install Target Specific Python Standard Library */, - 8B3F1A552CC4A00F00B85330 /* Prepare Python Binary Modules */, + 8B3F1A532CC49F9C00B85330 /* Install Python Standard Library and Payload */, 8B3F1A4F2CC498EB00B85330 /* Embed Frameworks */, ); buildRules = ( @@ -178,14 +176,13 @@ files = ( 058557D9273AAEEB004C7B11 /* Preview Assets.xcassets in Resources */, 058557BB273AAA24004C7B11 /* Assets.xcassets in Resources */, - 8B3F1A5B2CC4A49700B85330 /* dylib-Info-template.plist in Resources */, ); runOnlyForDeploymentPostprocessing = 0; }; /* End PBXResourcesBuildPhase section */ /* Begin PBXShellScriptBuildPhase section */ - 8B3F1A532CC49F9C00B85330 /* Install Target Specific Python Standard Library */ = { + 8B3F1A532CC49F9C00B85330 /* Install Python Standard Library and Payload */ = { isa = PBXShellScriptBuildPhase; buildActionMask = 2147483647; files = ( @@ -194,32 +191,14 @@ ); inputPaths = ( ); - name = "Install Target Specific Python Standard Library"; + name = "Install Python Standard Library and Payload"; outputFileListPaths = ( ); outputPaths = ( ); runOnlyForDeploymentPostprocessing = 0; shellPath = /bin/sh; - shellScript = "set -e\n\nmkdir -p \"$CODESIGNING_FOLDER_PATH/lib\"\nif [ \"$EFFECTIVE_PLATFORM_NAME\" = \"-iphonesimulator\" ]; then\n echo \"Installing Python modules for iOS Simulator\"\n rsync -au --delete \"$PROJECT_DIR/../sample/build/xcode-frameworks/Python.xcframework/ios-arm64_x86_64-simulator/lib/\" \"$CODESIGNING_FOLDER_PATH/lib/\" \nelse\n echo \"Installing Python modules for iOS Device\"\n rsync -au --delete \"$PROJECT_DIR/../sample/build/xcode-frameworks/Python.xcframework/ios-arm64/lib/\" \"$CODESIGNING_FOLDER_PATH/lib/\" \nfi\n"; - }; - 8B3F1A552CC4A00F00B85330 /* Prepare Python Binary Modules */ = { - isa = PBXShellScriptBuildPhase; - buildActionMask = 2147483647; - files = ( - ); - inputFileListPaths = ( - ); - inputPaths = ( - ); - name = "Prepare Python Binary Modules"; - outputFileListPaths = ( - ); - outputPaths = ( - ); - runOnlyForDeploymentPostprocessing = 0; - shellPath = /bin/sh; - shellScript = "set -e\n\ninstall_dylib () {\n INSTALL_BASE=$1\n FULL_EXT=$2\n\n # The name of the extension file\n EXT=$(basename \"$FULL_EXT\")\n # The location of the extension file, relative to the bundle\n RELATIVE_EXT=${FULL_EXT#$CODESIGNING_FOLDER_PATH/} \n # The path to the extension file, relative to the install base\n PYTHON_EXT=${RELATIVE_EXT/$INSTALL_BASE/}\n # The full dotted name of the extension module, constructed from the file path.\n FULL_MODULE_NAME=$(echo $PYTHON_EXT | cut -d \".\" -f 1 | tr \"/\" \".\"); \n # A bundle identifier; not actually used, but required by Xcode framework packaging\n FRAMEWORK_BUNDLE_ID=$(echo $PRODUCT_BUNDLE_IDENTIFIER.$FULL_MODULE_NAME | tr \"_\" \"-\")\n # The name of the framework folder.\n FRAMEWORK_FOLDER=\"Frameworks/$FULL_MODULE_NAME.framework\"\n\n # If the framework folder doesn't exist, create it.\n if [ ! -d \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER\" ]; then\n echo \"Creating framework for $RELATIVE_EXT\" \n mkdir -p \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER\"\n cp \"$CODESIGNING_FOLDER_PATH/dylib-Info-template.plist\" \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/Info.plist\"\n plutil -replace CFBundleExecutable -string \"$FULL_MODULE_NAME\" \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/Info.plist\"\n plutil -replace CFBundleIdentifier -string \"$FRAMEWORK_BUNDLE_ID\" \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/Info.plist\"\n fi\n \n echo \"Installing binary for $FRAMEWORK_FOLDER/$FULL_MODULE_NAME\" \n mv \"$FULL_EXT\" \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/$FULL_MODULE_NAME\"\n # Create a placeholder .fwork file where the .so was\n echo \"$FRAMEWORK_FOLDER/$FULL_MODULE_NAME\" > ${FULL_EXT%.so}.fwork\n # Create a back reference to the .so file location in the framework\n echo \"${RELATIVE_EXT%.so}.fwork\" > \"$CODESIGNING_FOLDER_PATH/$FRAMEWORK_FOLDER/$FULL_MODULE_NAME.origin\" \n}\n\nPYTHON_VER=$(ls -1 \"$CODESIGNING_FOLDER_PATH/lib\")\necho \"Install Python $PYTHON_VER standard library extension modules...\"\nfind \"$CODESIGNING_FOLDER_PATH/lib/$PYTHON_VER/lib-dynload\" -name \"*.so\" | while read FULL_EXT; do\n install_dylib lib/$PYTHON_VER/lib-dynload/ \"$FULL_EXT\"\ndone\n\n# Clean up dylib template \nrm -f \"$CODESIGNING_FOLDER_PATH/dylib-Info-template.plist\"\necho \"Signing frameworks as $EXPANDED_CODE_SIGN_IDENTITY_NAME ($EXPANDED_CODE_SIGN_IDENTITY)...\"\nfind \"$CODESIGNING_FOLDER_PATH/Frameworks\" -name \"*.framework\" -exec /usr/bin/codesign --force --sign \"$EXPANDED_CODE_SIGN_IDENTITY\" ${OTHER_CODE_SIGN_FLAGS:-} -o runtime --timestamp=none --preserve-metadata=identifier,entitlements,flags --generate-entitlement-der \"{}\" \\; \n"; + shellScript = "set -e\n# SPEC L-10 / issue #59: the CPython stdlib and this app's python/ payload into the .app, with\n# extension modules wrapped as frameworks. See docs/platforms/ios-app-bundle.md.\ncd \"$SRCROOT/..\"\nexport PYTHON_PAYLOAD_DIR=\"$SRCROOT/../sample/python/src/main\"\n/bin/bash tools/xcode/install-python.sh\n"; }; F36B1CEB2AD83DDC00CB74D5 /* Compile Kotlin Framework */ = { isa = PBXShellScriptBuildPhase; diff --git a/python-multiplatform-gradle-plugin/src/main/kotlin/python/multiplatform/gradle/IosPythonHome.kt b/python-multiplatform-gradle-plugin/src/main/kotlin/python/multiplatform/gradle/IosPythonHome.kt new file mode 100644 index 000000000..dce8d74a4 --- /dev/null +++ b/python-multiplatform-gradle-plugin/src/main/kotlin/python/multiplatform/gradle/IosPythonHome.kt @@ -0,0 +1,250 @@ +package python.multiplatform.gradle + +import org.gradle.api.DefaultTask +import org.gradle.api.GradleException +import org.gradle.api.file.DirectoryProperty +import org.gradle.api.provider.Property +import org.gradle.api.tasks.Input +import org.gradle.api.tasks.InputDirectory +import org.gradle.api.tasks.OutputDirectory +import org.gradle.api.tasks.PathSensitive +import org.gradle.api.tasks.PathSensitivity +import org.gradle.api.tasks.TaskAction +import java.io.File +import java.nio.file.Files +import java.nio.file.StandardCopyOption + +/** + * What an iOS app bundle carries of CPython's standard library, and where it comes from -- SPEC L-11, + * issue #59. + * + * `Python.xcframework` ships the interpreter and headers and nothing else (`iosMain/README.md`); the + * standard library sits *beside* the framework slices in the same archive: + * + * ``` + * Python.xcframework/ + * lib/python3.14/ pure-Python stdlib, shared by every slice + * ios-arm64/lib-arm64/python3.14/ lib-dynload/ (the .so files), _sysconfigdata (device) + * ios-arm64_x86_64-simulator/lib-arm64/... the same for the arm64 simulator + * ios-arm64_x86_64-simulator/lib-x86_64/... ...and the x86_64 simulator + * build/iOS-dylib-Info-template.plist Info.plist for an extension-module framework + * build/utils.sh upstream's own Xcode helper (install_python) + * ``` + * + * `build/utils.sh`'s `install_stdlib` merges the shared tree with one `lib-$ARCHS` tree into the app; + * this does the same merge at Gradle time, into a directory per slice, so the Xcode phase + * (`tools/xcode/install-python.sh`) copies a finished prefix and needs no path into the archive. + * Pure functions are separated out so the layout is pinned by tests without downloading anything. + */ +object IosPythonHomeLayout { + + /** + * The directory inside the app bundle the prefix is copied to, and that `IosPythonHome` in the + * library's `iosMain` looks for under `NSBundle.mainBundle.resourcePath`. The same name desktop + * packages under ([PACKAGED_HOME_DIRECTORY]); deliberately *not* upstream's `python/`, because + * `python/` is the consumer's payload root (`PythonPayload.PAYLOAD_ROOT`, SPEC L-6) and the + * payload phase `rsync --delete`s it. + */ + const val BUNDLE_DIRECTORY: String = PACKAGED_HOME_DIRECTORY + + /** The Info.plist template for an extension-module framework, relative to the xcframework. */ + const val DYLIB_INFO_TEMPLATE: String = "build/iOS-dylib-Info-template.plist" + + /** + * Top-level stdlib packages an iOS app never ships. + * + * Measured on 3.14 (`Python.xcframework/lib/python3.14`, 51 MB, 2,407 files): `test` alone is + * 36 MB and 1,641 files -- CPython's own regression suite. `tkinter`, `idlelib` and `turtledemo` + * need `_tkinter`, which no iOS slice builds (no `lib-dynload/_tkinter*`). `ensurepip` installs + * pip, which cannot run inside an app sandbox. Everything else is kept, as on desktop (L-9). + */ + val EXCLUDED_TOP_LEVEL_PACKAGES: Set = setOf("test", "idlelib", "tkinter", "turtledemo", "ensurepip") + + /** One xcframework slice + architecture: what Xcode builds for in one Run Script invocation. */ + data class Slice( + /** Xcode's platform name without the dash: `iphoneos`, `iphonesimulator`. */ + val sdk: String, + /** One `ARCHS` value: `arm64`, `x86_64`. */ + val arch: String, + /** The xcframework slice directory holding `lib-/`. */ + val xcframeworkSlice: String, + ) { + /** `iphonesimulator-arm64`: the staging directory name. */ + val name: String get() = "$sdk-$arch" + + /** `stageIosPythonHome_iphonesimulator_arm64`: the library build's task for this slice. */ + val stageTaskName: String get() = "stageIosPythonHome_" + name.replace('-', '_') + } + + /** + * Every slice this library stages. `iosArm64` is a device; `iosSimulatorArm64` and `iosX64` + * both live in the fat simulator slice, with separate `lib-` trees -- which is why `iosX64` + * needs its own entry rather than the hard-coded `lib-arm64` ROADMAP §13.4 records for the test + * task. + */ + val slices: List = listOf( + Slice("iphoneos", "arm64", "ios-arm64"), + Slice("iphonesimulator", "arm64", "ios-arm64_x86_64-simulator"), + Slice("iphonesimulator", "x86_64", "ios-arm64_x86_64-simulator"), + ) + + /** + * The slice an Xcode build phase is building for, from the two settings Xcode exports to it. + * + * Exactly one architecture: `ARCHS="arm64 x86_64"` (a universal simulator build, which Xcode + * does for Release unless `ONLY_ACTIVE_ARCH=YES`) would need two `lib-dynload` trees in one + * prefix, and CPython looks in one. Upstream's `install_stdlib` (`lib-$ARCHS`) has the same + * limit, silently; this says so. + */ + fun sliceFor(effectivePlatformName: String?, archs: String?): Slice { + val platform = effectivePlatformName?.trim()?.removePrefix("-") + require(!platform.isNullOrEmpty()) { + "EFFECTIVE_PLATFORM_NAME is not set. This task picks the iOS slice from the settings Xcode " + + "exports to a Run Script phase; run it from one (tools/xcode/install-python.sh does)." + } + val archList = archs?.trim()?.split(Regex("\\s+"))?.filter { it.isNotEmpty() }.orEmpty() + require(archList.isNotEmpty()) { "ARCHS is not set; run this from an Xcode Run Script phase." } + require(archList.size == 1) { + "ARCHS='${archs!!.trim()}' names ${archList.size} architectures, and one app prefix can hold " + + "one lib-dynload. Build one architecture at a time (ONLY_ACTIVE_ARCH=YES, or ARCHS=arm64)." + } + val arch = archList.single() + return slices.firstOrNull { it.sdk == platform && it.arch == arch } + ?: throw IllegalArgumentException( + "no CPython iOS slice for platform '$platform' and architecture '$arch'; this library " + + "stages ${slices.joinToString { it.name }}." + ) + } + + /** `3.14` from `3.14.7`. */ + fun stdlibTag(pythonVersion: String): String { + val parts = pythonVersion.split('.') + require(parts.size >= 2) { "python version '$pythonVersion' has no major.minor" } + return "${parts[0]}.${parts[1]}" + } + + /** + * The trees merged into `lib/python/` for [slice], relative to the xcframework, shared tree + * first: the slice tree only adds files (`lib-dynload/`, `_sysconfigdata__ios_*`), but if it ever + * overlapped, the slice-specific file is the one that must win, as in `install_stdlib`. + */ + fun sourceDirectories(slice: Slice, pythonVersion: String): List { + val tag = stdlibTag(pythonVersion) + return listOf("lib/python$tag", "${slice.xcframeworkSlice}/lib-${slice.arch}/python$tag") + } + + /** + * Whether a file or directory at [stdlibRelativePath] -- relative to `lib/python/`, `/` + * separated -- goes into the app. + * + * Not copied: any `__pycache__` (bytecode from whatever ran against the extracted tree, so the + * app would depend on the build machine's history), the [EXCLUDED_TOP_LEVEL_PACKAGES], and any + * `libpython*.dylib` (upstream excludes it too: the interpreter arrives as `Python.framework`, + * and a loose dylib in an app bundle is rejected by App Store validation). + */ + fun isStdlibPathStaged(stdlibRelativePath: String): Boolean { + val segments = stdlibRelativePath.trim('/').split('/').filter { it.isNotEmpty() } + if (segments.isEmpty()) return true + if (segments.first() in EXCLUDED_TOP_LEVEL_PACKAGES) return false + if (segments.any { it == "__pycache__" }) return false + val name = segments.last() + if (name.startsWith("libpython") && name.endsWith(".dylib")) return false + return true + } + + /** + * Every file the prefix for [slice] contains, as destination path (relative to the prefix) to + * source file. A later source overrides an earlier one at the same destination + * ([sourceDirectories]' order). Fails when a source tree is missing, rather than staging half a + * prefix that would abort `Py_Initialize()` on the device. + */ + fun plan(xcframework: File, slice: Slice, pythonVersion: String): Map { + val tag = stdlibTag(pythonVersion) + val result = sortedMapOf() + for (relative in sourceDirectories(slice, pythonVersion)) { + val root = File(xcframework, relative) + if (!root.isDirectory) { + throw GradleException( + "$root does not exist, so the ${slice.name} prefix would have no " + + (if (relative.startsWith("lib/")) "standard library" else "lib-dynload") + + ". The Python.xcframework layout may have changed; see IosPythonHomeLayout's KDoc." + ) + } + root.walkTopDown() + .onEnter { dir -> dir == root || isStdlibPathStaged(dir.relativeTo(root).invariantSeparatorsPath) } + .filter { it.isFile } + .forEach { file -> + val inStdlib = file.relativeTo(root).invariantSeparatorsPath + if (isStdlibPathStaged(inStdlib)) result["lib/python$tag/$inStdlib"] = file + } + } + return result + } +} + +/** + * Stages one slice's prefix for an iOS app bundle (SPEC L-11): `/home/lib/python/` + * as [IosPythonHomeLayout.plan] lists it, plus `/dylib-Info-template.plist` *beside* + * the prefix -- it is an input to the Xcode phase, not part of what the app carries. + * + * A plain copy rather than a `Sync` so that what [IosPythonHomeLayout.plan] returns -- which the + * tests pin -- is exactly what is copied, with no second notion of "excluded" in a copy spec. + */ +abstract class StageIosPythonHomeTask : DefaultTask() { + + /** `.../extracted//ios/Python.xcframework`. */ + @get:InputDirectory + @get:PathSensitive(PathSensitivity.RELATIVE) + abstract val xcframework: DirectoryProperty + + /** One of [IosPythonHomeLayout.slices]' names, e.g. `iphonesimulator-arm64`. */ + @get:Input + abstract val sliceName: Property + + @get:Input + abstract val pythonVersion: Property + + @get:OutputDirectory + abstract val destinationDir: DirectoryProperty + + @TaskAction + fun stage() { + val slice = IosPythonHomeLayout.slices.firstOrNull { it.name == sliceName.get() } + ?: throw GradleException("unknown iOS slice '${sliceName.get()}'") + val framework = xcframework.get().asFile + val version = pythonVersion.get() + val plan = IosPythonHomeLayout.plan(framework, slice, version) + + val root = destinationDir.get().asFile + val home = File(root, PREFIX_DIRECTORY) + // Whole directory replaced: a file dropped from the plan (an exclusion added, an upstream + // module removed) must not survive from the previous run into the next app. + home.deleteRecursively() + for ((relative, source) in plan) { + val target = File(home, relative) + target.parentFile.mkdirs() + Files.copy(source.toPath(), target.toPath(), StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.COPY_ATTRIBUTES) + } + + val template = File(framework, IosPythonHomeLayout.DYLIB_INFO_TEMPLATE) + if (!template.isFile) { + throw GradleException("$template is missing; the Xcode phase needs it to wrap extension modules as frameworks.") + } + Files.copy(template.toPath(), File(root, TEMPLATE_NAME).toPath(), StandardCopyOption.REPLACE_EXISTING) + + // The marker PythonHomeCheck probes on the device, checked here where it is cheap to fail. + val marker = File(home, "lib/python${IosPythonHomeLayout.stdlibTag(version)}/os.py") + if (!marker.isFile) { + throw GradleException("staged ${slice.name} into $home but $marker is missing; the app would abort at Py_Initialize().") + } + logger.info("Staged {} files of CPython {} for {} into {}", plan.size, version, slice.name, home) + } + + companion object { + /** The prefix inside [destinationDir]; what the Xcode phase copies into the app. */ + const val PREFIX_DIRECTORY: String = "home" + + /** The framework Info.plist template inside [destinationDir]. */ + const val TEMPLATE_NAME: String = "dylib-Info-template.plist" + } +} diff --git a/python-multiplatform-gradle-plugin/src/test/kotlin/python/multiplatform/gradle/IosPythonHomeLayoutTest.kt b/python-multiplatform-gradle-plugin/src/test/kotlin/python/multiplatform/gradle/IosPythonHomeLayoutTest.kt new file mode 100644 index 000000000..7f11258e3 --- /dev/null +++ b/python-multiplatform-gradle-plugin/src/test/kotlin/python/multiplatform/gradle/IosPythonHomeLayoutTest.kt @@ -0,0 +1,206 @@ +package python.multiplatform.gradle + +import org.gradle.api.GradleException +import org.gradle.testfixtures.ProjectBuilder +import java.io.File +import kotlin.test.AfterTest +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertFailsWith +import kotlin.test.assertFalse +import kotlin.test.assertTrue + +/** + * SPEC L-11 (issue #59): which slice an Xcode build stages, which files of `Python.xcframework` go + * into the app's prefix, and what the staging task leaves on disk. + * + * The run-time half -- the library finding `/python-multiplatform-home` and handing it + * to CPython -- is `python-multiplatform/src/iosSimulatorArm64Test/.../env/IosPythonHomeTest.kt`. + * The two share no code, so each pins the directory name as a literal. + * + * What this cannot find: whether CPython on a device really loads the result. The fake tree below + * mirrors the observed 3.14 archive's *shape*; the installed-app run in + * `docs/platforms/ios-app-bundle.md` is the check for the rest. + */ +class IosPythonHomeLayoutTest { + + private val version = "3.14.7" + // Under this module's build directory (the test task's working directory), not the system + // temporary directory: AGENTS.md rule 2. + private val scratch: File = + File(System.getProperty("user.dir"), "build/tmp/iosPythonHomeLayoutTest/${java.util.UUID.randomUUID()}") + .apply { mkdirs() } + + @AfterTest + fun cleanUp() { + scratch.deleteRecursively() + } + + // ---- names ---- + + @Test + fun theBundleDirectoryIsTheOneTheLibraryLooksFor() { + // `IosPythonHome.DIRECTORY_NAME` in python-multiplatform/src/iosMain. + assertEquals("python-multiplatform-home", IosPythonHomeLayout.BUNDLE_DIRECTORY) + } + + @Test + fun theBundleDirectoryIsNotThePayloadRoot() { + // `PythonPayload.PAYLOAD_ROOT` is "python", and the payload copy rsyncs it with --delete. + assertTrue(IosPythonHomeLayout.BUNDLE_DIRECTORY != "python") + } + + // ---- slice choice ---- + + @Test + fun xcodeSettingsSelectTheSlice() { + val l = IosPythonHomeLayout + assertEquals("iphonesimulator-arm64", l.sliceFor("-iphonesimulator", "arm64").name) + assertEquals("iphonesimulator-x86_64", l.sliceFor("-iphonesimulator", "x86_64").name) + assertEquals("iphoneos-arm64", l.sliceFor("-iphoneos", "arm64").name) + assertEquals("ios-arm64", l.sliceFor("-iphoneos", "arm64").xcframeworkSlice) + assertEquals("ios-arm64_x86_64-simulator", l.sliceFor("-iphonesimulator", "x86_64").xcframeworkSlice) + } + + @Test + fun aUniversalBuildIsRefusedWithTheFix() { + val e = assertFailsWith { IosPythonHomeLayout.sliceFor("-iphonesimulator", "arm64 x86_64") } + assertTrue("ONLY_ACTIVE_ARCH" in e.message.orEmpty(), e.message) + } + + @Test + fun outsideXcodeOrOnAnotherPlatformIsRefused() { + assertFailsWith { IosPythonHomeLayout.sliceFor(null, "arm64") } + assertFailsWith { IosPythonHomeLayout.sliceFor("-iphonesimulator", "") } + assertFailsWith { IosPythonHomeLayout.sliceFor("-appletvos", "arm64") } + assertFailsWith { IosPythonHomeLayout.sliceFor("-iphoneos", "x86_64") } + } + + @Test + fun everySliceHasItsOwnTask() { + assertEquals( + listOf( + "stageIosPythonHome_iphoneos_arm64", + "stageIosPythonHome_iphonesimulator_arm64", + "stageIosPythonHome_iphonesimulator_x86_64", + ), + IosPythonHomeLayout.slices.map { it.stageTaskName }, + ) + } + + // ---- what is copied ---- + + @Test + fun theSharedStdlibComesFirstAndTheSliceTreeSecond() { + val sim = IosPythonHomeLayout.sliceFor("-iphonesimulator", "x86_64") + assertEquals( + listOf("lib/python3.14", "ios-arm64_x86_64-simulator/lib-x86_64/python3.14"), + IosPythonHomeLayout.sourceDirectories(sim, version), + ) + } + + @Test + fun fileSelection() { + val staged = IosPythonHomeLayout::isStdlibPathStaged + assertTrue(staged("os.py")) + assertTrue(staged("encodings/__init__.py")) + assertTrue(staged("json/decoder.py")) + assertTrue(staged("lib-dynload/_json.cpython-314-iphonesimulator.so")) + assertTrue(staged("_sysconfigdata__ios_arm64-iphonesimulator.py")) + assertTrue(staged("site-packages/README.txt")) + // A package *named like* an excluded one deeper down is not excluded. + assertTrue(staged("email/test/__init__.py")) + + assertFalse(staged("test/test_os.py")) + assertFalse(staged("test")) + assertFalse(staged("tkinter/__init__.py")) + assertFalse(staged("idlelib/idle.py")) + assertFalse(staged("turtledemo/__main__.py")) + assertFalse(staged("ensurepip/_bundled/pip.whl")) + assertFalse(staged("encodings/__pycache__/aliases.cpython-314.pyc")) + assertFalse(staged("__pycache__")) + assertFalse(staged("libpython3.14.dylib")) + } + + @Test + fun aPlanMergesTheTwoTreesAndDropsTheExcludedFiles() { + val xcframework = fakeXcframework() + val slice = IosPythonHomeLayout.sliceFor("-iphonesimulator", "arm64") + val plan = IosPythonHomeLayout.plan(xcframework, slice, version) + assertEquals( + listOf( + "lib/python3.14/_sysconfigdata__ios_arm64-iphonesimulator.py", + "lib/python3.14/encodings/__init__.py", + "lib/python3.14/lib-dynload/_json.cpython-314-iphonesimulator.so", + "lib/python3.14/os.py", + ), + plan.keys.toList(), + ) + // The x86_64 tree of the same fat slice is not mixed in. + assertFalse(plan.values.any { "lib-x86_64" in it.path }) + } + + @Test + fun aMissingSliceTreeFailsInsteadOfStagingHalfAPrefix() { + val xcframework = fakeXcframework() + File(xcframework, "ios-arm64").deleteRecursively() + assertFailsWith { + IosPythonHomeLayout.plan(xcframework, IosPythonHomeLayout.sliceFor("-iphoneos", "arm64"), version) + } + } + + // ---- the task, against the fake tree ---- + + @Test + fun theTaskStagesThePrefixAndTheTemplateBesideIt() { + val xcframework = fakeXcframework() + val project = ProjectBuilder.builder() + .withProjectDir(File(scratch, "project").apply { mkdirs() }) + .withGradleUserHomeDir(File(scratch, "gradle-home")) + .build() + val out = File(scratch, "out") + // Left over from a previous run with a wider selection; must not survive. + File(out, "home/lib/python3.14/test/stale.py").apply { parentFile.mkdirs(); writeText("") } + + val task = project.tasks.create("stage", StageIosPythonHomeTask::class.java) + task.xcframework.set(xcframework) + task.sliceName.set("iphonesimulator-arm64") + task.pythonVersion.set(version) + task.destinationDir.set(out) + task.stage() + + val staged = File(out, "home").walkTopDown().filter { it.isFile } + .map { it.relativeTo(File(out, "home")).invariantSeparatorsPath }.sorted().toList() + assertEquals( + listOf( + "lib/python3.14/_sysconfigdata__ios_arm64-iphonesimulator.py", + "lib/python3.14/encodings/__init__.py", + "lib/python3.14/lib-dynload/_json.cpython-314-iphonesimulator.so", + "lib/python3.14/os.py", + ), + staged, + ) + assertTrue(File(out, StageIosPythonHomeTask.TEMPLATE_NAME).isFile) + assertFalse(File(out, "home/${StageIosPythonHomeTask.TEMPLATE_NAME}").exists()) + } + + /** The observed 3.14 archive's shape, a few files deep. */ + private fun fakeXcframework(): File { + val root = File(scratch, "Python.xcframework") + fun file(path: String) = File(root, path).apply { parentFile.mkdirs(); writeText(path) } + file("lib/python3.14/os.py") + file("lib/python3.14/encodings/__init__.py") + file("lib/python3.14/encodings/__pycache__/__init__.cpython-314.pyc") + file("lib/python3.14/test/test_os.py") + file("lib/python3.14/tkinter/__init__.py") + file("ios-arm64/lib-arm64/python3.14/lib-dynload/_json.cpython-314-iphoneos.so") + file("ios-arm64/lib-arm64/python3.14/_sysconfigdata__ios_arm64-iphoneos.py") + file("ios-arm64/lib/libpython3.14.dylib") + file("ios-arm64_x86_64-simulator/lib-arm64/python3.14/lib-dynload/_json.cpython-314-iphonesimulator.so") + file("ios-arm64_x86_64-simulator/lib-arm64/python3.14/_sysconfigdata__ios_arm64-iphonesimulator.py") + file("ios-arm64_x86_64-simulator/lib-x86_64/python3.14/lib-dynload/_json.cpython-314-iphonesimulator.so") + file("ios-arm64_x86_64-simulator/lib/libpython3.14.dylib") + file("build/iOS-dylib-Info-template.plist") + return root + } +} diff --git a/python-multiplatform/build.gradle.kts b/python-multiplatform/build.gradle.kts index f080aa6ad..b12066b10 100644 --- a/python-multiplatform/build.gradle.kts +++ b/python-multiplatform/build.gradle.kts @@ -1438,6 +1438,68 @@ tasks.withType + tasks.register(slice.stageTaskName, python.multiplatform.gradle.StageIosPythonHomeTask::class.java) { + group = "python" + description = "Stages CPython's standard library for an iOS app built for ${slice.name} (SPEC L-10)" + dependsOn(downloadPython_ios) + xcframework.set(file("$extractedDir/ios/Python.xcframework")) + sliceName.set(slice.name) + pythonVersion.set(configuredPythonVersion) + destinationDir.set(iosPythonHomeRoot.map { it.dir(slice.name) }) + } +} + +/** Every slice; the Xcode phase uses [stageIosPythonHomeForXcode], which stages only the one it builds. */ +val stageIosPythonHome by tasks.registering { + group = "python" + description = "Stages CPython's standard library for every iOS app slice (SPEC L-10)" + dependsOn(stageIosPythonHomeSlices) +} + +/** + * The Xcode phase's entry point. Picks the slice from `EFFECTIVE_PLATFORM_NAME` and `ARCHS` -- Xcode + * exports both to a Run Script phase, and `./gradlew` passes the client's environment to the build -- + * stages it, and prints two lines a `-q` run leaves alone on stdout: + * + * PYTHON_HOME_DIR=/python-ios-home//home + * PYTHON_DYLIB_INFO_TEMPLATE=/python-ios-home//dylib-Info-template.plist + * + * The same contract shape as toolchain's `stagePythonBundleIosForXcode` (`PYTHON_PAYLOAD_DIR=`). + * Run outside Xcode (no `EFFECTIVE_PLATFORM_NAME`), or for a universal `ARCHS`, it fails while the + * task graph is built, with the reason, rather than staging a guessed slice. + */ +val stageIosPythonHomeForXcode by tasks.registering { + group = "python" + description = "Stages the iOS app prefix for the slice Xcode is building and prints its path (SPEC L-10)" + val platformName = providers.environmentVariable("EFFECTIVE_PLATFORM_NAME") + val archs = providers.environmentVariable("ARCHS") + val selected = providers.provider { + python.multiplatform.gradle.IosPythonHomeLayout.sliceFor(platformName.orNull, archs.orNull) + } + // Only the slice being built: staging all three costs three lib-dynload trees per build. + dependsOn(selected.map { tasks.named(it.stageTaskName) }) + val root = iosPythonHomeRoot.map { it.asFile.absolutePath } + doLast { + val slice = selected.get().name + println("PYTHON_HOME_DIR=${root.get()}/$slice/${python.multiplatform.gradle.StageIosPythonHomeTask.PREFIX_DIRECTORY}") + println("PYTHON_DYLIB_INFO_TEMPLATE=${root.get()}/$slice/${python.multiplatform.gradle.StageIosPythonHomeTask.TEMPLATE_NAME}") + } +} + // ================================================================================================= // androidNative -- giving the target a test *run*, not just a test *link*. // diff --git a/python-multiplatform/src/artMain/kotlin/python/multiplatform/env/PackagedPythonHome.art.kt b/python-multiplatform/src/artMain/kotlin/python/multiplatform/env/PackagedPythonHome.art.kt new file mode 100644 index 000000000..47e6d3484 --- /dev/null +++ b/python-multiplatform/src/artMain/kotlin/python/multiplatform/env/PackagedPythonHome.art.kt @@ -0,0 +1,12 @@ +package python.multiplatform.env + +/** + * No-op: androidNative is the Kotlin/Native half of an Android app, and `PythonBootstrap` + * (`androidMain`) has already unpacked the stdlib and set `PYTHONHOME` with `Os.setenv` by the time + * anything here runs. + * + * The `actual` lives here rather than in `nativeMain` because `nativeMain` is shared with iOS, where + * the answer is a real lookup in the app bundle (SPEC L-10, `iosMain`) -- the same split + * `discoverStagedPayloadRoots` already has. + */ +internal actual fun applyPackagedPythonHome() = Unit diff --git a/python-multiplatform/src/commonMain/kotlin/python/multiplatform/env/PackagedPythonHome.kt b/python-multiplatform/src/commonMain/kotlin/python/multiplatform/env/PackagedPythonHome.kt index 086235884..e1653d239 100644 --- a/python-multiplatform/src/commonMain/kotlin/python/multiplatform/env/PackagedPythonHome.kt +++ b/python-multiplatform/src/commonMain/kotlin/python/multiplatform/env/PackagedPythonHome.kt @@ -1,14 +1,16 @@ package python.multiplatform.env /** - * Points CPython at the prefix a packaged application ships, when there is one -- SPEC L-9. + * Points CPython at the prefix a packaged application ships, when there is one -- SPEC L-9 + * (desktop) and L-11 (iOS). * * Called once by [python.multiplatform.ffi.Python3.initialize], after [PythonHomeCheck] and before - * `Py_Initialize()`, which is the only window in which a home can still be chosen. Only desktop - * does anything: a packaged desktop app is started by a launcher that cannot put `PYTHONHOME` in - * its environment (see `desktopMain/README.md`). Android sets `PYTHONHOME` with `Os.setenv` in - * `PythonBootstrap`, iOS and Android-native processes have a real environment, and wasm bakes its - * stdlib into the virtual filesystem, so each of those is a no-op. + * `Py_Initialize()`, which is the only window in which a home can still be chosen. Desktop and iOS + * do something: a packaged desktop app is started by a launcher that cannot put `PYTHONHOME` in its + * environment (see `desktopMain/README.md`), and an installed iOS app is started with none while its + * stdlib sits in its own bundle (`iosMain/README.md`). Android sets `PYTHONHOME` with `Os.setenv` in + * `PythonBootstrap` (and androidNative runs inside that process), and wasm bakes its stdlib into the + * virtual filesystem, so each of those is a no-op. * * Must throw, rather than continue, when the packaged prefix is unusable: the next call is * `Py_Initialize()`, which aborts the process instead of failing. diff --git a/python-multiplatform/src/iosMain/README.md b/python-multiplatform/src/iosMain/README.md index 2b54c393c..883b8e3a3 100644 --- a/python-multiplatform/src/iosMain/README.md +++ b/python-multiplatform/src/iosMain/README.md @@ -24,39 +24,34 @@ error anywhere in the log. `PythonHomeCheck`'s own filesystem probe cannot promi either: it reads the same directory through the same kind of syscall CPython's import machinery does, and a path whose access is *parked* rather than *denied* can park the check identically. -**The recipe below only works for the test binary, not the app — do not assume it carries over.** -The build extracts the stdlib from the BeeWare support archive into `build/python-stdlib/`, and -the simulator *test* task points `SIMCTL_CHILD_PYTHONHOME` at it directly (`simctl` only forwards -environment variables prefixed `SIMCTL_CHILD_`, which is what makes the test binary see it at -all). That path is still under this repo's workspace, which lives on an external volume (see -AGENTS.md) — fine for the test binary, which `simctl` launches with fewer sandbox restrictions -than an *installed app*, but exactly the shape that hangs an app rather than starting it. An app -needs its stdlib staged **inside its own bundle or installed container**, with `PYTHONHOME` -pointed at that installed path, not at the workspace — see the reproduction recipe in ROADMAP for -the two extra steps (`rsync`ing the stdlib into the built `.app` and resolving -`SIMCTL_CHILD_PYTHONHOME` from `simctl get_app_container` after install) this repository does not -yet automate. - -## The consumer's payload is staged and not attached — the runtime half is here, the Xcode half is not +**The test binary and an installed app get their stdlib differently.** The build extracts the +stdlib into `build/python-stdlib/`, and the simulator *test* task points `SIMCTL_CHILD_PYTHONHOME` +at it (`simctl` only forwards variables prefixed `SIMCTL_CHILD_`). That path is on this workspace's +external volume — fine for the test binary, which `simctl` launches with fewer sandbox restrictions +than an installed app, and exactly the shape that hangs an app rather than starting it. + +An installed **app** carries its own prefix inside its bundle (SPEC L-11, +`docs/platforms/ios-app-bundle.md`): + +- `stageIosPythonHome__` stages the stdlib for one slice; `tools/xcode/install-python.sh`, + an Xcode Run Script phase, copies it to `/python-multiplatform-home/`, copies the consumer's + payload to `/python/`, and wraps every `.so` as `Frameworks/.framework` with a + `.fwork` placeholder (Apple loads no loose binaries; CPython's `AppleFrameworkLoader` follows the + placeholder). +- `IosPythonHome` (`PackagedPythonHome.ios.kt`) resolves `/python-multiplatform-home` + when `PYTHONHOME` is unset, checks it with `PythonHomeCheck`, and hands it to `Py_SetPythonHome` + before `Py_Initialize()`. An environment `PYTHONHOME` still wins, which is what keeps the test task + working. + +## The consumer's payload `PythonPayload.discoverStagedPayloadRoots` looks for `python/` under -`NSBundle.mainBundle.resourcePath` and puts it on `sys.path` if it is there. Nothing puts it there. - -`toolchain`'s `49da1d8` produces `build/pythonStaging/ios/python` and stops, saying why: "a native -framework has no Gradle resource mechanism, and its resources go through an Xcode phase in a project -file this plugin does not own." So three things are missing, all of them packaging: - -1. A **Copy Bundle Resources** phase referencing that directory, with *folder reference* semantics — - a group flattens the package directories and every `__init__.py` collides. -2. A **build-order dependency** on the Gradle task that produces it, expressed in the Run Script - phase `iosApp/` already uses to drive Gradle. -3. A **run on device or simulator that proves it**, which is not desktop's check. Per the section - above, a path under this repo's external volume does not merely fail for an *installed* app — it - parks `open$NOCANCEL` forever at 0% CPU. Reading the payload out of the app's own bundle is what - avoids that, and is why the answer must be the bundle rather than a workspace path. - -Until (1) and (2) exist this returns an empty list on every run, and no test here can tell "the -phase is missing" from "the code is wrong". Desktop is the platform driven end to end instead. +`NSBundle.mainBundle.resourcePath` and puts it on `sys.path` if it is there. The Xcode phase above +puts it there, from `PYTHON_PAYLOAD_DIR` (a directory, e.g. the app's own Python sources) or +`PYTHON_PAYLOAD_TASK` (a Gradle task printing `PYTHON_PAYLOAD_DIR=`, e.g. `toolchain`'s +`stagePythonBundleIosForXcode`). It is copied with `rsync --delete`, which is why the stdlib does not +live in the same directory. Extension modules in the payload are wrapped as frameworks like the +stdlib's. ## No boundary, in either direction diff --git a/python-multiplatform/src/iosMain/kotlin/python/multiplatform/env/PackagedPythonHome.ios.kt b/python-multiplatform/src/iosMain/kotlin/python/multiplatform/env/PackagedPythonHome.ios.kt new file mode 100644 index 000000000..6398f4e82 --- /dev/null +++ b/python-multiplatform/src/iosMain/kotlin/python/multiplatform/env/PackagedPythonHome.ios.kt @@ -0,0 +1,103 @@ +package python.multiplatform.env + +import kotlinx.cinterop.BooleanVar +import kotlinx.cinterop.ExperimentalForeignApi +import kotlinx.cinterop.alloc +import kotlinx.cinterop.memScoped +import kotlinx.cinterop.ptr +import kotlinx.cinterop.value +import platform.Foundation.NSBundle +import platform.Foundation.NSFileManager + +/** + * Where an installed iOS app's CPython prefix is -- SPEC L-11, issue #59. + * + * `Python.xcframework` carries no standard library, and an installed app is launched by SpringBoard + * (or `simctl launch`) with no `PYTHONHOME`. The simulator *test* task sets one, pointing at the + * workspace (`SIMCTL_CHILD_PYTHONHOME`), but the same shape for an app parks it at 0% CPU when the + * path is on an external volume (`iosMain/README.md`). So the app carries its prefix inside its own + * bundle -- `tools/xcode/install-python.sh` copies it to `/`[DIRECTORY_NAME] -- and it is handed + * to CPython in-process, before `Py_Initialize`, by [applyPackagedPythonHome]. + * + * The desktop counterpart is `PackagedPythonHome` (`jvmMain`, SPEC L-9); same order, same name. + */ +internal object IosPythonHome { + + /** + * `IosPythonHomeLayout.BUNDLE_DIRECTORY` in the Gradle plugin, which stages it; the two builds + * share no code, so each pins the string in a test. Not upstream's `python/`: that is the + * consumer's payload root ([PythonPayload.PAYLOAD_ROOT]). + */ + const val DIRECTORY_NAME = "python-multiplatform-home" + + /** A resolved prefix and a human-readable account of where it came from, for error messages. */ + data class Resolved(val path: String, val source: String) + + /** [resolve] against this process's environment and main bundle. */ + fun resolve(): Resolved? = resolve( + environmentHome = readEnvVar("PYTHONHOME"), + resourcePath = NSBundle.mainBundle.resourcePath, + isDirectory = ::isDirectory, + ) + + /** + * The bundled prefix, or null when there is none to apply. + * + * 1. A non-blank `PYTHONHOME` in the environment wins, and the answer is null: CPython reads it + * itself and `PythonHomeCheck` already checked it. This is what keeps the simulator test task + * (`SIMCTL_CHILD_PYTHONHOME`) and an Xcode scheme's explicit variable working unchanged. + * 2. `/`[DIRECTORY_NAME], **only if that directory exists**: an app built without + * the Xcode phase must behave exactly as it did before this existed. + */ + fun resolve( + environmentHome: String?, + resourcePath: String?, + isDirectory: (String) -> Boolean, + ): Resolved? { + if (!environmentHome.isNullOrBlank()) return null + val resources = resourcePath?.takeIf { it.isNotBlank() } ?: return null + val candidate = "${resources.trimEnd('/')}/$DIRECTORY_NAME" + return if (isDirectory(candidate)) { + Resolved(candidate, "$DIRECTORY_NAME in the app bundle's resource directory") + } else { + null + } + } + + /** `NSFileManager`'s answer: a directory exists at [path] (a file there is not one). */ + @OptIn(ExperimentalForeignApi::class) + internal fun isDirectory(path: String): Boolean = memScoped { + val directory = alloc() + NSFileManager.defaultManager.fileExistsAtPath(path, directory.ptr) && directory.value + } +} + +/** + * iOS's half of SPEC L-11: hand the bundled prefix to CPython with `Py_SetPythonHome`. + * + * `Py_SetPythonHome` rather than `setenv("PYTHONHOME")`: it writes CPython's path configuration + * directly, so there is no process-wide variable for a child process or a later reader to inherit, + * and it is the same call desktop makes (L-9). `PyConfig.home` is the non-deprecated route, but the + * library carries no `PyConfig` layout (see `Python3.initialize`). + * + * The string is produced by `Py_DecodeLocale`, which CPython documents as safe before + * initialisation and which decodes UTF-8 on Apple platforms regardless of locale -- what upstream's + * own iOS testbed does for `config.home`. Its buffer is never freed: the C API asks for storage that + * lives as long as the program, and this runs once per process. + */ +@OptIn(ExperimentalForeignApi::class) +@Suppress("DEPRECATION") +internal actual fun applyPackagedPythonHome() { + val home = IosPythonHome.resolve() ?: return + try { + PythonHomeCheck.verifyOrThrow(home.path) + } catch (e: IllegalStateException) { + throw IllegalStateException( + "The Python home this app bundles (${home.source}) is unusable: ${e.message}", + e, + ) + } + val wide = python.native.ffi.bindings.Py_DecodeLocale(home.path, null) + ?: throw IllegalStateException("Py_DecodeLocale could not decode the bundled Python home '${home.path}'.") + python.native.ffi.bindings.Py_SetPythonHome(wide) +} diff --git a/python-multiplatform/src/iosMain/kotlin/python/multiplatform/env/PythonPayload.ios.kt b/python-multiplatform/src/iosMain/kotlin/python/multiplatform/env/PythonPayload.ios.kt index 102ab2075..2446ace6f 100644 --- a/python-multiplatform/src/iosMain/kotlin/python/multiplatform/env/PythonPayload.ios.kt +++ b/python-multiplatform/src/iosMain/kotlin/python/multiplatform/env/PythonPayload.ios.kt @@ -4,37 +4,15 @@ import platform.Foundation.NSBundle /** * iOS's answer to "where did the payload land": `python/` inside the app bundle's resource - * directory — **if** the Xcode project was told to copy it there, which nothing in this repository - * or in `toolchain` does yet. + * directory. * - * ### What exists and what does not + * `toolchain` produces the payload (`build/pythonStaging/ios/python`, or the app's own sources); + * `tools/xcode/install-python.sh`, an Xcode Run Script phase, copies it there and wraps any + * extension module in it as a framework (SPEC L-11, `docs/platforms/ios-app-bundle.md`). An app + * built without that phase has no `python/`, and this returns an empty list. * - * `toolchain`'s `49da1d8` *produces* the payload for iOS and says plainly that it stops there: - * "iOS is produced and not attached: a native framework has no Gradle resource mechanism, and its - * resources go through an Xcode phase in a project file this plugin does not own." So the staged - * directory sits at `build/pythonStaging/ios/python` in the consumer's Gradle build and nothing - * carries it into the `.app`. - * - * The runtime half is here and is complete: once the directory is in the bundle, this finds it and - * `Python3.initialize()` puts it on `sys.path` with no further work. What is missing is exactly - * three things, and they are packaging rather than runtime: - * - * 1. **A Copy Bundle Resources build phase** in the consumer's Xcode project referencing - * `$(SRCROOT)/..//build/pythonStaging/ios/python`, run after the Gradle build phase - * that produces it, with "Create folder references" semantics — a *group* would flatten the - * package directories and every `__init__.py` would collide. - * 2. **A build-order dependency** on that Gradle task, so the phase does not copy an empty or - * stale directory on a clean checkout. `iosApp/` in this repository drives Gradle from a Run - * Script phase already, which is where the ordering has to be expressed. - * 3. **A device/simulator run that proves it**, which is not the same check as desktop's. See - * `iosMain/README.md`: a path under this workspace's external volume does not merely fail for - * an *installed* app, it parks `open$NOCANCEL` forever with 0% CPU. A payload read out of the - * app's own bundle is not subject to that, which is precisely why the answer has to be the - * bundle and not a `PYTHONHOME`-adjacent workspace path. - * - * Until (1) and (2) exist this returns an empty list on every run, and no test in this repository - * can distinguish "the phase is missing" from "the code is wrong" — which is why the *desktop* - * path is the one driven end to end. + * A payload read out of the app's own bundle is also what avoids the external-volume hang + * `iosMain/README.md` records for a workspace path. * * ### `resourcePath`, not `bundlePath` * diff --git a/python-multiplatform/src/iosSimulatorArm64Test/kotlin/python/multiplatform/env/IosPythonHomeTest.kt b/python-multiplatform/src/iosSimulatorArm64Test/kotlin/python/multiplatform/env/IosPythonHomeTest.kt new file mode 100644 index 000000000..cdac05e73 --- /dev/null +++ b/python-multiplatform/src/iosSimulatorArm64Test/kotlin/python/multiplatform/env/IosPythonHomeTest.kt @@ -0,0 +1,77 @@ +package python.multiplatform.env + +import python.multiplatform.Versions +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertFalse +import kotlin.test.assertNotNull +import kotlin.test.assertNull +import kotlin.test.assertTrue + +/** + * SPEC L-11 (issue #59): where an installed iOS app's CPython prefix is resolved from. + * + * The build-time half -- what is staged and copied into the bundle under that name -- is + * `python-multiplatform-gradle-plugin/src/test/.../IosPythonHomeLayoutTest.kt`. + * + * What this cannot find: whether an *installed* app really starts from its bundle. The test binary + * is not an app bundle and always runs with `PYTHONHOME` set by the simulator test task, so the + * bundle branch is exercised here only through the pure overload; the installed-app run in + * `docs/platforms/ios-app-bundle.md` is the check for the rest. + */ +class IosPythonHomeTest { + + private val bundled = "/app/Demo.app/python-multiplatform-home" + + @Test + fun theDirectoryNameIsTheOneThePluginStages() { + // `IosPythonHomeLayout.BUNDLE_DIRECTORY` in python-multiplatform-gradle-plugin. + assertEquals("python-multiplatform-home", IosPythonHome.DIRECTORY_NAME) + } + + @Test + fun anEnvironmentPythonHomeWinsAndNothingIsApplied() { + assertNull(IosPythonHome.resolve("/somewhere/else", "/app/Demo.app") { true }) + } + + @Test + fun theBundledPrefixIsUsedWhenTheEnvironmentHasNone() { + for (environment in listOf(null, "", " ")) { + val resolved = IosPythonHome.resolve(environment, "/app/Demo.app") { it == bundled } + assertEquals(bundled, resolved?.path, "PYTHONHOME='$environment'") + assertTrue("app bundle" in resolved!!.source, resolved.source) + } + } + + @Test + fun aTrailingSlashOnTheResourcePathIsHarmless() { + assertEquals(bundled, IosPythonHome.resolve(null, "/app/Demo.app/") { it == bundled }?.path) + } + + @Test + fun anAppBuiltWithoutThePhaseBehavesAsBefore() { + // No directory in the bundle: nothing is set, and Py_Initialize() does what it always did. + assertNull(IosPythonHome.resolve(null, "/app/Demo.app") { false }) + assertNull(IosPythonHome.resolve(null, null) { true }) + assertNull(IosPythonHome.resolve(null, "") { true }) + } + + @Test + fun theSimulatorTestTaskStillGetsItsOwnPythonHome() { + // `KotlinNativeSimulatorTest` sets SIMCTL_CHILD_PYTHONHOME (python-multiplatform/build.gradle.kts). + // If that ever stopped reaching this process, the line below would be resolving the test + // binary's directory instead -- the regression this pins. + assertNotNull(readEnvVar("PYTHONHOME"), "the simulator test task no longer passes PYTHONHOME") + assertNull(IosPythonHome.resolve()) + } + + @Test + fun theDirectoryProbeTellsADirectoryFromAFile() { + val home = assertNotNull(readEnvVar("PYTHONHOME")) + val tag = Versions.currentVersion.taggedVersionString + assertTrue(IosPythonHome.isDirectory(home), home) + assertTrue(IosPythonHome.isDirectory("$home/lib/python$tag")) + assertFalse(IosPythonHome.isDirectory("$home/lib/python$tag/os.py")) + assertFalse(IosPythonHome.isDirectory("$home/does-not-exist")) + } +} diff --git a/python-multiplatform/src/nativeMain/kotlin/python/multiplatform/env/PackagedPythonHome.native.kt b/python-multiplatform/src/nativeMain/kotlin/python/multiplatform/env/PackagedPythonHome.native.kt deleted file mode 100644 index 8b02c1726..000000000 --- a/python-multiplatform/src/nativeMain/kotlin/python/multiplatform/env/PackagedPythonHome.native.kt +++ /dev/null @@ -1,4 +0,0 @@ -package python.multiplatform.env - -/** No-op: iOS and Android-native processes take `PYTHONHOME` from a real process environment. */ -internal actual fun applyPackagedPythonHome() = Unit diff --git a/sample/src/iosMain/kotlin/org/thisisthepy/python/multiplatform/demo/MainViewController.kt b/sample/src/iosMain/kotlin/org/thisisthepy/python/multiplatform/demo/MainViewController.kt index 965a00be6..df65bc651 100644 --- a/sample/src/iosMain/kotlin/org/thisisthepy/python/multiplatform/demo/MainViewController.kt +++ b/sample/src/iosMain/kotlin/org/thisisthepy/python/multiplatform/demo/MainViewController.kt @@ -10,8 +10,9 @@ import org.thisisthepy.python.multiplatform.demo.ui.App * from inside a composable body -- on every recomposition, and only once the user had expanded * the section that hosted it. * - * On iOS the shipped framework carries no stdlib, so `PYTHONHOME` has to point at one before this - * runs; see `python-multiplatform/src/iosMain/README.md`. + * On iOS the shipped framework carries no stdlib. An app built through `iosApp/`'s "Install Python + * Standard Library and Payload" phase carries one in its bundle and the library finds it there + * (SPEC L-10); see `python-multiplatform/src/iosMain/README.md`. */ fun MainViewController() = run { // Outside the content lambda on purpose: that lambda is `@Composable`, so anything in it runs @@ -59,5 +60,13 @@ private fun dumpDemoSections() { line("6 staticSurface") { PythonDemo.staticSurface() } line("7 awaitFast") { PythonDemo.awaitFastPath() } line("7 awaitSuspending") { PythonDemo.awaitSuspending() } + // SPEC L-10 (issue #59): a stdlib module, a stdlib extension module (a .fwork framework), and the + // payload's module, all read out of the installed bundle -- docs/platforms/ios-app-bundle.md. + line("8 bundle") { + PythonDemo.evaluate( + "(__import__('json').dumps(__import__('example_py').greeting()), __import__('_json').__file__, " + + "__import__('example_py').__file__, __import__('sys').prefix, __import__('sys').executable)" + ) + } println("DEMO ---- end ----") } diff --git a/tools/xcode/install-python.sh b/tools/xcode/install-python.sh new file mode 100755 index 000000000..261fb5f99 --- /dev/null +++ b/tools/xcode/install-python.sh @@ -0,0 +1,124 @@ +#!/usr/bin/env bash +# Xcode Run Script phase: put CPython's standard library and the app's python/ payload into the .app. +# SPEC L-11, issue #59. Procedure and reasoning: docs/platforms/ios-app-bundle.md. +# +# Place the phase AFTER "Copy Bundle Resources" and BEFORE "Embed Frameworks", in a target whose +# ENABLE_USER_SCRIPT_SANDBOXING is NO (the phase reads the Gradle build directory). It: +# 1. runs ` -q :python-multiplatform:stageIosPythonHomeForXcode`, which stages the stdlib +# for the slice Xcode is building (EFFECTIVE_PLATFORM_NAME + ARCHS) and prints +# PYTHON_HOME_DIR= and PYTHON_DYLIB_INFO_TEMPLATE=; +# 2. copies that prefix to /python-multiplatform-home/ (IosPythonHome looks there); +# 3. copies the payload, when one is configured, to /python/ (PythonPayload looks there); +# 4. moves every .so under both into /Frameworks/.framework, leaving a .fwork +# placeholder that CPython's AppleFrameworkLoader follows -- upstream's +# Python.xcframework/build/utils.sh `install_dylib`, because iOS loads no loose binaries. +# +# Environment (all optional except what Xcode sets): +# GRADLEW Gradle wrapper, default ./gradlew (the phase cd's to the Gradle root first) +# PYTHON_HOME_TASK default :python-multiplatform:stageIosPythonHomeForXcode +# PYTHON_PAYLOAD_DIR a directory whose *contents* become /python/ (e.g. the app's own +# Python sources, or toolchain's build/pythonStaging/ios/python) +# PYTHON_PAYLOAD_TASK instead: a Gradle task printing PYTHON_PAYLOAD_DIR= under -q, e.g. +# toolchain's :app:stagePythonBundleIosForXcode +# With neither payload variable set, no payload is copied and any old /python/ is removed. +set -euo pipefail + +: "${CODESIGNING_FOLDER_PATH:?run this from an Xcode build phase (CODESIGNING_FOLDER_PATH is unset)}" +: "${TARGET_BUILD_DIR:?run this from an Xcode build phase (TARGET_BUILD_DIR is unset)}" +: "${UNLOCALIZED_RESOURCES_FOLDER_PATH:?run this from an Xcode build phase}" +: "${EFFECTIVE_PLATFORM_NAME:?run this from an Xcode build phase}" +: "${ARCHS:?run this from an Xcode build phase}" + +bundle="$CODESIGNING_FOLDER_PATH" +resources="$TARGET_BUILD_DIR/$UNLOCALIZED_RESOURCES_FOLDER_PATH" +# On iOS the bundle root *is* the resource directory, and both the runtime lookup +# (NSBundle.resourcePath) and the .fwork paths (relative to the executable's directory) rely on it. +if [ "$(cd "$bundle" && pwd -P)" != "$(cd "$resources" && pwd -P)" ]; then + echo "error: $resources is not the bundle root $bundle; only the flat iOS app layout is supported" >&2 + exit 1 +fi + +gradlew="${GRADLEW:-./gradlew}" +home_task="${PYTHON_HOME_TASK:-:python-multiplatform:stageIosPythonHomeForXcode}" + +# Captured first, parsed second: a failing Gradle must stop the phase here (set -e on the +# assignment), never leave an empty string for rsync to treat as "/". +home_out=$("$gradlew" -q "$home_task") +home_dir=$(printf '%s\n' "$home_out" | sed -n 's/^PYTHON_HOME_DIR=//p') +template=$(printf '%s\n' "$home_out" | sed -n 's/^PYTHON_DYLIB_INFO_TEMPLATE=//p') +[ -n "$home_dir" ] && [ -d "$home_dir" ] || { echo "error: no PYTHON_HOME_DIR from $home_task" >&2; exit 1; } +[ -n "$template" ] && [ -f "$template" ] || { echo "error: no PYTHON_DYLIB_INFO_TEMPLATE from $home_task" >&2; exit 1; } + +stdlib_count=$(find "$home_dir"/lib -maxdepth 2 -path '*/python3.*/os.py' | wc -l | tr -d ' ') +[ "$stdlib_count" = "1" ] || { echo "error: $home_dir has $stdlib_count lib/python3.*/os.py, expected 1" >&2; exit 1; } +stdlib_rel=$(cd "$home_dir" && find lib -maxdepth 1 -type d -name 'python3.*' | head -n 1) + +echo "Installing the Python standard library for $EFFECTIVE_PLATFORM_NAME/$ARCHS from $home_dir" +rsync -a --delete "$home_dir/" "$bundle/python-multiplatform-home/" + +payload_dir="${PYTHON_PAYLOAD_DIR:-}" +if [ -z "$payload_dir" ] && [ -n "${PYTHON_PAYLOAD_TASK:-}" ]; then + payload_out=$("$gradlew" -q "$PYTHON_PAYLOAD_TASK") + payload_dir=$(printf '%s\n' "$payload_out" | sed -n 's/^PYTHON_PAYLOAD_DIR=//p') + [ -n "$payload_dir" ] || { echo "error: no PYTHON_PAYLOAD_DIR from $PYTHON_PAYLOAD_TASK" >&2; exit 1; } +fi +if [ -n "$payload_dir" ]; then + [ -d "$payload_dir" ] || { echo "error: Python payload directory $payload_dir does not exist" >&2; exit 1; } + echo "Installing the Python payload from $payload_dir" + rsync -a --delete --exclude '__pycache__' "$payload_dir/" "$bundle/python/" +else + echo "No Python payload configured (PYTHON_PAYLOAD_DIR / PYTHON_PAYLOAD_TASK); removing any old one" + rm -rf "$bundle/python" +fi + +# --- extension modules as frameworks (upstream utils.sh install_dylib, with the template staged) --- +sign_identity="${EXPANDED_CODE_SIGN_IDENTITY:-}" + +install_dylib() { # $1 = install base relative to the bundle, with trailing slash; $2 = absolute .so + local base="$1" ext_path="$2" + local relative="${ext_path#"$bundle"/}" + local in_base="${relative#"$base"}" + local module + module=$(printf '%s' "$in_base" | cut -d . -f 1 | tr / .) + local framework="Frameworks/$module.framework" + local bundle_id + bundle_id=$(printf '%s' "${PRODUCT_BUNDLE_IDENTIFIER:-python.multiplatform}.$module" | tr _ -) + + if [ ! -d "$bundle/$framework" ]; then + mkdir -p "$bundle/$framework" + cp "$template" "$bundle/$framework/Info.plist" + plutil -replace CFBundleExecutable -string "$module" "$bundle/$framework/Info.plist" + plutil -replace CFBundleIdentifier -string "$bundle_id" "$bundle/$framework/Info.plist" + fi + mv "$ext_path" "$bundle/$framework/$module" + printf '%s\n' "$framework/$module" > "${ext_path%.so}.fwork" + printf '%s\n' "${relative%.so}.fwork" > "$bundle/$framework/$module.origin" + + # /.xcprivacy beside /.cpython-*.so, as upstream looks for it. + local privacy + privacy="$(dirname "$ext_path")/$(basename "$ext_path" | cut -d . -f 1).xcprivacy" + if [ -e "$privacy" ]; then + mv -f "$privacy" "$bundle/$framework/PrivacyInfo.xcprivacy" + fi + + if [ -n "$sign_identity" ]; then + /usr/bin/codesign --force --sign "$sign_identity" ${OTHER_CODE_SIGN_FLAGS:-} -o runtime \ + --timestamp=none --preserve-metadata=identifier,entitlements,flags \ + --generate-entitlement-der "$bundle/$framework" + fi +} + +process_dylibs() { # $1 = directory relative to the bundle + local base="$1" + [ -d "$bundle/$base" ] || return 0 + local count=0 + while IFS= read -r -d '' ext_path; do + install_dylib "$base/" "$ext_path" + count=$((count + 1)) + done < <(find "$bundle/$base" -name '*.so' -print0) + echo "Wrapped $count extension module(s) under $base as frameworks" +} + +process_dylibs "python-multiplatform-home/$stdlib_rel/lib-dynload" +process_dylibs "python" +[ -n "$sign_identity" ] || echo "note: EXPANDED_CODE_SIGN_IDENTITY is empty; extension frameworks were not signed"