Repository navigation
docs(proposals): add offline build-parallel proposal #1361
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Xaenalt
wants to merge
1
commit into
python-wheel-build:main
Choose a base branch
from
Xaenalt:docs/offline-build-parallel-proposal
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,191 @@ | ||
| # Offline wheel builds from prepared artifacts | ||
|
|
||
| - Author: Sean Pryor | ||
| - Created: 2026-10-04 | ||
| - Status: Draft | ||
|
|
||
| ## What | ||
|
|
||
| Add an opt-in `--offline` mode to `build-parallel`. It consumes a dependency | ||
| graph and the prepared artifacts from an earlier bootstrap run to build wheels | ||
| without repeating source acquisition or source-preparation work. | ||
|
|
||
|
Xaenalt marked this conversation as resolved.
|
||
| ## Why | ||
|
|
||
| Preparing source distributions and their build environments can require network | ||
| access and package-specific hooks. Repeating those steps in every wheel-build | ||
|
Xaenalt marked this conversation as resolved.
|
||
| environment adds work and prevents builds from running in environments with no | ||
| network access. | ||
|
|
||
| Bootstrap already produces the graph and working files needed to separate | ||
| preparation from wheel building. An offline build mode lets operators prepare | ||
| those inputs once, then reuse them in a restricted build environment. | ||
|
|
||
| Separating preparation from building is the intended split between bootstrap | ||
| and the build commands | ||
| ([#995](https://github.com/python-wheel-build/fromager/issues/995)); the | ||
| build steps doing any preparation work at all is leftover coupling. The mode | ||
| is opt-in to preserve the existing one-step behavior, but it is meant to | ||
| become the default, and eventually the only, behavior of the build commands. | ||
|
|
||
| ## Goals | ||
|
|
||
| - Reuse an existing dependency graph, prepared source tree, build metadata, | ||
|
Xaenalt marked this conversation as resolved.
|
||
| source distribution, and build environment. | ||
| - Reuse local wheels for packages that are already built or configured as | ||
|
Xaenalt marked this conversation as resolved.
|
||
| pre-built. | ||
| - Skip `download_source`, `prepare_source`, `prepare_build_environment`, and | ||
| `build_sdist` when `--offline` is enabled. | ||
|
Xaenalt marked this conversation as resolved.
|
||
| - Fail with a package-specific error when a required local artifact is missing, | ||
| without falling back to a remote server. | ||
| - Preserve dependency ordering and the normal wheel-building outputs. | ||
|
|
||
| ## Non-goals | ||
|
|
||
| - Creating or exporting a portable bootstrap bundle. This proposal covers | ||
| consuming an existing prepared workspace. | ||
|
Xaenalt marked this conversation as resolved.
|
||
| - Resolving package versions or regenerating the dependency graph during the | ||
| offline build. | ||
|
Xaenalt marked this conversation as resolved.
|
||
| - Guaranteeing that arbitrary package build scripts cannot access the | ||
| network. Network isolation remains a property of the build environment. | ||
| - Defining artifact digests, provenance, or a cross-host transfer format. | ||
|
|
||
| ## Related issues | ||
|
|
||
| This proposal is partial work toward | ||
| [#995](https://github.com/python-wheel-build/fromager/issues/995) and | ||
| [#797](https://github.com/python-wheel-build/fromager/issues/797), the | ||
| broader goal of fully separating dependency discovery from building, and | ||
| does not close either issue. | ||
|
|
||
| - **Two-stage build** ([#995](https://github.com/python-wheel-build/fromager/issues/995), | ||
| [#797](https://github.com/python-wheel-build/fromager/issues/797)): | ||
| offline mode gives the build commands a way to avoid preparation work, | ||
| but they still perform that work by default, and neither a dedicated | ||
| `prepare` command nor a defined transferable workspace layout is part | ||
| of this proposal. | ||
| - **Discovery fallback** ([#996](https://github.com/python-wheel-build/fromager/issues/996)): | ||
| offline mode never reaches the fallback to PEP 517 discovery hooks | ||
| because it reuses the prepared build environments. Eliminating that | ||
| fallback for online builds remains open. | ||
| - **Sdist as build input** ([#1311](https://github.com/python-wheel-build/fromager/issues/1311), | ||
| [#696](https://github.com/python-wheel-build/fromager/issues/696)): | ||
| offline mode keeps building from the retained source tree, with the | ||
| prepared sdist as a required artifact. The expectation that every | ||
| package is buildable from its own sdist relies on the packaging best | ||
| practices described in #696. Building the wheel from the unpacked | ||
| downstream sdist instead remains follow-up work; the workspace already | ||
| retains the sdist either way. | ||
| - **Git source resolvers** ([#1205](https://github.com/python-wheel-build/fromager/issues/1205), | ||
| [#1206](https://github.com/python-wheel-build/fromager/issues/1206)): | ||
| offline mode depends on bootstrap producing source trees and sdists, so | ||
| packages using the git-clone source resolvers need that preparation | ||
| work to land first. Offline mode does not change how sources are | ||
| prepared. | ||
| - **Build-tag acceptance** ([#1360](https://github.com/python-wheel-build/fromager/issues/1360)): | ||
| offline mode applies the same wheel acceptance rules as an online | ||
| build, so it takes no position on what those rules should be. | ||
| Consolidating them remains follow-up work. | ||
|
|
||
| ## How | ||
|
|
||
| A workflow prepares a package graph and workspace by running | ||
| `fromager --no-cleanup bootstrap --sdist-only`, then runs | ||
| `build-parallel --offline` against that graph and workspace. For each graph | ||
| node, Fromager reuses a suitable local wheel when available. Otherwise, it | ||
| builds the wheel from the prepared source tree using the existing build | ||
| environment. Packages configured as pre-built require a local pre-built | ||
| wheel. Packages whose wheels were already built during preparation, such as | ||
| build requirements, are satisfied from the local wheel repository. | ||
|
|
||
| Every package is expected to be buildable from its own sdist; a package that | ||
| cannot be is a packaging bug. Offline mode reuses the prepared source tree to | ||
| avoid repeating the preparation work, and the prepared sdist remains a | ||
| required artifact for each node. | ||
|
|
||
| Offline mode skips the four source and environment preparation phases listed | ||
| above and does not use external source or wheel servers. It still runs the | ||
| wheel build and its applicable post-build hooks. The workspace therefore needs | ||
| to contain the prepared source, metadata, source distribution, and build | ||
| environment for each package that must be built from source. | ||
|
|
||
| The graph and workspace should come from the same preparation run and build | ||
| context. Fromager checks package and version metadata where available and | ||
| stops with a clear error if a required artifact is missing or inconsistent; it | ||
| does not retry the skipped preparation steps online. | ||
|
|
||
| The prepared sdist for a node is located by its expected filename, including | ||
| any package naming hooks, falling back to the name and version parsed from | ||
| the files present. The prepared source tree is validated against its build | ||
| metadata. Sdist filenames carry no build tag, so prepared sdists from | ||
| different runs with the same name and version are not distinguished from each | ||
| other; verifying content digests is a non-goal. | ||
|
|
||
| ## Preparing the workspace | ||
|
|
||
| The preparation run is `fromager --no-cleanup bootstrap --sdist-only`, | ||
| started from a clean output directory so that the offline inputs are exactly | ||
| what the preparation produces and nothing depends on leftover state. Both | ||
| options are required: | ||
|
|
||
| - `--sdist-only` prepares sdists and build environments without building the | ||
| remaining wheels, so there is work left for the offline build. | ||
| - `--no-cleanup` keeps the prepared source trees and build environments in | ||
| the work directory. By default Fromager removes each package's source tree | ||
| as soon as its build metadata is recorded. A workspace prepared without | ||
| `--no-cleanup` has no source trees to reuse, so every from-source package | ||
| fails with an "offline mode requires prepared source" error. | ||
|
|
||
| The preparation run and the offline build are coupled in three ways: | ||
|
|
||
| - **Package settings.** Expected wheel build tags are computed from the | ||
| package settings changelogs and hooks in both phases, so both phases must | ||
| use the same settings directory and overrides. A local wheel whose build | ||
| tag no longer matches the settings is ignored, and the build then fails if | ||
| no prepared source exists for the package. | ||
| - **Constraints.** Version resolution happens during preparation and the | ||
| resolved versions are recorded in the graph, so the offline build does not | ||
| need constraints and does not consult them. Fromager reads every | ||
| `--constraints-file` argument, including remote URLs, when it starts and | ||
| before any command runs, so the offline invocation must reference local | ||
| files only, or pass no constraints at all. | ||
| - **Build image.** Build environment directory names embed the interpreter | ||
| version, and the `python` inside each environment is a symlink to the base | ||
| interpreter that created it. The offline build must run in the same build | ||
| image (or an image with the same interpreter at the same path) as the | ||
| preparation run. The offline phase also invokes the `uv` and `wheel` | ||
| command line tools. | ||
|
|
||
| ## Limitations | ||
|
|
||
| `--offline` prevents Fromager from fetching sources, preparing them, creating | ||
| build environments, or building source distributions during `build-parallel`. | ||
| It does not prevent a package's wheel-build hook or build tool from attempting | ||
|
Xaenalt marked this conversation as resolved.
|
||
| network access. Plugins configured with `network_isolation: false` may still | ||
| attempt remote access, and tools such as Bazel may fetch dependencies during | ||
| the wheel build. Strict network isolation must be applied by the container or | ||
| build environment. | ||
|
|
||
| The offline build does not consult remote wheel caches or package indexes. | ||
| Each graph node is still checked against the local wheel repository, with the | ||
| same reuse rules as an online build, to decide whether it needs to be built. | ||
| A wheel that exists only remotely, including in the index the built wheels | ||
| will be published to, can only be reused by populating the local wheel | ||
| repository during preparation. | ||
|
|
||
| Offline mode does not invoke the package hooks for the phases it skips | ||
| (source download and preparation, build environment setup, and sdist | ||
| building). The `build_wheel` hook and the prebuilt-wheel and post-build hooks | ||
| still run. The plugin interface for distinguishing bootstrap behavior from | ||
| build behavior is unchanged; refining it is follow-up work. | ||
|
|
||
| ## Example | ||
|
|
||
| ```bash | ||
| # Preparation, with network access: resolves versions, prepares sdists, | ||
| # source trees, and build environments. | ||
| fromager --no-cleanup bootstrap --sdist-only 'stevedore==5.2.0' | ||
|
|
||
| # Offline wheel build: no external network access required. | ||
| fromager build-parallel --offline work-dir/graph.json | ||
| ``` | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.