feat(venv): support data, include, and scripts schemes#3726
feat(venv): support data, include, and scripts schemes#3726rickeylev wants to merge 26 commits intobazel-contrib:mainfrom
Conversation
ref https://packaging.python.org/en/latest/specifications/binary-distribution-format/#installing-a-wheel-distribution-1-0-py32-none-any-whl for generic data scheme: these are usually installed to the venv root, but that seems ill-advised. So we install into a data sub-directory of the venv.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request enhances wheel support by correctly handling and symlinking data/, bin/, and include/ directories within virtual environments. It updates the venv construction logic to recognize these directories, adds a new DATA symlink kind, and includes these paths in the wheel's data targets. Review feedback highlights a bug in path prefix matching for top-level directories, suggests removing internal 'TO CHECK' comments, and recommends refactoring how data targets are added to avoid redundancy and runfile bloat in non-venv environments.
|
Be sure you check the whl_library extraction starlark code, because it is handling some of the mapping already. |
…into whl.with.data
Uses os.name to correctly assert the Scripts/ and Include/ directory paths when creating the virtual environment on Windows.
Also add debug info to venvs_site_packages_libs_test to help diagnose Windows CI failures.
…addition These changes seemed to cause many regressions in WORKSPACE mode in CI.
Also revert redundant bazel_skylib additions to example WORKSPACE files.
The original target was intended to be internal-only, so it has been renamed and all references updated. No alias was added as per instructions.
…d because data files are always added
|
|
||
| config_setting( | ||
| name = "is_venvs_site_packages", | ||
| name = "_is_venvs_site_packages", |
There was a problem hiding this comment.
This is different from the PR description and the constant elsevhere.
2 participants
Currently, the files from the data, headers, and scripts portions of a
wheel don't end up in the proper sub-directories of the venv. This
means the full files of a distribution aren't available at the typical
location in the venv, making it harder to integrate with standard
tools.
To fix, simply map the directories to paths in the venv and give them
similar treatment as site-packages.
The spec says certain first-leve directories of the
.datadirectorymap to specific scheme paths, which
whl_libraryalready handles.Here's a listing of the
wheel data directory -> install scheme path key -> whl_library directoryrelationships
Relevant reading:
https://packaging.python.org/en/latest/specifications/binary-distribution-format
https://docs.python.org/3/library/sysconfig.html#posix-prefix
https://docs.python.org/3/library/sysconfig.html#nt
The whl_library rule uses posix names for extracting. When
materialized into a binary's venv, platform specific names are used.
Along the way ...
always included as part of depending on the library. They would be
in included in venv_site_packages=yes mode, so this better aligns
behavior of the two modes.
is_venv_site_packagesto_is_venv_site_packages_yestobetter represent the purpose and visibility of it.
DO NOT MERGE: Right now, the generic 'data' schema is being mapped into
the binary's venv under a "data" directory. This doesn't match the
prefix spec, so we probably shouldn't do it. It was just easier
because as it avoids generating conflicting files.