diff --git a/peps/pep-0832.rst b/peps/pep-0832.rst index 7790a5aca50..0a6e2436f4f 100644 --- a/peps/pep-0832.rst +++ b/peps/pep-0832.rst @@ -21,135 +21,70 @@ location. Motivation ========== -Typically, when someone is working on a project larger than a single file, a -long-lived virtual environment is desirable (the single file case is covered by -:pep:`723` and ephemeral virtual environments). As such, tools working on the -user's behalf may want to create and/or use the same virtual environment. These -tools could be custom scripts in the project like running the test script or -third-party tools like package installers. - -Unfortunately, there's no guidance on where tools should put a virtual -environment or how to find where one is ultimately put. There's somewhat of a -convention in the community to put a virtual environment locally at the root of -the project in a directory named :file:`.venv`, but being a convention means it -isn't consistently followed. As well, there is no mechanism to point to a -virtual environment regardless of its location. - -This lack of guidance on where tools can find a virtual environment makes the -developer experience less smooth than it could be. If you rely on shell -activation to use the proper virtual environment, then you have to make sure to -do that (either manually or by configuring some automatic shell integration) as -well as not to accidentally reuse that activated shell with another project (or -have set up shell automation to handle the deactivation). Otherwise tools need -to guess or have custom logic per tool that creates virtual environments or to -ask users to manually specify where the virtual environment is. - -For virtual environment creation, it leads to different instructions per -project on what to do. And those instructions can be critical if scripts and -project configuration rely on the virtual environment being in a certain place. -This can also be an issue when a tool creates a virtual environment -automatically but it isn't obvious where the environment was placed. - - -Rationale -========= - -There are three aspects to where a virtual environment is placed. The first is -whether the virtual environment is local to the project or stored globally with -other virtual environments. Keeping the virtual environment local means that it -is isolated and unique to the project. As well, it means that if you delete the -project you also delete the virtual environment. If you store the virtual -environment globally then you can share it among multiple projects and delete -all virtual environments at once by deleting the directory that contains them -all. Keeping virtual environments global also means it won't be backed up -automatically if a project is stored e.g. in a directory automatically backed -up to remote storage where you pay based on how much storage you use. - -Another aspect is the directory name used for the virtual environment -(although this really only affects local virtual environments). If one views -virtual environments as more of an implementation detail, a directory name -starting with :file:`.` seemingly makes sense to mark it hidden or de-emphasized -in various tools such as shells and code editors. But hiding it can make -accessing the directory harder via tools that don't expose paths starting with -a ``.``. - -Lastly, there's whether you have one virtual environment at a time or many. -Having only one can minimize disk space for some tools and keeps it simple by -not trying to manage multiple virtual environments. Having multiple virtual -environments, though, means not having to constantly recreate virtual -environments when e.g. needing to test against multiple Python versions. - -This PEP takes a two-pronged approach to making virtual environments easily -discoverable while supporting all aspects mentioned above. First, by default, -the virtual environment for the project could be put in the :file:`.venv` -directory of the project (this can be a hardlink, symlink, etc.). This name -has been chosen due to preexisting tool support: -`Poetry `__ -will detect a virtual environment in such a location, -`PDM `__ -and `uv `__ -create their environments there by default already ( -`Hatch can support `__ -a virtual environment there). `VS Code `__ -will select it automatically while you can configure -`PyCharm `__ -to use it. The default :file:`.gitignore` file for -`GitHub `__, -`GitLab `__, -and `Codeberg `__ -already ignore the path. - -But for various reasons (from personal preference to preexisting tool defaults), -the :file:`.venv` directory in the project root may not work. In those cases, a -:file:`.venv` **file** which points to the virtual environment by default -should be provided in the project's root directory (i.e. the same location as -specified above for the :file:`.venv` directory). This file should point to the -virtual environment to use by default; there can be other virtual environments -for the project, but the :file:`.venv` file should point to the virtual -environment to be used if no preference is specified. While a hardlink, -symlink, etc. for :file:`.venv` could serve the same purpose, not all file -systems support such links. As well, situations like the automatic backup case -mentioned previously require a level of indirection so that backup tools don't -implicitly follow into a virtual environment and back it up. - -The :file:`.venv` file is meant to represent the virtual environment a workflow -tool is expected to use that is external to the one that wrote the -:file:`.venv` file (e.g. Hatch wrote the file and VS Code is going to read it). -This means that a workflow tool shouldn't update the :file:`.venv` file as it -runs a test suite through multiple versions of Python. But if the workflow tool -has a command to control what virtual environment is used when running Python, -then the file should be updated as necessary to match what environment the -workflow tool would use (e.g. :file:`.venv` should always match what virtual -environment `'hatch run' `__ -would use). This is not expected to cause a "noisy neighbour" problem as it's -not expected to change that rapidly. - +Imagine you are on your Mac laptop and you double-click your desktop shortcut +to launch Emacs (feel free to substitute "Mac" and "Emacs" with your preferred +OS and editor, respectively). You open the directory for your project in Emacs. +Now, how is Emacs (or any other tool for that matter) supposed to know where +the virtual environment for your project is? There's no possible detection of +an activated virtual environment via the ``VIRTUAL_ENV`` environment variable +as you didn't launch from a terminal. You potentially could scan all +subdirectories for a :file:`pyvenv.cfg` file to find the virtual environment, +but that assumes the virtual environment is kept locally with the project and +that there is only one of them and not several to ambiguously choose from. What +are tools like code editors which need access to the virtual environment being +used to provide functionality like auto-complete to do when there is currently +no standardized way to tell anyone where the virtual environment is? Currently, +tools like editors have to hard-code a search algorithm for every tool that +they choose to support. As well, they can document any conventions they +support, but that assumes you or the tool you use to manage your virtual +environments follow those conventions. + +And this is not a hypothetical issue. The author of this PEP was the dev +manager for Python support in VS Code for 7 years and saw firsthand the +user struggles and constant feature requests involving trying to find one's +preferred/default virtual environment for a project. + +This issue is also not restricted to code editors. Other tools have a need to +access the virtual environment to know what is installed. One example is +type checkers which need access to the packages that are installed to +appropriately gather type annotations for 3rd-party code in order to type check +the user's code. + +The goal of this PEP is to provide a specification for tools which +create/manage virtual environments a way to tell other tools where the +(default) virtual environment for a project is. In the case of a project which +has multiple virtual environments, this PEP is meant to allow for specifying +the default or "active" virtual environment so users are not forced to make a +choice of virtual environment to use if one does not want to make such a +decision (e.g. at first launch of their code editor). Please note this PEP +neither condones nor discourages having multiple virtual environments for a +single project; it is neutral as to whether having a single virtual environment +or multiple ones is good or bad. Specification ============= -The virtual environment for a project SHOULD be in a directory named +This PEP does not define what the "root of a project" means, but the assumption +is it is the directory one would open in their code editor to work on a +project's code. This could be the directory where the project's +:file:`pyproject.toml` lives, or potentially the top directory of a monorepo. + +The virtual environment for a project MAY be in a directory named :file:`.venv` (i.e. :file:`.venv/pyvenv.cfg` will exist which can be used to -detect the existence of a virtual environment) in the root of the project -(typically next to the :file:`pyproject.toml` file) if it makes sense for it -to reside there (e.g. the tool creating the virtual environment doesn't already -use a different location by default or the user didn't specify a location). -This can be a hardlink or symlink to a virtual environment. +detect the existence of a virtual environment) in the root of the project. In all other situations where placing a virtual environment at the project root in a :file:`.venv` directory is not possible or desirable, a :file:`.venv` -**file** SHOULD be written in the project root instead. The file's contents -MUST be a single line containing the path to the directory containing the +**file** SHOULD be written in the project root instead. The file MUST contain +at least a single line recording the path to the directory of the virtual environment (i.e. the directory containing :file:`pyvenv.cfg`). The -file MUST be encoded in UTF-8. A single trailing newline in the file is -acceptable (either ``\r\n`` or ``\n``), and so code reading the file should -strip off any trailing newline. The path in the file is expected to be specific -to the machine it's on, so there are no requirements on path formatting (i.e. a -POSIX path is not required). The path MAY be relative to the :file:`.venv` file -to ease creation of the file by hand. Tools MUST verify that the directory the -file points to exists before using it to prevent tools from being tricked into -e.g. blindly passing the file contents into -``subprocess.run(..., shell=True)``. +file MUST be encoded in UTF-8. If ``\r\n`` or ``\n`` are contained in the file +then the path is considered from the start of the file until the first newline +in the file with the rest of the file's contents ignored but reserved for +future use cases, otherwise the path is the entire contents of the file. There +are NO requirements on path formatting (i.e. a POSIX path is not required). The +path MAY be relative to the :file:`.venv` file. Tools SHOULD verify that the +directory the file points to exists before using it. Tools looking for a virtual environment SHOULD look for the :file:`.venv` directory or file and handle them appropriately. Tools SHOULD NOT prefer one @@ -158,7 +93,9 @@ looks up through parent directories for a virtual environment, it shouldn't look for a directory first and then a file; the first thing found with the :file:`.venv` path name should be chosen). Sharing the same path name for both the directory and file means there is no precedence issue within the same -directory. +directory. If the found ``.venv`` is a symlink, it does NOT alter how the +resolved file is treated; symlinks SHOULD be treated as if they were a regular +entry in the file system. This PEP proposes some changes to the :mod:`venv` module to go along with the above recommendations: @@ -188,10 +125,10 @@ above recommendations: new parameter to :meth:`venv.EnvBuilder.create`. It will be an error to use the option when multiple *ENV_DIR* arguments are provided. #. A function named - ``executable(dir: os.PathLike, *, traverse: bool = False) -> pathlib.Path`` - will be added; it will look for a virtual environment in *dir* at - ``DEFAULT_NAME`` (directory or redirect file) and return the path to the - ``python`` executable for the virtual environment, raising + ``executable(dir: os.PathLike, name: str = DEFAULT_NAME, *, traverse: bool = False) -> pathlib.Path`` + will be added; it will look for a virtual environment in *dir* at *name* + (directory or redirect file; defaults to ``DEFAULT_NAME``) and return the + path to the ``python`` executable for the virtual environment, raising an exception if the path to a virtual environment is not found or the virtual environment is somehow corrupted. If *traverse* is true, then traversal through the parent directories of *dir* to look for @@ -209,11 +146,105 @@ and thus static on the file system. The guidance of NOT committing your actual virtual environment to version control is unchanged by this PEP. +Rationale +========= + +There are three aspects to where a virtual environment is placed. The first is +whether the virtual environment is local to the project or stored globally with +other virtual environments. Keeping the virtual environment local means that it +is isolated and unique to the project. As well, it means that if you delete the +project you also delete the virtual environment. If you store the virtual +environment globally then you can share it among multiple projects and delete +all virtual environments at once by deleting the directory that contains them +all. Keeping virtual environments global also means it won't be backed up +automatically if a project is stored e.g. in a directory automatically backed +up to remote storage where you pay based on how much storage you use. + +Another aspect is the directory name used for the virtual environment +(although this really only affects local virtual environments). If one views +virtual environments as more of an implementation detail, a directory name +starting with :file:`.` seemingly makes sense to mark it hidden or de-emphasized +in various tools such as shells and code editors. But hiding it can make +accessing the directory harder via tools that don't expose paths starting with +a ``.``. + +Lastly, there's whether you have one virtual environment at a time or many. +Having only one can minimize disk space for some tools and keeps it simple by +not trying to manage multiple virtual environments. Having multiple virtual +environments, though, means not having to constantly recreate virtual +environments when e.g. needing to test against multiple Python versions. + +This PEP takes a two-pronged approach to making virtual environments easily +discoverable while supporting all aspects mentioned above. First, this PEP +suggests putting the virtual environment in the :file:`.venv` +directory of the project (this can be a hardlink, symlink, etc.). This name +has been chosen due to preexisting tool support: + +- `Poetry `__ + will detect a virtual environment in such a location, +- `PDM `__ + creates virtual environments there already +- `uv `__ + creates environments there already +- `Hatch can support `__ + a virtual environment there) +- `VS Code `__ + will select it automatically, while still allowing configuration +- `PyCharm `__ + will use it +- `GitHub `__ + has a default :file:`.gitignore` which ignores :file:`.venv` +- `GitLab `__ + has a default :file:`.gitignore` which ignores :file:`.venv` +- `Codeberg `__ + has a default :file:`.gitignore` which ignores :file:`.venv` + +But for various reasons (from personal preference to preexisting tool defaults), +the :file:`.venv` directory in the project root may not work. In those cases, a +:file:`.venv` **file** which points to the virtual environment by default +should be provided in the project's root directory (i.e. the same location as +specified above for the :file:`.venv` directory). This file should point to the +virtual environment to use by default; there can be other virtual environments +for the project, but the :file:`.venv` file should point to the virtual +environment to be used if no preference is specified. While a symlink for +:file:`.venv` could serve the same purpose, not all file systems support +symlinks. As well, situations like automatic backup of a directory to a cloud +backup solution require a level of indirection so that backup tools don't +implicitly follow into a virtual environment and back it up. + +The :file:`.venv` file is meant to represent the virtual environment a workflow +tool is expected to use that is external to the one that wrote the +:file:`.venv` file (e.g. Hatch wrote the file and VS Code is going to read it). +This means that a workflow tool shouldn't update the :file:`.venv` file when +running a test suite through multiple versions of Python. But if the workflow +tool has a command to control what virtual environment is used when running +Python, then the file should be updated as necessary to match what environment +the workflow tool would use (e.g. :file:`.venv` should always match what +virtual environment `'hatch run' `__ +would use). This is not expected to cause a "noisy neighbour" problem as it's +not expected to change that rapidly. + +The format for the :file:`.venv` redirect file is for ease of use. Allowing +newlines in the file makes it easy to create or edit the file in a code editor +that automatically adds newlines to the end of a file. Only reading up to the +first newline, if one exists, also allows for adding more data to the file in +the future. It also allows for easy shell scripting to read the file, e.g. +``head -n 1 .venv | tr -d '\n'`` or ``Get-Content .venv -TotalCount 1``. + +Having tools check for the existence of the path before using it is to prevent +tools from being tricked into e.g. blindly passing the file contents into +``subprocess.run(..., shell=True)``. + + Project Support for this PEP ============================ Speaking to various tool maintainers about this PEP: +.. note:: + Any tool without a link to an expression of (no) support gave that + information privately, but with permission to state publicly. + - Supports 1. PDM (Frost Ming) @@ -222,6 +253,8 @@ Speaking to various tool maintainers about this PEP: 4. Virtualenv (Bernát Gábor) 5. Tox (Bernát Gábor) 6. Hatch (Cary Hawkins) + 7. `PyCharm `__ (Mark Smith) + 8. `library-skills `__ (Sebastián Ramírez) - Lukewarm @@ -268,7 +301,7 @@ How to Teach This For new users, they can be told that ``python -m venv`` creates a virtual environment in :file:`.venv`, and that any other tool that creates a virtual -environment on their behalf. +environment on their behalf can do the same. For experienced users, they should be taught the default location for a project's virtual environment is at the root of the project in :file:`.venv`. @@ -304,60 +337,16 @@ Discussing alternative names was viewed as bikeshedding. Open Issues =========== -An API for discovering environments ------------------------------------ - -While what the PEP currently proposes is simple, there has been some concern -that it's *too* simple. As such, there's concern that an API to allow -for broader environment discovery is necessary. - -A possible way to provide such an API is to introduce a ``[workflow]`` table to -:file:`pyproject.toml`. That could have an ``environments`` key that holds a -table specifying how to run a discovery app to get environment details in -JSON. For example: - -.. code-block:: toml - - [workflow] - environments = {tool = "...", command = ["..."]} - -The command could output JSON that lists any and all known environments. For -flexibility, the output could include more than just virtual environments, such as -global installs of Python. The output could also specify the environment to use -by default if a tool doesn't have a reason or way to make a choice of which -environment to use. - -.. code-block:: json - - { - "version": "1.0", // ... or whatever to version the JSON schema. - "default": 0, // Optional; index into - "environments": [ - { - "type": "virtual", - "path": ".venv", - "python_version": "...", // Optional - "name": "..." // Optional - }, - ] - } - -The discovery tool could also create a virtual environment as a side-effect -of being called. As well, it could write out a ``.venv`` file or directory if -it makes sense. Having ``.venv`` checked for prior to calling the discovery -tool would help avoid the overhead of calling the discovery tool. - -But this expands the complexity of this PEP. It also isn't necessary to -be a part of this PEP as it could be added later on. - - -"MAY" instead of "SHOULD" -------------------------- +List all known virtual environments in the redirect file +-------------------------------------------------------- -Some have suggested removing the "SHOULD" for ``.venv`` in either the directory -or file case and make it a "MAY". That would weaken the PEP, but that is -what some want in order to not feel like a specific workflow is being forced -upon tool authors. +While the redirect file format is designed for future usage, this PEP could +choose to just use that space now instead of in some future PEP. The extra data +in the file could record other virtual environments that the project has. +Optionally, the path could be separated from a labelled name by a ``\t``. The +default virtual environment would be allowed to be listed in the labelled +section if an explicit label was desired. Another option would to record such +data in a JSON trailer in the file. Acknowledgements @@ -372,7 +361,17 @@ this PEP. Change History ============== -N/A +- 23-Apr-2026: Add PyCharm and library-skills support; have redirect files + read up to the first newline; clarify there is no opinion to + having multiple virtual environments; explicitly use the + code editor example for the motivation; have + ``venv.executable()`` be configurable for the virtual + environment name; clarify symlinks are not to be treated in any + special way; move the Rationale after the Specification and + simplify the latter by moving details to the former; loosened + things involving "MAY", "SHOULD", and "NOT" so tools are not + required to do anything beyond how they interpret a redirect + file. Copyright