tmpdir: add tmp_path_layout ini for per-rootdir retention - #14938
Open
jawauntb wants to merge 3 commits into
Open
tmpdir: add tmp_path_layout ini for per-rootdir retention#14938jawauntb wants to merge 3 commits into
jawauntb wants to merge 3 commits into
Conversation
Each pytest-of-<user>/pytest-N/ session dir now contains a `.origin` file next to `.lock`, recording rootpath, pytest version, PID, and hostname. External cleanup tooling (workstation janitors, CI cleanup steps, editor temp sweeps) can attribute a session dir to its project by reading a file, instead of walking /proc or lsof. Writing is best-effort: any OSError during the write is swallowed so a read-only mount, permissions error, or full disk never breaks a test run. Refs pytest-dev#14935.
Adds a new ini option `tmp_path_layout` with values `"flat"` (the current default) and `"per-rootdir"`. Under `"per-rootdir"`, numbered directories are nested one level deeper under a stable token derived from the pytest `rootpath` (`<slug>-<8-char-hash>`), so `tmp_path_retention_count` applies per project instead of across every rootdir that shares a user account. Under `"flat"` layout, retention_count=3 with concurrent work in three git worktrees means each worktree effectively gets one slot, and any fourth run in one worktree evicts the oldest run of another. The per-rootdir layout gives each rootdir its own numbered sequence, so retention_count=3 means "3 most recent runs of this project" — which is what almost every user assumes it already means. Default is `"flat"` for one release for backwards compatibility. A follow-up will flip the default after user feedback. The slug is portable (`[A-Za-z0-9._-]`); the 8-char blake2b hash of the absolute rootpath disambiguates homonyms like `~/work/foo` vs `~/play/foo`. Refs pytest-dev#14935. Depends on pytest-dev#14936 (adds the `_rootpath` plumbing on `TempPathFactory` this PR needs). Rebase order: land the origin sidecar PR first.
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #14935 (part 3 of 3).
Depends on #14936 (the
.originsidecar PR, which adds the_rootpathplumbing on
TempPathFactorythis PR reuses). Please review after #14936;I'll rebase this branch off
mainonce #14936 lands.What
Adds a new ini option
tmp_path_layoutwith values:"flat"— the current default. Numbered dirs sit directly underpytest-of-<user>/pytest-N. Nothing changes."per-rootdir"— numbered dirs are nested one level deeper under astable token derived from
config.rootpath:Why
Under
flat,tmp_path_retention_count(default 3) applies to a singlepool shared across every rootdir a user has ever run pytest against. Two
practical consequences:
a fourth run in worktree B evicts the oldest run of worktree A, even
though the two projects are unrelated. Users who work in multiple
checkouts (worktrees, side clones, unrelated projects) lose scratch they
still needed, invisibly.
user reads
tmp_path_retention_count = 3as "keep the last 3 runs ofmy project", not "keep the last 3 pytest runs from any project under
my user account". The fact that it does the latter is the root cause of
several existing reports (Cleaning up tmpdir's #1120, flag to cleanup generated tmp_path objects #7465, Clarify whether and when
tmp_pathis deleted after each use #8036, discussion How can I completely cleanup temporary directories from tmp_path? #10325).per-rootdirmakes the layout match the mental model.Token design
<slug>-<8-char-hash>:.name, sanitised to[A-Za-z0-9._-], capped at32 characters. Keeps
ls /tmp/pytest-of-me/human-readable.Disambiguates homonyms:
~/work/fooand~/play/fooget differentsubdirectories.
_rootdir_slugand_rootdir_tokenare internal helpers, both covered bytests.
Backwards compatibility
"flat". No user sees any behaviour change without opting in.TempPathFactory.__init__gains alayoutkwarg with a"flat"default,so external callers constructing the factory directly (a handful of
tests do this, and any third-party plugins that might) are unaffected.
--basetemppath is untouched (given basetemp bypasses the layout entirely).PYTEST_DEBUG_TEMPROOTbehaviour is unchanged.pytest-of-<user>/pytest-N/paths under
flatlayout changes.Rollout plan
tmp_path_layoutlands with default"flat". Userswho want the new behaviour opt in.
"per-rootdir"(separate PR, informed byfeedback here).
"flat"if no dependents on the exact pathlayout have surfaced.
The rollout is deliberately conservative — the layout choice matters to any
external tool that hardcodes assumptions about the
pytest-of-*/pytest-Npath (there are a few in the ecosystem).
Interaction with #14936 (
.originsidecar) and #14937 (pid liveness)These three PRs are independent in the sense that each is useful without
the others, but they compound:
.origin(tmpdir: write .origin sidecar to record session ownership #14936) lets external tools attribute a session dir.Together they make retention correct and attributable and deterministic.
Tests
New
TestPerRootdirLayoutclass intesting/test_tmpdir.py:test_flat_is_default— default layout still writes directly underpytest-of-<user>/.test_per_rootdir_nests_under_token— opting in nests under thecomputed token.
test_per_rootdir_retention_is_scoped_per_project— the payoff test.Three sessions in proj-A, three in proj-B, retention_count=1: each
project keeps one dir, neither evicts the other.
test_rootdir_token_disambiguates_homonyms— same basename, differentabsolute path → different tokens.
test_rootdir_token_sanitises_unsafe_names— slug portion only containsfilesystem-safe characters.
test_tmp_path_layout_invalid— invalid ini value fails cleanly.Local runs:
test_tmpdir.py+test_pytester.py+test_config.py+test_pathlib.py→ 519 passed, 3 skipped, 2 xfailed.Changelog
changelog/14935.feature.rst.Not in scope
pytest-of-<user>/pytest-Ndirs into per-rootdir subtrees when a user opts in. The two layouts happily
coexist; the old flat pool ages out under its own retention. If a follow-up
wants a one-time migration on the default flip, that can be its own PR.
--basetemppath. Out of scope; users who set--basetemphave already committed to a specific path.
🤖 Generated with Claude Code
https://claude.ai/code/session_017o9EnCzHyTdYKdcXshMyMZ