Skip to content

tmpdir: add tmp_path_layout ini for per-rootdir retention - #14938

Open
jawauntb wants to merge 3 commits into
pytest-dev:mainfrom
jawauntb:jb/tmpdir-per-rootdir-layout
Open

tmpdir: add tmp_path_layout ini for per-rootdir retention#14938
jawauntb wants to merge 3 commits into
pytest-dev:mainfrom
jawauntb:jb/tmpdir-per-rootdir-layout

Conversation

@jawauntb

Copy link
Copy Markdown

Refs #14935 (part 3 of 3).

Depends on #14936 (the .origin sidecar PR, which adds the _rootpath
plumbing on TempPathFactory this PR reuses). Please review after #14936;
I'll rebase this branch off main once #14936 lands.

What

Adds a new ini option tmp_path_layout with values:

  • "flat" — the current default. Numbered dirs sit directly under
    pytest-of-<user>/pytest-N. Nothing changes.
  • "per-rootdir" — numbered dirs are nested one level deeper under a
    stable token derived from config.rootpath:
/tmp/pytest-of-me/
├── proj-a-8f3c1a2b/          # blake2b(rootpath), 8 chars
│   ├── pytest-0
│   ├── pytest-1
│   └── pytest-current -> pytest-1
├── proj-b-1d9f4e77/
│   ├── pytest-0
│   ├── pytest-1
│   └── pytest-2

Why

Under flat, tmp_path_retention_count (default 3) applies to a single
pool shared across every rootdir a user has ever run pytest against. Two
practical consequences:

  1. Cross-project eviction. With concurrent work in three git worktrees,
    a fourth run in worktree B evicts the oldest run of worktree A, even
    though the two projects are unrelated. Users who work in multiple
    checkouts (worktrees, side clones, unrelated projects) lose scratch they
    still needed, invisibly.
  2. The mental model doesn't match the implementation. Almost every
    user reads tmp_path_retention_count = 3 as "keep the last 3 runs of
    my project", not "keep the last 3 pytest runs from any project under
    my user account". The fact that it does the latter is the root cause of
    several existing reports (Cleaning up tmpdir's #1120, flag to cleanup generated tmp_path objects #7465, Clarify whether and when tmp_path is deleted after each use #8036, discussion How can I completely cleanup temporary directories from tmp_path? #10325).

per-rootdir makes the layout match the mental model.

Token design

<slug>-<8-char-hash>:

  • Slug: the rootpath's .name, sanitised to [A-Za-z0-9._-], capped at
    32 characters. Keeps ls /tmp/pytest-of-me/ human-readable.
  • Hash: 4 bytes of blake2b of the absolute rootpath (8 hex chars).
    Disambiguates homonyms: ~/work/foo and ~/play/foo get different
    subdirectories.

_rootdir_slug and _rootdir_token are internal helpers, both covered by
tests.

Backwards compatibility

  • Default is "flat". No user sees any behaviour change without opting in.
  • TempPathFactory.__init__ gains a layout kwarg with a "flat" default,
    so external callers constructing the factory directly (a handful of
    tests do this, and any third-party plugins that might) are unaffected.
  • --basetemp path is untouched (given basetemp bypasses the layout entirely).
  • PYTEST_DEBUG_TEMPROOT behaviour is unchanged.
  • Nothing about existing session directories or pytest-of-<user>/pytest-N/
    paths under flat layout changes.

Rollout plan

  • This release (N): tmp_path_layout lands with default "flat". Users
    who want the new behaviour opt in.
  • N+2: flip the default to "per-rootdir" (separate PR, informed by
    feedback here).
  • N+4: consider removing "flat" if no dependents on the exact path
    layout have surfaced.

The rollout is deliberately conservative — the layout choice matters to any
external tool that hardcodes assumptions about the pytest-of-*/pytest-N
path (there are a few in the ecosystem).

Interaction with #14936 (.origin sidecar) and #14937 (pid liveness)

These three PRs are independent in the sense that each is useful without
the others, but they compound:

Together they make retention correct and attributable and deterministic.

Tests

New TestPerRootdirLayout class in testing/test_tmpdir.py:

  • test_flat_is_default — default layout still writes directly under
    pytest-of-<user>/.
  • test_per_rootdir_nests_under_token — opting in nests under the
    computed token.
  • test_per_rootdir_retention_is_scoped_per_project — the payoff test.
    Three sessions in proj-A, three in proj-B, retention_count=1: each
    project keeps one dir, neither evicts the other.
  • test_rootdir_token_disambiguates_homonyms — same basename, different
    absolute path → different tokens.
  • test_rootdir_token_sanitises_unsafe_names — slug portion only contains
    filesystem-safe characters.
  • test_tmp_path_layout_invalid — invalid ini value fails cleanly.

Local runs: test_tmpdir.py + test_pytester.py + test_config.py +
test_pathlib.py → 519 passed, 3 skipped, 2 xfailed.

Changelog

changelog/14935.feature.rst.

Not in scope

  • Legacy migration. This PR does not move existing pytest-of-<user>/pytest-N
    dirs into per-rootdir subtrees when a user opts in. The two layouts happily
    coexist; the old flat pool ages out under its own retention. If a follow-up
    wants a one-time migration on the default flip, that can be its own PR.
  • Deprecation of flat. Not yet.
  • Migration of --basetemp path. Out of scope; users who set --basetemp
    have already committed to a specific path.

🤖 Generated with Claude Code

https://claude.ai/code/session_017o9EnCzHyTdYKdcXshMyMZ

Each pytest-of-<user>/pytest-N/ session dir now contains a `.origin`
file next to `.lock`, recording rootpath, pytest version, PID, and
hostname. External cleanup tooling (workstation janitors, CI cleanup
steps, editor temp sweeps) can attribute a session dir to its project
by reading a file, instead of walking /proc or lsof.

Writing is best-effort: any OSError during the write is swallowed so
a read-only mount, permissions error, or full disk never breaks a
test run.

Refs pytest-dev#14935.
Adds a new ini option `tmp_path_layout` with values `"flat"` (the
current default) and `"per-rootdir"`. Under `"per-rootdir"`, numbered
directories are nested one level deeper under a stable token derived
from the pytest `rootpath` (`<slug>-<8-char-hash>`), so
`tmp_path_retention_count` applies per project instead of across every
rootdir that shares a user account.

Under `"flat"` layout, retention_count=3 with concurrent work in three
git worktrees means each worktree effectively gets one slot, and any
fourth run in one worktree evicts the oldest run of another. The
per-rootdir layout gives each rootdir its own numbered sequence, so
retention_count=3 means "3 most recent runs of this project" — which
is what almost every user assumes it already means.

Default is `"flat"` for one release for backwards compatibility. A
follow-up will flip the default after user feedback.

The slug is portable (`[A-Za-z0-9._-]`); the 8-char blake2b hash of
the absolute rootpath disambiguates homonyms like `~/work/foo` vs
`~/play/foo`.

Refs pytest-dev#14935.

Depends on pytest-dev#14936 (adds the `_rootpath` plumbing on `TempPathFactory`
this PR needs). Rebase order: land the origin sidecar PR first.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided (automation) changelog entry is part of PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant