Retention and cleanup are user-scoped, not project-scoped: cross-project eviction and lossy provenance under /tmp/pytest-of-<user>/

[jawauntb]

Retention and cleanup are user-scoped, not project-scoped: cross-project eviction and lossy provenance under /tmp/pytest-of-<user>/

Summary

TempPathFactory.getbasetemp() keys the scratch root by user (pytest-of-<user>/), then applies tmp_path_retention_count and lock-based cleanup globally inside that root. Every rootdir a user runs pytest against — every checkout, every worktree, every unrelated project — shares the same numbered sequence and the same retention window. Two consequences:

1. Cross-project eviction. A run in project B rotates project A's most recent runs out from under it. Users who work in more than one checkout at a time (git worktrees, multiple clones, monorepos + side projects) lose scratch they may still want, and the loss is invisible until they go looking for it.
2. Lossy provenance. Nothing in the session dir records which rootdir produced it. The .lock file identifies the pid, but no file names the project. External cleanup tooling (or a human trying to reclaim disk) can't tell whose scratch is whose without walking /proc or lsof. Third-party "safe cleanup" scripts end up conservative-and-slow, or they delete something they shouldn't.

A related bug in the same neighborhood: ensure_deletable (_pytest/pathlib.py) treats a .lock as dead purely by st_mtime age (LOCK_TIMEOUT = 3 days). The pid is recorded but never consulted, so a still-running session whose lock is 4 days old is fair game, and a dead session whose lock is 2 days old survives. Fixing this changes ownership recovery from "guess by clock" to "read the file."

Reproduction

# Any two projects will do — git worktrees are the sharpest case.
$ cd ~/proj-a && pytest -x                # creates /tmp/pytest-of-me/pytest-0
$ cd ~/proj-b && pytest -x                # creates /tmp/pytest-of-me/pytest-1
$ cd ~/proj-b && pytest -x                # /tmp/pytest-of-me/pytest-2
$ cd ~/proj-b && pytest -x                # /tmp/pytest-of-me/pytest-3, evicts pytest-0 (proj-a)
$ ls /tmp/pytest-of-me/                   # proj-a's most recent run is gone

grep -R proj-a /tmp/pytest-of-me/ returns nothing — no path in the tree records that pytest-0 came from ~/proj-a.

Impact

• Users with git worktrees regularly hit 1–2 GB spikes under /tmp/pytest-of-<user>/ because each worktree's suite gets copied 3× into a shared pool (default tmp_path_retention_count=3), and the pool grows with the number of concurrent projects, not with retention_count. See Cleaning up tmpdir's #1120 (2015) for the same complaint reported as "disk fills up with pytest tempfiles".
• Debug artifacts from the run they cared about get evicted by an unrelated run in another checkout.
• Anything trying to safely reclaim pytest-of-<user>/ disk from the outside — a CI cleanup step, a workstation janitor, an editor's temp-sweep — has to reconstruct ownership at delete time from process state. Getting this wrong deletes live scratch; being conservative leaves the disk full.

Proposal

Two independent, backwards-compatible changes. Each is useful on its own; together they close both the retention and the provenance problem.

(1) Scope retention per rootdir

Inside pytest-of-<user>/, add a subdirectory named for the rootdir — a stable, filesystem-safe token derived from config.rootpath. The numbered dirs and the retention window live under that subdirectory.

/tmp/pytest-of-me/
├── proj-a-8f3c1a2b/          # short hash of /Users/me/proj-a
│   ├── pytest-0
│   ├── pytest-1
│   └── pytest-current -> pytest-1
├── proj-b-1d9f4e77/          # short hash of /Users/me/proj-b
│   ├── pytest-0
│   ├── pytest-1
│   └── pytest-2

• Retention is now per rootdir: tmp_path_retention_count=3 means "3 most recent runs of this project", which is what almost every user thinks it already means.
• Cross-project eviction stops as a category. Removing a git worktree's checkout no longer risks evicting an unrelated project's scratch.
• Human-legible layout: the subdirectory name should include a human-readable prefix (last path component or config.rootpath.name) followed by a short hash of the absolute rootdir to disambiguate homonyms (~/work/foo and ~/play/foo). Example: foo-8f3c1a2b/.
• Rollout: guarded by a new ini option, off by default for one release, then defaulted on. The old flat layout is inspected on cleanup for one more release so existing pytest-N/ dirs still get reaped.

(2) Provenance sidecar

Alongside .lock, write a .origin file containing (one per line):

• absolute rootpath
• pytest version
• pid (redundant with .lock, useful when .lock has been reaped)
• hostname (helpful on shared filesystems)

$ cat /tmp/pytest-of-me/foo-8f3c1a2b/pytest-2/.origin
rootpath=/Users/me/proj-a
version=8.3.3
pid=48211
host=laptop.local

Cost: a few dozen bytes per session. Benefit: any external tool can attribute a session dir with a cat. This closes the "safest to infer ownership from lsof" gap that forces cleanup tooling to be overly conservative.

(3) Liveness check on the lock (separate, smaller fix)

In ensure_deletable, when a .lock is present, check whether its pid is alive (os.kill(pid, 0) on POSIX; OpenProcess on Windows) before falling back to the mtime timeout. LOCK_TIMEOUT becomes a bound on how long we wait when the pid is unknown/uncheckable, not the primary signal. Removes the "dead session held its scratch for 3 days" and "long suite got its scratch reaped mid-run" failure modes at the same time.

Backwards compatibility

• (1) is opt-in for one release via tmp_path_layout = "per-rootdir" | "flat" (default "flat"). One release later, switch the default. Both layouts remain readable by cleanup so no orphaned bytes.
• (2) is additive; nothing reads .origin yet, so writing it can't break anyone.
• (3) is a behavior change but a strict improvement for both failure modes. Could go in the same release as (1)'s default flip.
• No public API changes to TempPathFactory, tmp_path, tmp_path_factory, or --basetemp semantics. PYTEST_DEBUG_TEMPROOT behavior unchanged.

Prior art / related

• Cleaning up tmpdir's #1120 — "Cleaning up tmpdir's": same disk-fill symptom, addressed with retention policy in RotatingFileHandler with delay=True does not create log file with pytest #10577 but the root cause (shared pool across all rootdirs) was never touched.
• flag to cleanup generated tmp_path objects #7465 — "flag to cleanup generated tmp_path objects": user wanted per-run cleanup; landed as tmp_path_retention_policy. Orthogonal to this proposal; both help.
• Clarify whether and when tmp_path is deleted after each use #8036 — "Clarify when tmp_path is deleted": doc issue that surfaced the same retention confusion.
• Discussion How can I completely cleanup temporary directories from tmp_path? #10325 — maintainer note that "deferred cleanup is intentional." This proposal keeps deferred cleanup; it just scopes the deferral per project.
• FileNotFoundError for lock file with tmpdir for old pytest-of-user root #4135 — FileNotFoundError for lock file with tmpdir for old pytest-of-user root: another symptom of the shared-pool assumption.

Files touched (sketch)

• src/_pytest/tmpdir.py — TempPathFactory.getbasetemp(): add per-rootdir subdir, plumb config.rootpath into the factory.
• src/_pytest/tmpdir.py — write .origin next to .lock when a session dir is created.
• src/_pytest/pathlib.py — ensure_deletable: pid-liveness check ahead of mtime timeout.
• New ini tmp_path_layout; changelog fragment; docs (doc/en/how-to/tmp_path.rst, doc/en/reference/reference.rst).

What I'm offering

I'll send this as three separate PRs so each can be reviewed on its own merits:

1. .origin sidecar (lowest risk, useful immediately for external tooling).
2. Pid-liveness in ensure_deletable.
3. Per-rootdir layout behind tmp_path_layout, followed by a default flip.

Happy to iterate on the layout naming, the ini surface, and Windows liveness before writing code. Wanted to open the discussion first because the design choices above (in particular the layout token and the rollout shape) benefit from maintainer input.

[jawauntb]

Update: opened the three PRs proposed above.

• tmpdir: write .origin sidecar to record session ownership #14936 — tmpdir: write .origin sidecar to record session ownership. Adds a .origin file next to .lock recording rootpath, pytest version, PID, and hostname. Lowest risk of the three; useful immediately for external cleanup tooling. Independent of the other two.
• pathlib: use PID liveness as primary stale-lock signal #14937 — pathlib: use PID liveness as primary stale-lock signal. Reads the PID out of .lock and probes it (os.kill(pid, 0) on POSIX, OpenProcess on Windows) before falling back to the 3-day LOCK_TIMEOUT. Fixes both "dead session's scratch survives 3 days" and "live long-running session gets reaped". Independent of the other two.
• tmpdir: add tmp_path_layout ini for per-rootdir retention #14938 — tmpdir: add tmp_path_layout ini for per-rootdir retention. New tmp_path_layout = "flat" | "per-rootdir" ini, defaulting to "flat" for one release. Under "per-rootdir" numbered dirs nest under <slug>-<hash> derived from rootpath, so retention is scoped per project. Depends on tmpdir: write .origin sidecar to record session ownership #14936 for the _rootpath plumbing; will rebase off main after tmpdir: write .origin sidecar to record session ownership #14936 lands.

All three ship with tests and changelog fragments (14935.improvement.1.rst, 14935.improvement.2.rst, 14935.feature.rst). Full sweep of test_tmpdir.py + test_pytester.py + test_config.py + test_pathlib.py passes locally on each branch.

Happy to split, reorder, or reshape any of them based on maintainer feedback.