[v1.0] Bound retention recovery work for large run caches

[codeforester]

Goal

Bound startup latency and filesystem work when recovering a large or previously unbounded run cache.

Background

prune_run_bundles() discovers every direct child, reads metadata, recursively sizes each eligible bundle, sorts all results, and removes excess bundles synchronously while holding the retention lock:

base-cli/lib/python/base_cli/_runtime.py

Lines 300 to 324 in 8a93d22

	try:
	with _retention_lock(runs_root):
	bundles = _discover_run_bundles(
	runs_root,
	protected=protected,
	max_age_seconds=effective.max_age_seconds,
	now=clock,
	)
	_apply_bundle_retention(
	runs_root,
	bundles,
	policy=effective,
	protected=protected,
	logger=log,
	now=clock,
	reserved_active_bundles=1 if current_run_root is not None else 0,
	)
	_write_run_index(
	runs_root,
	bundles,
	log,
	current_run_root=current_run_root,
	now=clock,
	)
	except (OSError, RuntimeError) as exc:

and

base-cli/lib/python/base_cli/_runtime.py

Lines 355 to 463 in 8a93d22

	def _discover_run_bundles(
	runs_root: Path,
	*,
	protected: set[Path],
	max_age_seconds: float | None,
	now: float,
	) -> list[dict[str, Any]]:
	bundles: list[dict[str, Any]] = []
	try:
	children = sorted(runs_root.iterdir(), key=lambda path: path.name)
	except OSError:
	return bundles
	for child in children:
	if child.name.startswith(".") or child.is_symlink() or not child.is_dir():
	continue
	metadata = _read_bundle_metadata(child)
	if metadata is None:
	# Partial startup directories are deliberately not considered
	# complete. They can be diagnosed or cleaned by a consumer's
	# explicit maintenance command without retention guessing.
	continue
	status = str(metadata.get("status", ""))
	started_at = _timestamp_to_epoch(metadata.get("started_at"))
	if started_at is None:
	try:
	started_at = child.stat().st_mtime
	except OSError:
	continue
	age = max(0.0, now - started_at)
	running = status == "running"
	stale_running = running and max_age_seconds is not None and age >= max_age_seconds
	if running and not stale_running:
	continue
	if status not in {"running", "ok", "aborted", "error"}:
	continue
	resolved = _safe_resolved_path(child)
	try:
	size = _bundle_size(child)
	except OSError:
	continue
	retention_metadata = metadata.get("retention")
	preserve = bool(metadata.get("preserve")) or (
	isinstance(retention_metadata, dict) and retention_metadata.get("preserve") is True
	)
	bundles.append(
	{
	"path": child,
	"resolved": resolved,
	"run_id": metadata.get("run_id"),
	"status": status,
	"started_at": started_at,
	"age": age,
	"size": size,
	"preserve": preserve,
	"protected": resolved in protected,
	}
	)
	bundles.sort(key=lambda bundle: (float(bundle["started_at"]), str(bundle["path"])))
	return bundles
	def _apply_bundle_retention(
	runs_root: Path,
	bundles: list[dict[str, Any]],
	*,
	policy: RetentionPolicy,
	protected: set[Path],
	logger: logging.Logger,
	now: float,
	reserved_active_bundles: int,
	) -> None:
	del now # retained for a stable extension point in policy implementations
	removable = [
	bundle
	for bundle in bundles
	if not bool(bundle["protected"])
	and not bool(bundle["preserve"])
	and _safe_resolved_path(bundle["path"]) not in protected
	]
	removable.sort(key=lambda bundle: (float(bundle["started_at"]), str(bundle["path"])))
	def remove(bundle: dict[str, Any]) -> bool:
	path = Path(bundle["path"])
	try:
	_remove_run_bundle(runs_root, path)
	except OSError as exc:
	logger.warning("Could not prune run bundle '%s': %s", path, exc)
	removable.remove(bundle)
	return False
	bundles.remove(bundle)
	removable.remove(bundle)
	return True
	if policy.max_age_seconds is not None:
	for bundle in list(removable):
	if float(bundle["age"]) >= policy.max_age_seconds:
	remove(bundle)
	if policy.max_bundles is not None:
	while len(bundles) + reserved_active_bundles > policy.max_bundles and removable:
	remove(removable[0])
	if policy.max_total_bytes is not None:
	total = sum(int(bundle["size"]) for bundle in bundles)
	while total > policy.max_total_bytes and removable:
	candidate = removable[0]
	if remove(candidate):
	total -= int(candidate["size"])

.

Defaults normally keep the directory small, but crash accumulation, a formerly disabled policy, migration from older releases, or same-user cache pollution can create a large recovery set. A local review probe with 2,000 minimal terminal bundles spent about 1.15 seconds in pruning before the command could run; work grows with bundle count and contents. The existing benchmark exercises an ordinary isolated invocation and does not cover recovery cardinality.

Scope

• Define a bounded-work recovery algorithm for large run roots.
• Decide how the run index can accelerate discovery without becoming a trusted source for unsafe deletion.
• Limit recursive sizing and deletion work per foreground invocation or move safe maintenance behind an explicit command.
• Expose actionable diagnostics when a cache remains above policy after a bounded pass.
• Preserve active/inherited/preserved and symlink/no-follow safety.

Acceptance Criteria

• Foreground startup has a documented work/latency bound for 20, 2,000, and 10,000 bundle fixtures.
• The algorithm does not recursively size every bundle on every invocation when no policy decision needs that data.
• Stale/corrupt/missing indexes fail safely and are incrementally reconciled.
• A bounded pass makes deterministic progress and reports remaining policy debt without corrupting state.
• Concurrent invocations remain serialized only for the minimum critical section.
• Benchmarks cover count, age, total bytes, deep trees, corrupt metadata, and slow/unreadable filesystems.
• [v1.0] Never prune a live run bundle based on age alone #266's live-run protection is preserved.

Validation

Add deterministic retention stress benchmarks and concurrency/security regressions; run them on Linux, macOS, and the portable Windows path.

Non-Goals

• Do not weaken configured retention safety silently.
• Do not trust index paths without revalidating ownership and no-follow boundaries.
• Do not put a background daemon in the core library.

Dependencies

• [v1.0] Benchmark lifecycle overhead against Click, Typer, and Cyclopts #251
• [v1.0] Never prune a live run bundle based on age alone #266

Project Fields

• Status: Backlog
• Priority: P2
• Area: Runtime
• Initiative: Adoption Polish
• Size: M

Ownership

• Assignee: @codeforester