ENH: NFS mount/export option pass-through + two stat-stability targets

[yarikoptic-gitmate]

Proposal, from chasing con/git-annex#293: git-annex fails intermittently on NFS with failed to link to annex / unlock failed, because it records (inode, size, high-resolution mtime) for a file, copies it, stats it again and compares the two exactly — and an NFS client can report two different mtimes for a file nothing wrote to. In con/git-annex CI the mtime of an unmodified file moved by 12 ms, 279 ms and 41 s across a copy, and mounting with actimeo=0 turned 3/3 failing runs into 3/3 passing ones.

Three things that investigation needed and this framework could not express. All small, and useful beyond that one bug.

1. --mount-opts / --export-opts on the NFS backend

NFS_OPTS was hardcoded to rw,async / rw,sync, so the client-side knobs that decide this class of question — actimeo=0, noac, lookupcache=none, nocto, vers=, rsize=/wsize= — were unreachable without editing the script. Now:

sudo bin/eval-under nfs --set-home --mount-opts actimeo=0 -- \
  bash -c 'cd "$HOME" && git annex test'

Mount and export options stay in separate variables (EVAL_UNDER_NFS_MOUNT_OPTS / EVAL_UNDER_NFS_EXPORT_OPTS), since an option valid in one namespace is rejected by the other — same reasoning as the existing sync/no_root_squash split. This is the highest-value part of the PR: it makes "the filesystem lied" vs "the code is wrong" a one-flag experiment.

2. New target: mtime-stability

Write a file, stat it, copy it the way git-annex copies (cp --reflink=auto -a --no-preserve=xattr), stat it again, compare exactly, report a rate. No git-annex involved, so a red cell says the filesystem, not the application. -j for parallel load (the race is a timing one), --delay to age the attribute cache past acregmin/acregmax, --no-copy as a control separating "the copy revalidates" from "the cache expired on its own". Output is in git-annex's own inode size secs nsecs format so it lines up with what git-annex prints.

This is the property Annex/Content.hs:linkAnnex and prepSendAnnex assume, so the cell answers the question for BeeGFS and vfat too — relevant to con/git-annex#288.

3. New target: git-annex-linkannex

The same question at the git-annex level: loop git annex unlock (linkFromAnnex') and unlocked git annex add (linkToAnnex), report a failure rate in minutes instead of a pass/fail of the ~20-minute suite. Useful for verifying a fix, and for watching the rate move as mount options change.

Both targets emit TAP and are scored per row by bin/ci/collect-results.py → known_issues.py, like stress-ng: mtime-stability reports one point per stat field (inode-stable, size-stable, mtime-stable, with the rate and worst mtime delta in the description), git-annex-linkannex one per direction (unlock, add-unlocked). Per-field/per-mode ids rather than per-round ones, so a known issue can name "this filesystem moves the mtime" instead of "round 417".

What CI found

All 30 cells green on run 36790703938 (f358985), in 18 minutes. The git-annex-linkannex measurements, since they are the point of the PR:

cell	rate	wall
NFS (localhost)	0/200 unlock, 0/200 add-unlocked	105 s
BeeGFS 7.4.6	0/200, 0/200	196 s
BeeGFS 8.1.0	0/200, 0/200	220 s (83 s + 64 s in the loop)
Loop ext4	0/200, 0/200	39 s
Loop vfat	0/200, 0/200	28 s

mtime-stability passes on all five backends: no filesystem here moves an unmodified file's (inode, size, mtime) across a copy.

The NFS cells being clean is itself the finding. CI installs the current daily build (10.20260901+git74-g3cdda5ef2a) — the same git-annex whose nfs-home cell fails in con/git-annex — so this is the exact failing operations on the exact failing version, with zero failures. The variable is the kernel: this workflow pins ubuntu-22.04 (kernel 6.8) because BeeGFS's DKMS modules cannot build against 6.17, while con/git-annex's test-annex moved to ubuntu-24.04, whose image now ships 6.17. See con/git-annex#294, where 22.04 passed 4/4 and 24.04 failed 4/4 on the same commit.

So this framework cannot currently reproduce that bug, by construction. A 24.04 row would have to skip the BeeGFS backends; I have not added one, as that is a matrix-shape decision rather than part of this PR.

Three things CI taught us about the harness along the way, each fixed here:

• annex.diskreserve vs small loop images. On a fresh ext4 image with 83 MB free, git annex unlock of a 14-byte file refuses with "not enough free space, need 13.73 MB more". Every round failed before linkAnnex was reached. The loop now sets annex.diskreserve=0 in its probe repos — it measures the inode-cache comparison, not git-annex's disk-space policy. This is the same root cause as the existing loop-annex-diskreserve known issue, which covers the git-annex target; that entry is untouched and still applies there.
• A full disk looked exactly like the bug. ENOSPC makes git-annex print the same failed to link to annex / unlock failed lines as an inode-cache mismatch, and the loop was counting them — reporting 800/800 "failures" on ext4, the negative control. It now recognises those messages, refuses to report a rate, and exits 4; the target turns that into an incomplete cell rather than a verdict the run cannot support.
• Sizing. At 200×4 rounds, BeeGFS spent 967 s on one of two modes against a 1200 s budget, so both BeeGFS cells timed out with only mode 1 reported. Rounds per worker are now 50 (200 per mode). The trade-off is detection power — 200 rounds per mode will not reliably show a rate below ~1% — so bin/ci/linkannex-loop.sh -n is the knob for hunting something rare, not this default.

Notes for review

• Both targets run on every backend, so the README grid goes 20 → 30 cells (gen-readme-matrix.sh re-run). If that is more than you want, dropping the git-annex-linkannex entry from evals/matrix.yaml is a one-line change — the script stays usable by hand.
• matrix-json.sh now carries needs-git-annex per cell and the workflow gates the daily-build fetch on it, instead of on matrix.target == 'git-annex'. Without that, a second git-annex-using target runs with no git-annex installed — which is exactly what happened. target_needs_git_annex() in matrix.sh had been dead code; it now has a caller, and tests/test_matrix_json.py holds both halves of the contract.
• mtime-stability needs nothing installed; git-annex-linkannex reuses install_git_annex.
• No known-issues.yaml entries added: every cell passes, so there is nothing to record.
• Observed but not addressed here: at the old round count the BeeGFS cluster degraded under the load (Receive failed from node_meta_1, then metadata and storage nodes to offline, with teardown wedged in FhgfsOps_flush). That also inflated the 967 s above — at 200 rounds per mode the same cell does both modes in 147 s, far better than linear scaling would predict, and the cluster stays healthy. Worth knowing if any future target leans harder on BeeGFS metadata, but it needs no change here.
• Not included, but proposed: a --tmpdir-local flag so TMPDIR can stay off the mount. con/git-annex's nfs-home flavor puts only HOME on NFS, and that difference is part of why the first local reproduction attempt diverged from CI. It belongs in all three backends rather than just this one, so it seemed better as its own PR.

Testing

• bin/ci/run-checks.sh fully green: shellcheck, pyflakes, bats (30 tests, two new for the NFS flags), known-issues, 33 unit tests (five new for the matrix contract).
• Both targets run end-to-end into a cell dir and collect with # complete: yes.
• The ENOSPC abort path exercised on a deliberately pre-filled image: real ENOSPC, exit 4, rate withheld, cell incomplete rather than failing.
• mtime-stability's failure path exercised with a stubbed cp that bumps the source mtime, which reproduces the NFS signature exactly (inode and size stable, mtime-stable not ok).
• The cleanup trap verified as an unprivileged user against real annex objects (dr-xr-xr-x): plain rm -rf leaves them behind and floods the log, chmod -R u+w first leaves nothing and says nothing. On the NFS cell that cut the uploaded log bundle from 78,288 to 1,573 bytes.
• vfat could not be exercised locally (no mkfs.vfat, and this container's apt is blocked), so its results come from CI only.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B89nUooZLfThcTA4fSMPGf

---

Generated by Claude Code

[yarikoptic-gitmate]

Rebased onto current master (60 commits, up to e8b1291 / #16). One textual conflict, in the matrix file — git followed its move to evals/matrix.yaml, and the collision was only that master reworded the pjdfstest comment where my two entries were inserted above it; master's wording kept, my entries kept. Everything else auto-merged, and the README grid regenerated to the same 30 cells.

The rebase also surfaced something a textual merge cannot: #14 changed how cells are judged, from the suite's exit code to per-test rows via bin/ci/collect-results.py → known_issues.py. That collector dispatches on target name and raises no results adapter for target … for anything it does not know, so as originally written both new targets would have produced incomplete cells. Fixed in the second commit — both now print TAP and are scored like stress-ng:

target	TAP points (row ids)
mtime-stability	inode-stable, size-stable, mtime-stable — rate and worst mtime delta in the description, first few mismatches as TAP comments
git-annex-linkannex	unlock, add-unlocked — one per direction

Per-field and per-mode ids rather than per-round ones, so a known issue can name "this filesystem moves the mtime" instead of "round 417". linkannex-loop.sh gained --report FILE so the wrapper reads a count instead of parsing prose, and a loop that dies before reporting becomes a failing point rather than a silent pass. collect_tap_target() serves both and takes the row id from the description's first word, as collect_stress_ng does.

No known-issues.yaml entries added on purpose: whether NFS fails mtime-stable on this CI's ubuntu-22.04 runners is genuinely open. The failure that started all of this only reproduces on kernel 6.17 (ubuntu-24.04); on 6.8 — what ubuntu-22.04 runs today, and what this workflow uses — the same git-annex build passes, 4/4 vs 4/4 in a one-variable run (con/git-annex#294). So this PR's own first run is the interesting data point, and pre-declaring a verdict would only hide it. Worth considering a 24.04 row for the NFS backend separately — on 22.04 alone this grid cannot see that class of bug at all.

Verification, all local:

• bin/ci/run-checks.sh fully green — shellcheck (29 scripts), pyflakes, bats (30 tests, two new ones covering --mount-opts/--export-opts), known-issues valid, 28 unit tests.
• Both targets run end-to-end into a cell dir and collect cleanly on ext4: inode-stable/size-stable/mtime-stable and unlock/add-unlocked, # complete: yes.
• The TAP failure path was exercised with a stubbed cp that bumps the source mtime — it yields exactly the NFS signature, not ok mtime-stable with inode and size stable.

---

Generated by Claude Code