From 93daf8e1b46732e0b1a8e216323adebeaa79c748 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Thu, 24 Sep 2026 20:18:32 +0000 Subject: [PATCH 1/6] ENH: NFS mount/export option pass-through + two stat-stability targets Chasing con/git-annex#293 (git-annex intermittently failing on NFS with "failed to link to annex" / "unlock failed") needed three things this framework could not express. All three are small: * `eval-under nfs --mount-opts OPTS` / `--export-opts OPTS`. NFS_OPTS was hardcoded to rw,async|rw,sync, so the client-side knobs that matter for this class of bug -- actimeo=0, noac, lookupcache=none, nocto, vers= -- were unreachable. Mount and export options stay in separate variables because an option valid in one is rejected by the other. * target `mtime-stability`: write, stat, copy the way git-annex copies, stat again, compare (inode, size, high-res mtime) exactly, report a rate. No git-annex involved, so a red cell says "the filesystem", not "the application". This is the property git-annex assumes in Annex/Content.hs:linkAnnex and prepSendAnnex, and that NFS attribute caching breaks: in con/git-annex CI the mtime of an unmodified file moved by 12 ms, 279 ms and 41 s across a copy, and mounting with actimeo=0 made 3/3 failing runs pass. * target `git-annex-linkannex`: the same question at the git-annex level -- loop `git annex unlock` (linkFromAnnex') and unlocked `git annex add` (linkToAnnex), report a failure rate in minutes rather than a pass/fail of the ~20 minute suite. Both targets run on every backend, so BeeGFS and vfat get answered for free; that grows the README grid from 20 cells to 30. Tested: shellcheck clean; both targets run end-to-end on ext4 (0%, the negative control); option strings verified for each flag combination. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01B89nUooZLfThcTA4fSMPGf --- README.md | 20 ++- bin/ci/install-target.sh | 4 +- bin/ci/linkannex-loop.sh | 136 ++++++++++++++++++ bin/ci/mtime-stability.py | 208 +++++++++++++++++++++++++++ bin/ci/target-git-annex-linkannex.sh | 61 ++++++++ bin/ci/target-mtime-stability.sh | 48 +++++++ bin/eval-under-nfs | 34 +++++ evals/matrix.yaml | 21 +++ 8 files changed, 524 insertions(+), 8 deletions(-) create mode 100755 bin/ci/linkannex-loop.sh create mode 100755 bin/ci/mtime-stability.py create mode 100755 bin/ci/target-git-annex-linkannex.sh create mode 100755 bin/ci/target-mtime-stability.sh diff --git a/README.md b/README.md index 23f7932..aa51479 100644 --- a/README.md +++ b/README.md @@ -23,13 +23,13 @@ itself is both backend- and suite-agnostic: new filesystems drop in as ## CI status -| Backend | git-annex test | git testsuite | stress-ng | pjdfstest | -| --- | --- | --- | --- | --- | -| BeeGFS 7.4.6 | [![BeeGFS 7.4.6 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-git-annex.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-git-annex) | [![BeeGFS 7.4.6 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-git.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-git) | [![BeeGFS 7.4.6 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-stress-ng.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-stress-ng) | [![BeeGFS 7.4.6 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-pjdfstest.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-pjdfstest) | -| BeeGFS 8.1.0 | [![BeeGFS 8.1.0 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-git-annex.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-git-annex) | [![BeeGFS 8.1.0 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-git.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-git) | [![BeeGFS 8.1.0 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-stress-ng.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-stress-ng) | [![BeeGFS 8.1.0 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-pjdfstest.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-pjdfstest) | -| NFS (localhost) | [![NFS (localhost) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git-annex.svg)](https://con.github.io/eval-under/#nfs-git-annex) | [![NFS (localhost) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git.svg)](https://con.github.io/eval-under/#nfs-git) | [![NFS (localhost) / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-stress-ng.svg)](https://con.github.io/eval-under/#nfs-stress-ng) | [![NFS (localhost) / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-pjdfstest.svg)](https://con.github.io/eval-under/#nfs-pjdfstest) | -| Loop vfat | [![Loop vfat / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git-annex.svg)](https://con.github.io/eval-under/#loop-vfat-git-annex) | [![Loop vfat / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git.svg)](https://con.github.io/eval-under/#loop-vfat-git) | [![Loop vfat / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-stress-ng.svg)](https://con.github.io/eval-under/#loop-vfat-stress-ng) | [![Loop vfat / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-pjdfstest.svg)](https://con.github.io/eval-under/#loop-vfat-pjdfstest) | -| Loop ext4 | [![Loop ext4 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git-annex.svg)](https://con.github.io/eval-under/#loop-ext4-git-annex) | [![Loop ext4 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git.svg)](https://con.github.io/eval-under/#loop-ext4-git) | [![Loop ext4 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-stress-ng.svg)](https://con.github.io/eval-under/#loop-ext4-stress-ng) | [![Loop ext4 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-pjdfstest.svg)](https://con.github.io/eval-under/#loop-ext4-pjdfstest) | +| Backend | git-annex test | git testsuite | stress-ng | mtime stability | git-annex linkAnnex loop | pjdfstest | +| --- | --- | --- | --- | --- | --- | --- | +| BeeGFS 7.4.6 | [![BeeGFS 7.4.6 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-git-annex.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-git-annex) | [![BeeGFS 7.4.6 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-git.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-git) | [![BeeGFS 7.4.6 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-stress-ng.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-stress-ng) | [![BeeGFS 7.4.6 / mtime stability](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-mtime-stability.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-mtime-stability) | [![BeeGFS 7.4.6 / git-annex linkAnnex loop](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-git-annex-linkannex.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-git-annex-linkannex) | [![BeeGFS 7.4.6 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-pjdfstest.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-pjdfstest) | +| BeeGFS 8.1.0 | [![BeeGFS 8.1.0 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-git-annex.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-git-annex) | [![BeeGFS 8.1.0 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-git.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-git) | [![BeeGFS 8.1.0 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-stress-ng.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-stress-ng) | [![BeeGFS 8.1.0 / mtime stability](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-mtime-stability.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-mtime-stability) | [![BeeGFS 8.1.0 / git-annex linkAnnex loop](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-git-annex-linkannex.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-git-annex-linkannex) | [![BeeGFS 8.1.0 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-pjdfstest.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-pjdfstest) | +| NFS (localhost) | [![NFS (localhost) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git-annex.svg)](https://con.github.io/eval-under/#nfs-git-annex) | [![NFS (localhost) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git.svg)](https://con.github.io/eval-under/#nfs-git) | [![NFS (localhost) / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-stress-ng.svg)](https://con.github.io/eval-under/#nfs-stress-ng) | [![NFS (localhost) / mtime stability](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-mtime-stability.svg)](https://con.github.io/eval-under/#nfs-mtime-stability) | [![NFS (localhost) / git-annex linkAnnex loop](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git-annex-linkannex.svg)](https://con.github.io/eval-under/#nfs-git-annex-linkannex) | [![NFS (localhost) / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-pjdfstest.svg)](https://con.github.io/eval-under/#nfs-pjdfstest) | +| Loop vfat | [![Loop vfat / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git-annex.svg)](https://con.github.io/eval-under/#loop-vfat-git-annex) | [![Loop vfat / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git.svg)](https://con.github.io/eval-under/#loop-vfat-git) | [![Loop vfat / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-stress-ng.svg)](https://con.github.io/eval-under/#loop-vfat-stress-ng) | [![Loop vfat / mtime stability](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-mtime-stability.svg)](https://con.github.io/eval-under/#loop-vfat-mtime-stability) | [![Loop vfat / git-annex linkAnnex loop](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git-annex-linkannex.svg)](https://con.github.io/eval-under/#loop-vfat-git-annex-linkannex) | [![Loop vfat / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-pjdfstest.svg)](https://con.github.io/eval-under/#loop-vfat-pjdfstest) | +| Loop ext4 | [![Loop ext4 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git-annex.svg)](https://con.github.io/eval-under/#loop-ext4-git-annex) | [![Loop ext4 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git.svg)](https://con.github.io/eval-under/#loop-ext4-git) | [![Loop ext4 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-stress-ng.svg)](https://con.github.io/eval-under/#loop-ext4-stress-ng) | [![Loop ext4 / mtime stability](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-mtime-stability.svg)](https://con.github.io/eval-under/#loop-ext4-mtime-stability) | [![Loop ext4 / git-annex linkAnnex loop](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git-annex-linkannex.svg)](https://con.github.io/eval-under/#loop-ext4-git-annex-linkannex) | [![Loop ext4 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-pjdfstest.svg)](https://con.github.io/eval-under/#loop-ext4-pjdfstest) | Rows are **backends** (which filesystem the work happens on), columns @@ -161,6 +161,12 @@ sudo bin/eval-under loop --fs xfs --size 200 --set-home -- \ # the fsync-heavy slow path) sudo bin/eval-under nfs --set-home -- bash -c 'cd "$HOME" && git annex test' +# Same, with client-side attribute caching off -- the knob that decides +# whether git-annex's "failed to link to annex" / "unlock failed" on NFS +# is the filesystem or the code (con/git-annex#293) +sudo bin/eval-under nfs --set-home --mount-opts actimeo=0 -- \ + bash -c 'cd "$HOME" && git annex test' + # Skip teardown to poke around after a failure sudo bin/eval-under beegfs --set-home --keep -- some-failing-command diff --git a/bin/ci/install-target.sh b/bin/ci/install-target.sh index 508eebc..2da0998 100755 --- a/bin/ci/install-target.sh +++ b/bin/ci/install-target.sh @@ -137,7 +137,9 @@ install_pjdfstest() { } case "$TARGET" in - git-annex) install_git_annex ;; + git-annex|git-annex-linkannex) install_git_annex ;; + # mtime-stability ships with this repo and needs nothing installed + mtime-stability) ;; git) install_git ;; stress-ng) install_stress_ng ;; pjdfstest) install_pjdfstest ;; diff --git a/bin/ci/linkannex-loop.sh b/bin/ci/linkannex-loop.sh new file mode 100755 index 0000000..4a7ba37 --- /dev/null +++ b/bin/ci/linkannex-loop.sh @@ -0,0 +1,136 @@ +#!/bin/bash +# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko +# SPDX-License-Identifier: MIT +# +# git-annex-level reproducer for the NFS LinkAnnexFailed flake +# (con/git-annex#293): loops the two operations that go through +# Annex/Content.hs:linkAnnex and counts how often they fail. +# +# unlock (default) git annex add + git annex unlock +# -> linkFromAnnex', the "unlock failed" case +# add-unlocked git -c annex.addunlocked=true annex add +# -> linkToAnnex, the "failed to link to annex" case +# +# Run it with cwd on the filesystem under test; it creates its own repos. +# Minutes rather than the ~20 that a full `git annex test` takes, and it +# reports a rate instead of a single pass/fail. +# +# usage: +# ./nfs-annex-linkannex-loop.sh [-n ROUNDS] [-j WORKERS] [-m MODE] [-d DIR] +# +# exits non-zero if any round failed. + +set -u -o pipefail + +ROUNDS=200 +WORKERS=1 +MODE=unlock +DIR=. + +usage() { sed -n '3,20p' "$0"; } + +while [ $# -gt 0 ]; do + case "$1" in + -n|--rounds) ROUNDS="$2"; shift 2 ;; + -j|--workers) WORKERS="$2"; shift 2 ;; + -m|--mode) MODE="$2"; shift 2 ;; + -d|--dir) DIR="$2"; shift 2 ;; + -h|--help) usage; exit 0 ;; + *) echo "unknown arg: $1" >&2; usage >&2; exit 2 ;; + esac +done + +case "$MODE" in + unlock|add-unlocked) ;; + *) echo "unknown mode: $MODE (expected unlock or add-unlocked)" >&2; exit 2 ;; +esac + +command -v git-annex >/dev/null 2>&1 || command -v git >/dev/null 2>&1 || { + echo "git-annex not found in PATH" >&2; exit 3; } + +mkdir -p "$DIR" +root="$(cd "$DIR" && pwd)" +work="$(mktemp -d "$root/linkannex-loop-XXXXXX")" +trap 'rm -rf "$work"' EXIT + +echo "# dir: $root" +echo "# mount: $(findmnt -no FSTYPE,OPTIONS --target "$root" 2>/dev/null || echo '(findmnt unavailable)')" +echo "# version: $(git annex version --raw 2>/dev/null || echo unknown)" +echo "# plan: $WORKERS worker(s) x $ROUNDS rounds, mode=$MODE" + +# Each worker gets its own repo, the way `git annex test` runs its parts. +run_worker() { + # Note: `local a=$1 b=$a` would expand $a before the assignment happens, + # which trips `set -u`; keep the dependent ones on their own lines. + local wid="$1" + local repo="$work/w$wid" + local failures=0 i out + mkdir -p "$repo" + ( + cd "$repo" || exit 3 + git init -q . 2>/dev/null + git config user.email test@example.com + git config user.name "NFS Probe" + git annex init -q "probe-$wid" >/dev/null 2>&1 + ) || { echo "worker $wid: repo setup failed" >&2; return 3; } + + cd "$repo" || return 3 + for ((i = 0; i < ROUNDS; i++)); do + printf 'content %s %s\n' "$wid" "$i" > "f$i" + if [ "$MODE" = add-unlocked ]; then + out="$(git -c annex.addunlocked=true annex add "f$i" 2>&1)" || { + failures=$((failures + 1)) + printf 'worker %s round %s: add failed\n%s\n' "$wid" "$i" "$out" >&2 + continue + } + # the To-direction failure is a warning + non-zero exit; also catch the + # message in case a future version only warns + case "$out" in *"failed to link to annex"*) + failures=$((failures + 1)) + printf 'worker %s round %s:\n%s\n' "$wid" "$i" "$out" >&2 ;; + esac + else + git annex add -q "f$i" >/dev/null 2>&1 || { + failures=$((failures + 1)) + printf 'worker %s round %s: add failed\n' "$wid" "$i" >&2 + continue + } + out="$(git annex unlock "f$i" 2>&1)" || { + failures=$((failures + 1)) + printf 'worker %s round %s:\n%s\n' "$wid" "$i" "$out" >&2 + continue + } + case "$out" in *"unlock failed"*) + failures=$((failures + 1)) + printf 'worker %s round %s:\n%s\n' "$wid" "$i" "$out" >&2 ;; + esac + fi + git commit -qm "round $i" >/dev/null 2>&1 || : + done + echo "$failures" > "$work/failures.$wid" +} + +start=$(date +%s) +for ((w = 0; w < WORKERS; w++)); do + run_worker "$w" & +done +wait +end=$(date +%s) + +total_failures=0 +for ((w = 0; w < WORKERS; w++)); do + if [ ! -e "$work/failures.$w" ]; then + echo "worker $w did not finish; its output above says why" >&2 + exit 3 + fi + f="$(cat "$work/failures.$w")" + total_failures=$((total_failures + f)) +done +total=$((ROUNDS * WORKERS)) + +printf '\n%s/%s rounds failed in linkAnnex (%s%%), in %ss\n' \ + "$total_failures" "$total" \ + "$(awk -v a="$total_failures" -v b="$total" 'BEGIN{printf "%.2f", b ? 100*a/b : 0}')" \ + "$((end - start))" + +[ "$total_failures" -eq 0 ] || exit 1 diff --git a/bin/ci/mtime-stability.py b/bin/ci/mtime-stability.py new file mode 100755 index 0000000..54a5089 --- /dev/null +++ b/bin/ci/mtime-stability.py @@ -0,0 +1,208 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko +# SPDX-License-Identifier: MIT +# +# Minimal reproducer for the git-annex "failed to link to annex" / +# "unlock failed" flake seen on NFS (con/git-annex#293). +# +# git-annex's Annex/Content.hs:linkAnnex stats the source file, copies it, +# then stats it again and compares the two InodeCaches with compareStrong, +# i.e. exact equality of (inode, size, high-resolution mtime). If they +# differ it assumes the file changed under it, deletes the destination and +# fails. On NFS the two stats of an *unmodified* file can disagree in the +# mtime, which is what this script measures -- no git-annex needed. +# +# Run it on the filesystem under test: +# +# ./nfs-mtime-stability.py -n 200 # in cwd +# ./nfs-mtime-stability.py -n 200 -j 8 # under parallel load +# ./nfs-mtime-stability.py -n 50 --delay 5 # let the attribute cache age +# +# Exits non-zero if any mismatch was observed, so it can be used as a check. + +from __future__ import annotations + +import argparse +import concurrent.futures +import os +import shutil +import subprocess +import sys +import tempfile +import time +from dataclasses import dataclass + + +@dataclass(frozen=True) +class InodeCache: + """What git-annex stores and compares (Utility/InodeCache.hs).""" + + inode: int + size: int + mtime_ns: int + + @classmethod + def of(cls, path: str) -> "InodeCache": + # git-annex uses lstat (getSymbolicLinkStatus) + modificationTimeHiRes + st = os.lstat(path) + return cls(st.st_ino, st.st_size, st.st_mtime_ns) + + def show(self) -> str: + # same field order git-annex's showInodeCache prints: + # " " + return f"{self.inode} {self.size} {self.mtime_ns // 10**9} {self.mtime_ns % 10**9}" + + +def compare_strong(a: InodeCache, b: InodeCache) -> bool: + return a == b + + +def differing_fields(a: InodeCache, b: InodeCache) -> list[str]: + out = [] + if a.inode != b.inode: + out.append("inode") + if a.size != b.size: + out.append("size") + if a.mtime_ns != b.mtime_ns: + out.append("mtime") + return out + + +def copy_like_git_annex(src: str, dest: str) -> None: + """Utility/CopyFile.hs:copyFileExternal CopyAllMetaData.""" + subprocess.run( + ["cp", "--reflink=auto", "-a", "--no-preserve=xattr", src, dest], + check=True, + capture_output=True, + ) + + +@dataclass +class Mismatch: + phase: str + path: str + before: InodeCache + after: InodeCache + + def describe(self) -> str: + fields = "+".join(differing_fields(self.before, self.after)) + delta_ns = self.after.mtime_ns - self.before.mtime_ns + return ( + f" [{self.phase}] {self.path}: {fields} changed; " + f"before: {self.before.show()}; after: {self.after.show()}" + f" (mtime delta {delta_ns / 1e9:+.9f}s)" + ) + + +def one_round(workdir: str, index: int, size: int, delay: float, do_copy: bool) -> list[Mismatch]: + """Write a file, stat it, copy it (as git-annex does), stat it again.""" + src = os.path.join(workdir, f"src-{index}") + dest = os.path.join(workdir, f"dest-{index}") + with open(src, "wb") as fh: + fh.write(os.urandom(size)) + # NFS flushes on close; git-annex's callers see the file after close too. + + if delay: + time.sleep(delay) + + before = InodeCache.of(src) + + if do_copy: + copy_like_git_annex(src, dest) + else: + # control: same wait, no copy, to tell "the copy revalidates" from + # "the attribute cache expires on its own" + time.sleep(0.05) + + after = InodeCache.of(src) + + found = [] + if not compare_strong(before, after): + found.append( + Mismatch("copy" if do_copy else "no-copy", os.path.relpath(src, workdir), before, after) + ) + + for f in (src, dest): + try: + os.unlink(f) + except FileNotFoundError: + pass + return found + + +def main() -> int: + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("-d", "--dir", default=".", help="directory on the filesystem under test (default: cwd)") + p.add_argument("-n", "--rounds", type=int, default=200, help="rounds per worker (default: 200)") + p.add_argument("-j", "--jobs", type=int, default=1, help="concurrent workers (default: 1)") + p.add_argument("-s", "--size", type=int, default=16, help="file size in bytes (default: 16)") + p.add_argument( + "--delay", + type=float, + default=0.0, + help="seconds to wait between writing and the first stat; >3 exceeds the " + "default acregmin, >60 the default acregmax", + ) + p.add_argument("--no-copy", action="store_true", help="control run: stat twice without copying") + p.add_argument("--quiet", action="store_true", help="only print the summary") + args = p.parse_args() + + root = os.path.abspath(args.dir) + os.makedirs(root, exist_ok=True) + workroot = tempfile.mkdtemp(prefix="mtime-stability-", dir=root) + + fsinfo = subprocess.run( + ["findmnt", "-no", "FSTYPE,OPTIONS", "--target", root], + capture_output=True, + text=True, + ).stdout.strip() + print(f"# directory: {root}") + print(f"# mount: {fsinfo or '(findmnt unavailable)'}") + print( + f"# plan: {args.jobs} worker(s) x {args.rounds} rounds, {args.size}B files, " + f"delay={args.delay}s, {'no copy (control)' if args.no_copy else 'copy via cp'}" + ) + + started = time.time() + mismatches: list[Mismatch] = [] + try: + def worker(wid: int) -> list[Mismatch]: + wdir = os.path.join(workroot, f"w{wid}") + os.makedirs(wdir, exist_ok=True) + found = [] + for i in range(args.rounds): + found.extend(one_round(wdir, i, args.size, args.delay, not args.no_copy)) + return found + + if args.jobs > 1: + with concurrent.futures.ThreadPoolExecutor(max_workers=args.jobs) as pool: + for res in pool.map(worker, range(args.jobs)): + mismatches.extend(res) + else: + mismatches.extend(worker(0)) + finally: + shutil.rmtree(workroot, ignore_errors=True) + + total = args.jobs * args.rounds + elapsed = time.time() - started + if mismatches and not args.quiet: + print(f"\n# mismatches ({len(mismatches)}):") + for m in mismatches: + print(m.describe()) + + rate = 100.0 * len(mismatches) / total if total else 0.0 + print( + f"\n{len(mismatches)}/{total} rounds saw the inode cache change under an " + f"unmodified file ({rate:.2f}%), in {elapsed:.1f}s" + ) + if mismatches: + worst = max(abs(m.after.mtime_ns - m.before.mtime_ns) for m in mismatches) + print(f"largest mtime jump: {worst / 1e9:.9f}s") + print("This is what makes git-annex report 'failed to link to annex' / 'unlock failed'.") + return 1 + print("No mismatch seen: compareStrong would have held for every round.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/bin/ci/target-git-annex-linkannex.sh b/bin/ci/target-git-annex-linkannex.sh new file mode 100755 index 0000000..a3cb0d7 --- /dev/null +++ b/bin/ci/target-git-annex-linkannex.sh @@ -0,0 +1,61 @@ +#!/bin/bash +# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko +# SPDX-License-Identifier: MIT +# +# eval-under *target*: the git-annex operations that go through +# Annex/Content.hs:linkAnnex, looped, reported as a failure rate. +# +# The full `git annex test` suite answers "did anything break?" in +# ~20 minutes and hides how often. This target loops just the two +# commands that hit the inode-cache comparison -- `git annex unlock` +# (linkFromAnnex') and an unlocked `git annex add` (linkToAnnex) -- +# so a filesystem that trips it shows up as a percentage in minutes. +# See con/git-annex#293 and bin/ci/target-mtime-stability.sh, which +# measures the same property without git-annex. +# +# Runs INSIDE the eval-under wrapper, i.e. with TMPDIR / HOME already +# pointing at the filesystem under test. Do not invoke directly for CI +# purposes -- go through bin/ci/run-under.sh \ +# git-annex-linkannex. +# +# usage: +# bin/ci/target-git-annex-linkannex.sh +# +# env (set by eval-under, honoured here): +# HOME /home -- the repos are created here, so they live +# on the filesystem under test +# TMPDIR +# +# env (optional): +# EVAL_UNDER_LINKANNEX_ROUNDS rounds per worker (default 200) +# EVAL_UNDER_LINKANNEX_WORKERS concurrent repos (default 4) + +set -euo pipefail + +here="$(cd "$(dirname "$0")" && pwd)" + +ROUNDS="${EVAL_UNDER_LINKANNEX_ROUNDS:-200}" +WORKERS="${EVAL_UNDER_LINKANNEX_WORKERS:-4}" + +cd "$HOME" + +# Same reason as target-git-annex.sh: --set-home repoints HOME at the +# mount, so the runner's ~/.gitconfig is out of scope and `git commit` +# inside the loop would fail without an identity. +git config --global --get user.email >/dev/null 2>&1 \ + || git config --global user.email test@github.land +git config --global --get user.name >/dev/null 2>&1 \ + || git config --global user.name "GitHub Almighty" + +git annex version | head -1 + +rc=0 +# Both directions, reported separately: `unlock` exercises linkAnnex +# From (annex object -> worktree), `add-unlocked` exercises To. +for mode in unlock add-unlocked; do + echo + echo "=== mode: $mode" + "$here/linkannex-loop.sh" --dir "$HOME" --mode "$mode" \ + --rounds "$ROUNDS" --workers "$WORKERS" || rc=1 +done +exit "$rc" diff --git a/bin/ci/target-mtime-stability.sh b/bin/ci/target-mtime-stability.sh new file mode 100755 index 0000000..2346bc2 --- /dev/null +++ b/bin/ci/target-mtime-stability.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko +# SPDX-License-Identifier: MIT +# +# eval-under *target*: does this filesystem keep a file's stat data +# stable while the file is not being written to? +# +# git-annex records (inode, size, high-resolution mtime) for a file, +# copies it, then stats it again and compares the two exactly; if they +# differ it concludes the file changed under it, deletes the +# destination and fails the operation. On NFS with attribute caching +# the two stats of an *unmodified* file can disagree -- sub-second +# within the same second, or by up to `acregmax` when a cached +# attribute goes stale -- which is what surfaces as "failed to link to +# annex" / "unlock failed" / "content changed while it was being sent". +# See con/git-annex#293. +# +# This target measures that property directly, with no git-annex +# involved, so a red cell says "the filesystem", not "the application". +# +# Runs INSIDE the eval-under wrapper, i.e. with TMPDIR / HOME already +# pointing at the filesystem under test. Do not invoke directly for CI +# purposes -- go through bin/ci/run-under.sh \ +# mtime-stability. +# +# usage: +# bin/ci/target-mtime-stability.sh +# +# env (set by eval-under, honoured here): +# HOME /home -- the probe runs here +# TMPDIR +# +# env (optional, for hunting a rare case by hand): +# EVAL_UNDER_MTIME_ROUNDS rounds per worker (default 500) +# EVAL_UNDER_MTIME_JOBS concurrent workers (default 4) + +set -euo pipefail + +here="$(cd "$(dirname "$0")" && pwd)" + +ROUNDS="${EVAL_UNDER_MTIME_ROUNDS:-500}" +JOBS="${EVAL_UNDER_MTIME_JOBS:-4}" + +cd "$HOME" + +# Under load, because the race is a timing one: a single sequential +# worker on an idle mount can miss it for a long time. +exec "$here/mtime-stability.py" --dir "$HOME" --rounds "$ROUNDS" --jobs "$JOBS" diff --git a/bin/eval-under-nfs b/bin/eval-under-nfs index 731e527..23134bf 100755 --- a/bin/eval-under-nfs +++ b/bin/eval-under-nfs @@ -30,6 +30,12 @@ SYNC="${EVAL_UNDER_NFS_SYNC:-${DATALAD_TESTS_NFS_SYNC:-}}" # instead of the default (root_squash + drop back to the invoking user). NO_ROOT_SQUASH="${EVAL_UNDER_NFS_NO_ROOT_SQUASH:-0}" +# Extra mount(8) / exportfs(8) options, comma separated, appended to the +# ones derived from the flags above. Two namespaces, two variables: an +# option valid in one is often rejected by the other. +EXTRA_MOUNT_OPTS="${EVAL_UNDER_NFS_MOUNT_OPTS:-}" +EXTRA_EXPORT_OPTS="${EVAL_UNDER_NFS_EXPORT_OPTS:-}" + usage() { cat <<'EOF' Usage: eval-under-nfs [OPTIONS] -- CMD [ARGS...] @@ -69,6 +75,26 @@ Options (flag / env var / default / purpose): during clone/save/get, and `sync` pushed datalad CI jobs past 6h. Enable to reproduce that slowdown locally. + --mount-opts OPTS EVAL_UNDER_NFS_MOUNT_OPTS (unset) + Extra options appended to the `mount -t nfs -o` list, comma + separated. This is how you reach the client-side knobs the + built-in flags do not cover -- `actimeo=0` or `noac` (attribute + caching), `lookupcache=none`, `nocto`, `vers=3`, `rsize=`/`wsize=`. + + Attribute caching is not a detail: git-annex compares a file's + (inode, size, high-resolution mtime) before and after copying it + and fails the operation if they differ, and an NFS client can + report two different mtimes for a file nothing wrote to. Mounting + with `--mount-opts actimeo=0` turns that class of failure off and + is the fastest way to tell "the filesystem lied" from "the code is + wrong". See bin/ci/mtime-stability.py. + + --export-opts OPTS EVAL_UNDER_NFS_EXPORT_OPTS (unset) + Extra options appended to the `exportfs -o` list, comma + separated, for server-side knobs such as `no_subtree_check`, + `sec=`, `fsid=`. `sync`/`async` and `no_root_squash` have their + own flags above; this is for everything else. + --no-root-squash EVAL_UNDER_NFS_NO_ROOT_SQUASH (unset) Export with `no_root_squash` and run the wrapped command as root rather than dropping back to the invoking user. @@ -108,6 +134,8 @@ while [ $# -gt 0 ]; do --mount-point) MNT="$2"; shift 2 ;; --set-home) SET_HOME=1; shift ;; --sync) SYNC=1; shift ;; + --mount-opts) EXTRA_MOUNT_OPTS="$2"; shift 2 ;; + --export-opts) EXTRA_EXPORT_OPTS="$2"; shift 2 ;; --no-root-squash) NO_ROOT_SQUASH=1; shift ;; -h|--help) usage; exit 0 ;; --) shift; break ;; @@ -130,6 +158,12 @@ EXPORT_OPTS="$NFS_OPTS" if [ "$NO_ROOT_SQUASH" != 0 ]; then EXPORT_OPTS="$EXPORT_OPTS,no_root_squash" fi +if [ -n "$EXTRA_MOUNT_OPTS" ]; then + NFS_OPTS="$NFS_OPTS,$EXTRA_MOUNT_OPTS" +fi +if [ -n "$EXTRA_EXPORT_OPTS" ]; then + EXPORT_OPTS="$EXPORT_OPTS,$EXTRA_EXPORT_OPTS" +fi if [ "$(id -u)" -ne 0 ]; then command -v sudo >/dev/null || { diff --git a/evals/matrix.yaml b/evals/matrix.yaml index 350a674..1e0fee6 100644 --- a/evals/matrix.yaml +++ b/evals/matrix.yaml @@ -79,6 +79,27 @@ targets: needs-root: true needs-git-annex: false + # No suite to install: a few hundred write/stat/copy/stat rounds, + # asking whether the filesystem keeps an unmodified file's stat data + # stable. Fast, and the one cell that distinguishes "the filesystem + # lied" from "the application is wrong" -- see con/git-annex#293. + - name: mtime-stability + label: mtime stability + timeout: 600 + loop-size-mb: 100 + needs-root: false + needs-git-annex: false + + # The git-annex half of the same question: loops unlock / unlocked + # add and reports a failure rate instead of a pass/fail of the whole + # `git annex test` suite. + - name: git-annex-linkannex + label: git-annex linkAnnex loop + timeout: 1200 + loop-size-mb: 100 + needs-root: false + needs-git-annex: true + # Run serially; a sync-heavy NFS or BeeGFS mount is an order of # magnitude slower than ext4. - name: pjdfstest From b8aa45a4a4560d5f6ceb223c32d2f5cea226fe37 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Wed, 30 Sep 2026 19:20:48 +0000 Subject: [PATCH 2/6] Score the two new targets with the known-issues machinery The rebase onto master brought in per-test cell judging (#14): each target's output goes through bin/ci/collect-results.py into results.tsv, and known_issues.py judges that instead of the suite's exit code. collect-results.py dispatches on the target name and raises `no results adapter for target ...` otherwise, so as first written these two targets would have produced incomplete cells. Both now print TAP and are scored like stress-ng: * mtime-stability: one point per stat field -- inode-stable, size-stable, mtime-stable -- with the rate and the worst mtime delta in the description, and the first few mismatches as TAP comments. Per-field ids rather than per-round ones because a known issue has to be able to name "this filesystem moves the mtime", not "round 417". * git-annex-linkannex: one point per direction, `unlock` and `add-unlocked`. linkannex-loop.sh gained --report FILE so the wrapper reads a count instead of parsing prose, and a loop that dies before reporting becomes a failing point rather than a silent pass. * collect_tap_target() serves both, taking the row id from the description's first word, as collect_stress_ng does. No known-issues entries are added: whether NFS actually fails mtime-stable on this CI's ubuntu-22.04 runners (kernel 6.8) is an open question -- con/git-annex only sees the git-annex failures on 24.04 (kernel 6.17), see con/git-annex#294 -- so let the first run answer it rather than pre-declaring a verdict. Tested: bin/ci/run-checks.sh fully green (shellcheck 29 scripts, pyflakes, bats 30 tests, known-issues, 28 unit tests). Both targets run end-to-end into a cell dir and collect cleanly on ext4; the TAP failure path was exercised with a stubbed cp that bumps the source mtime, which yields exactly the NFS signature (inode and size stable, mtime-stable not ok). Two bats tests cover the new flags. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01B89nUooZLfThcTA4fSMPGf --- bin/ci/collect-results.py | 26 +++++++++++++++++ bin/ci/linkannex-loop.sh | 11 ++++++- bin/ci/mtime-stability.py | 41 ++++++++++++++++++++++++-- bin/ci/target-git-annex-linkannex.sh | 43 +++++++++++++++++++++++----- bin/ci/target-mtime-stability.sh | 5 +++- tests/eval-under.bats | 21 ++++++++++++++ 6 files changed, 136 insertions(+), 11 deletions(-) diff --git a/bin/ci/collect-results.py b/bin/ci/collect-results.py index 24dbd7f..32233e0 100755 --- a/bin/ci/collect-results.py +++ b/bin/ci/collect-results.py @@ -212,6 +212,29 @@ def collect_stress_ng(lines: list[str]) -> list[Row]: return rows +def collect_tap_target(name: str, lines: list[str]) -> list[Row]: + """From the TAP a target prints itself, ids taken from the description. + + Shared by mtime-stability and git-annex-linkannex: both print a short + fixed plan whose descriptions start with a stable slug, so a known + issue can name "mtime-stable" or "add-unlocked" rather than a round + number that means nothing on the next run. + """ + tf = TapFile(name) + for ln in lines: + tf.feed(ln) + check_no_dupes([tf]) + if tf.plan is None: + raise Incomplete(f"no TAP plan from target-{name}.sh (suite died?)") + if tf.plan != len(tf.points): + raise Incomplete(f"{name} TAP planned {tf.plan}, parsed {len(tf.points)}") + rows = [] + for o, d in tf.points.values(): + slug, _, detail = d.partition(" ") + rows.append((slug, o, detail)) + return rows + + TASTY_RESULT = re.compile(r"^(?P *)(?P\S.*?):\s+(?POK|FAIL|SKIP)\b") TASTY_GROUP_SUMMARY = re.compile( r"^(?:All (?P\d+) tests passed|(?P\d+) out of (?P\d+) tests failed)") @@ -335,6 +358,9 @@ def main() -> int: "pjdfstest": collect_pjdfstest, "stress-ng": collect_stress_ng, "git-annex": collect_git_annex, + "mtime-stability": lambda lines: collect_tap_target("mtime-stability", lines), + "git-annex-linkannex": lambda lines: collect_tap_target( + "git-annex-linkannex", lines), } log = a.cell_dir / "suite.log" rows, reason = [], "" diff --git a/bin/ci/linkannex-loop.sh b/bin/ci/linkannex-loop.sh index 4a7ba37..fe5e47c 100755 --- a/bin/ci/linkannex-loop.sh +++ b/bin/ci/linkannex-loop.sh @@ -16,7 +16,10 @@ # reports a rate instead of a single pass/fail. # # usage: -# ./nfs-annex-linkannex-loop.sh [-n ROUNDS] [-j WORKERS] [-m MODE] [-d DIR] +# linkannex-loop.sh [-n ROUNDS] [-j WORKERS] [-m MODE] [-d DIR] [--report FILE] +# +# --report writes "\t" to FILE, so a caller can turn the +# result into TAP without parsing this script's prose. # # exits non-zero if any round failed. @@ -26,6 +29,7 @@ ROUNDS=200 WORKERS=1 MODE=unlock DIR=. +REPORT= usage() { sed -n '3,20p' "$0"; } @@ -35,6 +39,7 @@ while [ $# -gt 0 ]; do -j|--workers) WORKERS="$2"; shift 2 ;; -m|--mode) MODE="$2"; shift 2 ;; -d|--dir) DIR="$2"; shift 2 ;; + --report) REPORT="$2"; shift 2 ;; -h|--help) usage; exit 0 ;; *) echo "unknown arg: $1" >&2; usage >&2; exit 2 ;; esac @@ -133,4 +138,8 @@ printf '\n%s/%s rounds failed in linkAnnex (%s%%), in %ss\n' \ "$(awk -v a="$total_failures" -v b="$total" 'BEGIN{printf "%.2f", b ? 100*a/b : 0}')" \ "$((end - start))" +if [ -n "$REPORT" ]; then + printf '%s\t%s\n' "$total_failures" "$total" > "$REPORT" +fi + [ "$total_failures" -eq 0 ] || exit 1 diff --git a/bin/ci/mtime-stability.py b/bin/ci/mtime-stability.py index 54a5089..73ac427 100755 --- a/bin/ci/mtime-stability.py +++ b/bin/ci/mtime-stability.py @@ -145,6 +145,12 @@ def main() -> int: ) p.add_argument("--no-copy", action="store_true", help="control run: stat twice without copying") p.add_argument("--quiet", action="store_true", help="only print the summary") + p.add_argument( + "--tap", + action="store_true", + help="emit TAP for bin/ci/collect-results.py: three stable points, one per " + "stat field, with the rate in the description", + ) args = p.parse_args() root = os.path.abspath(args.dir) @@ -156,10 +162,12 @@ def main() -> int: capture_output=True, text=True, ).stdout.strip() + # Both modes prefix these with "#", which is a comment in TAP and just + # a comment to a human, so there is only one format to read. print(f"# directory: {root}") print(f"# mount: {fsinfo or '(findmnt unavailable)'}") print( - f"# plan: {args.jobs} worker(s) x {args.rounds} rounds, {args.size}B files, " + f"# rounds: {args.jobs} worker(s) x {args.rounds} rounds, {args.size}B files, " f"delay={args.delay}s, {'no copy (control)' if args.no_copy else 'copy via cp'}" ) @@ -185,12 +193,16 @@ def worker(wid: int) -> list[Mismatch]: total = args.jobs * args.rounds elapsed = time.time() - started - if mismatches and not args.quiet: + if mismatches and not args.quiet and not args.tap: print(f"\n# mismatches ({len(mismatches)}):") for m in mismatches: print(m.describe()) rate = 100.0 * len(mismatches) / total if total else 0.0 + if args.tap: + emit_tap(mismatches, total, elapsed) + return 1 if mismatches else 0 + print( f"\n{len(mismatches)}/{total} rounds saw the inode cache change under an " f"unmodified file ({rate:.2f}%), in {elapsed:.1f}s" @@ -204,5 +216,30 @@ def worker(wid: int) -> list[Mismatch]: return 0 +def emit_tap(mismatches: list[Mismatch], total: int, elapsed: float) -> None: + """One point per stat field, so the ids are stable across runs. + + Round numbers would not be: a known issue has to be able to name + "this filesystem moves the mtime", not "round 417 failed". + """ + print(f"# {len(mismatches)}/{total} rounds changed, in {elapsed:.1f}s") + fields = ("inode", "size", "mtime") + hits = {f: [m for m in mismatches if f in differing_fields(m.before, m.after)] + for f in fields} + print(f"1..{len(fields)}") + for n, f in enumerate(fields, start=1): + bad = hits[f] + detail = f"{f}-stable {len(bad)}/{total} rounds changed" + if bad: + worst = max(abs(m.after.mtime_ns - m.before.mtime_ns) for m in bad) + rate = 100.0 * len(bad) / total if total else 0.0 + detail += f" ({rate:.2f}%), worst mtime delta {worst / 1e9:.9f}s" + print(f"not ok {n} - {detail}") + for m in bad[:5]: + print(f"# {m.describe().strip()}") + else: + print(f"ok {n} - {detail}") + + if __name__ == "__main__": sys.exit(main()) diff --git a/bin/ci/target-git-annex-linkannex.sh b/bin/ci/target-git-annex-linkannex.sh index a3cb0d7..7e20d7d 100755 --- a/bin/ci/target-git-annex-linkannex.sh +++ b/bin/ci/target-git-annex-linkannex.sh @@ -47,15 +47,44 @@ git config --global --get user.email >/dev/null 2>&1 \ git config --global --get user.name >/dev/null 2>&1 \ || git config --global user.name "GitHub Almighty" -git annex version | head -1 +# As a TAP comment: everything this target prints but the plan and the +# points has to be a comment, or the collector would try to read it. +git annex version | head -1 | sed 's/^/# /' -rc=0 # Both directions, reported separately: `unlock` exercises linkAnnex -# From (annex object -> worktree), `add-unlocked` exercises To. -for mode in unlock add-unlocked; do - echo - echo "=== mode: $mode" +# From (annex object -> worktree), `add-unlocked` exercises To. Each +# mode becomes one TAP point, named after the mode so the id is stable +# and a known issue can name it; bin/ci/collect-results.py scores them. +reportdir="$(mktemp -d)" +trap 'rm -rf "$reportdir"' EXIT + +modes=(unlock add-unlocked) +rc=0 +for mode in "${modes[@]}"; do + echo "# === mode: $mode" "$here/linkannex-loop.sh" --dir "$HOME" --mode "$mode" \ - --rounds "$ROUNDS" --workers "$WORKERS" || rc=1 + --rounds "$ROUNDS" --workers "$WORKERS" \ + --report "$reportdir/$mode" 2>&1 | sed 's/^/# /' || rc=1 +done + +echo "1..${#modes[@]}" +n=0 +for mode in "${modes[@]}"; do + n=$((n + 1)) + if [ -r "$reportdir/$mode" ]; then + read -r failures total < "$reportdir/$mode" + else + failures=; total= + fi + if [ -z "${total:-}" ]; then + # The loop died before reporting: a result, not a harness error. + echo "not ok $n - $mode loop did not report (died before finishing?)" + rc=1 + elif [ "$failures" -eq 0 ]; then + echo "ok $n - $mode 0/$total rounds failed" + else + pct="$(awk -v a="$failures" -v b="$total" 'BEGIN{printf "%.2f", b ? 100*a/b : 0}')" + echo "not ok $n - $mode $failures/$total rounds failed ($pct%)" + fi done exit "$rc" diff --git a/bin/ci/target-mtime-stability.sh b/bin/ci/target-mtime-stability.sh index 2346bc2..0355a4f 100755 --- a/bin/ci/target-mtime-stability.sh +++ b/bin/ci/target-mtime-stability.sh @@ -45,4 +45,7 @@ cd "$HOME" # Under load, because the race is a timing one: a single sequential # worker on an idle mount can miss it for a long time. -exec "$here/mtime-stability.py" --dir "$HOME" --rounds "$ROUNDS" --jobs "$JOBS" +# --tap so bin/ci/collect-results.py can score the cell per stat field; +# the human-readable lines come through as TAP comments. +exec "$here/mtime-stability.py" --tap \ + --dir "$HOME" --rounds "$ROUNDS" --jobs "$JOBS" diff --git a/tests/eval-under.bats b/tests/eval-under.bats index f06595a..89434fe 100644 --- a/tests/eval-under.bats +++ b/tests/eval-under.bats @@ -316,6 +316,27 @@ FAKE done } +@test "nfs: --mount-opts and --export-opts are documented and parsed" { + run "$REPO_ROOT/bin/eval-under-nfs" --help + [ "$status" -eq 0 ] + for opt in --mount-opts --export-opts; do + if [[ "$output" != *"$opt"* ]]; then + echo "eval-under-nfs: --help does not mention $opt" >&2 + return 1 + fi + done + # Parsed rather than rejected, and consuming their argument: --help + # after them still reaches the usage text. + run "$REPO_ROOT/bin/eval-under-nfs" --mount-opts actimeo=0 --export-opts no_subtree_check --help + [ "$status" -eq 0 ] + [[ "$output" == *"Usage: eval-under-nfs"* ]] +} + +@test "nfs: --mount-opts without a value is an error, not a silent skip" { + run "$REPO_ROOT/bin/eval-under-nfs" --mount-opts + [ "$status" -ne 0 ] +} + @test "every installed backend documents the common options" { local b opt for b in $("$DISPATCHER" --list); do From ee2b15d3cec0352816f6489f3ab7efd44af870bc Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 20:38:32 +0000 Subject: [PATCH 3/6] Fetch git-annex for every cell that declares it needs it The five git-annex-linkannex cells of run 36765084880 all died the same way, on every backend including ext4: I: running: timeout 1200 .../bin/ci/target-git-annex-linkannex.sh git: 'annex' is not a git command. See 'git --help'. git-annex was never installed. evals/matrix.yaml declares `needs-git-annex: true` for the target and install-target.sh routes it to the git-annex arm, but the workflow step that actually fetches the daily build was gated on `matrix.target == 'git-annex'` -- the target's *name*, not the flag -- so it never ran. ext4 failing was the tell: it is the negative control, and a filesystem-independent failure is the harness. matrix-json.sh now carries `needs-git-annex` per cell, so the workflow gates on the data ("each entry carries everything the job body needs", per its own header) and target_needs_git_annex() in matrix.sh gets its first caller -- it was dead code, which is why the duplication went unnoticed. Adding a git-annex-using target stays a data edit. tests/test_matrix_json.py holds both halves of that contract: every cell's flag matches the data file, the flag is a JSON boolean (the string "0" is truthy in a GitHub expression), and no step is gated on a target name. Checked against master's workflow, where the last of those fails. Also drops the stale target list from install-target.sh's missing-arg message, which had gone stale exactly this way; target_known already prints the live list from evals/matrix.yaml. Tested: the failure reproduced locally by running the target with a PATH holding everything but git-annex -- same `git: 'annex' is not a git command`, same "no TAP plan from target-git-annex-linkannex.sh (suite died?)" from the collector -- and the same target emitting `1..2` with two passing points once git-annex is present. Old vs new gate over the rendered matrix: 5 cells vs 10, the extra 5 being exactly the cells that failed. bin/ci/run-checks.sh fully green (shellcheck, pyflakes, bats 30, known-issues, 33 unit tests). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01B89nUooZLfThcTA4fSMPGf --- .github/workflows/test.yaml | 11 +++-- bin/ci/install-target.sh | 8 +++- bin/ci/matrix-json.sh | 13 +++++- tests/test_matrix_json.py | 84 +++++++++++++++++++++++++++++++++++++ 4 files changed, 109 insertions(+), 7 deletions(-) create mode 100644 tests/test_matrix_json.py diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 71d15b1..f860ff9 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -91,10 +91,15 @@ jobs: - name: Install target ${{ matrix.target }} run: bin/ci/install-target.sh "${{ matrix.target }}" - # Only the git-annex target needs the daily build (and the token - # that fetching it requires). + # Only some targets need the daily build (and the token that + # fetching it requires). Which ones is data -- `needs-git-annex` + # in evals/matrix.yaml, carried through by matrix-json.sh -- not a + # target name spelled out here: a second git-annex-using target + # would silently run without git-annex installed. + # Indexed, not dotted: the documented form for a property name + # containing hyphens. - name: Fetch latest git-annex daily build from con/git-annex - if: matrix.target == 'git-annex' + if: ${{ matrix['needs-git-annex'] }} env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} GIT_ANNEX_RUN_ID: ${{ needs.matrix.outputs.git-annex-run }} diff --git a/bin/ci/install-target.sh b/bin/ci/install-target.sh index 2da0998..159cd89 100755 --- a/bin/ci/install-target.sh +++ b/bin/ci/install-target.sh @@ -15,7 +15,8 @@ # usage: # bin/ci/install-target.sh # -# target = git-annex | git | stress-ng | pjdfstest +# target = git-annex | git-annex-linkannex | git | stress-ng +# | mtime-stability | pjdfstest # # env overrides: # EVAL_UNDER_SRC_DIR where to clone/build (/opt/eval-under-src) @@ -34,7 +35,10 @@ here="$(cd "$(dirname "$0")" && pwd)" # shellcheck source=bin/ci/matrix.sh disable=SC1091 . "$here/matrix.sh" -TARGET="${1:?target required (git-annex|git|stress-ng|pjdfstest)}" +# No target list spelled out here: target_known below rejects an unknown +# one and prints the live list from evals/matrix.yaml, which cannot go +# stale the way this message just did. +TARGET="${1:?target required (see targets in evals/matrix.yaml)}" target_known "$TARGET" || { echo "unknown target: $TARGET (expected: ${EVAL_UNDER_TARGETS[*]})" >&2 exit 1 diff --git a/bin/ci/matrix-json.sh b/bin/ci/matrix-json.sh index ebf2948..e3f4ed2 100755 --- a/bin/ci/matrix-json.sh +++ b/bin/ci/matrix-json.sh @@ -18,6 +18,12 @@ # version backend version, or "n/a" # target suite to run under it # slug filename-safe cell id, for artifact names +# needs-git-annex +# whether this cell's suite needs the git-annex daily build, +# straight from evals/matrix.yaml. The workflow gates the +# fetch step on it, so adding a git-annex-using target stays +# a data edit. Referenced as matrix['needs-git-annex'], the +# documented index form for a property name with hyphens. # # usage: # bin/ci/matrix-json.sh # all cells @@ -36,7 +42,9 @@ entries=() for cell in "${EVAL_UNDER_BACKENDS[@]}"; do IFS='|' read -r backend version label <<< "$cell" for target in "${EVAL_UNDER_TARGETS[@]}"; do - entries+=("$backend|$version|$label|$target|$(target_label "$target")|$(cell_slug "$backend" "$version" "$target")") + needs_ga=0 + target_needs_git_annex "$target" && needs_ga=1 + entries+=("$backend|$version|$label|$target|$(target_label "$target")|$(cell_slug "$backend" "$version" "$target")|$needs_ga") done done @@ -48,13 +56,14 @@ for line in sys.stdin: line = line.rstrip("\n") if not line: continue - backend, version, blabel, target, tlabel, slug = line.split("|") + backend, version, blabel, target, tlabel, slug, needs_ga = line.split("|") include.append({ "name": "%s / %s" % (blabel, tlabel), "backend": backend, "version": version, "target": target, "slug": slug, + "needs-git-annex": needs_ga == "1", }) if not include: diff --git a/tests/test_matrix_json.py b/tests/test_matrix_json.py new file mode 100644 index 0000000..4a92b34 --- /dev/null +++ b/tests/test_matrix_json.py @@ -0,0 +1,84 @@ +# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko +# SPDX-License-Identifier: MIT +# +# Generated with Claude Code +# +# bin/ci/matrix-json.sh against evals/matrix.yaml, and the workflow's +# side of that contract. +# +# evals/matrix.yaml is the single source of truth for the matrix, and +# matrix-json.sh's promise is that "each entry carries everything the job +# body needs, so the workflow never has to recompute anything about a +# cell". These tests hold it to that, because a workflow that recomputes +# a cell's properties from the target *name* is how the +# git-annex-linkannex cells first ran without git-annex installed: they +# declared needs-git-annex, but the fetch step tested +# `matrix.target == 'git-annex'`, so it never ran and every such cell +# died with "git: 'annex' is not a git command". + +import json +import re +import subprocess +import sys +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +WORKFLOW = ROOT / ".github/workflows/test.yaml" +sys.path.insert(0, str(ROOT / "bin" / "ci")) + +import evals # noqa: E402 + + +def cells() -> list[dict]: + out = subprocess.run([ROOT / "bin/ci/matrix-json.sh"], + check=True, capture_output=True, text=True).stdout + return json.loads(out)["include"] + + +class TestMatrixJson(unittest.TestCase): + def setUp(self): + self.matrix = evals.load_matrix() + self.cells = cells() + + def test_every_backend_times_every_target(self): + want = len(self.matrix["backends"]) * len(self.matrix["targets"]) + self.assertEqual(want, len(self.cells)) + self.assertEqual(len({c["slug"] for c in self.cells}), want, + "cell slugs must be unique -- they name artifacts") + + def test_needs_git_annex_matches_the_data_file(self): + """The flag the workflow gates the daily build on comes from the + data file, for every cell, not just the ones we remembered.""" + declared = {t["name"]: bool(t["needs-git-annex"]) + for t in self.matrix["targets"]} + self.assertTrue(any(declared.values()), "fixture check: someone needs it") + for c in self.cells: + with self.subTest(cell=c["slug"]): + self.assertIn("needs-git-annex", c) + self.assertEqual(c["needs-git-annex"], declared[c["target"]]) + + def test_needs_git_annex_is_a_json_boolean(self): + """`if:` reads this directly, and the *string* "0" is truthy in a + GitHub expression -- it would fetch git-annex for every cell.""" + for c in self.cells: + with self.subTest(cell=c["slug"]): + self.assertIsInstance(c["needs-git-annex"], bool) + + +class TestWorkflowReadsTheFlag(unittest.TestCase): + def setUp(self): + self.workflow = WORKFLOW.read_text() + + def test_daily_build_is_gated_on_the_flag(self): + self.assertIn("matrix['needs-git-annex']", self.workflow) + + def test_no_step_is_gated_on_a_target_name(self): + """Any per-target behaviour belongs in evals/matrix.yaml as a + flag, so adding a target cannot silently skip a step it needs.""" + hits = re.findall(r"matrix\.target\s*[=!]=.*", self.workflow) + self.assertEqual(hits, [], "gate on a matrix.yaml flag instead") + + +if __name__ == "__main__": + unittest.main() From 69c6ada4301555487b2cbce39f6d6847c79f689c Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 20:57:06 +0000 Subject: [PATCH 4/6] linkAnnex loop: never report a full disk as a linkAnnex failure Run 36774070493 got git-annex installed (the previous commit's fix) and promptly reported the opposite nonsense: 800/800 rounds "failed" in both modes, 100.00%, on ext4 -- the negative control, where the true rate is 0. The log's reason: not enough free space, need 21.73 MB more (use --force to override this check or adjust annex.diskreserve) f106 failed to link to annex Two separate defects, and the first is the dangerous one. A full filesystem makes git-annex print exactly the same "failed to link to annex" / "unlock failed" lines as the inode-cache mismatch this loop exists to measure, and the loop tallied them. So a cell that ran out of room reported a 100% linkAnnex failure rate -- indistinguishable, in the results, from the NFS bug this target was written to quantify. Left in, it would have made the first red NFS cell unreadable. The loop now recognises the space messages, aborts instead of counting, and exits 4 (documented alongside the other statuses); the target turns that into an incomplete cell rather than a filesystem verdict the run cannot support. Second, the reserve. Measured on a fresh ext4 image with 83MB free, `git annex unlock` of a 14-byte file refuses with "need 13.73 MB more" -- it wants ~97MB free to rewrite 14 bytes, which a 100MB image cannot give, so every round failed from round 0 without linkAnnex being reached at all. The loop sets annex.diskreserve=0 in each probe repo: it is measuring the inode-cache comparison, not git-annex's disk-space policy, and real ENOSPC is still caught by the check above. loop-size-mb 100 -> 256 as well. With the reserve gone the cell passes at 100MB, but it consumed ~52MB of the ~83MB usable, and vfat keeps a worktree copy of every file because it has no symlinks, so it needs more than ext4 did. The backing image is sparse, so the headroom is free. Tested: the ext4 cell reproduced the 800/800 exactly at the old settings, and now reports `ok 1 - unlock 0/800` / `ok 2 - add-unlocked 0/800`, finishing with 153.8M of 256M free (was 30.9M of 100M). The new abort path was exercised on a deliberately pre-filled 60MB image: genuine ENOSPC, exit 4, the rate withheld, no report file, and through the target a cell that is "incomplete", not "failing". vfat could not be measured here -- no mkfs.vfat and apt is blocked -- so CI is its first run. bin/ci/run-checks.sh fully green (shellcheck, pyflakes, bats 30, known-issues, 33 unit tests). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01B89nUooZLfThcTA4fSMPGf --- bin/ci/linkannex-loop.sh | 65 ++++++++++++++++++++++++++-- bin/ci/target-git-annex-linkannex.sh | 14 +++++- evals/matrix.yaml | 6 ++- 3 files changed, 80 insertions(+), 5 deletions(-) diff --git a/bin/ci/linkannex-loop.sh b/bin/ci/linkannex-loop.sh index fe5e47c..b74cd28 100755 --- a/bin/ci/linkannex-loop.sh +++ b/bin/ci/linkannex-loop.sh @@ -21,7 +21,13 @@ # --report writes "\t" to FILE, so a caller can turn the # result into TAP without parsing this script's prose. # -# exits non-zero if any round failed. +# exit status: +# 0 every round linked cleanly +# 1 some rounds failed in linkAnnex -- the finding this looks for +# 2 bad usage +# 3 the harness could not run (repo setup, a worker that vanished) +# 4 the filesystem filled up, so the run measures free space rather +# than linkAnnex; the rate is withheld deliberately set -u -o pipefail @@ -63,6 +69,29 @@ echo "# mount: $(findmnt -no FSTYPE,OPTIONS --target "$root" 2>/dev/null || ec echo "# version: $(git annex version --raw 2>/dev/null || echo unknown)" echo "# plan: $WORKERS worker(s) x $ROUNDS rounds, mode=$MODE" +# A full filesystem produces the very same "failed to link to annex" / +# "unlock failed" lines as the inode-cache mismatch this loop exists to +# measure -- git-annex reports both as a failure to link. Counting them +# would report a filesystem that merely ran out of room as a 100% +# linkAnnex failure rate, which is exactly what a 100MB loop image did +# on ext4 (con/eval-under#11): 800/800 "failures", none of them real. +# So they abort the run instead of being tallied. +is_space_failure() { + case "$1" in + *"not enough free space"*|*"No space left on device"*|*"no space left on device"*) + return 0 ;; + esac + return 1 +} + +# Called from a worker on a space failure: mark it, so the aggregation +# below can refuse to report a rate, and stop this worker. +abort_no_space() { + printf 'worker %s round %s: OUT OF SPACE -- not a linkAnnex failure, aborting\n%s\n' \ + "$1" "$2" "$3" >&2 + : > "$work/nospace.$1" +} + # Each worker gets its own repo, the way `git annex test` runs its parts. run_worker() { # Note: `local a=$1 b=$a` would expand $a before the assignment happens, @@ -77,6 +106,17 @@ run_worker() { git config user.email test@example.com git config user.name "NFS Probe" git annex init -q "probe-$wid" >/dev/null 2>&1 + # This loop measures the inode-cache comparison, not git-annex's + # disk-space policy. Measured on a fresh ext4 image with 83MB free: + # `git annex unlock` of a 14-byte file still refuses, with "not + # enough free space, need 13.73 MB more" -- so it wanted ~97MB free + # to rewrite 14 bytes, and on a small backing image every round + # fails before linkAnnex is ever reached. (Whatever computes that + # figure, it is far above the 1MB annex.diskreserve is documented to + # default to; not investigated further, since this loop has no + # business enforcing a reserve at all.) Genuine ENOSPC is still + # caught -- see is_space_failure. + git config annex.diskreserve 0 ) || { echo "worker $wid: repo setup failed" >&2; return 3; } cd "$repo" || return 3 @@ -84,6 +124,7 @@ run_worker() { printf 'content %s %s\n' "$wid" "$i" > "f$i" if [ "$MODE" = add-unlocked ]; then out="$(git -c annex.addunlocked=true annex add "f$i" 2>&1)" || { + is_space_failure "$out" && { abort_no_space "$wid" "$i" "$out"; return 4; } failures=$((failures + 1)) printf 'worker %s round %s: add failed\n%s\n' "$wid" "$i" "$out" >&2 continue @@ -91,21 +132,25 @@ run_worker() { # the To-direction failure is a warning + non-zero exit; also catch the # message in case a future version only warns case "$out" in *"failed to link to annex"*) + is_space_failure "$out" && { abort_no_space "$wid" "$i" "$out"; return 4; } failures=$((failures + 1)) printf 'worker %s round %s:\n%s\n' "$wid" "$i" "$out" >&2 ;; esac else - git annex add -q "f$i" >/dev/null 2>&1 || { + out="$(git annex add -q "f$i" 2>&1)" || { + is_space_failure "$out" && { abort_no_space "$wid" "$i" "$out"; return 4; } failures=$((failures + 1)) - printf 'worker %s round %s: add failed\n' "$wid" "$i" >&2 + printf 'worker %s round %s: add failed\n%s\n' "$wid" "$i" "$out" >&2 continue } out="$(git annex unlock "f$i" 2>&1)" || { + is_space_failure "$out" && { abort_no_space "$wid" "$i" "$out"; return 4; } failures=$((failures + 1)) printf 'worker %s round %s:\n%s\n' "$wid" "$i" "$out" >&2 continue } case "$out" in *"unlock failed"*) + is_space_failure "$out" && { abort_no_space "$wid" "$i" "$out"; return 4; } failures=$((failures + 1)) printf 'worker %s round %s:\n%s\n' "$wid" "$i" "$out" >&2 ;; esac @@ -122,6 +167,20 @@ done wait end=$(date +%s) +# Exit 4, distinct from "some rounds failed" (1) and "the harness is +# broken" (3): the run is void rather than negative, and the caller +# (bin/ci/target-git-annex-linkannex.sh) turns it into an incomplete +# cell instead of a filesystem verdict the numbers do not support. +if compgen -G "$work/nospace.*" >/dev/null 2>&1; then + { + echo "ERROR: the filesystem under test ran out of space." + echo " Any rate from this run would measure free space, not linkAnnex." + echo " Give the cell a bigger backing image: loop-size-mb in" + echo " evals/matrix.yaml, or --size for bin/eval-under loop." + } >&2 + exit 4 +fi + total_failures=0 for ((w = 0; w < WORKERS; w++)); do if [ ! -e "$work/failures.$w" ]; then diff --git a/bin/ci/target-git-annex-linkannex.sh b/bin/ci/target-git-annex-linkannex.sh index 7e20d7d..18eeb75 100755 --- a/bin/ci/target-git-annex-linkannex.sh +++ b/bin/ci/target-git-annex-linkannex.sh @@ -62,9 +62,21 @@ modes=(unlock add-unlocked) rc=0 for mode in "${modes[@]}"; do echo "# === mode: $mode" + # pipefail is on and sed always succeeds, so the pipeline's status is + # the loop's own -- including its exit 4 for "out of space". + loop_rc=0 "$here/linkannex-loop.sh" --dir "$HOME" --mode "$mode" \ --rounds "$ROUNDS" --workers "$WORKERS" \ - --report "$reportdir/$mode" 2>&1 | sed 's/^/# /' || rc=1 + --report "$reportdir/$mode" 2>&1 | sed 's/^/# /' || loop_rc=$? + if [ "$loop_rc" = 4 ]; then + # Out of space. Exit without a plan, so the cell comes out + # "incomplete" (a harness problem, which it is) rather than + # pinning a verdict on the filesystem that the run cannot + # support -- see the exit-status list in linkannex-loop.sh. + echo "# ABORT: $mode ran out of disk space; no verdict from this cell" + exit 4 + fi + [ "$loop_rc" = 0 ] || rc=1 done echo "1..${#modes[@]}" diff --git a/evals/matrix.yaml b/evals/matrix.yaml index 1e0fee6..09503e2 100644 --- a/evals/matrix.yaml +++ b/evals/matrix.yaml @@ -96,7 +96,11 @@ targets: - name: git-annex-linkannex label: git-annex linkAnnex loop timeout: 1200 - loop-size-mb: 100 + # 800 rounds x 2 modes of add/commit is metadata-heavy: measured + # ~52MB of an ext4 image's ~83MB usable, and vfat keeps a worktree + # copy of every file because it has no symlinks, so it needs more. + # The image is sparse, so the headroom costs nothing until used. + loop-size-mb: 256 needs-root: false needs-git-annex: true From 5d52a67bbf67585f6e1f61b9ebfb7dca60afbc2e Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 21:02:45 +0000 Subject: [PATCH 5/6] linkAnnex loop: clean up repos whose object dirs git-annex made read-only The NFS cell of run 36774070493 passed (0/800 both modes) but printed dozens of these first: rm: cannot remove '.../annex/objects/x1/V2/SHA256E-s14--.../SHA256E-s14--...': Permission denied git-annex sets each object's directory to dr-xr-xr-x, and nothing can unlink through a directory it cannot write. Running as root hides this, which is why it never showed locally; under the root-squashed export this target uses (needs-root: false) it does not, so the EXIT trap's `rm -rf` failed on every object. Two costs: a screenful of noise ahead of the TAP output, in a target whose only job is to be read, and the probe repos left on the mount, where the next mode's 800 rounds still have to fit -- which is also why the ext4 headroom measured in the previous commit was pessimistic. chmod -R u+w first, as git-annex's own test suite does. Tested: reproduced as an unprivileged user on a repo with real annex objects (dr-xr-xr-x, owner nobody) -- plain `rm -rf` gives exactly the message above and leaves the object behind, chmod-then-rm leaves nothing and says nothing. The ext4 cell still reports `ok 1 - unlock 0/800` / `ok 2 - add-unlocked 0/800`, now with zero "cannot remove" lines and no leftover probe dirs. bin/ci/run-checks.sh fully green. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01B89nUooZLfThcTA4fSMPGf --- bin/ci/linkannex-loop.sh | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/bin/ci/linkannex-loop.sh b/bin/ci/linkannex-loop.sh index b74cd28..a926b92 100755 --- a/bin/ci/linkannex-loop.sh +++ b/bin/ci/linkannex-loop.sh @@ -62,7 +62,20 @@ command -v git-annex >/dev/null 2>&1 || command -v git >/dev/null 2>&1 || { mkdir -p "$DIR" root="$(cd "$DIR" && pwd)" work="$(mktemp -d "$root/linkannex-loop-XXXXXX")" -trap 'rm -rf "$work"' EXIT + +# git-annex makes each object's directory read-only (dr-xr-xr-x), and +# nothing can unlink through a directory it cannot write. As root that is +# invisible; under a root-squashed NFS export it is not, and a plain +# `rm -rf` then printed a screenful of "Permission denied" into the cell +# log -- burying the TAP output of a target whose only job is to be read +# -- and left the probe repos on the mount for the next mode's rounds to +# squeeze past. Make them writable first, as git-annex's own test suite +# does. +cleanup() { + chmod -R u+w "$work" 2>/dev/null || : + rm -rf "$work" +} +trap cleanup EXIT echo "# dir: $root" echo "# mount: $(findmnt -no FSTYPE,OPTIONS --target "$root" 2>/dev/null || echo '(findmnt unavailable)')" From f35898555fc9f8ddfcf113768810c3ce0d03af56 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 23:21:50 +0000 Subject: [PATCH 6/6] linkAnnex loop: 200 rounds per mode, sized by the slowest backend Both BeeGFS cells of run 36776837242 came back incomplete (exit 124). Not a filesystem verdict and not the cleanup: BeeGFS 8.1.0 finished mode `unlock` cleanly at 0/800 (0.00%) -- and took 967s of the target's 1200s budget to do it, so mode `add-unlocked` was killed four minutes in. NFS, ext4 and vfat each finish a mode in 100-200s; BeeGFS is an order of magnitude slower, exactly as evals/matrix.yaml already says of it. Rounds per worker 200 -> 50, so 200 rounds per mode instead of 800. That brings BeeGFS to roughly 240s per mode, fits the existing timeout with room to spare, and restores what this target is for: a rate in minutes rather than a pass/fail after twenty. Raising the timeout instead would have cost up to ~50min per BeeGFS cell and risked the 60-minute job cap -- teardown alone hung 15 minutes in that run -- turning a timeout into a bare GitHub cancellation with no logs. The cost is detection power: 200 rounds per mode will not reliably show a rate below ~1%. Both the target's header and this message say so, and point at bin/ci/linkannex-loop.sh -n for hunting something rare by hand, because that is the knob to reach for rather than this default. Separately, and not this PR's to fix: the BeeGFS cluster degraded during that run -- "Receive failed from node_meta_1" from t=927s, then metadata and storage nodes to `offline`, and teardown wedged in FhgfsOps_flush. A metadata-heavy workload appears to be enough to upset it. loop-size-mb stays 256: its comment now says the 52MB measurement was at 800 rounds per mode, and that the sparse image leaves room to raise the rounds by hand without editing the matrix too. Tested: the ext4 cell runs in 38s wall (was ~4min), reporting `ok 1 - unlock 0/200` / `ok 2 - add-unlocked 0/200`, 19s and 13s per mode, and collects with `# complete: yes`. bin/ci/run-checks.sh fully green. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01B89nUooZLfThcTA4fSMPGf --- bin/ci/target-git-annex-linkannex.sh | 18 ++++++++++++++++-- evals/matrix.yaml | 10 ++++++---- 2 files changed, 22 insertions(+), 6 deletions(-) diff --git a/bin/ci/target-git-annex-linkannex.sh b/bin/ci/target-git-annex-linkannex.sh index 18eeb75..384a05c 100755 --- a/bin/ci/target-git-annex-linkannex.sh +++ b/bin/ci/target-git-annex-linkannex.sh @@ -27,14 +27,28 @@ # TMPDIR # # env (optional): -# EVAL_UNDER_LINKANNEX_ROUNDS rounds per worker (default 200) +# EVAL_UNDER_LINKANNEX_ROUNDS rounds per worker (default 50) # EVAL_UNDER_LINKANNEX_WORKERS concurrent repos (default 4) +# +# 50 x 4 = 200 rounds per mode. Sized by the slowest backend: BeeGFS +# 8.1.0 took 967s for one mode at 200 x 4 (con/eval-under#11, run +# 36776837242), so two modes did not fit the 1200s budget and the cell +# timed out with only mode 1 reported -- while NFS, ext4 and vfat each +# finished a mode in 100-200s. A quarter of the rounds brings BeeGFS to +# roughly 240s per mode and keeps this target's point: a rate in +# minutes, not a pass/fail after twenty. +# +# The cost is detection power, and it is the reason to raise this rather +# than the timeout when hunting something rare: 200 rounds per mode will +# not reliably show a failure rate below ~1%. For that, run the loop by +# hand with more rounds (bin/ci/linkannex-loop.sh -n) instead of waiting +# on a matrix cell. set -euo pipefail here="$(cd "$(dirname "$0")" && pwd)" -ROUNDS="${EVAL_UNDER_LINKANNEX_ROUNDS:-200}" +ROUNDS="${EVAL_UNDER_LINKANNEX_ROUNDS:-50}" WORKERS="${EVAL_UNDER_LINKANNEX_WORKERS:-4}" cd "$HOME" diff --git a/evals/matrix.yaml b/evals/matrix.yaml index 09503e2..35c0807 100644 --- a/evals/matrix.yaml +++ b/evals/matrix.yaml @@ -96,10 +96,12 @@ targets: - name: git-annex-linkannex label: git-annex linkAnnex loop timeout: 1200 - # 800 rounds x 2 modes of add/commit is metadata-heavy: measured - # ~52MB of an ext4 image's ~83MB usable, and vfat keeps a worktree - # copy of every file because it has no symlinks, so it needs more. - # The image is sparse, so the headroom costs nothing until used. + # Rounds are metadata-heavy: 800 per mode measured ~52MB of an ext4 + # image's ~83MB usable, and vfat keeps a worktree copy of every file + # because it has no symlinks, so it needs more. 200 per mode (the + # current default) needs a quarter of that, but the image is sparse, + # so the headroom costs nothing until used and leaves room to raise + # the rounds by hand without also editing this. loop-size-mb: 256 needs-root: false needs-git-annex: true