Filesystem probing: what breaks git-annex, and what can be stood up here

[yarikoptic-gitmate]

The sshfs backend has moved out to #13 so this stays reviewable. What's left is the probing / capability / survey work: 1727 insertions across 17 files, no new matrix cells and no new badges.

Reconnaissance: what can we even stand up?

• bin/ci/probe-backend.sh — tries to bootstrap a candidate filesystem on a GitHub-hosted runner and reports what it got: 24 candidates (gfs2, ocfs2, f2fs, exfat, ntfs3, udf, nilfs2, bcachefs, zfs, overlay, glusterfs, cephfs, cifs, sshfs, gocryptfs, encfs, ecryptfs, s3-rclone, lustre-client, openafs, …). "NOT BOOTSTRAPPABLE" is a finding, not a failure, so a verdict exits 0 while a genuine script error does not.
• .github/workflows/probe-filesystems.yaml + bin/ci/render-probe-table.sh — runs the probes (dispatch, plus master-only push) and rolls the summaries into one table.

Capability profiling: why it broke

bin/ci/fs-capabilities.sh emits key=value lines for the dimensions git-annex and DataLad actually trip over — sqlite-wal, fcntl-lock, flock, link-eexist, rename-over, unlink-open, symlink, hardlink, hardlink-same-inode, hardlink-nlink, exec-bit, perm-bits, xattr-user, case-sensitive, special-chars, mtime-distinct-of-5, … Ten seconds, and it usually explains a suite result before the suite is worth starting: hardlink-same-inode=no predicts the failed to link to annex failure that takes a 20-minute git annex test to reach.

It runs under any backend as the on-demand capabilities target (bin/ci/target-capabilities.sh), which is deliberately not a matrix column — update-status.py and render-report.py filter on-demand targets so it publishes no permanently-unknown badge and no never-fillable column.

On-demand reproduction

.github/workflows/reproduce.yaml — the support entry point: someone reports a problem, and you want that combination running now with their mount options, without waiting for the schedule or adding a cell nobody watches. Every field is a dispatch input; no badge, no cron. It does not touch the published status, so a reproduction can never rewrite what the README shows for master.

Survey

FILESYSTEMS.md — three parts: filesystems with documented git-annex / DataLad breakage, then what the probes actually measured here, then feasibility and recommendations. The second half is measured, not guessed.

GOTCHAS.md — the loop backend's new flavours: single-node locking for gfs2/ocfs2, image size floors for journal overhead, and how far "cluster filesystem without a cluster" can be trusted.

Framework

bin/eval-under-loop gains gfs2, ocfs2, f2fs, exfat and nilfs2, a --mkfs-opts override, per-filesystem size floors, and integer validation of --size (a non-numeric value previously reached dd count=40M, i.e. 40 TiB).

CI

Every one of the 20 matrix cells matches master's published baseline cell-for-cell — 10 red by design, 10 green — plus matrix, checks (shellcheck + 28 bats) and codespell green, with publish correctly skipped.

🤖 Generated with Claude Code

https://claude.ai/code/session_01E1fDiGVWKywpbhVps5hzQK

[yarikoptic-gitmate]

CI on d0c387e: 10 red, and all 10 are red on master too

test run #9 finished 10/20. Comparing cell-by-cell against the published status for master (eb99fc6, run #7, from gh-pages/status.json) rather than at run level, because a run-level "failure" is the normal state of this matrix and says nothing:

Cell	master eb99fc6	this PR d0c387e
beegfs-7.4.6-git-annex	❌	❌
beegfs-7.4.6-pjdfstest	❌	❌
beegfs-8.1.0-git-annex	❌	❌
beegfs-8.1.0-pjdfstest	❌	❌
loop-ext4-git-annex	❌	❌
loop-vfat-git	❌	❌
loop-vfat-git-annex	❌	❌
loop-vfat-pjdfstest	❌	❌
loop-vfat-stress-ng	❌	❌
nfs-pjdfstest	❌	❌
beegfs-7.4.6-git, beegfs-7.4.6-stress-ng	✅	✅
beegfs-8.1.0-git, beegfs-8.1.0-stress-ng	✅	✅
loop-ext4-git, loop-ext4-pjdfstest, loop-ext4-stress-ng	✅	✅
nfs-git, nfs-git-annex, nfs-stress-ng	✅	✅

An exact match in both directions: no cell that is green on master is red here, and this PR turns nothing red that was not already red. matrix and codespell pass; publish is skipped, as intended on a pull request.

Most of the red is documented in GOTCHAS.md and is the point of the row rather than a defect — all four Loop vfat cells (no symlinks, no ownership, and a filename charset that git's own suite deliberately violates), and the pjdfstest cells. Two are worth chasing on their own, and neither belongs to this PR:

• loop-ext4-git-annex — ext4 is the control row, so GOTCHAS.md already calls this a real bug rather than a filesystem property. It ran with a 100 MB image here (loop3: detected capacity change from 0 to 204800 in dmesg), i.e. the per-filesystem size floor added in this PR is inert for ext4 and did not change what the cell does.
• The BeeGFS git-annex pair, red on master since before this branch.

I have not spent a re-run on any of these: comparing against master's own per-cell status is stronger evidence than a second attempt at a cell that is documented as reliably red.

One correction to the PR description above, which was written before this branch was merged with master: _test-under.yaml is not "updated" — master deleted it when the per-cell dispatchers were replaced by the single matrix workflow, so reproduce.yaml now carries those steps directly. Likewise capabilities is defined in .github/matrix.yaml (flagged on-demand: true), not in matrix.sh; the grid stays at exactly 20 cells.

---

Generated by Claude Code

[yarikoptic-gitmate]

Updated after review: CI on b917d58 matches master cell-for-cell

Superseding my earlier comparison, which cited a pre-review head and a master baseline that has since moved. Three things changed under it: master merged #10 (the per-run mountpoint rework), this branch was merged with it, and two independent reviews found real defects that are now fixed.

Baseline refreshed, and it did not move

master is now aa98679; the published status for it (run 19, gh-pages/status.json) is red/green in exactly the same 10/10 split as eb99fc6 was a month ago. Worth stating because #10 rewrote every backend's mountpoint handling and could plausibly have shifted a cell — it didn't.

This PR, run 24 on b917d58

Cell	master aa98679	this PR b917d58
beegfs-7.4.6-git-annex, beegfs-7.4.6-pjdfstest	❌	❌
beegfs-8.1.0-git-annex, beegfs-8.1.0-pjdfstest	❌	❌
loop-ext4-git-annex	❌	❌
loop-vfat-git, loop-vfat-git-annex, loop-vfat-pjdfstest, loop-vfat-stress-ng	❌	❌
nfs-pjdfstest	❌	❌
beegfs-7.4.6-git, beegfs-7.4.6-stress-ng	✅	✅
beegfs-8.1.0-git, beegfs-8.1.0-stress-ng	✅	✅
loop-ext4-git, loop-ext4-pjdfstest, loop-ext4-stress-ng	✅	✅
nfs-git, nfs-git-annex, nfs-stress-ng	✅	✅

20/20 identical in both directions: nothing green on master is red here, and nothing red here was green there. matrix and codespell pass; publish is skipped, as intended on a pull request. No re-run spent — comparing against master's own per-cell status is better evidence than a second attempt at a cell documented as reliably red.

The reds are the point of those rows, not defects in them: all four Loop vfat cells (no symlinks, no ownership, and a filename charset git's own suite deliberately violates) plus the pjdfstest cells, all documented in GOTCHAS.md. Two are worth chasing separately and neither belongs to this PR — loop-ext4-git-annex (ext4 is the control row, so GOTCHAS.md already calls it a real bug) and the BeeGFS git-annex pair, red on master since before this branch.

What the reviews changed

Two things are worth flagging to a reviewer of this PR specifically, because they were wrong in ways CI could not have caught:

1. The sshfs backend had never worked in the mode it documents. Under sudo bin/eval-under sshfs … — its own --help example, and what bin/ci/run-under.sh uses — it exited 127 on "${SUDO[@]}" -u, because that array is empty exactly when we are already root and need the privilege drop. Every earlier test had run as root directly, which takes the other branch. Fixed and re-verified against a real unprivileged invoker.
2. A false claim at the centre of the documentation. GOTCHAS.md said v10 unlocked repos were fine and only the adjusted unlocked branch broke. Re-measured: a plain v10 repo with annex.addunlocked=true fails identically. The boundary is who ingests — git-annex hardlinking into the annex fails, git's filter writing a pointer does not. The green Repo Tests v10 unlocked group in git-annex's suite is green because that mode adds with git add; generalising from it was the error. This matters for DataLad, which calls git annex add.

Also fixed: the publish side would have written 5 permanently-unknown *-capabilities badges and a never-fillable column, contradicting the "no badge" promise for on-demand targets (now 20 cells / 21 badges, measured); reproduce.yaml offered sshfs × root-requiring targets that cannot work; a --size 40M reached dd count=40M (40 TiB); fs-capabilities.sh reported a missing python3 as missing filesystem features, sqlite-wal included; and the cifs probe left a mode-777 force user = root Samba share plus a root SMB password behind, reverting neither.

Full detail is in the two commit messages (e46b3ce, b917d58).

---

Generated by Claude Code