Add an sshfs backend to eval-under

GOTCHAS.md

@@ -75,6 +75,117 @@ So `--no-root-squash` (env: `EVAL_UNDER_NFS_NO_ROOT_SQUASH`) exports with
 user would. The loop and BeeGFS backends already run the wrapped command
 as root, so the flag is a no-op there.
 
+### sshfs (`bin/eval-under-sshfs`)
+
+A throwaway sshd is started on the first free port at or above 2222 --
+its own host key, its own `authorized_keys`, its own pid file, all inside
+the run's scratch directory -- and a fresh backing directory is
+sshfs-mounted back over it. The system sshd is not used and
+`~/.ssh/authorized_keys` is never written to. With `--host` it mounts a
+real remote instead, using the caller's ssh config.
+
+| Knob | Value | Why |
+| --- | --- | --- |
+| Port | first free `>= 2222` | Two runs at once (a reproduction while a suite is going, two CI cells on one runner) must not collide. `--port` pins it. |
+| `-o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3` | always | A dropped connection should fail the command, not wedge it forever. |
+| `-o cache=no` | only with `--no-cache` | sshfs caches attributes by default, which hides stale-stat behaviour. This turns off sshfs's own cache, not all caching -- the kernel's 1-second attribute timeout still applies, and the stale-size effect above survives it. On is what users actually have. |
+| `-o workaround=rename` | only with `--workaround rename` | Makes sshfs emulate rename-over-existing by unlinking first -- non-atomic, which is what SFTP servers without the POSIX-rename extension force. |
+| Mount/command user | the invoking user | A FUSE mount belongs to whoever ran `sshfs`; root cannot read it without `allow_other`. Same treatment `eval-under-nfs` gives `root_squash`. |
+
+**Hardlinks exist but are not observable, and that is the whole story
+for git-annex.** `ln a b` succeeds over SFTP, and then `a` and `b` report
+*different* inode numbers and `nlink=1` each:
+
+```
+ext4     name=a ino=1884275 nlink=2      name=b ino=1884275 nlink=2
+sshfs    name=a ino=3       nlink=1      name=b ino=4       nlink=1
+```
+
+This is not a misconfiguration, and the link is not fake: in the backing
+directory on the server both names really do share one inode with
+`nlink=2`. What SFTP cannot carry is *identity* -- its attribute record
+has neither an inode number nor a link count -- so sshfs synthesises an
+`st_ino` per path and reports `nlink=1` for everything. There is no knob
+for it: `use_ino` (removed in libfuse 3) only ever passed through inode
+numbers that a filesystem supplies, and sshfs has none to supply, so it
+would not have helped under FUSE2 either; sshfs 3.7 offers only
+`disable_hardlink`, which makes `link()` fail outright.
+
+Worse than invisible, and worth knowing when a report mentions truncated
+files: immediately after writing one name, the *other* name still reads
+back with size 0 through the mount -- with `-o cache=no` as well. The
+capability probe reports the inode half as
+`hardlink-same-inode=no` / `hardlink-nlink=no` while `hardlink=yes` --
+which is exactly why those two checks exist. `hardlink` alone called
+sshfs healthy.
+
+What it costs, measured with git-annex 10.20240129:
+
+- `git annex add` on a **locked** branch: fine. The file becomes a
+  symlink into `.git/annex/objects`, and symlinks work.
+- **Any unlocked `git annex add`: fails** -- `foo failed to link to
+  annex`. add hardlinks the content into the annex and then verifies the
+  link, and the verification cannot succeed on a filesystem where the
+  link is invisible. This is not limited to an adjusted branch: a plain
+  v10 repo fails identically with `annex.addunlocked=true` in git
+  config, set through `git annex config`, or passed as `-c`.
+- `git add` through git's own filter (with `annex.largefiles` matching):
+  **fine** -- the pointer is committed and `git annex fsck` is clean.
+  So is `git annex unlock` of an already-committed file.
+- `git clone` of a local repo: **fails** --
+  `fatal: hardlink different from source at '...'`. git's local-clone
+  path hardlinks objects and runs the same check.
+
+So the boundary is not locked-vs-unlocked, and not adjusted-vs-plain:
+it is **who does the ingest**. git-annex hardlinking content into the
+annex fails; git's filter writing a pointer does not.
+
+That distinction is easy to get backwards from the test suite alone,
+because `git annex test` shows `Repo Tests v10 unlocked` **green** and
+only `v10 adjusted unlocked branch` red (11 of 12 in its Init Tests
+group, once `add` fails). The green group is not evidence that unlocked
+repos are fine here -- the suite's unlocked mode ingests with `git add`.
+An earlier revision of this file drew exactly that inference and was
+wrong.
+
+It matters for DataLad, which calls `git annex add`: plain v10 unlocked
+repos break for it too, not just adjusted ones.
+
+**`git annex test` wedges partway through, reproducibly.** Both
+full-suite runs stopped at the same place -- `Remote Tests / unavailable
+remote / removeKey` -- and sat there until killed (15+ minutes on the
+second). On ext4 that same test takes **0.02s and passes**, and the whole
+suite finishes in 1m21s.
+
+It is not an I/O hang. While wedged:
+
+- the mount stays responsive (`ls` returns immediately),
+- git-annex holds **no open files on the mount and no sockets**,
+- its threads sit in `futex_do_wait` / `ep_poll`, with no child
+  processes outstanding.
+
+That is a process waiting on something internal, not one blocked on the
+filesystem. Note also that git-annex sets `annex.sshcaching = false` here
+on its own, because ssh control sockets need unix sockets and this mount
+has none -- so the run is already on a different code path from a normal
+one.
+
+Two caveats before anyone reports this upstream: the same test passes in
+seconds when selected on its own with `-p '/unavailable remote/'`, so it
+needs the full-suite context; and this was git-annex 10.20240129 from
+Ubuntu 24.04, not a daily build. Re-run it through the reproduce
+workflow, which installs the daily build from con/git-annex, before
+filing anything.
+
+**Timestamps are quantised to the second.** Five files created back to
+back get one or two distinct mtimes, where every other filesystem
+measured gives five. Nothing else in the matrix has a clock this coarse,
+which makes sshfs the row that exercises git's racy-timestamp handling.
+
+**No fifos, no unix sockets.** git-annex says so itself at init
+("Detected a filesystem without fifo support") and adapts. Worth knowing
+before reading it as a failure.
+
 ### BeeGFS (`bin/eval-under-beegfs`)
 
 A containerised cluster (`fixtures/beegfs/docker-compose-v{7,8}.yml`) plus
@@ -205,6 +316,104 @@ See: <https://git-annex.branchable.com/bugs/35_failed_tests_on_beegfs/>, [BeeGFS
 
 See: [Loop git-annex cells: annex.diskreserve](#loop-git-annex-cells-annexdiskreserve)
 
+<a id="sshfs-git-local-clone-hardlink"></a>
+### `sshfs-git-local-clone-hardlink`: local `git clone` verifies its hardlinks, and sshfs synthesises st_ino
+
+**Cells:** `sshfs-git` \
+**Tags:** `fs-divergence` \
+**Tests:** `t0001-init.sh#37`, `t0003-attributes.sh#24-25,29,32-34`, `t0021-conversion.sh#28-30`, `t0033-safe-directory.sh#16`, `t0035-safe-bare-repository.sh#1,13`, `t0410-partial-clone.sh#34,38`, `t0610-reftable-basics.sh#26-28`, `t1013-read-tree-submodule.sh#1-8,10-15,18-28,30-48,51-60,65-68`, `t1060-object-corruption.sh#12`, `t1091-sparse-checkout-builtin.sh#31-32,36-38,49,77`, `t1350-config-hooks-path.sh#4`, `t1423-ref-backend.sh#36`, `t1460-refs-migrate.sh#9,24`, `t1500-rev-parse.sh#77`, `t1507-rev-parse-upstream.sh#1-7,9-11,13-14,17-18,21,23-27`, `t1600-index.sh#6`
+
+SFTP's `ATTRS` carries no inode number, so sshfs synthesises `st_ino`
+per path. `git clone <local path>` hardlinks each object and then
+compares `st_mode`/`st_ino`/`st_dev`/`st_size`/`st_uid`/`st_gid`
+against the source (`builtin/clone.c`), so the check fails and the
+clone dies with `fatal: hardlink different from source`. **Plain
+`git clone` of a local path does not work on sshfs at all.**
+
+Each script here clones, or adds a submodule (which clones), in its
+setup, so one failed setup cascades through the script; the scripts
+were attributed by that message appearing in their own logs.
+`git clone --no-hardlinks`, `git clone file://...` and mounting with
+`-o disable_hardlink` all work -- with the option, sshfs fails
+`link()` with `EPERM` instead of pretending, and git falls back to
+copying.
+
+Test ids seeded from run 36476334300 (git v2.55.0), 110 assertions
+across 16 scripts.
+
+See: [sshfs (`bin/eval-under-sshfs`)](#sshfs-bineval-under-sshfs)
+
+<a id="sshfs-git-no-unix-sockets"></a>
+### `sshfs-git-no-unix-sockets`: git's IPC and credential-cache tests need Unix sockets on the work tree
+
+**Cells:** `sshfs-git` \
+**Tags:** `fs-limitation` \
+**Tests:** `t0052-simple-ipc.sh#1-9`, `t0301-credential-cache.sh#2-3,7-8,10-11,13-23,25-26,28,30,32-33,37-38,40-41,43-52`
+
+sshfs has no Unix sockets: `bind()` on the mount fails with
+`Operation not permitted`, so the credential-cache daemon never
+starts (`unable to bind to .../credential/socket`) and simple-ipc
+finds `no server listening`. `unix-socket=no` in
+`bin/ci/fs-capabilities.sh` predicts both. The same limitation is
+recorded for vfat as `vfat-git-no-unix-sockets`.
+
+See: [sshfs (`bin/eval-under-sshfs`)](#sshfs-bineval-under-sshfs)
+
+<a id="sshfs-git-untriaged"></a>
+### `sshfs-git-untriaged`: remaining git failures on sshfs, not yet attributed
+
+**Cells:** `sshfs-git` \
+**Tags:** `needs-triage` \
+**Tests:** `t0003-attributes.sh#41,48`, `t0017-env-helper.sh#4`, `t0040-parse-options.sh#37`, `t0061-run-command.sh#6,18`, `t0302-credential-store.sh#57`, `t0450-txt-doc-vs-help.sh#131,647,797`, `t0610-reftable-basics.sh#61`, `t1091-sparse-checkout-builtin.sh#21,43,48`, `t1092-sparse-checkout-compatibility.sh#55`, `t1300-config.sh#194,197,237,285,494`, `t1403-show-ref.sh#9`, `t1430-bad-ref-name.sh#26`, `t1450-fsck.sh#36`, `t1461-refs-list.sh#415`, `t1503-rev-parse-verify.sh#4`, `t1700-split-index.sh#10-12,14-15`
+
+**These are flaky, not fixed divergences, and this entry cannot
+gate the cell.** Two mechanisms above are deterministic; this
+residue is not. Measured three ways:
+
+- The same cell on two CI runs of near-identical code
+  (36476334300, then 36485450259) reported 12 and 18 residual
+  failures with **no overlap**: every id the first run flagged
+  passed in the second, and vice versa. The two mechanism entries
+  meanwhile reproduced exactly both times, 110 and 46.
+- Locally, six runs of `t0003 t0017 t0040 t1700` under sshfs:
+  `t0003`'s six hardlink assertions failed in all six runs, while
+  `t0017#4` failed in one and `t0040#37`, `t1700#10-15` and
+  `t0003#41`/`#48` in none -- although CI has flagged each of them.
+- `prove --jobs 1` is no cleaner than `--jobs 4`, and 14 runs of
+  `t0017` alone were all clean, so it takes the fuller suite's
+  concurrent load to show up at all.
+
+Some of these tests touch no filesystem semantics whatsoever --
+`t0040#37` "OPT_CALLBACK() and OPT_BIT() work" and `t0017#4`
+"test-tool env-helper --type=ulong" parse arguments and
+environment variables, and `t0450` compares documentation against
+`-h` output. What they do share is capturing output into `>out` /
+`2>err` inside the trash directory on the mount and then grepping
+it, which points at the mount losing or delaying writes under
+concurrent load rather than at any semantic divergence.
+
+So a `script#N` list is the wrong instrument here: each run draws a
+different sample, and pinning one run's sample is what made this
+cell report `failing-new` twice.
+
+Mounting with `-o cache=no` (`--no-cache`) was measured, not
+guessed: five runs of `t0*.sh` at `--jobs 4`, counting failures
+beyond the 64 this subset's two mechanisms own. Baseline drew 4, 8
+and 8; `cache=no` drew 1 and 2, for about 7% more wall clock (107s
+-> 114s). So the cache is implicated and `cache=no` is worth having
+as a mitigation -- but it does not fix the gate, because
+`known_issues.py` sets `failing-new` on the *first* uncovered
+failure and has no tolerance for a flake budget. A rate of one per
+run still fails the cell most runs.
+
+Which leaves a decision rather than a patch: cover the whole cell
+with `tests: ["*"]` as `vfat-pjdfstest` does (green, but it masks
+the 110 and 46 findings and any future regression), make the cell
+non-gating, or give the verdict machinery a flake budget. That last
+one belongs in the machinery, not in this backend's PR.
+
+See: [sshfs (`bin/eval-under-sshfs`)](#sshfs-bineval-under-sshfs)
+
 <!-- END KNOWN ISSUES -->
 
 ## Root-cause notes

README.md

@@ -28,6 +28,7 @@ itself is both backend- and suite-agnostic: new filesystems drop in as
 | BeeGFS 7.4.6 | [![BeeGFS 7.4.6 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-git-annex.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-git-annex) | [![BeeGFS 7.4.6 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-git.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-git) | [![BeeGFS 7.4.6 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-stress-ng.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-stress-ng) | [![BeeGFS 7.4.6 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-7.4.6-pjdfstest.svg)](https://con.github.io/eval-under/#beegfs-7.4.6-pjdfstest) |
 | BeeGFS 8.1.0 | [![BeeGFS 8.1.0 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-git-annex.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-git-annex) | [![BeeGFS 8.1.0 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-git.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-git) | [![BeeGFS 8.1.0 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-stress-ng.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-stress-ng) | [![BeeGFS 8.1.0 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/beegfs-8.1.0-pjdfstest.svg)](https://con.github.io/eval-under/#beegfs-8.1.0-pjdfstest) |
 | NFS (localhost) | [![NFS (localhost) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git-annex.svg)](https://con.github.io/eval-under/#nfs-git-annex) | [![NFS (localhost) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-git.svg)](https://con.github.io/eval-under/#nfs-git) | [![NFS (localhost) / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-stress-ng.svg)](https://con.github.io/eval-under/#nfs-stress-ng) | [![NFS (localhost) / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/nfs-pjdfstest.svg)](https://con.github.io/eval-under/#nfs-pjdfstest) |
+| sshfs (loopback) | [![sshfs (loopback) / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/sshfs-git-annex.svg)](https://con.github.io/eval-under/#sshfs-git-annex) | [![sshfs (loopback) / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/sshfs-git.svg)](https://con.github.io/eval-under/#sshfs-git) | n/a | n/a |
 | Loop vfat | [![Loop vfat / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git-annex.svg)](https://con.github.io/eval-under/#loop-vfat-git-annex) | [![Loop vfat / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-git.svg)](https://con.github.io/eval-under/#loop-vfat-git) | [![Loop vfat / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-stress-ng.svg)](https://con.github.io/eval-under/#loop-vfat-stress-ng) | [![Loop vfat / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-vfat-pjdfstest.svg)](https://con.github.io/eval-under/#loop-vfat-pjdfstest) |
 | Loop ext4 | [![Loop ext4 / git-annex test](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git-annex.svg)](https://con.github.io/eval-under/#loop-ext4-git-annex) | [![Loop ext4 / git testsuite](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-git.svg)](https://con.github.io/eval-under/#loop-ext4-git) | [![Loop ext4 / stress-ng](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-stress-ng.svg)](https://con.github.io/eval-under/#loop-ext4-stress-ng) | [![Loop ext4 / pjdfstest](https://raw.githubusercontent.com/con/eval-under/gh-pages/badges/loop-ext4-pjdfstest.svg)](https://con.github.io/eval-under/#loop-ext4-pjdfstest) |
 <!-- END CI MATRIX -->
@@ -161,6 +162,14 @@ sudo bin/eval-under loop --fs xfs --size 200 --set-home -- \
 # the fsync-heavy slow path)
 sudo bin/eval-under nfs --set-home -- bash -c 'cd "$HOME" && git annex test'
 
+# Under sshfs, with the attribute cache off
+sudo bin/eval-under sshfs --no-cache --set-home -- \
+  bash -c 'cd "$HOME" && git annex test'
+
+# ...or against the reporter's own server, with their mount options
+sudo bin/eval-under sshfs --host store.example.org --remote-dir /data/scratch \
+  --workaround rename --set-home -- git annex fsck
+
 # Skip teardown to poke around after a failure
 sudo bin/eval-under beegfs --set-home --keep -- some-failing-command
 
@@ -204,6 +213,7 @@ into `bin/eval-under`, bumped with each release tag.
 | `bin/eval-under`                         | Dispatcher: routes to `bin/eval-under-<backend>`                                   |
 | `bin/eval-under-beegfs`                  | BeeGFS backend (containerised cluster + kernel client mount)                       |
 | `bin/eval-under-nfs`                     | NFS backend (localhost loopback export)                                            |
+| `bin/eval-under-sshfs`                   | sshfs backend (throwaway loopback sshd, or a remote you name)                       |
 | `bin/eval-under-loop`                    | Loop-device backend (dd + losetup + mkfs.<fs> + mount)                             |
 | `fixtures/beegfs/docker-compose-v7.yml`  | BeeGFS v7 test cluster (mgmtd + meta + storage), `network_mode: host`              |
 | `fixtures/beegfs/docker-compose-v8.yml`  | Same, for BeeGFS v8.x (different mgmtd command style / gRPC control plane)         |