Skip to content

nginx-systemd units: fix service start and socket removal, so CI runs test_nginx_dispatch - #225

Merged
ThomasWaldmann merged 2 commits into
borgbackup:mainfrom
ThomasWaldmann:nginx-dispatch-ci-skip
Sep 19, 2026
Merged

ThomasWaldmann merged 2 commits into
borgbackup:mainfrom
ThomasWaldmann:nginx-dispatch-ci-skip

Conversation

@ThomasWaldmann

@ThomasWaldmann ThomasWaldmann commented Sep 19, 2026

Copy link
Copy Markdown
Member

test_nginx_dispatch (and all the generic backend tests for the two nginx-proxied REST backends) were silently skipped in every Linux CI job, e.g. "SKIPPED [1] tests/test_backends.py:984: REST1 and REST2 backends not available" in this run on main. The REST servers behind nginx never worked there, because of two bugs in the contrib/server/nginx-systemd/ unit files.

Reproduced by running the CI setup step verbatim in an Ubuntu 24.04 systemd container.

Causes

Wrong executable path. borgstore@.service had ExecStart=/usr/bin/borgstore-server-rest, but pip install as root puts the script into /usr/local/bin on Debian/Ubuntu. The service failed with status=203/EXEC in a restart loop and the first request hung until the client's 30 s timeout (that is most of the 33 s test collection takes in the CI log).

RuntimeDirectory=borgstore in both units. /run/borgstore/ is shared by the sockets of all instances, but systemd removes a runtime directory when the unit declaring it stops or fails, together with the sockets of all other instances:

connect() to unix:/run/borgstore/repo1.sock failed (2: No such file or directory)

So the failing repo1 service also deleted repo2.sock, and restarting one .socket unit deletes its sibling's socket the same way. This also affects real deployments with more than one repo. Additionally, starting a service chown'ed the directory recursively to borgstore:borgstore, so the sockets lost SocketGroup=www-data - CI worked around that with usermod -aG borgstore www-data.

Changes

  • borgstore@.service: give the executable without a path, systemd then finds it in /usr/local/bin or /usr/bin. No RuntimeDirectory=.
  • borgstore@.socket: no RuntimeDirectory=, systemd creates the parent directory of a ListenStream= socket by itself.
  • README.md: updated to match.
  • CI: create and destroy a store in both repos via nginx at the end of the setup step, so a broken setup fails the job (and shows the existing diagnostics step) instead of skipping the tests. The usermod workaround is removed, the sockets now keep their www-data group.
  • CI: also install the blake3 extra into the system Python the socket-activated servers run in. They compute hashes server-side, so test_hash_blake3 failed on rest1/rest2 as soon as these backends were not skipped any more (first CI run of this PR).

Testing

In a fresh container with the new units and the new step: the step passes with www-data in no extra group, the sockets stay borgstore:www-data across service starts and sibling stops/restarts, test_nginx_dispatch passes and 46 parametrized tests additionally run on the rest1/rest2 backends.

Negative control: main's units with the new step make the step fail (curl exit 28) instead of going green.

CI of this PR: test_nginx_dispatch PASSED in all Linux py3xx jobs, 390 passed / 1 skipped (main: 330 passed / 2 skipped).

🤖 Generated with Claude Code

ThomasWaldmann and others added 2 commits September 20, 2026 00:16
… test_nginx_dispatch

test_nginx_dispatch (and all generic backend tests for the 2 nginx-proxied
REST backends) were silently skipped in all Linux CI jobs, because the REST
servers behind nginx never worked there:

- borgstore@.service had ExecStart=/usr/bin/borgstore-server-rest, but
  "pip install" as root puts the script into /usr/local/bin on Debian/Ubuntu,
  so the service failed with status=203/EXEC in a restart loop and the first
  request hung until the client timed out. Give the executable without a path,
  systemd then finds it in /usr/local/bin or /usr/bin.

- both units declared RuntimeDirectory=borgstore. /run/borgstore/ is shared by
  the sockets of all instances, but systemd removes a runtime directory when
  the unit declaring it stops or fails - together with the sockets of all other
  instances (nginx: "connect() to unix:/run/borgstore/repo1.sock failed (2: No
  such file or directory)"). Also, starting a service chown'ed the directory
  recursively to borgstore:borgstore, so the sockets lost SocketGroup=www-data.
  systemd creates the parent directory of a ListenStream= socket by itself, so
  just do not declare a RuntimeDirectory.

CI:

- create and destroy a store in both repos via nginx at the end of the setup
  step, so a broken setup fails the job instead of skipping the tests.
- do not add www-data to the borgstore group: not needed, the sockets now keep
  their www-data group.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
They run in the system Python and compute the hashes server-side, so
test_hash_blake3 failed on the rest1/rest2 backends as soon as these
backends were not skipped any more.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ThomasWaldmann
ThomasWaldmann merged commit 4eba23c into borgbackup:main Sep 19, 2026
9 checks passed
@ThomasWaldmann
ThomasWaldmann deleted the nginx-dispatch-ci-skip branch September 19, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant