Nextcloud stuck in maintenance mode: nextcloud-app starts before Redis has loaded its AOF

[stephdl]

Steps to reproduce

• Install a Nextcloud instance and let its Redis AOF grow (a nextcloud-redis-data volume with ~20M keys loads in ~14s)
• Make a Nextcloud upgrade pending, as the nightly auto-update does
• Restart the module so all units start together

Expected behavior

nextcloud-app waits for Redis to be usable before running occ upgrade.

Actual behavior

nextcloud-redis.service is Type=forking with a PIDFile, so systemd marks it active as soon as the container forks, not when redis-server has finished loading the AOF. nextcloud-app.service only declares After=nextcloud-redis.service, which orders the start but does not wait for availability.

occ upgrade therefore runs while Redis still answers LOADING Redis is loading the dataset in memory. The schema step succeeds, a repair step throws RedisException, the update fails, and maintenance mode is deliberately kept on. Nothing retries, so the instance stays in maintenance until someone runs occ upgrade by hand.

Sep 03 15:05:51 R2-pve.rocky9-pve2.org systemd[736845]: Started Podman nextcloud-redis.service.   <- systemd marks redis ready: the container forked
Sep 03 15:05:53 R2-pve.rocky9-pve2.org systemd[736845]: Starting Podman nextcloud-app.service...
Sep 03 15:05:53 R2-pve.rocky9-pve2.org nextcloud-app[746340]: Upgrading nextcloud from 33.0.6.2 ...   <- entrypoint starts occ upgrade
Sep 03 15:05:54 R2-pve.rocky9-pve2.org nextcloud-app[746340]: [nextcloud][no app in context][4] {... "Exception":"RedisException","Message":"Could not boot webhook_listeners: LOADING Redis is loading the dataset in memory" ...}
Sep 03 15:05:54 R2-pve.rocky9-pve2.org nextcloud-app[746340]: [nextcloud][no app in context][3] {... "Exception":"RedisException","Message":"Could not boot workflowengine: LOADING Redis is loading the dataset in memory" ...}
Sep 03 15:05:58 R2-pve.rocky9-pve2.org nextcloud-app[746340]: Turned on maintenance mode
Sep 03 15:05:58 R2-pve.rocky9-pve2.org nextcloud-app[746340]: [nextcloud][core][0] {... "message":"starting upgrade from 33.0.6.2 to 33.0.7.1" ...}
Sep 03 15:05:58 R2-pve.rocky9-pve2.org nextcloud-app[746340]: Updated database
Sep 03 15:05:58 R2-pve.rocky9-pve2.org nextcloud-app[746340]: RedisException: LOADING Redis is loading the dataset in memory
Sep 03 15:05:58 R2-pve.rocky9-pve2.org nextcloud-app[746340]: [nextcloud][updater][3] {... "message":"\\OC\\Updater::failure: RedisException: LOADING Redis is loading the dataset in memory" ...}
Sep 03 15:05:58 R2-pve.rocky9-pve2.org nextcloud-app[746340]: Update failed
Sep 03 15:05:58 R2-pve.rocky9-pve2.org nextcloud-app[746340]: Maintenance mode is kept active   <- instance stuck here, nothing retries
Sep 03 15:06:06 R2-pve.rocky9-pve2.org nextcloud-redis[746190]: 1:M 03 Sep 2026 13:06:06.328 * DB loaded from append only file: 14.423 seconds
Sep 03 15:06:06 R2-pve.rocky9-pve2.org nextcloud-redis[746190]: 1:M 03 Sep 2026 13:06:06.328 * Ready to accept connections tcp   <- redis is only usable now, 15s after systemd said it was

systemd[736845], the module user manager, reports the redis unit started at 15:05:51. nextcloud-redis[746190] only accepts connections at 15:06:06, 15 seconds later. Everything nextcloud-app[746340] does in between runs against a Redis that answers LOADING, including the whole occ upgrade. Six RedisException in that window, the last one carrying the update to failure. Long JSON entries are elided at ....

occ status then reports maintenance: true and needsDbUpgrade: true while config/config.php still holds the old version.

nextcloud-db.service already guards against this with a mariadb-admin ping loop in ExecStartPost. The Redis unit has no equivalent.

Components

• ns8-nextcloud 1.7.4, reproduced on Rocky Linux 9.8
• docker.io/redis:7.4.5-alpine, started with --appendonly yes

See also

• Fix: fix(systemd): wait for redis to be ready before starting the app ns8-nextcloud#261
• Detailed reproduction procedure and the control run on the patched image follow in a comment.

[stephdl]

How to reproduce, for anyone interested

Two things have to line up: a Redis load window long enough to matter, and a pending Nextcloud upgrade that fires on app start.

Tested on Rocky Linux 9.8, cluster leader, node 1. Budget about 2.5 GB of free RAM for the largest dataset below.

1. Install an unpatched instance

A stable released tag does not contain the fix:

add-module ghcr.io/nethserver/nextcloud:1.7.4 1

Confirm the units really are unpatched — the redis unit must have no ExecStartPost, and the app unit no Requires=:

grep -c ExecStartPost /home/nextcloud2/.config/systemd/user/nextcloud-redis.service   # expect 0
grep -E '^(Requires|After)=' /home/nextcloud2/.config/systemd/user/nextcloud-app.service

2. Configure the instance

Send the complete payload. config.json is a verbatim copy of what you pass, and setup-smtp reads config['internal_smarthost'] without a guard, so omitting it makes every app start fail with KeyError: 'internal_smarthost' — a dead end unrelated to this bug:

api-cli run module/nextcloud2/configure-module --data - <<'JSON'
{"host":"nextcloud2.example.org","lets_encrypt":false,"domain":"","internal_smarthost":false,
 "is_collabora":false,"collabora_host":"","tls_verify_collabora":true,"password":"Nethesis,1234"}
JSON

3. Widen the Redis load window

DEBUG POPULATE is refused by the image (enable-debug-command is not set), so populate with a Lua script instead, in batches of 2 million:

for b in 0 1 2 3 4 5 6 7 8 9; do
  runagent -m nextcloud2 podman exec nextcloud-redis redis-cli -t 300 \
    eval "for i=1,2000000 do redis.call('set', 'loadtest:'..($b*2000000+i), i) end return 1" 0
done
runagent -m nextcloud2 podman exec nextcloud-redis redis-cli BGREWRITEAOF

Wait for the rewrite to finish, then check the base file and note its size:

runagent -m nextcloud2 podman exec nextcloud-redis sh -c 'ls -la /data/appendonlydir'

Measured on the test node, one key per SET, tiny values:

keys	used_memory	base.rdb	load time	reproduces?
2 M	~150 MB	45 MB	1.3 s	no
8 M	584 MB	183 MB	5.3 s	no
20 M	1.61 GB	468 MB	14.1 s	yes

The window has to cover the whole entrypoint path, not just the app start. At 5.3 s the LOADING exceptions do appear during app bootstrap, but occ upgrade itself only reaches Turned on maintenance mode about 6.5 s after the redis unit starts, by which point Redis is ready and the upgrade succeeds. 20 million keys gives ~14 s and lands the repair step squarely inside the window.

4. Make an upgrade pending

The upstream entrypoint decides whether to upgrade by comparing /var/www/html/version.php (installed code) against the image's /usr/src/nextcloud/version.php. occ upgrade then compares config.php's version against the code. Both files must be lowered, in the same shot, immediately before the restart. Three traps here, each of which cost us a failed attempt:

• occ config:system:set version --value <lower> is refused: Detected downgrade attempt. Edit config.php with sed.
• Lowering only config.php does nothing: the entrypoint sees no version change and never calls occ upgrade.
• Lowering only version.php also does nothing useful: the entrypoint rsyncs the code back to the image version, then occ upgrade finds config.php already current. It also restores version.php, so a second attempt needs the edit again.

For an instance whose image is 33.0.7.1:

runagent -m nextcloud2 podman exec nextcloud-app sh -c \
  "sed -i \"s/'version' => '33.0.7.1'/'version' => '33.0.6.2'/\" /var/www/html/config/config.php
   sed -i 's/array(33,0,7,1)/array(33,0,6,2)/; s/= .33\.0\.7./= '\''33.0.6'\''/' /var/www/html/version.php
   grep -E \"'version'\" /var/www/html/config/config.php; head -3 /var/www/html/version.php"

5. Restart the whole module

Start every unit in one command so systemd applies its own ordering, the way a node reboot or a module update does:

runagent -m nextcloud2 systemctl --user stop nextcloud-notify_push nextcloud-nginx \
  nextcloud-app nextcloud-db nextcloud-redis nextcloud.service
sleep 2
runagent -m nextcloud2 systemctl --user start nextcloud.service nextcloud-db \
  nextcloud-redis nextcloud-app nextcloud-nginx nextcloud-notify_push

6. Expected result: the bug

runagent -m nextcloud2 podman exec --user www-data nextcloud-app php ./occ status

  - installed: true
  - version: 33.0.7.1
  - maintenance: true
  - needsDbUpgrade: true

config/config.php still holds 'version' => '33.0.6.2' while the code is 33.0.7.1 — the exact state reported from the field. The journal trace is the one in the issue description; read it with journalctl _UID=$(id -u nextcloud2).

Control run: same scenario, patched image

Repeat steps 1 to 5 with ghcr.io/nethserver/nextcloud:fix-redis-readiness-gate, same 20 million keys, same version rollback:

Sep 03 15:11:02 R2-pve nextcloud1 systemd: Starting Podman nextcloud-redis.service...
Sep 03 15:11:17 R2-pve nextcloud-redis: DB loaded from append only file: 14.119 seconds
Sep 03 15:11:17 R2-pve nextcloud-redis: Ready to accept connections tcp
Sep 03 15:11:17 R2-pve nextcloud1 systemd: Started Podman nextcloud-redis.service.   <- probe held the unit 14.8s
Sep 03 15:11:17 R2-pve nextcloud1 systemd: Starting Podman nextcloud-app.service...
Sep 03 15:11:18 R2-pve nextcloud-app: Upgrading nextcloud from 33.0.7.1 ...
Sep 03 15:11:23 R2-pve nextcloud-app: Turned on maintenance mode
Sep 03 15:11:23 R2-pve nextcloud-app: Updated database
Sep 03 15:11:44 R2-pve nextcloud-app: Update successful
Sep 03 15:11:44 R2-pve nextcloud-app: Turned off maintenance mode

occ status: maintenance: false, needsDbUpgrade: false. Zero LOADING errors in the whole run.

Cleanup

runagent -m nextcloud2 podman exec nextcloud-redis redis-cli FLUSHALL
runagent -m nextcloud2 podman exec nextcloud-redis redis-cli BGREWRITEAOF   # AOF back to 8 KB
runagent -m nextcloud2 podman exec --user www-data nextcloud-app php ./occ upgrade
runagent -m nextcloud2 podman exec --user www-data nextcloud-app php ./occ maintenance:mode --off

Then remove-module --no-preserve nextcloud2 if the instance is disposable.

Notes for anyone repeating this

Two instances on one 8 GB node is tight at 20 million keys — stop the units of the instance you are not testing before populating Redis. Load time scales with key count, not byte count, so many tiny keys are the cheapest way to buy a long window. And read timings from the journal rather than from wall-clock guesses: Started Podman nextcloud-redis.service versus Ready to accept connections tcp is the single pair of lines that shows the bug, with or without an upgrade in play.

Two side findings, unrelated to the fix but worth recording:

• redis-cli PING is the right probe because PING does not carry the loading command flag: COMMAND INFO ping returns fast only, and a poller sampling every 50 ms during the load window returned LOADING Redis is loading the dataset in memory on every sample, never PONG. Redis error replies do not change redis-cli's exit code (0 on -LOADING, 1 only on connection failure), so a pg_isready-style exit-code probe would not gate this.
• ns8-core's redis-wait-ready answers a different question. It probes ROLE for replication topology on the cluster Redis, with privileged credentials and 3 retries by default, so it is neither reusable from a module pod nor sized for a 14 s load window.

[stephdl]

QA

Image: ghcr.io/nethserver/nextcloud:1.7.5-dev.1

Goal: prove nextcloud-app starts only once both Redis and MariaDB are up, never before.

Needs ~1 GB of free RAM on the node for the test dataset.

1. Install

• add-module ghcr.io/nethserver/nextcloud:1.7.5-dev.1 1, then configure from the UI Settings form
• Set mid to the module id printed by add-module — it is not always nextcloud1:

mid=nextcloud1

2. Make Redis slow to load

Fills Redis with 5M keys and rewrites the AOF in command format, so it takes ~7s to load instead of milliseconds. Takes about a minute.

cat > /tmp/genload.sh <<'EOF'
redis-cli CONFIG SET aof-use-rdb-preamble no
awk 'BEGIN{ v = sprintf("%0100d", 7); for (i = 0; i < 5000000; i++) printf "SET bench:%d %s\r\n", i, v; }' | redis-cli --pipe
redis-cli BGREWRITEAOF
EOF
runagent -m $mid podman exec -i nextcloud-redis sh < /tmp/genload.sh
sleep 30
runagent -m $mid podman exec nextcloud-redis du -sh /data/appendonlydir

• AOF is ~670 MB. If it is only a few KB, the rewrite is not finished — wait and check again.

3. Restart

Stop all units in one command — stopping nextcloud.service alone leaves the others failed. Mark the time first, so the journal shows only this restart.

M=$(date +%H:%M:%S)
runagent -m $mid systemctl --user stop nextcloud-app.service nextcloud-nginx.service \
  nextcloud-notify_push.service nextcloud-db.service nextcloud-redis.service nextcloud.service
runagent -m $mid systemctl --user start nextcloud-app.service nextcloud-nginx.service \
  nextcloud-notify_push.service

4. Read the journal

journalctl _UID=$(id -u $mid) --no-pager -o short-precise --since "$M" \
  | grep -E "DB loaded from append only|ready for connections|Ready to accept connections|Started Podman nextcloud-(db|redis)|Starting Podman nextcloud-app"

• DB loaded from append only file: N seconds with N above 1 — otherwise step 2 failed and the rest proves nothing
• mariadbd: ready for connections. comes before Started Podman nextcloud-db.service.
• Ready to accept connections tcp comes before Started Podman nextcloud-redis.service.
• Starting Podman nextcloud-app.service... comes after both Started lines

nextcloud-db and nextcloud-redis start in parallel, so their order relative to each other does not matter. What matters is that each reports itself ready before systemd marks it started, and that the app starts after both.

Reference run on 1.7.5-dev.1:

12:35:25.269 nextcloud-db:    mariadbd: ready for connections.
12:35:26.270 systemd:         Started Podman nextcloud-db.service.
12:35:32.548 nextcloud-redis: DB loaded from append only file: 7.489 seconds
12:35:32.548 nextcloud-redis: Ready to accept connections tcp
12:35:33.207 systemd:         Started Podman nextcloud-redis.service.
12:35:33.223 systemd:         Starting Podman nextcloud-app.service...

MariaDB was ready at 12:35:26 but the app waited until 12:35:33, held back by Redis.

5. Counter-test (optional)

Shows what 1.7.4 does. Remove the probe, repeat 3 and 4, restore it.

runagent -m $mid sh -lc 'cp $AGENT_INSTALL_DIR/systemd/user/nextcloud-redis.service /tmp/bak; \
  sed -i "/^ExecStartPost=/d" $AGENT_INSTALL_DIR/systemd/user/nextcloud-redis.service; systemctl --user daemon-reload'

• The order inverts — the app is fully started before Redis can answer:

12:35:57.047 systemd:         Started Podman nextcloud-redis.service.
12:35:58.283 systemd:         Starting Podman nextcloud-app.service...
12:36:00.543 systemd:         Started Podman nextcloud-app.service.
12:36:04.539 nextcloud-redis: DB loaded from append only file: 7.488 seconds
12:36:04.539 nextcloud-redis: Ready to accept connections tcp

• Restore: runagent -m $mid sh -lc 'cp /tmp/bak $AGENT_INSTALL_DIR/systemd/user/nextcloud-redis.service; systemctl --user daemon-reload'

Cleanup

• remove-module --no-preserve $mid