Repository navigation
fix: only wait on the machines an update brings up - #66
Open
riadelhajjaji-scality wants to merge 1 commit into
Open
riadelhajjaji-scality wants to merge 1 commit into
riadelhajjaji-scality wants to merge 1 commit into
Conversation
Updating a platform waited again on every machine, the ones left alone included. On a Rocky machine where anything installed a package since it booted, `cloud-init status` then failed for the SSH user with a PermissionError on /run/cloud-init/cloud.cfg, and the update failed with "booted badly, cloud-init exited 1". The cloud-init package keeps /run/cloud-init to root through a tmpfiles rule, and the systemd package runs every tmpfiles rule again whenever a package ships one, so cloud-init can no longer read its own runtime config for anybody else (canonical/cloud-init#6680). Freshly booted machines do not show it, which is why spawns went through. Asking as root is no way out: on ARTESCA the SSH user is not allowed to run cloud-init through sudo once the product is installed. Each machine now has a wait of its own, triggered by that machine only. An update waits on the machines it brought up, which are fresh, and leaves the others alone. The waits are named after the machine, so an update of a platform spawned before this change waits once more on every machine. Refs: MK8S-519
riadelhajjaji-scality
requested review from
TeddyAndrieux,
anthony-treuillier-scality and
m4nch0t
and removed request for
m4nch0t
October 29, 2026 11:08
riadelhajjaji-scality
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Updating a platform fails at the
wait-for-bootstep on machines that were left alone:node-2 booted badly, cloud-init exited 1
PermissionError: [Errno 13] Permission denied: '/run/cloud-init/cloud.cfg'
Example: https://github.com/scality/metalk8s/actions/runs/40701329682 (
bootstrap-restore, red on MetalK8s 134.0 and 136.0 since it moved to platform-spawner)wait-for-bootwas a single command checking every machine, re-run whenever any instance changed. Removing node-1 therefore asked cloud-init again on the bastion, node-2 and node-3, which had been in use for an hour.On Rocky 9, cloud-init ships a tmpfiles rule keeping
/run/cloud-initto root (0700), and systemd runs every tmpfiles rule again whenever a package ships one. Once MetalK8s has installed its packages,cloud-init statuscan no longer read its runtime config as the SSH user (canonical/cloud-init#6680). A freshly booted machine does not show it, which is why spawns always went through. The bastion, where nothing is installed, passes too.Running it through
sudois no fix: on ARTESCA,artesca-osis not allowed to run cloud-init through sudo once the product is installed, and ARTESCA spawns from snapshots of installed clusters.What it does
Gives each machine its own
wait-for-boot-<machine>command, triggered by that machine's instance only. An update now waits on the machines it brings up, which are fresh, and leaves the others alone. The check itself is unchanged and still runs as the SSH user. The bastion setup waits on all of them as before.Since the waits are renamed, updating a platform spawned before this change waits once more on every machine. Only a MetalK8s nightly running across the release would see it.
Once merged, a GA patch release (
v3.2.3) movesv3so that callers get the fix without changing anything on their side.This PR overrites the closed #65 .
Closes MK8S-519