Skip to content

feat: add armhf (32-bit ARM) architecture support [ISD-5856]#233

Merged
florentianayuwono merged 28 commits into
mainfrom
feat/armhf-arch-support-isd-5856
Jul 11, 2026
Merged

feat: add armhf (32-bit ARM) architecture support [ISD-5856]#233
florentianayuwono merged 28 commits into
mainfrom
feat/armhf-arch-support-isd-5856

Conversation

@florentianayuwono

@florentianayuwono florentianayuwono commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Overview

Adds armhf (32-bit ARM / linux-arm) as a buildable image architecture so the charm can
produce GitHub Actions runner images that run the 32-bit runner agent on arm64 (aarch64)
hardware — no emulation.

The key design point: an armhf image is an arm64 boot image (64-bit kernel) that carries a
32-bit linux-arm runner payload plus an armhf multiarch userland
. The 32-bit runner executes
via the host's native AArch32 support (AArch32 at EL0). A native armhf image (32-bit kernel) does
not boot on the aarch64 virt machine, which is why the boot/glance architecture is aarch64
while the runner payload is 32-bit arm.

The runner binary is the linux-arm build published by the
canonical/github-actions-runner fork
(FORK_RUNNER_BINARY_REPO): actions-runner-linux-arm-<version>.tar.gz (arm32 actions wired onto
Node.js 22, from fork PRs #155/#169/#171).

How an armhf image is built

  1. Boot arch ≠ payload arch. Arch.ARM.to_openstack()aarch64 and the base cloud image
    download arch (_get_supported_runner_arch) → arm64, so the base image and the resulting
    snapshot schedule and boot on arm64 hosts. RUNNER_ARCH renders to arm, so cloud-init
    downloads actions-runner-linux-arm-<version>.tar.gz.
  2. armhf multiarch userland (enable_armhf_multiarch in cloud-init.sh.j2, armhf-only):
    • dpkg --add-architecture armhf.
    • Pin the stock ubuntu.sources to the native arch (Architectures: <native>), because the
      arm64 mirrors only serve the native arch.
    • Add an armhf-ports.sources pointing at ports.ubuntu.com (armhf is a "ports" architecture),
      so the 32-bit runtime dependencies resolve.
    • Write /etc/cloud/cloud.cfg.d/99-armhf-preserve-apt-sources.cfg (preserve_sources_list: true)
      so the pinned/ports sources survive on the runner VM (see "Why preserve_sources_list").
  3. armhf runtime + build deps (ARM_ADDITIONAL_APT_PACKAGES, installed only for Arch.ARM):
    libc6:armhf (provides ld-linux-armhf.so.3), libatomic1:armhf, rustup, docker-buildx,
    plus the release-specific armhf libicu (libicu74:armhf on noble, libicu78:armhf on
    resolute — ARM_LIBICU_APT_PACKAGE_BY_BASE). The distro cargo/rustc are dropped
    (ARM_EXCLUDED_DEFAULT_APT_PACKAGES) because rustup provides them and conflicts otherwise.
    libicu + libatomic are the runner's bundled .NET runtime dependencies.
  4. Rust toolchain: cloud-init runs sudo -u ubuntu rustup default stable (armhf-only; a no-op
    elsewhere since rustup is only installed on armhf images).

Why preserve_sources_list: true

Before snapshotting, the builder runs cloud-init clean --logs --machine-id --seed --configs all.
On the runner VM's first boot cloud-init re-runs and regenerates ubuntu.sources, dropping the
Architectures: pin — while the dpkg --add-architecture armhf state persists. Apt would then
request binary-armhf from the arm64-only primary/security mirrors and 404 on apt-get update.
The drop-in lives in /etc/cloud/cloud.cfg.d/ (which cloud-init clean does not touch) and
tells cloud-init not to manage apt sources, so the baked multiarch config survives.

armhf is restricted to noble+ base images

rustup, docker-buildx and the pinned libicu sonames are only available from noble (24.04)
onwards. _generate_cloud_init_script raises UnsupportedArchitectureError up front when
arch == ARM and the base is older than noble (ARM_SUPPORTED_BASE_IMAGES = (NOBLE, RESOLUTE)),
instead of failing midway through the build.

Changes

Charm layer

  • src/state.py: add ARM to the Arch enum and ARCHITECTURES_ARM = {"armhf", "arm", "armv7l"};
    map those config values to Arch.ARM in Arch.from_charm.
  • charmcraft.yaml: document the armhf/arm architecture option (experimental; notes it builds
    on arm64 hardware with a 32-bit userspace and uses the fork's runner binary).

App layer

  • config.py: add Arch.ARM; to_openstack()aarch64; ARM_ADDITIONAL_APT_PACKAGES,
    ARM_EXCLUDED_DEFAULT_APT_PACKAGES, ARM_LIBICU_APT_PACKAGE_BY_BASE.
  • cloud_image.py: Arch.ARM base download arch → arm64; add armhf to
    SupportedBaseImageArch.
  • openstack_builder.py: assemble the armhf apt package set (drop cargo/rustc, add armhf multiarch
    libs + release-matched libicu); fail fast with UnsupportedArchitectureError on pre-noble bases;
    ARM_SUPPORTED_BASE_IMAGES.
  • templates/cloud-init.sh.j2: enable_armhf_multiarch() (dpkg arch, source pinning,
    ports.ubuntu.com source, preserve_sources_list drop-in) run before package install for
    arch == arm; rustup default stable for the ubuntu user.
  • store.py: minor refactor of the image-properties dict (no behaviour change).

Tests

Unit (TDD) — app suite: 133 passed, black/isort/flake8 clean.

  • test_config.pyArch.ARM → "aarch64".
  • test_cloud_image.pyArch.ARM base download arch → arm64.
  • test_state.pyarmhf/arm/armv7lArch.ARM.
  • test_openstack_builder.py::test__generate_cloud_init_script — armhf on a supported base renders
    the multiarch setup, the correct apt set (cargo/rustc removed; armhf libs + matched libicu added),
    and the rustup step.
  • test_openstack_builder.py — armhf on focal/jammy raises UnsupportedArchitectureError.
  • test_integration_commands.pycommands_for_arch(Arch.ARM) includes the armhf assertions.

Integration — a new armhf matrix leg (noble + resolute) in
.github/workflows/integration_test_app.yaml, sharing the arm64 tenant creds. It builds a real
snapshot, boots a runner VM from it, and runs the base runner command suite plus armhf-specific
assertions (ARM_RUNNER_COMMANDS): rustc/cargo/docker buildx present, and crucially
file .../Runner.Listener is an ELF 32-bit ... ARM binary (interpreter
/lib/ld-linux-armhf.so.3). Both armhf legs pass end-to-end.

Notes / follow-ups

  • Deployment wiring (Terraform/Juju in the prod repo) is a separate change: reuse the arm64
    OpenStack creds/project/network and set architecture = "armhf".
  • The arm64/armhf shared staging tenant (stg-ps7-...-arm64) has an external periodic resource
    reaper that can transiently fail keypair assertions across arches; unrelated to this change.

Add "armhf"/"arm" as a buildable image architecture. armhf images are built
on arm64 (aarch64) hosts running a 32-bit userspace and use the GitHub runner
binary published by the canonical/github-actions-runner fork
(actions-runner-linux-arm-<version>.tar.gz).

Changes:
- charm state.py: add ARM to the Arch enum and map the "armhf"/"arm"/"armv7l"
  config values to it in Arch.from_charm.
- app config.py: add ARM to the Arch enum (drives the runner download arch
  "arm") with to_openstack() -> "armhf"; add ARM_ADDITIONAL_APT_PACKAGES
  (libicu74, libatomic1) required at runtime by the self-contained .NET runtime
  bundled in the linux-arm runner tarball.
- app cloud_image.py: map Arch.ARM to the "armhf" Ubuntu cloud-image arch.
- app openstack_builder.py: install the armhf runtime apt packages when
  building for Arch.ARM.
- charmcraft.yaml: document the "armhf" architecture option.

Unit tests added for every new arch mapping (TDD).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds armhf (32-bit ARM) as a supported build architecture across the charm and the image-builder app, enabling production of Ubuntu cloud images intended to run a 32-bit userspace on arm64 hardware.

Changes:

  • Extend charm-side architecture parsing to recognize armhf/arm/armv7l and map them to a new Arch.ARM.
  • Extend app-side architecture support to map Arch.ARM to OpenStack/cloud-image arch armhf, and install additional runtime apt packages for armhf builds.
  • Add/extend unit tests covering the new architecture mappings and cloud-init package selection.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/state.py Adds ARM arch recognition (ARCHITECTURES_ARM) and maps to Arch.ARM.
tests/unit/test_state.py Extends charm unit tests for new ARM aliases.
charmcraft.yaml Documents the new armhf/arm architecture option.
app/src/github_runner_image_builder/config.py Adds Arch.ARM, OpenStack conversion to armhf, and ARM-specific apt package list.
app/src/github_runner_image_builder/cloud_image.py Allows armhf as a supported Ubuntu cloud-image architecture and maps Arch.ARM -> armhf.
app/src/github_runner_image_builder/openstack_builder.py Appends ARM additional apt packages in generated cloud-init for Arch.ARM.
app/tests/unit/test_config.py Tests Arch.ARM -> "armhf" for OpenStack conversion.
app/tests/unit/test_cloud_image.py Tests Arch.ARM -> "armhf" mapping for cloud-image selection.
app/tests/unit/test_openstack_builder.py Tests ARM-specific additional apt packages in generated cloud-init.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/unit/test_state.py
Comment thread charmcraft.yaml Outdated
Comment thread app/src/github_runner_image_builder/openstack_builder.py Outdated
florentianayuwono and others added 3 commits July 2, 2026 16:21
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Add rustup and docker-buildx to the armhf-only additional apt packages and
set the default Rust toolchain for the ubuntu user during image build.

- config.py: ARM_ADDITIONAL_APT_PACKAGES gains rustup and docker-buildx
  (docker.io is already in the default package set).
- cloud-init.sh.j2: after configuring system users, run
  `sudo -u ubuntu rustup default stable`, guarded on the arm arch so it is a
  no-op on other architectures (rustup is only installed on armhf images).

Test updated: the armhf cloud-init assertion now expects the extra packages
and the rustup toolchain step.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The armhf additional apt packages (libicu74, rustup, docker-buildx) are only
available from the noble (24.04) archive onwards. On focal/jammy these package
names do not exist, so an apt-get install would fail midway through the image
build with a confusing error.

Fail fast in _generate_cloud_init_script: when arch == ARM and the requested
base image is older than noble, raise UnsupportedArchitectureError with a clear
message instead of producing a cloud-init script that is guaranteed to fail.

Tests: the cloud-init render test now exercises armhf on a supported (noble)
base, and a new test asserts UnsupportedArchitectureError is raised for armhf
on focal/jammy.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@florentianayuwono florentianayuwono marked this pull request as ready for review July 2, 2026 10:51
Comment thread app/src/github_runner_image_builder/templates/cloud-init.sh.j2
Comment thread app/src/github_runner_image_builder/templates/cloud-init.sh.j2
Comment thread app/src/github_runner_image_builder/openstack_builder.py
florentianayuwono and others added 9 commits July 6, 2026 16:50
…cript

Add the missing Raises: section for the fail-fast guard that rejects
armhf images on pre-noble base images, addressing review feedback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Breaks the commands.py <-> helpers.py circular import so command
constants can be imported without the heavy OpenStack/fabric deps,
enabling unit tests for arch-conditional command selection.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add ARM_RUNNER_COMMANDS (rustc/cargo, docker buildx, 32-bit
Runner.Listener) and a pure commands_for_arch() selector, with unit
tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Thread arch through run_openstack_tests so armhf images additionally
assert rustup, docker-buildx, and the 32-bit runner binary.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reuses the arm64 OpenStack tenant (creds/network/flavor) with
--arch armhf. armhf is restricted to noble and resolute to match
ARM_SUPPORTED_BASE_IMAGES.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The pytest --arch option restricted choices to amd64/arm64/s390x/ppc64le,
so pytest rejected 'armhf' before arch_fixture could map it. Add armhf to
the allowed choices.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Nova validates the glance image 'architecture' property against its
architecture enum, which uses 'armv7l' for 32-bit ARM; 'armhf' (the
Ubuntu userland ABI name) is rejected with 'Architecture name armhf is
not valid'. The cloud-image download filename still uses 'armhf' via
cloud_image.py, which is unaffected.

Surfaced by the new armhf integration test when booting an OpenStack
server from the built image.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The armhf base image is tagged architecture=armv7l and booted on aarch64
compute hosts, which use the QEMU "virt" machine type. That machine type
has no IDE bus, so libvirt defaulted to an IDE controller for the root disk
and the config-drive CD-ROM and rejected the domain with "IDE controllers
are unsupported for this QEMU binary or machine type", leaving the builder
VM in ERROR.

Set hw_machine_type=virt and virtio/scsi disk/cdrom buses on ARM image
uploads so the guest boots. amd64 keeps the default PC machine type.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@yhaliaw yhaliaw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR threads 32-bit ARM (armhf) support throughout the image builder stack — Arch.ARM, OpenStack image properties, ARM-specific apt packages, a rustup cloud-init step, and a base-image guard are all added consistently across the charm and app layers, with thorough unit and integration test coverage. The implementation is correct; the distinct arch strings (arm/armhf/armv7l) are intentional and round-tripped correctly across lookup points. The comments below are minor polish suggestions with no correctness implications.


🤝 Human review with AI assistance.

Comment thread app/src/github_runner_image_builder/config.py Outdated
florentianayuwono and others added 2 commits July 8, 2026 18:48
The armhf builder VMs booted on the aarch64 virt machine type but never
became SSH-reachable. The root disk used virtio-blk (so the VM booted), but
the config drive was attached as a virtio-scsi CD-ROM. The 32-bit armhf guest
kernel lacks the virtio-scsi driver, so it could not read the config drive,
cloud-init received no network metadata, and networking never came up.

Attach the config drive on the same virtio-blk bus as the root disk and drop
the now-unneeded virtio-scsi model.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@florentianayuwono florentianayuwono self-assigned this Jul 9, 2026
@florentianayuwono florentianayuwono marked this pull request as draft July 9, 2026 12:26
virtio-blk cannot back ejectable CD-ROM media (libvirt rejects the domain
with "disk type of 'vdb' does not support ejectable media"), so the
config-drive CD-ROM must ride a virtio-scsi controller. This restores the
bus layout that at least boots the guest so its serial console can be
inspected to root-cause the SSH-unreachable behaviour on the aarch64 hosts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
florentianayuwono and others added 11 commits July 9, 2026 21:05
Native armhf VMs (32-bit kernel) do not boot on the ps7 arm64 hosts (empty
console). The proven-working model from runner PRs #155/#169/#171 is an arm64
kernel running 32-bit armhf userland via native AArch32. This spec redefines
the image-builder armhf path to build an arm64 boot image carrying the 32-bit
linux-arm runner agent plus armhf multiarch userland.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The armhf runner image is an arm64 boot image carrying a 32-bit runner
payload; native armhf (32-bit kernel) images do not boot on the aarch64
hosts. Schedule/boot the base image and snapshot as aarch64.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The builder VM must boot a 64-bit kernel on the aarch64 hosts. Download the
arm64 base cloud image; the 32-bit runner payload is added via multiarch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With the armhf runner image now built from an arm64 base (64-bit kernel), it
boots with the cloud's normal defaults. The virtio-scsi/machine-type
workaround for booting a native armhf (32-bit kernel) image is no longer
needed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The 32-bit linux-arm runner agent needs the armhf loader (libc6:armhf) and
armhf builds of its .NET runtime deps (libicu74:armhf, libatomic1:armhf) to
exec on the arm64 image via native AArch32.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add dpkg --add-architecture armhf (guarded to the arm runner) before apt
install so the arm64 image can install and run the 32-bit linux-arm runner
and its armhf runtime deps via native AArch32. armhf packages are served by
ports.ubuntu.com, which the arm64 base already uses.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The docs/superpowers plan and spec are internal agent planning artifacts,
not product documentation. Untrack them and gitignore the directory so the
Vale docs linter does not scan them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The arm64 cloud image's apt mirrors (clouds.archive/security.ubuntu.com)
only carry the native architecture; armhf is a ports architecture and its
package indices 404 there. Pin the stock sources to the native arch and add
a ports.ubuntu.com source for armhf so the 32-bit runner's runtime deps
(libc6:armhf, libicu74:armhf, libatomic1:armhf) resolve.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two armhf package-selection bugs surfaced once the builder VM boots and runs
apt:

- rustup conflicts with the distro cargo/rustc packages ('held broken
  packages'). rustup provides the armhf/armv7 Rust toolchain, so drop cargo
  and rustc from the default apt set on armhf.
- libicu's soname is release-specific (noble ships libicu74, resolute ships
  libicu78); the hardcoded libicu74:armhf is 'Unable to locate' on resolute.
  Select the armhf libicu matching the base image.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The runner VM boots from a snapshot whose cloud-init state was cleared
via "cloud-init clean --configs all" before snapshotting. On first boot
cloud-init regenerates the stock apt sources, dropping the native-arch
pin added by enable_armhf_multiarch, while the armhf dpkg architecture
persists. apt-get update then requests binary-armhf indexes from the
primary/security mirrors (which only serve arm64) and 404s, failing the
runner's apt-get update.

Bake an /etc/cloud/cloud.cfg.d drop-in with apt.preserve_sources_list:
true so cloud-init keeps the pinned sources baked into the image. The
drop-in lives outside /var/lib/cloud so it survives cloud-init clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@florentianayuwono florentianayuwono marked this pull request as ready for review July 10, 2026 08:10

@yanksyoon yanksyoon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a pretty complex task, thank you for making it happen @florentianayuwono !

Comment thread app/src/github_runner_image_builder/templates/cloud-init.sh.j2
Comment thread app/src/github_runner_image_builder/templates/cloud-init.sh.j2
Comment thread app/src/github_runner_image_builder/templates/cloud-init.sh.j2
Comment thread app/tests/unit/test_integration_commands.py Outdated
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@florentianayuwono florentianayuwono enabled auto-merge (squash) July 10, 2026 11:27
@florentianayuwono florentianayuwono merged commit b60a2bd into main Jul 11, 2026
74 of 91 checks passed
@florentianayuwono florentianayuwono deleted the feat/armhf-arch-support-isd-5856 branch July 11, 2026 05:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants