fix(garm): deliver aproxy bootstrap as pre-install script so it runs before GARM's install wrapper#272
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Moves aproxy bootstrap out of the per-scaleset runner template and into GARM scaleset extra_specs.pre_install_scripts (as 00-aproxy) so it runs before GARM’s compiled-in install wrapper can fetch template content—fixing runner provisioning on proxy-only networks. It also base64-encodes all pre_install_scripts values to match GARM’s map[string][]byte wire type and sets disable_updates when a proxy is configured.
Changes:
- Deliver aproxy bootstrap via scaleset
extra_specs.pre_install_scripts["00-aproxy"](base64-encoded) and setextra_specs.disable_updates=truewhen proxying. - Update aproxy script rendering to configure
snap set aproxy proxy=with a barehost:portform. - Retarget/add unit + integration assertions to validate extra-spec injection, encoding, and drift behavior.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/changelog.md | Documents the operational change for proxy-only networks and the new delivery mechanism. |
| charms/tests/integration/test_garm.py | Updates integration assertions to validate aproxy delivery via extra_specs (decoded) instead of template injection. |
| charms/garm/tests/unit/test_scaleset_reconciler.py | Adds coverage for base64 encoding, disable_updates, aproxy script injection, and drift/no-op cases. |
| charms/garm/tests/unit/test_runner_template.py | Retargets tests away from template-injected aproxy; adds direct tests for standalone aproxy script rendering and scheme stripping. |
| charms/garm/src/templates/pre_install.j2 | Removes aproxy block from the template pre-install snippet. |
| charms/garm/src/templates/aproxy.j2 | Converts aproxy template into a standalone best-effort bash script and switches aproxy config to proxy_host_port. |
| charms/garm/src/scaleset_reconciler.py | Centralizes extra_specs generation via _effective_extra_specs() for create/update/drift and injects 00-aproxy + disable_updates. |
| charms/garm/src/runner_template.py | Introduces render_aproxy_pre_install_script() and _proxy_host_port(); updates template injection scope to exclude proxy/aproxy. |
cbartz
force-pushed
the
fix/garm-aproxy-pre-install-bootstrap
branch
from
f7ea017 to
b7943c4
Compare
cbartz
force-pushed
the
fix/garm-aproxy-pre-install-bootstrap
branch
from
b7943c4 to
42ee6a0
Compare
…before GARM's install wrapper When a scaleset uses a server-side template, GARM injects a compiled-in wrapper as the cloud-init install script; the wrapper must reach the GARM API to fetch the template before any template content runs. Proxy bootstrap living inside the template therefore can never work on proxy-only networks: the wrapper's fetch has no egress until the proxy is up. The aproxy setup is now rendered as a standalone script and delivered via the pre_install_scripts extra-spec (key "00-aproxy", sorting before operator scripts), which GARM runs as root before the wrapper. When a proxy is configured, apt updates are also disabled on boot (disable_updates extra-spec): cloud-init's package upgrade runs before any pre-install script, so it can never use the proxy and only burns boot time on timeouts. Two related fixes: `snap set aproxy proxy=` now receives a bare host:port (aproxy rejects URLs; the new `snap set system proxy.http/https=` calls take the full URL), and all pre_install_scripts values are base64-encoded on the wire as required by GARM's map[string][]byte extra-spec type.
cbartz
force-pushed
the
fix/garm-aproxy-pre-install-bootstrap
branch
from
42ee6a0 to
ef4d1be
Compare
Add 'aproxy' to the docs wordlist and reword 'prepends' (absent from the base dictionary) to 'injects' so the docs spellcheck passes.
The nftables DNAT sends :80/:443 to aproxy's :54969; applying it when the aproxy snap failed to install or start would black-hole all egress into a dead port rather than leaving the box on whatever path it had. Only install the redirect once 'snap services aproxy' reports active, tolerate an already-baked-in snap, and log every step (and both bail-out paths) to the cloud-init console output so a broken proxy bootstrap is diagnosable. Also correct the header comment: cloud-init runs each pre-install script as its own runcmd entry and continues past a non-zero exit, so strict mode would not have protected GARM's install sequence anyway.
cbartz
requested review from
florentianayuwono,
javierdelapuente,
weiiwang01,
yanksyoon and
yhaliaw
as code owners
yanksyoon
approved these changes
Jul 9, 2026
weiiwang01
approved these changes
Jul 9, 2026
Review follow-ups on the aproxy pre-install script: - Runner VMs are single-use and never reboot, so persisting the ruleset to /etc/nftables.conf and enabling nftables.service is dead weight. Pipe the ruleset straight into the kernel with 'nft -f -' instead, keeping the aproxy-active guard around it. - Hoist the repeated function-local test imports (RunnerConfig, _effective_extra_specs, build_template_data) to the module top.
…nstall-bootstrap # Conflicts: # docs/changelog.md
|
Fyi, Proxy settings get added to package managers using the cloud-init bootcmd section, so it runs before the package upgrade. GARM itself has separate proxy settings in the config file. I will update docs before the next release. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Delivers the aproxy proxy bootstrap as a GARM
pre_install_scriptsextra-spec entry (00-aproxy) instead of injecting it into the per-scaleset runner template. When a proxy is configured, cloud-init package upgrades are also disabled on boot. Two related fixes:snap set aproxy proxy=now gets a barehost:port(aproxy rejects URLs), and allpre_install_scriptsvalues are base64-encoded as GARM'smap[string][]byteextra-spec type requires.Why we need it
On proxy-only networks (no direct egress), runners never registered. When a scaleset uses a server-side template, GARM prepends a compiled-in wrapper as the cloud-init install script; that wrapper must reach the GARM metadata API to fetch the template before any template content runs. Proxy bootstrap living inside the template therefore can never help the wrapper's fetch succeed — the fetch times out (~134 s/attempt) with no egress. GARM's
pre_install_scriptsrun as root before the wrapper, so moving aproxy there brings egress up in time. The apt upgrade runs even earlier (before any pre-install script), so it's disabled when a proxy is set rather than left to time out.Test plan
render_aproxy_pre_install_script, added happy-path + scheme-stripping (http:///https:/// scheme-less →host:port) coverage, and added scaleset-reconciler tests for the00-aproxy+disable_updatescreate path and three drift cases (encoded match → no-op; legacy raw-text → update; missingdisable_updates→ update).tox -c tox.toml -e fmt,lint,static,unitall pass; touched-module coverage 98% / 89%.test_runner_options_render_into_scaleset_templateto assert the aproxy script now lands in the scaleset'sextra_specs(base64-decoded00-aproxy,disable_updates: true) rather than the template. Not run here (needs a live Juju model).Review focus
_effective_extra_specs()inscaleset_reconciler.pyis the single source of truth feeding create, update, and drift detection — worth confirming the drift comparison (encoded scripts +disable_updates) is symmetric with what's sent.00-aproxykey name is load-bearing: GARM runs pre-install scripts in lexicographic order, so it must sort before the configurator'spre_install.sh.Potential breaking changes
Existing scalesets get a one-time update on first reconcile as their
extra_specsconverge (encoding change + aproxy entry). No config-surface change. Operational prerequisite: the egress proxy must allow CONNECT to the GARM API host, since all GARM API traffic from runners now rides through aproxy.Checklist
CONTRIBUTING.mdhas been updated upon changes to the contribution/development process (e.g. changes to the way tests are run) — n/adocs/changelog.mdwith user-relevant changestest_garmmodule, no new module)terraform fmtpasses andtflintreports no errors — n/aAGENTS.md— n/a (no convention change).copilot-collections.yamlor.github/instructions/: I re-checked theAGENTS.md"12-factor divergences" guidance — n/a