fix(idle-shutdown): base CPU activity on the application container rather than whole-VM load - #465
Closed
june-hua wants to merge 1 commit into
Closed
fix(idle-shutdown): base CPU activity on the application container rather than whole-VM load#465june-hua wants to merge 1 commit into
june-hua wants to merge 1 commit into
Conversation
…ther than whole-VM load probe-user-access.sh derived the `last-active/cpu` signal from the whole-VM 15-minute load average. That attributes CPU burned by sidecar containers to the user: a template running a continuously busy sidecar keeps the load average above the 0.1 threshold permanently, so `last-active/cpu` is refreshed on every probe, idle-shutdown.sh never fires, and the VM bills indefinitely even when nobody is using it. Measure only the container running the user's workload instead. The container's cgroup CPU accounting (`cpu.stat` `usage_usec` on cgroup v2, `cpuacct.usage` on v1) is diffed against the sample taken by the previous probe run, which averages CPU over the whole inter-probe interval for free. The unit changes: `/proc/loadavg` is a runnable-task count that includes I/O wait, while the new metric is a percentage of one CPU core, so the threshold is re-derived rather than reused. The default is 5.0% of one core. The measured container defaults to `application-server` and is overridable per VM with the new `idle-cpu-container-name` instance attribute, for templates where the workload runs in a differently named container. If the container cannot be measured -- absent, not running, or no cgroup accounting -- the script falls back to the previous load-average behaviour using the existing `$1` threshold, so an unmeasurable container cannot cause a premature shutdown. `last-active/proxy` and `last-active/ssh` are unchanged. Adds a bats suite covering the threshold boundary, idle/busy containers, cgroup v1 and v2, missing and stopped containers, container restart, first-probe sampling and the metadata override, and runs it in CI.
Contributor
Author
|
current idle timeout cpu threshold is sufficient |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
probe-user-access.shderives the CPU activity signal from whole-VM load average:That attributes any container's CPU to user activity, not just the application's. A template with
a continuously-busy sidecar keeps the load average above the
0.1threshold indefinitely, solast-active/cpuis refreshed on every probe,idle-shutdown.shnever fires, and the VM runs untilsomeone notices. On a 4-vCPU VM a sidecar using ~1 core produces a load average around 1.0 — 10x the
threshold.
Change
CPU activity is now measured from the application container's own cgroup accounting
(
cpu.stat/usage_usecon v2,cpuacct.usageon v1), diffed against the previous probe's sample.Container is resolved by name → PID →
/proc/<pid>/cgroup, which is cgroup-driver agnostic.idle-cpu-container-name, defaultapplication-server.Metadata (rather than a new positional arg) because the systemd timer that invokes this script lives
in another repo, and this is already how per-workspace config reaches these scripts
(
idle-shutdown.shreadsidle-timeout-secondsthe same way). Works on both clouds — the AWSmetadata-utils.shmaps the key into thevwbusr:tag namespace.thing on a 2-vCPU and a 64-vCPU VM (values >100 are expected for multi-core work). Derivation: the
old
0.1load average ≈ 10% of one core measured across the whole box; with sidecar and host noisenow excluded, 5% keeps clear margin over app-server background chatter (ioloop heartbeats, autosave
— well under 2%) while staying far below real work.
$1keeps its original meaning (whole-VM load average) and is used only by the fallback path.Reusing it for the new metric would have been a trap: an existing caller passing
0.1would haveset a 0.1%-of-a-core threshold and silently preserved the bug.
stale sample after restart all fall back to the previous load-average behaviour rather than failing.
last-active/proxyandlast-active/sshare unchanged.Tests
New
tests/test-probe-user-access.bats, 20 cases, stub-driven with a fake clock,/proc, and cgrouptree: threshold boundary, idle/busy/multi-core, cgroup v1 and v2, missing/stopped container, missing
accounting, restart with stale sample, first-probe sampling, sub-second gap rejection, metadata
override, and assertions that the proxy/ssh signals still behave. Wired into
test-scripts.yamlwithpath filters. shellcheck clean under the repo's CI flags.
The existing
tests/harness is integration-only (docker execinto a live container) so it couldn'tcover this.
Two bugs found while testing:
emit()wrote to stdout and was being captured into the CPU reading bycommand substitution (now stderr); and bare
!negative assertions in bats silently pass unlessfinal, so those are wrapped in a helper.
Reviewers please check
JupyterLab and a light workload before this leaves draft.
/proc/loadavgcounts I/O wait, so an I/O-bound orGPU-bound job with light CPU used to keep the VM alive. Container CPU accounting does not.
last-active/proxy/sshcover interactive cases but not a detached long-running job. Worth decidingwhether to also treat container block-I/O (
io.stat) as an activity signal.get_metadata_valuespawns an AWS CLI container per call, so this adds onespawn per probe interval (
idle-shutdown.shalready does this, so there's precedent, but it roughlydoubles it). I deliberately did not cache: a transient metadata failure returns the default,
indistinguishable from a real read, and caching that would pin the wrong container name for the VM's
lifetime and reintroduce the never-stops bug.
METADATA_UTILS,CGROUP_ROOT,PROC_ROOT,STATE_DIR) exist only for fixtureinjection and default to real paths. No other butane script parameterizes
/home/core/metadata-utils.sh, so a second opinion would be welcome.Flatcar's systemd cgroup-v2 layout (
/system.slice/docker-<id>.scope).