Skip to content

fix(idle-shutdown): base CPU activity on the application container rather than whole-VM load - #465

Closed
june-hua wants to merge 1 commit into
masterfrom
june/idle-detect-app-cpu
Closed

fix(idle-shutdown): base CPU activity on the application container rather than whole-VM load#465
june-hua wants to merge 1 commit into
masterfrom
june/idle-detect-app-cpu

Conversation

@june-hua

Copy link
Copy Markdown
Contributor

Problem

probe-user-access.sh derives the CPU activity signal from whole-VM load average:

LOAD="$(awk '{print $3}' /proc/loadavg)"   # 15-min average, entire VM

That attributes any container's CPU to user activity, not just the application's. A template with
a continuously-busy sidecar keeps the load average above the 0.1 threshold indefinitely, so
last-active/cpu is refreshed on every probe, idle-shutdown.sh never fires, and the VM runs until
someone notices. On a 4-vCPU VM a sidecar using ~1 core produces a load average around 1.0 — 10x the
threshold.

Change

CPU activity is now measured from the application container's own cgroup accounting
(cpu.stat/usage_usec on v2, cpuacct.usage on v1), diffed against the previous probe's sample.
Container is resolved by name → PID → /proc/<pid>/cgroup, which is cgroup-driver agnostic.

  • Target container — VM metadata key idle-cpu-container-name, default application-server.
    Metadata (rather than a new positional arg) because the systemd timer that invokes this script lives
    in another repo, and this is already how per-workspace config reaches these scripts
    (idle-shutdown.sh reads idle-timeout-seconds the same way). Works on both clouds — the AWS
    metadata-utils.sh maps the key into the vwbusr: tag namespace.
  • Threshold: 5.0% of one core, deliberately not normalized by core count so it means the same
    thing on a 2-vCPU and a 64-vCPU VM (values >100 are expected for multi-core work). Derivation: the
    old 0.1 load average ≈ 10% of one core measured across the whole box; with sidecar and host noise
    now excluded, 5% keeps clear margin over app-server background chatter (ioloop heartbeats, autosave
    — well under 2%) while staying far below real work.
  • $1 keeps its original meaning (whole-VM load average) and is used only by the fallback path.
    Reusing it for the new metric would have been a trap: an existing caller passing 0.1 would have
    set a 0.1%-of-a-core threshold and silently preserved the bug.
  • Graceful degradation — missing container, stopped container, missing cgroup accounting, or a
    stale sample after restart all fall back to the previous load-average behaviour rather than failing.
  • last-active/proxy and last-active/ssh are unchanged.

Tests

New tests/test-probe-user-access.bats, 20 cases, stub-driven with a fake clock, /proc, and cgroup
tree: threshold boundary, idle/busy/multi-core, cgroup v1 and v2, missing/stopped container, missing
accounting, restart with stale sample, first-probe sampling, sub-second gap rejection, metadata
override, and assertions that the proxy/ssh signals still behave. Wired into test-scripts.yaml with
path filters. shellcheck clean under the repo's CI flags.

The existing tests/ harness is integration-only (docker exec into a live container) so it couldn't
cover this.

Two bugs found while testing: emit() wrote to stdout and was being captured into the CPU reading by
command substitution (now stderr); and bare ! negative assertions in bats silently pass unless
final, so those are wrapped in a helper.

Reviewers please check

  • The threshold is reasoned, not measured. 5.0% needs a sanity check against a real idle
    JupyterLab and a light workload before this leaves draft.
  • Regression risk: low-CPU-but-real workloads. /proc/loadavg counts I/O wait, so an I/O-bound or
    GPU-bound job with light CPU used to keep the VM alive. Container CPU accounting does not.
    last-active/proxy/ssh cover interactive cases but not a detached long-running job. Worth deciding
    whether to also treat container block-I/O (io.stat) as an activity signal.
  • AWS metadata costget_metadata_value spawns an AWS CLI container per call, so this adds one
    spawn per probe interval (idle-shutdown.sh already does this, so there's precedent, but it roughly
    doubles it). I deliberately did not cache: a transient metadata failure returns the default,
    indistinguishable from a real read, and caching that would pin the wrong container name for the VM's
    lifetime and reintroduce the never-stops bug.
  • New env seams (METADATA_UTILS, CGROUP_ROOT, PROC_ROOT, STATE_DIR) exist only for fixture
    injection and default to real paths. No other butane script parameterizes
    /home/core/metadata-utils.sh, so a second opinion would be welcome.
  • Untested on real hardware — cgroup path resolution is verified only against fixtures modeled on
    Flatcar's systemd cgroup-v2 layout (/system.slice/docker-<id>.scope).

…ther than whole-VM load

probe-user-access.sh derived the `last-active/cpu` signal from the whole-VM
15-minute load average. That attributes CPU burned by sidecar containers to the
user: a template running a continuously busy sidecar keeps the load average
above the 0.1 threshold permanently, so `last-active/cpu` is refreshed on every
probe, idle-shutdown.sh never fires, and the VM bills indefinitely even when
nobody is using it.

Measure only the container running the user's workload instead. The container's
cgroup CPU accounting (`cpu.stat` `usage_usec` on cgroup v2, `cpuacct.usage` on
v1) is diffed against the sample taken by the previous probe run, which averages
CPU over the whole inter-probe interval for free.

The unit changes: `/proc/loadavg` is a runnable-task count that includes I/O
wait, while the new metric is a percentage of one CPU core, so the threshold is
re-derived rather than reused. The default is 5.0% of one core.

The measured container defaults to `application-server` and is overridable per
VM with the new `idle-cpu-container-name` instance attribute, for templates
where the workload runs in a differently named container.

If the container cannot be measured -- absent, not running, or no cgroup
accounting -- the script falls back to the previous load-average behaviour using
the existing `$1` threshold, so an unmeasurable container cannot cause a
premature shutdown. `last-active/proxy` and `last-active/ssh` are unchanged.

Adds a bats suite covering the threshold boundary, idle/busy containers, cgroup
v1 and v2, missing and stopped containers, container restart, first-probe
sampling and the metadata override, and runs it in CI.
@june-hua

Copy link
Copy Markdown
Contributor Author

current idle timeout cpu threshold is sufficient

@june-hua june-hua closed this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant