ci-runner

ci-runner is the Windows host controller and disposable Linux worker image for a small, demand-scaled GitHub Actions fleet. Its purpose is to move eligible private-repository CI from paid GitHub-hosted runners onto melo-desk-001 and melo-lap-001 without creating a second CI contract. There is no routing policy: an eligible private job names the governed fleet label melodic-ubuntu-24.04-x64 as a literal, and GitHub queues it for that label.

Both hosts passed their acceptance gates and serve every private job that names the fleet label, which is every eligible private job that does not hold a declared hosted exception. The former Compose/restart-in-place implementation is retired and its files are deleted; the only production credential entry point is ci-runner secret import --file PATH.

Runtime architecture

flowchart LR
    W["Workflow job"] -->|"names melodic-ubuntu-24.04-x64"| S["Host-owned scale sets"]
    W -->|"names an approved hosted label"| H["GitHub-hosted ubuntu-24.04"]
    S --> C1["Windows controller\nmelo-desk-001"]
    S --> C2["Windows controller\nmelo-lap-001"]
    C1 --> D1["Fresh one-job Linux containers"]
    C2 --> D2["Fresh one-job Linux containers"]

Loading

Each host owns an independent scale-set ID and listener session. Organization hosts advertise the same workflow label, so GitHub can distribute work and reassign a job before acquisition if one host disappears. The controller uses GitHub's official Runner Scale Set Client outside Kubernetes and scales from its authoritative TotalAssignedJobs statistic.

Every admitted job gets a new container from the digest-pinned official actions/actions-runner image. The controller streams the one-job JIT payload over attached stdin; the entrypoint exposes it to the official runner through the documented ACTIONS_RUNNER_INPUT_JITCONFIG input. The payload is absent from Docker's persistent config and docker inspect. The container has no host mount, Docker socket, device, GPU, persistent home, work directory, temp, or tool cache, and is removed after terminal diagnostics are copied out.

The Windows process is the only control plane. GitHub App keys are protected with current-user DPAPI and never enter worker containers. The controller talks only to the fixed local Docker Engine endpoint and requires a Linux/amd64 engine; DOCKER_HOST, TLS, and API-version environment overrides are ignored.

Routing contract

There is no selector and no fallback. ci-perf Phase 7 retired the central select-runner reusable workflow in melodic-software/ci-workflows (ci-workflows#569, merged as 541ee4e90d12d77a90a3ddd72a3af9bc78634ea7, released as v0.23.0), and melodic-software/standards#556 (merged as 771a796628f325c3c418c7b397d09fb7211e2972) removed the selector grammar from the runner-policy component. The three policies this section used to document, hosted-only, prefer-self-hosted and self-hosted-only, no longer exist. The CI_RUNNER_POLICY organization variable that chose between them, and its declaration in github-iac OrgCiRouting.cs, were deleted on 2026-09-08 (github-iac#467).

An eligible private-repository job names the governed fleet label melodic-ubuntu-24.04-x64 as a literal in its own runs-on. GitHub queues the job for that label, and this controller supplies runners inside its governed name and label namespaces exactly as before. What changed is upstream of the controller: nothing decides at run time where a job goes. The standards runner policy admits that literal under its managed-literal routing kind and fails a job that names anything else without a declared exception.

A busy runner does not divert work. GitHub queues the job until matching capacity is available, which was already the behavior under the retired liveness rule and is now structural rather than policy-dependent.

There is no consumer-side recovery from an unavailable fleet. The audited github-iac routing-control procedure is deleted with the workflow that implemented it (github-iac#453, merged as 4c5937e6b6067552d11b87abef04620191dda503), and nothing replaces it. Affected jobs queue on the fleet label until a host returns.

The exceptions mechanism does not cover this case, and reaching for it will fail. A job that needs hosted capacity declares an entry under exceptions in its repository's own .github/runner-policy.json, keyed <workflow path>#<jobId>, carrying a justification and a reason that must be a member of the closed hostedExceptionReasons set: dependabot, docker-socket, job-container, privileged-control-plane, publication, service-container, windows. None of those expresses "the fleet is down", and a reason outside the set is rejected by the analyzer. Granting a fleet-outage escape would mean adding a member to that set, which is a governed change to melodic-software/standards and its synced policy.json, not something a consumer repository can do for itself. (hosted-exception-required is the finding the analyzer raises when a required entry is missing, not a key anyone writes.) A rerun changes nothing about placement: there is no selector verdict to recompute, so Re-run all jobs and a failed-job rerun are equivalent as far as routing is concerned. Do not re-run a stale run on a superseded head SHA in a pull request whose concurrency group key does not vary with the head; that hazard is unrelated to routing and is recorded in github-iac docs/topics/ci-perf/POSTURE.md under "Stale re-runs". A workflow_dispatch still creates a separate run with different event and ref context and does not recover the original pull-request check. GitHub documents the distinct full and partial rerun operations and workflow_dispatch event context.

The two-minute ubuntu-slim selector control job is gone with the selector, so it no longer sits in front of a build. The decision record is github-iac ADR 0014, which supersedes ADR 0004. It is added by melodic-software/github-iac#466 and is not on that repository's main yet, so cite the pull request until it merges.

Authoritative behavior:

• GitHub self-hosted routing
• Runner scale-set high availability
• runs-on syntax
• ubuntu-slim limits

Host lifecycle

ci-runner.exe is the only operator interface. With no subcommand it opens the interactive menu; the same operations are available for automation:

ci-runner host
ci-runner host status [--json]
ci-runner host enable [--wait]
ci-runner host disable [--wait|--detach]
ci-runner host game [--wait|--detach]
ci-runner host doctor [--json] [--include-elevated]
ci-runner host logs [--follow|--job ID|--cleanup]
ci-runner host force-stop
ci-runner host controller restart
ci-runner host controller stop-for-update
ci-runner host reboot [--timeout DURATION] [--force] [--dry-run]
ci-runner secret import --file PATH

Modes are persisted separately from checked-in configuration:

• enabled starts Docker when policy allows, advertises capacity, and maintains configured warm workers;
• disabled advertises zero, drains CI, and removes idle workers without touching unrelated Docker or WSL workloads;
• gaming drains CI, stops Docker Desktop, shuts down every WSL distribution, verifies both are down, and remains down across logons.

Desired mode is state\desired.json, and that file survives reboot. This module does not rewrite it to enabled when the machine boots or when the ci-runner-fleet logon task starts the controller. A host left disabled by host disable therefore comes back disabled, and ci-runner host enable is required before CI resumes. gaming persists the same way. host reboot drains capacity and restarts the machine without changing the file.

host enable writes desired.json even when a reconcile pass is already in flight. The controller compares admission intent (mode, schema version, and any temporary capacity override) with the file it loaded for that pass and cancels the pass when they differ. That cancellation is not a scale-set failure. Step reruns immediately so the new mode is advertised without waiting out the listener poll.

If the file changes while that poll is still open, the pass returns before the final Docker Desktop status and managed-worker inventory calls. It does not record desktop-final-status-error or worker-final-inventory-error, and it does not persist phase=degraded. observed.json stays at the previous checkpoint until the retry writes a new one.

If the file changes after that poll-time check, including during worker cleanup, the rest of the pass still runs. The final probes then see a canceled context and can record those two problem codes, and the checkpoint is persisted as phase=degraded before the retry replaces it. host doctor in that window reports the degraded checkpoint even though enable succeeded.

No normal timeout implies force. Busy jobs finish naturally, including drains longer than the warning threshold. force-stop is a separate destructive path that inventories affected jobs and requires typed confirmation. Ctrl+C while watching a drain detaches from the display; it does not cancel the drain.

host doctor is non-elevating by default. It reports the BitLocker check as skipped while still running every ordinary diagnostic. --include-elevated explicitly enables the full BitLocker verification and writes a warning before the verifier can open an Administrator UAC prompt; use it only in an interactive session where that prompt is expected.

host doctor also surfaces pending-OS-reboot state from the standard registry signals (component servicing, session-manager pending file renames, and Windows Update). Because updates auto-install but the host never auto-reboots while a session exists, a pending reboot is expected operational state the operator finishes during a deliberate drain window: the check is advisory, renders as WARN, and never degrades the doctor exit code.

host reboot is the operator command that finishes that drain window: it reuses the stop-for-update capacity-zero drain so desired mode is untouched, then restarts the machine through trusted shutdown.exe /r. --timeout bounds the drain wait; on expiry the controller keeps draining on its own. The command refuses to restart after an unclean drain unless --force is passed. --dry-run drains and reports without requesting a restart. The command does not pass shutdown.exe /f; --force means "reboot after an unclean drain", not "forcibly close applications".

The windowless ci-runner-controller.exe runs from a current-user logon task because Docker Desktop is user-session software. A controller restart first uses the authenticated control plane to bind the shutdown reservation to the preflight PID, version, and job counts, then advertises zero capacity and lets every assigned or active job finish. Only after every pool has two confirmed zero-capacity observations, no active worker remains, and the message sessions, worker runtime, and control server close successfully does the controller write an ACL-hardened, atomic completion receipt bound to the authenticated request ID, old PID, and exact version and emit its dedicated restart exit code. The CLI waits on that exact process handle and requires both the exit code and matching durable receipt before it can touch Task Scheduler.

The CLI then asks Task Scheduler to run the canonical ci-runner-fleet task at most four times with exponential backoff. Those bounded retries close the MultipleInstances=IgnoreNew race while Task Scheduler finishes the prior instance. Success still requires the authenticated control plane to report a different PID at the exact expected version. An unavailable initial control plane, an ordinary controller failure, or a missing or mismatched receipt never authorizes a task start.

The CLI invokes the existing task through a direct hidden schtasks.exe /Run child process. It does not launch the controller directly, open another console window, invoke a shell, request elevation or UAC, or terminate the draining controller. Battery, resource admission, drain, Docker Desktop, and WSL policy remain in the shared Go state machine.

The logon task ci-runner-enable-on-logon is installed by melodic-software/provisioning, not by this module, and this repository does not define its exit codes. When that task was started while Docker Desktop's engine was down, it exited 5: wsl.exe in Docker's context failed with "The file cannot be accessed by the system". The recovery that restored the host was docker desktop restart, then a stop and start of the ci-runner-fleet task, then ci-runner host enable --wait.

Configuration and ownership

The checked-in, nonsecret host YAML is owned by melodic-software/provisioning and is installed as %LOCALAPPDATA%\ci-runner\config.yaml. The strict parser rejects unknown properties, unsupported schemas, invalid units, duplicate targets, unsafe paths, inconsistent thresholds, and explicit null or blank target worker resource sections and fields. YAML merge keys (<<) are rejected everywhere so inheritance cannot bypass those checks.

Provisioning verifies ownership boundaries through the product rather than parsing YAML itself. config validate --json returns the normalized release.compatibilityManifest and paths (secrets, state, logs, and diagnostics) contract. host status --json returns the authenticated live controller's PID, exact version, phase, shutdown state, and job counts under controller; provisioning requires a new nonzero PID and the requested version before committing an install transaction. observed.controller carries the controller's own cost: lastTickDurationSeconds (the previous reconcile tick's wall time) and processCpuSeconds (cumulative process CPU; diff two samples for a rate). Human host status prints both as Controller cost. Tick duration is wall time that includes the listener long poll, so processCpuSeconds deltas are the measure of controller cost. observed.presence carries inputIdleSeconds (age of the last keyboard or mouse input in the controller's session, omitted when the host cannot report it) and interactiveCapActive; human host status prints both as Presence.

Mutable local state is separate:

%LOCALAPPDATA%\ci-runner\
  state\desired.json       # user-owned mode; survives reboot (enable to leave disabled)
  state\observed.json      # controller heartbeat, pools, workers, problems, drain reason
  state\jobs.json          # exact job-to-artifact correlation
  state\restart-completed.json # last authenticated restart completion receipt
  secrets\                 # current-user DPAPI-protected App keys
  logs\controller\         # structured JSON Lines controller events
  logs\workers\            # externally captured runner stdout/stderr
  diagnostics\             # copied runner _diag and terminal cgroup evidence

Permanent capacity and threshold changes are YAML changes, never source-code changes. A menu capacity override is local state and can be reset to the checked-in value. Provisioning must not overwrite desired mode.

state\desired.json keeps the last operator mode across reboot. Boot and logon do not flip a persisted disabled mode to enabled; host enable is required to resume runners. A missing file is not created as enabled: the next reconcile writes disabled as a fail-safe, which still does not resume CI.

One diagnostics policy governs both copied runner stdout and compressed _diag archives. maxFileSize, rawDiagnosticMaxInput, retention, total-cap, and cleanup cadence are strict host configuration. Startup retention runs only after every managed active or exited container has been adopted. The explicit host logs --cleanup command first inventories the fixed local Docker endpoint and refuses to run if that safety inventory is unavailable.

Optional OpenTelemetry export uses a reviewed telemetry: host configuration block, with standard OTEL_* environment variables retained for legacy and exporter-specific settings. It is disabled without explicit configuration and emits controller reconcile spans and low-cardinality fleet, capacity, worker, job, host-pressure, gate, and lifecycle metrics. Exporter failures are locally logged and cannot change capacity, drain workers, cancel jobs, or stop the controller. Runner names, container IDs, job IDs, credentials, and arbitrary error text never become metric attributes. See OpenTelemetry observability for the exact enablement, metric, cancellation, and failure-isolation contracts.

Default worker parity is 2 CPU, 8 GiB memory, no additional swap, 4096 PIDs, and no devices. Admission also honors configurable host memory/CPU thresholds, hysteresis, global worker limits, and laptop AC-only policy. Active workers are never killed to reclaim resources.

An enabled pool can keep one excess healthy idle runner as bounded burst inventory instead of advertising zero capacity for the whole pool to retire it. All pools share the host-wide advertised-capacity budget; when no safe slot is available, the excess runner falls back to quiescence. The next admitted job naturally consumes retained ephemeral inventory. Larger downscales and explicit zero-capacity modes retain the two-poll quiescence requirement before exact runner deregistration, so assignment races still fail closed.

While a GitHub listener poll is open, the controller refreshes managed worker inventory on the normal reconciliation cadence. A completed ephemeral container cancels the stale poll immediately so authoritative assignments can start replacement workers without waiting for the listener timeout.

Targets may optionally override individual fields from the global resources.worker profile. Every omitted field inherits the global value; the effective profile must still satisfy the same CPU, memory, total memory-plus-swap, and PID validation. For example, this relevant excerpt keeps the ordinary pool at 2 CPU/8 GiB while giving a CodeQL pool a larger profile:

schemaVersion: 2

resources:
  maximumConcurrentWorkers: 5
  minimumAvailableMemoryPercent: 25
  memoryCapacityIncreaseMarginPercent: 25
  worker:
    cpus: 2
    memory: 8GiB
    memorySwap: 8GiB
    pids: 4096
github:
  targets:
    - id: ordinary
      # The remaining required target identity and capacity fields are omitted.
    - id: codeql
      resources:
        worker:
          cpus: 4
          memory: 24GiB
          memorySwap: 24GiB
          pids: 8192

Host admission computes the byte headroom above the physical-memory floor and allocates prospective starts in target-priority order, charging each start's effective memory profile. A larger target that does not fit is skipped so a smaller eligible target can still run. Successful mixed-profile starts are also reserved by their exact memory limits against fresh-but-potentially-stale host observations for the rest of that reconciliation step. The reservation sum saturates fail-closed, and maximumConcurrentWorkers remains the host-wide ceiling. Insufficient memory headroom caps only prospective starts; it does not activate the global resource gate or retire healthy existing capacity. Invalid resource observations and sustained high CPU remain global gates. Target profiles change only Docker CPU, memory, memory-plus-swap, and PID limits. They cannot add the Docker socket, privileged mode, devices, or host mounts.

On an interactive development host, set resources.maximumConcurrentWorkers below the sum of pool maxCapacity values, lower resources.worker.cpus, or both, so full CI load leaves CPU for editors, terminals, and other interactive tools. The cpuBlockPercent gate does not cover that case: it reacts only to host-wide CPU held above the threshold for cpuObservationWindow, not to interactive latency.

To lower the cap only while someone is at the keyboard, set resources.interactiveMaximumConcurrentWorkers (1 to maximumConcurrentWorkers) and resources.interactiveIdleThreshold (for example 10m). Each reconcile reads the last keyboard or mouse input in the controller's Windows session; the interactive cap applies until input has been idle for the threshold, which also keeps short pauses from flapping it. A temporary capacity override wins over it, lowering the cap never stops running jobs, and a host that cannot report input keeps the full cap. Omit both fields (schema version 2 only) to turn the feature off; there is no zero "pause while present" value. On melo-lap-001, 8 is a starting value (judgment, from the #391 replay).

Listener capacity uses a per-pool memory Schmitt trigger. A decrease crosses the raw worker-memory boundary immediately, preserving the fail-closed admission contract. An increase must additionally clear memoryCapacityIncreaseMarginPercent of that target's effective worker memory; inside the band, the last advertised capacity is retained. The margin applies to capacity growth only and is not charged as a permanent memory reservation. Schema version 1 remains accepted without the margin and preserves the legacy zero-margin behavior; the margin is required only by schema version 2.

Credential boundary

The organization host App has only organization Self-hosted runners: write; each physical host gets a distinct private key. Personal-repository host credentials are introduced only after the organization soak gate.

ci-runner secret import validates RSA PKCS#1/PKCS#8 PEM, BitLocker protection, current-user DPAPI protection, and exact current-user/SYSTEM ACLs. Before it reports success, it reads the protected destination back under the current identity and verifies its fingerprint and import metadata. On Windows, the import keeps the exact source file open with read and delete access while withholding write/delete sharing. It rechecks that file identity at commit and uses handle-bound deletion, so a moved, missing, or replacement pathname is never deleted as though it were the imported PEM. If that identity check or deletion is ambiguous, the verified DPAPI destination is retained and the command fails with explicit manual-cleanup instructions. Failures before the source-deletion commit leave the PEM untouched and roll back the destination where possible. Any failed rollback or incomplete exclusive write that cannot be removed is also reported as requiring manual cleanup and is never presented as success.

The reported fingerprint is standard Base64 of the SHA-256 digest of the DER public key. This is the exact format produced by GitHub's documented openssl verification command, so the CLI value can be compared directly with the fingerprint in the GitHub App's settings.

Protected-secret schema v2 records that Base64 fingerprint. The loader remains backward compatible with schema v1 files, which stored the same SPKI SHA-256 digest as lowercase hexadecimal, and reports their fingerprint in canonical Base64 after validation. New imports never write the legacy representation.

Source cleanup is ordinary filesystem deletion. On Windows, deletion can fail for reasons such as an incompatible open handle, a read-only file, or insufficient access, and this product does not weaken those protections or silently continue. The identity lock follows Microsoft's documented CreateFile sharing contract, where withholding delete sharing also prevents rename, and commit uses SetFileInformationByHandle with FileDispositionInfo, which requires delete access and deletes the opened file object when its handle closes. Import is deliberately Windows-only; it does not emulate this guarantee with a pathname-only unlink on other systems. File deletion is not a forensic secure erase or a claim of media sanitization. Use an appropriate sanitization process when the storage medium is reused or disposed of, following NIST SP 800-88 Rev. 2. The key is decrypted only in controller memory. Rotation overlaps old and new keys until the new key can create a JIT runner, then revokes the old key.

The disposable worker necessarily contains its own decoded one-job runner credentials because that is how the stock GitHub runner connects and executes a job under one identity. Workflow code must be treated as able to inspect those ephemeral credentials. The enforced boundary is that no reusable App key, observer key, controller JWT, installation token, Docker control socket, or host filesystem enters the worker.

See the worker-image contract and GitHub's self-hosted-runner security guidance.

Supported workload

V1 is Linux x64 only. Versioned runtimes come from the same official setup actions used on ubuntu-24.04; the image adds only the documented compatibility baseline. GitHub-managed Actions caches continue to work across hosted and self-hosted execution, but cache content is untrusted and must contain no secrets.

These workloads stay GitHub-hosted:

• public repositories and fork pull requests;
• Windows jobs;
• service containers, job containers, Testcontainers, Docker actions, or jobs requiring a Docker socket;
• GPU/device/privileged workloads;
• broad cross-repository write/control-plane jobs;
• publication of this controller and worker image.

See GitHub dependency caching and Docker resource constraints.

Build, release, and rollback

Pull requests and pushes delegate Go quality to the exact-SHA-pinned reusable workflow in melodic-software/ci-workflows. It runs the reviewed analyzer set on native Linux and Windows, module tidiness and verification plus race-enabled tests on Linux, ordinary tests on Windows, and authenticated govulncheck analysis. The repository-local Go build lane only cross-compiles the two Windows executables on Ubuntu; it does not repeat those checks. Committed fuzz seeds run in ordinary tests, while the weekly schedule and manual dispatch actively fuzz both targets for 30 seconds each.

Other gates include Actionlint, Zizmor, strict configuration tests, and the live worker-image verifier. Release tags first rerun the complete read-only gate against the exact tagged source. Only the dependent publication job receives the combined job-scoped contents:write, packages:write, attestations:write, artifact-metadata:write, and id-token:write grant; the later image-promotion job retains only contents:read and packages:write. Worker provenance explicitly requests an organization artifact storage record, and publication fails closed unless the pinned attestation action returns at least one numeric storage-record ID.

Releases produce an immutable pair:

• versioned Windows ZIP and SHA-256 checksums;
• exact OCI worker image digest;
• controller and worker SBOM/provenance attestations;
• a compatibility manifest tying source SHA, controller, image, runner, Scale Set Client, Go, PowerShell, Buildx, BuildKit, and SBOM-generator pins together.

Dependencies and Actions are exact pins. Release publication resolves official sources and fails when a pin has been behind for more than 14 days or a critical release is pending; updates remain reviewed and are never auto-merged. Deployment uses a versioned install directory plus current junction; how many known-good pairs are retained is a floor, stated once in the freshness policy.

Rollback order is: drain without killing work, restore the prior immutable pair, restore the prior reusable-workflow SHA if needed, then use Re-run all jobs for affected workflows. The first step used to be flipping routing to hosted-only; there is no routing variable to flip and no consumer-side substitute, so a fleet the rollback cannot restore means the affected jobs queue until it is back. Plan the rollback window accordingly: the fleet's availability is now the availability of every private job that routes to it, which is all of them except the ones already holding a hosted exception. There is no quick shortcut either, because widening the reason set is a governed change to the standards runner policy and each affected repository still needs its own per-job exceptions entry and a runs-on change before anything routes differently.

Troubleshooting

A completed one-job worker's captured runner log can contain an ERR BrokerServer line carrying a TaskCanceledException when the stock runner cancels its broker long-poll at teardown. This is expected teardown noise, not a failed job or worker; see Cancellation and runner shutdown noise.

Further documentation

• Worker image and isolation contract
• OpenTelemetry observability
• Immutable releases, freshness, and rollback
• Deferred capabilities and non-workaround boundaries