Skip to content

CI runner CPU requests overpack the workers and starve colocated workloads #14

Description

@linear-code

Problem

The GitHub Actions runner templates request 500m (lean) / 250m+250m (dind) CPU while their limits allow 4-12 CPUs. The comment even records the intent: "many of these schedule where one dind runner did." On 16-core workers the scheduler therefore stacks up to eight bursting runners beside the dev environment, and under concurrent pull requests the node saturates.

A day of creativetoolkits CI produced the full symptom set, each observed repeatedly:

  • [vitest-pool]: Failed to start forks worker ... Timeout waiting for worker to respond in runs where all 723 tests passed
  • module imports measured at 5.3s and a checkout+setup that blew a 10 minute budget
  • fatal: unable to access 'https://github.com/...': SSL connection timeout during the runner's own fetch
  • GoTrue Processing this request timed out and Next The destination stream closed early from the dev Supabase/app the e2e suite depends on - the CI load starves the very environment it tests against

Change

Raise the runner CPU/memory requests to what a job actually sustains (lean 2 CPU/4Gi; dind 1+2 CPU) so kube-scheduler becomes the concurrency throttle. Surplus demand then PENDS visibly instead of degrading every neighbour. Limits unchanged; memory is abundant (~128Gi workers) and stays modest.

Acceptance

  • Runner pods carry the new requests after ArgoCD sync.
  • Under a multi-PR burst, excess runner pods go Pending rather than nodes saturating.
  • The failure classes above stop appearing in runs whose tests pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions