Skip to content

Benchmark PRs on machines chosen by machine:<ec2-type> labels#1010

Merged
alexey-milovidov merged 2 commits into
mainfrom
pr-benchmark-machine-labels
Jul 20, 2026
Merged

Benchmark PRs on machines chosen by machine:<ec2-type> labels#1010
alexey-milovidov merged 2 commits into
mainfrom
pr-benchmark-machine-labels

Conversation

@alexey-milovidov

@alexey-milovidov alexey-milovidov commented Jul 20, 2026

Copy link
Copy Markdown
Member

Summary

  • The automatic PR benchmark always ran on the default c6a.4xlarge, with no way to configure it. Some systems cannot work there — e.g. Hyrise (Add Hyrise #883), an in-memory DBMS whose encoded hits table alone likely exceeds the machine's 32 GB.
  • A maintainer can now add one or more machine:<ec2-type> labels to a PR (e.g. machine:c6a.metal) to override the default machine — one run per machine. Adding such a label relaunches the benchmark immediately (the labeled trigger is filtered so other labels don't launch anything, and a state guard keeps closed PRs inert).
  • machine:all, machine:all-amd and machine:all-arm expand to the machine sets of the daily runs (mirroring benchmark-daily.yml), deduplicated against explicitly labeled machines.
  • Labels for every machine type plus the three aliases already exist on the repository.
  • Security/cost posture is unchanged: labels require triage access, pull_request_target still takes the workflow from the base branch, and every launch remains gated by the benchmark-approval environment.

Test plan

  • YAML validated; the jq label extraction and alias expansion tested for: a single machine, all-arm, all, all-amd + all-arm, all + explicit duplicate (deduplicated), two explicit machines, and no labels (falls back to c6a.4xlarge)
  • After merge: add machine:c6a.metal to Add Hyrise #883 and check that the launched run uses c6a.metal

🤖 Generated with Claude Code

alexey-milovidov and others added 2 commits July 20, 2026 21:56
The PR benchmark always ran on c6a.4xlarge, but some systems cannot
work there — e.g. Hyrise (#883), an in-memory DBMS whose encoded hits
table alone exceeds the machine's 32 GB. The manual workflow can target
any machine, but its runs are invisible on the PR until the results
sink picks them up, and every re-run needs a manual dispatch.

A maintainer can now add one or more `machine:<ec2-type>` labels to a
PR (e.g. `machine:c6a.metal`) to override the default machine, one run
per label. Adding such a label relaunches the benchmark immediately;
other labels don't trigger anything. Labels can only be set by users
with triage access, and every launch stays gated by the
`benchmark-approval` environment, so the cost control is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`machine:all`, `machine:all-amd` and `machine:all-arm` labels expand
to the machine sets of the daily runs (the lists mirror
benchmark-daily.yml), deduplicated against explicitly labeled machines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alexey-milovidov alexey-milovidov self-assigned this Jul 20, 2026
@alexey-milovidov
alexey-milovidov merged commit 800b101 into main Jul 20, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant