Record the pull request number in the results as clickbench_pr#994
Merged
Conversation
The log gets a 'ClickBench PR:' line (empty for runs of main), parsed into the new clickbench_pr column of sink.results, and PR runs are excluded from collect-results.sh so they cannot be published as official results. repo and branch are now empty by default: when clickbench_pr is set, run-benchmark.sh derives them from the PR's head via the GitHub API; otherwise they fall back to ClickHouse/ClickBench and main. The manual workflow gets a pr input, so any PR can be benchmarked by number. The '(\d*)' in the parser matters: the header line comes first in the log with no digits for runs of main, and the empty match there stops the search before any spoofed 'ClickBench PR: N' in untrusted output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #993: benchmark runs launched for a pull request are now marked, so the future collection step can tell them apart from the runs of main — without this, a PR run of an existing system would land in
sink.resultsindistinguishable from the daily run and could be published as the official result.cloud-init.sh.inwrites aClickBench PR: <number>line into the log header (empty for runs of main).sink.parserextracts it into a newclickbench_pr UInt32column ofsink.results(prepare-database.sql). The pattern is(\d*), not(\d+): the header line comes first in the log, and its empty match for runs of main stops the search before any spoofedClickBench PR: Nin untrusted benchmark output.collect-results.shfiltersclickbench_pr = 0, so PR runs are never published.repoandbranchare now empty by default: whenclickbench_pris set,run-benchmark.shderives the PR's head repository and branch from the GitHub API; otherwise they fall back toClickHouse/ClickBenchandmain. The PR workflow therefore passes only the PR number, and the manual workflow takes an optionalprinput — any PR can be benchmarked by number.Requires two statements on the sink database before merging (add the column, update the parser view); until the view is updated, PR runs would be recorded with
clickbench_pr = 0.🤖 Generated with Claude Code