Share benchmark suite across weekly and nightly runs - #42
Merged
Merged
Conversation
gauravtiwari
marked this pull request as ready for review
October 5, 2026 12:20
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run source checks hourly at :11 and fan out independently by case. Weekly stable and nightly canary fresh runs preflight all 31 targets, resolve one CLI tag, and dispatch a matrix with individual and aggregate receipts. The maintained suite plus the new families contains 24 source cases.
Source publication reconstructs permitted pin changes, checks trusted history and the inspected harness, records dispatch intent, and conditionally commits against the branch head. Each case reconciles its previous runs independently. Deno retains adjacent commits, OBS retains build-relevant pairs, and Zed promotes only after verified builds. Recipe changes for Immich, Qdrant, msgpack and Zed are documented.
All 24 cases now declare rolling paths. Native, REAPI, Nix and OBS workflows retain the case/variant/provider/phase labels and canonical comparison reports. Nix workers import one checksummed dependency seed so strict provider comparisons do not depend on independently rebuilding a nondeterministic dependency. The earlier failed Zed Nix observation remains retained.
Central activation remains off: merging registers workflows for manual qualification without replacing historical schedule owners. Activation is pending live changed-source qualification, publication rehearsal, retirement of replaced historical triggers, and a compatible stable CLI release. Stable v1.33.0 lacks --reapi-port; canary vcli-canary-7a5b27146ebe supports the suite. Cache capacity is 200 GB.
Validation: 245 Ruby tests / 1,955 assertions passed; case, registry, workflow, report-contract and Actionlint checks passed. All rolling targets plan successfully. Hosted CodeQL passes after validating controller revisions against trusted branch history. The 31-target canary dry run passed without requesting builds. The complete source controller rehearsal, corrected Nix screens and isolated native rolling qualification are in progress. These are execution gaps, not completed qualification. See docs/cadence.md for the cutover gates.