Add instructions: research experiment discipline - #2854
Open
GuoCheng24 wants to merge 1 commit into
Open
Conversation
Contributor
🔒 PR Risk Scan ResultsScanned 1 changed file(s).
✅ No matching risk patterns were detected in changed files.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
instructions/research-experiment-discipline.instructions.md— guidance for repositories where Copilot writes evaluation harnesses, benchmark sweeps, or any code that reports a performance number.Falls under "Development methodologies and best practices" / "Testing strategies and quality assurance" in CONTRIBUTING. The gap it addresses: in research code the common failure is not a crash but a number that looks great and is wrong — models happily write evaluation code that tunes on the test set, compares against unmatched baselines, or reports point estimates inflated by group leakage. The instructions are specific and mechanical: null models scored first (freeze on a well-scoring null), calibration/held-out separation on the unit of independence, a held-out access budget, published-baseline reproduction before improvement claims, paired error bars, and every new check demonstrated to fail on a deliberately broken input.
Distilled from real failures in agent-run experiment campaigns; the longer-form methodology with runnable demonstrations lives at https://github.com/GuoCheng24/breakthrough-harness.
mainnpm startrun;docs/README.instructions.mdupdated (the script also produced unrelated churn indocs/README.agents.mdfrom live MCP-registry data — left out to keep the diff minimal; happy to include if the check requires it)