You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Spec-review Trials have no Trial inputs or PR service #1251
#1018 imported 30 approved Spec-review Eval tasks with no [trial] table. spec-review depends on src/prompt.rs, src/session.rs, the Harness files and others, so most releases select it (#1022). Each of its Trials then stops with missing Trial inputs, after both Arms are built. Grilled in docs/grill-with-docs/done/1251-spec-review-trial-inputs.md.
The import fills Spec-review [trial] tables from the replay manifest, as it already does for Code review: issue_url and pr_url are the Spec's original URLs, base its Base branch, branch its Spec branch. It writes the table only when it is missing and keeps later local edits, so a rerun completes tasks already imported (Q1.2).
An Experiment refuses before building either Arm when any selected approved task has no [trial] table, naming each one. Release Experiments included (Q1.3).
No PR service.add evals -integrate with langfuse / oss eval tool? #698's Q3.11 stands: a Spec-review Trial keeps the production Spec review prompt, its gh calls fail inside the sandbox, and only its reports are graded (Q1.1). Remove "supply fixture services or disposable PRs" from docs/eval-tasks.md and docs/trial.md, and say this instead.
Out of scope: a live Spec-review Trial, which is a separate ready-for-human issue (Q1.4, Q2.1).
Activity
Merge pull request #1256 from JacobStephens2/grill-1251
JacobStephens2 commentedon Oct 24, 2026
Closed by #1272, merged into issue-977 by a thirdshift Ticket run with Self-merge.