Skip to content

Spec-review Trials have no Trial inputs or PR service #1251

Closed

Description

@JacobStephens2

This was generated by AI during triage.

Parent

#977

What to build

#1018 imported 30 approved Spec-review Eval tasks with no [trial] table. spec-review depends on src/prompt.rs, src/session.rs, the Harness files and others, so most releases select it (#1022). Each of its Trials then stops with missing Trial inputs, after both Arms are built. Grilled in docs/grill-with-docs/done/1251-spec-review-trial-inputs.md.

  • The import fills Spec-review [trial] tables from the replay manifest, as it already does for Code review: issue_url and pr_url are the Spec's original URLs, base its Base branch, branch its Spec branch. It writes the table only when it is missing and keeps later local edits, so a rerun completes tasks already imported (Q1.2).
  • An Experiment refuses before building either Arm when any selected approved task has no [trial] table, naming each one. Release Experiments included (Q1.3).
  • No PR service. add evals -integrate with langfuse / oss eval tool? #698's Q3.11 stands: a Spec-review Trial keeps the production Spec review prompt, its gh calls fail inside the sandbox, and only its reports are graded (Q1.1). Remove "supply fixture services or disposable PRs" from docs/eval-tasks.md and docs/trial.md, and say this instead.

Out of scope: a live Spec-review Trial, which is a separate ready-for-human issue (Q1.4, Q2.1).

Acceptance criteria

  • Importing made-up Multi-family review: have another model family review a Run's work #429-style Spec manifests writes each Spec-review task's [trial] table from them; a rerun fills a missing table and leaves an edited one unchanged.
  • An Experiment selecting an approved task without a [trial] table fails before any Arm is built, naming the task.
  • A Spec-review Trial with a fake Harness runs through to its reports with no PR service.
  • The docs no longer ask for fixture services or disposable PRs.

Activity

  1. added
    ready-for-agentFully specified, ready for an AFK agent
    and removed
    needs-triageMaintainer needs to evaluate this issue
    on Oct 24, 2026
  2. added a commit that references this issue on Oct 24, 2026
  3. JacobStephens2 commented on Oct 24, 2026

    @JacobStephens2
    OwnerAuthor

    Closed by #1272, merged into issue-977 by a thirdshift Ticket run with Self-merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestready-for-agentFully specified, ready for an AFK agent

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions