Skip to content

[rush reporter][R8] Qualify AI reporter auto-selection #5981

Description

Parent: #5858

Blocked by: #5977, #5978

Goal

Prove that the AI reporter improves agent outcomes before enabling environment-based automatic selection.

Scope

  • Build a deterministic corpus of representative install, configuration, dependency, build, cache, network/auth, and plugin failures.
  • Compare legacy/plaintext/AI output for remediation success, context consumption, determinism, secret handling, and full-log discoverability.
  • Validate built-in COPILOT_CLI detection plus configured agentEnvironmentVariables.
  • Define pass/fail thresholds and store machine-readable gate results.
  • Keep auto-selection disabled until the corpus gates pass.

Acceptance criteria

  • The corpus is deterministic and runnable in CI.
  • AI output never includes classified secrets and does not regress producer privacy.
  • Thresholds for remediation quality, output size, determinism, and log-path correctness are documented and enforced.
  • Passing the gate enables agent auto-selection only on the opt-in reporter path before Rush 6.
  • RUSH_REPORTER=legacy remains authoritative.

DAG

Unblocks R9.

Part of #5858.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementThe issue is asking for a new feature or design change

    Type

    No type

    Projects

    Status
    Needs triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions