Plan AgentV -> Opik export integration

[christso]

Context

We evaluated whether Opik can replace AgentV for CI gating. Decision: do not replace AgentV. AgentV should remain the CI authority and portable coding-agent harness; Opik should be the observability/eval/repair platform AgentV exports to.

Durable handoff findings are in the research wiki:

https://github.com/tsoyang-org/ai-research-wiki/blob/ai-research-wiki-coordinator/concepts/agentv-opik-integration-findings.md

Decision

Build AgentV -> Opik first, not opik run agentv.

Expected flow:

agentv run evals/*.yaml
  -> AgentV computes pass/fail and exits nonzero on gates
  -> AgentV exports traces/scores/run metadata to Opik
  -> Opik stores experiments, traces, feedback scores, dashboards
  -> selected Opik failures can become future AgentV regression cases

Braintrust is out of scope for this workstream. For the current need, AgentV + Opik covers CI gating, observability, experiment history, dashboards, trace-to-test-suite growth, and repair-loop workflows.

Proposed workstreams

1. Document the boundary

   • AgentV is the runner/gate.
   • Opik is an export target and repair/observability platform.
   • Keep AgentV's existing CI exit semantics authoritative.

2. Add Opik OTLP export recipe

   • Use AgentV's OpenTelemetry direction as the first trace path.
   • Document Opik endpoint config for local self-host and cloud/enterprise.
   • Add required span attributes: agentv.run_id, agentv.eval_id, agentv.test_id, agentv.target, agentv.provider, agentv.workspace_mode, agentv.gate_status.

3. Add agentv export opik for completed runs

   • Input: .agentv/results/runs/<run-id>/index.jsonl.
   • Output: Opik experiment with per-test items, feedback scores, and trace links.
   • Keep SDK/API usage isolated and version-pinned.

4. Add explicit Opik failure importer

   • Proposed shape: agentv import opik --project <name> --filter <query> --out cases/opik-failures.jsonl.
   • Convert selected failed Opik traces/test-suite items into AgentV regression cases.
   • Require review before commit because production traces may contain secrets/customer data.

5. Publish CI recipe

   • CI still fails on AgentV gates.
   • Opik upload runs best-effort/always() after the AgentV run so failures are visible.
   • Missing Opik credentials should not block local OSS users unless export is explicitly required.

Acceptance criteria

• agentv run ... --export-opik or agentv export opik <run-id> uploads one completed run to Opik.
• Opik shows one experiment for the AgentV run with per-case scores.
• Each case links back to AgentV run/test IDs and local artifact paths.
• Required evaluator failures are visible as feedback scores and metadata.
• AgentV CI behavior is unchanged: gates still fail the job via AgentV's own exit code.
• Export can be disabled, best-effort, or required through explicit config.

Request for coordinator/agents

Please brainstorm the implementation shape, then break this down into beads. Suggested initial beads:

• docs: AgentV/Opik boundary and CI authority
• otel: Opik OTLP export recipe
• exporter: agentv export opik adapter
• schema: export metadata mapping for run/test/score/gate fields
• importer: Opik failure -> AgentV regression cases
• ci: GitHub Actions example with best-effort export
• security: redaction and metadata-only export mode

Source anchors

• Wiki handoff: https://github.com/tsoyang-org/ai-research-wiki/blob/ai-research-wiki-coordinator/concepts/agentv-opik-integration-findings.md
• Opik: https://github.com/comet-ml/opik
• Opik docs: https://www.comet.com/docs/opik/

[christso]

Coordination is now active in Beads for the WTG Opik workstream.

Active Beads:

• Worker: av-vwa.15 (P2 Opik: install Opik and port WTG PR 679 E2E evals)
• Coordinator: av-vwa.15.1 (Coordinate WTG Opik E2E eval workstream)

Worker WorkMux session: wtg-opik-e2e-evals
Worker branch: private/av-vwa-15-opik-e2e
Worker path: /home/entity/projects/EntityProcess/wtg-ai-prompts-experiment__worktrees/wtg-opik-e2e-evals

Current status: worker is active, has read the AgentV/Opik handoff via authenticated GitHub access after anonymous wiki access 404ed, and is pursuing a dry-run-first Opik adapter path because the local opik package is not installed. AgentV remains the CI authority; Opik remains the trace/experiment/test-suite/feedback/dashboard target. Braintrust remains out of scope.

[christso]

Coordination update: a dedicated WTG AI Prompts coordinator has been spawned for this workstream.

Beads:

• Coordinator bead: av-vwa.15.1 — Coordinate WTG Opik E2E eval workstream
• Worker bead: av-vwa.15 — P2 Opik: install Opik and port WTG PR 679 E2E evals

WorkMux sessions:

• Coordinator: wtg-ai-prompts-coordinator
  • branch: private/av-vwa-15-1-opik-coordinator
  • path: /home/entity/projects/EntityProcess/wtg-ai-prompts-experiment__worktrees/wtg-ai-prompts-coordinator
• Worker: wtg-opik-e2e-evals
  • branch: private/av-vwa-15-opik-e2e
  • path: /home/entity/projects/EntityProcess/wtg-ai-prompts-experiment__worktrees/wtg-opik-e2e-evals

The coordinator is responsible for monitoring the worker, resolving safe blockers, breaking follow-up findings into beads, and posting only material milestones here. Braintrust remains out of scope for this workstream; the target stack is AgentV + Opik.

[christso]

WTG Opik PR 679 worker milestone for Bead av-vwa.15.

Branch/worktree:

• branch: private/av-vwa-15-opik-e2e
• path: /home/entity/projects/EntityProcess/wtg-ai-prompts-experiment__worktrees/wtg-opik-e2e-evals

Artifacts added in EntityProcess/wtg-ai-prompts-experiment:

• framework-parity/pr-679/opik/pr50857_opik_adapter.py
• framework-parity/pr-679/opik/README.md
• framework-parity/tests/test_pr679_opik_adapter.py

Result:

• Installed and inspected opik==2.0.64 in ignored .agentv/opik-venv only; no repo-level dependency was added.
• Created a dry-runable Opik test-suite seed for both PR 679 E2E cases.
• Created a completed-run dry-run export path against local AgentV run pr679-pr50857-clean-2026-06-08T05-42-55Z, preserving AgentV case IDs, scores, grader scores, assertion verdicts, artifact paths, source PR provenance, and redacted tool trace summaries.
• Live upload remains blocked in this shell because OPIK_API_KEY, OPIK_WORKSPACE, OPIK_PROJECT_NAME, and OPIK_URL_OVERRIDE were unset; no raw secrets were printed or persisted.

Mapping findings useful for agentv export opik:

• Eval run -> Opik experiment plus root trace metadata; AgentV remains CI gate authority.
• Test/case -> Opik test-suite item/dataset item, preserving AgentV eval path, case ID, source PR refs, workspace template, and type:file source paths.
• Final score, per-grader scores, and assertion pass/fail -> Opik feedback scores.
• Skill-trigger -> Opik text surrogate/global assertion plus feedback score from completed runs; the AgentV skill-trigger remains authoritative for tool trajectory.
• Transcript tool calls -> Opik tool-span plan with tool name, call ID, timing, counts, and SHA-256 hashes of redacted inputs/outputs.

[christso]

Coordinator close-out for WTG Opik workstream.

Worker av-vwa.15 has a final result comment and WorkMux session wtg-opik-e2e-evals is done. Coordinator av-vwa.15.1 is closed.

Follow-up Beads under av-vwa.15:

• av-vwa.15.2 — first-class agentv export opik completed-run exporter
• av-vwa.15.3 — redaction and metadata-only export policy
• av-vwa.15.4 — safe credentialed live-upload smoke path
• av-vwa.15.5 — PR 679 adapter raw-output edge hardening, resolved by the worker follow-up

Post-review verification update: the worker added the raw-output missing-path guard and regression test; focused Opik tests now pass 4 cases and combined parity tests pass 8 cases.

Main milestone remains: #1388 (comment)