Review Workshop eval trajectory changes - #477
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Preview:
|
|
Author identity unknown *** Please tell me who you are. Run git config --global user.email "you@example.com" to set your account's default identity. fatal: empty ident name (for runner@runnervmlun5p.22vpi1papmiehlbrkehlsftqdc.bx.internal.cloudapp.net) not allowed |
|
@AshishKumar4 Bonk workflow failed. Check the logs for details. View workflow run · To retry, trigger Bonk again. |
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
| uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0 | ||
| with: | ||
| name: workshop-evals-baseline | ||
| path: artifacts/baseline |
There was a problem hiding this comment.
[P1] Keep downloaded artifacts out of the git worktree
These downloads make artifacts/ untracked before opencode github run starts. The action treats that dirty worktree as agent changes and enters its commit path; in this PR’s run, it then failed with Author identity unknown and posted no trajectory synthesis. Download under an ignored path such as .wrangler/... (and update the prompt paths), or exclude this directory from Git before invoking Bonk.
|
Posted 1 actionable inline finding. |
79348b1 to
f3b1ef6
Compare
567e19d to
251a515
Compare
|
Performance is inconclusive: all nine trials failed in both runs, with null pass-rate deltas. Candidate mean durations increased by 0.51–0.84 seconds across tasks. Both runs stopped before any model turn or tool call due to connection errors. These timing shifts likely reflect failure timing, and model variance cannot be assessed. |
f3b1ef6 to
2cf0dcb
Compare
251a515 to
9269040
Compare
|
LGTM! |
2cf0dcb to
3c8d621
Compare
9269040 to
bf4ef34
Compare
| opencode_version: 1.18.24 | ||
| prompt: | | ||
| This is an advisory trajectory synthesis, not a code review. Read exactly | ||
| artifacts/baseline/results.json, artifacts/candidate/results.json, |
There was a problem hiding this comment.
[P1] Reformat trajectories before asking OpenCode to read them
The uploaded Vitest reports are minified JSON: in a real run each results.json was a single roughly 35 KB line. OpenCode's read tool truncates lines after 2,000 characters, and with every other tool denied it cannot recover the remainder, so this prompt exposes only the start of the first baseline/candidate trial and the requested trajectory-level explanation is not based on the other trials. Pretty-print or otherwise split both downloaded reports into readable lines before invoking Bonk (and point the prompt at those files).
|
Posted 1 actionable inline finding. |
3c8d621 to
cb4d71b
Compare
bf4ef34 to
2296df5
Compare
|
LGTM! |
cb4d71b to
52f4704
Compare
2296df5 to
6f4c82c
Compare
52f4704 to
31fd93b
Compare
|
LGTM! |
6f4c82c to
273a43d
Compare
| - name: Check out candidate | ||
| uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
| ref: ${{ github.event.pull_request.head.sha }} |
There was a problem hiding this comment.
[P1] Do not start OpenCode from the candidate checkout
opencode github run automatically loads .opencode/plugins/* and project opencode.json from its current worktree before model tool permissions apply. Because this job checks out the PR head and then supplies the Cloudflare gateway secrets and a NO_PUSH App token, a PR that also changes any eval-trigger path can add a plugin whose initialization executes arbitrary code and reads those credentials; the inline permission map does not sandbox plugin code. Check out the trusted base SHA for this job (the generated diff can still target HEAD_SHA), or invoke OpenCode from a trusted worktree that cannot contain candidate config or plugins.
|
Posted 1 actionable inline finding. |
31fd93b to
1a9a280
Compare
273a43d to
8185820
Compare
1a9a280 to
cad4717
Compare
8185820 to
9a6fc8c
Compare
cad4717 to
108e487
Compare
|
LGTM! |
9a6fc8c to
b3759de
Compare
1 participant
Stacked on #476.
After the deterministic baseline/candidate comparison completes, download and pretty-print both raw trajectory artifacts, then run the existing Bonk action with a fixed 3–4 sentence prompt. The report covers direction, measured changes, the strongest trajectory-level explanation, and uncertainty from model variance.
Bonk starts from the trusted base checkout, reads a precomputed baseline-to-candidate diff, and receives a no-push token. OpenCode denies every tool except file reads, downloaded inputs are excluded from Git status, and the advisory step does not affect the deterministic eval result.