Skip to content

Pin tool-capable OpenRouter free models in OpenCode review pool#619

Merged
seonghobae merged 18 commits into
mainfrom
seonghobae-add-openrouter-free-model-support
Jul 24, 2026
Merged

Pin tool-capable OpenRouter free models in OpenCode review pool#619
seonghobae merged 18 commits into
mainfrom
seonghobae-add-openrouter-free-model-support

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Why

After #614 landed, live review runs showed the openrouter/openrouter/free candidate failing every pool cycle with output did not include a valid control conclusion. The openrouter/free slug is a router, not a model: it picks an arbitrary free model per request (observed: content-safety and small instruct models) that cannot produce the reviewer's required structured control block. With GitHub Models and direct OpenAI currently failing, this left the whole org review queue exhausting its 12000s budget per PR and returning MODEL_OUTPUT_UNAVAILABLE.

What

  • Replace the router candidate with two pinned, tool-capable OpenRouter free models:
    • openrouter/nvidia/nemotron-3-ultra-550b-a55b:free (1M context)
    • openrouter/poolside/laguna-m.1:free (262k context)
  • Update the openrouter provider models block in opencode-review.yml accordingly (tool_call enabled, real context limits).
  • Update every contract test that pins the exact candidate list: tests/test_opencode_agent_contract.py (candidate pairs + both exact OPENCODE_MODEL_CANDIDATES strings) and the three assertions in scripts/ci/test_strix_quick_gate.sh.
  • Remove the temporary openrouter-diag.yml diagnostic workflow used to root-cause the failure (free-tier 50 req/day limit, since resolved by purchasing credits; verified is_free_tier: false and all OpenCode flag variants succeed).

Strix is untouched: its opt-in openrouter/free gate works via LiteLLM and stays as-is.

Verification

  • Targeted contract tests pass locally (test_opencode_model_pool_sets_high_effort_for_capable_candidates, test_workflow_provisions_sandbox_tool_and_reviewer_agent, job-timeout test).
  • Full suite + strix quick gate run in CI on Linux (the bash-subprocess tests hang on the local Windows machine).

Note: this PR is reviewed by main's workflow, which still uses the failing router candidate; if review cannot conclude, admin intervention may be needed as with #614.

seonghobae and others added 16 commits July 23, 2026 10:46
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Write trimmed provider API keys to llm_api_key_file in Strix workflow and lock this behavior in the workflow contract self-test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The openrouter/free router selects arbitrary free models, most of which
cannot produce the reviewer's structured control block, so every cycle
failed with 'output did not include a valid control conclusion'. Pin
nvidia/nemotron-3-ultra-550b-a55b:free (1M context) and
poolside/laguna-m.1:free (262k) instead, update the openrouter provider
model entries, and update all contract tests that pin the candidate
list. Remove the temporary OpenRouter diagnostic workflow.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 24, 2026 03:34

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Pins OpenRouter’s tool-capable free models in the OpenCode review model pool to avoid the nondeterministic openrouter/*/free router selecting models that can’t emit the required structured control block (which was causing repeated pool-cycle failures/timeouts).

Changes:

  • Replace the OpenCode pool’s OpenRouter router candidate with two pinned OpenRouter free models and add an explicit openrouter provider block in opencode-review.yml.
  • Extend the OpenCode model-pool runner to skip openrouter/* candidates when OPENROUTER_API_KEY is not configured.
  • Update Strix workflow + self-test contracts to recognize OpenRouter (openrouter/free) in the allowlist/errors and route provider-scoped key/base inputs accordingly.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tests/test_opencode_agent_contract.py Updates contract expectations for OpenCode model candidates/provider blocks to include pinned OpenRouter models.
scripts/ci/test_strix_quick_gate.sh Updates Strix/OpenCode workflow contract assertions for OpenRouter strings/provider-mode routing and candidate lists.
scripts/ci/strix_quick_gate.sh Adjusts API-base resolution to support cross-provider fallback routing for github_models/* models (e.g., when primary uses OpenRouter).
scripts/ci/run_opencode_review_model_pool.sh Adds OpenRouter candidate detection + credential-missing skip logic.
scripts/ci/emit_opencode_failed_check_fallback_findings.sh Updates known-missing-string evidence matcher to the new Strix unsupported-model error text.
requirements-strix-ci.txt Adds an explicit pyasn1 pin.
requirements-strix-ci-hashes.txt Updates hash-locked pyasn1 version/hashes to match the new pin.
.github/workflows/strix.yml Adds OpenRouter provider-mode handling (secrets gating, key/base preparation, model file normalization) and updates error strings.
.github/workflows/opencode-review.yml Adds OpenRouter provider configuration + pinned free models, injects OPENROUTER_API_KEY, and updates OPENCODE_MODEL_CANDIDATES.
Comments suppressed due to low confidence (1)

.github/workflows/strix.yml:612

  • The “Prepare GitHub Models fallback credentials” step now runs for both direct-OpenAI and OpenRouter provider modes, but the inline comment + ::notice:: message still says “Direct-OpenAI” only. Also, the fallback token is written using the CR/LF-sanitized value rather than the trimmed value, which is inconsistent with the earlier LLM key handling and can break auth if the secret has leading/trailing whitespace.
        if: steps.gate.outputs.provider_mode == 'openai_direct' || steps.gate.outputs.provider_mode == 'openrouter'
        env:
          GITHUB_MODELS_FALLBACK_TOKEN: ${{ secrets.STRIX_GITHUB_MODELS_TOKEN || github.token }}
        run: |
          # Direct-OpenAI scans keep GitHub Models candidates as fallbacks, so

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/strix.yml
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@seonghobae
seonghobae enabled auto-merge (squash) July 24, 2026 06:09
8192 output tokens can truncate a long review before the control block,
reproducing the exact no-control-conclusion failure this PR fixes. The
live provider caps are 65536 (nemotron-3-ultra) and 32768 (laguna-m.1).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 24, 2026 07:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread .github/workflows/opencode-review.yml
@opencode-agent
opencode-agent Bot disabled auto-merge July 24, 2026 07:45
@seonghobae
seonghobae enabled auto-merge July 24, 2026 07:56
@seonghobae
seonghobae merged commit bf1d60b into main Jul 24, 2026
54 of 55 checks passed
@seonghobae
seonghobae deleted the seonghobae-add-openrouter-free-model-support branch July 24, 2026 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants