Repository navigation
fix: support GPT-5.6 across Codex authentication - #4056
Conversation
Co-authored-by: openhands <openhands@all-hands.dev>
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Coverage Report •
|
||||||||||||||||||||||||||||||||||||||||||||||||||
Co-authored-by: openhands <openhands@all-hands.dev>
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
❌ QA Report: FAIL
The model-list and subscription surfaces now expose GPT-5.6, but a real Codex ACP run with the newly exposed GPT-5.6 models did not complete while the same path with GPT-5.5 did.
Does this PR achieve its stated goal?
No, not yet based on functional QA. The PR does update the SDK and Agent Server surfaces from the old Codex package/IDs to @agentclientprotocol/codex-acp@1.1.2, api-key, agent-full-access, and GPT-5.6 model entries. However, when I exercised the actual Codex ACP adapter with configured LLM credentials, gpt-5.5 returned the expected response, while gpt-5.6/gpt-5.6-sol failed before producing the requested answer.
| Phase | Result |
|---|---|
| Environment Setup | ✅ make build completed on the PR checkout and isolated origin/main worktree |
| CI Status | 🟡 34 checks successful; qa-changes was still in progress; cleanup-on-approval skipped |
| Functional Verification | ❌ SDK/API exposure works, but real Codex ACP GPT-5.6 execution failed |
Functional Verification
Test 1: SDK subscription and ACP provider surfaces before/after
Step 1 — Establish baseline on origin/main:
Ran cd /tmp/oh-qa-main && uv run python /tmp/oh_qa_sdk_probe.py:
{
"codex_default_command": ["npx", "-y", "@zed-industries/codex-acp@0.16.0"],
"codex_default_session_mode": "full-access",
"codex_gpt56_models": [],
"subscription_gpt56_models": [],
"create_subscription_llm_gpt56": {
"ok": false,
"error_type": "ValueError",
"error": "Model 'gpt-5.6' is not supported for subscription access..."
}
}This confirms the old behavior: GPT-5.6 was not exposed through the Codex provider or subscription helper.
Step 2 — Apply the PR changes:
Used the PR checkout at fix/codex-acp-gpt-5-6 / e1db501883efebee4be67249c4b705aa6f7a729d.
Step 3 — Re-run with the fix in place:
Ran uv run python /tmp/oh_qa_sdk_probe.py:
{
"codex_default_command": ["npx", "-y", "@agentclientprotocol/codex-acp@1.1.2"],
"codex_default_session_mode": "agent-full-access",
"codex_gpt56_models": ["gpt-5.6", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna"],
"subscription_gpt56_models": ["gpt-5.6", "gpt-5.6-luna", "gpt-5.6-sol", "gpt-5.6-terra"],
"create_subscription_llm_gpt56": {
"ok": true,
"model": "openai/gpt-5.6",
"auth_type": "subscription"
}
}This confirms the SDK-facing registry/subscription behavior changed as intended.
Test 2: Agent Server model endpoints before/after
Step 1 — Establish baseline on origin/main:
Started uv run agent-server --host 127.0.0.1 --port 8766, then called:
GET /api/llm/models?provider=openaiGET /api/llm/models/verifiedGET /api/llm/subscription/openai/models
Observed:
{"gpt56_models": [], "legacy_codex": ["gpt-5.1-codex-mini"]}
{"openai_gpt56_models": [], "openai_has_gpt55": true}
{"gpt56_models": [], "legacy_codex": ["gpt-5.1-codex-max", "gpt-5.1-codex-mini", "gpt-5.2", "gpt-5.2-codex", "gpt-5.3-codex"]}This confirms the server did not expose GPT-5.6 before the PR.
Step 2 — Apply the PR changes:
Started the PR server with uv run agent-server --host 127.0.0.1 --port 8765.
Step 3 — Re-run with the fix in place:
Called the same endpoints on port 8765 and observed:
{"gpt56_models": ["gpt-5.6", "gpt-5.6-luna", "gpt-5.6-sol", "gpt-5.6-terra"], "count": 25}
{"openai_gpt56_models": ["gpt-5.6", "gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna"], "openai_has_gpt55": true}
{"gpt56_models": ["gpt-5.6", "gpt-5.6-luna", "gpt-5.6-sol", "gpt-5.6-terra"], "has_gpt55": true}This confirms the Agent Server HTTP surfaces now include the new GPT-5.6 entries.
Test 3: Real Codex ACP conversation execution
Step 1 — Establish a working control path:
Using the PR checkout, configured Codex ACP with the available LLM_API_KEY/LLM_BASE_URL as OPENAI_API_KEY/OPENAI_BASE_URL, then ran a real conversation through @agentclientprotocol/codex-acp@1.1.2 with acp_model="gpt-5.5":
ACP server initialized: agent_name='@agentclientprotocol/codex-acp', agent_version='1.1.2'
Authenticating with ACP method: api-key
Setting ACP session mode: agent-full-access
ACP_QA_CONTROL_OK
{
"attempted_model": "gpt-5.5",
"current_model_id": "gpt-5.5",
"run_completed": true
}
This proves the Codex ACP adapter, API-key auth, base-url routing, and the available credentials can complete a real conversation for the existing model.
Step 2 — Exercise the new GPT-5.6 entries:
Ran the same flow with acp_model="gpt-5.6" and prompt Reply exactly ACP_QA_GPT56_OK:
ACP server initialized: agent_name='@agentclientprotocol/codex-acp', agent_version='1.1.2'
Authenticating with ACP method: api-key
Could not set model 'gpt-5.6' on ACP server @agentclientprotocol/codex-acp (Invalid params); the session will use the server default
ACP prompt failed: 'NoneType' object has no attribute 'get'
{
"attempted_model": "gpt-5.6",
"current_model_id": "gpt-5.6-sol",
"error_type": "ConversationRunError",
"run_completed": false
}
I also ran the same flow with acp_model="gpt-5.6-sol":
ACP server initialized: agent_name='@agentclientprotocol/codex-acp', agent_version='1.1.2'
Authenticating with ACP method: api-key
Setting ACP session mode: agent-full-access
ACP prompt failed: 'NoneType' object has no attribute 'get'
{
"attempted_model": "gpt-5.6-sol",
"current_model_id": "gpt-5.6-sol",
"error_type": "ConversationRunError",
"run_completed": false
}
This shows the newly exposed GPT-5.6 path did not produce the requested response in the same environment where the GPT-5.5 Codex ACP control succeeded.
Unable to Verify
I did not have a local ChatGPT/Codex subscription auth file (~/.codex/auth.json was absent), so I could not independently verify the chat-gpt OAuth/subscription auth path or Canvas UI selection. I verified the API-key ACP path and the HTTP/SDK model surfaces instead.
Issues Found
- 🟠 Issue: Real Codex ACP execution with the newly exposed GPT-5.6 models failed.
gpt-5.5completed successfully through the same adapter/credentials, butgpt-5.6was rejected asInvalid paramsandgpt-5.6-solfailed withConversationRunError: 'NoneType' object has no attribute 'get'before returning the requested text.
This review was created by an AI agent (OpenHands) on behalf of the user.
VascoSch92
left a comment
VascoSch92
left a comment
There was a problem hiding this comment.
@neubig LGTM just one thing: _LEGACY_OPENAI_CODEX_MODELS is removed, i.e., it is a breaking change. Is that ok?
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
|
I think it's OK to remove that as it's a private member, thanks! |
Co-authored-by: Graham Neubig <gneubig@all-hands.dev> Co-authored-by: openhands <openhands@all-hands.dev>
5 participants
HUMAN:
I have tested this in the agent canvas, and GPT-5.6 Sol is displayed and works.
AGENT:
Why
Agent Server and Canvas pinned
@zed-industries/codex-acp@0.16.0, whose embedded Codex does not support the GPT-5.6 model family. The ordinary LLM picker also omitted GPT-5.6 because Canvas assigns bare LiteLLM model IDs to providers through the SDK's verified-model registry.Summary
@agentclientprotocol/codex-acp@1.1.2and use its current auth, session-mode, model-config, andCODEX_CONFIGcontractsIssue Number
N/A
How to Test
uv run pytest -q tests/sdk/agent/test_acp_agent.py tests/sdk/settings/test_acp_providers.py tests/sdk/llm/auth/test_openai.py tests/sdk/llm/test_model_list.py tests/sdk/test_settings.py tests/agent_server/test_llm_router.py(626 passed)uv run pre-commit run --all-files --show-diff-on-failure(passed)uv run pyright openhands-sdk/openhands/sdk/agent/acp_agent.py openhands-sdk/openhands/sdk/llm/auth/openai.py openhands-sdk/openhands/sdk/llm/llm.py openhands-sdk/openhands/sdk/llm/utils/verified_models.py openhands-sdk/openhands/sdk/settings/acp_providers.py(0 errors)src/models/acp-providers.jsonin the companion TypeScript client branch matchesACP_PROVIDERSgpt-5.6-solthrough Codex ACP, and received the exact responseACP_GPT56_MODERN_OKin conversation 5f900271. Server state reported@agentclientprotocol/codex-acp1.1.2,agent-full-access, andgpt-5.6-solviaconfigOptions.Live evidence
Setup: local checkout of
fix/codex-acp-gpt-5-6atcfb13d06f0b24e0cab79e6bf077ec62a0571f473aftermake build; Nodev22.23.1/ npm10.9.8. The live GPT-5.6 run used the existingCODEX_HOME/auth.jsonChatGPT-format auth file, withOPENAI_API_KEY,OPENAI_BASE_URL,CODEX_API_KEY, andCODEX_CONFIGunset so Codex ACP selected subscription auth.Commands and observed results:
uv run pytest -q tests/sdk/agent/test_acp_agent.py -k ExtractTokenUsage->6 passed.uv run pytest -q tests/sdk/llm/test_model_list.py::test_gpt_5_6_models_are_verified_for_openai->1 passed.uv run pytest -q tests/agent_server/test_llm_router.py->16 passed.uv run pre-commit run --files openhands-sdk/openhands/sdk/agent/acp_agent.py tests/sdk/agent/test_acp_agent.py-> passed Ruff format/lint, pycodestyle, Pyright, import rules, and tool registration checks.uv run pre-commit run --files openhands-sdk/openhands/sdk/llm/utils/verified_models.py-> passed after resolving themainmerge conflict.uv run pre-commit run --files openhands-agent-server/openhands/agent_server/llm_router.py-> passed after fixing the provider-filter CI failure.ACP_PROMPT_MAX_RETRIES=1 OPENHANDS_SUPPRESS_BANNER=1 uv run python ...launchedACPAgent(acp_command=["npx", "-y", "@agentclientprotocol/codex-acp@1.1.2"], acp_server="codex", acp_model="gpt-5.6-sol")and sentReply exactly ACP_PR4056_CHATGPT_GPT56_SOL_FINAL_OK. Do not include any other text.Observed live ACP result:
@agentclientprotocol/codex-acp1.1.2chat-gptagent-full-accesscurrent_model_id="gpt-5.6-sol",model_override_applied=true,model_via_config_option=true4.17swith exact final responseACP_PR4056_CHATGPT_GPT56_SOL_FINAL_OKand accumulated cost0.015305000000000001Additional API-key probe: using
LLM_API_KEY_EVALwithOPENAI_BASE_URL=https://llm-proxy.eval.all-hands.dev, the same adapter completed agpt-5.5control run and returnedACP_PR4056_GPT55_CONTROL_OK. The eval proxy currently has no healthygpt-5.6-soldeployment, so that API-key path returns the proxy's 400 error forgpt-5.6-sol; after this PR's usage-accounting fix, the conversation reachesFINISHEDwith the proxy error instead of crashing in SDK token parsing.Video/Screenshots
Not applicable; the live conversation link above records the end-to-end result.
Type
Notes
Companion TypeScript client PR: OpenHands/typescript-client#272
LiteLLM was not upgraded: both the pinned 1.84.1 and current 1.91.1 already contain metadata for all four GPT-5.6 IDs.
Agent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:cfb13d0-pythonRun
All tags pushed for this build
About Multi-Architecture Support
cfb13d0-python) is a multi-arch manifest supporting both amd64 and arm64cfb13d0-python-amd64) are also available if neededCloses #4060