Skip to content

Release v1.51.1 - #5483

Closed
all-hands-bot wants to merge 2 commits into
mainfrom
rel-1.51.1
Closed

all-hands-bot wants to merge 2 commits into
mainfrom
rel-1.51.1

Conversation

@all-hands-bot

@all-hands-bot all-hands-bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

Release v1.51.1

This PR prepares the release for version 1.51.1.

Started by: @malhotra5

Release Checklist

  • Version set to 1.51.1
  • Fix any deprecation deadlines if they exist
  • Integration tests pass (tagged with integration-test)
  • Behavior tests pass (tagged with behavior-test)
  • Example tests pass (tagged with test-examples)
  • Security scan passes (tagged with security-scan)
  • Confirm any release-note-required PRs are accurately called out in the final release notes

What happens on merge

When this PR is merged, the create-release.yml workflow will automatically:

  1. Create a GitHub release with tag v1.51.1 and auto-generated notes, plus an explicit preamble for merged release-note-required PRs
  2. Explicitly dispatch PyPI, npm, GitHub Packages, agent-server image, and binary publishers against tag v1.51.1
  3. Trigger version-bump-prs.yml after successful PyPI publication

🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:3d2172e-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-3d2172e-python \
  ghcr.io/openhands/agent-server:3d2172e-python

All tags pushed for this build

ghcr.io/openhands/agent-server:3d2172e-golang-amd64
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-golang-amd64
ghcr.io/openhands/agent-server:rel-1.51.1-golang-amd64
ghcr.io/openhands/agent-server:3d2172e-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:3d2172e-golang-arm64
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-golang-arm64
ghcr.io/openhands/agent-server:rel-1.51.1-golang-arm64
ghcr.io/openhands/agent-server:3d2172e-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:3d2172e-java-amd64
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-java-amd64
ghcr.io/openhands/agent-server:rel-1.51.1-java-amd64
ghcr.io/openhands/agent-server:3d2172e-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:3d2172e-java-arm64
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-java-arm64
ghcr.io/openhands/agent-server:rel-1.51.1-java-arm64
ghcr.io/openhands/agent-server:3d2172e-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:3d2172e-python-amd64
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-python-amd64
ghcr.io/openhands/agent-server:rel-1.51.1-python-amd64
ghcr.io/openhands/agent-server:3d2172e-python-node-runtime-amd64
ghcr.io/openhands/agent-server:3d2172e-python-arm64
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-python-arm64
ghcr.io/openhands/agent-server:rel-1.51.1-python-arm64
ghcr.io/openhands/agent-server:3d2172e-python-node-runtime-arm64
ghcr.io/openhands/agent-server:3d2172e-python-minimal-amd64
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-python-minimal-amd64
ghcr.io/openhands/agent-server:rel-1.51.1-python-minimal-amd64
ghcr.io/openhands/agent-server:3d2172e-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:3d2172e-python-minimal-arm64
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-python-minimal-arm64
ghcr.io/openhands/agent-server:rel-1.51.1-python-minimal-arm64
ghcr.io/openhands/agent-server:3d2172e-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:3d2172e-python-slim-amd64
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-python-slim-amd64
ghcr.io/openhands/agent-server:rel-1.51.1-python-slim-amd64
ghcr.io/openhands/agent-server:3d2172e-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:3d2172e-python-slim-arm64
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-python-slim-arm64
ghcr.io/openhands/agent-server:rel-1.51.1-python-slim-arm64
ghcr.io/openhands/agent-server:3d2172e-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:3d2172e-golang
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-golang
ghcr.io/openhands/agent-server:rel-1.51.1-golang
ghcr.io/openhands/agent-server:3d2172e-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:3d2172e-java
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-java
ghcr.io/openhands/agent-server:rel-1.51.1-java
ghcr.io/openhands/agent-server:3d2172e-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:3d2172e-python-minimal
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-python-minimal
ghcr.io/openhands/agent-server:rel-1.51.1-python-minimal
ghcr.io/openhands/agent-server:3d2172e-python-node-runtime-minimal
ghcr.io/openhands/agent-server:3d2172e-python-slim
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-python-slim
ghcr.io/openhands/agent-server:rel-1.51.1-python-slim
ghcr.io/openhands/agent-server:3d2172e-python-node-runtime-slim
ghcr.io/openhands/agent-server:3d2172e-python
ghcr.io/openhands/agent-server:3d2172e3362b85daa404a566e805b7ea62e8b865-python
ghcr.io/openhands/agent-server:rel-1.51.1-python
ghcr.io/openhands/agent-server:3d2172e-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 3d2172e-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 3d2172e-python-amd64) are also available if needed

Co-authored-by: openhands <openhands@all-hands.dev>
@all-hands-bot all-hands-bot added integration-test Runs the integration tests and comments the results test-examples Run all applicable "examples/" files. Expensive operation. behavior-test security-scan Run the release security scan (approval-drift + supply-chain) on this PR labels Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

🔒 Release Security Scan

🔒 Approval drift (time-of-check vs time-of-use)

❌ 3 finding(s)

Baseline: v1.51.0 — 5 merged PR(s) in range.

⚠️ 1 first-parent commit(s) in range did not match the (#N) squash convention and were not audited (a merge/rebase merge, or a lost (#N) suffix). These are a blind spot — inspect them:

  • Release v1.51.1
PR finding detail
#5462 changed-after-approval last approval by @all-hands-bot was on d6418f8, but merged head was 6784a4d7d — commits landed after review
#5362 changed-after-approval last approval by @all-hands-bot was on a4cdbd9, but merged head was a4498572b — commits landed after review
#4630 changed-after-approval last approval by @VascoSch92 was on 24723a1, but merged head was 43eb0f0ce — commits landed after review

Audited 5 PR(s): 2 clean, 3 flagged, 0 un-auditable.

📦 Supply-chain dependency diff

✅ no findings

Baseline: v1.51.0
Added: 0, bumped: 0, removed: 0.
(plus 4 internal openhands-* bump(s))

OSV: no known vulns across 4 new/bumped dep(s).

Internal `openhands-*` bumps
  • openhands-agent-server: 1.51.0 → 1.51.1
  • openhands-sdk: 1.51.0 → 1.51.1
  • openhands-tools: 1.51.0 → 1.51.1
  • openhands-workspace: 1.51.0 → 1.51.1

Deterministic scanners: approval-drift + supply-chain dependency diff. Read-only; no PR code executed.

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor
  ✅ **PR Artifacts Cleaned Up**

  The `.pr/` directory is no longer present.

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Hi! I started running the behavior tests on your PR. You will receive a comment with the results shortly.

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly.

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Hi! I started running the behavior tests on your PR. You will receive a comment with the results shortly.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
TOTAL48276832083% 
report-only-changed-files is enabled. No changed files were found in the coverage report :)

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

🧪 Integration Tests Results

Overall Success Rate: 98.0%
Total Cost: $2.03
Models Tested: 5
Timestamp: 2026-10-03 20:04:18 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

📊 Summary

Model Overall Tests Passed Skipped Total Cost Tokens
litellm_proxy_openai_gpt_5.5 100.0% 10/10 0 10 $0.94 326,911
litellm_proxy_gemini_3.1_pro_preview 100.0% 10/10 0 10 $0.48 350,148
litellm_proxy_anthropic_claude_sonnet_4_6 90.0% 9/10 0 10 $0.59 400,335
litellm_proxy_deepseek_deepseek_v4_flash 100.0% 10/10 0 10 $0.02 428,341
litellm_proxy_minimax_MiniMax_M2.7 100.0% 9/9 1 10 $0.00 481,386

📋 Detailed Results

litellm_proxy_openai_gpt_5.5

  • Success Rate: 100.0% (10/10)
  • Total Cost: $0.94
  • Token Usage: prompt: 322,044, completion: 4,867, cache_read: 181,760, reasoning: 1,419
  • Run Suffix: litellm_proxy_openai_gpt_5.5_716eaf2_gpt_5_5_run_N10_20261003_200248

litellm_proxy_gemini_3.1_pro_preview

  • Success Rate: 100.0% (10/10)
  • Total Cost: $0.48
  • Token Usage: prompt: 345,405, completion: 4,743, cache_read: 149,353, reasoning: 2,581
  • Run Suffix: litellm_proxy_gemini_3.1_pro_preview_716eaf2_gemini_3_1_pro_run_N10_20261003_200211

litellm_proxy_anthropic_claude_sonnet_4_6

  • Success Rate: 90.0% (9/10)
  • Total Cost: $0.59
  • Token Usage: prompt: 394,989, completion: 5,346, cache_read: 282,109, cache_write: 112,821, reasoning: 470
  • Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_716eaf2_claude_sonnet_4_6_run_N10_20261003_200227

Failed Tests:

  • t02_add_bash_hello: Shell script is not executable (Cost: $0.06)

litellm_proxy_deepseek_deepseek_v4_flash

  • Success Rate: 100.0% (10/10)
  • Total Cost: $0.02
  • Token Usage: prompt: 421,773, completion: 6,568, cache_read: 321,536, reasoning: 1,683
  • Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_716eaf2_deepseek_v4_flash_run_N10_20261003_200225

litellm_proxy_minimax_MiniMax_M2.7

  • Success Rate: 100.0% (9/9)
  • Total Cost: $0.00
  • Token Usage: prompt: 475,631, completion: 5,755, cache_read: 389,525
  • Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_716eaf2_minimax_m2_7_run_N10_20261003_195855
  • Skipped Tests: 1

Skipped Tests:

  • t08_image_file_viewing: This test requires a vision-capable LLM model. Please use a model that supports image input.

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

🧪 Integration Tests Results

Overall Success Rate: 98.0%
Total Cost: $2.01
Models Tested: 5
Timestamp: 2026-10-03 20:04:46 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

📊 Summary

Model Overall Tests Passed Skipped Total Cost Tokens
litellm_proxy_openai_gpt_5.5 100.0% 10/10 0 10 $0.93 359,679
litellm_proxy_gemini_3.1_pro_preview 100.0% 10/10 0 10 $0.48 355,179
litellm_proxy_anthropic_claude_sonnet_4_6 90.0% 9/10 0 10 $0.58 380,592
litellm_proxy_deepseek_deepseek_v4_flash 100.0% 10/10 0 10 $0.02 509,199
litellm_proxy_minimax_MiniMax_M2.7 100.0% 9/9 1 10 $0.00 517,639

📋 Detailed Results

litellm_proxy_openai_gpt_5.5

  • Success Rate: 100.0% (10/10)
  • Total Cost: $0.93
  • Token Usage: prompt: 354,990, completion: 4,689, cache_read: 219,136, reasoning: 1,322
  • Run Suffix: litellm_proxy_openai_gpt_5.5_716eaf2_gpt_5_5_run_N10_20261003_200319

litellm_proxy_gemini_3.1_pro_preview

  • Success Rate: 100.0% (10/10)
  • Total Cost: $0.48
  • Token Usage: prompt: 349,820, completion: 5,359, cache_read: 156,808, reasoning: 3,255
  • Run Suffix: litellm_proxy_gemini_3.1_pro_preview_716eaf2_gemini_3_1_pro_run_N10_20261003_200303

litellm_proxy_anthropic_claude_sonnet_4_6

  • Success Rate: 90.0% (9/10)
  • Total Cost: $0.58
  • Token Usage: prompt: 375,447, completion: 5,145, cache_read: 263,140, cache_write: 112,250, reasoning: 467
  • Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_716eaf2_claude_sonnet_4_6_run_N10_20261003_200308

Failed Tests:

  • t02_add_bash_hello: Shell script is not executable (Cost: $0.06)

litellm_proxy_deepseek_deepseek_v4_flash

  • Success Rate: 100.0% (10/10)
  • Total Cost: $0.02
  • Token Usage: prompt: 503,543, completion: 5,656, cache_read: 402,688, reasoning: 1,992
  • Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_716eaf2_deepseek_v4_flash_run_N10_20261003_200255

litellm_proxy_minimax_MiniMax_M2.7

  • Success Rate: 100.0% (9/9)
  • Total Cost: $0.00
  • Token Usage: prompt: 511,678, completion: 5,961, cache_read: 434,758
  • Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_716eaf2_minimax_m2_7_run_N10_20261003_200250
  • Skipped Tests: 1

Skipped Tests:

  • t08_image_file_viewing: This test requires a vision-capable LLM model. Please use a model that supports image input.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

🔄 Running Examples with openhands/claude-haiku-4-5-20251001

Generated: 2026-10-03 20:25:32 UTC

Example Status Duration Cost
01_standalone_sdk/02_custom_tools.py ✅ PASS 24.6s $0.03
01_standalone_sdk/03_activate_skill.py ✅ PASS 22.8s $0.03
01_standalone_sdk/05_use_llm_registry.py ✅ PASS 12.6s $0.01
01_standalone_sdk/07_mcp_integration.py ✅ PASS 34.1s $0.03
01_standalone_sdk/09_pause_example.py ✅ PASS 12.6s $0.01
01_standalone_sdk/10_persistence.py ✅ PASS 29.6s $0.03
01_standalone_sdk/11_async.py ✅ PASS 31.0s $0.02
01_standalone_sdk/12_custom_secrets.py ✅ PASS 13.6s $0.01
01_standalone_sdk/13_get_llm_metrics.py ✅ PASS 26.1s $0.03
01_standalone_sdk/14_context_condenser.py ✅ PASS 1m 32s $0.09
01_standalone_sdk/17_image_input.py ✅ PASS 26.9s $0.02
01_standalone_sdk/18_send_message_while_processing.py ✅ PASS 25.0s $0.01
01_standalone_sdk/19_llm_routing.py ✅ PASS 17.5s $0.02
01_standalone_sdk/20_stuck_detector.py ✅ PASS 17.1s $0.02
01_standalone_sdk/21_generate_extraneous_conversation_costs.py ✅ PASS 12.4s $0.00
01_standalone_sdk/22_anthropic_thinking.py ✅ PASS 15.3s $0.01
01_standalone_sdk/23_responses_reasoning.py ✅ PASS 40.3s $0.01
01_standalone_sdk/24_planning_agent_workflow.py ✅ PASS 5m 17s $0.30
01_standalone_sdk/25_agent_delegation.py ✅ PASS 1m 8s $0.06
01_standalone_sdk/26_custom_visualizer.py ✅ PASS 21.0s $0.02
01_standalone_sdk/28_ask_agent_example.py ✅ PASS 32.4s $0.04
01_standalone_sdk/29_llm_streaming.py ✅ PASS 47.0s $0.03
01_standalone_sdk/30_tom_agent.py ✅ PASS 12.5s $0.01
01_standalone_sdk/31_iterative_refinement.py ✅ PASS 4m 44s $0.31
01_standalone_sdk/32_configurable_security_policy.py ✅ PASS 20.9s $0.01
01_standalone_sdk/33_hooks/main.py ✅ PASS 34.4s $0.04
01_standalone_sdk/34_critic_example.py ✅ PASS 2m 34s $0.02
01_standalone_sdk/36_event_json_to_openai_messages.py ✅ PASS 11.0s $0.00
01_standalone_sdk/37_llm_profile_store/main.py ✅ PASS 8.0s $0.00
01_standalone_sdk/38_browser_session_recording.py ✅ PASS 39.7s $0.03
01_standalone_sdk/39_llm_fallback.py ✅ PASS 13.4s $0.00
01_standalone_sdk/40_acp_agent_example.py ✅ PASS 48.2s $0.32
01_standalone_sdk/41_task_tool_set.py ✅ PASS 29.8s $0.03
01_standalone_sdk/42_file_based_subagents.py ✅ PASS 30.6s $0.04
01_standalone_sdk/44_model_switching_in_convo.py ✅ PASS 10.6s $0.01
01_standalone_sdk/45_parallel_tool_execution.py ✅ PASS 4m 34s $0.55
01_standalone_sdk/46_agent_settings.py ✅ PASS 13.0s $0.00
01_standalone_sdk/47_defense_in_depth_security.py ✅ PASS 5.2s $0.00
01_standalone_sdk/48_conversation_fork.py ✅ PASS 23.1s $0.01
01_standalone_sdk/49_switch_llm_tool.py ✅ PASS 8.5s $0.04
01_standalone_sdk/50_async_cancellation.py ✅ PASS 15.0s $0.00
01_standalone_sdk/51_agent_hooks/main.py ✅ PASS 45.8s $0.05
01_standalone_sdk/52_dynamic_workflow.py ✅ PASS 4m 8s $0.17
01_standalone_sdk/53_client_defined_tools.py ✅ PASS 12.9s $0.01
01_standalone_sdk/54_goal_completion_loop.py ✅ PASS 22.6s $0.02
01_standalone_sdk/55_persistent_memory.py ✅ PASS 18.8s $0.02
01_standalone_sdk/56_structured_output.py ✅ PASS 47.6s $0.05
01_standalone_sdk/57_prompt_hooks/main.py ✅ PASS 17.9s $0.00
01_standalone_sdk/58_ask_oracle_tool/main.py ✅ PASS 18.3s $0.01
01_standalone_sdk/59_route_task_to_model.py ✅ PASS 11.1s $0.05
02_remote_agent_server/01_convo_with_local_agent_server.py ✅ PASS 35.5s $0.02
02_remote_agent_server/02_convo_with_docker_sandboxed_server.py ✅ PASS 1m 25s $0.04
02_remote_agent_server/03_browser_use_with_docker_sandboxed_server.py ✅ PASS 2m 29s $0.18
02_remote_agent_server/04_convo_with_api_sandboxed_server.py ✅ PASS 1m 14s $0.03
02_remote_agent_server/06_custom_tool/main.py ✅ PASS 5m 46s $0.06
02_remote_agent_server/07_convo_with_cloud_workspace.py ✅ PASS 49.8s $0.04
02_remote_agent_server/08_convo_with_apptainer_sandboxed_server.py ✅ PASS 4m 4s $0.03
02_remote_agent_server/09_acp_agent_with_remote_runtime.py ✅ PASS 1m 10s $0.21
02_remote_agent_server/10_cloud_workspace_share_credentials.py ✅ PASS 1m 44s $0.00
02_remote_agent_server/11_conversation_fork.py ✅ PASS 1m 3s $0.00
02_remote_agent_server/12_settings_and_secrets_api.py ✅ PASS 2m 22s $0.02
02_remote_agent_server/13_workspace_get_llm.py ✅ PASS 27.3s $0.00
02_remote_agent_server/14_client_defined_tools.py ✅ PASS 36.5s $0.02
02_remote_agent_server/15_openai_compatible_gateway.py ✅ PASS 23.8s $0.01
02_remote_agent_server/16_deferred_init.py ✅ PASS 28.9s $0.00
04_llm_specific_tools/01_gpt5_apply_patch_preset.py ✅ PASS 23.5s $0.02
04_llm_specific_tools/02_gemini_file_tools.py ✅ PASS 34.1s $0.08
05_skills_and_plugins/01_loading_agentskills/main.py ✅ PASS 14.4s $0.02
05_skills_and_plugins/02_loading_plugins/main.py ✅ PASS 29.6s $0.03
05_skills_and_plugins/04_mixed_marketplace_skills/main.py ✅ PASS 8.2s $0.00

✅ All tests passed!

Total: 70 | Passed: 70 | Failed: 0 | Total Cost: $3.48

View full workflow run

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

🔄 Running Examples with openhands/claude-haiku-4-5-20251001

Generated: 2026-10-03 20:23:00 UTC

Example Status Duration Cost
01_standalone_sdk/02_custom_tools.py ✅ PASS 24.0s $0.02
01_standalone_sdk/03_activate_skill.py ✅ PASS 21.0s $0.01
01_standalone_sdk/05_use_llm_registry.py ✅ PASS 13.2s $0.00
01_standalone_sdk/07_mcp_integration.py ✅ PASS 30.8s $0.02
01_standalone_sdk/09_pause_example.py ✅ PASS 10.9s $0.01
01_standalone_sdk/10_persistence.py ✅ PASS 27.7s $0.03
01_standalone_sdk/11_async.py ✅ PASS 27.0s $0.03
01_standalone_sdk/12_custom_secrets.py ✅ PASS 12.3s $0.00
01_standalone_sdk/13_get_llm_metrics.py ✅ PASS 24.3s $0.02
01_standalone_sdk/14_context_condenser.py ✅ PASS 2m 29s $0.16
01_standalone_sdk/17_image_input.py ✅ PASS 20.5s $0.02
01_standalone_sdk/18_send_message_while_processing.py ✅ PASS 31.1s $0.02
01_standalone_sdk/19_llm_routing.py ✅ PASS 17.0s $0.01
01_standalone_sdk/20_stuck_detector.py ✅ PASS 19.6s $0.01
01_standalone_sdk/21_generate_extraneous_conversation_costs.py ✅ PASS 10.7s $0.00
01_standalone_sdk/22_anthropic_thinking.py ✅ PASS 15.3s $0.01
01_standalone_sdk/23_responses_reasoning.py ✅ PASS 36.0s $0.01
01_standalone_sdk/24_planning_agent_workflow.py ✅ PASS 4m 16s $0.22
01_standalone_sdk/25_agent_delegation.py ✅ PASS 1m 2s $0.06
01_standalone_sdk/26_custom_visualizer.py ✅ PASS 22.4s $0.02
01_standalone_sdk/28_ask_agent_example.py ✅ PASS 34.0s $0.03
01_standalone_sdk/29_llm_streaming.py ✅ PASS 35.6s $0.02
01_standalone_sdk/30_tom_agent.py ✅ PASS 10.9s $0.01
01_standalone_sdk/31_iterative_refinement.py ✅ PASS 3m 53s $0.23
01_standalone_sdk/32_configurable_security_policy.py ✅ PASS 16.3s $0.02
01_standalone_sdk/33_hooks/main.py ✅ PASS 32.3s $0.04
01_standalone_sdk/34_critic_example.py ✅ PASS 2m 41s $0.02
01_standalone_sdk/36_event_json_to_openai_messages.py ✅ PASS 10.7s $0.00
01_standalone_sdk/37_llm_profile_store/main.py ✅ PASS 8.1s $0.00
01_standalone_sdk/38_browser_session_recording.py ✅ PASS 41.6s $0.04
01_standalone_sdk/39_llm_fallback.py ✅ PASS 13.3s $0.01
01_standalone_sdk/40_acp_agent_example.py ✅ PASS 40.7s $0.15
01_standalone_sdk/41_task_tool_set.py ✅ PASS 27.3s $0.02
01_standalone_sdk/42_file_based_subagents.py ✅ PASS 44.7s $0.03
01_standalone_sdk/44_model_switching_in_convo.py ✅ PASS 10.5s $0.01
01_standalone_sdk/45_parallel_tool_execution.py ✅ PASS 4m 6s $0.61
01_standalone_sdk/46_agent_settings.py ✅ PASS 11.3s $0.00
01_standalone_sdk/47_defense_in_depth_security.py ✅ PASS 3.3s $0.00
01_standalone_sdk/48_conversation_fork.py ✅ PASS 21.8s $0.00
01_standalone_sdk/49_switch_llm_tool.py ✅ PASS 6.5s $0.02
01_standalone_sdk/50_async_cancellation.py ✅ PASS 13.7s $0.00
01_standalone_sdk/51_agent_hooks/main.py ✅ PASS 42.0s $0.06
01_standalone_sdk/52_dynamic_workflow.py ✅ PASS 5m 30s $0.19
01_standalone_sdk/53_client_defined_tools.py ✅ PASS 18.0s $0.01
01_standalone_sdk/54_goal_completion_loop.py ✅ PASS 25.1s $0.03
01_standalone_sdk/55_persistent_memory.py ✅ PASS 16.4s $0.02
01_standalone_sdk/56_structured_output.py ✅ PASS 45.7s $0.06
01_standalone_sdk/57_prompt_hooks/main.py ✅ PASS 9.5s $0.00
01_standalone_sdk/58_ask_oracle_tool/main.py ✅ PASS 13.8s $0.01
01_standalone_sdk/59_route_task_to_model.py ✅ PASS 10.0s $0.05
02_remote_agent_server/01_convo_with_local_agent_server.py ✅ PASS 37.3s $0.02
02_remote_agent_server/02_convo_with_docker_sandboxed_server.py ✅ PASS 1m 25s $0.05
02_remote_agent_server/03_browser_use_with_docker_sandboxed_server.py ✅ PASS 1m 1s $0.09
02_remote_agent_server/04_convo_with_api_sandboxed_server.py ✅ PASS 1m 48s $0.04
02_remote_agent_server/06_custom_tool/main.py ✅ PASS 4m 24s $0.05
02_remote_agent_server/07_convo_with_cloud_workspace.py ✅ PASS 33.9s $0.02
02_remote_agent_server/08_convo_with_apptainer_sandboxed_server.py ✅ PASS 2m 53s $0.02
02_remote_agent_server/09_acp_agent_with_remote_runtime.py ✅ PASS 1m 15s $0.44
02_remote_agent_server/10_cloud_workspace_share_credentials.py ✅ PASS 1m 3s $0.00
02_remote_agent_server/11_conversation_fork.py ✅ PASS 38.0s $0.00
02_remote_agent_server/12_settings_and_secrets_api.py ✅ PASS 2m 14s $0.02
02_remote_agent_server/13_workspace_get_llm.py ✅ PASS 26.2s $0.01
02_remote_agent_server/14_client_defined_tools.py ✅ PASS 31.0s $0.02
02_remote_agent_server/15_openai_compatible_gateway.py ✅ PASS 18.4s $0.02
02_remote_agent_server/16_deferred_init.py ✅ PASS 17.9s $0.00
04_llm_specific_tools/01_gpt5_apply_patch_preset.py ✅ PASS 28.9s $0.02
04_llm_specific_tools/02_gemini_file_tools.py ✅ PASS 44.7s $0.08
05_skills_and_plugins/01_loading_agentskills/main.py ✅ PASS 14.0s $0.02
05_skills_and_plugins/02_loading_plugins/main.py ✅ PASS 17.8s $0.02
05_skills_and_plugins/04_mixed_marketplace_skills/main.py ✅ PASS 3.5s $0.00

✅ All tests passed!

Total: 70 | Passed: 70 | Failed: 0 | Total Cost: $3.33

View full workflow run

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

🧪 Behavior Tests Results

Overall Success Rate: 100.0%
Total Cost: $8.07
Models Tested: 4
Timestamp: 2026-10-03 20:15:40 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

📊 Summary

Model Overall Tests Passed Skipped Total Cost Tokens
litellm_proxy_gemini_3.1_pro_preview 100.0% 5/5 0 5 $3.80 7,141,959
litellm_proxy_anthropic_claude_sonnet_4_6 100.0% 5/5 0 5 $3.61 5,384,080
litellm_proxy_deepseek_deepseek_v4_flash 100.0% 5/5 0 5 $0.49 7,326,727
litellm_proxy_minimax_MiniMax_M2.7 100.0% 5/5 0 5 $0.17 2,856,091

📋 Detailed Results

litellm_proxy_gemini_3.1_pro_preview

  • Success Rate: 100.0% (5/5)
  • Total Cost: $3.80
  • Token Usage: prompt: 7,107,959, completion: 34,000, cache_read: 5,924,759, reasoning: 14,453
  • Run Suffix: litellm_proxy_gemini_3.1_pro_preview_716eaf2_gemini_3_1_pro_run_N5_20261003_200246

litellm_proxy_anthropic_claude_sonnet_4_6

  • Success Rate: 100.0% (5/5)
  • Total Cost: $3.61
  • Token Usage: prompt: 5,316,514, completion: 67,566, cache_read: 4,920,480, cache_write: 284,276, reasoning: 32,295
  • Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_716eaf2_claude_sonnet_4_6_run_N5_20261003_200242

litellm_proxy_deepseek_deepseek_v4_flash

  • Success Rate: 100.0% (5/5)
  • Total Cost: $0.49
  • Token Usage: prompt: 7,271,037, completion: 55,690, cache_read: 6,872,832, reasoning: 22,604
  • Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_716eaf2_deepseek_v4_flash_run_N5_20261003_200239

litellm_proxy_minimax_MiniMax_M2.7

  • Success Rate: 100.0% (5/5)
  • Total Cost: $0.17
  • Token Usage: prompt: 2,817,528, completion: 38,563, cache_read: 2,450,125
  • Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_716eaf2_minimax_m2_7_run_N5_20261003_200240

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

🧪 Behavior Tests Results

Overall Success Rate: 100.0%
Total Cost: $12.62
Models Tested: 5
Timestamp: 2026-10-03 20:26:43 UTC

📁 Detailed Logs & Artifacts

Click the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.

📊 Summary

Model Overall Tests Passed Skipped Total Cost Tokens
litellm_proxy_openai_gpt_5.5 100.0% 5/5 0 5 $3.47 2,882,456
litellm_proxy_gemini_3.1_pro_preview 100.0% 5/5 0 5 $5.56 9,437,132
litellm_proxy_anthropic_claude_sonnet_4_6 100.0% 5/5 0 5 $3.03 4,302,699
litellm_proxy_deepseek_deepseek_v4_flash 100.0% 5/5 0 5 $0.43 5,487,664
litellm_proxy_minimax_MiniMax_M2.7 100.0% 5/5 0 5 $0.14 2,610,182

📋 Detailed Results

litellm_proxy_openai_gpt_5.5

  • Success Rate: 100.0% (5/5)
  • Total Cost: $3.47
  • Token Usage: prompt: 2,847,896, completion: 34,560, cache_read: 2,525,696, reasoning: 10,282
  • Run Suffix: litellm_proxy_openai_gpt_5.5_716eaf2_gpt_5_5_run_N5_20261003_200349

litellm_proxy_gemini_3.1_pro_preview

  • Success Rate: 100.0% (5/5)
  • Total Cost: $5.56
  • Token Usage: prompt: 9,381,037, completion: 56,095, cache_read: 7,588,838, reasoning: 23,553
  • Run Suffix: litellm_proxy_gemini_3.1_pro_preview_716eaf2_gemini_3_1_pro_run_N5_20261003_200354

litellm_proxy_anthropic_claude_sonnet_4_6

  • Success Rate: 100.0% (5/5)
  • Total Cost: $3.03
  • Token Usage: prompt: 4,247,763, completion: 54,936, cache_read: 3,878,276, cache_write: 263,401, reasoning: 18,286
  • Run Suffix: litellm_proxy_anthropic_claude_sonnet_4_6_716eaf2_claude_sonnet_4_6_run_N5_20261003_200353

litellm_proxy_deepseek_deepseek_v4_flash

  • Success Rate: 100.0% (5/5)
  • Total Cost: $0.43
  • Token Usage: prompt: 5,432,562, completion: 55,102, cache_read: 5,050,368, reasoning: 21,966
  • Run Suffix: litellm_proxy_deepseek_deepseek_v4_flash_716eaf2_deepseek_v4_flash_run_N5_20261003_200357

litellm_proxy_minimax_MiniMax_M2.7

  • Success Rate: 100.0% (5/5)
  • Total Cost: $0.14
  • Token Usage: prompt: 2,581,831, completion: 28,351, cache_read: 2,275,405
  • Run Suffix: litellm_proxy_minimax_MiniMax_M2.7_716eaf2_minimax_m2_7_run_N5_20261003_200343

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Review: Release v1.51.1

This release PR bumps the four Python packages and the TypeScript client from 1.51.0 to 1.51.1 (+11/-11 across 7 files), with the matching uv.lock entries. The version-consistency half of the version-bump guard is fine: .github/scripts/check_version_bumps.py confirms all packages agree on 1.51.1 and the release markers match the rel-1.51.1 branch.

However, two required checks are red on the current head, and the release-specific test suites have not run against this head. Per .agents/skills/sdk-release/SKILL.md (Phase 3) and the review guide's "Release PRs" section, the latest PR-specific Run tests, Run Examples Scripts, and Run Integration Tests must have run on the current head and be green before merge.

Material findings

1. Check package versions fails: a 1.51.1 patch bump is not valid for the breaking SDK API changes present in this head.

Job Check package versions (run 37173812127, job 111352129504) fails at the Check Python API compatibility step:

::error Breaking changes detected (6); require at least minor version bump from 1.51.x, but new is 1.51.1
::warning LocalConversation.__init__(prompt_cache_key)::Positional parameter was moved
::warning LocalConversation.__init__(file_store)::Positional parameter was moved
::warning LocalConversation.__init__(mcp_tool_provider)::Positional parameter was moved
::warning LocalConversation.__init__(profile_store_dir)::Positional parameter was moved
::warning LocalConversation.__init__(stream_callbacks)::Positional parameter was moved
::warning LocalConversation.__init__(_parent_llm_call_context)::Positional parameter was moved

Reproduced in the workspace at head 3d2172e:

$ ACP_VERSION_CHECK_BASE_REF=main uv run python .github/scripts/check_sdk_api_breakage.py
... Breaking changes detected (6); require at least minor version bump from 1.51.x, but new is 1.51.1
EXIT:1

Root cause is in openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py: observability_parent_span_context was inserted into LocalConversation.__init__ before prompt_cache_key, shifting six existing keyword-or-positional parameters out of the positional slots they held in 1.51.0. This is a public-API compatibility break, and AGENTS.md requires at least a MINOR SemVer bump for breaking SDK API changes. The regression landed via #5462 after v1.51.0; the v1.51.0 release PR (#5470) passed this same check. Fix by shipping as 1.52.0 (preferred), or restore the previous parameter order so the break disappears. Until then the release cannot pass its own version guard.

2. The release test suites are stale for this head.

Run Examples Scripts (workflow 201191657) and Run Integration Tests (workflow 188677861) have no runs for head_sha=3d2172e3362b85daa404a566e805b7ea62e8b865 (API query returns total_count: 0). Their most recent runs are against 716eaf2 — the pre-merge Release v1.51.1 commit — and every integration/behavior/examples result comment on this PR is timestamped 2026-10-03, before this head was created (2026-10-04T03:20:48Z). Run tests did run and pass on the head, but the examples and integration validations required by the release checklist are missing/stale. Re-trigger them against the current head and confirm the posted comments agree with the check results.

3. build-binary-and-test (windows-latest) fails.

Job 111352129891 (run 37173812154) fails at the Test --extra-python-path custom tool import step: the negative test and the OH_EXTRA_PYTHON_PATH positive test both pass, then the step exits 1 before the --extra-python-path CLI-flag test output appears. The identical job passes on main at b66c724 (run 37205812053, 2026-10-04T13:29Z), so this is head-specific rather than a pre-existing main failure. It is not attributable to the version-only diff and may be a flaky Windows runner, but it is a red required check on this head and must be green or explicitly waived by a maintainer.

Note

Release Security Scan ran only against 716eaf2 and reported 3 approval-drift findings (changed-after-approval on #5462, #5362, #4630); no scan has run against 3d2172e, so the security-scan label is also stale for this head.

Verdict

The version strings are internally consistent, but the release head fails the repository's version/API-compatibility guard and the Windows binary check, and its release-specific suites are stale. Deferring approval to a human maintainer.

🔄 CHANGES REQUESTED

[project]
name = "openhands-sdk"
version = "1.51.0"
version = "1.51.1"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bumping openhands-sdk to 1.51.1 fails the repository's own guard: the Check Python API compatibility step of Check package versions (job 111352129504) reports Breaking changes detected (6); require at least minor version bump from 1.51.x, but new is 1.51.1. The six breaks are the LocalConversation.__init__ parameters shifted by the insertion of observability_parent_span_context (see review body). Per AGENTS.md, breaking SDK API changes require at least a MINOR bump - ship as 1.52.0 or restore the previous parameter order.

@malhotra5 malhotra5 closed this Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

behavior-test integration-test Runs the integration tests and comments the results security-scan Run the release security scan (approval-drift + supply-chain) on this PR test-examples Run all applicable "examples/" files. Expensive operation.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants