Repository navigation
feat(sdk): support custom title generation prompts - #4564
DevinVinson wants to merge 7 commits into
Conversation
Co-authored-by: openhands <openhands@all-hands.dev>
|
Python API breakage checks — ✅ PASSEDResult: ✅ PASSED |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
DevinVinson
added
integration-test
|
Hi! I started running the integration tests on your PR. You will receive a comment with the results shortly. |
Coverage Report •
|
|||||||||||||||||||||||||||||||||||
🧪 Integration Tests ResultsOverall Success Rate: 97.7% 📁 Detailed Logs & ArtifactsClick the links below to access detailed agent/LLM logs showing the complete reasoning process for each model. On the GitHub Actions page, scroll down to the 'Artifacts' section to download the logs.
📊 Summary
📋 Detailed Resultslitellm_proxy_deepseek_deepseek_v4_flash
Skipped Tests:
litellm_proxy_anthropic_claude_sonnet_4_6
Failed Tests:
litellm_proxy_gemini_3.1_pro_preview
litellm_proxy_openai_gpt_5.5
litellm_proxy_minimax_MiniMax_M2.7
Skipped Tests:
|
|
🚦 CI is currently failing on this PR's latest commit. Please fix the failing checks before OpenHands reviews it - this is re-checked automatically once you push a new commit. (A maintainer can also request This is an automated check - no AI was used to generate this comment. |
| if title_generation_prompt and title_generation_prompt.strip(): | ||
| prompt = title_generation_prompt.strip() | ||
| includes_conversation = CONVERSATION_CONTENT_PLACEHOLDER in prompt | ||
| user_prompt = prompt.replace( | ||
| CONVERSATION_CONTENT_PLACEHOLDER, truncated_message | ||
| ).replace(MAX_LENGTH_PLACEHOLDER, str(max_length)) | ||
| if not includes_conversation: | ||
| user_prompt = f"{user_prompt}\n\nConversation content:\n{truncated_message}" | ||
| else: | ||
| user_prompt = ( | ||
| f"Generate a title (maximum {max_length} characters) " | ||
| f"for a conversation that starts with this message:\n\n" | ||
| f"{truncated_message}." | ||
| "Also make sure to include ONE most relevant emoji at " | ||
| "the start of the title." | ||
| f" Choose the emoji from this list:{emojis_descriptions} " | ||
| ) | ||
|
|
There was a problem hiding this comment.
template = (title_generation_prompt or "").strip()
if template:
user_prompt = template.replace(
CONVERSATION_CONTENT_PLACEHOLDER, truncated_message
).replace(MAX_LENGTH_PLACEHOLDER, str(max_length))
if CONVERSATION_CONTENT_PLACEHOLDER not in template:
user_prompt += f"\n\nConversation content:\n{truncated_message}"
else:
...
| {"emoji": "♻️", "name": "refactor", "description": "Code refactoring"}, | ||
| ] | ||
|
|
||
| CONVERSATION_CONTENT_PLACEHOLDER = "{conversation_content}" |
There was a problem hiding this comment.
Final[str] in both from typing.
Co-authored-by: Vasco Schiavo <115561717+VascoSch92@users.noreply.github.com>
github-actions
Bot
removed
the
release-note-required
Co-authored-by: Vasco Schiavo <115561717+VascoSch92@users.noreply.github.com>
…vice.py Co-authored-by: Vasco Schiavo <115561717+VascoSch92@users.noreply.github.com>
Co-authored-by: Vasco Schiavo <115561717+VascoSch92@users.noreply.github.com>
Co-authored-by: openhands <openhands@all-hands.dev>
|
Thank you @VascoSch92 ! I've updated to |
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Reviewed against head 4afe9a979fda23b125388e898d8c3f7895f2696e with repo guidance (AGENTS.md, openhands-agent-server/AGENTS.md, openhands-sdk/openhands/sdk/AGENTS.md, .agents/skills/custom-codereview-guide.md).
Scope: In scope. The change extends ConversationConfig / StoredConversation and the agent-server auto-title path, which is owned by this repo. No repository-move or product-direction gate applies.
What I verified
- The default title prompt is reproduced byte-for-byte: the wrapper only swaps the interpolated strings for an identical
user_promptliteral, and the new tests assert the default branch forNone/""/" ". Blank or absent overrides therefore keep existing behavior. AutoTitleSubscriberforwardsself.service.stored.promptpositionally into_generate_title_traced->generate_title_from_message->generate_title_with_llm; the argument order matches the new signature, andpromptis an optional keyword on the public SDK helpers, so this is backward compatible.promptis onConversationConfig, so it round-trips throughmeta.jsonviaStoredConversation(covered bytest_save_meta_round_trips_title_prompt). Adding an optional field is compatible and thePersisted settings/Check OpenAPI Schema/ Python-API breakage checks pass on this head.- Focused suite passes locally:
uv run pytest tests/sdk/conversation/test_generate_title.py tests/agent_server/test_conversation_service.py-> 124 passed.
Concerns (defer to a human maintainer)
-
Eval risk - prompt-template change. The repo's own review guide lists "prompt templates" as an eval-risk category: an
APPROVEis withheld until eval evidence exists (anopenhands-eval-monitor.vercel.apprun plus a maintainer confirming it). This PR changes the title-generation user message and lets callers replace it entirely, which can shift auto-title output. There is no eval-monitor link or maintainer eval confirmation on the PR, and the description states no live provider-backed run was performed. This needs a human maintainer's eval sign-off before merge. -
CI is red on this exact head.
PR Description Check(Validate PR description) andReview Thread Gateboth concludedfailurefor4afe9a9, andall-hands-botalready flagged the failing checks. The description check reports:- the first visible line must be
HUMAN:and needs a short human note betweenHUMAN:andAGENT:; and - linked issue #4561 carries neither
ready-for-devnor a pre-rollout creation date (it was created 2026-08-20, after the 2026-08-13 rollout cutoff), so the linked issue must meet readiness criteria before the PR can proceed.
The review-thread gate also reports 2 unresolved threads (theFinal[str]/typing thread and the placeholder-template thread). These are process gates rather than code defects, but they currently block the PR and need the author/maintainer to address.
- the first visible line must be
-
Non-blocking design note. The field is exposed on the public
/api/**contract as a barepromptonStartConversationRequest, even though it only affects title generation. The linked issue proposedtitle_generation_prompt; I understand this rename came from an existing review suggestion and that the companion UI PR (OpenHands/OpenHands#16771) sendsprompt, so I am not treating it as a defect - flagging only so the maintainer confirms the public field name deliberately departs from the issue's proposal and the docs PR.
No code-level correctness or security defect was found in the changed logic itself.
🔄 CHANGES REQUESTED
3 participants
HUMAN: Will need some 👀 to check but hopefully this is the right direction.
AGENT:
Implemented and exercised the complete Agent Server path: request validation and
conversation metadata persistence, AutoTitleSubscriber forwarding, prompt rendering,
and the LLM completion message. A live provider-backed run was not performed because
this environment has no LLM credentials; the PR remains a draft for human E2E testing.
Why
Users can choose a title LLM profile, but cannot customize the prompt used to generate
automatic conversation titles. This adds the Agent Server contract required by
#4561 and OpenHands/OpenHands#16761.
Summary
title_generation_promptconversation setting with a 2,000-character limit.{conversation_content}and{max_length}placeholders while safely appending conversation content when its placeholder is omitted.REST API contract changes
Compared with base OpenAPI
1de2e6d1bfcffor public/api/**paths.Issue Number
Closes #4561
How to Test
uv run pytest tests/sdk/conversation/test_generate_title.py tests/agent_server/test_conversation_service.py tests/agent_server/test_event_service.py.autotitle: trueand a prompt such asWrite a title under {max_length} characters for: {conversation_content}; after the first user message, confirm the completion receives the rendered custom user prompt andmeta.jsonretains the template.Validation performed:
make buildVideo/Screenshots
Not applicable to this backend-only change. Live LLM E2E evidence is pending human testing.
Design Doc
Not included; the API behavior and fallback rules are documented in the companion docs PR linked below.
Type
Notes
🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.13-nodejs22-slimgolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:4afe9a9-pythonRun
All tags pushed for this build
About Multi-Architecture Support
4afe9a9-python) is a multi-arch manifest supporting both amd64 and arm644afe9a9-python-amd64) are also available if neededJev-Fast-Audit
⚡ Jev fast audit · estimates · 0.68s · commit 4afe9a9
Strongest signal: Untrusted instruction authority · 9% estimated likelihood.
Evidence: No direct hunk selected.
Coverage: complete supplied coverage; 18/18 hunks, 7/7 files.
All estimates and evidence