Python: Support prompt cache breakpoints for GPT-5.6 models in OpenAI clients#7163
Open
Mordris wants to merge 2 commits into
Open
Python: Support prompt cache breakpoints for GPT-5.6 models in OpenAI clients#7163Mordris wants to merge 2 commits into
Mordris wants to merge 2 commits into
Conversation
… clients Add request-level prompt_cache_options to OpenAIChatOptions and OpenAIChatCompletionOptions, and forward a per-part prompt_cache_breakpoint from Content.additional_properties onto the content blocks each API supports. Text parts that carry a breakpoint keep typed list content, since the plain-string form cannot hold one; without a breakpoint the existing string forms are unchanged.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds Python OpenAI client support for GPT‑5.6 prompt caching controls by (a) exposing request-level prompt_cache_options in the typed options and (b) forwarding per-part prompt_cache_breakpoint metadata from Content.additional_properties into the outgoing request payloads, while preserving existing message-shape behavior when no breakpoint is present.
Changes:
- Introduces
PromptCacheOptionsand a sharedattach_prompt_cache_breakpoint(...)helper in_shared.py. - Adds
prompt_cache_optionsto both Responses (OpenAIChatOptions) and Chat Completions (OpenAIChatCompletionOptions) typed option sets. - Updates Chat Completions content/message preparation to keep list-form text parts when a breakpoint is present, and adds unit tests covering breakpoint propagation and options passthrough.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| python/packages/openai/agent_framework_openai/_shared.py | Adds a typed PromptCacheOptions shape and a helper to copy prompt_cache_breakpoint from Content.additional_properties into outgoing parts. |
| python/packages/openai/agent_framework_openai/_chat_completion_client.py | Plumbs breakpoint copying into Chat Completions parts and adjusts string-vs-list flattening so breakpoints are not lost. |
| python/packages/openai/agent_framework_openai/_chat_client.py | Plumbs breakpoint copying into Responses API parts and adds typed support for prompt_cache_options. |
| python/packages/openai/tests/openai/test_openai_chat_completion_client.py | Adds unit tests ensuring breakpoints are preserved (list-form) and prompt_cache_options passes through. |
| python/packages/openai/tests/openai/test_openai_chat_client.py | Adds unit tests ensuring breakpoint propagation for Responses parts and prompt_cache_options passthrough. |
Author
@microsoft-github-policy-service agree |
Contributor
Python Test Coverage Report •
Python Unit Test Overview
|
||||||||||||||||||||||||||||||||||||||||
eavanvalkenburg
left a comment
eavanvalkenburg
left a comment
Member
There was a problem hiding this comment.
would be good to have a sample for this as well
| AzureTokenProvider = Callable[[], str | Awaitable[str]] | ||
|
|
||
|
|
||
| class PromptCacheOptions(TypedDict, total=False): |
Member
There was a problem hiding this comment.
is there no built-in dict for this in the openai library? if so we should leverage that, it is OpenAI specific anyway!
| PROMPT_CACHE_BREAKPOINT_KEY = "prompt_cache_breakpoint" | ||
|
|
||
|
|
||
| def attach_prompt_cache_breakpoint(part: dict[str, Any], content: "Content") -> dict[str, Any]: |
Member
There was a problem hiding this comment.
let's make this is private method, it's not meant to be used outside of the two chat clients...
Suggested change
| def attach_prompt_cache_breakpoint(part: dict[str, Any], content: "Content") -> dict[str, Any]: | |
| def _attach_prompt_cache_breakpoint(part: dict[str, Any], content: "Content") -> dict[str, Any]: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation & Context
GPT-5.6 models support explicit prompt cache breakpoints, and cache writes are billed on these models, so controlling where a prefix is cached now matters for cost. Today the clients silently drop a
prompt_cache_breakpointset throughContent.additional_properties, andprompt_cache_optionsis not part of the typed chat options, so neither knob is usable from the framework.Description & Review Guide
What are the major changes?
PromptCacheOptionsTypedDict in_shared.py(same shape as the openai SDK type:mode,ttl) and aprompt_cache_optionsfield onOpenAIChatOptionsandOpenAIChatCompletionOptions._prepare_optionsalready forwards options generically, so no extra plumbing was needed.attach_prompt_cache_breakpointhelper copiesprompt_cache_breakpointfromContent.additional_propertiesonto the outgoing part, following the existingdetail/file_id/filenamepattern. It is applied to the blocks each API accepts:input_text/input_image/input_filefor the Responses client, and text/image/audio/file parts for the Chat Completions client._prepare_content_for_openaigets an explicittextcase (text previously fell through to theto_dictfallback), and text parts that carry a breakpoint keep list-form content, because the plain-string flattening cannot hold one. The same applies to system/developer messages. Without a breakpoint, the existing string forms are preserved unchanged.What is the impact of these changes?
mode: "explicit"(which disables the implicit breakpoint), a repeated ~1.6k-token prefix reportscached_tokens≈ 3100 on the second call acrossgpt-5.6-luna,gpt-5.6-solandgpt-5.6-terraon the Responses API, and ongpt-5.6-lunavia Chat Completions; the same setup without a breakpoint reports 0. Older models are unaffected; opting in on one surfaces the API's own 400 (prompt_cache_breakpoint is not supported on this model).openai>=2.45.0at runtime (the version that introduced these params). I left the dependency floor untouched since floors appear to be raised at release time.What do you want reviewers to focus on?
additional_propertiesis the surface you want for the per-part breakpoint, or if you'd rather have a first-class field onContent(happy to rework it).Related Issue
Fixes #7157
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.