Tool call history tracking invalidates implicit cache during chat stream session

[shawn-maybush]

Environment details

• Programming language: Python
• OS: Debian
• Language runtime version: Python 3.13.5
• Package version: 1.73.1

python-genai/google/genai/chats.py

Line 318 in cd66b68

output_contents.append(chunk.candidates[0].content)

In a streaming response, the model returns data in multiple chunks. If the model streams a thought followed by a tool call, they arrive in separate chunks. Because output_contents is populated with output_contents.append(chunk.candidates[0].content) during the stream, it is a list of Content objects.

By calling .extend(output_contents) in record_history, the SDK is dumping every single chunk as a separate Content entry into the history array. Instead of rolling up the parts into a single Content(role="model", parts=[...]), it’s producing history that looks like this:

[
  Content(role="user", parts=["Do a task..."]),
  Content(role="model", parts=["<thought>..."]),
  Content(role="model", parts=["</thought>"]),
  Content(role="model", parts=[FunctionCall(...)])
]

Gemini's Context Caching relies on strict left-to-right byte matching of the context payload. The cache expects alternating user and model turns. Even though the API might technically tolerate back-to-back model turns by silently collapsing them on the backend during inference, the raw JSON structure of the request has changed.

Because the serialized JSON array now contains multiple consecutive {"role": "model", ...} objects instead of one merged object, the byte-prefix of the request differs from whatever standard single-turn format was cached, resulting in a 100% cache miss (Cached Read = 0).

[shawn-maybush]

Was able to resolve by cleaning the history up in my client for Gemini SDK.

    def _clean_session_history(self) -> None:
        """In-place sanitization of the Google GenAI ChatSession history.
        
        Removes empty parts and merges consecutive duplicate roles to preserve
        strict alternating structures (user -> model -> user) for optimal prefix caching.
        """
        if not self.chat_session:
            return

        for history_attr in ["_curated_history", "_comprehensive_history"]:
            if not hasattr(self.chat_session, history_attr):
                continue
            history_list = getattr(self.chat_session, history_attr)
            if not history_list:
                continue

            cleaned_list = []
            for content in history_list:
                if not hasattr(content, "parts"):
                    continue

                # Keep only non-empty parts
                new_parts = []
                for part in content.parts:
                    # If it's a Pydantic model and has no fields set, it's empty
                    if hasattr(part, "model_fields_set"):
                        if part.model_fields_set:
                            new_parts.append(part)
                    else:
                        new_parts.append(part)

                if not new_parts:
                    # To preserve strict alternating roles, don't drop the content turn.
                    # Instead, add a dummy whitespace part.
                    new_parts.append(types.Part.from_text(text=" "))

                content.parts = new_parts
                cleaned_list.append(content)

            # Merge consecutive contents with the same role
            merged_list = []
            for content in cleaned_list:
                if not merged_list:
                    merged_list.append(content)
                else:
                    prev = merged_list[-1]
                    if getattr(prev, "role", None) == getattr(content, "role", None):
                        prev.parts.extend(content.parts)
                    else:
                        merged_list.append(content)

            # Update the list in-place
            history_list.clear()
            history_list.extend(merged_list)