Repository navigation
fix(openai): record refusals in chat completion outputs - #2018
Open
pcbeingused333 wants to merge 2 commits into
Open
pcbeingused333 wants to merge 2 commits into
pcbeingused333 wants to merge 2 commits into
Conversation
When a model refuses (for example with structured outputs), OpenAI returns the text in message.refusal and leaves content empty. The integration did not read refusal, so the generation output showed an empty assistant message; streamed refusals (delta.refusal) were dropped the same way. Record the refusal in both the non-streamed and the streamed output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a model refuses a request (for example with structured outputs), OpenAI puts the explanation in
message.refusaland leavescontentempty. The OpenAI integration doesn't readrefusal:_extract_chat_responsekeepsrole,function_call,tool_calls,audioandcontent, so the generation output is{"role": "assistant", "content": null};_extract_streamed_openai_responseonly accumulatesdelta.content(plus function and tool calls), so a streamed refusal is recorded as no output at all.In Langfuse the refused call then looks like an empty answer.
Changes
_extract_chat_responseaddsrefusalto the output when the message has one._extract_streamed_openai_responseaccumulatesdelta.refusaland returns{"role": "assistant", "refusal": ...}(pluscontentif any text was streamed), the same shape as the non-streamed output.Tests
Two unit tests in
tests/unit/test_openai.py, built with the openai SDK'sChatCompletionChunk/ChatCompletionMessage: a streamed refusal split over two chunks, and a non-streamed message withrefusal. Both fail onmainand pass with this change.pytest tests/unit/test_openai.py: 39 passed (37 onmain).pytest tests/unit --continue-on-collection-errors: 690 passed (688 onmain). The 2 failures and 24 collection errors are the same onmain(optional dependencies such as langchain are not installed in this environment).ruff format,ruff checkandmypy langfuse/openai.py: clean.This change was written with Claude Code; the tests and checks above were run before opening the PR.
The refusal changes appear sound, but the two imports must move to module level to satisfy the repository rule before merging.
Summary
The PR records OpenAI refusal text in chat completion outputs.
Reviews (1) · Last reviewed commit: "fix(openai): record refusals in chat com..." · Reviewed by Greptile