Skip to content

fix(openai): persist refusal explanations in OpenAIAgent response - #663

Open
shobhitagnihotri69 wants to merge 1 commit into
hud-evals:mainfrom
shobhitagnihotri69:fix/openai-refusal-explanations
Open

shobhitagnihotri69 wants to merge 1 commit into
hud-evals:mainfrom
shobhitagnihotri69:fix/openai-refusal-explanations

Conversation

@shobhitagnihotri69

@shobhitagnihotri69 shobhitagnihotri69 commented Sep 14, 2026

Copy link
Copy Markdown

Summary

Follow-up to PR #660 (fix(agents): persist Anthropic refusal explanations). This PR surfaces and persists model refusal explanations in OpenAIAgent, bringing it into full parity with ClaudeAgent and OpenAICompatibleAgent.

Problem

When OpenAI models refuse a prompt or tool call due to content policies or safety guardrails:

  1. ResponseOutputRefusal items in message.content were skipped because only ResponseOutputText was extracted.
  2. incomplete_details.reason == "content_filter" was recorded in finish_reason, but the refusal field was never populated.
  3. OpenAIAgent.get_response() returned refusal=None, causing downstream evaluation pipelines and benchmark traces to lose the refusal explanation.

Solution

  1. Extract ResponseOutputRefusal blocks from message content into refusal.
  2. Fall back to a standard content filter notice when incomplete_details.reason == "content_filter" and no explicit refusal text was emitted.
  3. Forward refusal to the returned AgentStep.
  4. Add unit tests covering both refusal extraction and content filter fallbacks.

Test Coverage

  • Added test_get_response_surfaces_refusal_content and test_get_response_surfaces_refusal_on_content_filter to hud/agents/tests/test_openai_agent.py.
  • Verified all 12 tests pass cleanly with uv run pytest hud/agents/tests/test_openai_agent.py.

Note

Low Risk
Localized response-parsing change with no auth, persistence, or tool-execution behavior changes; mainly improves observability for blocked prompts.

Overview
OpenAIAgent now populates AgentStep.refusal when the Responses API refuses a turn, aligning behavior with the other agents so eval traces keep the explanation.

During get_response, refusal text is read from ResponseOutputRefusal (or type="refusal") blocks in message content instead of being dropped. If the response ends with incomplete_details.reason == "content_filter" and no refusal text was returned, a standard safety-filter message is used. finish_reason is unchanged aside from being stored in a local variable before return.

Unit tests cover explicit refusal content and the content-filter fallback.

Reviewed by Cursor Bugbot for commit aa8cb13. Bugbot is set up for automated code reviews on this repo. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants