Skip to content

Commit 6065380

Browse files
AkhileshNegiclaude
andcommitted
fix(evaluation): force file_search so KB metric scores
Fast eval runs left tool_choice at auto. Models that decline to call file_search — gpt-4.1 in practice — return no chunks, so the knowledge_base metric is dropped on every row and "Adherence to Knowledge Base" comes back blank for the whole run. Force the tool wherever we already request file_search_call.results. The metric now measures groundedness given retrieval instead of the model's willingness to retrieve, so scores shift on every model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent a2e44ab commit 6065380

1 file changed

Lines changed: 5 additions & 2 deletions

File tree

  • backend/app/crud/evaluations

backend/app/crud/evaluations/fast.py

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -505,10 +505,13 @@ def run_response_chunk(
505505
)
506506

507507
# Ask OpenAI to return the file_search hits so knowledge_base can judge them.
508-
# tool_choice stays at the model default (auto) — consistent with normal calls;
509-
# a row where the model doesn't query the KB is scored N/A, not forced to search.
508+
# tool_choice is forced rather than left at auto: a model that declines to search
509+
# (gpt-4.1 usually does) returns no chunks, which drops the knowledge_base metric
510+
# for every row and leaves the score blank run-wide. Forcing the call makes the
511+
# metric measure groundedness given retrieval, not eagerness to retrieve.
510512
if any(t.get("type") == "file_search" for t in base_params.get("tools", [])):
511513
base_params["include"] = ["file_search_call.results"]
514+
base_params["tool_choice"] = "required"
512515

513516
results: list[dict[str, Any]] = []
514517
max_workers = max(

0 commit comments

Comments
 (0)