fix(evaluation): Force tool choice for metrics

[AkhileshNegi]

Issue

Closes #PLEASE_TYPE_ISSUE_NUMBER

Summary

• Before: run_response_chunk requested file_search_call.results but left tool_choice at the model default, leading to empty retrieved_chunks and dropping the knowledge_base metric.
• Now: Forces the tool choice to ensure hits are returned, allowing the metric to reflect groundedness based on retrieval.
• Requires changing the tool choice when requesting results.
• Adjusted behaviour to clarify empty chunks indicate no hits from the store instead of not being queried.

Checklist

Before submitting a pull request, please ensure that you mark these task.

• Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

Notes

Single-line behavioural change plus the comment that documented the old choice. Worth a real gpt-4.1 run against a KB-backed config before merging, to confirm the chunks come back and the metric populates.

Original PR description

Issue

No issue filed — reported directly: "Adherence to Knowledge Base" comes back empty on every fast evaluation run using gpt-4.1.

Summary

run_response_chunk requested file_search_call.results but left tool_choice at the model default (auto). A model that declines to call file_search returns no hits, so retrieved_chunks is empty, _applicable_metrics drops the knowledge_base metric for that row, and the trace records an N/A with "Knowledge base not queried." gpt-4.1 declines in practice, so the whole run shows a blank KB score while ground-truth and prompt adherence score normally.

The judge itself is fine — it runs on the shared EVAL_JUDGE_MODEL, which is why the symptom could only originate on the response-generation side.

This forces the tool in the same place we already request the results.

Behaviour change to be aware of: the metric now measures groundedness given retrieval rather than the model's willingness to retrieve. Scores will move on every model, not just gpt-4.1, and previously-blank rows will now carry a number.

Follow-up, deliberately not in this PR: with retrieval forced, empty chunks no longer mean "not queried" — they mean the store returned no hits. The "Knowledge base not queried." reason string in the trace builder is now misleading and should be reworded separately.

Checklist

• Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

Notes

Single-line behavioural change plus the comment that documented the old choice. Worth a real gpt-4.1 run against a KB-backed config before merging, to confirm the chunks come back and the metric populates.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)

• ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 4d63fcfb-803c-458d-8168-10c40aa9ce5f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

OpenAPI changes   ⚪ No API surface changes

Note

This PR does not modify the API contract.

_{main ↔ 52be5c42 · generated by oasdiff}