feat(judge): Add new metrics support - #279
kaapi-agent-bot[bot] wants to merge 1 commit into
Conversation
![Kaapi-Agent [Bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
…results Adds traffic-light coloring for the v2 judge metrics (Adherence to Ground Truth/Prompt/Knowledge Base) on a 0-5 scale, alongside the existing 0-1 cosine/correctness scale. Also surfaces the new per-trace `reasoning` field in score tooltips and CSV exports, falling back to the legacy `comment` field. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
changed the title
| /** | ||
| * Returns the judge's explanation for a score, preferring the v2 `reasoning` | ||
| * field over the legacy `comment` field. | ||
| */ |
There was a problem hiding this comment.
not needed this comment.
| return { value: catValue, color, bg }; | ||
| }; | ||
|
|
||
| // Traffic light thresholds for v2 judge metrics: 0-1 needs improvement, 2-3 could improve, 4-5 good. |
There was a problem hiding this comment.
able to understand via function name. so need to remove this.
| const numValue = Number(score.value); | ||
| const formattedValue = numValue.toFixed(2); | ||
| const formatCategoricalScore = (value: number | string) => { | ||
| const catValue = String(value); |
There was a problem hiding this comment.
Instead of using the catValue variable name, we should use a more descriptive and meaningful name that clearly indicates what the value represents.
| color = "#dc2626"; | ||
| bg = "#fee2e2"; | ||
| } | ||
| // v2 judge metrics score 0-5 (integers) instead of the legacy 0-1 cosine/correctness scale. |
There was a problem hiding this comment.
in every function, not needed these types of the comments.
This branch has not been deployed
1 participant
Issue
Closes #269
Summary
reasoningfield toTraceScorefor explanations.reasoningalongsidecomment.reasoningand modified headers.Checklist
Before submitting a pull request, please ensure that you mark these task.
npm run devandnpm run buildin the repository root and test.Original PR description
Closes #269
What changed
The v2 judge model reports three new metrics — Adherence to Ground Truth, Adherence to Prompt, Adherence to Knowledge Base — as integers on a 0–5 scale, applied per row only when the necessary inputs are present. The existing score-rendering code assumed every numeric score was a cosine-similarity-style value on a 0–1 scale, so these new metrics would have rendered with the wrong traffic-light color.
app/lib/types/evaluation.ts: added an optionalreasoningfield toTraceScorefor the judge model's per-trace explanation (the v2 counterpart to the existingcommentfield).app/lib/utils.ts: splitformatScoreValueinto per-scale helpers and added 0–5 traffic-light thresholds (0–1 red/needs improvement, 2–3 yellow/could improve, 4–5 green/good) for the three named "Adherence to ..." metrics, while keeping the existing 0–1 thresholds for cosine/correctness-style scores. Added agetScoreNotehelper that prefersreasoningovercomment.app/components/evaluations/DetailedResultsTable.tsxandGroupedResultsTable.tsx: score tooltips now showgetScoreNote(score)so the v2reasoningfield surfaces alongside the legacycommentfield.app/lib/utils/evaluationExport.ts: CSV exports now includereasoning(falling back tocomment) and the column headers were updated toComment/Reasoningto reflect that.Assumptions
"adherence to ground truth","adherence to prompt","adherence to knowledge base"(from the issue's screenshot/description) to decide which traffic-light scale (0–5 vs. legacy 0–1) applies to a givenTraceScore. If the backend uses different exact names, this list inapp/lib/utils.tswill need updating.reasoningfield onTraceScore, separate from (but analogous to) the existingcommentfield. Where both are present,reasoningtakes precedence for display purposes.TraceScore/scoring patterns already in the codebase.Test plan
npm run lintpasses with no new warnings/errorsnpm run buildpasses