fix(evaljob): Hide cosine similarity metrics - #282
Open
kaapi-agent-bot[bot] wants to merge 1 commit into
Open
kaapi-agent-bot[bot] wants to merge 1 commit into
kaapi-agent-bot[bot] wants to merge 1 commit into
Conversation
Judge runs (v2) never compute cosine similarity or interact with Langfuse, so run cards, run list, and the detail page should not show cosine metrics or the Langfuse-based resync flow for them. A new native score_trace_url per item is surfaced as a "View Trace" link instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Closes #272
Summary
is_judge_runflag onEvalJob:ScoreDisplay.score_trace_urlis present.Checklist
Before submitting a pull request, please ensure that you mark these task.
npm run devandnpm run buildin the repository root and test.Original PR description
Summary
Judge runs (v2) never compute cosine similarity and never touch Langfuse, but the run cards, run list, and detail page were still built to render cosine metrics and a Langfuse-based trace resync flow unconditionally. This branches all of that rendering on a new
is_judge_runflag onEvalJob, and preserves a v1-only "Resync" (Langfuse trace) flow while introducing a native per-item trace link for judge runs.EvalRunCard: hides the "Cosine similarity calculation" cost line item and passesisJudgeRundown toScoreDisplay, which filters any summary score named with "cosine" out of the chip row.MetricsOverview(run detail page): filters cosine-named summary scores out of the metrics overview, and hides the "Resync" button (which resyncs scores via Langfuse trace lookup) for judge runs.CategoryMetricsTable: hides the "Avg Cosine" column for judge runs, keeping "Avg Correctness".DetailedResultsTable/GroupedResultsTable: filter out cosine-named per-item score columns for judge runs, and add a new "Trace" column that links out via the nativescore_trace_urlfield when present (this only appears when the backend supplies it, so v1 rows are unaffected).job.is_judge_run, which defaults to falsy for existing v1 jobs.Assumptions
The backend v2 API (parent #265) hasn't landed in this repo yet — there is no
is_judge_runfield on evaluation jobs, noscore_trace_urlfield on trace items, and no existing "Langfuse trace link" UI element anywhere in the codebase today (the closest analog is the "Resync" button + "No Langfuse Traces Available" modal in the run detail page, which callsget_trace_info=true&resync_score=true). Given that, I made these assumptions for the reviewer to confirm/correct once the v2 contract is finalized:is_judge_run?: booleantoEvalJobandscore_trace_url?: stringtoTraceItem/IndividualScoreas the shape the v2 API will send.CategoryMetricsTable) and any summary/per-item score whosenamematches/cosine/ias the "cosine similarity" data to hide for judge runs, since scores are rendered generically by name rather than as a dedicated cosine field elsewhere.DetailedResultsTabledriven byscore_trace_urlas the "native" replacement path — this is net-new UI since no per-item trace link existed before.EvalRunsListunchanged since it only forwardsjobobjects toEvalRunCardand has no cosine/Langfuse rendering of its own.Test plan
npm run lintpasses (0 errors; pre-existing complexity warnings unrelated to this change)npm run buildpassesis_judge_run: true/score_trace_urlpayload once the v2 backend contract is available (not possible in this sandboxed run — no backend/network access)Closes #272
🤖 Generated with Claude Code