Skip to content

feat(improve-prompt): Update BFF route for evaluations - #281

Open
kaapi-agent-bot[bot] wants to merge 1 commit into
mainfrom
agent/ui-fix-273
Open

kaapi-agent-bot[bot] wants to merge 1 commit into
mainfrom
agent/ui-fix-273

Conversation

@kaapi-agent-bot

@kaapi-agent-bot kaapi-agent-bot Bot commented Sep 18, 2026

Copy link
Copy Markdown

Issue

Closes #273

Summary

  • Repoint the improve-prompt BFF route to POST /api/v2/evaluations/{evaluation_id}/improve-prompt instead of v1.
  • Maintain existing callback_url resolution and /api/webhooks/prompt-improvement behavior.
  • Type webhook callback payload as PromptRecommendationJobPublic with recommendation_type: "prompt" and reject other types.
  • Display a friendly toast for 422 not_a_judge_run response from the backend.
  • Disable "Iterate on Prompt" button for v1 runs with a tooltip explaining the reason.

Checklist

Before submitting a pull request, please ensure that you mark these task.

  • Ran npm run dev and npm run build in the repository root and test.
  • If you've fixed a bug or added code that is tested
Original PR description

Summary

  • Repoint the improve-prompt BFF route at POST /api/v2/evaluations/{evaluation_id}/improve-prompt instead of v1.
  • Keep the existing callback_url resolution and /api/webhooks/prompt-improvement receiver behavior unchanged (body shape is the same).
  • Type the webhook callback payload as PromptRecommendationJobPublic with a recommendation_type: "prompt" discriminant (ready to widen to a union of recommendation types later), and reject payloads where it isn't "prompt".
  • Surface a friendly toast for a 422 not_a_judge_run response from the backend.
  • Disable the "Iterate on Prompt" button client-side for v1 (cosine-only) runs — judged runs are detected via the existing isNewScoreObjectV2 check (v2 score objects carry summary_scores/traces, v1 only has cosine_similarity) — with a tooltip explaining why.

Assumptions

  • The v2 endpoint accepts the same request body shape as v1 (callback_url plus any pass-through fields), per the issue's "body shape unchanged" note. Not verified against the live backend since this run has no network access.
  • The 422 error response for non-judged runs follows the existing pattern used elsewhere in this route ({ error: "not_a_judge_run" }), matching how other sync-validation codes (e.g. evaluation_not_found) are already surfaced by apiFetch/extractErrorMessage.
  • The webhook callback's data object is PromptRecommendationJobPublic with recommendation_type: "prompt" as the only current variant (per the issue's note "widening to a union later"); only job_id, status, recommendation_type, config_version, and error_message are read, mirroring the previous PromptImprovementJobPublic shape.
  • Client-side "judged run" detection reuses isNewScoreObjectV2 (already used elsewhere in this page to distinguish v2/judge score shapes from legacy cosine-only v1 shapes) rather than adding a new backend field, since no explicit is_judge_run/judge_type field exists in EvalJob today.

Closes #273

🤖 Generated with Claude Code

Route the improve-prompt BFF endpoint to POST /api/v2/evaluations/{id}/improve-prompt
instead of v1, and type the webhook callback as PromptRecommendationJobPublic with a
recommendation_type: "prompt" discriminant so it's ready to widen to other
recommendation types later. The Iterate on Prompt button is now disabled for v1
(cosine-only) runs, and a 422 not_a_judge_run response surfaces a clear toast.
@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)
  • ready-for-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 7f8fc5fd-f3ed-49ec-ac9f-7d8aa4f1db00

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot changed the title fix: point improve-prompt at v2 and gate it to judged runs feat(improve-prompt): Update BFF route for evaluations Sep 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluation: Update improve-prompt to v2

0 participants