Skip to content

Evaluation: Update improve-prompt to v2 #273

Description

@AkhileshNegi

Is your feature request related to a problem?
The improve-prompt currently uses v1, which relies on cosine scores. This limits the ability to provide users with a more comprehensive recommendation based on better metrics available in v2.

Describe the solution you'd like

  • Redirect app/api/evaluations/[id]/improve-prompt/route.ts to POST /api/v2/evaluations/{evaluation_id}/improve-prompt
  • Maintain existing callback_url resolution and /api/webhooks/prompt-improvement receiver with unchanged body shape
  • Implement handling for PromptRecommendationJobPublic callback: recommendation_type: "prompt"
  • Restrict action to judged runs; return 422 not_a_judge_run for v1 runs
Original issue

Is your feature request related to a problem?
Improve-prompt proxies to v1, which drafts from cosine scores. v2 drafts from the three-metric judge trace (score + reasoning per metric) and returns a typed recommendation, so the console gets a better prompt and a reason for it.

Describe the solution you'd like

  • Point app/api/evaluations/[id]/improve-prompt/route.ts at POST /api/v2/evaluations/{evaluation_id}/improve-prompt
  • Keep the existing callback_url resolution and /api/webhooks/prompt-improvement receiver — the body shape is unchanged
  • Handle the PromptRecommendationJobPublic callback: recommendation_type: "prompt" (widening to a union later)
  • Only offer the action on judged runs; surface 422 not_a_judge_run if it's attempted on a v1 run

Parent: #265

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions