Is your feature request related to a problem?
The improve-prompt currently uses v1, which relies on cosine scores. This limits the ability to provide users with a more comprehensive recommendation based on better metrics available in v2.
Describe the solution you'd like
- Redirect
app/api/evaluations/[id]/improve-prompt/route.ts to POST /api/v2/evaluations/{evaluation_id}/improve-prompt
- Maintain existing
callback_url resolution and /api/webhooks/prompt-improvement receiver with unchanged body shape
- Implement handling for
PromptRecommendationJobPublic callback: recommendation_type: "prompt"
- Restrict action to judged runs; return
422 not_a_judge_run for v1 runs
Original issue
Is your feature request related to a problem?
Improve-prompt proxies to v1, which drafts from cosine scores. v2 drafts from the three-metric judge trace (score + reasoning per metric) and returns a typed recommendation, so the console gets a better prompt and a reason for it.
Describe the solution you'd like
- Point
app/api/evaluations/[id]/improve-prompt/route.ts at POST /api/v2/evaluations/{evaluation_id}/improve-prompt
- Keep the existing
callback_url resolution and /api/webhooks/prompt-improvement receiver — the body shape is unchanged
- Handle the
PromptRecommendationJobPublic callback: recommendation_type: "prompt" (widening to a union later)
- Only offer the action on judged runs; surface
422 not_a_judge_run if it's attempted on a v1 run
Parent: #265
Is your feature request related to a problem?
The improve-prompt currently uses v1, which relies on cosine scores. This limits the ability to provide users with a more comprehensive recommendation based on better metrics available in v2.
Describe the solution you'd like
app/api/evaluations/[id]/improve-prompt/route.tstoPOST /api/v2/evaluations/{evaluation_id}/improve-promptcallback_urlresolution and/api/webhooks/prompt-improvementreceiver with unchanged body shapePromptRecommendationJobPubliccallback:recommendation_type: "prompt"422 not_a_judge_runfor v1 runsOriginal issue
Is your feature request related to a problem?
Improve-prompt proxies to v1, which drafts from cosine scores. v2 drafts from the three-metric judge trace (score + reasoning per metric) and returns a typed recommendation, so the console gets a better prompt and a reason for it.
Describe the solution you'd like
app/api/evaluations/[id]/improve-prompt/route.tsatPOST /api/v2/evaluations/{evaluation_id}/improve-promptcallback_urlresolution and/api/webhooks/prompt-improvementreceiver — the body shape is unchangedPromptRecommendationJobPubliccallback:recommendation_type: "prompt"(widening to a union later)422 not_a_judge_runif it's attempted on a v1 runParent: #265