Evaluation: Fix AI summary truncation

[AkhileshNegi]

Is your feature request related to a problem?
AI summaries in EvaluationRun.score.overall.ai_summary are being truncated mid-sentence, affecting data integrity. This results in incomplete information being stored and reported.

Describe the solution you'd like

• Modify generate_run_ai_summary to handle quotes properly in the structured output.
• Ensure that the full summary, including complete sentences, is stored regardless of quotation marks.
• Implement logging for genuinely truncated summaries to provide visibility into issues.

Original issue

Describe the bug

Eval-v2 run AI summaries (EvaluationRun.score.overall.ai_summary) are stored truncated mid-sentence. 11 of 74 rows in a local ai_platform DB are affected.

... All five repeats omit the required     <- ends here
... because the model never names          <- ends here
... adds Linnaeus's nationality (          <- ends here

Root cause

generate_run_ai_summary (backend/app/crud/evaluations/summary.py) requests Anthropic structured output for what is a single prose field:

output_config={"format": {"type": "json_schema", "schema": _OUTPUT_SCHEMA}}
# _OUTPUT_SCHEMA = {"summary": {"type": "string"}}

Structured outputs use constrained decoding. Inside the string value of {"summary": "..."} a bare " is a grammatically legal token — it is the string terminator. When the summary reaches a quoted phrase (omit the required "wellness message", nationality ("Swedish")) and the sampler emits the raw " instead of \ + ", the grammar reads it as end-of-string. The object closes, generation finishes normally, and json.loads succeeds on valid JSON holding an amputated sentence.

Every layer reports success: valid JSON, stop_reason == "end_turn", and a non-fatal except Exception with nothing to log.

Ruled out

• max_tokens — cap is 3000; truncated rows are 354-842 chars (~90-220 output tokens). Complete summaries in the same table reach 2000+ chars.
• Column limit — score is JSONB, containing columns are Text. No constraint in the write path (save_score, crud/evaluations/core.py).
• Display clamp — ruled out by SELECT length(score->'overall'->>'ai_summary') in psql rather than reading a DB-client grid cell.

Evidence

	rows	contain a "
complete	58	11
truncated	16	1

Quotes are near-absent from truncated summaries, because the first quote is what ended generation. Cut points land exactly where a quote would open (twice immediately after (, which in this prompt's style precedes a quoted example). Per-quote coin flip: ~11 rows got lucky with \", ~11 did not.

To Reproduce

1. Run a v2 (is_judge_run) evaluation to completion so the terminal hook calls generate_run_ai_summary.
2. Query the stored summary:

   SELECT id, length(score->'overall'->>'ai_summary'), right(score->'overall'->>'ai_summary', 40)
   FROM evaluation_run
   WHERE score->'overall'->>'ai_summary' IS NOT NULL
   ORDER BY id DESC LIMIT 20;

3. Rows whose summary would have contained a quoted phrase end mid-sentence, with no closing reminder line.

Expected behavior

The full summary is stored — overall read, Top 3 to check: items, and the closing go-verify line — regardless of quotation marks in the prose. A summary that is genuinely cut short should be visible in logs, not silent.

Additional context

Two related defects in the same call, both fixed by removing the schema:

• next(b.text for b in response.content if b.type == "text") takes only the first text block and discards the rest.
• Any malformed JSON degrades the whole summary to None with no operator-facing signal.

The same json_schema pattern is used by _call_prompt_drafting_llm (backend/app/services/evaluations/prompt_improvement.py), where the blast radius is larger: improved_instructions is persisted as a new config_version, so a truncated draft becomes a live system prompt. That call has two required fields, so after an early string close the grammar forces ,"rationale":... and json.loads returns both keys with the instructions amputated — zero signal.

Not covered by the fix

• The already-truncated rows stay truncated; a backfill would mean re-running the summary call against each run's stored traces.
• The quote-termination class remains open on prompt_improvement, whose schema earns its place (two fields + a maxLength cap). Closing it there means switching to delimiter-separated output.

[github-actions]

🎉 This issue has been resolved in version 1.6.0-main.4 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀