Skip to content

fix: Improved evals - #155

Merged
schiwekM merged 20 commits into
mainfrom
fix/better-evals
Oct 2, 2026
Merged

schiwekM merged 20 commits into
mainfrom
fix/better-evals

Conversation

@schiwekM

Copy link
Copy Markdown
Collaborator
  • Include static test failures in eval results
  • Correctly aggregate metrics
  • Link prompts to eval run
  • Link model params and model to eval run
  • Correctly handle Summarize on timeout prompt
  • Avoid overcounting input tokens for anthropic (caused by calling _generate at the end of stream and usage object being a completely new one)

Have you...

  • Added relevant entry to the change log?

@schiwekM
schiwekM requested review from a team as code owners September 28, 2026 09:05
@schiwekM
schiwekM requested a review from Akatuoro September 28, 2026 10:51
Comment thread lib/telemetry/mlflow/evaluation.js Outdated
Comment thread lib/telemetry/mlflow/evaluation.js
Comment thread lib/telemetry/mlflow/evaluation.js
Comment thread lib/telemetry/mlflow/evaluation.js
@schiwekM
schiwekM enabled auto-merge October 2, 2026 13:27
@schiwekM
schiwekM added this pull request to the merge queue Oct 2, 2026
Merged via the queue into main with commit 896b8f1 Oct 2, 2026
9 checks passed
@schiwekM
schiwekM deleted the fix/better-evals branch October 2, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants