评测:建立 blind-run 与 one-shot baseline 框架 - #7
Conversation
|
主控契约/安全复核退回:reference 与 package 可重叠进入 solve;evaluation contract 未绑定结果;token/cost budget 未执行;adapter note/identity 可泄漏敏感值;编译后的 benchmark 入口找不到 schemas。完整复现与必须补的测试见 Issue #3 最新评论。请通过新的 Orca regression Dispatch 更新本 PR。 |
|
第二轮主控复核仍需退回:adapter throw/timeout/invalid-output 在读取 scoring reference 前提前构造 hard-error,导致 failure evaluation digest 不绑定实际 reference,既无法与同契约 baseline 比较,reference 改变也不改变 failure digest。另需拒绝同 case 重复 variant/run id。完整复现见 Issue #3 最新评论。 |
当前 head 的主控复核通过
完整复核和人验收步骤在 Issue #3。代码与自动化验证通过,待人验收;请勿在此前合并。 |
Refs #3
Finalizes benchmark scoring-reference lifecycle handling: every terminal adapter path binds the final reference state and observed digest into deterministic evaluation and run identities, scoring failures are sanitized, and aggregate reports reject duplicate run IDs or case/variant entries.
Validation: focused benchmark finalization tests,
npm run check,npm run build, source/dist deterministic synthetic report comparison, andnpm audit --audit-level=high.