Add Skill vs no-Skill behavioral eval comparison - #21

Closed

GeoGeekLab wants to merge 34 commits into

mainfrom

eval/skill-ab-comparison

Commits on Sep 23, 2026

Add clean no-skill host eval mode
GeoGeekLab
committed
bffdc04
Test clean no-skill adapter behavior
GeoGeekLab
committed
8c80e85
Support repeated eval runs and timing evidence
GeoGeekLab
committed
3c2f407
Test repeated eval metadata
GeoGeekLab
committed
b69e203
Add Skill A/B eval comparison report
GeoGeekLab
committed
4341d2b
Test Skill A/B result comparison
GeoGeekLab
committed
8144e02
Document repeated eval run index
GeoGeekLab
committed
b03462f
Document Skill A/B evaluation workflow
GeoGeekLab
committed
434a0f6
Require paired repetitions in A/B comparisons
GeoGeekLab
committed
1e94c9a
Test paired A/B repetition requirements
GeoGeekLab
committed
9214276
Add paired counterbalanced A/B eval runner
GeoGeekLab
committed
cddaf88
Keep paired runner environment handling self-contained
GeoGeekLab
committed
ec6c4b2
Test paired counterbalanced A/B runner
GeoGeekLab
committed
935b915
Add manual real-host Skill A/B workflow
GeoGeekLab
committed
e10d1da
Harden real-host workflow inputs and diagnostics
GeoGeekLab
committed
c26fe7d
Document paired real-host A/B phase
GeoGeekLab
committed
f9ba730
Require A/B evaluation tooling in repository validation
GeoGeekLab
committed
69b2bbd
Revert real-host experiment additions in scripts/compare_eval_results.py
GeoGeekLab
committed
6c6a2ab
Revert real-host experiment additions in tests/test_compare_eval_results.py
GeoGeekLab
committed
e1a2cae
Revert real-host experiment additions in evals/README.md
GeoGeekLab
committed
f3e1d65
Revert real-host experiment additions in scripts/validate_skill.py
GeoGeekLab
committed
424c520
Remove real-host experiment file .github/workflows/real-host-ab.yml
GeoGeekLab
committed
1f4e568
Remove real-host experiment file scripts/run_ab_evals.py
GeoGeekLab
committed
4cc5330
Remove real-host experiment file tests/test_run_ab_evals.py
GeoGeekLab
committed
7041e3a
Add semantic claim and generator-drift checks
GeoGeekLab
committed
ed1dbbc
Document stronger hidden eval checks
GeoGeekLab
committed
cb4f43b
Strengthen evals for second-order engineering risks
GeoGeekLab
committed
8bc2c82
Test second-order eval check semantics
GeoGeekLab
committed
9d80768
Document strengthened second-order eval coverage
GeoGeekLab
committed
114b574
Fix negation-aware final claim matching
GeoGeekLab
committed
7341a26
Validate hidden inline Python checks without executing them
GeoGeekLab
committed
a155ff1
Test hidden-check validation and negation edge case
GeoGeekLab
committed
ce54e91
Regression-test flaky fixture module isolation
GeoGeekLab
committed
2dc81d1
Sharpen README for GitHub engineering style
GeoGeekLab
committed
ab882e3

Navigation Menu

Add Skill vs no-Skill behavioral eval comparison - #21

Add Skill vs no-Skill behavioral eval comparison#21
GeoGeekLab wants to merge 34 commits into
mainfrom
eval/skill-ab-comparison

Commits on Sep 23, 2026

Add clean no-skill host eval mode

Test clean no-skill adapter behavior

Support repeated eval runs and timing evidence

Test repeated eval metadata

Add Skill A/B eval comparison report

Test Skill A/B result comparison

Document repeated eval run index

Document Skill A/B evaluation workflow

Require paired repetitions in A/B comparisons

Test paired A/B repetition requirements

Add paired counterbalanced A/B eval runner

Keep paired runner environment handling self-contained

Test paired counterbalanced A/B runner

Add manual real-host Skill A/B workflow

Harden real-host workflow inputs and diagnostics

Document paired real-host A/B phase

Require A/B evaluation tooling in repository validation

Revert real-host experiment additions in scripts/compare_eval_results.py

Revert real-host experiment additions in tests/test_compare_eval_results.py

Revert real-host experiment additions in evals/README.md

Revert real-host experiment additions in scripts/validate_skill.py

Remove real-host experiment file .github/workflows/real-host-ab.yml

Remove real-host experiment file scripts/run_ab_evals.py

Remove real-host experiment file tests/test_run_ab_evals.py

Add semantic claim and generator-drift checks

Document stronger hidden eval checks

Strengthen evals for second-order engineering risks

Test second-order eval check semantics

Document strengthened second-order eval coverage

Fix negation-aware final claim matching

Validate hidden inline Python checks without executing them

Test hidden-check validation and negation edge case

Regression-test flaky fixture module isolation

Sharpen README for GitHub engineering style