Hi — I maintain EvalPort, an open JSON spec for portable LLM eval datasets and results. benchmarks/ already includes GSM8K; MATH is a natural complement (harder, worked-solution format) and isn't in there yet.
I loaded the dataset this repo's README points to (the HF mirror linked as "MATH dataset here") and read the real fields: problem, level (e.g. "Level 5"), type (e.g. "Algebra"), solution (the full worked solution, with the final answer inside a \boxed{...} — which is how this repo's own modeling/math_equivalence.py extracts and compares answers).
That maps onto EvalPort cleanly: problem -> TestCase.input, the \boxed{} extraction of solution -> TestCase.expected_output, level/type -> metadata, graded with a grader mirroring math_equivalence.py's own equivalence check rather than naive string match.
Would you be open to an EvalPort-format export living in EvalPort's benchmarks/, with attribution and license preserved? Happy to do the conversion and open a PR for review first. No worries if this isn't a fit for a repo that's mostly loaders/eval code at this point.
Hi — I maintain EvalPort, an open JSON spec for portable LLM eval datasets and results.
benchmarks/already includes GSM8K; MATH is a natural complement (harder, worked-solution format) and isn't in there yet.I loaded the dataset this repo's README points to (the HF mirror linked as "MATH dataset here") and read the real fields:
problem,level(e.g."Level 5"),type(e.g."Algebra"),solution(the full worked solution, with the final answer inside a\boxed{...}— which is how this repo's ownmodeling/math_equivalence.pyextracts and compares answers).That maps onto EvalPort cleanly:
problem->TestCase.input, the\boxed{}extraction ofsolution->TestCase.expected_output,level/type-> metadata, graded with a grader mirroringmath_equivalence.py's own equivalence check rather than naive string match.Would you be open to an EvalPort-format export living in EvalPort's
benchmarks/, with attribution and license preserved? Happy to do the conversion and open a PR for review first. No worries if this isn't a fit for a repo that's mostly loaders/eval code at this point.