refactor(rwkv): restore analogy-principle compliance for scoring and task prompt - #1382
Closed
Cai-z-us wants to merge 41 commits into
Closed
refactor(rwkv): restore analogy-principle compliance for scoring and task prompt#1382Cai-z-us wants to merge 41 commits into
Cai-z-us wants to merge 41 commits into
Conversation
Signed-off-by: Caizus <18930215187@163.com>
Serialize identical Hugging Face dataset cache writes across concurrent RWKV evaluation processes while preserving parallel loading for distinct datasets.
Require exactly one field marker per publishable task, publish singular task.field, strip marker tags, and make field part of canonical publication identity.
Complete native RWKV G1j evaluation and publication pipeline.
Remove framework-specific Scoreboard publication from the evaluator and retain native producer facts with standard LightEval outputs.
refactor(rwkv): align evaluation changes with upstream contracts
fix(rwkv): preserve verified rollout answer details
…task prompt Revert the RWKV-introduced math_verify scoring, field:xxx task tags, and task_prompt override feature back to their upstream LightEval equivalents across generic (non-RWKV) task definitions and pipeline code, since they had no upstream/authoritative prototype and touched shared, non-RWKV code paths. Relocate the RWKV-specific math judge (mirroring Albatross's eval_math500.py verify_one) into rwkv_free_response.py, gated by task name, alongside the renamed rwkv_single_choice.py (formerly rwkv_answer_extractor.py, mirroring Albatross's eval_gpqa_diamond.py). Delete the dead task_prompt feature (rwkv_prompt.py and its tests) which was unused outside its own tests, and the self-referential field:xxx tagging test that had no production consumer. Deduplicate the truncation-summary aggregation shared between Pipeline.evaluate() and RWKVPipeline.evaluate() into Pipeline._finalize_metrics(). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Author
|
Opened against the wrong repository by mistake (gh CLI defaulted to the fork parent). Closing — the actual PR targets our fork at rwkv-rs/lighteval-rwkv. Apologies for the noise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
math_verifyscoring,field:xxxtask tags, and thetask_promptoverride feature back to upstream LightEval equivalents in generic (non-RWKV) task files and pipeline code — none of these had an upstream/authoritative prototype and they touched shared, non-RWKV code paths (对照性原则 / 无关性原则).eval_math500.pyverify_one) into a newrwkv_free_response.py, gated by task name, alongside the renamedrwkv_single_choice.py(formerlyrwkv_answer_extractor.py, mirroring Albatross'seval_gpqa_diamond.py).task_promptfeature (rwkv_prompt.py+ its dedicated tests), which was unused outside its own test suite, and a self-referentialfield:xxxtagging test with no production consumer.Pipeline.evaluate()andRWKVPipeline.evaluate()intoPipeline._finalize_metrics().Test plan
uv run pytest tests/unit/rwkv/— 163 passeduv run pytest tests/unit/— 583 passed, 2 skippeduv run ruff check/ruff format --checkon all touched files