feat(rwkv): isolate per-run eval outputs and harden restart loop - #1383
Closed
Cai-z-us wants to merge 42 commits into
Closed
feat(rwkv): isolate per-run eval outputs and harden restart loop#1383Cai-z-us wants to merge 42 commits into
Cai-z-us wants to merge 42 commits into
Conversation
Signed-off-by: Caizus <18930215187@163.com>
Serialize identical Hugging Face dataset cache writes across concurrent RWKV evaluation processes while preserving parallel loading for distinct datasets.
Require exactly one field marker per publishable task, publish singular task.field, strip marker tags, and make field part of canonical publication identity.
Complete native RWKV G1j evaluation and publication pipeline.
Remove framework-specific Scoreboard publication from the evaluator and retain native producer facts with standard LightEval outputs.
refactor(rwkv): align evaluation changes with upstream contracts
fix(rwkv): preserve verified rollout answer details
…task prompt (#18) Revert the RWKV-introduced math_verify scoring, field:xxx task tags, and task_prompt override feature back to their upstream LightEval equivalents across generic (non-RWKV) task definitions and pipeline code, since they had no upstream/authoritative prototype and touched shared, non-RWKV code paths. Relocate the RWKV-specific math judge (mirroring Albatross's eval_math500.py verify_one) into rwkv_free_response.py, gated by task name, alongside the renamed rwkv_single_choice.py (formerly rwkv_answer_extractor.py, mirroring Albatross's eval_gpqa_diamond.py). Delete the dead task_prompt feature (rwkv_prompt.py and its tests) which was unused outside its own tests, and the self-referential field:xxx tagging test that had no production consumer. Deduplicate the truncation-summary aggregation shared between Pipeline.evaluate() and RWKVPipeline.evaluate() into Pipeline._finalize_metrics(). Co-authored-by: Caizus <user@example.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Give each RWKV eval run its own output_dir keyed by --run-id so concurrent/interrupted runs stop clobbering each other, and record a run manifest to guard against resuming a run-id with mismatched config or manifests. Fix the auto-restart wrapper scripts to actually support resume: dropping `exec` for a retry loop had detached the wrapper's PID from the worker process, so a TERM sent to the wrapper no longer reached the child (orphaning it instead of shutting it down) and the `status -ge 128` skip made the loop refuse to restart after an OOM-kill/crash, defeating the point of auto-restart. The wrapper now traps TERM/INT, forwards the signal to the child, waits for it to exit, and only skips restarting on a clean exit. Also stop the runner test from writing real files into the repo's results/ directory, which made it flaky depending on prior runs.
Author
|
Opened by mistake against the wrong repository (gh inferred the upstream remote instead of the fork). Closing; the correct PR targets rwkv-rs/lighteval-rwkv. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
output_dirkeyed by--run-id/--output-root, with a run manifest guarding against resuming a run-id whose config or manifests changed.temp/run_rwkv.sh/temp/run_g1j.sh: droppingexecfor a restart loop had detached the wrapper's PID from the worker, so aTERMsent to the wrapper never reached the child (orphaning it) andstatus -ge 128made the loop refuse to restart after an OOM-kill/crash. The wrapper now traps TERM/INT, forwards the signal to the child, waits for it to exit, and only skips restarting on a clean exit — restoring both graceful shutdown and crash resilience.test_four_model_evaluation_runner_forwards_signals_to_every_processfrom writing real files into the repo'sresults/directory (it used the default--output-root/--run-id), which made the test flaky depending on prior runs.Test plan
tests/unit/rwkv/— 163 passedTERMto a wrapper-style script's PID now reaches and terminates the backgrounded child instead of orphaning ittest_four_model_evaluation_runner_forwards_signals_to_every_processpasses deterministically across repeated runs with noresults/pollution