Conversation
Two of the 4,576 archive rows carry a bare `gpt2` with no publishing namespace, so the `org/model` check rejected them and the adapter raised on incomplete conversion. That is a correct refusal, and it blocked the whole backfill over two rows. They are not junk rows. Both report sha 607a30d783df, the same weights as the `openai-community/gpt2` rows beside them, submitted separately and scoring differently, so dropping them would lose two real measurements. The repository those weights live at is namespaced now, and the datastore already holds `data/hfopenllm_v2/openai-community/gpt2`, so they are addressed there. An exclusion would have been the wrong mechanism. That records a row which is not an evaluation, and these are evaluations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two of the 4,576 HF Open LLM Leaderboard v2 archive rows carry a bare
gpt2with no publishing namespace. Theorg/modelcheck rejected them, soraise_if_incompletefired and the adapter produced nothing at all — 4,574 convertible rows lost to two bad names.They are not junk rows. All four
gpt2rows in the archive report sha607a30d783df, so the bare pair is the same weights as theopenai-community/gpt2pair beside it, but submitted separately and scoring differently (6.3910 against 6.5108 at bfloat16). Dropping them would lose two real measurements.The repository those weights live at is namespaced now, and the datastore already holds
data/hfopenllm_v2/openai-community/gpt2, so they are addressed there. The map is applied in both places the identity is derived, sinceconvert_modelandconvert_modelsbuild the id and the directory independently and disagreeing would trip the path warning.SourceRecordExclusionwould have been the wrong mechanism. It records a source row that is not an evaluation, and these are evaluations.gpt2is the only org-less name in all 4,576 rows, so the map has one entry rather than a heuristic.