pretraining-data

Here are 5 public repositories matching this topic...

Yaxin9Luo / LLMSurgeon

(ACL 2026 Main) LLMSurgeon recovers the pretraining data mixture of any LLM from only its generated text — no weights, no training data. A calibrated domain classifier plus label-shift correction de-blurs biased predictions. Ships with LLMScan, a benchmark on 8 open-source LLMs.

data-provenance label-shift membership-inference large-language-models llm-analysis pretraining-data data-mixture acl2026

Updated May 29, 2026
Python

Madjakul / HALvesting

Star

[EMNLP 2026] HALvest-Contrastive: Retrieval-like Authorship Attribution with Patch-level Late Interaction

language-modeling dataset-generation pretraining-data

Updated May 21, 2026
Python

liujin99 / climbmix

Star

Unofficial reproduction of Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training (NeurIPS 2025, arXiv:2504.13161) — search-found mixtures beat uniform baselines +0.014–0.031 STEM at d28; novel finding: selection-mechanism winner's curse; Ascend NPU backend.

lightgbm reproduction data-curation climb paper-reproduction llm ascend-npu pretraining-data mid-training data-mixing data-mixture nanochat llm-pretraining nemotron-climb

Updated Sep 22, 2026
Python

emfreeburg / announcement-carries-cue

Star

Notation as an unrecorded pre-training variable: the paper, the instruments, the corpus census, the reader/writer studies, and the pure-frame format. E. M. Freeburg, 2026.

nlp corpus-linguistics language-models data-curation pretraining-data

Updated Aug 11, 2026
Python

Rian-T / benchmark-in-a-haystack

Star

Do quality filters pull benchmark questions into pretraining corpora? Injects MMLU, GSM8K, GPQA, ARC, HellaSwag, PIQA and TruthfulQA items into a corpus and ranks them with six quality classifiers, including DCLM, FineWeb-Edu and Gaperon.

python nlp data-filtering data-curation data-contamination gsm8k mmlu llm-evaluation fineweb dataset-quality pretraining-data llm-benchmark benchmark-contamination quality-filter

Updated Aug 19, 2026
Python

Add this topic to your repo

To associate your repository with the pretraining-data topic, visit your repo's landing page and select "manage topics."

Learn more