Useful Pretrain-Datasets - a Leon-Leee Collection

Leon-Leee 's Collections

awesome-zh-corpus

GPT-4 generated datasets

Useful Pretrain-Datasets

Code, Math, and Reasoning related Instruct datasets

High Quality Instruct Collections

Code Benchmarks

Useful Pretrain-Datasets

updated Mar 12, 2025

pretrain-datasets with (maybe) good quality

Zyphra/Zyda

Viewer • Updated Jun 19, 2024 • 4.35B • 4.31k • 84

Note English only
HuggingFaceFW/fineweb

Viewer • Updated Jul 11, 2025 • 52.5B • 262k • 2.73k

Note English only
HuggingFaceFW/fineweb-edu

Viewer • Updated Jul 11, 2025 • 3.5B • 313k • 1.01k

Note English only
m-a-p/Matrix

Viewer • Updated Feb 25, 2025 • 6.43B • 5.65k • 174

Note zh-en bilingual
bigcode/the-stack-v2

Viewer • Updated Apr 23, 2024 • 5.45B • 10.4k • 525

Note code
bigcode/the-stack-v2-train-full-ids

Viewer • Updated Jun 6, 2024 • 60.5M • 1.23k • 58

Note code
togethercomputer/RedPajama-Data-V2

Updated Nov 21, 2024 • 6.09k • 399
allenai/c4

Viewer • Updated Jan 9, 2024 • 10.4B • 625k • 540
EleutherAI/pile

Updated May 3, 2023 • 1.74k • 492

Note english
Skywork/SkyPile-150B

Viewer • Updated Dec 7, 2023 • 1.76M • 33.4k • 403

Note chinese
Tele-AI/TeleChat-PTD

Updated Mar 20, 2024 • 447 • 175

Note chinese
wenge-research/yayi2_pretrain_data

Viewer • Updated Dec 29, 2023 • 1.68M • 508 • 59

Note chinese and english
wikimedia/wikipedia

Viewer • Updated Jan 9, 2024 • 61.6M • 95.6k • 1.17k

Note multi-lingual
oscar-corpus/oscar

Updated Sep 4, 2025 • 531 • 206

Note multi-lingual
oscar-corpus/colossal-oscar-1.0

Preview • Updated Oct 26, 2023 • 12 • 36

Note multi-lingual
uonlp/CulturaX

Viewer • Updated Dec 16, 2024 • 7.18B • 15.3k • 604

Note multi-lingual
JosephusCheung/GuanacoDataset

Viewer • Updated Apr 15, 2024 • 1.17M • 8.75k • 515
allenai/MADLAD-400

Updated Sep 9, 2024 • 626k • 160

Note multi-lingual
Leon-Leee/unofficial-pyedu

Viewer • Updated Mar 12, 2025 • 7.68M • 99 • 3