Hugging Face's logo

dignity045
/
grandline

dataset-preprocessing
llm-pretraining
tokenization
deduplication
data-pipeline
ml-intern
Model card Files
xet
 Community
grandline / scripts /inspect_dataset.py

Commit History

Initial GrandLine implementation: deterministic shard-first dataset preprocessing for LLM pretraining
ed59144
verified

dignity045 commited on