Hugging Face's logo

dignity045
/
grandline

dataset-preprocessing
llm-pretraining
tokenization
deduplication
data-pipeline
ml-intern
Model card Files
xet
 Community
grandline / tests
21.5 kB
  • 1 contributor
 History: 1 commit
dignity045's picture
dignity045
Initial GrandLine implementation: deterministic shard-first dataset preprocessing for LLM pretraining
ed59144 verified