Hugging Face's logo

dignity045
/
grandline

dataset-preprocessing
llm-pretraining
tokenization
deduplication
data-pipeline
ml-intern
Model card Files
xet
 Community
grandline / README.md

Commit History

Update ML Intern artifact metadata
f39f181
verified

dignity045 commited on

Add selective HF parquet shard download support (--hf-files, --hf-subdir, --max-shards, --list-shards)
ab68c56
verified

dignity045 commited on

Update ML Intern artifact metadata
0d9ecc4
verified

dignity045 commited on

Add YAML metadata to README
c5799d7
verified

dignity045 commited on

Update ML Intern artifact metadata
c89217e
verified

dignity045 commited on

Initial GrandLine implementation: deterministic shard-first dataset preprocessing for LLM pretraining
ed59144
verified

dignity045 commited on

Update ML Intern artifact metadata
d4af6f6
verified

dignity045 commited on