dignity045
/

grandline

dataset-preprocessing

llm-pretraining

Model card Files Files and versions

grandline / tests /test_tokenize.py

Commit History

Initial GrandLine implementation: deterministic shard-first dataset preprocessing for LLM pretraining

ed59144
verified

dignity045 commited on 2 days ago