nanochat-d20

Training Pipeline

1.Base-training PreTraining on FineWeb-EDU dataset using nanochat framework

2. Mid-training: General instruction tuning on SmolTalk, MMLU, GSM8K, Spelling tasks

3. SFT (Supervised Fine-Tuning): Chat-specific training on ARC, GSM8K, SmolTalk

4. RL (Reinforcement Learning): Optional GRPO-style training on GSM8K (if included)

Repository Structure

├── tokenizer/
│   ├── tokenizer.pkl          # Tokenizer
│   └── token_bytes.pt         # Token byte mappings
├── mid_checkpoints/d34/       # Mid-training checkpoint
│   ├── model_*.pt
│   └── meta_*.json
├── chatsft_checkpoints/d20/   # SFT checkpoint
│   ├── model_*.pt
│   └── meta_*.json
├── chatsft_checkpoints_int8/d20/   # SFT checkpoint
│   ├── model_*.pt
│   └── meta_*.json
├── chatrl_checkpoints/d20/    # RL checkpoint (if available)
│   ├── model_*.pt
│   └── meta_*.json
├── report/                    # Evaluation reports
│   └── report.md
└── logs/                      # Training logs

License

MIT License (same as nanochat)

Acknowledgments

• Andrej Karpathy for the nanochat framework

@misc{nanochat,
  author = {Andrej Karpathy},
  title = {nanochat: The best ChatGPT that $100 can buy},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}

• The nanochat community