pankajmathur/nanochat-d34-sft

nanochat-d34-finetuned

This is SFT only checkpoints of karpathy/nanochat-d34, trained using the nanochat framework.

Model Description

• Base Model: karpathy/nanochat-d34 (2.2B parameters, pre-trained on 88B tokens)
• Architecture: GPT-style transformer with depth=34
• Training Pipeline: Mid-training → SFT → RL (optional)
• Hardware: 8x A100-80GB GPUs

Training Details

Base Model (Pre-trained by Karpathy)

• Parameters: 2,217,082,880
• Training tokens: 88,683,315,200 (40x param:token ratio)
• Max sequence length: 2048
• Base CORE score: 0.3382

Dataset

It's same smol-smoltalk which is mentioned in Karpathy OG nanochat discussion

Eval Reports

Full details in report/reports.md

Metric	BASE	MID	SFT	RL
ARC-Challenge	-	0.5367	0.5418	-
ARC-Easy	-	0.6961	0.7210	-
GSM8K	-	0.1137	0.1327	-
HumanEval	-	0.1098	0.1037	-
MMLU	-	0.4229	0.4304	-
ChatCORE	-	0.4045	0.4157	-

Fine-tuning Pipeline

1. Mid-training: General instruction tuning on SmolTalk, MMLU, GSM8K, Spelling tasks
2. SFT (Supervised Fine-Tuning): Chat-specific training on ARC, GSM8K, SmolTalk
3. RL (Reinforcement Learning): Optional GRPO-style training on GSM8K (if included)

WandB Reports

Full Report

Repository Structure

├── tokenizer/
│   ├── tokenizer.pkl          # Tokenizer
│   └── token_bytes.pt         # Token byte mappings
├── mid_checkpoints/d34/       # Mid-training checkpoint
│   ├── model_*.pt
│   └── meta_*.json
├── chatsft_checkpoints/d34/   # SFT checkpoint
│   ├── model_*.pt
│   └── meta_*.json
├── chatsft_checkpoints_int8/d34/   # SFT checkpoint
│   ├── model_*.pt
│   └── meta_*.json
├── chatrl_checkpoints/d34/    # RL checkpoint (if available)
│   ├── model_*.pt
│   └── meta_*.json
├── report/                    # Evaluation reports
│   └── report.md
└── logs/                      # Training logs

License

MIT License (same as nanochat)

Acknowledgments

• Andrej Karpathy for the nanochat framework and pre-trained base model

@misc{nanochat,
  author = {Andrej Karpathy},
  title = {nanochat: The best ChatGPT that $100 can buy},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}

• The nanochat community