picochat

A 90M parameter GPT trained from scratch in Rust using the picochat framework.

Model details

• Architecture: Decoder-only transformer with grouped-query attention, RoPE, sliding window attention, ReLU-squared MLP
• Parameters: 31.5M (depth=8: 8 layers, 512 dim, 8 heads, 4 KV heads)
• Vocab size: 4,096 (BPE tokenizer)
• Context length: 2048 tokens
• Training: Pretrained on OpenWebText (10k steps), then supervised fine-tuned on UltraChat + no_robots (2k steps)
• Framework: candle (Rust)
• Trained on: CPU only

Usage

# Clone the framework
git clone https://github.com/Nu11ified/picochat.git
cd picochat

# Download weights
mkdir -p runs/model
# Download model.safetensors, config.json, and tokenizer.json from this repo
# into runs/model/

# Chat
cargo run --release -- \
  --chat --load runs/model --tokenizer runs/model/tokenizer.json \
  --temperature 0.8 --max-tokens 256

# Web UI
cargo run --release -- \
  --serve --load runs/model --tokenizer runs/model/tokenizer.json --port 8000

Limitations

This model was trained on CPU with limited data (~5M tokens vs GPT-2's 8B). It produces coherent text on topics seen during training but will generate garbled output on novel questions. The value of this project is the from-scratch Rust training framework, not the resulting model.

Files

• model.safetensors -- model weights (120MB)
• config.json -- model architecture config
• tokenizer.json -- BPE tokenizer (32K vocab)