---
license: cc0-1.0
language:
- hi
- en
tags:
- hinglish
- text-generation
- from-scratch
pipeline_tag: text-generation
---

# Chad

A small from scratch Hinglish chat model, around 100M params. Built and trained end to end: a byte level BPE tokenizer, a Llama style decoder (RoPE, grouped query attention, SwiGLU, RMSNorm), pretrained on romanized Hinglish, then fine tuned to chat in a chill Gen Z voice.

This repo holds every version in one place so the lineage is easy to follow.

## The lineage

The base model is pretrained once. Everything after that is a fine tune on top of it.

- **base (v22)**: the pretrained model, around 4B tokens of romanized Hinglish. Knows the language but not how to chat. Val loss about 3.77.
- **sft-v2**: the first good chat version, pure distillation. Clean and on character. This is the champion of the SFT runs.
- **sft-v1**: an earlier mining attempt that came out wholesome instead of funny. Kept for the record.
- **sft-v3**: v2 plus a big general Hinglish set. Broader, but the voice got diluted.
- **sft-v4 / v4.2 / v4.3**: distill heavy blends with a calculator tool and gif tags added. v4 over roasts, v4.2 dials it back, v4.3 is the widest blend.
- **dpo-final**: preference tuned on top of SFT so it leans toward the sharper reply on its own. The latest stage.

## What is in here

- `base/v22.pt`: the pretrained base (PyTorch checkpoint).
- `checkpoints/`: the raw trained weights, one per version. These are the source of truth.
- `transformers/`: the same models exported to Hugging Face format (load with `from_pretrained`). These are what the browser app runs after quantization.
- `experiments/`: side runs and A/B arms, kept so nothing is lost (the v1 continued branches, older DPO iterations).

## How to load one

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("vermarjun/chad", subfolder="transformers/v2")
t = AutoTokenizer.from_pretrained("vermarjun/chad", subfolder="transformers/v2")
```

The architecture is plain LlamaForCausalLM (hidden 768, 12 layers, 12 query heads, 3 kv heads, vocab 32000, context 1024).

The raw `.pt` files use the project's own model class, not transformers. Use the `transformers/` exports if you just want to run the model.

No private chat data was used at any stage.