| --- |
| license: cc0-1.0 |
| language: |
| - hi |
| - en |
| tags: |
| - hinglish |
| - text-generation |
| - from-scratch |
| pipeline_tag: text-generation |
| --- |
| |
| # Chad |
|
|
| A small from scratch Hinglish chat model, around 100M params. Built and trained end to end: a byte level BPE tokenizer, a Llama style decoder (RoPE, grouped query attention, SwiGLU, RMSNorm), pretrained on romanized Hinglish, then fine tuned to chat in a chill Gen Z voice. |
|
|
| This repo holds every version in one place so the lineage is easy to follow. |
|
|
| ## The lineage |
|
|
| The base model is pretrained once. Everything after that is a fine tune on top of it. |
|
|
| - **base (v22)**: the pretrained model, around 4B tokens of romanized Hinglish. Knows the language but not how to chat. Val loss about 3.77. |
| - **sft-v2**: the first good chat version, pure distillation. Clean and on character. This is the champion of the SFT runs. |
| - **sft-v1**: an earlier mining attempt that came out wholesome instead of funny. Kept for the record. |
| - **sft-v3**: v2 plus a big general Hinglish set. Broader, but the voice got diluted. |
| - **sft-v4 / v4.2 / v4.3**: distill heavy blends with a calculator tool and gif tags added. v4 over roasts, v4.2 dials it back, v4.3 is the widest blend. |
| - **dpo-final**: preference tuned on top of SFT so it leans toward the sharper reply on its own. The latest stage. |
|
|
| ## What is in here |
|
|
| - `base/v22.pt`: the pretrained base (PyTorch checkpoint). |
| - `checkpoints/`: the raw trained weights, one per version. These are the source of truth. |
| - `transformers/`: the same models exported to Hugging Face format (load with `from_pretrained`). These are what the browser app runs after quantization. |
| - `experiments/`: side runs and A/B arms, kept so nothing is lost (the v1 continued branches, older DPO iterations). |
|
|
| ## How to load one |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| m = AutoModelForCausalLM.from_pretrained("vermarjun/chad", subfolder="transformers/v2") |
| t = AutoTokenizer.from_pretrained("vermarjun/chad", subfolder="transformers/v2") |
| ``` |
|
|
| The architecture is plain LlamaForCausalLM (hidden 768, 12 layers, 12 query heads, 3 kv heads, vocab 32000, context 1024). |
|
|
| The raw `.pt` files use the project's own model class, not transformers. Use the `transformers/` exports if you just want to run the model. |
|
|
| No private chat data was used at any stage. |
|
|
