README.md

metadata

license: cc0-1.0
language:
  - hi
  - en
tags:
  - hinglish
  - text-generation
  - from-scratch
pipeline_tag: text-generation

Chad

A small from scratch Hinglish chat model, around 100M params. Built and trained end to end: a byte level BPE tokenizer, a Llama style decoder (RoPE, grouped query attention, SwiGLU, RMSNorm), pretrained on romanized Hinglish, then fine tuned to chat in a chill Gen Z voice.

This repo holds every version in one place so the lineage is easy to follow.

The lineage

The base model is pretrained once. Everything after that is a fine tune on top of it.

• base (v22): the pretrained model, around 4B tokens of romanized Hinglish. Knows the language but not how to chat. Val loss about 3.77.
• sft-v2: the first good chat version, pure distillation. Clean and on character. This is the champion of the SFT runs.
• sft-v1: an earlier mining attempt that came out wholesome instead of funny. Kept for the record.
• sft-v3: v2 plus a big general Hinglish set. Broader, but the voice got diluted.
• sft-v4 / v4.2 / v4.3: distill heavy blends with a calculator tool and gif tags added. v4 over roasts, v4.2 dials it back, v4.3 is the widest blend.
• dpo-final: preference tuned on top of SFT so it leans toward the sharper reply on its own. The latest stage.

What is in here

• base/v22.pt: the pretrained base (PyTorch checkpoint).
• checkpoints/: the raw trained weights, one per version. These are the source of truth.
• transformers/: the same models exported to Hugging Face format (load with from_pretrained). These are what the browser app runs after quantization.
• experiments/: side runs and A/B arms, kept so nothing is lost (the v1 continued branches, older DPO iterations).

How to load one

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("vermarjun/chad", subfolder="transformers/v2")
t = AutoTokenizer.from_pretrained("vermarjun/chad", subfolder="transformers/v2")

The architecture is plain LlamaForCausalLM (hidden 768, 12 layers, 12 query heads, 3 kv heads, vocab 32000, context 1024).

The raw .pt files use the project's own model class, not transformers. Use the transformers/ exports if you just want to run the model.

No private chat data was used at any stage.