Dhevenddra/slm-lab-base-15m
Text Generation • Updated
SLM Lab
A 12.6M LM built from scratch on free Colab, then a 1B fine-tuned, DPO'd and quantized by hand. In build order - start at slm-lab-base-15m.
Note START HERE. 12.6M params, written from scratch and pretrained on TinyStories on one free T4. Val loss 1.5581, and at chance on every benchmark - measured, not omitted.
Dhevenddra/slm-lab-chat-15m
Text Generation • UpdatedNote Step 2. The same model continued on chat text. It buys 2.52 nats on conversation, pays 0.55 nats of forgetting, and learns 0% of the chat format. Mid-training alone is not enough.
Dhevenddra/slm-lab-sft-15m
Text Generation • UpdatedNote Step 3. Supervised fine-tuning, loss masked to the assistant turns. Format compliance 0% -> 72%; usefulness barely moves. Learning the protocol and being useful are separate things.
Dhevenddra/slm-lab-json-r8
Text Generation • Updated • 160Note Step 4. First real model: a QLoRA adapter on Llama-3.2-1B that turns messy expense lines into strict JSON. r=8, the rank this project's own sweep selected.
Dhevenddra/slm-lab-json-r16
Text Generation • Updated • 28Note Rank sweep arm: r=16.
Dhevenddra/slm-lab-json-r32
Text Generation • Updated • 24Note Rank sweep arm: r=32. Four times r=8's trainable parameters for no gain this project could resolve.
Dhevenddra/slm-lab-json-r8_s1
Text Generation • Updated • 25Note Rank sweep control: r=8 at a second seed, so the sweep's differences can be judged against the noise instead of read off a ranking.
Dhevenddra/slm-lab-json-dpo_b01
Text Generation • Updated • 64Note Step 5. DPO on top of r=8, beta=0.1 - the arm the write-up reports. Preference optimization could not teach the one field fine-tuning missed either.
Dhevenddra/slm-lab-json-dpo_b005
Text Generation • Updated • 23Note DPO sweep arm: beta=0.05.
Dhevenddra/slm-lab-json-dpo_b03
Text Generation • Updated • 24Note DPO sweep arm: beta=0.3.
Dhevenddra/slm-lab-gguf
Text Generation • 1B • Updated • 386Note Step 6. Both models as GGUF, runnable on a CPU. Scored on KL divergence from fp16, not just perplexity - which is how you find that Q5_K_M beats fp16 on perplexity while being further away.
Dhevenddra/slm-lab-data
Viewer • Updated • 12.2k • 3.28k • 1Note The data and every measurement. Browse the expense-to-JSON task in the viewer; read results/ for all 9 write-ups, including the ablations that found nothing.