Kotichitturu/slm-125m-base

SLM-125M — Base

A 125M-parameter LLaMA-architecture language model pretrained from scratch on 2.04B tokens of US case law, SEC filings, and educational web text. Trained in 52 minutes on 8×H100 for $31.25 (~$34.80 including the data pipeline).

⚠️ This is a BASE model. It completes text. It does not answer questions.

Ask it a question and it will typically hand the question back to you, or recite your own prompt. Both of these are real outputs:

You send	It returns
What is the minimum net worth?	"What is the minimum net worth????????…"
Answer correctly and concisely. What is the main legal issue?	"Answer correctly and concisely. Answer correctly and concisely…"

This is not a defect — it is what a base model is. It was trained on exactly one objective: predict the next token. Give it a prefix to continue, never an instruction to obey.

Want one that answers? Same weights, fine-tuned:

Model	What it does
slm-125m-sft-raft	Answers from a context passage you supply, and refuses when the answer isn't in it (96.7%). The deployable one — but read its card: it reports the right figure only 37% of the time.
slm-125m-sft-qa	Follows instructions — and invents facts. A demonstration, not an answering service.

Try all three side by side: https://slm-125m-app.vercel.app/

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Kotichitturu/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("Kotichitturu/slm-125m-base")

prompt = "The plaintiff alleges that the defendant breached"   # a PREFIX, not a question
ids = tok(prompt, return_tensors="pt")

out = model.generate(
    **ids, max_new_tokens=90,
    do_sample=True, temperature=0.8, top_p=0.95, top_k=50,   # sampling matters — see below
)
print(tok.decode(out[0], skip_special_tokens=True))

Use sampling, not greedy. With do_sample=False this model degenerates into loops ("…minimum net worth????????"). temperature≈0.8, top_p≈0.95 is a sane default. That is normal for a small base model, not a bug in these weights.

⚠️ The tokenizer has chat tokens. The base cannot use them.

<|system|>, <|user|>, <|assistant|> exist in the vocabulary — they were added when the tokenizer was trained, so that fine-tuning later would not need a vocab resize (which would invalidate every embedding). But the pretraining corpus is raw text and never contains those strings, so their embeddings received essentially no gradient. They are untrained vectors in this model. Sending a chat-formatted prompt here produces noise. Use the SFT models above for chat format.

Model details

Architecture	LLaMA (decoder-only, pre-norm)
Parameters	125,848,320 (~125.8M), tied embeddings
Layers / hidden / heads	12 / 768 / 12 (head dim 64, MHA — not GQA)
MLP	SwiGLU, inner 3,072
Positions	RoPE (theta 10,000)
Norm	RMSNorm, pre-norm
Context	1,024 tokens
Vocab	16,384 byte-level BPE, trained on this corpus
Precision	bf16

Training

Data seen	2.04B unique tokens × 3 epochs = 6.12B
Optimizer	AdamW, lr 6e-4 → 6e-5 cosine, 200M-token warmup
Batch	524,288 tokens global
Hardware	8×H100 (Modal), ~1.96M tok/s
Steps	11,667
Wall time	~52 minutes

Evaluation

Final train loss	2.1899
Final val loss	2.2521
Perplexity	9.51

Perplexity 9.51 means that, on held-out text from this domain, the model is on average choosing between about 9.5 equally likely next tokens. Reasonable for 125M parameters on specialised text.

Re-measured later on an independent harness at 9.50 (loss 2.2511) — the 0.001 difference is bf16 rounding. That number is the control for the alignment-tax comparison of the fine-tuned children: Q&A costs +1.6% perplexity, RAFT +6.0%.

Training data

Source	Share	Dataset
US case law	~40%	HFforLegal/case-law
SEC filings (10-K, 10-Q)	~40%	PleIAs/SEC
Educational web	~20%	HuggingFaceFW/fineweb-edu (sample-10BT)

Cleaned (language ID, OCR gate, repetition filters), deduplicated (exact + MinHash near-dup), and decontaminated against CaseHOLD and LexGLUE by 13-gram overlap, so downstream evaluation on those benchmarks is not compromised.

Limitations

• It invents citations, case names, dates, and figures. At ~2 bits/parameter a 125M model cannot store a legal corpus; it produces text that looks like the training distribution. Do not treat any specific fact it emits as real.
• Not legal or financial advice. Do not use it for anything consequential.
• 1,024-token context.
• English, US-centric. Trained on US case law and SEC filings; it knows nothing of other jurisdictions.
• No alignment, no safety tuning, no RLHF. This is raw pretrained output.
• Web-text share means general fluency, but the domain skew is heavy.

How it was built

Full write-up — data pipeline, tokenizer, packing, the 8×H100 run, deployment, and the fine-tuning that followed — is in the project's study guide. Total spend for the base: $34.80 ($3.55 data pipeline + $31.25 pretraining).