slm-125m-base

A 126M-parameter Llama-architecture base model pretrained from scratch on US case law, SEC filings, and educational web text. Trained on 8xH100 for ~28 minutes at a total project cost of roughly $18.

This is a base model. It has had no instruction tuning, no RLHF, and no safety alignment. It completes text; it does not follow instructions.

Architecture

Parameters	125,847,552
Layers / hidden / heads	12 / 768 / 12
Context length	1024
Vocab	16384 (byte-level BPE trained on this corpus)
Tied embeddings	True

Training data

1.92B tokens, ~76% legal:

Source	Tokens	Share
PleIAs/SEC	819M	43%
HFforLegal/case-law	639M	33%
HuggingFaceFW/fineweb-edu	463M	24%

Pipeline: deterministic rule-based cleaning (line filters, boilerplate stripping, repetition and language gates, plus a dictionary-based OCR gate on case law) then MinHash near-dedup, exact dedup, and 13-gram decontamination against the CaseHOLD evaluation set.

Training

2 epochs (3.84B tokens seen), AdamW, cosine schedule to a 6e-05 cooldown, bf16, sequence length 1024, global batch 524,288 tokens.

Results

Metric	Value
Val perplexity (held-out 1%)	9.07
CaseHOLD (zero-shot, 5-way)	19.5%

CaseHOLD is measured on data excluded from training by 13-gram decontamination, so it is not inflated by leakage. Expect near-chance zero-shot performance: a 125M base model with no task tuning is not competitive on multiple-choice legal reasoning, and this number is a baseline for fine-tuning, not a capability claim.

Limitations

Small, base-only, English-only, 1024-token context. Case-law inputs contain OCR noise that survives filtering. Not legal or financial advice and not suitable for any use where an incorrect statement of law carries consequences.