license: apache-2.0
language:
- en
tags:
- llama
- from-scratch
- legal
- finance
pipeline_tag: text-generation
slm-125m-base
A 125M-parameter, Llama-style, decoder-only language model, pretrained completely from scratch (fresh weights, fresh 16K byte-level BPE tokenizer) on a domain-specific corpus of US case law, SEC filings, and general web text.
This is a base completer, not a chat model — give it a passage prefix and it continues it. It has not been instruction-tuned, aligned, or fine-tuned for any downstream task.
Architecture
| Parameters | ~125M |
| Layers | 12 |
| Hidden size | 768 |
| Attention heads | 12 (head dim 64), plain MHA |
| MLP | SwiGLU, inner 3072 |
| Normalization | RMSNorm, pre-norm |
| Position embeddings | RoPE |
| Context length | 1024 |
| Vocab | 16,384 (fresh byte-level BPE, no OOV token ever) |
| Embeddings | tied |
Training data
~3.53B packed tokens across three sources, deduplicated (MinHash-LSH on the legal corpus) and decontaminated against the LexGLUE/CaseHOLD held-out splits:
| Source | Tokens | Share |
|---|---|---|
HFforLegal/case-law (US case law, us split) |
~1.1B | 31% |
PleIAs/SEC (SEC filings) |
~2.0B | 57% |
HuggingFaceFW/fineweb-edu (general web text) |
~1.0B | 28% |
(Shares don't sum to 100% due to rounding; case-law's split was capped at its
full-dataset ceiling, not a target percentage — see the build's docs/01-data.md.)
Training
AdamW (β 0.9/0.95, weight decay 0.1), linear warmup + cosine LR, bf16 (autocast, fp32 master weights), single-node DDP across 8x H100. Trained to step 7,500 (~4.1B tokens processed, more than one full epoch over the training set) before the run was stopped.
Evaluation
Held-out validation perplexity: 10.14 (full validation set, 34,513 windows). This is the honest headline metric at this scale — not MMLU or similar benchmarks, which are near-random for a 125M model.
Perplexity varies a lot by domain, reflecting the training mix:
| Domain | Perplexity |
|---|---|
| SEC filings | ~6.0 |
| Case law | ~15.3 |
| General web text (fineweb-edu) | ~27.4 |
The model is noticeably better at the two legal/financial domains it mostly trained on than at general web text.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("mickey5k/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("mickey5k/slm-125m-base")
prompt = "The court held that"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
output_ids = model.generate(
input_ids,
max_new_tokens=100,
min_new_tokens=20, # a base model can emit EOS almost immediately otherwise
do_sample=True,
temperature=0.8,
top_p=0.9,
top_k=50,
)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
Limitations
- Base completer, not instruction-tuned. It continues text; it does not follow instructions or answer questions in a chat format.
- Domain-skewed. Strongest on legal/financial register, weaker on general text (see per-domain perplexity above).
- Small. 125M parameters is far below the scale where benchmark-style reasoning emerges — evaluate it on perplexity/completion quality, not accuracy benchmarks.
- Training was cost-capped, not run to convergence — see the build's
docs/05-pretrain.mdfor the full training log and the reasoning behind stopping at step 7,500.