slm-125m-base
A 125.8M-parameter Llama-architecture language model pretrained from scratch on a legal/financial corpus. No weights or tokenizer were inherited from any existing model.
Model
| params | 125,847,552 (125.8M, tied embeddings) |
| layers / d_model / heads | 12 / 768 / 12 (MHA) |
| context length | 1024 |
| vocab | 16384 (byte-level BPE, trained from scratch) |
| activation / norm | SwiGLU / RMSNorm, RoPE |
Training data (2.04B tokens)
Streamed from HuggingFace, cleaned, deduplicated and decontaminated:
| source | role |
|---|---|
HFforLegal/case-law |
US court opinions |
PleIAs/SEC |
SEC filings (10-K etc.) |
HuggingFaceFW/fineweb-edu |
general educational web text |
Realized mix is roughly 40/40/20 (case-law / SEC / web), i.e. ~78% legal. The legal sources together contain only ~2B clean tokens, so a legal-heavy mix at a larger scale is not achievable with these corpora.
Pipeline: 6-step deterministic cleaning (line filters, boilerplate strip,
repetition, language ID, OCR-garble gate on scanned case law) -> MinHash LSH
near-dedup + exact dedup -> 13-gram decontamination against the CaseHOLD /
LexGLUE case_hold evaluation set -> 16K byte-level BPE -> packed uint16
1024-token windows, 99/1 train/val split.
Training
2 epochs over the fixed corpus (~4.08B tokens seen)
8x H100, DDP, bf16 autocast,
torch.compile, fused AdamW, SDPA attentionglobal batch 524,288 tokens, cosine LR 0.0006 -> 6e-05, 200M warmup tokens
measured throughput ~3.19M tok/s (35% MFU)
final val loss: 2.2083
final val perplexity: 9.10
Intended use and limitations
A small base model for research on legal/financial text. It is not instruction-tuned, has no safety alignment, and will produce fluent but unreliable legal or financial statements. Do not use it for legal or financial advice. Case-law inputs include OCR noise; SEC filings skew to older filings. The corpus is decontaminated against CaseHOLD, so CaseHOLD evaluation is meaningful; no other benchmark was held out.
- Downloads last month
- 6