slm-125m-base

A 125-million-parameter Llama-style language model trained completely from scratch — random initialization to a working legal/financial text model, with its own tokenizer, its own data pipeline, and its own pretraining run. No base checkpoint was borrowed from anywhere.

• Repo / full build: github.com/DeependraVerma/legal-slm-125M
• Live demo (Playground): deependraverma-ai-legal-slm-125-m.vercel.app
• Author: Deependra Verma — Generative AI Researcher / AI Engineer (Hugging Face)
• ONNX / in-browser version: DeependraVerma/slm-125m-base-onnx
• Fine-tuned Q&A version: DeependraVerma/legal-slm-125m-sft

This is a base (completion) model, not an assistant. It continues text; it does not follow instructions or answer questions reliably. For a Q&A-capable model, use legal-slm-125m-sft. Never use this model's output as legal, financial, or factual advice — it invents specifics (names, numbers, citations) like any base language model.

Model description

slm-125m-base is a decoder-only transformer trained end-to-end for legal and financial English: a byte-level BPE tokenizer trained on the target corpus, a Llama-architecture model trained from random weights, and full evaluation — all built and run by one person, on-prem, as documented in the project README.

Parameters	125,848,320 (~125.8M, tied embeddings)
Architecture	Llama-style decoder (maps 1:1 to transformers.LlamaConfig)
Layers / hidden / heads	12 / 768 / 12 (head dim 64, full multi-head attention, no GQA/MQA)
MLP	SwiGLU, intermediate size 3,072
Positional encoding	RoPE, θ = 10,000
Normalization	RMSNorm, ε = 1e-5
Vocabulary	16,384, byte-level BPE, trained on this project's own corpus
Context length	1,024 tokens
Embeddings	tied input/output
Precision	bf16 compute (autocast) · fp32 master weights and saved checkpoint

Intended uses & limitations

Intended use: text continuation / completion in a legal and financial register — drafting-style prose, exploring how a small from-scratch model represents this domain, research and educational purposes (tokenizer design, small-model pretraining, domain-adapted LMs).

Not intended for: question answering, instruction following, or any use where factual accuracy matters. This is a base language model — it completes text plausibly, and at 125M parameters over a 2B-token corpus it will fabricate case names, statute citations, dollar figures, and other specifics that sound right but are not grounded in anything. Do not use its output as legal, financial, or factual advice, or present it as such to end users without a human review step. If you need a model that at least attempts to answer questions, see the fine-tuned legal-slm-125m-sft — and note that model has its own documented hallucination limitations too.

Training data

Three public, ungated Hugging Face datasets, streamed (never fully downloaded), cleaned, deduplicated, and decontaminated against the CaseHOLD / LexGLUE benchmarks before training:

Source	HF dataset	Split / field
US case law	HFforLegal/case-law	split us, field document
SEC filings	PleIAs/SEC	split train, field text
Educational web	HuggingFaceFW/fineweb-edu (sample-10BT)	split train, field text

The mix is legal-first, not an arbitrary ratio: the two legal sources only yield ~2B clean tokens total, so the pipeline takes all of case law and all of SEC, then tops up with a small educational-web slice for general fluency. Realized token shares after cleaning, dedup, and tokenization:

Source	Tokens	Share
US case law	715M	35%
SEC filings	860M	42%
Educational web	464M	23%
Total (train)	2.04B	100%

Held-out validation set: 20.6M tokens (20,119 windows), a clean 99/1 split (every 100th packed window), so validation documents never appear in training.

Training procedure

Initialization	random (from scratch — no pretrained base)
Hardware	8× NVIDIA B200 GPUs, on-prem (not cloud-rented)
Distribution	8-way DDP, bf16 autocast, SDPA/flash attention, torch.compile
Epochs / steps	2 epochs, 38,890 steps
Global batch	524,288 tokens (512 windows/step)
Optimizer	AdamW, β=(0.9, 0.95), weight decay 0.1 (2D params only), grad-clip 1.0
LR schedule	6e-4 → 6e-5 cosine decay, 200M-token linear warmup
Sequence packing	documents packed into contiguous 1,024-token windows separated by `<

Evaluation

Held-out perplexity was tracked throughout training and independently recomputed over the full 20.6M-token validation set (not a sample) at the final checkpoint:

Step	1,000	5,000	10,000	15,000	20,000	25,000	30,000	35,000	final (38,889)
Perplexity	16.4	10.1	10.0	9.3	8.9	8.6	8.4	8.3	7.76

Final held-out perplexity: 7.76 (validation loss 2.049), full-set, not a periodic in-training sample. Sample completions are coherent, on-domain legal/financial prose (see the live demo) while, as a base model, inventing all specifics.

How to use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("DeependraVerma/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained(
    "DeependraVerma/slm-125m-base", torch_dtype=torch.bfloat16
)

prompt = "The plaintiff filed a motion for summary judgment, arguing that"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(
    **inputs,
    max_new_tokens=150,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),
    pad_token_id=tok.convert_tokens_to_ids("<|pad|>"),
)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

For browser/edge inference (transformers.js, no server), use the ONNX int8-quantized export: DeependraVerma/slm-125m-base-onnx.

Citation

@misc{verma2026legalslm125m,
  author = {Deependra Verma},
  title  = {legal-slm-125M: A 125M-Parameter Legal and Financial Language Model Trained From Scratch},
  year   = {2026},
  url    = {https://huggingface.co/DeependraVerma/slm-125m-base},
  note   = {Code: https://github.com/DeependraVerma/legal-slm-125M}
}

Author

Deependra Verma — Generative AI Researcher / AI Engineer. GitHub · Hugging Face

License

MIT — see LICENSE in the source repo. This is a research artifact, not a source of legal or financial advice.