akhil-2019/slm-125m-base

SLM-125M · a legal & financial base model

A 125-million-parameter, Llama-style decoder for a legal-and-financial-first corpus: US case law, SEC filings, and educational web text. It is a compact base model, fluent in the register of law and finance.

This is a research artifact, not an instruction-tuned assistant. It continues text; it does not answer questions or follow instructions, and it has no safety alignment. Do not rely on it for legal, financial, or other professional advice.

Highlights

Parameters	125.8M
Vocabulary	16,384 byte-level BPE (custom)
Context length	1,024 tokens
Training tokens	2.04B unique (+20.6M held-out val)
Epochs	6 (5 base + 1 continuation)
Validation perplexity	8.41
Continuation compute	~20 min on 8×H100 ($10.53)

Architecture

Standard Llama-family decoder, mapped 1:1 to transformers.LlamaConfig:

Field	Value
Hidden size	768
Layers	12
Attention heads	12 (multi-head; 12 KV heads)
Head dimension	64
Feed-forward	SwiGLU, intermediate 3,072
Positional encoding	RoPE (θ = 10,000)
Normalization	RMSNorm (ε = 1e-5)
Embeddings	tied input/output
Precision	bfloat16

Training data

The two legal sources are small by nature, so the mix is legal-first (~40 / 40 / 20) rather than the usual web-heavy blend. Realized token counts over the continuation corpus:

Source	Share	Tokens	What it is
SEC filings (PleIAs/SEC)	42.2%	861M	10-K / 10-Q, born-digital
US case law (HFforLegal/case-law)	35.0%	713M	court opinions
Educational web (HuggingFaceFW/fineweb-edu)	22.8%	465M	fluency filler

Data pipeline: stream three public datasets → 6-step deterministic cleaning (line filtering, boilerplate stripping, repetition and language checks, an OCR-garble gate on scanned opinions) → dedup + decontamination (MinHash-LSH near-duplicate removal + 13-gram eval-set stripping) → a fresh 16,384 byte-level BPE tokenizer → pack into 1,024-token windows with a 99/1 train-val split. The full pipeline runs on CPU for under $1.

Training procedure

The full data pipeline and the 16K tokenizer here were built from scratch; the weights are warm-started from another base checkpoint.

This release continues pretraining from another release where the base checkpoint was reportedly pretrained for 5 epochs. This release continues it for 1 additional epoch via warm-start: weights loaded from the base checkpoint with a fresh AdamW optimizer and a gentle cosine schedule (peak LR 1e-4 → 6e-5, 20M-token warmup) on 8×H100. Validation perplexity improved to 8.41. Special-token ids: <|bos|>=0, <|eos|>=1, <|pad|>=2.

The continuation epoch used our own re-cleaned corpus - the same source distribution as the base's data, but not byte-identical - so it is best described as "5 epochs on the base data plus 1 on ours," not a literal 6th pass over the identical token stream.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("akhil-2019/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("akhil-2019/slm-125m-base")

prompt = "The plaintiff alleges that the defendant"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=80, do_sample=True,
                     temperature=0.8, top_p=0.95, top_k=50,
                     repetition_penalty=1.3, no_repeat_ngram_size=3,
                     eos_token_id=1, pad_token_id=2)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Limitations & intended use

• Base completer only - no instruction-following, chat, or safety tuning.
• Can produce fluent-sounding but incorrect or outdated statements; knowledge is bounded by the training data.
• English-only; specialized to legal/financial prose and weaker on general text.
• Not for legal, financial, or other professional advice. Outputs are text continuations, not verified facts.

License

Released under Apache-2.0. Underlying datasets retain their own licenses.