slm-125m-base

A 125M-parameter Llama-architecture small language model pretrained from scratch on a legal/financial corpus. This is a base (pretrained) model — it is not instruction- or chat-tuned.

Model details

Architecture	Llama (via transformers.LlamaForCausalLM)
Parameters	125.8M (tied embeddings)
Vocab	16,384 (byte-level BPE trained on this corpus)
Layers / hidden / heads	12 / 768 / 12 (head dim 64, MHA)
Context length	1,024
Positional	RoPE (θ=10,000)
Norm / activation	RMSNorm (1e-5) / SwiGLU (silu)
Precision	bf16 training, weights saved fp32

Training data (~2.04B tokens, "legal-first" mix)

Streamed, cleaned, deduplicated (MinHash LSH + exact), and decontaminated against CaseHOLD / LexGLUE before tokenization. Realized token mix:

Source	Share	HF dataset
US case law	~35%	HFforLegal/case-law (split us)
SEC filings	~42%	PleIAs/SEC
Educational web	~23%	HuggingFaceFW/fineweb-edu (sample-10BT)

Training

• 2 epochs (7,778 steps) on 8×H100 (DDP, torch.compile, SDPA/flash attention)
• Global batch 524,288 tokens; AdamW (β=0.9/0.95, wd 0.1, clip 1.0)
• LR 6e-4 → 6e-5 cosine, 200M-token linear warmup; seed 1337
• Throughput ~3.19M tok/s @ ~30% MFU

Evaluation

Held-out validation perplexity: 9.13 (loss 2.211, full 1% held-out split, 20,581,737 tokens / 20,119 packed 1024-token windows).

Validation loss over training (subset eval): 2.796 → 2.232 (steps 1000 → 7778).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("s2211252/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("s2211252/slm-125m-base", torch_dtype=torch.bfloat16)

prompt = "Pursuant to the terms of this Agreement, the parties"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=120, do_sample=True, top_k=50, top_p=0.95, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

Small base model, English only, 1,024-token context. Trained on legal/financial + web text; it is not instruction-tuned and can produce inaccurate or fabricated legal/financial statements. Not for legal advice or production decisions without review. Domain contamination against CaseHOLD/LexGLUE was filtered, but standard LM caveats apply.