slm-500m-base

A 528.5-million-parameter Llama-style language model trained completely from scratch — random initialization, its own tokenizer (shared with this author's earlier 125M build), its own data pipeline, its own pretraining run. This is the scaled-up successor to DeependraVerma/slm-125m-base, built to test whether more parameters and more (better-balanced) legal/financial data would produce a genuinely stronger model, not just a bigger one.

• Repo / full build: github.com/DeependraVerma/legal-slm-125M
• Author: Deependra Verma — Generative AI Researcher / AI Engineer (Hugging Face)
• Fine-tuned Q&A version: DeependraVerma/legal-slm-500m-sft
• Predecessor (125M): DeependraVerma/slm-125m-base

This is a base (completion) model, not an assistant. It continues text; it does not follow instructions or answer questions reliably. For a Q&A-capable model, use legal-slm-500m-sft. Never use this model's output as legal, financial, or factual advice — it invents specifics (names, numbers, citations) like any base language model, and this is a genuine, measured limitation, not a hypothetical one (see "Known limitations" below).

Model description

Parameters	528,538,752 (~528.5M, tied embeddings)
Architecture	Llama-style decoder (12→24 layers, 768→1152 hidden vs. the 125M build), full multi-head attention (no GQA/MQA)
Layers / hidden / heads	24 / 1152 / 18 (head dim 64)
MLP	SwiGLU, intermediate size 4,608
Positional encoding	RoPE, θ = 10,000
Normalization	RMSNorm, ε = 1e-5
Vocabulary	16,384, byte-level BPE — the same tokenizer as the 125M build, reused deliberately rather than retrained
Context length	1,024 tokens
Embeddings	tied input/output
Precision	bf16 compute (autocast) · fp32 master weights and saved checkpoint

Training data

Seven sources, expanded in two stages. The original five (~14.7B unique tokens): US case law, SEC 10-K filings, educational web text, bulk SEC 10-K/10-Q/8-K filings, and bulk US case law. A subsequent continuation run added two new government sources plus more SEC filing types:

Source	HF dataset / origin	License
US case law	HFforLegal/case-law	CC BY 4.0
SEC 10-K filings	PleIAs/SEC	—
Educational web	HuggingFaceFW/fineweb-edu	ODC-BY
SEC 10-K/10-Q/8-K/S-1/20-F filings (bulk)	TeraflopAI/SEC-EDGAR	Apache-2.0
US case law (bulk)	common-pile/caselaw_access_project	Public domain
Code of Federal Regulations (CFR)	govinfo.gov bulk XML API	Public domain (US government work)
Federal Register	govinfo.gov bulk XML API	Public domain (US government work)

The mix isn't static. A flat/proportional mix of the original five sources was tried first and measurably diluted the original case-law/SEC/fineweb-edu "identity" of the 125M build down to ~13% of training data, causing a real regression on general text (fineweb-edu perplexity got worse than the 125M model, 22.82 vs 20.73). The fix — adopted from precedent in MiniCPM's and SmolLM2's published training recipes — is a staged (Warmup-Stable-Decay) schedule: broad, balanced per-source sampling weights for most of training, then a decay phase near the end that upweights the curated core sources specifically. The schedule was recomputed for all seven sources when CFR and the Federal Register were added, and the training budget was extended from a single-epoch, 28,054-step run to roughly 200 tokens/parameter (201,620 steps) via a checkpoint-resumed continuation.

Evaluation

Full held-out perplexity, per source (not just one blended number, since the seven sources are genuinely different registers of text):

Source	Perplexity (7-source)	Perplexity (original 5-source)
SEC 10-K/10-Q/8-K/S-1/20-F (bulk)	3.38	3.96
SEC-EDGAR (bulk)	3.83	3.84
Federal Register	4.18	(new source)
CFR	4.95	(new source)
US case law	6.62	7.23
US case law (bulk)	7.63	8.23
Educational web	18.07	17.81
Aggregate (all 7 sources)	4.642	5.072

Both new sources (CFR, Federal Register) score better than case law individually — regulatory/administrative prose is comparatively predictable text, and this is a genuinely encouraging sign the expansion is being learned as intended rather than just diluting the mix. Compared against two current small open-weight general-purpose models (OLMo-2-1B, Gemma-3-270M) on external benchmarks (HellaSwag, ARC-Easy, PIQA, MMLU-legal, CaseHOLD, LegalBench), this checkpoint closes most of the gap to the ~2x-larger OLMo-2-1B on legal-domain-specific tasks specifically (CaseHOLD, LegalBench), while trailing on general commonsense reasoning as expected for a narrow-domain model at this scale — and beats same-parameter-count Gemma-3-270M on every benchmark measured. Full tables, methodology, and raw logs: github.com/DeependraVerma/legal-slm-125M.

Intended uses & limitations

Intended use: text continuation / completion in a legal and financial register; research and education on small-model pretraining, data-mix design, and staged training schedules.

Not intended for: question answering, instruction following, or any use where factual accuracy matters. Like the 125M base model (and any base model at this scale), it will fabricate case names, statute citations, dollar figures, and other specifics that sound right but are not grounded in anything. A 528.5M-parameter model can store, at most, roughly 2 bits of knowledge per parameter (Allen-Zhu & Li, "Physics of Language Models: Knowledge Capacity Scaling Laws") — about 132MB of total compressible fact storage, shared across grammar, style, and every fact it knows. This is a hard, measured ceiling, not a training defect. Do not use its output as legal, financial, or factual advice, or present it as such to end users without a human review step. For a model that at least attempts to answer questions, see legal-slm-500m-sft — and note that model has its own documented hallucination limitations too.

Citation

@misc{verma2026slm500mbase,
  author = {Deependra Verma},
  title  = {slm-500m-base: A 528.5M-Parameter Legal/Financial Language Model Trained From Scratch},
  year   = {2026},
  url    = {https://huggingface.co/DeependraVerma/slm-500m-base},
  note   = {Code: https://github.com/DeependraVerma/legal-slm-125M}
}

Author

Deependra Verma — Generative AI Researcher / AI Engineer. GitHub · Hugging Face