README.md

metadata

license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
  - llama
  - legal
  - finance
  - continued-pretraining
  - small-language-model

SLM-125M-e2 — a 125M legal/financial language model (2-epoch continued pretraining)

A 125.8M-parameter LLaMA-architecture base language model for US legal and SEC-filing text, and the strongest checkpoint in the slm-125m line. It is Ace-2504/slm-125m-extended continued for 2 epochs over a rebuilt, cleaner 2.50B-token corpus.

• Validation perplexity: 9.86 (down from 11.72 for slm-125m-extended)
• Not instruction-tuned — this is a base model that continues text, it does not answer questions.

What changed vs. the earlier checkpoints

This model exists to answer a research question: is the catastrophic forgetting seen in one-shot continued pretraining reversible? The slm-125m-extended run had trained on 515M tokens containing no SEC filings, and SEC ability degraded +8.1% as a result.

This run trained on a corpus that is ~35% SEC. Every domain improved, and the earlier forgetting reversed — measured on the base model's own held-out validation set (a fair within-family comparison, identical tokenizer, neither model trained on it):

Domain	slm-125m-extended → e2 (perplexity)	change
SEC filings	7.41 → 5.74	−12.8% (was +8.1% before)
US case law	11.51 → 9.80	−6.6%
Educational web	27.65 → 23.52	−4.9%

Conclusion: continued-pretraining forgetting is a function of data composition, not an inevitable cost — reintroduce a domain and the model recovers it. SEC, the model's strongest register, is now better than in any previous checkpoint.

Model

Architecture	LlamaForCausalLM, 125.8M params
Layers / hidden / heads	12 / 768 / 12 (head dim 64)
Context length	1,024
Vocabulary	16,384 (byte-level BPE, trained on the domain corpus)
Positional / norm / MLP	RoPE (θ=10,000) · RMSNorm · SwiGLU · tied embeddings
Precision	bf16

Training

Initialised from	Ace-2504/slm-125m-extended (weights only; optimizer reset)
Corpus	2.500B tokens — SEC 869M / US case law 855M / FineWeb-Edu 776M
Epochs / steps	2 epochs · 9,440 steps · 4.95B tokens processed
Schedule	100-step warmup → peak LR 3e-4 → cosine decay → 3e-5 floor
Batch	524,288 tokens/step (micro-batch 32 × grad-accum 16)
Optimizer	AdamW (β 0.9/0.95, wd 0.1), grad-clip 1.0
Hardware	1× A100 (SXM4), ~12.5 h

Model lineage

Checkpoint	cumulative tokens seen	val perplexity
slm-125m-base (from scratch, 1 epoch)	2.04B	~11.3
slm-125m-extended (+515M)	2.55B	11.72
slm-125m-e2 (this model, +4.95B / 2 epochs)	~7.5B	9.86

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("Ace-2504/slm-125m-e2")
model = AutoModelForCausalLM.from_pretrained("Ace-2504/slm-125m-e2", torch_dtype=torch.float32).eval()

prompt = "Pursuant to Section 10(b) of the Securities Exchange Act,"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.95, top_k=50)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Intended use & limitations

• Base model, not a chatbot. It completes text in a legal/financial register; it does not follow instructions or answer questions. For that, fine-tune it.
• Specialist, not general. It is strongest on SEC-filing / contract text (where it beats similarly-sized general models such as GPT-2 and Pythia-160M on bits-per-byte) and weaker on general web and educational text.
• No world knowledge guarantees. At 125M parameters with a 1,024-token context it holds little factual knowledge; anything factual should come from retrieval (RAG), not the weights.
• Trained on public US case law, SEC filings, and educational web text; it inherits their biases and coverage gaps.

Related models

• Ace-2504/slm-125m-base — the original from-scratch model.
• Ace-2504/slm-125m-extended — the 1-epoch +515M checkpoint this one continues from.