README.md

---
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
  - llama
  - legal
  - finance
  - continued-pretraining
  - small-language-model
---

# SLM-125M-e2 — a 125M legal/financial language model (2-epoch continued pretraining)

A 125.8M-parameter LLaMA-architecture base language model for **US legal and SEC-filing
text**, and the strongest checkpoint in the `slm-125m` line. It is
`Ace-2504/slm-125m-extended` continued for **2 epochs** over a rebuilt, cleaner **2.50B-token**
corpus.

- **Validation perplexity: 9.86** (down from 11.72 for `slm-125m-extended`)
- Not instruction-tuned — this is a base model that *continues* text, it does not answer questions.

## What changed vs. the earlier checkpoints

This model exists to answer a research question: **is the catastrophic forgetting seen in
one-shot continued pretraining reversible?** The `slm-125m-extended` run had trained on
515M tokens containing **no SEC filings**, and SEC ability degraded +8.1% as a result.

This run trained on a corpus that is ~35% SEC. Every domain improved, and the earlier
forgetting **reversed** — measured on the base model's own held-out validation set (a fair
within-family comparison, identical tokenizer, neither model trained on it):

| Domain | slm-125m-extended → e2 (perplexity) | change |
|---|---|---|
| **SEC filings** | 7.41 → **5.74** | **−12.8%** (was +8.1% before) |
| US case law | 11.51 → 9.80 | −6.6% |
| Educational web | 27.65 → 23.52 | −4.9% |

**Conclusion:** continued-pretraining forgetting is a function of data composition, not an
inevitable cost — reintroduce a domain and the model recovers it. SEC, the model's strongest
register, is now better than in any previous checkpoint.

## Model

| | |
|---|---|
| Architecture | LlamaForCausalLM, 125.8M params |
| Layers / hidden / heads | 12 / 768 / 12 (head dim 64) |
| Context length | 1,024 |
| Vocabulary | 16,384 (byte-level BPE, trained on the domain corpus) |
| Positional / norm / MLP | RoPE (θ=10,000) · RMSNorm · SwiGLU · tied embeddings |
| Precision | bf16 |

## Training

| | |
|---|---|
| Initialised from | `Ace-2504/slm-125m-extended` (weights only; optimizer reset) |
| Corpus | 2.500B tokens — SEC 869M / US case law 855M / FineWeb-Edu 776M |
| Epochs / steps | 2 epochs · 9,440 steps · 4.95B tokens processed |
| Schedule | 100-step warmup → peak LR 3e-4 → cosine decay → 3e-5 floor |
| Batch | 524,288 tokens/step (micro-batch 32 × grad-accum 16) |
| Optimizer | AdamW (β 0.9/0.95, wd 0.1), grad-clip 1.0 |
| Hardware | 1× A100 (SXM4), ~12.5 h |

### Model lineage

| Checkpoint | cumulative tokens seen | val perplexity |
|---|---|---|
| `slm-125m-base` (from scratch, 1 epoch) | 2.04B | ~11.3 |
| `slm-125m-extended` (+515M) | 2.55B | 11.72 |
| **`slm-125m-e2` (this model, +4.95B / 2 epochs)** | **~7.5B** | **9.86** |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("Ace-2504/slm-125m-e2")
model = AutoModelForCausalLM.from_pretrained("Ace-2504/slm-125m-e2", torch_dtype=torch.float32).eval()

prompt = "Pursuant to Section 10(b) of the Securities Exchange Act,"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.8, top_p=0.95, top_k=50)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
```

## Intended use & limitations

- **Base model, not a chatbot.** It completes text in a legal/financial register; it does
  not follow instructions or answer questions. For that, fine-tune it.
- **Specialist, not general.** It is strongest on SEC-filing / contract text (where it beats
  similarly-sized general models such as GPT-2 and Pythia-160M on bits-per-byte) and weaker
  on general web and educational text.
- **No world knowledge guarantees.** At 125M parameters with a 1,024-token context it holds
  little factual knowledge; anything factual should come from retrieval (RAG), not the
  weights.
- Trained on public US case law, SEC filings, and educational web text; it inherits their
  biases and coverage gaps.

## Related models

- `Ace-2504/slm-125m-base` — the original from-scratch model.
- `Ace-2504/slm-125m-extended` — the 1-epoch +515M checkpoint this one continues from.