File size: 1,659 Bytes
12c39bc 3854527 12c39bc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | ---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- llama
- small-language-model
- pretrained-from-scratch
- legal
---
# slm-125m-base
A **125.8M-parameter** Llama-shaped language model pretrained
**from scratch** on a legal-first corpus (US case law + SEC filings + a slice of
FineWeb-Edu). Built with a custom 16,384-token BPE tokenizer.
This is a **base / foundation** model -- it does next-token continuation, not
instruction following or chat. Expect rough, domain-flavored completions; it was
trained on a small budget (step 19,334, ~10.14B tokens seen).
## Architecture
| | |
|---|---|
| Params | ~125.8M (tied embeddings) |
| Layers | 12 |
| Hidden size | 768 |
| Heads / KV heads | 12 / 12 (MHA) |
| Context length | 1,024 |
| Vocab | 16,384 (custom BPE) |
| Activation | SwiGLU (silu) |
| Position | RoPE (theta 10000) |
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("analyticspro/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("analyticspro/slm-125m-base")
model.eval()
ids = tok("The court held that", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=120, do_sample=True,
temperature=0.8, top_p=0.95,
pad_token_id=tok.eos_token_id)
print(tok.decode(out[0], skip_special_tokens=True))
```
## Limitations
Small model, small pretraining budget, and a legal-heavy corpus: outputs can be
factually wrong, repetitive, or biased toward legal/regulatory phrasing. Not
suitable for production or any high-stakes use. For research and demos only.
|