| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - llama |
| - from-scratch |
| - legal |
| - finance |
| pipeline_tag: text-generation |
| --- |
| |
| # slm-125m-base |
|
|
| A **125M-parameter, Llama-style, decoder-only language model**, pretrained |
| **completely from scratch** (fresh weights, fresh 16K byte-level BPE |
| tokenizer) on a domain-specific corpus of US case law, SEC filings, and |
| general web text. |
|
|
| This is a **base completer, not a chat model** — give it a passage prefix and |
| it continues it. It has not been instruction-tuned, aligned, or fine-tuned for |
| any downstream task. |
|
|
| ## Architecture |
|
|
| | | | |
| |---|---| |
| | Parameters | ~125M | |
| | Layers | 12 | |
| | Hidden size | 768 | |
| | Attention heads | 12 (head dim 64), plain MHA | |
| | MLP | SwiGLU, inner 3072 | |
| | Normalization | RMSNorm, pre-norm | |
| | Position embeddings | RoPE | |
| | Context length | 1024 | |
| | Vocab | 16,384 (fresh byte-level BPE, no OOV token ever) | |
| | Embeddings | tied | |
|
|
| ## Training data |
|
|
| ~3.53B packed tokens across three sources, deduplicated (MinHash-LSH on the |
| legal corpus) and decontaminated against the LexGLUE/CaseHOLD held-out splits: |
|
|
| | Source | Tokens | Share | |
| |---|---|---| |
| | `HFforLegal/case-law` (US case law, `us` split) | ~1.1B | 31% | |
| | `PleIAs/SEC` (SEC filings) | ~2.0B | 57% | |
| | `HuggingFaceFW/fineweb-edu` (general web text) | ~1.0B | 28% | |
|
|
| (Shares don't sum to 100% due to rounding; case-law's split was capped at its |
| full-dataset ceiling, not a target percentage — see the build's `docs/01-data.md`.) |
|
|
| ## Training |
|
|
| AdamW (β 0.9/0.95, weight decay 0.1), linear warmup + cosine LR, bf16 |
| (autocast, fp32 master weights), single-node DDP across 8x H100. Trained to |
| step 7,500 (~4.1B tokens processed, more than one full epoch over the |
| training set) before the run was stopped. |
|
|
| ## Evaluation |
|
|
| **Held-out validation perplexity: 10.14** (full validation set, 34,513 |
| windows). This is the honest headline metric at this scale — not MMLU or |
| similar benchmarks, which are near-random for a 125M model. |
|
|
| Perplexity varies a lot by domain, reflecting the training mix: |
|
|
| | Domain | Perplexity | |
| |---|---| |
| | SEC filings | ~6.0 | |
| | Case law | ~15.3 | |
| | General web text (fineweb-edu) | ~27.4 | |
|
|
| The model is noticeably better at the two legal/financial domains it mostly |
| trained on than at general web text. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("mickey5k/slm-125m-base") |
| model = AutoModelForCausalLM.from_pretrained("mickey5k/slm-125m-base") |
| |
| prompt = "The court held that" |
| input_ids = tokenizer(prompt, return_tensors="pt").input_ids |
| output_ids = model.generate( |
| input_ids, |
| max_new_tokens=100, |
| min_new_tokens=20, # a base model can emit EOS almost immediately otherwise |
| do_sample=True, |
| temperature=0.8, |
| top_p=0.9, |
| top_k=50, |
| ) |
| print(tokenizer.decode(output_ids[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Limitations |
|
|
| - **Base completer, not instruction-tuned.** It continues text; it does not |
| follow instructions or answer questions in a chat format. |
| - **Domain-skewed.** Strongest on legal/financial register, weaker on general |
| text (see per-domain perplexity above). |
| - **Small.** 125M parameters is far below the scale where benchmark-style |
| reasoning emerges — evaluate it on perplexity/completion quality, not |
| accuracy benchmarks. |
| - **Training was cost-capped**, not run to convergence — see the build's |
| `docs/05-pretrain.md` for the full training log and the reasoning behind |
| stopping at step 7,500. |
|
|
