---
license: odc-by
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- legal
- finance
- llama
- small-language-model
---

# slm125MLIVE-base

A 125M-parameter LLaMA-architecture small language model pretrained from scratch
on a legal + financial corpus. Base model (no instruction tuning).

## Model
- Architecture: LLaMA (transformers `LlamaForCausalLM`), 12 layers / 768 hidden /
  12 heads (MHA), SwiGLU, RoPE, RMSNorm, tied embeddings.
- Params: ~125.8M. Context length: 1024. Vocab: 16,384 (byte-level BPE trained
  on this corpus).

## Training data (~2.04B tokens, 1 epoch)
Legal-first mix, cleaned / deduplicated / decontaminated:
- `HFforLegal/case-law` (US court opinions) — ~40%
- `PleIAs/SEC` (SEC filings) — ~40%
- `HuggingFaceFW/fineweb-edu` (educational web) — ~20%

Decontaminated against CaseHOLD / LexGLUE (13-gram overlap removal).

## Results
- Validation perplexity: **10.44** (full held-out set, 20.56M tokens).
- Final val loss: 2.346. Trained 3,889 steps at a 524,288-token global batch.

## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("sumitguha13/slm125MLIVE-base")
model = AutoModelForCausalLM.from_pretrained("sumitguha13/slm125MLIVE-base")
ids = tok("<|bos|>The plaintiff shall bear the burden of", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=60)[0], skip_special_tokens=True))
```

## Limitations
Base model: generates fluent, domain-appropriate text but invents facts and does
not follow instructions. Not for production or legal/financial advice.