| license: apache-2.0 | |
| language: en | |
| tags: | |
| - legal | |
| - finance | |
| - from-scratch | |
| - llama | |
| # slm-125m-base | |
| A 125.8M-parameter Llama-architecture language model, trained from scratch on a legal/financial corpus. | |
| ## Data | |
| Trained on ~2.19B tokens: US case law (~40%), SEC filings (~40%), and FineWeb-Edu web text (~20%). See the project's `Agent_brief.md` for the full data pipeline (cleaning, dedup, decontamination). | |
| ## Architecture | |
| - vocab_size: 16384 | |
| - hidden_size: 768 | |
| - num_hidden_layers: 12 | |
| - num_attention_heads: 12 | |
| - max_position_embeddings: 1024 | |
| ## Training | |
| Pretrained on 8xH100 via Modal, budget-capped at $40. | |