skhotta/slm-125m-base
A ~125.8M parameter Llama-architecture base language model trained from scratch on a legal/financial corpus (US case law + SEC filings + a fineweb-edu slice), with a custom 16,384-token byte-level BPE tokenizer.
- Params: ~125.8M (12L / 768d / 12h, context 1024)
- Vocab: 16384 (byte-level BPE)
- Training data:
2.0B tokens (40% case law / ~40% SEC / ~20% web), deduplicated and decontaminated against CaseHOLD/LexGLUE. - Objective: causal language modeling.
This is a base model (no instruction tuning). It is a research artifact.
- Downloads last month
- 408