---
language: en
license: mit
tags:
- slm
- llama
- from-scratch
- it-support
- call-centre
datasets:
- HuggingFaceFW/fineweb-edu
- ArmelR/the-pile-splitted
- uonlp/CulturaX
- mlfoundations/dclm-baseline-1.0
pipeline_tag: text-generation
library_name: transformers
---

# Support 125M SLM - Base

A **125M parameter Llama-style language model** trained from scratch on ~2.6B tokens of curated IT support and technical data. This is the **base (pretrained)** model — it completes text but does not follow instructions.

## Training Data

| Source | Tokens | Description |
|--------|--------|-------------|
| FineWeb-Edu | 900M | High-quality educational web text |
| Ubuntu IRC | 600M | Technical support chat logs |
| StackExchange | 1.05B | Q&A from StackExchange network |
| DCLM | 300M | Filtered web text |

### Total: ~2.85B tokens (6 epochs = ~17B tokens seen)

## Model Architecture

| Parameter | Value |
|-----------|-------|
| Parameters | 125,847,552 |
| Layers | 12 |
| Hidden dim | 768 |
| FFN dim | 3072 (SwiGLU) |
| Attention heads | 12 |
| KV heads | 12 (MHA) |
| Vocab size | 16,384 |
| Context length | 1,024 |
| Position encoding | RoPE |
| Norm | RMSNorm |
| Tie embeddings | Yes |

## Training Details

- **Hardware:** 8x H100 (Modal cloud)
- **Framework:** PyTorch + DDP
- **Optimizer:** AdamW (lr=6e-4, warmup 200M tokens, cosine decay)
- **Mixed precision:** bfloat16
- **Total cost:** ~$31
- **Val perplexity:** 15.06

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("applegrew/support-125M-slm-base")
tokenizer = AutoTokenizer.from_pretrained("applegrew/support-125M-slm-base")

prompt = "The VPN connection keeps dropping"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Special Tokens

This model uses custom chat tokens: `<|bos|>`, `<|eos|>`, `<|pad|>`, `<|unk|>`, `<|system|>`, `<|user|>`, `<|assistant|>`

## SFT Version

For instruction following, use the SFT version: [applegrew/support-125M-slm-sft](https://huggingface.co/applegrew/support-125M-slm-sft)