README.md

---
license: cc-by-sa-3.0
language: en
tags:
  - text-generation
  - instruction-tuning
  - from-scratch
  - pytorch
datasets:
  - databricks/databricks-dolly-15k
---

# SmoLLM-109M-instruct

A **109M-parameter, Llama-style language model built entirely from scratch** — custom BPE tokenizer, RoPE, RMSNorm, SwiGLU, and multi-head causal attention — pretrained on FineWeb-Edu and instruction-tuned on `databricks-dolly-15k`.

This is the **instruction-following** variant. For the raw base model, see [SmoLLM-109M-base](https://huggingface.co/rohit-upadhya/SmoLLM-109M-base).

## Prompt template

This model uses a **custom chat template**. Format inputs exactly like this (note the trailing space after `[ASSISTANT]`):

```
[SYSTEM] You are a helpful bot [/SYSTEM]
[USER] your question here [/USER]
[ASSISTANT] 
```

The model was trained to emit `[EOS]` at the end of each response, so it terminates on its own for focused questions.

## Evaluation

| Metric | Value |
|---|---|
| Perplexity (WikiText-2 test) | 80.27 |
| Base model perplexity | 74.57 |
| Tokens evaluated | 290,889 |

Instruction-tuning shifts the model toward the chat format, so raw-text perplexity rises slightly (74.6 → 80.3). The small gap indicates base language ability was preserved — no catastrophic forgetting.

## Usage

This is a **custom architecture**, not a `transformers` `AutoModel`. Clone the repo for the model code, then load the weights from this repo.

**1. Clone and set up** (uses [uv](https://github.com/astral-sh/uv)):

```bash
git clone https://github.com/rohit-upadhya/smol-llm.git
cd smol-llm
uv sync
```

**2. Create `run.py` in the repo root:**

```python
from huggingface_hub import hf_hub_download
from src.inference.inference import Inference

repo = "rohit-upadhya/SmoLLM-109M-instruct"
weights = hf_hub_download(repo, "pytorch_model.bin")
tokenizer = hf_hub_download(repo, "tokenizer.json")

inf = Inference(model_name_or_path=weights, tokenizer_path=tokenizer)

prompt = (
    "[SYSTEM] You are a helpful bot [/SYSTEM]\n"
    "[USER] What is machine learning? [/USER]\n"
    "[ASSISTANT] "
)
print(inf.generate(prompt, max_tokens=100, temperature=0.7,
                   top_k=50, top_p=0.95, repetition_penalty=1.2))
```

**3. Run it:**

```bash
uv run python run.py
```

`hf_hub_download` pulls the weights and tokenizer straight from this repo — no manual downloads needed.

## Example outputs

**What is machine learning?**
> Machine Learning (ML) is the branch of computer science that focuses on building models and algorithms to perform tasks more efficiently, in order to create better services.

**Why is exercise important?**
> Exercise can help people who suffer from depression and anxiety. Exercise releases endorphins which may reduce symptoms of stress, depression, and anxiety.

**List three colors.**
> Red, Green and Blue

Real, unedited generations (temperature 0.8). The model is fluent and stops cleanly on focused questions — but at 109M parameters it will **confidently hallucinate facts** (inventing dates, people, or details). Treat it as a demonstration of small-model instruction-following, not a knowledge source.

## Architecture

| | |
|---|---|
| Parameters | 109.5M |
| Layers | 12 |
| Hidden dim | 768 |
| Attention heads | 12 |
| Context length | 512 |
| Tokenizer | Custom BPE (~32k vocab) |
| Components | RoPE, RMSNorm, SwiGLU, multi-head causal attention |

## Training

- **Base:** pretrained from scratch on FineWeb-Edu (`sample-10BT`), Chinchilla-optimal token budget (~20 tokens/param, ~2.2B tokens).
- **EOS continued-pretraining:** additional continued-pretraining to install end-of-sequence behavior at document boundaries (the base model originally never learned to stop).
- **Instruction tuning:** SFT on `databricks-dolly-15k` with prompt-token masking (loss computed only on response tokens) and `[EOS]`-terminated responses. LR 2e-5, cosine schedule, best checkpoint selected at epoch 2 by held-out eval loss (before overfitting onset).

## Recommended decoding

```python
generate(prompt, max_tokens=100, temperature=0.7,
         top_k=50, top_p=0.95, repetition_penalty=1.2)
```

## Limitations

- **109M parameters** — fluent but factually unreliable. Strong at short, focused generation; weak on facts, multi-step reasoning, and summarization.
- **In-context induction is weak** — relies on frequency-based memorization rather than in-context pattern learning.
- **Open-ended prompts ramble** — vague prompts give no natural stopping point, so termination is less reliable than for focused questions.
- Built as an **educational / research artifact** to understand LLM mechanics from the ground up, not for production use.

## Links

- Author: Rohit Upadhya
- GitHub: https://github.com/rohit-upadhya/smol-llm