SmoLLM-109M-base

A 109M-parameter, Llama-style language model built entirely from scratch — custom BPE tokenizer, RoPE, RMSNorm, SwiGLU, and multi-head causal attention — pretrained on FineWeb-Edu, with continued pretraining to install end-of-sequence (EOS) behavior at document boundaries.

This is the base model (text continuation). For the instruction-following version, see SmoLLM-109M-instruct.

Evaluation

Metric	Value
Perplexity (WikiText-2 test)	74.57
Avg NLL	4.31
Tokens evaluated	290,889

SmoLLM was pretrained on FineWeb-Edu, so WikiText is out-of-distribution — this is a fair but conservative measure.

Usage

This is a custom architecture, not a transformers AutoModel. Clone the repo for the model code, then load the weights from this repo.

1. Clone and set up (uses uv):

git clone https://github.com/rohit-upadhya/smol-llm.git
cd smol-llm
uv sync

2. Create run.py in the repo root:

from huggingface_hub import hf_hub_download
from src.inference.inference import Inference

repo = "rohit-upadhya/SmoLLM-109M-base"
weights = hf_hub_download(repo, "pytorch_model.bin")
tokenizer = hf_hub_download(repo, "tokenizer.json")

inf = Inference(model_name_or_path=weights, tokenizer_path=tokenizer)

# the base model continues text — it does not follow instructions
print(inf.generate("Machine learning is ", max_tokens=60, temperature=0.7,
                   top_k=50, top_p=0.95, repetition_penalty=1.2))

3. Run it:

uv run python run.py

hf_hub_download pulls the weights and tokenizer straight from this repo — no manual downloads needed.

Architecture


Parameters	109.5M
Layers	12
Hidden dim	768
Attention heads	12
Context length	512
Tokenizer	Custom BPE (~32k vocab)
Components	RoPE, RMSNorm, SwiGLU, multi-head causal attention

Training

Pretrained from scratch on FineWeb-Edu (sample-10BT), Chinchilla-optimal token budget (~20 tokens/param, ~2.2B tokens).
Continued-pretraining pass to install [EOS] emission at document boundaries. The original pretraining never appended EOS at document boundaries, so the model never learned to stop — this checkpoint corrects that, raising EOS from effectively unreachable (rank ~30,000) to a samplable position at natural boundaries.

Behavior notes

This model continues text; it does not follow instructions or reliably stop on its own (that behavior is added in the instruct variant).

Memorization vs. induction: the model reliably completes high-frequency memorized sequences (common phrases) but fails at in-context pattern induction — a repeated a b a b prompt collapses rather than continuing. Consistent with weak induction-head formation at this scale.
Fluent, but factually unreliable — it memorizes patterns rather than facts.

Limitations

109M parameters — fluent but factually limited. Memorizes frequent patterns; weak on facts and reasoning. Built as an educational / research artifact for understanding LLM pretraining mechanics from the ground up, not a production model.

rohit-upadhya
/

SmoLLM-109M-base