Upload folder using huggingface_hub

b560ff5 verified 3 days ago

4.79 kB

license: cc-by-sa-3.0
language: en
tags:
  - text-generation
  - instruction-tuning
  - from-scratch
  - pytorch
datasets:
  - databricks/databricks-dolly-15k

SmoLLM-109M-instruct

A 109M-parameter, Llama-style language model built entirely from scratch — custom BPE tokenizer, RoPE, RMSNorm, SwiGLU, and multi-head causal attention — pretrained on FineWeb-Edu and instruction-tuned on databricks-dolly-15k.

This is the instruction-following variant. For the raw base model, see SmoLLM-109M-base.

Prompt template

This model uses a custom chat template. Format inputs exactly like this (note the trailing space after [ASSISTANT]):

[SYSTEM] You are a helpful bot [/SYSTEM]
[USER] your question here [/USER]
[ASSISTANT]

The model was trained to emit [EOS] at the end of each response, so it terminates on its own for focused questions.

Evaluation

Metric	Value
Perplexity (WikiText-2 test)	80.27
Base model perplexity	74.57
Tokens evaluated	290,889

Instruction-tuning shifts the model toward the chat format, so raw-text perplexity rises slightly (74.6 → 80.3). The small gap indicates base language ability was preserved — no catastrophic forgetting.

Usage

This is a custom architecture, not a transformers AutoModel. Clone the repo for the model code, then load the weights from this repo.

1. Clone and set up (uses uv):

git clone https://github.com/rohit-upadhya/smol-llm.git
cd smol-llm
uv sync

2. Create run.py in the repo root:

from huggingface_hub import hf_hub_download
from src.inference.inference import Inference

repo = "rohit-upadhya/SmoLLM-109M-instruct"
weights = hf_hub_download(repo, "pytorch_model.bin")
tokenizer = hf_hub_download(repo, "tokenizer.json")

inf = Inference(model_name_or_path=weights, tokenizer_path=tokenizer)

prompt = (
    "[SYSTEM] You are a helpful bot [/SYSTEM]\n"
    "[USER] What is machine learning? [/USER]\n"
    "[ASSISTANT] "
)
print(inf.generate(prompt, max_tokens=100, temperature=0.7,
                   top_k=50, top_p=0.95, repetition_penalty=1.2))

3. Run it:

uv run python run.py

hf_hub_download pulls the weights and tokenizer straight from this repo — no manual downloads needed.

Example outputs

What is machine learning?

Machine Learning (ML) is the branch of computer science that focuses on building models and algorithms to perform tasks more efficiently, in order to create better services.

Why is exercise important?

Exercise can help people who suffer from depression and anxiety. Exercise releases endorphins which may reduce symptoms of stress, depression, and anxiety.

List three colors.

Red, Green and Blue

Real, unedited generations (temperature 0.8). The model is fluent and stops cleanly on focused questions — but at 109M parameters it will confidently hallucinate facts (inventing dates, people, or details). Treat it as a demonstration of small-model instruction-following, not a knowledge source.

Architecture


Parameters	109.5M
Layers	12
Hidden dim	768
Attention heads	12
Context length	512
Tokenizer	Custom BPE (~32k vocab)
Components	RoPE, RMSNorm, SwiGLU, multi-head causal attention

Training

Base: pretrained from scratch on FineWeb-Edu (sample-10BT), Chinchilla-optimal token budget (~20 tokens/param, ~2.2B tokens).
EOS continued-pretraining: additional continued-pretraining to install end-of-sequence behavior at document boundaries (the base model originally never learned to stop).
Instruction tuning: SFT on databricks-dolly-15k with prompt-token masking (loss computed only on response tokens) and [EOS]-terminated responses. LR 2e-5, cosine schedule, best checkpoint selected at epoch 2 by held-out eval loss (before overfitting onset).

Recommended decoding

generate(prompt, max_tokens=100, temperature=0.7,
         top_k=50, top_p=0.95, repetition_penalty=1.2)

Limitations

109M parameters — fluent but factually unreliable. Strong at short, focused generation; weak on facts, multi-step reasoning, and summarization.
In-context induction is weak — relies on frequency-based memorization rather than in-context pattern learning.
Open-ended prompts ramble — vague prompts give no natural stopping point, so termination is less reliable than for focused questions.
Built as an educational / research artifact to understand LLM mechanics from the ground up, not for production use.

rohit-upadhya
/

SmoLLM-109M-Instruct