HobbyLM-Base (500M sparse-MoE foundation LM)

HobbyLM-Base is the foundation the whole family is built on: a 500M-parameter sparse Mixture-of-Experts decoder trained from scratch on FineWeb — no distillation, no borrowed weights. It exists to answer a simple question: how far can you get at the ~500M scale if you sweat the architecture and the training recipe instead of throwing tokens at the problem?

It's part of the HobbyLM family — a 500M sparse-MoE model (and its variants) built from scratch on a hobby budget: FineWeb, a handful of Modal H100 hours, a lot of ablations, and a from-scratch Rust engine (hobby-rs) to run it on a laptop CPU.

Intended use

A pretrained base model for text completion, and the checkpoint you fine-tune for downstream tasks. It is not instruction-tuned — for chat, use HobbyLM-Chat.

Architecture

Every HobbyLM variant shares one core: a sparse Mixture-of-Experts (MoE) decoder in the modern small-MoE style (DeepSeek-V3 / OLMoE lineage), where each design choice was picked by ablation rather than by guesswork.

Component	Value
Total parameters	~500M (only a fraction is active per token)
Hidden size / layers	768 / 16 (first FFN dense, the rest MoE)
Routed experts / active	36 / top-6 (+ 1 always-on shared expert)
Attention	GQA, 12 query / 3 KV heads, decoupled head-dim 128, per-head QK-norm
Router	sigmoid gating, DeepSeek-V3 aux-loss-free load balancing, no top-k renorm
Positional	RoPE (θ up to 1e6 for the 8k-context checkpoints)
Tokenizer	GPT-2 byte-level BPE (50,304 vocab, sentinel-padded)
Optimizer	Muon on the 2-D + per-expert matrices, AdamW on everything else

The full ablation log (QK-norm is the single biggest lever; aux-loss-free beats classic aux-loss; ≥32 experts and top-6 help; embedding-scaling hurt) lives in the project's architecture notes.

Benchmarks

0-shot, 7-task average through our harness (see note below). HobbyLM was trained on 40B tokens — a tiny budget next to the comparison models — so the right way to read this table is per training token.

Model	Params	Pretrain tokens	Avg (7-task)
SmolLM2-360M	360M	~4T	56.29
Qwen3-0.6B	600M	~36T	54.78
gemma-3-270m	270M	—	48.09
pythia-410m	410M	300B	45.34
HobbyLM-Base (500M)	500M	40B	44.05
opt-350m	350M	180B	43.61
HobbyLM-130M (sibling)	130M	10B	42.97
MicroLlama-300M	300M	50B	42.23
gpt2	124M	—	40.62
pythia-160m	160M	300B	38.60

Per-task (0-shot): HellaSwag 41.5 · LAMBADA 40.0 · SciQ 70.3 · PIQA 69.6 · ARC-easy 42.7 (ARC-challenge / WinoGrande sit near chance, as expected at this scale). Validation loss: 3.03 at 1k context, 2.94 after the 8k context-extension.

The ranking tracks pretraining tokens, not parameters: the top models see 50–900× more data than we do. In the classic ≤300B-token regime, HobbyLM leads per token — the 130M (10B tokens) beats MicroLlama-300M (50B), opt-350m (180B) and pythia-160m (300B). Token budget, not architecture, is the gap.

How these were measured. All language-model scores are 0-shot through our own port of EleutherAI's lm-evaluation-harness (a custom MoELMWrapper that runs log-likelihood scoring over the HobbyLM MoE + GPT-2 tokenizer). Reference models in the comparison table were run through the identical harness and task set, so the numbers are apples-to-apples with ours — they are not copied from other model cards. We validated the harness against published cards (e.g. TinyLlama 52.75 vs card 52.99). These are small research models: read the numbers in context, not as leaderboard claims.

Usage

Python (PyTorch reference implementation)

HobbyLM is a custom sparse-MoE architecture — there's no transformers AutoModel for it, so load it with the small reference implementation from the GitHub repo:

# HobbyLM is a CUSTOM sparse-MoE architecture, so load it with the reference implementation —
# NOT transformers.AutoModelForCausalLM (there is no AutoModel mapping for this arch).
# pip install torch safetensors tiktoken huggingface_hub
# git clone https://github.com/harishsg993010/HobbyLM && cd HobbyLM

import json, torch, tiktoken
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from hobbylm.config import ModelConfig
from hobbylm.model import MoETransformer
from hobbylm.generate import generate

repo = "rootxhacker/HobbyLM-Base"
cfg = ModelConfig(**{k: v for k, v in json.load(open(hf_hub_download(repo, "config.json"))).items() if k != "preset"})
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
cfg.expert_backend = "grouped" if device.type == "cuda" else "bmm"

model = MoETransformer(cfg).to(device).eval()
model.load_state_dict(load_file(hf_hub_download(repo, "model.safetensors")))

enc = tiktoken.get_encoding("gpt2")
prompt = "The capital of France is"
ids = torch.tensor([enc.encode_ordinary(prompt)], device=device)
out = generate(model, ids, max_new_tokens=64, temperature=0.7, top_k=0, device=device,
               repetition_penalty=1.3)               # temperature=0.0 for greedy
print(enc.decode(out[0].tolist()))

GGUF + hobby-rs (CPU)

GGUF builds (architecture hobbylm) live in rootxhacker/HobbyLM-gguf. They load directly in the from-scratch hobby-rs CPU engine — stock llama.cpp won't load them without registering the hobbylm architecture first.

hobby-rs --model HobbyLM-Base.gguf --prompt "..." --n 64

Training

Pretrained on ~40B unique FineWeb tokens (8×H100), then context-extended 1k→8k (RoPE θ 1e4→1e6). Muon on the hidden + per-expert matrices, AdamW on the router/embeddings/norms; fp32 router; chunked-checkpointed cross-entropy to fit a larger batch.

Limitations

• It's a ~500M base model on a 40B-token budget: fluent and factually-okay on easy questions, but it hallucinates and can repeat without a repetition penalty at decode time.
• Trained on English FineWeb; other languages and code are out of distribution.
• Not aligned or safety-tuned.

License

Apache-2.0. Weights aren't a substitute for judgement — this is a research / hobby model at the 500M scale, not a production system.