HobbyLM-1B
Base model, zero function-calling training. Corrected Task 1 elicitation results show strong latent tool-use behavior: single-sample pass@1 is 88.1% format validity, 94.8% function selection, and 54.8% argument extraction. At pass@64, the same probes reach 100% / 100% / 96%.
HobbyLM-1B is a base pretrained sparse MoE language model. It is not instruction-tuned, not RLHF-tuned, and not safety-aligned. Any function-calling ability comes from pretraining, especially the code slice of the corpus, not from tool-call SFT or tool-use benchmark data.
Architecture
HobbyLM-1B is a sparse Mixture-of-Experts transformer with 1.037B total parameters and about 305M active parameters per token (29.4% active).
| Component | Value |
|---|---|
| Layers | 20 total: layer 0 dense, layers 1-19 MoE |
| Experts | 64 routed experts + 1 shared expert |
| Active experts | Top-8 routed experts + shared expert |
| Router | Sigmoid router with aux-loss-free bias, DeepSeek-V3 style |
| Attention | GQA, 16 query heads / 8 KV heads |
| QK norm | Per-head QK-RMSNorm |
| Position | Full RoPE, base 10000 |
| FFN | SwiGLU |
| Norm | RMSNorm, eps 1e-6 |
| Embeddings | Tied token embedding / LM head |
| Tokenizer | GPT-2 tokenizer |
| Vocab size | 50304 |
| Context length | 1024 tokens; context extension is in progress |
Training
HobbyLM-1B was trained from scratch on 100B tokens using 4x H200 GPUs for about 76 hours.
Training mix:
| Source | Share |
|---|---|
| FineWeb-Edu | 60% |
| DCLM | 15% |
| Code | 10% |
| Math | 10% |
| Anneal: Cosmopedia + high-quality FineWeb-Edu | 5% |
Training used the Muon optimizer and a trapezoidal learning-rate schedule.
Final main-phase validation loss: 3.4112.
Benchmarks
7-task lm-eval, 0-shot:
| Task | Score |
|---|---|
| HellaSwag | 43.66 |
| PIQA | 68.34 |
| WinoGrande | 52.41 |
| ARC Easy | 54.59 |
| OpenBookQA | 35.20 |
| ARC Challenge | 29.52 |
| BoolQ | 49.54 |
| Average | 47.61 |
Function-Calling Elicitation
Synthetic probes only; no xLAM, BFCL, ToolACE, Glaive, or other function-calling benchmark/training data was used.
Corrected HF generate() run with the KV-cache fix, k=64, temperature=1.0, top_p=1.0, max 64 new tokens:
| Probe | pass@1 | pass@64 |
|---|---|---|
| Format validity | 88.14% | 100.00% |
| Argument extraction | 54.83% | 96.00% |
| Function selection | 94.81% | 100.00% |
Earlier lower numbers for this test were measured through a KV-cache bug and should not be used for planning or comparison.
Precision Warning
This model is fp32 and must stay fp32. Converting the router to bf16 changes which experts are selected: router top-1 agreement drops to 62%, and full-model output agreement drops from 97.6% to 61%.
With top-8-of-64 routing, the score gap between rank 8 and rank 9 is below bf16's representable precision. Quantizing this model will silently change its behavior.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "harims95/hobbylm-1b-hf"
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
)
This repository uses custom modeling code, so trust_remote_code=True is required.
Links
- Training code: https://github.com/harims95/hobbylm-1b
- Base architecture: https://github.com/harishsg993010/HobbyLM
- Routing dynamics: https://huggingface.co/datasets/harims95/hobbylm-routing-dynamics
- Long-context corpus: https://huggingface.co/datasets/harims95/hobbylm-longctx-2p5b
Credits
- Harish: base architecture
- Hariharan: training pipeline, data engineering, infra
- Prajan: validation and analysis
Limitations
- Base model only; not instruction-tuned or aligned.
- 1024-token context window in this release.
- GPT-2 tokenizer is roughly 30% less token-efficient than modern tokenizers.
- Not evaluated for safety.
- vLLM support is not yet validated; use the HF implementation for correctness.
- Downloads last month
- 123