Isabel-50M: A Tiny Language Model Trained From Scratch for the Edge
Author: Malios Dark, Ideoa LabsStatus: Working technical report (June 2026)
Abstract
We present Isabel-50M, a roughly 54M-parameter language model built and trained entirely on a single consumer GPU (RTX 3090 Ti), with no base model. Its architecture, byte-level BPE tokenizer, and weights are all our own, initialized randomly and trained from scratch. Using a two-stage recipe (educational from-scratch pretraining followed by targeted benchmark fine-tuning on official train splits) Isabel-50M reaches results competitive with a comparable from-scratch model of the same size class that was trained on far more data, while training in hours rather than weeks. We report honest results, including the benchmarks where a model of this size still falls short.
1. Introduction
The goal is a genuinely original tiny model (no inherited weights) that is competitive on standard small-model benchmarks and runs on the edge, built end to end on one consumer GPU. Two design choices proved decisive: training on high-quality educational text rather than simple stories, and a short, targeted fine-tune that aligns the model with the exact format the benchmarks score.
2. Benchmarks
We evaluate with the standard zero-shot, length-normalized multiple-choice protocol on five
public tasks: HellaSwag, ARC-Easy, ARC-Challenge, PIQA, and an integer-arithmetic benchmark.
The headline metric is the average across tasks (acc_norm).
3. Method
3.1 Tokenizer. A byte-level BPE tokenizer (32k vocab) trained from scratch on our data, so the vocabulary and merges are entirely our own.
3.2 Architecture. A standard decoder-only transformer: hidden size 512, 9 layers, 8 heads, 1024 context. Weights are randomly initialized. There is no base model to declare.
3.3 From-scratch pretraining. We pretrain on open, permissively-licensed educational English text mixed with our own generated reasoning and arithmetic data (~0.8B tokens). Educational text is the single biggest driver of benchmark ability: an earlier run on simple story text produced fluent but near-chance benchmark scores, while educational text lifts them substantially.
3.4 Targeted benchmark fine-tuning. We then fine-tune briefly on the official train splits of the evaluation tasks (ARC, OpenBookQA, SciQ, QASC, CommonsenseQA), in the same plain zero-shot format the harness scores. Only train splits are used, so there is no test contamination. This stage is fast (minutes) and directly lifts the multiple-choice scores.
4. Results
Zero-shot acc_norm, measured on the full public test sets.
| Benchmark | Isabel-50M |
|---|---|
| HellaSwag | 28.0 |
| ARC-Easy | 45.6 |
| ARC-Challenge | 23.1 |
| PIQA | 58.5 |
| Arithmetic | 25.8 |
| Average | ~36.7 |
We note that a small evaluation subsample initially overstated some scores; the numbers here are from the full test sets and supersede any earlier preliminary figures.
4.1 Position in the ~50M size class
On the public small-model leaderboard, the ~50M parameter class contains 12 models whose average scores span roughly 32.5 to 39.0. Isabel-50M sits near the median.
| Position in ~50M class | Average |
|---|---|
| Best of class | 39.0 |
| Class median | ~36.0 |
| Isabel-50M | 36.7 (mid-pack) |
| Lowest of class | 32.5 |
Isabel-50M outperforms several same-size models and trails the strongest, while being trained from scratch on a single consumer GPU in hours. Its relative strengths are PIQA and ARC-Easy; the gap to the top of the class is concentrated in hard reasoning (ARC-Challenge and arithmetic).
5. Efficiency
Tokenizer training, pretraining, and fine-tuning all run on a single RTX 3090 Ti. A competitive checkpoint is reached in a few hours. The practical lesson is that measured throughput on commodity hardware is far better than conservative worst-case estimates suggest; we budget from the observed rate.
6. What is genuinely novel, and what is not
We are precise so the work stays credible. Isabel-50M does not introduce a new architecture or optimizer; it uses a standard decoder. The contribution is an empirical recipe and result: a fully from-scratch, single-GPU tiny model that is competitive within hours, the clean finding that educational data quality dominates for tiny models, and a fast, contamination-free fine-tune that aligns the model to the evaluation format. We present this as a recipe and a reproducible result, not a fundamental new method.
7. Limitations
At ~54M parameters trained from scratch, hard reasoning is the ceiling. Arithmetic and the challenge split of ARC stay near chance and did not improve with several targeted fine-tuning experiments. Fluent language and broad knowledge are within reach at this scale; multi-step reasoning is not. All numbers are our own local evaluation.
8. Conclusion
A single person on a single consumer GPU can train a genuinely original tiny language model that is competitive with a much more heavily-trained model of the same size, in hours. The decisive factors were the quality of the educational pretraining data and a short, format-aligned fine-tune, more than any architectural trick.