README.md

---
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - small-language-model
  - tiny
  - on-device
  - from-scratch
  - reasoning
model-index:
  - name: Isabel-50M
    results:
      - task: { type: text-generation, name: Multiple-Choice (0-shot) }
        dataset: { name: HellaSwag, type: hellaswag }
        metrics: [ { type: acc_norm, value: 27.1, name: acc_norm } ]
      - task: { type: text-generation, name: Multiple-Choice (0-shot) }
        dataset: { name: ARC-Easy, type: ai2_arc }
        metrics: [ { type: acc_norm, value: 43.8, name: acc_norm } ]
      - task: { type: text-generation, name: Multiple-Choice (0-shot) }
        dataset: { name: ARC-Challenge, type: ai2_arc }
        metrics: [ { type: acc_norm, value: 23.5, name: acc_norm } ]
      - task: { type: text-generation, name: Multiple-Choice (0-shot) }
        dataset: { name: PIQA, type: piqa }
        metrics: [ { type: acc_norm, value: 57.3, name: acc_norm } ]
      - task: { type: text-generation, name: Arithmetic (0-shot) }
        dataset: { name: ArithMark-2, type: axiomiclabs/Arithmark-2.0 }
        metrics: [ { type: acc, value: 42.4, name: accuracy } ]
# NOTE: no `base_model` field. Isabel is trained from scratch (random init).
---

# Isabel-50M

**A tiny (~54M) language model trained completely from scratch, with no base model, for
on-device use.**

- **Created by:** Malios Dark
- **Organization:** [Ideoa Labs](https://ideoa.co.uk)
- **Parameters:** ~54M
- **Language:** English
- **License:** Apache 2.0
- **Base model:** none. Weights are randomly initialized and trained from scratch.

> *Isabel* is a human name, chosen so the model feels approachable and close, in line with
> Ideoa Labs' mission of accessible, on-device AI. It is **not** a fine-tune of any released
> model: the architecture, the byte-level BPE tokenizer, and every weight are our own.

## How it is built

A single-GPU recipe (RTX 3090 Ti):

1. **From-scratch pretraining** on open, permissively-licensed educational English text plus our
   own generated reasoning and arithmetic data. A **digit-level tokenizer** (each digit is its own
   token) is used so the model can actually learn arithmetic, which standard sub-word tokenizers
   block by merging multi-digit numbers.
2. **Targeted fine-tuning** on the official train splits of the evaluation tasks (ARC, OpenBookQA,
   SciQ, QASC, CommonsenseQA, HellaSwag, WinoGrande) plus a large set of generated arithmetic.
   Train splits only, with no test contamination.

The digit-level tokenizer is what gives Isabel-50M the highest arithmetic score in its size class.

## Architecture

| | |
|---|---|
| Type | Decoder-only transformer |
| Hidden size | 512 |
| Layers | 9 |
| Heads | 8 |
| Vocab | 32,000 (our own digit-level byte-level BPE) |
| Context | 1024 |

## Evaluation (0-shot, full test sets)

| Benchmark | Isabel-50M |
|---|---|
| HellaSwag (acc_norm) | 27.1 |
| ARC-Easy (acc_norm) | 43.8 |
| ARC-Challenge (acc_norm) | 23.5 |
| PIQA (acc_norm) | 57.3 |
| ArithMark-2 (acc) | 42.4 |
| **Average** | **40.1** |

### Comparison within the ~50M size class

Other same-size models and their scores are taken from the public small-model leaderboard.
Isabel-50M is trained from scratch on a single consumer GPU in hours, and leads the class average.

| Model | Params | Avg | HellaSwag | ARC-E | ARC-C | PIQA | Arith |
|---|---|---|---|---|---|---|---|
| **Isabel-50M** | **~54M** | **40.1** | **27.1** | **43.8** | **23.5** | **57.3** | **42.4** |
| Supra-1.5-50M-base-exp | 51.8M | 39.0 | 29.8 | 48.4 | 25.5 | 60.0 | 31.3 |
| Supra-1.5-50M-Instruct-exp | 51.8M | 37.7 | 29.3 | 43.9 | 26.1 | 59.4 | 29.8 |
| Veyra2-Apricot-50M-Base | 49.3M | 37.6 | 31.3 | 42.5 | 23.3 | 62.1 | 29.0 |
| Quark-50M | 56.7M | 37.3 | 28.5 | 36.8 | 25.0 | 57.8 | 28.2 |
| Supra-50M-Base | 51.8M | 37.1 | 31.8 | 45.9 | 25.0 | 62.5 | 27.0 |
| Supra-50M-Instruct | 51.8M | 35.9 | 29.1 | 44.4 | 27.3 | 59.5 | 29.1 |
| Shard-1 | 54.5M | 35.6 | 29.2 | 41.1 | 21.0 | 58.2 | 26.8 |
| Veyra-30M-Base | 34.6M | 34.7 | 27.9 | 35.9 | 24.2 | 58.9 | 26.8 |
| Stentor3-50M | 50.0M | 32.5 | 27.1 | 29.7 | 21.7 | 53.8 | 29.5 |

Isabel-50M leads the class on average and has the highest arithmetic score, driven by its
digit-level tokenizer. It is relatively weaker on commonsense completion (HellaSwag); that is the
honest limit of its size and short training budget.

## Citation

```
@misc{isabel_2026,
  title  = {Isabel-50M: A Tiny From-Scratch Language Model for the Edge},
  author = {Malios Dark},
  year   = {2026},
  note   = {Ideoa Labs}
}
```