---
title: NanoDex
emoji: 🧬
colorFrom: green
colorTo: blue
sdk: docker
app_port: 7860
pinned: true
license: apache-2.0
short_description: Train a decoder-only LM from scratch on fineweb-edu
hf_oauth: true
hf_oauth_scopes:
  - read-repos
  - write-repos
  - manage-repos
hf_oauth_expiration_minutes: 480
startup_duration_timeout: 45m
datasets:
  - HuggingFaceFW/fineweb-edu
---

# 🧬 NanoDex

**Train your first decoder-only language model from scratch, in the cloud.**

Not a fine-tune. Not an adapter. Random weights, a real corpus, and a real loss
curve you watch fall in real time.

Pick a size (**492k / 1.06M / 5.17M / 8.06M** parameters), pick a token budget
(**200M – 1.5B** tokens of fineweb-edu), and queue it. A worker picks it up, you
watch the loss fall live, and when it's done the model is pushed to *your*
Hugging Face account with a generated model card.

## Pages

| Route | What it is |
|---|---|
| `/` | Landing page when signed out; your dashboard when signed in |
| `/train` | Four-step run wizard — size, tokens, name, review |
| `/models` | Everything you've trained: training, queued (with position), ready |
| `/models/{id}` | One run in detail — live loss chart, metrics, logs, publish, playground |
| `/queue` | Global queue and worker status |
| `/profile` | Account, totals, permissions |
| `/about` | The full recipe |

## Architecture

A standard modern decoder-only transformer (`LlamaForCausalLM`): SiLU MLP,
RMSNorm (ε=1e-5), rotary position embeddings (θ=10000), grouped-query attention,
tied input/output embeddings, no biases. The four tiers are that same recipe
scaled down in width and depth.

| Tier | Parameters | Layers | Hidden | Heads (KV) | FFN | Tokens / step |
|---|---:|---:|---:|---|---:|---:|
| NanoDex-500k | 492,192 | 3 | 96 | 6 (2) | 256 | 131,072 |
| NanoDex-1M | 1,062,272 | 5 | 128 | 8 (4) | 288 | 262,144 |
| NanoDex-5M | 5,172,384 | 9 | 224 | 8 (2) | 592 | 393,216 |
| NanoDex-8M | 8,060,256 | 9 | 288 | 9 (3) | 704 | 524,288 |

A typical small-LM vocabulary (~49k tokens) would be 28M embedding parameters on
its own — more than three times the biggest model here. So NanoDex ships its own
**2,048-token byte-level BPE** trained on fineweb-edu (~2.7 chars/token). The
parameter counts above are real totals, embeddings included.

## Data

[`HuggingFaceFW/fineweb-edu`](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu),
config `sample-10BT`. A 1.505B-token slice is tokenized **once** into a flat
`uint16` file (~2.8 GB) that every run reads from, so workers are never
data-bound. Each run starts at its own random offset. Building the cache is a
one-time cost per container; a run asking for more tokens than are cached yet
waits for the writer and reports it in the run log.

## Training

AdamW (β=0.9/0.95, wd=0.1, grad-clip 1.0) · 2% warmup then cosine decay to 10%
of peak · bf16 autocast · 512-token context · 128–256 sequences per forward
pass · 131k–524k tokens per optimizer step, with automatic micro-batch backoff
if CUDA runs out of memory. One worker thread per GPU claims the oldest queued run
atomically, so a 4-GPU Space trains four models at once.

## Layout

```
main.py                  FastAPI app — pages, JSON API, Hugging Face OAuth
Dockerfile
web/templates/           landing, home, train wizard, models, detail, queue,
                         profile, about, error
web/static/              app.css, app.js (dependency-free charts + polling)
nanodex/config.py        tier definitions + exact parameter counting
nanodex/data.py          shared fineweb-edu token cache
nanodex/trainer.py       the training loop
nanodex/worker.py        one worker per GPU
nanodex/hub.py           push a finished run to the user's account
nanodex/estimate.py      ETA, self-calibrating from completed runs
nanodex/db.py            SQLite job store
tokenizer/               the 2,048-token BPE
```

## A note on scale

1.5 billion tokens into an 8M-parameter model is ~9× past Chinchilla-optimal,
and into a 500k-parameter model over 150× past it — on purpose. Tiny models keep
improving long past the compute-optimal point, and the goal here isn't FLOP
efficiency — it's watching cross-entropy fall from 7.6
(uniform over 2,048 tokens) toward 4, and seeing a network that started as pure
noise begin to emit English words.

Nothing trained here is a useful assistant. That was never the point.

Built with 🤗 by [hugging-science](https://huggingface.co/hugging-science).