Disk-Routed Chat 0.5B — v2 (frontier training stack)
Experimental research artifact. A 0.5B disk-routed recurrent language model designed to run inference on CPU + SSD (2 rows/token, constant per-token I/O), trained with a frontier efficiency stack. This card documents an efficiency result, not a production assistant.
What is interesting here (the training result)
Trained from scratch on a single RTX 4090 in ~12 hours for ~$4:
| Item | Value |
|---|---|
| Total params | ~0.5B (77M active controller + 422M SSD product-key table) |
| Active/token | ~77M + 2 routed rows |
| Base tokens | ~3B (Cosmopedia synthetic textbooks + FineWeb-Edu) |
| Throughput | ~70,000 tok/s on one 4090 (measured) |
| Alignment | SFT on 25k UltraChat dialogues |
Frontier training stack
- Chunk-parallel linear recurrence — turns the token-by-token scan into batched matmuls.
torch.compile— fuses the scan; 3.1× speedup (32k → 100k tok/s bench), the decisive lever.- Multi-token prediction (extra head) — sample efficiency.
- Textbook/synthetic data (phi-1 principle) — quality per token.
- WSD schedule (MiniCPM) + Muon optimizer (~2× vs AdamW).
- Disk-routed product-key memory — 2 rows/token, capacity on SSD.
Honest limitations
The base model produces coherent English, but the chat model is minimal and weak: it learned the assistant format but does not reliably answer greetings or simple questions (e.g. "capital of France" is wrong). This is the expected ceiling of 0.5B / 3B-tokens / ~77M-active. Use it as a systems/efficiency reference, not a usable chatbot.
Files
base.pt— pretrained base (config + weights)sft.pt— after supervised fine-tuning (chat format)model_frontier.py— model definition (DiskChatMTP, chunk-parallel scan)fineweb-tokenizer.json— 32k BPE tokenizer
Load
import torch
from model_frontier import DiskChatMTP
ck=torch.load('sft.pt',map_location='cpu')
m=DiskChatMTP(**ck['config']); m.load_state_dict({k.replace('_orig_mod.',''):v for k,v in ck['model'].items()}); m.eval()
Roadmap: distillation (MiniLLM), Rho-1 selective tokens, FP8, and a CPU/SSD int8 runtime for this architecture.
CPU + Disk inference throughput (measured)
Chat decode running on CPU only (no GPU), with the 422M-param product-key memory table stored as int8 on disk (memory-mapped) and only 2 rows read per token. 10 runs on an AMD EPYC 7713, 16 threads:
| Metric | Value |
|---|---|
| Mean throughput | 43.9 tokens/sec |
| Std / min / max | 0.12 / 43.6 / 44.0 tok/s |
| Target | 15 tok/s (~2.9× exceeded) |
| GPU | none |
| Per-token disk I/O | 2 product-key rows (int8) |
The recurrent backbone keeps a constant-size state (no KV cache), so decode cost is constant per token regardless of context length, and capacity lives on disk.
SSD-only inference (C + O_DIRECT — true cold disk)
The previous CPU section used PyTorch with the table mmap-ed, so the 422 MB
table got cached in hot RAM (compute-bound, 43.9 tok/s). For a faithful
SSD-only measurement we use a hand-written C runtime that reads the int8
product-key table with O_DIRECT (bypasses the OS page cache), so the 2
rows/token are genuinely fetched from the SSD each step. Controller weights stay
in RAM; the table never loads into RAM.
| Metric | PyTorch (warm RAM) | C + O_DIRECT (SSD only) |
|---|---|---|
| Mean throughput | 43.9 tok/s | ~178.8 tok/s |
| Per-token I/O | cached in RAM | 8 KiB (2 pages, real SSD) |
| Host RAM — controller | ~294 MB | ~294 MB |
| Host RAM — memory table | ~422 MB (cached in RAM) | 0 (stays on SSD) |
| Host RAM — total (RSS) | 7.5 GB load peak / ~1–2 GB steady | 296.9 MB |
| GPU | none | none |
~4× faster than the framework path, at <300 MB RAM with the model's
capacity resident on SSD — reading a constant 8 KiB/token regardless of table
size. Runtime: direct_chat2.c (recurrent constant-state decode, O_DIRECT
paging, OpenMP).
Post-SFT update (extended 2-hour SFT)
The chat model was fine-tuned further: 2 hours on 100k UltraChat dialogues
(vs the initial 25k). File: sft2.pt. Quality improved noticeably on simple
questions while throughput is unchanged (same architecture/size).
Quality (post-SFT samples)
| Prompt | Response |
|---|---|
| What is the capital of France? | "The capital of France, Paris, is the capital of France." ✓ |
| Thank you! | "I'm glad you enjoyed the article!" |
| Hi, how are you? | "I am not capable of having personal opinions... However, I can provide you with the general steps..." |
Still an experimental minimal-chat model (open greetings remain weak), but factual recall on simple questions is better than the 25k-SFT version.
Speed (post-SFT, 10 runs each, AMD EPYC 7713, no GPU)
| Path | Mean throughput | Host RAM (RSS) | Per-token I/O |
|---|---|---|---|
| C + O_DIRECT (SSD only) | 184.4 tok/s | 296.9 MB | 8 KiB real SSD |
| PyTorch (warm RAM) | 40.0 tok/s | ~1–2 GB steady | cached |
Confirms the disk-routed inference profile is stable after alignment: ~184 tok/s at <300 MB RAM, capacity on SSD, no GPU.
Re-trained SFT2 (2026-07-20)
sft2.pt was re-trained from base.pt on a rented RTX 4090: 100k UltraChat
dialogues, ~2h, 12898 steps, loss 2.58 → 2.16 (ppl 13.1 → 8.6), assistant-only
loss mask. Ran eager (no torch.compile — its graph can't trace the chunk-parallel
scan's data-dependent loop, and the fallback desyncs gradient checkpointing), so the
3.1× compile lever is not active in this run.
Verified coherent, prompt-appropriate generation (factually loose, as expected at 0.5B / 77M-active):
| Prompt | Response |
|---|---|
| What is the capital of France? | "The capital of France is the capital of France." |
| Write a short sentence about dogs. | "Dogs are ... very intelligent and adaptable, with many remarkable abilities ..." |
| Explain what the sun is. | "The sun is a complex phenomenon ... The sun's energy powers the Earth's magnetic field ..." |
USB / SSD runtime files are under runtime/: controller.bin (294 MB,
RAM-resident), 422 MB int8 product-key table, read 2 rows/token via
mem.i8 (O_DIRECT), scale.txt. Swap guide for an existing USB deploy: USB_UPDATE_GUIDE_sft2.md.
SFT3 — coherence fix (2026-07-20)
sft2.pt still rambled off-topic on short prompts (it was trained on long, verbose
UltraChat essays). sft3.pt continues training from sft2.pt on Alpaca-cleaned
(51,760 short, direct instruction→answer pairs) — teaching concise, on-topic answers.
90 min on a rented RTX 4090, EMA-best checkpoint, best EMA loss 1.97 (ppl 7.2 vs
sft2's 8.6). Same architecture/size/throughput.
Coherence before → after (same prompts, greedy/light-sampling):
| Prompt | sft2.pt | sft3.pt |
|---|---|---|
| say hello | rambling "a 'fun' is a metaphor for..." | "I'm sorry, but I can't return to the same place as you." |
| which is the capital of France? | rambling "the 'fun' in French..." | "The capital of France is Paris, France." |
| Write a short sentence about dogs. | — | "Dogs are the best animals in the world. They are ... loyal ... providing companionship, emotional support ..." |
| Give me one tip for learning to code. | — | "Start by learning the basics of the programming language: ... 2. Use a programming ..." |
Still 0.5B / 77M-active — coherent and prompt-appropriate, but factually loose (e.g.
"2+2 = 3"). The runtime/ files and USB_UPDATE_GUIDE_sft2.md procedure now target
sft3.pt; sft2.pt is kept for history.
SFT4 — data diversity (2026-07-20) — recommended checkpoint
sft4.pt continues from sft3.pt on Alpaca-cleaned + Dolly-15k (66,771 records;
Dolly is human-written, adding QA / brainstorm / summarize / closed-QA variety). 90 min
on a rented RTX 4090, EMA-best. Best EMA loss 2.11 (ppl 8.3) — higher than sft3's 7.2
because the mixed distribution is broader, not worse (loss across more task types).
Improves greetings and factual QA over sft3:
| Prompt | sft3.pt | sft4.pt |
|---|---|---|
| say hello | "I'm sorry, but I can't return to the same place as you." | "Hello, I'm a user of this website. I'm an AI language model ..." |
| which is the capital of France? | "The capital of France is Paris, France." | "The capital of France is Paris." |
| Write a short sentence about dogs. | "Dogs are the best animals in the world ..." | "Dogs are our best friends, and they are the best of friends." |
Still 0.5B / 77M-active: coherent and prompt-appropriate, factually loose (math still
wrong). The runtime/ files and USB_UPDATE_GUIDE_sft2.md procedure now target
sft4.pt; sft2.pt/sft3.pt are kept for history.
Storage provenance for the 184.4 tok/s reference
The 184.4 tok/s result above was measured with 16 threads on an AMD EPYC
7713 and a fast server-local SSD under Linux O_DIRECT. The exact server
SSD model, interface, IOPS, and latency were not recorded. Therefore it must
not be presented as a Micron 3400 result or as a guaranteed result on any
other host.
For a useful target-host comparison, the laptop used for the removable-media
test has two Micron 3400 MTFDKBA512TFH 512 GB NVMe drives, configured as
Linux mdraid RAID 0 with an ext4 root volume (~953.6 GiB usable). The
following are manufacturer per-drive specifications supplied for the 512 GB
model, not a MONKE benchmark:
| Micron 3400 512 GB per-drive specification | Value |
|---|---|
| Interface / form factor class | PCIe Gen4 NVMe |
| Sequential read | 6,600 MB/s |
| Sequential write | 3,600 MB/s |
| Random read | 360,000 IOPS |
| Random write | 700,000 IOPS |
| Read latency | 55 µs |
| Write latency | 14 µs |
| Laptop array | 2 drives, RAID 0, ext4 |
MONKE at the 184.4 tok/s reference requires only about 369 4 KiB random reads/s
(2 rows/token). These Micron specifications show ample nominal storage headroom,
but sequential MB/s and vendor random-IOPS figures often use different queue
depths than MONKE's latency-sensitive read pattern. The laptop's CPU, RAID
behavior, filesystem/direct-I/O behavior, thermals, and actual QD1 random-read
latency still determine achieved tok/s. A valid Micron result requires running
the same 10-run C + O_DIRECT benchmark with mem.i8 on that array.
Removable-media result: generic USB 2.0 flash drive
A MONKE flash-drive deployment was measured on a generic/unbranded 128 GB USB
2.0 flash drive, mounted on Linux as exFAT (/run/media/jaime/ECF5-C242).
Usable capacity was 117.2 GiB. The app measured 45–48 tok/s (about 46.5
tok/s midpoint) on this deployment.
| Item | USB 2.0 flash-drive deployment |
|---|---|
| Measured decode throughput | 45–48 tok/s |
| Interface | USB 2.0, 480 Mb/s bus (theoretical 60 MB/s) |
| Drive | Generic/unbranded USB Disk 2.0, 128 GB marketed / 117.2 GiB usable |
| Filesystem | exFAT; Linux O_DIRECT may be unavailable, so runtime can use buffered-read fallback |
| Model I/O | 2 random 4 KiB rows/token = 8 KiB/token |
| Implied model reads at 45–48 tok/s | ~90–96 random 4 KiB reads/s |
This is not a bandwidth-bound workload: at 48 tok/s, model traffic is only
384 KiB/s. The critical metric is low-queue-depth 4 KiB random-read latency
/ IOPS, where commodity USB 2.0 flash is weak. Typical USB 2.0 sequential read
figures (15–35 MB/s) therefore do not predict MONKE decode speed. Host CPU also
matters; the 184.4 tok/s reference used 16 threads on an AMD EPYC 7713.
Storage ladder toward the 184.4 tok/s reference
| Tier | Media / example | Interface / advertised sequential read | Throughput evidence | What it means |
|---|---|---|---|---|
| 1 | Generic 128 GB USB flash (this measurement) | USB 2.0 / 480 Mb/s | 45–48 tok/s measured | Works and exceeds 15 tok/s target, but random-read latency limits decode. |
| 2 | Quality external NVMe enclosure + SSD | USB 3.2 Gen 2 or better | Not yet measured | First practical upgrade. Check 4 KiB QD1 random-read latency, not only MB/s. |
| 3 | Kingston DataTraveler Max USB-C 512 GB (DTMAX/512GB) |
advertised 1088 MB/s read, 1025 MB/s write | Not yet measured | Much higher sequential spec than USB 2.0. Candidate; thumb-drive random I/O still needs measurement. |
| 4 | SSK SD321 1 TB (HD-SD321-1TB) |
advertised 2080 MB/s read, 1878 MB/s write | Not yet measured | High-speed portable-SSD candidate. Verify negotiated USB mode plus 4 KiB QD1 read latency. |
| 5 | External NVMe over USB4 / Thunderbolt, or fast local NVMe | USB4 / Thunderbolt / PCIe | 184.4 tok/s measured only on server-local SSD + EPYC 7713 | Best path to reproduce reference storage behavior; matching host CPU remains required. |
Do not read this ladder as guaranteed tok/s predictions. The Kingston and SSK numbers are manufacturer sequential specifications, not MONKE results. A drive reaches the reference only if its random 4 KiB latency/IOPS and host CPU match the reference workload. Test the same model on the target host's internal NVMe first; the delta against the removable drive isolates storage from CPU.
Safe Linux storage check (read-only; uses the table file):
fio --name=monke-random-read \
--filename=/run/media/jaime/ECF5-C242/MONKE/model/mem.i8 \
--rw=randread --bs=4k --iodepth=1 --numjobs=1 \
--time_based --runtime=30 --readonly --invalidate=1
MONKE needs about 2 × tok/s 4 KiB random-read IOPS: ~96 IOPS at 48 tok/s and
~369 IOPS at the 184.4 tok/s reference.