jaivial/disk-routed-ssd-chat-0.5b

Disk-Routed SSD Chat 0.5B

Experimental disk-routed recurrent language model designed to run inference on CPU + SSD only, with host RSS well under 1 GiB and no GPU. Model capacity lives in an int8 external memory table paged from SSD via O_DIRECT (2 rows/token); a small selective-recurrent controller stays resident in RAM.

Facts

Item	Value
Total weights	505,437,952
External (SSD) weights	499,968,000
Controller weights	5,469,952
Tokenizer	32,000 BPE (FineWeb-Edu)
Base training tokens	500,000,016 (FineWeb-Edu)
Alignment	20k UltraChat SFT + 500-step DPO

Measured inference speed (CPU only, no GPU)

Hardware: AMD Ryzen 9 5950X, NVMe SSD, int8 external table paged with O_DIRECT. Runtime is the C decoder direct_chat.c. "BPE tok/s" = generated 32k-BPE tokens per second.

Mode	Output speed	I/O per token	Process RSS	GPU
Cold, SSD-persisted recurrent state (disk-state)	774 BPE tok/s	16 KiB	23.0 MB	none
In-RAM recurrent state (state kept in RAM, weights on SSD)	2,916 BPE tok/s	8 KiB	23.3 MB	none
Chat eval worst-case sustained	1,172 BPE tok/s	8-16 KiB	<25 MB	none

Notes:

• Every configuration is >50x the 15 tok/s target and holds RSS far below the 1 GiB budget.
• Only 2 int8 rows (2 x 4 KiB pages) are read per token regardless of table size; optional 2 more pages persist recurrent state to SSD.
• Speed is bounded by per-token SSD page reads plus small controller compute, not by total model size.

Honest limitations

This is a systems proof, not a quality model. Chat evaluation semantic pass rate was ~0.40 with visible repetition and BPE spacing markers. 500M tokens and a ~5.5M-active controller are insufficient for reliable chat. It demonstrates that ~0.5B-weight capacity can be served from SSD at hundreds-to-thousands of tok/s under 25 MB RAM.

Files

• memory.i8 - int8 external table, page-aligned (499,970,048 bytes)
• controller.f32 - controller weights, CDR1 header
• scale.txt - int8 dequant scale
• manifest.json - config and byte counts
• fineweb-tokenizer.json - tokenizer

Runtime: a C O_DIRECT decoder (direct_chat.c) in the project repo.