jaivial/dna-diskchat-2b-v1

DNA-DiskChat-2B

A 2.24B-parameter disk-routed language model whose 2.15B-parameter memory table is stored natively compressed ("codons": 128 bytes/row, 75% smaller than int8) and trained from scratch on a single 16 GB RTX 5060 Ti. It runs inference on CPU + SSD only, reading a constant 256 bytes/token from disk.

Think of it as a zip file you never unzip: the memory stays compressed 4x and the model still learns and infers in that form. A small "librarian" (96.8M-param controller) fetches exactly 2 rows out of 4.19M per token from a 512 MB table that can live on SSD.

Model

Field	Value
Total params	2.24B (96.8M controller + 2.15B memory table)
d / layers / ff	512 / 16 / 1536
Product-key rows	2048 x 2048 = 4.19M
Codon storage	128 B/row (vs 512 B int8) -> 512 MB table on disk
Positional codebooks	4 reading-frames x 256 codewords x 4-dim
Tokenizer	32k BPE (FineWeb)
Per-token SSD I/O	256 bytes (2 routed rows)

Training (RTX 5060 Ti 16 GB, vast.ai)

• ~45,600 tok/s, GPU 100% util, 15.7/16.3 GB VRAM
• Stopped at the 6B-token budget -> step 419,346
• Final valid CE 3.5719 (valid ppl ~35.6); curve CE 9.83 -> 3.57
• Optimizer: fused AdamW + Muon (Newton-Schulz) + sparse shadow + codon mutation + WSD
• Cost: ~$3 total on a $0.13/hr instance

Inference - CPU vs SSD-only (10 runs each, 12-core CPU, no GPU)

Metric	CPU (table warm in RAM)	CPU + SSD (table on disk)
Throughput	48.95 tok/s (+/-0.42)	47.47 tok/s (+/-2.15)
C runtime (direct_dna.c)	-	~92 tok/s (O_DIRECT, 310 MB RSS)
Per-token I/O	cached in RAM	256 B real SSD
Table location	RAM (512 MB)	SSD (0 in RAM)
GPU	none	none

Inference is compute-bound, so putting the 512 MB table on SSD costs only ~3% throughput while reading a constant 256 B/token regardless of table size - 32x less per-token I/O than the earlier DiskChat-0.5B-v2 (8 KiB/token).

Instruction-tuned (SFT) checkpoint

An SFT pass on 200k UltraChat dialogues (assistant-only loss, chat format <s><|user|>...<\/s><|assistant|>...) gives a chat model at sft/sft.pt. It now follows the instruction format (a clear shift from the base continuation model):

• "What is the capital of France?" -> "The capital of France is ... Paris."
• "Give me one tip for learning to code." -> "...1. Learn a few basic programming languages: - Python ..."
• "Say hello." -> "Sure, I'm glad you're here."

Honest ceiling: it learns the format and answers on-topic, but content is often shallow or hallucinated (e.g. arithmetic, exact facts) - the expected limit of a ~97M active-parameter controller. See sft_quality.json. The roadmap to raise this at fixed inference cost is a ternary + PEER expert controller (grow effective capacity toward ~500M while holding active FLOPs and per-token bytes ~flat).

Ready-to-run C runtime (CPU + SSD, no GPU)

Prebuilt artifacts live in c-runtime/ - drop-in for the MONKE flash-drive runtime:

file	what
c-runtime/controller.bin	chat (SFT) controller, ~322 MB, loaded into RAM
c-runtime/controller_base.bin	base LM controller (text continuation)
c-runtime/codons.u8	512 MB codon memory table, stays on SSD (256 B/token)
c-runtime/tokenizer.json	32k ByteLevel-BPE
c-runtime/monke_runtime_dna.c	O_DIRECT streaming runtime source

huggingface-cli download jaivial/dna-diskchat-2b-v1 c-runtime/controller.bin \
  c-runtime/codons.u8 c-runtime/tokenizer.json c-runtime/monke_runtime_dna.c --local-dir .
gcc -O3 -march=native -fopenmp c-runtime/monke_runtime_dna.c -o monke_runtime_dna -lm
printf 'PROMPT 1 459 3411 284 3771 316\nGEN 24 316\n' | \
  ./monke_runtime_dna c-runtime/controller.bin c-runtime/codons.u8 1 8
# -> streams T <id> tokens then: E 86.29 8192 309.8   (tok_s, bytes/token, RSS MB)

Bit-exact with PyTorch; ~86-92 tok/s, ~310 MB RSS.

Base-model quality (honest)

Base LM (FineWeb pretraining only, no SFT). Greedy continuations are coherent, grammatical English and often factually correct, but repetitive - expected for a 2B base model at greedy decoding. It is a systems/efficiency artifact, not a chat assistant.

• "The capital of France is" -> "the French capital, Paris. It is the capital of France..."
• "The three primary colors are" -> "red, green, and blue."
• "Water is important because" -> "it is the most important source of water for all life on Earth..."
• "The sun is a star that" -> "is the source of all life on earth..."

Full samples in quality_samples.json; raw numbers in cpu_bench.json / ssd_bench.json. Latest checkpoint: checkpoints/ckpt_419346.pt.