tiny1bit — a 1-bit (binary) TinyStories model
A ~7.7M-parameter TinyStories generator with 1-bit binary weights {-1, +1} (BitNet-style, per-tensor
absmean scale) on all projection layers, trained quantization-aware. Runs in a browser tab, offline, no
GPU via the Sprapp WASM engine (the 1-bit weights pack into the ternary 2-bit code, so the existing kernel
runs them directly).
What it is
- Architecture: decoder-only transformer (dim 320, 6 layers, GQA, RoPE, SwiGLU, tied embeddings), 4096-vocab BPE tokenizer trained on TinyStories.
- 1-bit QAT (sign + absmean straight-through estimator) + knowledge distillation. Embeddings f16, lm_head int8, norms fp.
- ~5.5 MB on disk (KNM1 v3).
Sample
Once upon a time, there was a little → girl named Lily. She loved to play outside ...
Coherent for its size and bit-width; use temperature ≈ 0.65.
Files
| file | what |
|---|---|
model_tiny1bit.knm |
1-bit weights, KNM1 v3 (~5.5 MB) |
tokenizer.json |
4096-vocab BPE tokenizer |
Family
Part of Sprapp (offline on-device tiny LMs in the browser). See also eeny (999K, int8, beats TinyStories-1M). Trained on TinyStories.