| --- |
| license: apache-2.0 |
| datasets: |
| - roneneldan/TinyStories |
| language: |
| - en |
| tags: |
| - 1-bit |
| - binary |
| - bitnet |
| - tinystories |
| - wasm |
| - on-device |
| - sprapp |
| pipeline_tag: text-generation |
| --- |
| |
| # tiny1bit — a 1-bit (binary) TinyStories model |
|
|
| A ~7.7M-parameter TinyStories generator with **1-bit binary weights** `{-1, +1}` (BitNet-style, per-tensor |
| absmean scale) on all projection layers, trained quantization-aware. Runs **in a browser tab, offline, no |
| GPU** via the Sprapp WASM engine (the 1-bit weights pack into the ternary 2-bit code, so the existing kernel |
| runs them directly). |
|
|
| ## What it is |
|
|
| - Architecture: decoder-only transformer (dim 320, 6 layers, GQA, RoPE, SwiGLU, tied embeddings), |
| 4096-vocab BPE tokenizer trained on TinyStories. |
| - 1-bit QAT (sign + absmean straight-through estimator) + knowledge distillation. Embeddings f16, lm_head |
| int8, norms fp. |
| - ~5.5 MB on disk (KNM1 v3). |
| |
| ## Sample |
| |
| ``` |
| Once upon a time, there was a little → girl named Lily. She loved to play outside ... |
| ``` |
| |
| Coherent for its size and bit-width; use temperature ≈ 0.65. |
| |
| ## Files |
| |
| | file | what | |
| |---|---| |
| | `model_tiny1bit.knm` | 1-bit weights, KNM1 v3 (~5.5 MB) | |
| | `tokenizer.json` | 4096-vocab BPE tokenizer | |
|
|
| ## Family |
|
|
| Part of **Sprapp** (offline on-device tiny LMs in the browser). See also |
| [eeny](https://huggingface.co/sprapp/eeny-tinystories-999k) (999K, int8, beats TinyStories-1M). |
| Trained on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). |
|
|
