File size: 1,495 Bytes
6bb8060 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | ---
license: apache-2.0
datasets:
- roneneldan/TinyStories
language:
- en
tags:
- 1-bit
- binary
- bitnet
- tinystories
- wasm
- on-device
- sprapp
pipeline_tag: text-generation
---
# tiny1bit — a 1-bit (binary) TinyStories model
A ~7.7M-parameter TinyStories generator with **1-bit binary weights** `{-1, +1}` (BitNet-style, per-tensor
absmean scale) on all projection layers, trained quantization-aware. Runs **in a browser tab, offline, no
GPU** via the Sprapp WASM engine (the 1-bit weights pack into the ternary 2-bit code, so the existing kernel
runs them directly).
## What it is
- Architecture: decoder-only transformer (dim 320, 6 layers, GQA, RoPE, SwiGLU, tied embeddings),
4096-vocab BPE tokenizer trained on TinyStories.
- 1-bit QAT (sign + absmean straight-through estimator) + knowledge distillation. Embeddings f16, lm_head
int8, norms fp.
- ~5.5 MB on disk (KNM1 v3).
## Sample
```
Once upon a time, there was a little → girl named Lily. She loved to play outside ...
```
Coherent for its size and bit-width; use temperature ≈ 0.65.
## Files
| file | what |
|---|---|
| `model_tiny1bit.knm` | 1-bit weights, KNM1 v3 (~5.5 MB) |
| `tokenizer.json` | 4096-vocab BPE tokenizer |
## Family
Part of **Sprapp** (offline on-device tiny LMs in the browser). See also
[eeny](https://huggingface.co/sprapp/eeny-tinystories-999k) (999K, int8, beats TinyStories-1M).
Trained on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).
|