---
license: apache-2.0
datasets:
  - roneneldan/TinyStories
language:
  - en
tags:
  - 1-bit
  - binary
  - bitnet
  - tinystories
  - wasm
  - on-device
  - sprapp
pipeline_tag: text-generation
---

# tiny1bit — a 1-bit (binary) TinyStories model

A ~7.7M-parameter TinyStories generator with **1-bit binary weights** `{-1, +1}` (BitNet-style, per-tensor
absmean scale) on all projection layers, trained quantization-aware. Runs **in a browser tab, offline, no
GPU** via the Sprapp WASM engine (the 1-bit weights pack into the ternary 2-bit code, so the existing kernel
runs them directly).

## What it is

- Architecture: decoder-only transformer (dim 320, 6 layers, GQA, RoPE, SwiGLU, tied embeddings),
  4096-vocab BPE tokenizer trained on TinyStories.
- 1-bit QAT (sign + absmean straight-through estimator) + knowledge distillation. Embeddings f16, lm_head
  int8, norms fp.
- ~5.5 MB on disk (KNM1 v3).

## Sample

```
Once upon a time, there was a little  →  girl named Lily. She loved to play outside ...
```

Coherent for its size and bit-width; use temperature ≈ 0.65.

## Files

| file | what |
|---|---|
| `model_tiny1bit.knm` | 1-bit weights, KNM1 v3 (~5.5 MB) |
| `tokenizer.json` | 4096-vocab BPE tokenizer |

## Family

Part of **Sprapp** (offline on-device tiny LMs in the browser). See also
[eeny](https://huggingface.co/sprapp/eeny-tinystories-999k) (999K, int8, beats TinyStories-1M).
Trained on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).