rewrite the model card with how it works and how it was trained
Browse files
README.md
CHANGED
|
@@ -64,50 +64,101 @@ model-index:
|
|
| 64 |
</p>
|
| 65 |
|
| 66 |
<p align="center">
|
| 67 |
-
<img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="params:
|
| 68 |
</p>
|
| 69 |
|
| 70 |
-
<h3 align="center">
|
| 71 |
|
| 72 |
<p align="center">
|
| 73 |
-
<a href="https://huggingface.co/spaces/cfahlgren1/enron-autocomplete"><b>Try the demo →</b></a>
|
| 74 |
-
·
|
| 75 |
<a href="https://offlinepage.app"><b>Write with it in offlinepage.app →</b></a>
|
| 76 |
</p>
|
| 77 |
|
| 78 |
<br>
|
| 79 |
|
| 80 |
-
tinynext is the autocomplete in [offlinepage.app](https://offlinepage.app).
|
| 81 |
|
| 82 |
-
|
| 83 |
|
| 84 |
-
|
| 85 |
|
| 86 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
|
| 88 |
-
|
| 89 |
|
| 90 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 91 |
|---|---|---|---|
|
| 92 |
| Articles (Wikipedia) | 1.5% | **3.9%** | 8.1% |
|
| 93 |
| Messages (OpenAssistant) | 2.0% | **8.8%** | 16.8% |
|
| 94 |
| Email (Enron, unseen senders) | 3.0% | **4.0%** | 5.7% |
|
| 95 |
|
| 96 |
-
|
|
|
|
|
|
|
| 97 |
|---|---|---|---|---|
|
| 98 |
| Articles (Wikipedia) | 10% | 78% | 4% | 71% |
|
| 99 |
| Messages (OpenAssistant) | 19% | 75% | 9% | 70% |
|
| 100 |
| Email (Enron, unseen senders) | 11% | 78% | 4% | 51% |
|
| 101 |
|
| 102 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 103 |
|
| 104 |
| Run | Training tokens | Fill-in-the-middle | Typing | Editing | Val loss |
|
| 105 |
|---|---|---|---|---|---|
|
| 106 |
| **v2 (this one)** | **3B** | half the examples | **3.9 / 8.8 / 4.0** | **3.1 / 7.6 / 4.2** | **2.89** |
|
| 107 |
| First v2 | 1.5B | half the examples | 4.0 / 7.3 / 4.3 | 2.8 / 6.3 / 3.6 | 2.99 |
|
| 108 |
-
| [No
|
| 109 |
|
| 110 |
-
At 1.
|
| 111 |
|
| 112 |
<details>
|
| 113 |
<summary><b>Would a bigger model help?</b></summary>
|
|
@@ -123,27 +174,20 @@ At 1.5B tokens, fill-in-the-middle cost the model more than it gave: half its tr
|
|
| 123 |
| SmolLM2 | 135M | 8.1% | 16.8% | 5.7% |
|
| 124 |
| **tinynext v2** | **12.5M** | **3.9%** | **8.8%** | **4.0%** |
|
| 125 |
|
| 126 |
-
tinynext matches Pythia models
|
| 127 |
|
| 128 |
</details>
|
| 129 |
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
| | |
|
| 133 |
-
|---|---|
|
| 134 |
-
| Download | **13.3 MB**: int8 weights with one scale per 64 values, expanded to float32 when the page loads |
|
| 135 |
-
| Matches full precision | same next word 96% of the time, same 5-word path 90% |
|
| 136 |
-
| Speed | **5 ms** per suggestion (13 ms p95) in a Web Worker, Chrome on an M3 Max; typing stays under one frame |
|
| 137 |
-
|
| 138 |
-
The browser computes in float32 on purpose: onnxruntime-web's int8 kernels were 8–11× slower than its float kernels for this model, and int8 activations changed the top word too often. Only the download is compressed.
|
| 139 |
-
|
| 140 |
-
**Training.** 3B tokens (60% [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), 30% [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk), 10% Enron email from training senders), half of it as fill-in-the-middle examples. 56 minutes on one A100 at 920K tokens a second, from data tokenized once on a CPU job. Validation loss 2.89. [Training curves on Trackio](https://huggingface.co/spaces/cfahlgren1/tinynext-trackio).
|
| 141 |
|
| 142 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 143 |
|
| 144 |
-
|
| 145 |
-
- It still leans toward office email, and it only sees keyboard characters.
|
| 146 |
-
- It's English only.
|
| 147 |
|
| 148 |
<details>
|
| 149 |
<summary><b>v2 files</b></summary>
|
|
@@ -162,20 +206,14 @@ The browser computes in float32 on purpose: onnxruntime-web's int8 kernels were
|
|
| 162 |
|
| 163 |
</details>
|
| 164 |
|
| 165 |
-
|
| 166 |
-
<summary><b>Train it yourself</b></summary>
|
| 167 |
|
| 168 |
-
``
|
| 169 |
-
# Prepare data once on a CPU job, then train on a GPU from the prepared files.
|
| 170 |
-
hf jobs uv run --flavor cpu-xl --secrets HF_TOKEN train.py --stage prepare --tokens 3e9 --fineweb-files 4
|
| 171 |
-
hf jobs uv run --flavor a100-large --secrets HF_TOKEN train.py --stage train --tokens 3e9 --fineweb-files 4
|
| 172 |
-
```
|
| 173 |
|
| 174 |
-
<
|
| 175 |
-
|
| 176 |
-
## v1: the original character model
|
| 177 |
|
| 178 |
-
|
| 179 |
|
| 180 |
It finishes the word or short phrase you're typing. Press <kbd>Tab</kbd> to take the suggestion.
|
| 181 |
|
|
@@ -186,7 +224,7 @@ It finishes the word or short phrase you're typing. Press <kbd>Tab</kbd> to take
|
|
| 186 |
| The books are fine. Please call me i | **f you have any** |
|
| 187 |
| Enron N | **orth America Corp.** |
|
| 188 |
|
| 189 |
-
|
| 190 |
|
| 191 |
The model knows how sure it is of each guess. You pick a cutoff, and it only suggests when it's at least that sure. A higher cutoff means fewer suggestions, and more of them right.
|
| 192 |
|
|
@@ -202,21 +240,21 @@ The model knows how sure it is of each guess. You pick a cutoff, and it only sug
|
|
| 202 |
|---|---|---|
|
| 203 |
| 20% | 30% of the time | 48% |
|
| 204 |
| 35% | 18% of the time | 66% |
|
| 205 |
-
| **50%**
|
| 206 |
| 70% | 7% of the time | 90% |
|
| 207 |
|
| 208 |
-
The demo uses 40%
|
| 209 |
|
| 210 |
These numbers come from 5,000 places in emails from people the model never saw in training. “Right” means the suggestion exactly matched what the person typed next.
|
| 211 |
|
| 212 |
-
|
| 213 |
|
| 214 |
- It only knows office email from 2001. Outside that it has little to say.
|
| 215 |
- With no cutoff it rambles. “Dear Santa,” becomes “I am attaching a copy of the agreement to the Enron Corp.”
|
| 216 |
- It learned real names and sign-offs from the emails, so keep suggestions short.
|
| 217 |
- English only.
|
| 218 |
|
| 219 |
-
|
| 220 |
|
| 221 |
The model takes one character and its memory of the text so far. It returns its guess for the next character and an updated memory. So each key press is one small step.
|
| 222 |
|
|
@@ -264,7 +302,7 @@ I also tried a bigger version with 3.46M parameters and 3.5 MB. It wasn't better
|
|
| 264 |
|
| 265 |
</details>
|
| 266 |
|
| 267 |
-
|
| 268 |
|
| 269 |
It read 40 million characters from 119,859 emails sent from enron.com addresses, 30 times over. Typos were added to what it read, so it keeps predicting when you make one. The run took 14 minutes on one A10G with Hugging Face Jobs and cost 35 cents.
|
| 270 |
|
|
@@ -274,6 +312,9 @@ It read 40 million characters from 119,859 emails sent from enron.com addresses,
|
|
| 274 |
hf jobs uv run --flavor a10g-large --secrets HF_TOKEN train.py --repo your-name/your-model
|
| 275 |
```
|
| 276 |
|
|
|
|
|
|
|
|
|
|
| 277 |
<br>
|
| 278 |
|
| 279 |
<p align="center">
|
|
|
|
| 64 |
</p>
|
| 65 |
|
| 66 |
<p align="center">
|
| 67 |
+
<img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="params: 12.5M" src="https://img.shields.io/badge/params-12.5M-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="size: 13.3 MB" src="https://img.shields.io/badge/size-13.3%20MB-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="speed: ~5 ms" src="https://img.shields.io/badge/speed-~5%20ms-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="runs: in browser" src="https://img.shields.io/badge/runs-in%20browser-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="data sent: none" src="https://img.shields.io/badge/data%20sent-none-6e6e73?style=flat-square&labelColor=1d1d1f">
|
| 68 |
</p>
|
| 69 |
|
| 70 |
+
<h3 align="center">Gray text that finishes your sentence.<br>A 13 MB model that runs in a browser tab.</h3>
|
| 71 |
|
| 72 |
<p align="center">
|
|
|
|
|
|
|
| 73 |
<a href="https://offlinepage.app"><b>Write with it in offlinepage.app →</b></a>
|
| 74 |
</p>
|
| 75 |
|
| 76 |
<br>
|
| 77 |
|
| 78 |
+
tinynext is the autocomplete in [offlinepage.app](https://offlinepage.app). It reads the text around the cursor and, when it's sure enough, shows the next few words in gray. Press <kbd>Tab</kbd> to take them. It runs in a Web Worker with [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/), so nothing you type is sent anywhere.
|
| 79 |
|
| 80 |
+
The model the app uses is in [`v2/`](https://huggingface.co/paperweight/tinynext/tree/main/v2). The files at the root of this repo are v1, an earlier and much smaller character model, described [at the end](#v1-the-original-character-model).
|
| 81 |
|
| 82 |
+
## What it does
|
| 83 |
|
| 84 |
+
<p align="center">
|
| 85 |
+
<img src="what-it-does.png" alt="You type: Please accept my resignation. I will be o. You get: the same text with “ut of the office” offered in gray, and Tab to take it." width="880">
|
| 86 |
+
</p>
|
| 87 |
+
|
| 88 |
+
| You type | It suggests |
|
| 89 |
+
|---|---|
|
| 90 |
+
| Please accept my resignation. I will be o | **ut of the office** |
|
| 91 |
+
| See you on the ground. I look f | **orward to** |
|
| 92 |
+
|
| 93 |
+
It stays quiet most of the time. A suggestion only appears when the model is at least 40% sure of the whole phrase, and it has to be four characters or longer.
|
| 94 |
|
| 95 |
+
## How well it works
|
| 96 |
|
| 97 |
+
Every model here is scored the same way by [`v2/eval.py`](v2/eval.py): type held-out text character by character, pause every 12 characters, and count a suggestion only if the text really continues that way. These numbers use a 50% cutoff; the app now uses 40%, so it suggests a little more often and is right a little less often.
|
| 98 |
+
|
| 99 |
+
**How much typing it saves:**
|
| 100 |
+
|
| 101 |
+
| Writing | v1 | **v2** | SmolLM2-135M (about 100 MB) |
|
| 102 |
|---|---|---|---|
|
| 103 |
| Articles (Wikipedia) | 1.5% | **3.9%** | 8.1% |
|
| 104 |
| Messages (OpenAssistant) | 2.0% | **8.8%** | 16.8% |
|
| 105 |
| Email (Enron, unseen senders) | 3.0% | **4.0%** | 5.7% |
|
| 106 |
|
| 107 |
+
**How often it speaks up, and how often it's right:**
|
| 108 |
+
|
| 109 |
+
| Writing | Suggests at | Right | 2+ words suggested at | 2+ words right |
|
| 110 |
|---|---|---|---|---|
|
| 111 |
| Articles (Wikipedia) | 10% | 78% | 4% | 71% |
|
| 112 |
| Messages (OpenAssistant) | 19% | 75% | 9% | 70% |
|
| 113 |
| Email (Enron, unseen senders) | 11% | 78% | 4% | 51% |
|
| 114 |
|
| 115 |
+
“Suggests at” is the share of pauses where it offered anything. “Right” means the suggestion exactly matched what the person typed next.
|
| 116 |
+
|
| 117 |
+
### Limits
|
| 118 |
+
|
| 119 |
+
- offlinepage.app only suggests at the end of a paragraph. Mid-line suggestions, where reading the text after the cursor should help most, aren't built yet.
|
| 120 |
+
- It still leans toward office email, and it only sees keyboard characters.
|
| 121 |
+
- English only.
|
| 122 |
+
|
| 123 |
+
## In the browser
|
| 124 |
+
|
| 125 |
+
| Setting | Value |
|
| 126 |
+
|---|---|
|
| 127 |
+
| Download | **13.3 MB**: 8-bit weights with one scale per 64 values, expanded to float32 when the page loads |
|
| 128 |
+
| Matches full precision | Same next word 96% of the time, same 5-word path 90% |
|
| 129 |
+
| Speed | **5 ms** per suggestion (13 ms at the slow end) in a Web Worker, Chrome on an M3 Max |
|
| 130 |
+
|
| 131 |
+
The browser computes in float32 on purpose. ONNX Runtime Web's 8-bit kernels were 8 to 11 times slower than its float kernels for this model, and 8-bit activations changed the top word too often. Only the download is compressed.
|
| 132 |
+
|
| 133 |
+
## How it was trained
|
| 134 |
+
|
| 135 |
+
**The task.** Predict the next word piece. Half of the training examples are fill-in-the-middle: the model also sees the text after a gap, so it can suggest in the middle of a paragraph as well as at the end.
|
| 136 |
+
|
| 137 |
+
**The data.** 3 billion tokens of writing like what people type: 60% [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) articles, 30% [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) conversations and 10% [Enron email](https://huggingface.co/datasets/corbt/enron-emails), with the emails split by sender so the test senders are never seen.
|
| 138 |
+
|
| 139 |
+
**The model.** A small transformer that reads up to about 1,500 characters before the cursor.
|
| 140 |
+
|
| 141 |
+
| Setting | Value |
|
| 142 |
+
|---|---|
|
| 143 |
+
| Parameters | 12.5 million |
|
| 144 |
+
| Layers / width / heads | 8 / 320 / 5 |
|
| 145 |
+
| Tokenizer | 8,192 byte-level BPE pieces, so one step can be a whole word |
|
| 146 |
+
| Context | 512 tokens |
|
| 147 |
+
| Training | One pass over 3 billion tokens, batch 512, learning rate 0.003 |
|
| 148 |
+
| Time | 56 minutes on one A100 at 920,000 tokens a second, with [Hugging Face Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs) |
|
| 149 |
+
| Validation loss | 2.89 |
|
| 150 |
+
|
| 151 |
+
The data is tokenized once on a CPU job, and the GPU job trains from the prepared files. [Training curves are on Trackio](https://huggingface.co/spaces/cfahlgren1/tinynext-trackio).
|
| 152 |
+
|
| 153 |
+
**What the runs showed.** Three runs of the same model, scored as typing saved on articles / messages / email. *Editing* is the same test with text after the cursor.
|
| 154 |
|
| 155 |
| Run | Training tokens | Fill-in-the-middle | Typing | Editing | Val loss |
|
| 156 |
|---|---|---|---|---|---|
|
| 157 |
| **v2 (this one)** | **3B** | half the examples | **3.9 / 8.8 / 4.0** | **3.1 / 7.6 / 4.2** | **2.89** |
|
| 158 |
| First v2 | 1.5B | half the examples | 4.0 / 7.3 / 4.3 | 2.8 / 6.3 / 3.6 | 2.99 |
|
| 159 |
+
| [No fill-in-the-middle](https://huggingface.co/paperweight/tinynext/tree/main/v2/experiments/tinynext-v2-nofim) | 1.5B | none | 3.9 / 8.8 / 4.2 | 3.0 / 6.9 / 4.1 | 2.94 |
|
| 160 |
|
| 161 |
+
At 1.5 billion tokens, fill-in-the-middle cost more than it gave: half the training went to a task the model rarely uses. With twice the tokens it stops costing anything on typing and helps when editing.
|
| 162 |
|
| 163 |
<details>
|
| 164 |
<summary><b>Would a bigger model help?</b></summary>
|
|
|
|
| 174 |
| SmolLM2 | 135M | 8.1% | 16.8% | 5.7% |
|
| 175 |
| **tinynext v2** | **12.5M** | **3.9%** | **8.8%** | **4.0%** |
|
| 176 |
|
| 177 |
+
tinynext matches Pythia models two to three times its size because it trained on writing like what people type. Size starts to pay off between 30M and 70M parameters, and better data (SmolLM2) matters as much as size.
|
| 178 |
|
| 179 |
</details>
|
| 180 |
|
| 181 |
+
<details>
|
| 182 |
+
<summary><b>Train it yourself</b></summary>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 183 |
|
| 184 |
+
```bash
|
| 185 |
+
# Prepare data once on a CPU job, then train on a GPU from the prepared files.
|
| 186 |
+
hf jobs uv run --flavor cpu-xl --secrets HF_TOKEN train.py --stage prepare --tokens 3e9 --fineweb-files 4
|
| 187 |
+
hf jobs uv run --flavor a100-large --secrets HF_TOKEN train.py --stage train --tokens 3e9 --fineweb-files 4
|
| 188 |
+
```
|
| 189 |
|
| 190 |
+
</details>
|
|
|
|
|
|
|
| 191 |
|
| 192 |
<details>
|
| 193 |
<summary><b>v2 files</b></summary>
|
|
|
|
| 206 |
|
| 207 |
</details>
|
| 208 |
|
| 209 |
+
## v1: the original character model
|
|
|
|
| 210 |
|
| 211 |
+
The files at the root of this repo. v1 guesses one letter at a time, so its suggestions are short word endings. It has 944,000 parameters, is 0.98 MB, and learned only from Enron's emails. It is the same model as [`cfahlgren1/enron-autocomplete`](https://huggingface.co/cfahlgren1/enron-autocomplete), and it has [its own demo](https://huggingface.co/spaces/cfahlgren1/enron-autocomplete).
|
|
|
|
|
|
|
|
|
|
|
|
|
| 212 |
|
| 213 |
+
<details>
|
| 214 |
+
<summary><b>Everything about v1</b></summary>
|
|
|
|
| 215 |
|
| 216 |
+
**What it does**
|
| 217 |
|
| 218 |
It finishes the word or short phrase you're typing. Press <kbd>Tab</kbd> to take the suggestion.
|
| 219 |
|
|
|
|
| 224 |
| The books are fine. Please call me i | **f you have any** |
|
| 225 |
| Enron N | **orth America Corp.** |
|
| 226 |
|
| 227 |
+
**How sure it has to be**
|
| 228 |
|
| 229 |
The model knows how sure it is of each guess. You pick a cutoff, and it only suggests when it's at least that sure. A higher cutoff means fewer suggestions, and more of them right.
|
| 230 |
|
|
|
|
| 240 |
|---|---|---|
|
| 241 |
| 20% | 30% of the time | 48% |
|
| 242 |
| 35% | 18% of the time | 66% |
|
| 243 |
+
| **50%** | **11% of the time** | **80%** |
|
| 244 |
| 70% | 7% of the time | 90% |
|
| 245 |
|
| 246 |
+
The demo uses 40%.
|
| 247 |
|
| 248 |
These numbers come from 5,000 places in emails from people the model never saw in training. “Right” means the suggestion exactly matched what the person typed next.
|
| 249 |
|
| 250 |
+
**Limits**
|
| 251 |
|
| 252 |
- It only knows office email from 2001. Outside that it has little to say.
|
| 253 |
- With no cutoff it rambles. “Dear Santa,” becomes “I am attaching a copy of the agreement to the Enron Corp.”
|
| 254 |
- It learned real names and sign-offs from the emails, so keep suggestions short.
|
| 255 |
- English only.
|
| 256 |
|
| 257 |
+
**Use it**
|
| 258 |
|
| 259 |
The model takes one character and its memory of the text so far. It returns its guess for the next character and an updated memory. So each key press is one small step.
|
| 260 |
|
|
|
|
| 302 |
|
| 303 |
</details>
|
| 304 |
|
| 305 |
+
**Training**
|
| 306 |
|
| 307 |
It read 40 million characters from 119,859 emails sent from enron.com addresses, 30 times over. Typos were added to what it read, so it keeps predicting when you make one. The run took 14 minutes on one A10G with Hugging Face Jobs and cost 35 cents.
|
| 308 |
|
|
|
|
| 312 |
hf jobs uv run --flavor a10g-large --secrets HF_TOKEN train.py --repo your-name/your-model
|
| 313 |
```
|
| 314 |
|
| 315 |
+
|
| 316 |
+
</details>
|
| 317 |
+
|
| 318 |
<br>
|
| 319 |
|
| 320 |
<p align="center">
|