cfahlgren1 HF Staff commited on
Commit
698642e
·
verified ·
1 Parent(s): 735b3dc

rewrite the model card with how it works and how it was trained

Browse files
Files changed (1) hide show
  1. README.md +88 -47
README.md CHANGED
@@ -64,50 +64,101 @@ model-index:
64
  </p>
65
 
66
  <p align="center">
67
- <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="params: 944K" src="https://img.shields.io/badge/params-944K-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="size: 0.98 MB" src="https://img.shields.io/badge/size-0.98%20MB-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="speed: ~4 ms" src="https://img.shields.io/badge/speed-~4%20ms-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="trained: $0.35" src="https://img.shields.io/badge/trained-$0.35-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="data: Enron email" src="https://img.shields.io/badge/data-Enron%20email-6e6e73?style=flat-square&labelColor=1d1d1f">
68
  </p>
69
 
70
- <h3 align="center">A tiny email autocomplete that runs in your browser.<br>It learned to write from Enron's emails.</h3>
71
 
72
  <p align="center">
73
- <a href="https://huggingface.co/spaces/cfahlgren1/enron-autocomplete"><b>Try the demo&nbsp;→</b></a>
74
- &nbsp;&nbsp;·&nbsp;&nbsp;
75
  <a href="https://offlinepage.app"><b>Write with it in offlinepage.app&nbsp;→</b></a>
76
  </p>
77
 
78
  <br>
79
 
80
- tinynext is the autocomplete in [offlinepage.app](https://offlinepage.app). The app uses **v2**, in [`v2/`](https://huggingface.co/paperweight/tinynext/tree/main/v2). The files at the root are v1, the original character model, the same as [`cfahlgren1/enron-autocomplete`](https://huggingface.co/cfahlgren1/enron-autocomplete).
81
 
82
- ## v2: the next few words, with context
83
 
84
- <sub>In [`v2/`](https://huggingface.co/paperweight/tinynext/tree/main/v2). The files at the root of this repo are still v1.</sub>
85
 
86
- v1 guesses one letter at a time, so its suggestions are short word endings. v2 is a **12.5M-parameter transformer that predicts word pieces**, so one step can be a whole word, and it reads up to about 1,500 characters before the cursor (and, when there is any, text after it). It's the autocomplete in offlinepage.app, running in a Web Worker.
 
 
 
 
 
 
 
 
 
87
 
88
- **How much typing it saves.** Every model is scored by [`v2/eval.py`](v2/eval.py) on the same held-out text: type it character by character, pause every 12 characters, and count a suggestion only if the text really continues that way. Same decoding rule and 50% cutoff as the app, suggestions of 4+ characters.
89
 
90
- | Writing | v1 | v2 | SmolLM2-135M (ceiling, ~100 MB) |
 
 
 
 
91
  |---|---|---|---|
92
  | Articles (Wikipedia) | 1.5% | **3.9%** | 8.1% |
93
  | Messages (OpenAssistant) | 2.0% | **8.8%** | 16.8% |
94
  | Email (Enron, unseen senders) | 3.0% | **4.0%** | 5.7% |
95
 
96
- | Writing (v2) | Suggests at | Right | 2+ words suggested at | 2+ words right |
 
 
97
  |---|---|---|---|---|
98
  | Articles (Wikipedia) | 10% | 78% | 4% | 71% |
99
  | Messages (OpenAssistant) | 19% | 75% | 9% | 70% |
100
  | Email (Enron, unseen senders) | 11% | 78% | 4% | 51% |
101
 
102
- **Which v2.** Three runs of the same 12.5M-parameter model, typing saved on articles / messages / email. *Editing* is the same test with text after the cursor, where fill-in-the-middle can use it.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
103
 
104
  | Run | Training tokens | Fill-in-the-middle | Typing | Editing | Val loss |
105
  |---|---|---|---|---|---|
106
  | **v2 (this one)** | **3B** | half the examples | **3.9 / 8.8 / 4.0** | **3.1 / 7.6 / 4.2** | **2.89** |
107
  | First v2 | 1.5B | half the examples | 4.0 / 7.3 / 4.3 | 2.8 / 6.3 / 3.6 | 2.99 |
108
- | [No FIM](https://huggingface.co/paperweight/tinynext/tree/main/v2/experiments/tinynext-v2-nofim) | 1.5B | none | 3.9 / 8.8 / 4.2 | 3.0 / 6.9 / 4.1 | 2.94 |
109
 
110
- At 1.5B tokens, fill-in-the-middle cost the model more than it gave: half its training went to a task it rarely uses. With twice the tokens it stops costing anything on typing and helps when editing. The first v2 is in this repo's history.
111
 
112
  <details>
113
  <summary><b>Would a bigger model help?</b></summary>
@@ -123,27 +174,20 @@ At 1.5B tokens, fill-in-the-middle cost the model more than it gave: half its tr
123
  | SmolLM2 | 135M | 8.1% | 16.8% | 5.7% |
124
  | **tinynext v2** | **12.5M** | **3.9%** | **8.8%** | **4.0%** |
125
 
126
- tinynext matches Pythia models 2–3× its size because it trained on writing like what people type. Size starts to pay off between 30M and 70M parameters, and better data (SmolLM2) matters as much as size.
127
 
128
  </details>
129
 
130
- **In the browser.**
131
-
132
- | | |
133
- |---|---|
134
- | Download | **13.3 MB**: int8 weights with one scale per 64 values, expanded to float32 when the page loads |
135
- | Matches full precision | same next word 96% of the time, same 5-word path 90% |
136
- | Speed | **5 ms** per suggestion (13 ms p95) in a Web Worker, Chrome on an M3 Max; typing stays under one frame |
137
-
138
- The browser computes in float32 on purpose: onnxruntime-web's int8 kernels were 8–11× slower than its float kernels for this model, and int8 activations changed the top word too often. Only the download is compressed.
139
-
140
- **Training.** 3B tokens (60% [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), 30% [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk), 10% Enron email from training senders), half of it as fill-in-the-middle examples. 56 minutes on one A100 at 920K tokens a second, from data tokenized once on a CPU job. Validation loss 2.89. [Training curves on Trackio](https://huggingface.co/spaces/cfahlgren1/tinynext-trackio).
141
 
142
- **Limits.**
 
 
 
 
143
 
144
- - offlinepage.app only suggests at the end of a paragraph, so the text after the cursor is the next paragraph. Mid-line suggestions, where fill-in-the-middle should help most, aren't built yet.
145
- - It still leans toward office email, and it only sees keyboard characters.
146
- - It's English only.
147
 
148
  <details>
149
  <summary><b>v2 files</b></summary>
@@ -162,20 +206,14 @@ The browser computes in float32 on purpose: onnxruntime-web's int8 kernels were
162
 
163
  </details>
164
 
165
- <details>
166
- <summary><b>Train it yourself</b></summary>
167
 
168
- ```bash
169
- # Prepare data once on a CPU job, then train on a GPU from the prepared files.
170
- hf jobs uv run --flavor cpu-xl --secrets HF_TOKEN train.py --stage prepare --tokens 3e9 --fineweb-files 4
171
- hf jobs uv run --flavor a100-large --secrets HF_TOKEN train.py --stage train --tokens 3e9 --fineweb-files 4
172
- ```
173
 
174
- </details>
175
-
176
- ## v1: the original character model
177
 
178
- ### What it does
179
 
180
  It finishes the word or short phrase you're typing. Press <kbd>Tab</kbd> to take the suggestion.
181
 
@@ -186,7 +224,7 @@ It finishes the word or short phrase you're typing. Press <kbd>Tab</kbd> to take
186
  | The books are fine. Please call me i | **f you have any** |
187
  | Enron N | **orth America Corp.** |
188
 
189
- ### How sure it has to be
190
 
191
  The model knows how sure it is of each guess. You pick a cutoff, and it only suggests when it's at least that sure. A higher cutoff means fewer suggestions, and more of them right.
192
 
@@ -202,21 +240,21 @@ The model knows how sure it is of each guess. You pick a cutoff, and it only sug
202
  |---|---|---|
203
  | 20% | 30% of the time | 48% |
204
  | 35% | 18% of the time | 66% |
205
- | **50%** *(offlinepage.app)* | **11% of the time** | **80%** |
206
  | 70% | 7% of the time | 90% |
207
 
208
- The demo uses 40%, and offlinepage.app uses 50%.
209
 
210
  These numbers come from 5,000 places in emails from people the model never saw in training. “Right” means the suggestion exactly matched what the person typed next.
211
 
212
- #### Limits
213
 
214
  - It only knows office email from 2001. Outside that it has little to say.
215
  - With no cutoff it rambles. “Dear Santa,” becomes “I am attaching a copy of the agreement to the Enron Corp.”
216
  - It learned real names and sign-offs from the emails, so keep suggestions short.
217
  - English only.
218
 
219
- ### Use it
220
 
221
  The model takes one character and its memory of the text so far. It returns its guess for the next character and an updated memory. So each key press is one small step.
222
 
@@ -264,7 +302,7 @@ I also tried a bigger version with 3.46M parameters and 3.5 MB. It wasn't better
264
 
265
  </details>
266
 
267
- ### Training
268
 
269
  It read 40 million characters from 119,859 emails sent from enron.com addresses, 30 times over. Typos were added to what it read, so it keeps predicting when you make one. The run took 14 minutes on one A10G with Hugging Face Jobs and cost 35 cents.
270
 
@@ -274,6 +312,9 @@ It read 40 million characters from 119,859 emails sent from enron.com addresses,
274
  hf jobs uv run --flavor a10g-large --secrets HF_TOKEN train.py --repo your-name/your-model
275
  ```
276
 
 
 
 
277
  <br>
278
 
279
  <p align="center">
 
64
  </p>
65
 
66
  <p align="center">
67
+ <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="params: 12.5M" src="https://img.shields.io/badge/params-12.5M-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="size: 13.3 MB" src="https://img.shields.io/badge/size-13.3%20MB-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="speed: ~5 ms" src="https://img.shields.io/badge/speed-~5%20ms-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="runs: in browser" src="https://img.shields.io/badge/runs-in%20browser-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="data sent: none" src="https://img.shields.io/badge/data%20sent-none-6e6e73?style=flat-square&labelColor=1d1d1f">
68
  </p>
69
 
70
+ <h3 align="center">Gray text that finishes your sentence.<br>A 13 MB model that runs in a browser tab.</h3>
71
 
72
  <p align="center">
 
 
73
  <a href="https://offlinepage.app"><b>Write with it in offlinepage.app&nbsp;→</b></a>
74
  </p>
75
 
76
  <br>
77
 
78
+ tinynext is the autocomplete in [offlinepage.app](https://offlinepage.app). It reads the text around the cursor and, when it's sure enough, shows the next few words in gray. Press <kbd>Tab</kbd> to take them. It runs in a Web Worker with [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/), so nothing you type is sent anywhere.
79
 
80
+ The model the app uses is in [`v2/`](https://huggingface.co/paperweight/tinynext/tree/main/v2). The files at the root of this repo are v1, an earlier and much smaller character model, described [at the end](#v1-the-original-character-model).
81
 
82
+ ## What it does
83
 
84
+ <p align="center">
85
+ <img src="what-it-does.png" alt="You type: Please accept my resignation. I will be o. You get: the same text with “ut of the office” offered in gray, and Tab to take it." width="880">
86
+ </p>
87
+
88
+ | You type | It suggests |
89
+ |---|---|
90
+ | Please accept my resignation. I will be o | **ut of the office** |
91
+ | See you on the ground. I look f | **orward to** |
92
+
93
+ It stays quiet most of the time. A suggestion only appears when the model is at least 40% sure of the whole phrase, and it has to be four characters or longer.
94
 
95
+ ## How well it works
96
 
97
+ Every model here is scored the same way by [`v2/eval.py`](v2/eval.py): type held-out text character by character, pause every 12 characters, and count a suggestion only if the text really continues that way. These numbers use a 50% cutoff; the app now uses 40%, so it suggests a little more often and is right a little less often.
98
+
99
+ **How much typing it saves:**
100
+
101
+ | Writing | v1 | **v2** | SmolLM2-135M (about 100 MB) |
102
  |---|---|---|---|
103
  | Articles (Wikipedia) | 1.5% | **3.9%** | 8.1% |
104
  | Messages (OpenAssistant) | 2.0% | **8.8%** | 16.8% |
105
  | Email (Enron, unseen senders) | 3.0% | **4.0%** | 5.7% |
106
 
107
+ **How often it speaks up, and how often it's right:**
108
+
109
+ | Writing | Suggests at | Right | 2+ words suggested at | 2+ words right |
110
  |---|---|---|---|---|
111
  | Articles (Wikipedia) | 10% | 78% | 4% | 71% |
112
  | Messages (OpenAssistant) | 19% | 75% | 9% | 70% |
113
  | Email (Enron, unseen senders) | 11% | 78% | 4% | 51% |
114
 
115
+ “Suggests at” is the share of pauses where it offered anything. “Right” means the suggestion exactly matched what the person typed next.
116
+
117
+ ### Limits
118
+
119
+ - offlinepage.app only suggests at the end of a paragraph. Mid-line suggestions, where reading the text after the cursor should help most, aren't built yet.
120
+ - It still leans toward office email, and it only sees keyboard characters.
121
+ - English only.
122
+
123
+ ## In the browser
124
+
125
+ | Setting | Value |
126
+ |---|---|
127
+ | Download | **13.3 MB**: 8-bit weights with one scale per 64 values, expanded to float32 when the page loads |
128
+ | Matches full precision | Same next word 96% of the time, same 5-word path 90% |
129
+ | Speed | **5 ms** per suggestion (13 ms at the slow end) in a Web Worker, Chrome on an M3 Max |
130
+
131
+ The browser computes in float32 on purpose. ONNX Runtime Web's 8-bit kernels were 8 to 11 times slower than its float kernels for this model, and 8-bit activations changed the top word too often. Only the download is compressed.
132
+
133
+ ## How it was trained
134
+
135
+ **The task.** Predict the next word piece. Half of the training examples are fill-in-the-middle: the model also sees the text after a gap, so it can suggest in the middle of a paragraph as well as at the end.
136
+
137
+ **The data.** 3 billion tokens of writing like what people type: 60% [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) articles, 30% [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) conversations and 10% [Enron email](https://huggingface.co/datasets/corbt/enron-emails), with the emails split by sender so the test senders are never seen.
138
+
139
+ **The model.** A small transformer that reads up to about 1,500 characters before the cursor.
140
+
141
+ | Setting | Value |
142
+ |---|---|
143
+ | Parameters | 12.5 million |
144
+ | Layers / width / heads | 8 / 320 / 5 |
145
+ | Tokenizer | 8,192 byte-level BPE pieces, so one step can be a whole word |
146
+ | Context | 512 tokens |
147
+ | Training | One pass over 3 billion tokens, batch 512, learning rate 0.003 |
148
+ | Time | 56 minutes on one A100 at 920,000 tokens a second, with [Hugging Face Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs) |
149
+ | Validation loss | 2.89 |
150
+
151
+ The data is tokenized once on a CPU job, and the GPU job trains from the prepared files. [Training curves are on Trackio](https://huggingface.co/spaces/cfahlgren1/tinynext-trackio).
152
+
153
+ **What the runs showed.** Three runs of the same model, scored as typing saved on articles / messages / email. *Editing* is the same test with text after the cursor.
154
 
155
  | Run | Training tokens | Fill-in-the-middle | Typing | Editing | Val loss |
156
  |---|---|---|---|---|---|
157
  | **v2 (this one)** | **3B** | half the examples | **3.9 / 8.8 / 4.0** | **3.1 / 7.6 / 4.2** | **2.89** |
158
  | First v2 | 1.5B | half the examples | 4.0 / 7.3 / 4.3 | 2.8 / 6.3 / 3.6 | 2.99 |
159
+ | [No fill-in-the-middle](https://huggingface.co/paperweight/tinynext/tree/main/v2/experiments/tinynext-v2-nofim) | 1.5B | none | 3.9 / 8.8 / 4.2 | 3.0 / 6.9 / 4.1 | 2.94 |
160
 
161
+ At 1.5 billion tokens, fill-in-the-middle cost more than it gave: half the training went to a task the model rarely uses. With twice the tokens it stops costing anything on typing and helps when editing.
162
 
163
  <details>
164
  <summary><b>Would a bigger model help?</b></summary>
 
174
  | SmolLM2 | 135M | 8.1% | 16.8% | 5.7% |
175
  | **tinynext v2** | **12.5M** | **3.9%** | **8.8%** | **4.0%** |
176
 
177
+ tinynext matches Pythia models two to three times its size because it trained on writing like what people type. Size starts to pay off between 30M and 70M parameters, and better data (SmolLM2) matters as much as size.
178
 
179
  </details>
180
 
181
+ <details>
182
+ <summary><b>Train it yourself</b></summary>
 
 
 
 
 
 
 
 
 
183
 
184
+ ```bash
185
+ # Prepare data once on a CPU job, then train on a GPU from the prepared files.
186
+ hf jobs uv run --flavor cpu-xl --secrets HF_TOKEN train.py --stage prepare --tokens 3e9 --fineweb-files 4
187
+ hf jobs uv run --flavor a100-large --secrets HF_TOKEN train.py --stage train --tokens 3e9 --fineweb-files 4
188
+ ```
189
 
190
+ </details>
 
 
191
 
192
  <details>
193
  <summary><b>v2 files</b></summary>
 
206
 
207
  </details>
208
 
209
+ ## v1: the original character model
 
210
 
211
+ The files at the root of this repo. v1 guesses one letter at a time, so its suggestions are short word endings. It has 944,000 parameters, is 0.98 MB, and learned only from Enron's emails. It is the same model as [`cfahlgren1/enron-autocomplete`](https://huggingface.co/cfahlgren1/enron-autocomplete), and it has [its own demo](https://huggingface.co/spaces/cfahlgren1/enron-autocomplete).
 
 
 
 
212
 
213
+ <details>
214
+ <summary><b>Everything about v1</b></summary>
 
215
 
216
+ **What it does**
217
 
218
  It finishes the word or short phrase you're typing. Press <kbd>Tab</kbd> to take the suggestion.
219
 
 
224
  | The books are fine. Please call me i | **f you have any** |
225
  | Enron N | **orth America Corp.** |
226
 
227
+ **How sure it has to be**
228
 
229
  The model knows how sure it is of each guess. You pick a cutoff, and it only suggests when it's at least that sure. A higher cutoff means fewer suggestions, and more of them right.
230
 
 
240
  |---|---|---|
241
  | 20% | 30% of the time | 48% |
242
  | 35% | 18% of the time | 66% |
243
+ | **50%** | **11% of the time** | **80%** |
244
  | 70% | 7% of the time | 90% |
245
 
246
+ The demo uses 40%.
247
 
248
  These numbers come from 5,000 places in emails from people the model never saw in training. “Right” means the suggestion exactly matched what the person typed next.
249
 
250
+ **Limits**
251
 
252
  - It only knows office email from 2001. Outside that it has little to say.
253
  - With no cutoff it rambles. “Dear Santa,” becomes “I am attaching a copy of the agreement to the Enron Corp.”
254
  - It learned real names and sign-offs from the emails, so keep suggestions short.
255
  - English only.
256
 
257
+ **Use it**
258
 
259
  The model takes one character and its memory of the text so far. It returns its guess for the next character and an updated memory. So each key press is one small step.
260
 
 
302
 
303
  </details>
304
 
305
+ **Training**
306
 
307
  It read 40 million characters from 119,859 emails sent from enron.com addresses, 30 times over. Typos were added to what it read, so it keeps predicting when you make one. The run took 14 minutes on one A10G with Hugging Face Jobs and cost 35 cents.
308
 
 
312
  hf jobs uv run --flavor a10g-large --secrets HF_TOKEN train.py --repo your-name/your-model
313
  ```
314
 
315
+
316
+ </details>
317
+
318
  <br>
319
 
320
  <p align="center">