ONNX
English
autocorrect
spelling-correction
browser
offline-page
cfahlgren1 HF Staff commited on
Commit
2d1da4c
·
verified ·
1 Parent(s): bbe52c1

rewrite the model card with how it works and how it was trained

Browse files
Files changed (1) hide show
  1. README.md +59 -14
README.md CHANGED
@@ -24,9 +24,9 @@ datasets:
24
  <h3 align="center">Spell check can't tell that “form” was meant to be “from”,<br>since both are real words. This 2.1 MB model can.</h3>
25
 
26
  <p align="center">
27
- <a href="https://huggingface.co/spaces/cfahlgren1/tinyfix"><b>Try the demo&nbsp;→</b></a>
28
- &nbsp;&nbsp;·&nbsp;&nbsp;
29
  <a href="https://offlinepage.app"><b>Write with it in offlinepage.app&nbsp;→</b></a>
 
 
30
  </p>
31
 
32
  <br>
@@ -36,14 +36,15 @@ tinyfix reads the words on both sides of a correctly spelled word and picks the
36
  ## What it does
37
 
38
  <p align="center">
39
- <img src="tinyfix.png" alt="The tinyfix demo correcting “I recieve the report tomorow and we saw there new house.” as it is typed" width="760">
40
  </p>
41
 
42
  | You type | It becomes |
43
  |---|---|
44
  | I will send it <del>form</del> home tomorrow. | I will send it **from** home tomorrow. |
45
- | Let me <del>now</del> if that works. | Let me **know** if that works. |
46
- | We saw <del>there</del> new house. | We saw **their** new house. |
 
47
  | Please fill out the **form** and return it. | *No change. “form” is right here.* |
48
 
49
  ## How well it works
@@ -51,12 +52,12 @@ tinyfix reads the words on both sides of a correctly spelled word and picks the
51
  <table>
52
  <tr>
53
  <td align="center" width="33%"><h2>94%</h2>of its fixes are right</td>
54
- <td align="center" width="33%"><h2>40%</h2>of wrong words fixed<br><sub>24% before this version</sub></td>
55
  <td align="center" width="33%"><h2>~3 ms</h2>per check in Chrome</td>
56
  </tr>
57
  </table>
58
 
59
- Measured by typing test text into the code [offlinepage.app](https://github.com/cfahlgren1/offline-page) runs, on text the model never trained on and that nothing was tuned on. The 40% is the average of five test sets:
60
 
61
  | Test set | Wrong words fixed | Fixes that were right |
62
  |---|---|---|
@@ -66,7 +67,7 @@ Measured by typing test text into the code [offlinepage.app](https://github.com/
66
  | Real errors from children's writing | **28%** | 91% |
67
  | Real typos from GitHub commits | **18%** | 96% |
68
 
69
- On about 110,000 words of correct email, chat, encyclopedia, news, documentation and fiction it wrongly changed about 1 word in 1,800, the same as the version before it.
70
 
71
  The first two sets are built from the mix-up pairs the model was trained to watch for, so they flatter it. The last three are the fairer picture of everyday typing.
72
 
@@ -79,6 +80,54 @@ The first two sets are built from the mix-up pairs the model was trained to watc
79
  - It needs the words that come after, so these fixes land a couple of words later.
80
  - English only.
81
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
82
  ## Using the model
83
 
84
  The model scores candidate words for one position in a sentence. Higher scores fit better, and comparing each candidate to the typed word tells you whether a swap is worth making.
@@ -104,17 +153,13 @@ IDs are line numbers in `wrong-word-vocab.txt`, starting from 0. Words are lower
104
  |---|---|
105
  | `wrong-word.onnx` | The quantized model |
106
  | `wrong-word-vocab.txt` | The 20,000 words it knows, one per line |
107
- | `wrong-word-confusions.txt` | 234 words people mix up, like “their” and “there”, each with the words they are mistaken for |
108
  | `train_realword.py` | The training script |
109
 
110
- The model only scores words. The rules that decide when to make a swap (which words to check, the extra confidence needed at the start of a sentence, and so on) are in offlinepage.app's [`src/tinyfix`](https://github.com/cfahlgren1/offline-page/tree/main/src/tinyfix).
111
 
112
  </details>
113
 
114
- ## Training
115
-
116
- A word-level GRU that reads 16 words before and up to 4 words after, trained for 120,000 steps on one GPU. It learned from [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) and [OpenAssistant](https://huggingface.co/datasets/OpenAssistant/oasst1) text, by swapping real words for nearby real words and learning to swap them back. About one example in four is centered on a known mix-up, half of them used correctly, so it also learns when to leave a word alone.
117
-
118
  <br>
119
 
120
  <p align="center">
 
24
  <h3 align="center">Spell check can't tell that “form” was meant to be “from”,<br>since both are real words. This 2.1 MB model can.</h3>
25
 
26
  <p align="center">
 
 
27
  <a href="https://offlinepage.app"><b>Write with it in offlinepage.app&nbsp;→</b></a>
28
+ &nbsp;&nbsp;·&nbsp;&nbsp;
29
+ <a href="https://huggingface.co/spaces/cfahlgren1/tinyfix"><b>Try the demo&nbsp;→</b></a>
30
  </p>
31
 
32
  <br>
 
36
  ## What it does
37
 
38
  <p align="center">
39
+ <img src="what-it-does.png" alt="You type: I got a letter form the bank. Their is a problem, and I am to tired to recieve it. You get: I got a letter from the bank. There is a problem, and I am too tired to receive it." width="880">
40
  </p>
41
 
42
  | You type | It becomes |
43
  |---|---|
44
  | I will send it <del>form</del> home tomorrow. | I will send it **from** home tomorrow. |
45
+ | <del>Their</del> is a problem with the invoice. | **There** is a problem with the invoice. |
46
+ | I am <del>to</del> tired to cook tonight. | I am **too** tired to cook tonight. |
47
+ | I think <del>your</del> right about that. | I think **you're** right about that. |
48
  | Please fill out the **form** and return it. | *No change. “form” is right here.* |
49
 
50
  ## How well it works
 
52
  <table>
53
  <tr>
54
  <td align="center" width="33%"><h2>94%</h2>of its fixes are right</td>
55
+ <td align="center" width="33%"><h2>40%</h2>of wrong words fixed<br><sub>24% in the first version</sub></td>
56
  <td align="center" width="33%"><h2>~3 ms</h2>per check in Chrome</td>
57
  </tr>
58
  </table>
59
 
60
+ Measured by typing test text into the same code [offlinepage.app](https://github.com/cfahlgren1/offline-page) runs, on text the model never trained on and that nothing was tuned on. The 40% is the average of five test sets:
61
 
62
  | Test set | Wrong words fixed | Fixes that were right |
63
  |---|---|---|
 
67
  | Real errors from children's writing | **28%** | 91% |
68
  | Real typos from GitHub commits | **18%** | 96% |
69
 
70
+ On about 110,000 words of correct email, chat, encyclopedia, news, documentation and fiction, it wrongly changed about 1 word in 2,000.
71
 
72
  The first two sets are built from the mix-up pairs the model was trained to watch for, so they flatter it. The last three are the fairer picture of everyday typing.
73
 
 
80
  - It needs the words that come after, so these fixes land a couple of words later.
81
  - English only.
82
 
83
+ ## How it works
84
+
85
+ Autocorrect in offlinepage.app is two parts, and this model is the second one.
86
+
87
+ 1. **Misspellings** (“recieve”) are caught without a model: an 80,000-word list, a search for words within one or two edits, and a cost for each edit based on how likely that slip is on a keyboard.
88
+ 2. **Real words in the wrong place** (“form” for “from”) are what tinyfix is for. For each finished word, the app gathers up to eight nearby real words: known mix-ups first (their/there/they're), then words one edit away. tinyfix scores the typed word and each candidate against the 16 words before and up to 4 words after.
89
+ 3. **The swap is only made when the gap is clear.** The best candidate has to beat the typed word by a set margin, and a less likely slip needs a bigger margin. That margin is what keeps wrong changes to about 1 word in 2,000.
90
+
91
+ The app waits for two more words, or the end of the sentence, before it checks a word. That is why a fix lands a moment after the typo.
92
+
93
+ ## How it was trained
94
+
95
+ **The task.** Take a sentence of ordinary text, pick a word, and half the time swap it for a nearby real word. The model sees the words around the gap and a short list of candidates, and learns to score the word that was really there highest. The other half of the time nothing is swapped, which teaches it to leave correct words alone.
96
+
97
+ **The data.** About 120 million words of plain text, with no labels: 70 million from [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) articles and 50 million from [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) conversations, plus [OpenAssistant](https://huggingface.co/datasets/OpenAssistant/oasst1) messages.
98
+
99
+ **The recipe that mattered.** One training example in four is centered on a word from 160 groups of words people mix up, and half of those show the word used correctly. This one change did most of the work: it took the fix rate on classic mix-ups from 25% to 69%.
100
+
101
+ **The model.** Two small GRUs, one reading left to right up to the word and one reading right to left back to it, joined and compared against each candidate's embedding.
102
+
103
+ | Setting | Value |
104
+ |---|---|
105
+ | Parameters | 1.9 million |
106
+ | Vocabulary | 20,000 words, lowercased; names and numbers each become one token |
107
+ | Embedding / hidden size | 64 / 256 |
108
+ | Context | 16 words before, 4 after |
109
+ | Training | 120,000 steps, batch 4,096, learning rate 0.002 |
110
+ | Time | about 1 hour 45 minutes on one A10G, with [Hugging Face Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs) |
111
+ | Download | 2.1 MB after 8-bit quantization (the GRU is built from plain matrix multiplies so ONNX Runtime can quantize it) |
112
+
113
+ **How it was measured.** A bench that types frozen test text through the app's real code, one keystroke at a time, and counts what got fixed and what got wrongly changed. Settings were tuned on a separate development split. Two runs of the same recipe differ by 2 to 3 points, so smaller differences were treated as noise.
114
+
115
+ <details>
116
+ <summary><b>What was tried and didn't help</b></summary>
117
+
118
+ Each of these scored the same as the recipe above, within the noise between two identical runs:
119
+
120
+ - A wider embedding (96 instead of 64), which also made the file bigger
121
+ - Distilling from BERT
122
+ - More context after the word
123
+ - Training on real typo pairs from GitHub commits
124
+ - Simulated keyboard slips as training swaps
125
+ - Training for longer, and averaging weights across the run
126
+
127
+ Most of the remaining misses come from two limits outside training. The right word often isn't among the candidates offered, and a typo outside the 20,000-word vocabulary can't be judged at all.
128
+
129
+ </details>
130
+
131
  ## Using the model
132
 
133
  The model scores candidate words for one position in a sentence. Higher scores fit better, and comparing each candidate to the typed word tells you whether a swap is worth making.
 
153
  |---|---|
154
  | `wrong-word.onnx` | The quantized model |
155
  | `wrong-word-vocab.txt` | The 20,000 words it knows, one per line |
156
+ | `wrong-word-confusions.txt` | About 235 words people mix up, like “their” and “there”, each with the words they are mistaken for |
157
  | `train_realword.py` | The training script |
158
 
159
+ The model only scores words. The rules that decide when to make a swap are in offlinepage.app's [`src/tinyfix`](https://github.com/cfahlgren1/offline-page/tree/main/src/tinyfix).
160
 
161
  </details>
162
 
 
 
 
 
163
  <br>
164
 
165
  <p align="center">