Update README.md

README.md

@@ -1,3 +1,25 @@
 ---
 license: mit
 ---
+
+## GPT-2 Tokenizer with unmerged digits
+
+```python
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained('cyrilzhang/gpt2-numfix')
+```
+
+A fork of the GPT-2 tokenizer, which **removes multi-digit tokens**:
+```python
+tokenizer('123.45')  # [16, 17, 18, 13, 19, 20]
+gpt2_tokenizer('123.45')  # [10163, 13, 2231]
+```
+
+Backwards-compatible:
+```python
+tokenizer.decode([10163, 46387])  # '<unused123> pigeon'
+gpt2_tokenizer.decode([10163, 46387])  # '123 pigeon'
+```
+
+[PaLM](https://arxiv.org/abs/2204.02311) does this.