Update README.md
Browse files
README.md
CHANGED
|
@@ -67,4 +67,4 @@ toks = tokenizer("Всі красиві зберігають оптимізм",
|
|
| 67 |
print(len(toks.input_ids)) -only 4 tokens 💪🏻
|
| 68 |
```
|
| 69 |
|
| 70 |
-
### "fixed" - means that we remove condition that allow to add empty
|
|
|
|
| 67 |
print(len(toks.input_ids)) -only 4 tokens 💪🏻
|
| 68 |
```
|
| 69 |
|
| 70 |
+
### "fixed" - means that we remove condition that allow to add empty `<think></think>` for hybrid approach. This significantly speeds up tokenization.
|