Update README.md
Browse files
README.md
CHANGED
|
@@ -35,6 +35,13 @@ This is the model card of a 🤗 transformers model that has been pushed on the
|
|
| 35 |
|
| 36 |
## Uses
|
| 37 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
|
| 39 |
|
| 40 |
### Direct Use
|
|
|
|
| 35 |
|
| 36 |
## Uses
|
| 37 |
|
| 38 |
+
from transformers import AutoTokenizer
|
| 39 |
+
|
| 40 |
+
shakspear_tokenizer = AutoTokenizer.from_pretrained("otmanheddouch/shakespear-tokenizer")
|
| 41 |
+
|
| 42 |
+
tokens = tokenizer.tokenize(example)
|
| 43 |
+
tokens_ids = tokenizer.encode(example)
|
| 44 |
+
|
| 45 |
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
|
| 46 |
|
| 47 |
### Direct Use
|