otmanheddouch commited on
Commit
22f4458
·
verified ·
1 Parent(s): 8f49b51

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +7 -0
README.md CHANGED
@@ -35,6 +35,13 @@ This is the model card of a 🤗 transformers model that has been pushed on the
35
 
36
  ## Uses
37
 
 
 
 
 
 
 
 
38
  <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
39
 
40
  ### Direct Use
 
35
 
36
  ## Uses
37
 
38
+ from transformers import AutoTokenizer
39
+
40
+ shakspear_tokenizer = AutoTokenizer.from_pretrained("otmanheddouch/shakespear-tokenizer")
41
+
42
+ tokens = tokenizer.tokenize(example)
43
+ tokens_ids = tokenizer.encode(example)
44
+
45
  <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
46
 
47
  ### Direct Use