Corianas commited on
Commit
0e30278
·
verified ·
1 Parent(s): 4281937

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -13
README.md CHANGED
@@ -36,13 +36,11 @@ No PyTorch/TensorFlow/Flax required to use the tokenizer itself.
36
 
37
  ## Usage
38
 
39
- ### Load from local folder
40
-
41
  ```python
42
  from transformers import AutoTokenizer
43
 
44
- # Load local tokenizer folder
45
- tok = AutoTokenizer.from_pretrained("char128_shift_tokenizer")
46
 
47
  print(tok.vocab_size) # 128
48
  ids = tok.encode("Hello, There!\n<eos>")
@@ -51,15 +49,6 @@ print(tok.decode(ids, skip_special_tokens=True, clean_up_tokenization_spaces=Fal
51
  # → "↨hello, ↨there!\n<eos>"
52
  ```
53
 
54
- ### Load from Hugging Face Hub
55
-
56
- ```python
57
- from transformers import AutoTokenizer
58
-
59
- # Replace with your Hub repo
60
- tok = AutoTokenizer.from_pretrained("Corianas/char128_shift_tokenizer")
61
- ```
62
-
63
  ---
64
 
65
  ## Restoring Uppercase
 
36
 
37
  ## Usage
38
 
 
 
39
  ```python
40
  from transformers import AutoTokenizer
41
 
42
+ # Replace with your Hub repo
43
+ tok = AutoTokenizer.from_pretrained("Corianas/char128_shift_tokenizer")
44
 
45
  print(tok.vocab_size) # 128
46
  ids = tok.encode("Hello, There!\n<eos>")
 
49
  # → "↨hello, ↨there!\n<eos>"
50
  ```
51
 
 
 
 
 
 
 
 
 
 
52
  ---
53
 
54
  ## Restoring Uppercase