4ldk commited on
Commit
89fa38d
·
verified ·
1 Parent(s): f480d02

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -2
README.md CHANGED
@@ -22,14 +22,15 @@ This model is a fine-tuned version of [roberta-base](https://huggingface.co/robe
22
  ## Model Usage
23
 
24
  We made and used the original tokenizer with [BPE-Dropout](https://aclanthology.org/2020.acl-main.170/).
 
25
 
26
  Example and Tokenizer Repository: [github](https://github.com/4ldk/CoNLL2003_Choices)
27
 
28
  ```python
29
- from transformers import AutoTokenizer, AutoModelForTokenClassification
30
  from transformers import pipeline
31
 
32
- tokenizer = AutoTokenizer.from_pretrained("4ldk/Roberta-Base-CoNLL2003")
33
  model = AutoModelForTokenClassification.from_pretrained("4ldk/Roberta-Base-CoNLL2003")
34
 
35
  nlp = pipeline("ner", model=model, tokenizer=tokenizer, grouped_entities=True)
 
22
  ## Model Usage
23
 
24
  We made and used the original tokenizer with [BPE-Dropout](https://aclanthology.org/2020.acl-main.170/).
25
+ So, you can't use AutoTokenizer but if subword normalization is not used, original RobertaTokenizer can be substituted.
26
 
27
  Example and Tokenizer Repository: [github](https://github.com/4ldk/CoNLL2003_Choices)
28
 
29
  ```python
30
+ from transformers import RobertaTokenizer, AutoModelForTokenClassification
31
  from transformers import pipeline
32
 
33
+ tokenizer = RobertaTokenizer.from_pretrained("4ldk/Roberta-Base-CoNLL2003")
34
  model = AutoModelForTokenClassification.from_pretrained("4ldk/Roberta-Base-CoNLL2003")
35
 
36
  nlp = pipeline("ner", model=model, tokenizer=tokenizer, grouped_entities=True)