4ldk
/

Roberta-Base-CoNLL2003

Token Classification

Model card Files Files and versions

4ldk commited on Feb 14, 2024

Commit

89fa38d

·

verified ·

1 Parent(s): f480d02

Update README.md

Files changed (1) hide show

README.md +3 -2

README.md CHANGED Viewed

@@ -22,14 +22,15 @@ This model is a fine-tuned version of [roberta-base](https://huggingface.co/robe
 ## Model Usage
 We made and used the original tokenizer with [BPE-Dropout](https://aclanthology.org/2020.acl-main.170/).
 Example and Tokenizer Repository: [github](https://github.com/4ldk/CoNLL2003_Choices)
 ```python
-from transformers import AutoTokenizer, AutoModelForTokenClassification
 from transformers import pipeline
-tokenizer = AutoTokenizer.from_pretrained("4ldk/Roberta-Base-CoNLL2003")
 model = AutoModelForTokenClassification.from_pretrained("4ldk/Roberta-Base-CoNLL2003")
 nlp = pipeline("ner", model=model, tokenizer=tokenizer, grouped_entities=True)

 ## Model Usage
 We made and used the original tokenizer with [BPE-Dropout](https://aclanthology.org/2020.acl-main.170/).
+So, you can't use AutoTokenizer but if subword normalization is not used, original RobertaTokenizer can be substituted.
 Example and Tokenizer Repository: [github](https://github.com/4ldk/CoNLL2003_Choices)
 ```python
+from transformers import RobertaTokenizer, AutoModelForTokenClassification
 from transformers import pipeline
+tokenizer = RobertaTokenizer.from_pretrained("4ldk/Roberta-Base-CoNLL2003")
 model = AutoModelForTokenClassification.from_pretrained("4ldk/Roberta-Base-CoNLL2003")
 nlp = pipeline("ner", model=model, tokenizer=tokenizer, grouped_entities=True)