Instructions to use krinal/BertWordPieceTokenizer-hi with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use krinal/BertWordPieceTokenizer-hi with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="krinal/BertWordPieceTokenizer-hi")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("krinal/BertWordPieceTokenizer-hi", dtype="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -19,7 +19,7 @@ from transformers import AutoTokenizer
|
|
| 19 |
|
| 20 |
hi_tokenizer = AutoTokenizer.from_pretrained('krinal/BertWordPieceTokenizer-hi')
|
| 21 |
|
| 22 |
-
hi_str = "आज का सूर्य देखो, कितना प्यारा, कितना शीतल है
|
| 23 |
|
| 24 |
# encode text
|
| 25 |
encoded_str = hi_tokenizer.encode(hi_str)
|
|
|
|
| 19 |
|
| 20 |
hi_tokenizer = AutoTokenizer.from_pretrained('krinal/BertWordPieceTokenizer-hi')
|
| 21 |
|
| 22 |
+
hi_str = "आज का सूर्य देखो, कितना प्यारा, कितना शीतल है"
|
| 23 |
|
| 24 |
# encode text
|
| 25 |
encoded_str = hi_tokenizer.encode(hi_str)
|