File size: 349 Bytes
5d6a12e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
---
language: kaa
license: mit
---
## How to use
You can load this tokenizer directly using the `transformers` library in Python:
\`\`\`python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bekan/testkaa-tokenizer")
text = "C++ programmalastırıw tili."
tokens = tokenizer.tokenize(text)
print(tokens)
\`\`\` |