metadata
language: kaa
license: mit
How to use
You can load this tokenizer directly using the transformers library in Python:
```python from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bekan/testkaa-tokenizer")
text = "C++ programmalastırıw tili." tokens = tokenizer.tokenize(text) print(tokens) ```