| language: kaa | |
| license: mit | |
| ## How to use | |
| You can load this tokenizer directly using the `transformers` library in Python: | |
| \`\`\`python | |
| from transformers import AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("bekan/testkaa-tokenizer") | |
| text = "C++ programmalastırıw tili." | |
| tokens = tokenizer.tokenize(text) | |
| print(tokens) | |
| \`\`\` |