|
|
--- |
|
|
license: mit |
|
|
task_categories: |
|
|
- text-generation |
|
|
language: |
|
|
- tr |
|
|
tags: |
|
|
- tokenizer |
|
|
- bpe |
|
|
- turkish |
|
|
- wikipedia |
|
|
size_categories: |
|
|
- 10K<n<100K |
|
|
--- |
|
|
|
|
|
# Turkish BPE Tokenizer for Mamba & GPT Models |
|
|
|
|
|
This is a custom **Byte-Pair Encoding (BPE)** tokenizer trained specifically for the Turkish language to support the `mamba-tr-project` research. |
|
|
|
|
|
## Model Details |
|
|
- **Vocabulary Size:** 32,000 tokens |
|
|
- **Training Corpus:** Turkish Wikipedia (2023 Dump) |
|
|
- **Algorithm:** BPE (Byte-Level) |
|
|
- **Special Tokens:** `<|endoftext|>`, `<|padding|>` |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import PreTrainedTokenizerFast |
|
|
|
|
|
tokenizer = PreTrainedTokenizerFast.from_pretrained("oguzatas/mamba-tr-project-tokenizer") |
|
|
|
|
|
text = "Yapay zeka günümüzde çok gelişti." |
|
|
tokens = tokenizer.encode(text) |
|
|
decoded = tokenizer.decode(tokens) |
|
|
|
|
|
print(tokens) |
|
|
# Output: [Token IDs...] |
|
|
print(decoded) |
|
|
# Output: "Yapay zeka günümüzde çok gelişti." |
