| ---
|
| license: apache-2.0
|
| language:
|
| - tr
|
| tags:
|
| - tokenizer
|
| - bpe
|
| - e-commerce
|
| - turkish
|
| pretty_name: TR E-Commerce Customer Support Tokenizer
|
| ---
|
|
|
| # TR E-Commerce Customer Support Tokenizer 🇹🇷
|
|
|
| A custom-trained **Byte-Pair Encoding (BPE)** tokenizer optimized specifically for Turkish e-commerce customer support dialogues. Trained on the `Mer1Alii/TR-ECommerce-CustomerSupport-Instructions` corpus, this tokenizer drastically improves token efficiency and semantic comprehension for Turkish conversational AI.
|
|
|
| ---
|
|
|
| ## 1. The Challenge of Turkish Tokenization
|
|
|
| Turkish is an **agglutinative language** with a rich morphological structure. Words are constructed by attaching multiple suffixes to a root (e.g., *kar-go-lar-ı-mız-dan*).
|
|
|
| Standard English-centric tokenizers (like GPT-2 or LLaMA) do not have these Turkish roots/suffixes in their pre-trained vocabularies. As a result, they fragment basic Turkish words into tiny, meaningless character groups. This leads to:
|
| * **High Token Counts**: Turkish texts take up to 2.5x to 3x more tokens than English counterparts.
|
| * **Context Window Waste**: Models hit their context limits much faster.
|
| * **Poorer Semantic Learning**: The model spends capacity learning character-level combinations instead of word meanings.
|
|
|
| This custom tokenizer solves these issues by learning a vocabulary derived directly from real Turkish customer service dialogues.
|
|
|
| ---
|
|
|
| ## 2. Tokenization Performance Benchmark
|
|
|
| Here is a comparison of how different tokenizers split the sample Turkish e-commerce query:
|
| **`"kargom teslim edilmedi iade istiyorum"`** *(my package was not delivered, I want a return)*
|
|
|
| | Tokenizer | Tokenized Representation | Token Count | Efficiency Gain |
|
| | :--- | :--- | :---: | :---: |
|
| | **GPT-2 (Standard)** | `['k', 'arg', 'om', ' t', 'es', 'lim', ' ed', 'il', 'medi', ' i', 'ade', ' is', 't', 'iy', 'orum']` | **15** | Baseline |
|
| | **Our Custom Tokenizer** | `['kargom', ' teslim', ' edil', 'medi', ' iade', ' istiyorum']` | **6** | **2.5x Fewer Tokens (60% Savings)** |
|
|
|
| ---
|
|
|
| ## 3. Specifications
|
|
|
| * **Vocabulary Size**: 8192 ($2^{13}$ tokens)
|
| * **Algorithm**: Byte-Level BPE (`ByteLevelBPETokenizer`)
|
| * **Base Training Corpus**: 558 lines of Turkish e-commerce customer support dialogues (`Mer1Alii/TR-ECommerce-CustomerSupport-Instructions`)
|
| * **Special Tokens Map**:
|
| * `<s>`: Beginning of Sequence (BOS)
|
| - `<pad>`: Padding (PAD)
|
| - `</s>`: End of Sequence (EOS)
|
| - `<unk>`: Unknown token (UNK)
|
| - `<mask>`: Masking token (MASK)
|
|
|
| ---
|
|
|
| ## 5. Quick Start (Usage)
|
|
|
| You can load and use this tokenizer directly in Python using the Hugging Face `transformers` library:
|
|
|
| ```python
|
| from transformers import AutoTokenizer
|
|
|
| # Load custom tokenizer
|
| tokenizer = AutoTokenizer.from_pretrained("Mer1Alii/TR-ECommerce-CustomerSupport-Tokenizer")
|
|
|
| # Test Sentence
|
| text = "kargom teslim edilmedi iade istiyorum"
|
| tokens = tokenizer.encode(text)
|
|
|
| print("Token IDs:", tokens)
|
| print("Decoded Tokens:", tokenizer.convert_ids_to_tokens(tokens))
|
| ```
|
|
|
| ## Developer
|
| **[Mert Ali Alkan](https://github.com/MertAlii)**
|
|
|
