README.md

---
language:
- tig
license: apache-2.0
tags:
- tokenizer
- sentencepiece
- ge'ez
- tigre
- beit-tigre-ai
library_name: transformers
---

# Tigre SentencePiece Tokenizer (16K Vocab)

A standalone **SentencePiece (Unigram/BPE)** subword tokenizer built and optimized for the **Tigre language (ትግሬ)** written in Ge'ez script.

## Tokenizer Statistics & Verification

| Metric | Value | Status |
| :--- | :--- | :--- |
| **Vocabulary Size** | `16,384` | ✅ Standard |
| **Round-Trip Integrity** | `100% Lossless` | ✅ Verified |
| **Unknown Token (`<unk>`) Rate** | `0.00%` | ✅ Verified |
| **Punctuation Isolation** | `Clean` | ✅ Isolated |
| **Ge'ez Wordspace (`፡`) Handling** | `Atomic Token` | ✅ Verified |
| **Base Architecture** | `SentencePiece` | ✅ Native |

## Quickstart Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("BeitTigreAI/tigre-spm-tokenizer")

text = "ሰላም፡ እሊ ናይ ሂጋ ትግሬ ክታበት ቱ።"

# Encode to tokens & IDs
tokens = tokenizer.tokenize(text)
input_ids = tokenizer.encode(text)

print("Tokens:", tokens)
print("IDs   :", input_ids)

# Lossless Decoding
decoded_text = tokenizer.decode(input_ids, clean_up_tokenization_spaces=False)
print("Decoded:", decoded_text)
assert text == decoded_text
```

## Repository Structure

- `tokenizer.model`: Native SentencePiece binary model file.
- `tokenizer.json`: Serialized fast tokenizer representation for Python/Rust environments.
- `tokenizer_config.json`: Metadata and special token configuration mapping.
- `special_tokens_map.json`: Explicit `<pad>`, `<s>`, `</s>`, and `<unk>` assignments.