You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Tigre SentencePiece Tokenizer (16K Vocab)

A standalone SentencePiece (Unigram/BPE) subword tokenizer built and optimized for the Tigre language (ትግሬ) written in Ge'ez script.

Tokenizer Statistics & Verification

Metric	Value	Status
Vocabulary Size	`16,384`	✅ Standard
Round-Trip Integrity	`100% Lossless`	✅ Verified
Unknown Token (`<unk>`) Rate	`0.00%`	✅ Verified
Punctuation Isolation	`Clean`	✅ Isolated
Ge'ez Wordspace (`፡`) Handling	`Atomic Token`	✅ Verified
Base Architecture	`SentencePiece`	✅ Native

Quickstart Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("BeitTigreAI/tigre-spm-tokenizer")

text = "ሰላም፡ እሊ ናይ ሂጋ ትግሬ ክታበት ቱ።"

# Encode to tokens & IDs
tokens = tokenizer.tokenize(text)
input_ids = tokenizer.encode(text)

print("Tokens:", tokens)
print("IDs   :", input_ids)

# Lossless Decoding
decoded_text = tokenizer.decode(input_ids, clean_up_tokenization_spaces=False)
print("Decoded:", decoded_text)
assert text == decoded_text

Repository Structure

tokenizer.model: Native SentencePiece binary model file.
tokenizer.json: Serialized fast tokenizer representation for Python/Rust environments.
tokenizer_config.json: Metadata and special token configuration mapping.
special_tokens_map.json: Explicit <pad>, <s>, </s>, and <unk> assignments.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support