README.md

metadata

language:
  - tig
license: apache-2.0
tags:
  - tokenizer
  - sentencepiece
  - ge'ez
  - tigre
  - beit-tigre-ai
library_name: transformers

Tigre SentencePiece Tokenizer (16K Vocab)

A standalone SentencePiece (Unigram/BPE) subword tokenizer built and optimized for the Tigre language (ትግሬ) written in Ge'ez script.

Tokenizer Statistics & Verification

Metric	Value	Status
Vocabulary Size	16,384	✅ Standard
Round-Trip Integrity	100% Lossless	✅ Verified
Unknown Token (<unk>) Rate	0.00%	✅ Verified
Punctuation Isolation	Clean	✅ Isolated
Ge'ez Wordspace (፡) Handling	Atomic Token	✅ Verified
Base Architecture	SentencePiece	✅ Native

Quickstart Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("BeitTigreAI/tigre-spm-tokenizer")

text = "ሰላም፡ እሊ ናይ ሂጋ ትግሬ ክታበት ቱ።"

# Encode to tokens & IDs
tokens = tokenizer.tokenize(text)
input_ids = tokenizer.encode(text)

print("Tokens:", tokens)
print("IDs   :", input_ids)

# Lossless Decoding
decoded_text = tokenizer.decode(input_ids, clean_up_tokenization_spaces=False)
print("Decoded:", decoded_text)
assert text == decoded_text

Repository Structure

• tokenizer.model: Native SentencePiece binary model file.
• tokenizer.json: Serialized fast tokenizer representation for Python/Rust environments.
• tokenizer_config.json: Metadata and special token configuration mapping.
• special_tokens_map.json: Explicit <pad>, <s>, </s>, and <unk> assignments.