flexitok/bpe_ltr_fas_Arab_32000_v2

Byte-Level BPE Tokenizer: fas_Arab (32K)

A Byte-Level BPE tokenizer trained on fas_Arab data from Fineweb-2-HQ.

Training Details

Parameter	Value
Algorithm	Byte-Level BPE
Language	fas_Arab
Target Vocab Size	32,000
Final Vocab Size	32,808
Pre-tokenizer	custom:fas_Arab
Number handling	ltr_3digit
Contraction handling	True
Normalizer	NONE
Special Tokens	<s>, </s>, <pad>, <unk>
Training Shards	2

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("flexitok/bpe_ltr_fas_Arab_32000_v2")
tokens = tokenizer.encode("Hello, world!")

Files

• tokenizer.json — Full HuggingFace tokenizer
• vocab.json — Vocabulary mapping
• merges.txt — BPE merge rules

Sample Encoding

Text	Tokens	Token IDs
Hello, world! 12345 This is a test. こんにちは	H, ello, ,, Ġ, world, !, Ġ, 123, 45, Ġ, This, Ġ, is, Ġ, a, Ġ, test, ., Ġ, ãģ	42, 25612, 14, 223, 22224, 3, 223, 18213, 5459, 223, 15727, 223, 1809, 223, 67, 223, 21816, 16, 223, 18685