Upload folder using huggingface_hub

README.md

@@ -0,0 +1,43 @@
+---
+license: mit
+language:
+- fas
+tags:
+- tokenizer
+- unigram
+- flexitok
+- fineweb2
+---
+
+# UnigramLM Tokenizer: fas_Arab (64K)
+
+A **UnigramLM** tokenizer trained on **fas_Arab** data from Fineweb-2-HQ.
+
+## Training Details
+
+| Parameter | Value |
+|-----------|-------|
+| Algorithm | UnigramLM |
+| Language | `fas_Arab` |
+| Target Vocab Size | 64,000 |
+| Final Vocab Size | 0 |
+| Pre-tokenizer | ByteLevel |
+| Normalizer | NFC |
+| Special Tokens | `<s>`, `</s>`, `<pad>`, `<unk>` |
+| Training Shards | 2 |
+| Data Source | `/scratch/gsa/data/flexitok//fas_Arab/` |
+
+## Usage
+
+```python
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("<repo_id>")
+tokens = tokenizer.encode("Hello, world!")
+```
+
+## Files
+
+- `tokenizer.json` — Full HuggingFace tokenizer
+- `vocab.json` — Vocabulary mapping
+- `tokenizer.model` — SentencePiece protobuf (if available)