Update README for v1.1 release

README.md

@@ -4,32 +4,48 @@ tags:
 - tokenizer
 - morphology
 - nlp
+- dialect
 license: apache-2.0
 language:
 - ar
 datasets:
 - dataflare/arabic-dialect-corpus
+- fr3on/egyptian-dialogue
+- fr3on/egyptian-songs
+- fr3on/arabic-feedback-corpus
 ---
 
-# DF-Arc: Morphology-Aware Arabic Tokenizer
+# DF-Arc v1.1: Morphology-Aware Arabic Tokenizer
 
-DF-Arc is a specialized tokenizer for Arabic LLMs that achieves **1.0 fertility** (one token per word) on average, eliminating the "Arabic Token Tax".
+DF-Arc is a specialized tokenizer for Arabic LLMs that minimizes the "Arabic Token Tax". By combining **Morphological Pre-tokenization** with **PMI-based Phrase Merging**, it achieves near 1:1 fertility (0.83 fertility on dialects), preserving semantic coherence better than GPT-4 or standard BERT tokenizers.
 
-## Features
-- **Morphological Pre-tokenization**: Splits words into prefix-stem-suffix units.
-- **Phrase Merging**: Automatically merges common multi-word expressions (e.g., "in the name of God") into single tokens.
-- **Dialect Support**: Optimized for Egyptian, Gulf, and Levantine dialects.
+## New in v1.1
+- **PMI-Powered Phrase Merging**: Learning phrases based on statistical coupling (Pointwise Mutual Information) rather than just frequency.
+- **Embedded Protections**: Built-in protection for sensitive entities (e.g., "Allah", "Mohamed") and common trademarks without external files.
+- **Enhanced Dialect Support**: Trained on a broader corpus including Egyptian dialogue, songs, and feedback datasets.
+- **Self-Contained**: No extra config files needed; just load and go.
+
+## Performance
+| Model | Fertility (lower is better) | Efficiency vs GPT-4 |
+|-------|-----------------------------|---------------------|
+| **DF-Arc v1.1** | **0.83** | **+77.6%** |
+| GPT-4 (cl100k) | 3.69 | Baseline |
+| AraBERT v2 | 1.56 | - |
 
 ## Usage
 
 ```python
 from transformers import AutoTokenizer
 
+# trust_remote_code=True is required for custom logic
 tokenizer = AutoTokenizer.from_pretrained("dataflare/df-arc", trust_remote_code=True)
 
-text = "الكتابة بالعربية ممتعة جدا"
+# Example: Dialectal + MSA
+text = "بسم الله الرحمن الرحيم، انا بحب الذكاء الاصطناعي جدا"
 tokens = tokenizer.tokenize(text)
 print(tokens)
+# Output: ['ب_سم', 'الله', 'ال_رحمن', 'ال_رحيم', '،', 'انا', 'ب_حب', 'ال_ذكاء_ال_اصطناع_ي', 'جدا']
+# Note "الله" preserved, phrases like "بسم الله" handled naturally.
 ```
 
 ## Citation