Upload all models and assets for ar (latest)

README.md

@@ -36,727 +36,195 @@ metrics:
     value: 4.347
   - name: best_isotropy
     type: isotropy
-    value: 0.7394
+    value: 0.8111
+  - name: best_alignment_r10
+    type: alignment
+    value: 0.7660
   - name: vocabulary_size
     type: vocab
-    value: 0
-generated: 2026-01-07
+    value: 986324
+generated: 2026-03-04
 ---
 
-# Arabic - Wikilangs Models
-## Comprehensive Research Report & Full Ablation Study
+# Arabic — Wikilangs Models
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Arabic** Wikipedia data.
-We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
+Open-source tokenizers, n-gram & Markov language models, vocabulary stats, and word embeddings trained on **Arabic** Wikipedia by [Wikilangs](https://wikilangs.org).
 
-## 📋 Repository Contents
+🌐 [Language Page](https://wikilangs.org/languages/ar/) · 🎮 [Playground](https://wikilangs.org/playground/?lang=ar) · 📊 [Full Research Report](RESEARCH_REPORT.md)
 
-### Models & Assets
+## Language Samples
 
-- Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4, 5-gram)
-- Markov chains (context of 1, 2, 3, 4 and 5)
-- Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions (aligned and unaligned)
-- Language Vocabulary
-- Language Statistics
+Example sentences drawn from the Arabic Wikipedia corpus:
 
-![Performance Dashboard](visualizations/performance_dashboard.png)
+> تصغير K \ كي \ هو الحرف الحادي العشر في الأبجدية The Oxford English Dictionary, 2nd ed., online ويمثل هذا الحرف الصوت الطبقي الوقفي المهموس في الكيمياء، يرمز K لعنصر البوتاسيوم مراجع لاتينية
 
-### Analysis and Evaluation
+> : إحدى ولايات الولايات المتحدة الأمريكية. مدينة نيويورك: أكبر مدن الولايات المتحدة الأمريكية وإحدى أكبرها في العالم. مقاطعة نيويورك: إحدى مقاطعات ولاية نيويورك. توضيح أسماء أماكن
 
-- [1. Tokenizer Evaluation](#1-tokenizer-evaluation)
-- [2. N-gram Model Evaluation](#2-n-gram-model-evaluation)
-- [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
-- [4. Vocabulary Analysis](#4-vocabulary-analysis)
-- [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
-- [7. Summary & Recommendations](#7-summary--recommendations)
-- [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
-- [Visualizations Index](#visualizations-index)
+> أبو إبراهيم الفارابي أديب نحوي لغوي أبو نصر محمد الفارابي فيلسوف مشائي مسلم وطبيب
 
----
-## 1. Tokenizer Evaluation
+> إسحاق نيوتن عالم إنجليزي نيوتن وحدة قياس القوة. ذكور إنجليزية توضيح أسماء أماكن
 
-![Tokenizer Compression](visualizations/tokenizer_compression.png)
+> بوتان (مملكة) بوتان مملكة في جبال الهمالايا بين الهند والصين. بوتان (كيمياء) أحد الألكانات، يتكون من أربع ذرات كربون.
 
-![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+## Quick Start
 
-![Tokenizer OOV](visualizations/tokenizer_oov.png)
+### Load the Tokenizer
 
-![Total Tokens](visualizations/tokenizer_total_tokens.png)
+```python
+import sentencepiece as spm
 
-### Results
+sp = spm.SentencePieceProcessor()
+sp.Load("ar_tokenizer_32k.model")
 
-| Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
-|------------|-------------|---------------|----------|--------------|
-| **8k** | 3.252x | 3.25 | 0.0704% | 5,499,500 |
-| **16k** | 3.655x | 3.65 | 0.0791% | 4,893,689 |
-| **32k** | 4.034x | 4.03 | 0.0873% | 4,433,903 |
-| **64k** | 4.347x 🏆 | 4.35 | 0.0941% | 4,114,555 |
+text = "استوديوهات أفلام والت ديزني أفلام والت ديزني منتجع والت ديزني العالمي ديزني لاند"
+tokens = sp.EncodeAsPieces(text)
+ids    = sp.EncodeAsIds(text)
 
-### Tokenization Examples
+print(tokens)  # subword pieces
+print(ids)     # integer ids
 
-Below are sample sentences tokenized with each vocabulary size:
+# Decode back
+print(sp.DecodeIds(ids))
+```
 
-**Sample 1:** `بيغجة خاتون هي قرية في مقاطعة شبستر، إيران. يقدر عدد سكانها بـ 635 نسمة بحسب إحص...`
+<details>
+<summary><b>Tokenization examples (click to expand)</b></summary>
+
+**Sample 1:** `استوديوهات أفلام والت ديزني أفلام والت ديزني منتجع والت ديزني العالمي ديزني لاند…`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁بي غ جة ▁خ ات ون ▁هي ▁قرية ▁في ▁مقاطعة ... (+26 more)` | 36 |
-| 16k | `▁بي غ جة ▁خ ات ون ▁هي ▁قرية ▁في ▁مق��طعة ... (+23 more)` | 33 |
-| 32k | `▁بيغ جة ▁خاتون ▁هي ▁قرية ▁في ▁مقاطعة ▁شب ستر ، ... (+20 more)` | 30 |
-| 64k | `▁بيغ جة ▁خاتون ▁هي ▁قرية ▁في ▁مقاطعة ▁شب ستر ، ... (+20 more)` | 30 |
+| 8k | `▁است ودي وه ات ▁أفلام ▁والت ▁دي ز ني ▁أفلام … (+22 more)` | 32 |
+| 16k | `▁است ودي وهات ▁أفلام ▁والت ▁ديزني ▁أفلام ▁والت ▁ديزني ▁منت … (+10 more)` | 20 |
+| 32k | `▁استوديوهات ▁أفلام ▁والت ▁ديزني ▁أفلام ▁والت ▁ديزني ▁منتجع ▁والت ▁ديزني … (+7 more)` | 17 |
+| 64k | `▁استوديوهات ▁أفلام ▁والت ▁ديزني ▁أفلام ▁والت ▁ديزني ▁منتجع ▁والت ▁ديزني … (+7 more)` | 17 |
 
-**Sample 2:** `IL18BP (Interleukin 18 binding protein) هوَ بروتين يُشَفر بواسطة جين IL18BP في ا...`
+**Sample 2:** `باسكال قد تعني: الباسكال، وحدة قياس الضغط لغة باسكال، لغة برمجة الفيلسوف باسكال،…`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ il 1 8 b p ▁( in ter le ... (+51 more)` | 61 |
-| 16k | `▁il 1 8 b p ▁( in ter le uk ... (+44 more)` | 54 |
-| 32k | `▁il 1 8 b p ▁( inter le uk in ... (+39 more)` | 49 |
-| 64k | `▁il 1 8 b p ▁( inter le uk in ... (+36 more)` | 46 |
+| 8k | `▁با سك ال ▁قد ▁تعني : ▁البا سك ال ، … (+29 more)` | 39 |
+| 16k | `▁باسكال ▁قد ▁تعني : ▁الباسك ال ، ▁وحدة ▁قياس ▁الضغط … (+18 more)` | 28 |
+| 32k | `▁باسكال ▁قد ▁تعني : ▁الباسك ال ، ▁وحدة ▁قياس ▁الضغط … (+15 more)` | 25 |
+| 64k | `▁باسكال ▁قد ▁تعني : ▁الباسك ال ، ▁وحدة ▁قياس ▁الضغط … (+15 more)` | 25 |
 
-**Sample 3:** `هي مقاطعة في ولاية قشقداريا في أوزبكستان، ومركزها مدينة شهرسبز. المصادر مأهولة ف...`
+**Sample 3:** `جمهورية الكونغو الديمقراطية، زائير سابقًا، عاصمتها كينشاسا. جمهورية الكونغو، عاص…`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁هي ▁مقاطعة ▁في ▁ولاية ▁ق ش قد اريا ▁في ▁أوزب ... (+18 more)` | 28 |
-| 16k | `▁هي ▁مقاطعة ▁في ▁ولاية ▁ق ش قد اريا ▁في ▁أوزبكستان ... (+16 more)` | 26 |
-| 32k | `▁هي ▁مقاطعة ▁في ▁ولاية ▁قش قد اريا ▁في ▁أوزبكستان ، ... (+13 more)` | 23 |
-| 64k | `▁هي ▁مقاطعة ▁في ▁ولاية ▁قش قد اريا ▁في ▁أوزبكستان ، ... (+13 more)` | 23 |
-
-
-### Key Findings
-
-- **Best Compression:** 64k achieves 4.347x compression
-- **Lowest UNK Rate:** 8k with 0.0704% unknown tokens
-- **Trade-off:** Larger vocabularies improve compression but increase model size
-- **Recommendation:** 32k vocabulary provides optimal balance for production use
-
----
-## 2. N-gram Model Evaluation
-
-![N-gram Perplexity](visualizations/ngram_perplexity.png)
-
-![N-gram Unique](visualizations/ngram_unique.png)
-
-![N-gram Coverage](visualizations/ngram_coverage.png)
-
-### Results
-
-| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|---------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | Word | 452,226 | 18.79 | 5,760,373 | 5.7% | 16.3% |
-| **2-gram** | Subword | 436 🏆 | 8.77 | 70,700 | 55.9% | 96.1% |
-| **3-gram** | Word | 1,074,568 | 20.04 | 10,101,258 | 4.3% | 14.7% |
-| **3-gram** | Subword | 4,203 | 12.04 | 528,264 | 23.7% | 56.2% |
-| **4-gram** | Word | 1,869,871 | 20.83 | 16,693,684 | 3.8% | 14.3% |
-| **4-gram** | Subword | 26,613 | 14.70 | 2,851,427 | 13.2% | 31.9% |
-| **5-gram** | Word | 1,422,629 | 20.44 | 12,591,346 | 4.2% | 15.4% |
-| **5-gram** | Subword | 126,300 | 16.95 | 9,618,770 | 6.2% | 19.5% |
-
-### Top 5 N-grams by Size
-
-**2-grams (Word):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `كرة قدم` | 754,062 |
-| 2 | `في القرن` | 693,987 |
-| 3 | `في عام` | 580,274 |
-| 4 | `الولايات المتحدة` | 468,192 |
-| 5 | `وصلات خارجية` | 357,388 |
-
-**3-grams (Word):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `في القرن 20` | 274,915 |
-| 2 | `مراجع وصلات خارجية` | 255,117 |
-| 3 | `في الولايات المتحدة` | 245,241 |
-| 4 | `في القرن 21` | 238,844 |
-| 5 | `أمريكيون في القرن` | 166,269 |
-
-**4-grams (Word):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `كرة قدم مغتربون في` | 94,639 |
-| 2 | `تحت سن الثامنة عشر` | 93,897 |
-| 3 | `هو لاعب كرة قدم` | 93,478 |
-| 4 | `أمريكيون في ��لقرن 20` | 87,276 |
-| 5 | `في الألعاب الأولمبية الصيفية` | 66,167 |
-
-**5-grams (Word):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `تعداد عام بلغ عدد سكان` | 38,914 |
-| 2 | `بحسب تعداد عام وبلغ عدد` | 38,787 |
-| 3 | `تعداد عام وبلغ عدد الأسر` | 38,786 |
-| 4 | `نسمة بحسب تعداد عام وبلغ` | 38,783 |
-| 5 | `في الفئة العمرية ما بين` | 38,744 |
-
-**2-grams (Subword):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `ا ل` | 88,022,277 |
-| 2 | `_ ا` | 75,496,816 |
-| 3 | `ة _` | 45,404,729 |
-| 4 | `ي _` | 32,155,198 |
-| 5 | `ن _` | 31,357,117 |
-
-**3-grams (Subword):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `_ ا ل` | 71,328,243 |
-| 2 | `_ ف ي` | 15,404,541 |
-| 3 | `ف ي _` | 15,103,296 |
-| 4 | `ي ة _` | 14,752,185 |
-| 5 | `ا ل م` | 13,544,149 |
-
-**4-grams (Subword):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `_ ف ي _` | 14,189,454 |
-| 2 | `ة _ ا ل` | 12,269,528 |
-| 3 | `_ ا ل م` | 11,772,138 |
-| 4 | `_ م ن _` | 8,237,350 |
-| 5 | `ي _ ا ل` | 7,703,248 |
-
-**5-grams (Subword):**
-
-| Rank | N-gram | Count |
-|------|--------|-------|
-| 1 | `ف ي _ ا ل` | 4,810,645 |
-| 2 | `_ ف ي _ ا` | 4,774,417 |
-| 3 | `ا ت _ ا ل` | 3,857,996 |
-| 4 | `ي ة _ ا ل` | 3,696,976 |
-| 5 | `_ ع ل ى _` | 3,259,756 |
-
-
-### Key Findings
-
-- **Best Perplexity:** 2-gram (subword) with 436
-- **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~19% of corpus
-- **Recommendation:** 4-gram or 5-gram for best predictive performance
-
----
-## 3. Markov Chain Evaluation
-
-![Markov Entropy](visualizations/markov_entropy.png)
-
-![Markov Contexts](visualizations/markov_contexts.png)
-
-![Markov Branching](visualizations/markov_branching.png)
-
-### Results
-
-| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.9908 | 1.987 | 17.58 | 4,471,621 | 0.9% |
-| **1** | Subword | 1.3702 | 2.585 | 13.33 | 18,570 | 0.0% |
-| **2** | Word | 0.3659 | 1.289 | 2.31 | 78,540,786 | 63.4% |
-| **2** | Subword | 0.7295 | 1.658 | 5.21 | 247,596 | 27.1% |
-| **3** | Word | 0.1310 | 1.095 | 1.29 | 181,002,468 | 86.9% |
-| **3** | Subword | 0.6782 | 1.600 | 4.14 | 1,290,623 | 32.2% |
-| **4** | Word | 0.0499 🏆 | 1.035 | 1.09 | 233,679,791 | 95.0% |
-| **4** | Subword | 0.6490 | 1.568 | 3.51 | 5,343,485 | 35.1% |
-
-### Generated Text Samples (Word-based)
-
-Below are text samples generated from each word-based Markov chain model:
-
-**Context Size 1:**
-
-1. `في المدائن وهي منتزه نيقولا الصايغ أميناً عاماً ونسبة 22 مايو حين سجلت في مجال تعليم`
-2. `من مونتريال اسمه إلى الساحل في الإصدار الرابع قبل الرابطة مع نادي ثون نادي سيون ببطولة`
-3. `على الصيد فلا يطالب بتنفيذها أو وجود منافسة ألعاب البحر في حين احتفظت بهويتها الجديدة بقيمة`
-
-**Context Size 2:**
-
-1. `كرة قدم من قصرش مقاطعة إسبان من كتالونيا إسبانيات في القرن 20 استمر التعليم التطوري أو التنموي`
-2. `في القرن 11 في وقتٍ واحد غابرييلا قرنفل وقرفة ترجمة عوض أحمد بن عبد الله الأميرة منيرة`
-3. `في عام أن تكلفة الوجبة البسيطة في نسج الظهارية ثخانة الجلد وتصلبه المترافقين مع المشكلات التي تنشأ`
-
-**Context Size 3:**
-
-1. `في القرن 20 أمريكيون أفارقة في القرن 21 كرة قدم رجالية أحياء دوري الدرجة الأولى الأرجنتيني فيليز سار...`
-2. `مراجع وصلات خارجية كرة قدم رجالية مغتربون في روسيا على أنها قوة بحرية صغيرة إلى مدينة تشهد حركة`
-3. `في الولايات المتحدة مراجع وصلات خارجية تلفزيونية مصرية بدأ عرضها في كوميديا سوداء تلفزيونية بريطانية...`
-
-**Context Size 4:**
-
-1. `كرة قدم مغتربون في السلفادور كرة قدم هندوراسيون كرة قدم هندوراسيون مغتربون كوبا سينتروأمريكانا منتخب...`
-2. `تحت سن الثامنة عشر تعيش معهم وبلغت نسبة الأزواج القاطنين مع بعضهم البعض 46 3 من أصل المجموع الكلي`
-3. `هو لاعب كرة قدم بريطاني في مركز لعب مع برادفورد سيتي وريث روفرز ونادي بارتيك ثيسل ونادي رينجرز ونادي`
-
-
-### Generated Text Samples (Subword-based)
-
-Below are text samples generated from each subword-based Markov chain model:
-
-**Context Size 1:**
-
-1. `_فيا،_دارب_ي_أمر`
-2. `اقصالمعب_ع_حمالم`
-3. `لبطة_قالمندواب_ا`
-
-**Context Size 2:**
-
-1. `الأخرها_تشت_علية_`
-2. `_الممثل_أصدققه_حا`
-3. `ة_لدعار_الة)_جوزي`
-
-**Context Size 3:**
+| 8k | `▁جمهورية ▁الكون غو ▁الديمقراطية ، ▁ز ائ ير ▁سابق ًا … (+21 more)` | 31 |
+| 16k | `▁جمهورية ▁الكونغو ▁الديمقراطية ، ▁ز ائ ير ▁سابقًا ، ▁عاصمتها … (+16 more)` | 26 |
+| 32k | `▁جمهورية ▁الكونغو ▁الديمقراطية ، ▁زائ ير ▁سابقًا ، ▁عاصمتها ▁كينشاسا … (+12 more)` | 22 |
+| 64k | `▁جمهورية ▁الكونغو ▁الديمقراطية ، ▁زائير ▁سابقًا ، ▁عاصمتها ▁كينشاسا . … (+10 more)` | 20 |
 
-1. `_الذين_حليلار_رُزِق_`
-2. `_في_إحصاءات_الله)،`
-3. `في_الوالصحيحًا_كرة_`
+</details>
 
-**Context Size 4:**
+### Load Word Embeddings
 
-1. `_في_جمهور._جسدت_ديك`
-2. `ة_البلدي_في_اخترعه_`
-3. `_المتحدة._يقدمه_في_`
+```python
+from gensim.models import KeyedVectors
 
+# Aligned embeddings (cross-lingual, mapped to English vector space)
+wv = KeyedVectors.load("ar_embeddings_128d_aligned.kv")
 
-### Key Findings
-
-- **Best Predictability:** Context-4 (word) with 95.0% predictability
-- **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (5,343,485 contexts)
-- **Recommendation:** Context-3 or Context-4 for text generation
-
----
-## 4. Vocabulary Analysis
-
-![Zipf's Law](visualizations/zipf_law.png)
-
-![Top Words](visualizations/top20_words.png)
-
-![Coverage Curve](visualizations/vocab_coverage.png)
-
-### Statistics
-
-| Metric | Value |
-|--------|-------|
-| Vocabulary Size | 1,950,572 |
-| Total Tokens | 322,254,287 |
-| Mean Frequency | 165.21 |
-| Median Frequency | 4 |
-| Frequency Std Dev | 12979.56 |
-
-### Most Common Words
-
-| Rank | Word | Frequency |
-|------|------|-----------|
-| 1 | في | 14,286,084 |
-| 2 | من | 8,287,878 |
-| 3 | على | 3,284,746 |
-| 4 | إلى | 2,443,493 |
-| 5 | عام | 1,621,280 |
-| 6 | أن | 1,387,527 |
-| 7 | مع | 1,153,439 |
-| 8 | عن | 1,144,208 |
-| 9 | أو | 1,098,905 |
-| 10 | التي | 1,084,821 |
-
-### Least Common Words (from vocabulary)
-
-| Rank | Word | Frequency |
-|------|------|-----------|
-| 1 | dekréty | 2 |
-| 2 | تادينا | 2 |
-| 3 | بوكسوري | 2 |
-| 4 | نموذجاالأدب | 2 |
-| 5 | كنونالأدب | 2 |
-| 6 | وليتاز | 2 |
-| 7 | حكمٌّ | 2 |
-| 8 | أسديراكي | 2 |
-| 9 | إنتركوليجيت | 2 |
-| 10 | للفيزيولوجية | 2 |
-
-### Zipf's Law Analysis
-
-| Metric | Value |
-|--------|-------|
-| Zipf Coefficient | 0.9488 |
-| R² (Goodness of Fit) | 0.991144 |
-| Adherence Quality | **excellent** |
-
-### Coverage Analysis
-
-| Top N Words | Coverage |
-|-------------|----------|
-| Top 100 | 23.1% |
-| Top 1,000 | 45.9% |
-| Top 5,000 | 66.1% |
-| Top 10,000 | 74.2% |
-
-### Key Findings
-
-- **Zipf Compliance:** R²=0.9911 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 23.1% of corpus
-- **Long Tail:** 1,940,572 words needed for remaining 25.8% coverage
-
----
-## 5. Word Embeddings Evaluation
-
-![Embedding Isotropy](visualizations/embedding_isotropy.png)
-
-![Similarity Matrix](visualizations/embedding_similarity.png)
-
-![t-SNE Words](visualizations/tsne_words.png)
-
-![t-SNE Sentences](visualizations/tsne_sentences.png)
-
-
-### 5.1 Cross-Lingual Alignment
-
-![Alignment Quality](visualizations/embedding_alignment_quality.png)
-
-![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
-
-
-### 5.2 Model Comparison
-
-| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
-|-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.7379 | 0.3519 | N/A | N/A |
-| **mono_64d** | 64 | 0.7394 🏆 | 0.2816 | N/A | N/A |
-| **mono_128d** | 128 | 0.7002 | 0.2259 | N/A | N/A |
-| **aligned_32d** | 32 | 0.7379 | 0.3528 | 0.2700 | 0.6440 |
-| **aligned_64d** | 64 | 0.7394 | 0.2881 | 0.4140 | 0.8200 |
-| **aligned_128d** | 128 | 0.7002 | 0.2283 | 0.6000 | 0.8940 |
+similar = wv.most_similar("word", topn=5)
+for word, score in similar:
+    print(f"  {word}: {score:.3f}")
+```
 
-### Key Findings
+### Load N-gram Model
 
-- **Best Isotropy:** mono_64d with 0.7394 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.2881. Lower values indicate better semantic separation.
-- **Alignment Quality:** Aligned models achieve up to 60.0% R@1 in cross-lingual retrieval.
-- **Recommendation:** 128d aligned for best cross-lingual performance
+```python
+import pyarrow.parquet as pq
 
----
-## 6.  Morphological Analysis (Experimental)
-
-This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
-
-### 6.1 Productivity & Complexity
-
-| Metric | Value | Interpretation | Recommendation |
-|--------|-------|----------------|----------------|
-| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
-| Idiomaticity Gap | **-0.210** | Low formulaic content | - |
-
-### 6.2 Affix Inventory (Productive Units)
-
-These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
-
-#### Productive Prefixes
-| Prefix | Examples |
-|--------|----------|
-| `-ال` | الألمانينصف, الاعتياديّ, الباكترية |
-| `-وا` | والشجرية, والكاحِل, والميلانين |
-| `-وال` | والشجرية, والكاحِل, والميلانين |
-| `-الم` | المُحاضرة, المورينو, الممنوعة |
-
-#### Productive Suffixes
-| Suffix | Examples |
-|--------|----------|
-| `-ين` | ضوئيتين, بقلبين, نحوين |
-| `-ات` | وخصوصيات, نانديات, دويركات |
-| `-ية` | والشجرية, الباكترية, الّدودية |
-| `-ها` | هاماريتيها, اختها, أُصولها |
-
-### 6.3 Bound Stems (Lexical Roots)
-
-Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
-
-| Stem | Cohesion | Substitutability | Examples |
-|------|----------|------------------|----------|
-| `تخدا` | 2.86x | 173 contexts | متخدا, كتخدا, متخداً |
-| `ستخد` | 2.18x | 623 contexts | مستخد, استخد, تستخد |
-| `ألعا` | 2.68x | 82 contexts | ألعاد, ألعاب, ألعالم |
-| `والع` | 1.74x | 629 contexts | والعز, والعي, والعى |
-| `اطعة` | 3.13x | 28 contexts | قاطعة, ساطعة, ساطعةً |
-| `التع` | 1.63x | 578 contexts | التعة, التعس, التعب |
-| `رنسي` | 1.82x | 179 contexts | درنسي, رنسيس, فرنسي |
-| `استخ` | 1.79x | 192 contexts | استخم, استخد, استخر |
-| `ريطا` | 2.08x | 85 contexts | غريطا, شريطا, وشريطا |
-| `لمنا` | 1.37x | 729 contexts | تلمنا, ظلمنا, ألمنا |
-| `غترب` | 2.44x | 39 contexts | اغترب, مغترب, يغترب |
-| `الحا` | 1.34x | 693 contexts | الحاء, مالحا, الحاص |
-
-### 6.4 Affix Compatibility (Co-occurrence)
-
-This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
-
-| Prefix | Suffix | Frequency | Examples |
-|--------|--------|-----------|----------|
-| `-ال` | `-ية` | 95 words | الائتمانية, الويبرية |
-| `-ال` | `-ات` | 76 words | الهباءات, الكوميديات |
-| `-ال` | `-ين` | 68 words | البحـرين, المتوارثين |
-| `-وا` | `-ية` | 35 words | والعضدية, والهانرية |
-| `-وا` | `-ات` | 24 words | والمطرزات, والسلوريات |
-| `-وا` | `-ين` | 17 words | والمُغنين, والميكرونيزيين |
-| `-وا` | `-ها` | 4 words | واعترضتها, واستبعدتها |
-
-### 6.5 Recursive Morpheme Segmentation
-
-Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
-
-| Word | Suggested Split | Confidence | Stem |
-|------|-----------------|------------|------|
-| البروتينين | **`ال-بروت-ين-ين`** | 7.5 | `بروت` |
-| والكاظمية | **`وال-كاظم-ية`** | 6.0 | `كاظم` |
-| والسرورية | **`وال-سرور-ية`** | 6.0 | `سرور` |
-| الغيلوغية | **`ال-غيلوغ-ية`** | 6.0 | `غيلوغ` |
-| والحطابين | **`وال-حطاب-ين`** | 6.0 | `حطاب` |
-| والمقدسيين | **`وال-مقدسي-ين`** | 6.0 | `مقدسي` |
-| والنجومية | **`وال-نجوم-ية`** | 6.0 | `نجوم` |
-| والرباعيات | **`وال-رباعي-ات`** | 6.0 | `رباعي` |
-| الكلابشات | **`ال-كلابش-ات`** | 6.0 | `كلابش` |
-| السبعينات | **`ال-سبعين-ات`** | 6.0 | `سبعين` |
-| لاحتجاجاتها | **`لاحتجاج-ات-ها`** | 6.0 | `لاحتجاج` |
-| والمكسّرات | **`وال-مكسّر-ات`** | 6.0 | `مكسّر` |
-| والسكيريين | **`وال-سكيري-ين`** | 6.0 | `سكيري` |
-| إسقاطاتها | **`إسقاط-ات-ها`** | 6.0 | `إسقاط` |
-| واستثمارها | **`وا-ستثمار-ها`** | 6.0 | `ستثمار` |
-
-### 6.6 Linguistic Interpretation
-
-> **Automated Insight:**
-The language Arabic shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+df = pq.read_table("ar_3gram_word.parquet").to_pandas()
+print(df.head())
+```
 
----
-## 7. Summary & Recommendations
+## Models Overview
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
-### Production Recommendations
-
-| Component | Recommended | Rationale |
-|-----------|-------------|-----------|
-| Tokenizer | **64k BPE** | Best compression (4.35x) |
-| N-gram | **2-gram** | Lowest perplexity (436) |
-| Markov | **Context-4** | Highest predictability (95.0%) |
-| Embeddings | **100d** | Balanced semantic capture and isotropy |
-
-
----
-## Appendix: Metrics Glossary & Interpretation Guide
-
-This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report.
-
-### Tokenizer Metrics
-
-**Compression Ratio**
-> *Definition:* The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text.
->
-> *Intuition:* Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average.
->
-> *What to seek:* Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information.
-
-**Average Token Length (Fertility)**
-> *Definition:* Mean number of characters per token produced by the tokenizer.
->
-> *Intuition:* Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length.
->
-> *What to seek:* Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens.
-
-**Unknown Token Rate (OOV Rate)**
-> *Definition:* Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent.
->
-> *Intuition:* Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences.
->
-> *What to seek:* Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback.
-
-### N-gram Model Metrics
-
-**Perplexity**
-> *Definition:* Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction.
->
-> *Intuition:* If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options.
->
-> *What to seek:* Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size.
-
-**Entropy**
-> *Definition:* Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy.
->
-> *Intuition:* High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character.
->
-> *What to seek:* Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases.
-
-**Coverage (Top-K)**
-> *Definition:* Percentage of corpus occurrences explained by the top K most frequent n-grams.
->
-> *Intuition:* High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage.
->
-> *What to seek:* Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text.
-
-### Markov Chain Metrics
-
-**Average Entropy**
-> *Definition:* Mean entropy across all contexts, measuring average uncertainty in next-word prediction.
->
-> *Intuition:* Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations).
->
-> *What to seek:* Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions.
-
-**Branching Factor**
-> *Definition:* Average number of unique next tokens observed for each context.
->
-> *Intuition:* High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive).
->
-> *What to seek:* Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains.
-
-**Predictability**
-> *Definition:* Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are.
->
-> *Intuition:* 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes.
->
-> *What to seek:* Higher predictability for text generation quality, but too high (>98%) may produce repetitive output.
-
-### Vocabulary & Zipf's Law Metrics
-
-**Zipf's Coefficient**
-> *Definition:* The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1.
->
-> *Intuition:* A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare.
->
-> *What to seek:* Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text.
-
-**R² (Coefficient of Determination)**
-> *Definition:* Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1.
->
-> *Intuition:* R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns.
->
-> *What to seek:* R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora.
-
-**Vocabulary Coverage**
-> *Definition:* Cumulative percentage of corpus tokens accounted for by the top N words.
->
-> *Intuition:* Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words.
->
-> *What to seek:* Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary.
-
-### Word Embedding Metrics
-
-**Isotropy**
-> *Definition:* Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values.
->
-> *Intuition:* High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness.
->
-> *What to seek:* Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy.
-
-**Average Norm**
-> *Definition:* Mean magnitude (L2 norm) of word vectors in the embedding space.
->
-> *Intuition:* Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained.
->
-> *What to seek:* Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation).
-
-**Cosine Similarity**
-> *Definition:* Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction).
->
-> *Intuition:* Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings.
->
-> *What to seek:* Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7.
-
-**t-SNE Visualization**
-> *Definition:* t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization.
->
-> *Intuition:* Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence.
->
-> *What to seek:* Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure.
-
-### General Interpretation Guidelines
-
-1. **Compare within model families:** Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer).
-2. **Consider trade-offs:** Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate).
-3. **Context matters:** Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification.
-4. **Corpus influence:** All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature.
-5. **Language-specific patterns:** Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages.
-
-
-### Visualizations Index
-
-| Visualization | Description |
-|---------------|-------------|
-| Tokenizer Compression | Compression ratios by vocabulary size |
-| Tokenizer Fertility | Average token length by vocabulary |
-| Tokenizer OOV | Unknown token rates |
-| Tokenizer Total Tokens | Total tokens by vocabulary |
-| N-gram Perplexity | Perplexity by n-gram size |
-| N-gram Entropy | Entropy by n-gram size |
-| N-gram Coverage | Top pattern coverage |
-| N-gram Unique | Unique n-gram counts |
-| Markov Entropy | Entropy by context size |
-| Markov Branching | Branching factor by context |
-| Markov Contexts | Unique context counts |
-| Zipf's Law | Frequency-rank distribution with fit |
-| Vocab Frequency | Word frequency distribution |
-| Top 20 Words | Most frequent words |
-| Vocab Coverage | Cumulative coverage curve |
-| Embedding Isotropy | Vector space uniformity |
-| Embedding Norms | Vector magnitude distribution |
-| Embedding Similarity | Word similarity heatmap |
-| Nearest Neighbors | Similar words for key terms |
-| t-SNE Words | 2D word embedding visualization |
-| t-SNE Sentences | 2D sentence embedding visualization |
-| Position Encoding | Encoding method comparison |
-| Model Sizes | Storage requirements |
-| Performance Dashboard | Comprehensive performance overview |
+| Category | Assets |
+|----------|--------|
+| Tokenizers | BPE at 8k, 16k, 32k, 64k vocab sizes |
+| N-gram models | 2 / 3 / 4 / 5-gram (word & subword) |
+| Markov chains | Context 1–5 (word & subword) |
+| Embeddings | 32d, 64d, 128d — mono & aligned |
+| Vocabulary | Full frequency list + Zipf analysis |
+| Statistics | Corpus & model statistics JSON |
+
+## Metrics Summary
+
+| Component | Model | Key Metric | Value |
+|-----------|-------|------------|-------|
+| Tokenizer | 8k BPE | Compression | 3.25x |
+| Tokenizer | 16k BPE | Compression | 3.65x |
+| Tokenizer | 32k BPE | Compression | 4.03x |
+| Tokenizer | 64k BPE | Compression | 4.35x 🏆 |
+| N-gram | 2-gram (subword) | Perplexity | 426 🏆 |
+| N-gram | 2-gram (word) | Perplexity | 359,826 |
+| N-gram | 3-gram (subword) | Perplexity | 4,163 |
+| N-gram | 3-gram (word) | Perplexity | 775,988 |
+| N-gram | 4-gram (subword) | Perplexity | 27,277 |
+| N-gram | 4-gram (word) | Perplexity | 1,494,234 |
+| N-gram | 5-gram (subword) | Perplexity | 133,736 |
+| N-gram | 5-gram (word) | Perplexity | 1,059,510 |
+| Markov | ctx-1 (subword) | Predictability | 0.0% |
+| Markov | ctx-1 (word) | Predictability | 0.0% |
+| Markov | ctx-2 (subword) | Predictability | 17.3% |
+| Markov | ctx-2 (word) | Predictability | 67.4% |
+| Markov | ctx-3 (subword) | Predictability | 29.5% |
+| Markov | ctx-3 (word) | Predictability | 89.5% |
+| Markov | ctx-4 (subword) | Predictability | 35.2% |
+| Markov | ctx-4 (word) | Predictability | 96.5% 🏆 |
+| Vocabulary | full | Size | 986,324 |
+| Vocabulary | full | Zipf R² | 0.9920 |
+| Embeddings | mono_32d | Isotropy | 0.8111 |
+| Embeddings | mono_64d | Isotropy | 0.7841 |
+| Embeddings | mono_128d | Isotropy | 0.7556 |
+| Embeddings | aligned_32d | Isotropy | 0.8111 🏆 |
+| Embeddings | aligned_64d | Isotropy | 0.7841 |
+| Embeddings | aligned_128d | Isotropy | 0.7556 |
+| Alignment | aligned_32d | R@1 / R@5 / R@10 | 13.4% / 35.0% / 48.6% |
+| Alignment | aligned_64d | R@1 / R@5 / R@10 | 28.6% / 54.0% / 65.6% |
+| Alignment | aligned_128d | R@1 / R@5 / R@10 | 37.2% / 65.0% / 76.6% 🏆 |
+
+📊 **[Full ablation study, per-model breakdowns, and interpretation guide →](RESEARCH_REPORT.md)**
 
 ---
-## About This Project
-
-### Data Source
 
-Models trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) - a monthly snapshot of Wikipedia articles across 300+ languages.
+## About
 
-### Project
+Trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) — monthly snapshots of 300+ Wikipedia languages.
 
-A project by **[Wikilangs](https://wikilangs.org)** - Open-source NLP models for every Wikipedia language.
-
-### Maintainer
-
-[Omar Kamali](https://omarkamali.com) - [Omneity Labs](https://omneitylabs.com)
+A project by **[Wikilangs](https://wikilangs.org)** · Maintainer: [Omar Kamali](https://omarkamali.com) · [Omneity Labs](https://omneitylabs.com)
 
 ### Citation
 
-If you use these models in your research, please cite:
-
 ```bibtex
 @misc{wikilangs2025,
-  author = {Kamali, Omar},
-  title = {Wikilangs: Open NLP Models for Wikipedia Languages},
-  year = {2025},
-  doi = {10.5281/zenodo.18073153},
+  author    = {Kamali, Omar},
+  title     = {Wikilangs: Open NLP Models for Wikipedia Languages},
+  year      = {2025},
+  doi       = {10.5281/zenodo.18073153},
   publisher = {Zenodo},
-  url = {https://huggingface.co/wikilangs}
+  url       = {https://huggingface.co/wikilangs},
   institution = {Omneity Labs}
 }
 ```
 
-### License
-
-MIT License - Free for academic and commercial use.
-
 ### Links
 
-- 🌐 Website: [wikilangs.org](https://wikilangs.org)
-- 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
-- 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
-- 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🌐 [wikilangs.org](https://wikilangs.org)
+- 🌍 [Language page](https://wikilangs.org/languages/ar/)
+- 🎮 [Playground](https://wikilangs.org/playground/?lang=ar)
+- 🤗 [HuggingFace models](https://huggingface.co/wikilangs)
+- 📊 [wikipedia-monthly dataset](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
+- 👤 [Omar Kamali](https://huggingface.co/omarkamali)
 - 🤝 Sponsor: [Featherless AI](https://featherless.ai)
----
-*Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-07 13:14:53*
+**License:** MIT — free for academic and commercial use.
+
+---
+*Generated by Wikilangs Pipeline · 2026-03-04 13:56:39*

RESEARCH_REPORT.md

@@ -0,0 +1,686 @@
+# Arabic — Full Ablation Study & Research Report
+
+Detailed evaluation of all model variants trained on **Arabic** Wikipedia data by [Wikilangs](https://wikilangs.org).
+
+👈 [Back to README](README.md)
+
+## 📋 Repository Contents
+
+### Models & Assets
+
+- Tokenizers (8k, 16k, 32k, 64k)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
+- Subword N-gram and Markov chains
+- Embeddings in various sizes and dimensions (aligned and unaligned)
+- Language Vocabulary
+- Language Statistics
+
+![Performance Dashboard](visualizations/performance_dashboard.png)
+
+### Analysis and Evaluation
+
+- [1. Tokenizer Evaluation](#1-tokenizer-evaluation)
+- [2. N-gram Model Evaluation](#2-n-gram-model-evaluation)
+- [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
+- [4. Vocabulary Analysis](#4-vocabulary-analysis)
+- [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
+- [7. Summary & Recommendations](#7-summary--recommendations)
+- [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
+- [Visualizations Index](#visualizations-index)
+
+---
+## 1. Tokenizer Evaluation
+
+![Tokenizer Compression](visualizations/tokenizer_compression.png)
+
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
+### Results
+
+| Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
+|------------|-------------|---------------|----------|--------------|
+| **8k** | 3.251x | 3.25 | 0.0702% | 5,509,050 |
+| **16k** | 3.654x | 3.65 | 0.0788% | 4,901,830 |
+| **32k** | 4.033x | 4.03 | 0.0870% | 4,440,712 |
+| **64k** | 4.347x 🏆 | 4.35 | 0.0938% | 4,120,770 |
+
+### Tokenization Examples
+
+Below are sample sentences tokenized with each vocabulary size:
+
+**Sample 1:** `استوديوهات أفلام والت ديزني أفلام والت ديزني منتجع والت ديزني العالمي ديزني لاند...`
+
+| Vocab | Tokens | Count |
+|-------|--------|-------|
+| 8k | `▁است ودي وه ات ▁أفلام ▁والت ▁دي ز ني ▁أفلام ... (+22 more)` | 32 |
+| 16k | `▁است ودي وهات ▁أفلام ▁والت ▁ديزني ▁أفلام ▁والت ▁ديزني ▁منت ... (+10 more)` | 20 |
+| 32k | `▁استوديوهات ▁أفلام ▁والت ▁ديزني ▁أفلام ▁والت ▁ديزني ▁منتجع ▁والت ▁ديزني ... (+7 more)` | 17 |
+| 64k | `▁استوديوهات ▁أفلام ▁والت ▁ديزني ▁أفلام ▁والت ▁ديزني ▁منتجع ▁والت ▁ديزني ... (+7 more)` | 17 |
+
+**Sample 2:** `باسكال قد تعني: الباسكال، وحدة قياس الضغط لغة باسكال، لغة برمجة الفيلسوف باسكال،...`
+
+| Vocab | Tokens | Count |
+|-------|--------|-------|
+| 8k | `▁با سك ال ▁قد ▁تعني : ▁البا سك ال ، ... (+29 more)` | 39 |
+| 16k | `▁باسكال ▁قد ▁تعني : ▁الباسك ال ، ▁وحدة ▁قياس ▁الضغط ... (+18 more)` | 28 |
+| 32k | `▁باسكال ▁قد ▁تعني : ▁الباسك ال ، ▁وحدة ▁قياس ▁الضغط ... (+15 more)` | 25 |
+| 64k | `▁باسكال ▁قد ▁تعني : ▁الباسك ال ، ▁وحدة ▁قياس ▁الضغط ... (+15 more)` | 25 |
+
+**Sample 3:** `جمهورية الكونغو الديمقراطية، زائير سابقًا، عاصمتها كينشاسا. جمهورية الكونغو، عاص...`
+
+| Vocab | Tokens | Count |
+|-------|--------|-------|
+| 8k | `▁جمهورية ▁الكون غو ▁الديمقراطية ، ▁ز ائ ير ▁سابق ًا ... (+21 more)` | 31 |
+| 16k | `▁جمهورية ▁الكونغو ▁الديمقراطية ، ▁ز ائ ير ▁سابقًا ، ▁عاصمتها ... (+16 more)` | 26 |
+| 32k | `▁جمهورية ▁الكونغو ▁الديمقراطية ، ▁زائ ير ▁سابقًا ، ▁عاصمتها ▁كينشاسا ... (+12 more)` | 22 |
+| 64k | `▁جمهورية ▁الكونغو ▁الديمقراطية ، ▁زائير ▁سابقًا ، ▁عاصمتها ▁كينشاسا . ... (+10 more)` | 20 |
+
+
+### Key Findings
+
+- **Best Compression:** 64k achieves 4.347x compression
+- **Lowest UNK Rate:** 8k with 0.0702% unknown tokens
+- **Trade-off:** Larger vocabularies improve compression but increase model size
+- **Recommendation:** 32k vocabulary provides optimal balance for production use
+
+---
+## 2. N-gram Model Evaluation
+
+![N-gram Perplexity](visualizations/ngram_perplexity.png)
+
+![N-gram Unique](visualizations/ngram_unique.png)
+
+![N-gram Coverage](visualizations/ngram_coverage.png)
+
+### Results
+
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 359,826 | 18.46 | 2,030,200 | 4.4% | 13.3% |
+| **2-gram** | Subword | 426 🏆 | 8.73 | 44,225 | 56.3% | 96.2% |
+| **3-gram** | Word | 775,988 | 19.57 | 2,900,317 | 3.0% | 10.9% |
+| **3-gram** | Subword | 4,163 | 12.02 | 321,654 | 24.3% | 56.3% |
+| **4-gram** | Word | 1,494,234 | 20.51 | 4,693,107 | 2.8% | 10.2% |
+| **4-gram** | Subword | 27,277 | 14.74 | 1,666,030 | 13.3% | 31.5% |
+| **5-gram** | Word | 1,059,510 | 20.01 | 3,368,028 | 3.6% | 11.9% |
+| **5-gram** | Subword | 133,736 | 17.03 | 5,324,551 | 5.8% | 18.5% |
+
+### Top 5 N-grams by Size
+
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `في عام` | 137,432 |
+| 2 | `في القرن` | 92,611 |
+| 3 | `كرة قدم` | 88,053 |
+| 4 | `العديد من` | 65,695 |
+| 5 | `الولايات المتحدة` | 63,417 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `في القرن 20` | 27,502 |
+| 2 | `في الولايات المتحدة` | 25,188 |
+| 3 | `على الرغم من` | 25,111 |
+| 4 | `في القرن 21` | 20,515 |
+| 5 | `بما في ذلك` | 18,931 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `كرة قدم مغتربون في` | 15,717 |
+| 2 | `تحت سن الثامنة عشر` | 13,585 |
+| 3 | `على الرغم من أن` | 8,756 |
+| 4 | `في الألعاب الأولمبية الصيفية` | 5,980 |
+| 5 | `عام بلغ عدد سكان` | 5,886 |
+
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `تعداد عام بلغ عدد سكان` | 5,588 |
+| 2 | `بحسب تعداد عام وبلغ عدد` | 5,569 |
+| 3 | `تعداد عام وبلغ عدد الأسر` | 5,569 |
+| 4 | `نسمة بحسب تعداد عام وبلغ` | 5,566 |
+| 5 | `في الفئة العمرية ما بين` | 5,561 |
+
+**2-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ا ل` | 27,516,669 |
+| 2 | `_ ا` | 23,616,110 |
+| 3 | `ة _` | 13,152,069 |
+| 4 | `ن _` | 9,255,735 |
+| 5 | `ي _` | 9,009,959 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ ا ل` | 22,248,047 |
+| 2 | `ا ل م` | 4,149,844 |
+| 3 | `ي ة _` | 4,126,642 |
+| 4 | `_ ف ي` | 4,065,816 |
+| 5 | `ف ي _` | 3,976,227 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ ف ي _` | 3,688,677 |
+| 2 | `ة _ ا ل` | 3,625,657 |
+| 3 | `_ ا ل م` | 3,573,633 |
+| 4 | `ن _ ا ل` | 2,468,103 |
+| 5 | `_ م ن _` | 2,362,149 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ف ي _ ا ل` | 1,266,206 |
+| 2 | `_ ف ي _ ا` | 1,245,053 |
+| 3 | `ا ت _ ا ل` | 1,085,180 |
+| 4 | `_ ع ل ى _` | 1,078,435 |
+| 5 | `ي ة _ ا ل` | 1,036,752 |
+
+
+### Key Findings
+
+- **Best Perplexity:** 2-gram (subword) with 426
+- **Entropy Trend:** Decreases with larger n-grams (more predictable)
+- **Coverage:** Top-1000 patterns cover ~18% of corpus
+- **Recommendation:** 4-gram or 5-gram for best predictive performance
+
+---
+## 3. Markov Chain Evaluation
+
+![Markov Entropy](visualizations/markov_entropy.png)
+
+![Markov Contexts](visualizations/markov_contexts.png)
+
+![Markov Branching](visualizations/markov_branching.png)
+
+### Results
+
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 1.0468 | 2.066 | 15.08 | 2,190,668 | 0.0% |
+| **1** | Subword | 1.2063 | 2.307 | 11.28 | 11,477 | 0.0% |
+| **2** | Word | 0.3256 | 1.253 | 2.03 | 33,010,787 | 67.4% |
+| **2** | Subword | 0.8269 | 1.774 | 5.80 | 129,485 | 17.3% |
+| **3** | Word | 0.1052 | 1.076 | 1.21 | 67,054,969 | 89.5% |
+| **3** | Subword | 0.7049 | 1.630 | 4.15 | 751,177 | 29.5% |
+| **4** | Word | 0.0350 🏆 | 1.025 | 1.06 | 81,123,579 | 96.5% |
+| **4** | Subword | 0.6481 | 1.567 | 3.38 | 3,113,652 | 35.2% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `في الألعاب الآسيوية خاض جهادًا فالأقدر قتالًا شديدًا تولَّى من حيث كانوا في المناهج العلاجية في`
+2. `من حيث منعت استخدام مصطلح من الخلايا battery of america bureau of the baskervilles العديد من`
+3. `على التلال وهي عضو النادي مبارياته الدولية بعد فوز فرنسا بيافرا التي عززت المظهر الخارجي للمبنى`
+
+**Context Size 2:**
+
+1. `في عام وقد انتقل بعض أفراد فرقته إلى فرقة المسرح الكويتي مسرح الرواد في هذا المجال غونار`
+2. `في القرن 20 يابانيون في القرن 20 ذكور في سينيما ماراثية من دلهي النحات الرئيسي والمسؤول الرئيسي`
+3. `كرة قدم مغتربون في الولايات المتحدة وبريطانيا العظمى والهجينة على مركبة فضائية مأهولة في منطقة كوم ا...`
+
+**Context Size 3:**
+
+1. `في القرن 20 أمريكيون في القرن 21 هـ في القاهرة 923 هـ في القاهرة بالعربية في القرن 7`
+2. `على الرغم من محدودية علمهم ومستواهما الثقافي إلا أنهما كانا تابعين لأمير بلدة فيدين البلغاري ميخائيل...`
+3. `في الولايات المتحدة تصغير يسار ترجمة لاتينية عمرها خمس مائة عام لكتاب القانون في الطب لابن سينا وقال`
+
+**Context Size 4:**
+
+1. `كرة قدم مغتربون في فرنسا كيداه منتخب ماليزيا لكرة القدم روابط خارجية مراجع رجال ناميبيون في القرن 21...`
+2. `تحت سن الثامنة عشر ونسبة 18 3 في الخامسة والستين من العمر وما فوق تعداد عام بلغ عدد سكان`
+3. `على الرغم من أن الاكتشافات الأثرية لا تدعم هذه النظرية حيث أن تسمية الألوان الأساسية طبقا للتطور الت...`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_التفيهالرزين_ول`
+2. `اقاتية_إلوعتحروب`
+3. `ل_ب_ي_الجة_الجة_`
+
+**Context Size 2:**
+
+1. `البية_عصرية_على_أ`
+2. `_الزهربية._إره_مق`
+3. `ة_التعلى_المية_ال`
+
+**Context Size 3:**
+
+1. `_البحر_من_أصبحت_حر`
+2. `المصر_السنّة_-_فقد_`
+3. `ية_في_wirtugust_ha`
+
+**Context Size 4:**
+
+1. `_في_جنوبيَّة_من_ناثـر`
+2. `ة_الممثلين_على_نفسه`
+3. `_المسيحيون_فلوريدا.`
+
+
+### Key Findings
+
+- **Best Predictability:** Context-4 (word) with 96.5% predictability
+- **Branching Factor:** Decreases with context size (more deterministic)
+- **Memory Trade-off:** Larger contexts require more storage (3,113,652 contexts)
+- **Recommendation:** Context-3 or Context-4 for text generation
+
+---
+## 4. Vocabulary Analysis
+
+![Zipf's Law](visualizations/zipf_law.png)
+
+![Top Words](visualizations/top20_words.png)
+
+![Coverage Curve](visualizations/vocab_coverage.png)
+
+### Statistics
+
+| Metric | Value |
+|--------|-------|
+| Vocabulary Size | 986,324 |
+| Total Tokens | 94,902,130 |
+| Mean Frequency | 96.22 |
+| Median Frequency | 4 |
+| Frequency Std Dev | 4980.31 |
+
+### Most Common Words
+
+| Rank | Word | Frequency |
+|------|------|-----------|
+| 1 | في | 3,714,132 |
+| 2 | من | 2,378,870 |
+| 3 | على | 1,085,920 |
+| 4 | إلى | 833,112 |
+| 5 | أن | 489,978 |
+| 6 | عام | 455,946 |
+| 7 | التي | 369,985 |
+| 8 | عن | 368,235 |
+| 9 | أو | 366,818 |
+| 10 | مع | 331,151 |
+
+### Least Common Words (from vocabulary)
+
+| Rank | Word | Frequency |
+|------|------|-----------|
+| 1 | وساريكات | 2 |
+| 2 | نهايةالمدةفترة | 2 |
+| 3 | valachi | 2 |
+| 4 | فالمختصون | 2 |
+| 5 | المتأسفين | 2 |
+| 6 | والمنشغلين | 2 |
+| 7 | انحسبت | 2 |
+| 8 | غيوان | 2 |
+| 9 | moji | 2 |
+| 10 | إيمجوي | 2 |
+
+### Zipf's Law Analysis
+
+| Metric | Value |
+|--------|-------|
+| Zipf Coefficient | 0.9151 |
+| R² (Goodness of Fit) | 0.992048 |
+| Adherence Quality | **excellent** |
+
+### Coverage Analysis
+
+| Top N Words | Coverage |
+|-------------|----------|
+| Top 100 | 22.0% |
+| Top 1,000 | 43.4% |
+| Top 5,000 | 63.5% |
+| Top 10,000 | 72.1% |
+
+### Key Findings
+
+- **Zipf Compliance:** R²=0.9920 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 22.0% of corpus
+- **Long Tail:** 976,324 words needed for remaining 27.9% coverage
+
+---
+## 5. Word Embeddings Evaluation
+
+![Embedding Isotropy](visualizations/embedding_isotropy.png)
+
+![Similarity Matrix](visualizations/embedding_similarity.png)
+
+![t-SNE Words](visualizations/tsne_words.png)
+
+![t-SNE Sentences](visualizations/tsne_sentences.png)
+
+
+### 5.1 Cross-Lingual Alignment
+
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8111 | 0.3617 | N/A | N/A |
+| **mono_64d** | 64 | 0.7841 | 0.2928 | N/A | N/A |
+| **mono_128d** | 128 | 0.7556 | 0.2345 | N/A | N/A |
+| **aligned_32d** | 32 | 0.8111 🏆 | 0.3646 | 0.1340 | 0.4860 |
+| **aligned_64d** | 64 | 0.7841 | 0.2939 | 0.2860 | 0.6560 |
+| **aligned_128d** | 128 | 0.7556 | 0.2339 | 0.3720 | 0.7660 |
+
+### Key Findings
+
+- **Best Isotropy:** aligned_32d with 0.8111 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2969. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 37.2% R@1 in cross-lingual retrieval.
+- **Recommendation:** 128d aligned for best cross-lingual performance
+
+---
+## 6.  Morphological Analysis (Experimental)
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **-0.353** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ال` | الماكر, الويجرية, العرقيه |
+| `-وال` | والمسكيت, والمأذون, والرسو |
+| `-و` | وَصِيف, وخلافاً, وبالكيفية |
+| `-الم` | الماكر, المحيطان, المتوسِّط |
+| `-بال` | بالمدين, بالجماع, بالتأثر |
+| `-ب` | بِالنيابة, بالمدين, بقاءة |
+| `-ل` | للتمتع, لجرح, لقزم |
+| `-م` | مصصم, معاملات, مناظرا |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-ا` | تعرضها, مناظرا, فقابلا |
+| `-ن` | بالمدين, والمأذون, المحيطان |
+| `-ة` | بِالنيابة, الويجرية, وبالكيفية |
+| `-ت` | معاملات, والمسكيت, ومُؤسسات |
+| `-ي` | اخصابي, الثانيةفي, بيبيمي |
+| `-ين` | بالمدين, الغلامين, للهيروجين |
+| `-ات` | معاملات, ومُؤسسات, الألقابسنوات |
+| `-م` | تسكينهم, مصصم, لقزم |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `ستخد` | 2.56x | 420 contexts | ستخدم, يستخد, تستخد |
+| `التع` | 1.70x | 417 contexts | التعس, التعب, التعمد |
+| `مجمو` | 2.12x | 120 contexts | مجموة, مجمود, مجموع |
+| `استخ` | 1.97x | 149 contexts | استخف, استخم, استخد |
+| `تحدة` | 2.82x | 26 contexts | متحدة, ومتحدة, لمتحدة |
+| `المق` | 1.38x | 607 contexts | المقد, المقل, المقص |
+| `ارات` | 1.31x | 739 contexts | كارات, تارات, دارات |
+| `لمنا` | 1.38x | 514 contexts | ظلمنا, حلمنا, لمنار |
+| `المج` | 1.39x | 473 contexts | المجل, المجد, المجن |
+| `امعة` | 2.14x | 53 contexts | قامعة, دامعة, سامعة |
+| `لعال` | 1.76x | 115 contexts | العال, لعالم, لعالي |
+| `الحا` | 1.34x | 492 contexts | الحاد, الحاق, الحاف |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-ال` | `-ة` | 297 words | المحميّة, العقاقيرية |
+| `-ال` | `-ن` | 179 words | القطبيتان, المحتشدين |
+| `-ال` | `-ي` | 167 words | السيجومي, الازدي |
+| `-و` | `-ا` | 138 words | والكوسا, ومجتهدًا |
+| `-ال` | `-ية` | 129 words | العقاقيرية, المُغطية |
+| `-ال` | `-ت` | 113 words | الأستكشافات, المتغيِّرات |
+| `-ال` | `-ين` | 98 words | المحتشدين, المتخاذلين |
+| `-ال` | `-ات` | 97 words | الأستكشافات, المتغيِّرات |
+| `-وال` | `-ة` | 72 words | والحرورية, والضرورية |
+| `-م` | `-ا` | 64 words | مُؤديًا, مأمونًا |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| الهوسيتية | **`الهوسي-ت-ية`** | 7.5 | `ت` |
+| وتحويلتين | **`وتحويل-ت-ين`** | 7.5 | `ت` |
+| القراخانيين | **`القراخان-ي-ين`** | 7.5 | `ي` |
+| فیروزآباد | **`فیروزآب-ا-د`** | 7.5 | `ا` |
+| الارسالية | **`الا-رسال-ية`** | 6.0 | `رسال` |
+| والمتعلمة | **`وال-متعلم-ة`** | 6.0 | `متعلم` |
+| والكيكونغو | **`و-ال-كيكونغو`** | 6.0 | `كيكونغو` |
+| والسويسريين | **`و-ال-سويسريين`** | 6.0 | `سويسريين` |
+| والنازحون | **`و-ال-نازحون`** | 6.0 | `نازحون` |
+| القترائية | **`الق-ترائ-ية`** | 6.0 | `ترائ` |
+| للنوميديين | **`لل-نوميدي-ين`** | 6.0 | `نوميدي` |
+| والفاندال | **`و-ال-فاندال`** | 6.0 | `فاندال` |
+| وبالإجراءات | **`و-بال-إجراءات`** | 6.0 | `إجراءات` |
+| بالهليكوبتر | **`ب-ال-هليكوبتر`** | 6.0 | `هليكوبتر` |
+| والاستقلابية | **`و-ال-استقلابية`** | 6.0 | `استقلابية` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language Arabic shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+---
+## 7. Summary & Recommendations
+
+![Performance Dashboard](visualizations/performance_dashboard.png)
+
+### Production Recommendations
+
+| Component | Recommended | Rationale |
+|-----------|-------------|-----------|
+| Tokenizer | **64k BPE** | Best compression (4.35x) |
+| N-gram | **2-gram** | Lowest perplexity (426) |
+| Markov | **Context-4** | Highest predictability (96.5%) |
+| Embeddings | **100d** | Balanced semantic capture and isotropy |
+
+
+---
+## Appendix: Metrics Glossary & Interpretation Guide
+
+This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report.
+
+### Tokenizer Metrics
+
+**Compression Ratio**
+> *Definition:* The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text.
+>
+> *Intuition:* Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average.
+>
+> *What to seek:* Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information.
+
+**Average Token Length (Fertility)**
+> *Definition:* Mean number of characters per token produced by the tokenizer.
+>
+> *Intuition:* Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length.
+>
+> *What to seek:* Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens.
+
+**Unknown Token Rate (OOV Rate)**
+> *Definition:* Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent.
+>
+> *Intuition:* Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences.
+>
+> *What to seek:* Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback.
+
+### N-gram Model Metrics
+
+**Perplexity**
+> *Definition:* Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction.
+>
+> *Intuition:* If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options.
+>
+> *What to seek:* Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size.
+
+**Entropy**
+> *Definition:* Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy.
+>
+> *Intuition:* High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character.
+>
+> *What to seek:* Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases.
+
+**Coverage (Top-K)**
+> *Definition:* Percentage of corpus occurrences explained by the top K most frequent n-grams.
+>
+> *Intuition:* High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage.
+>
+> *What to seek:* Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text.
+
+### Markov Chain Metrics
+
+**Average Entropy**
+> *Definition:* Mean entropy across all contexts, measuring average uncertainty in next-word prediction.
+>
+> *Intuition:* Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations).
+>
+> *What to seek:* Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions.
+
+**Branching Factor**
+> *Definition:* Average number of unique next tokens observed for each context.
+>
+> *Intuition:* High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive).
+>
+> *What to seek:* Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains.
+
+**Predictability**
+> *Definition:* Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are.
+>
+> *Intuition:* 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes.
+>
+> *What to seek:* Higher predictability for text generation quality, but too high (>98%) may produce repetitive output.
+
+### Vocabulary & Zipf's Law Metrics
+
+**Zipf's Coefficient**
+> *Definition:* The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1.
+>
+> *Intuition:* A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare.
+>
+> *What to seek:* Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text.
+
+**R² (Coefficient of Determination)**
+> *Definition:* Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1.
+>
+> *Intuition:* R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns.
+>
+> *What to seek:* R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora.
+
+**Vocabulary Coverage**
+> *Definition:* Cumulative percentage of corpus tokens accounted for by the top N words.
+>
+> *Intuition:* Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words.
+>
+> *What to seek:* Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary.
+
+### Word Embedding Metrics
+
+**Isotropy**
+> *Definition:* Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values.
+>
+> *Intuition:* High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness.
+>
+> *What to seek:* Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy.
+
+**Average Norm**
+> *Definition:* Mean magnitude (L2 norm) of word vectors in the embedding space.
+>
+> *Intuition:* Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained.
+>
+> *What to seek:* Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation).
+
+**Cosine Similarity**
+> *Definition:* Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction).
+>
+> *Intuition:* Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings.
+>
+> *What to seek:* Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7.
+
+**t-SNE Visualization**
+> *Definition:* t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization.
+>
+> *Intuition:* Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence.
+>
+> *What to seek:* Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure.
+
+### General Interpretation Guidelines
+
+1. **Compare within model families:** Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer).
+2. **Consider trade-offs:** Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate).
+3. **Context matters:** Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification.
+4. **Corpus influence:** All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature.
+5. **Language-specific patterns:** Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages.
+
+
+### Visualizations Index
+
+| Visualization | Description |
+|---------------|-------------|
+| Tokenizer Compression | Compression ratios by vocabulary size |
+| Tokenizer Fertility | Average token length by vocabulary |
+| Tokenizer OOV | Unknown token rates |
+| Tokenizer Total Tokens | Total tokens by vocabulary |
+| N-gram Perplexity | Perplexity by n-gram size |
+| N-gram Entropy | Entropy by n-gram size |
+| N-gram Coverage | Top pattern coverage |
+| N-gram Unique | Unique n-gram counts |
+| Markov Entropy | Entropy by context size |
+| Markov Branching | Branching factor by context |
+| Markov Contexts | Unique context counts |
+| Zipf's Law | Frequency-rank distribution with fit |
+| Vocab Frequency | Word frequency distribution |
+| Top 20 Words | Most frequent words |
+| Vocab Coverage | Cumulative coverage curve |
+| Embedding Isotropy | Vector space uniformity |
+| Embedding Norms | Vector magnitude distribution |
+| Embedding Similarity | Word similarity heatmap |
+| Nearest Neighbors | Similar words for key terms |
+| t-SNE Words | 2D word embedding visualization |
+| t-SNE Sentences | 2D sentence embedding visualization |
+| Position Encoding | Encoding method comparison |
+| Model Sizes | Storage requirements |
+| Performance Dashboard | Comprehensive performance overview |
+
+---
+👈 [Back to README](README.md)
+
+*Generated by Wikilangs Pipeline · 2026-03-04 14:57:25*

models/embeddings/aligned/ar_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ef219238cabd8a9ea12da66dbcd80332fbd38875433dc953df85a37a71b899aa
-size 2486763168
+oid sha256:a23418872eb4060d92c5a30858431c4a0d120f4313a4677ad9e2337907f80414
+size 1705713534

models/embeddings/aligned/ar_128d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2d61b78158be557df612ca2ef8343029b39babaae6f038fcd428534279135917
+oid sha256:d27f01e32d46732cff34bb048290910d32d944952b6d6c6066be98c12640f6c2
 size 65664

models/embeddings/aligned/ar_128d_metadata.json

@@ -3,6 +3,6 @@
   "dimension": 128,
   "version": "aligned",
   "hub_language": "en",
-  "seed_vocab_size": 200166,
-  "vocab_size": 1398324
+  "seed_vocab_size": 77116,
+  "vocab_size": 651662
 }

models/embeddings/aligned/ar_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:70e78275bfd0f11c47d9f144be703b7d9277b2e2698a9f3c400035b92f124640
-size 644850336
+oid sha256:30a05f3683a9b7bde4c6d30bd4edbfc63345f6c7ee83a0d8a6c90d80224b270d
+size 437237118

models/embeddings/aligned/ar_32d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:baedb1283a79af0fe218dcef59dcfa703ea3550812bcf17a0edcc064c20d6b39
+oid sha256:a98989d89641c5f1cd3758f6c607fe38b791974e69cf7a38a8796868fbd89880
 size 4224

models/embeddings/aligned/ar_32d_metadata.json

@@ -3,6 +3,6 @@
   "dimension": 32,
   "version": "aligned",
   "hub_language": "en",
-  "seed_vocab_size": 200166,
-  "vocab_size": 1398324
+  "seed_vocab_size": 77116,
+  "vocab_size": 651662
 }

models/embeddings/aligned/ar_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:02bc5f2f59c273f6f58b680abc0742d269afaafa47c336c6429c6a81304b006c
-size 1258821280
+oid sha256:feb5b0e7bf35dfc12b4304889348f19a160e32236c8ea50ace3714ad06dedda2
+size 860062590

models/embeddings/aligned/ar_64d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9efa73eb7ec91ab64fdf021d74e55418d0906b47c6bc624aab5b6a37dbd88bb9
+oid sha256:b99ebcc070a11027cca93e3acb53ba76d4eb41fe624d515260d3a6746602381d
 size 16512

models/embeddings/aligned/ar_64d_metadata.json

@@ -3,6 +3,6 @@
   "dimension": 64,
   "version": "aligned",
   "hub_language": "en",
-  "seed_vocab_size": 200166,
-  "vocab_size": 1398324
+  "seed_vocab_size": 77116,
+  "vocab_size": 651662
 }

models/embeddings/monolingual/ar_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ef219238cabd8a9ea12da66dbcd80332fbd38875433dc953df85a37a71b899aa
-size 2486763168
+oid sha256:a23418872eb4060d92c5a30858431c4a0d120f4313a4677ad9e2337907f80414
+size 1705713534

models/embeddings/monolingual/ar_128d_metadata.json

@@ -9,7 +9,8 @@
     "negative": 5,
     "epochs": 5,
     "encoding_method": "rope",
-    "dim": 128
+    "dim": 128,
+    "threads": 40
   },
-  "vocab_size": 1398324
+  "vocab_size": 651662
 }

models/embeddings/monolingual/ar_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:70e78275bfd0f11c47d9f144be703b7d9277b2e2698a9f3c400035b92f124640
-size 644850336
+oid sha256:30a05f3683a9b7bde4c6d30bd4edbfc63345f6c7ee83a0d8a6c90d80224b270d
+size 437237118

models/embeddings/monolingual/ar_32d_metadata.json

@@ -9,7 +9,8 @@
     "negative": 5,
     "epochs": 5,
     "encoding_method": "rope",
-    "dim": 32
+    "dim": 32,
+    "threads": 40
   },
-  "vocab_size": 1398324
+  "vocab_size": 651662
 }

models/embeddings/monolingual/ar_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:02bc5f2f59c273f6f58b680abc0742d269afaafa47c336c6429c6a81304b006c
-size 1258821280
+oid sha256:feb5b0e7bf35dfc12b4304889348f19a160e32236c8ea50ace3714ad06dedda2
+size 860062590

models/embeddings/monolingual/ar_64d_metadata.json

@@ -9,7 +9,8 @@
     "negative": 5,
     "epochs": 5,
     "encoding_method": "rope",
-    "dim": 64
+    "dim": 64,
+    "threads": 40
   },
-  "vocab_size": 1398324
+  "vocab_size": 651662
 }

models/subword_markov/ar_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:574edae048b28cd710d49d346838e995d3075aa7e9404f1962349f0f2b606f4f
-size 1510871
+oid sha256:695411e3e0fa61a7791e28a45830bee9ac3b8b467563fa3e341cf37c56f9eba9
+size 885004

models/subword_markov/ar_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "ar",
-  "unique_contexts": 18570,
-  "total_transitions": 1931607401
+  "unique_contexts": 11477,
+  "total_transitions": 570748046
 }

models/subword_markov/ar_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4c019e23cfe8e01e37162bcd6b57200033e2d76cdc0cc88a628d6e7fbd6649b8
-size 10138087
+oid sha256:021c447fa0d5480ddaa3f705fc2c2572210793992622f9af98ee6a8103ecd310
+size 5670221

models/subword_markov/ar_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "ar",
-  "unique_contexts": 247596,
-  "total_transitions": 1930342017
+  "unique_contexts": 129485,
+  "total_transitions": 570548046
 }

models/subword_markov/ar_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1ef7a7d5ec14ab20dc8f9e56c4e3c940701ba628aca3d6c03fe556dff843d17d
-size 44109485
+oid sha256:0ac54f42aa1f3b3b4d28fd393c117d51f4756740f0b1ce01f5925e6d367d3ad2
+size 24893296

models/subword_markov/ar_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "ar",
-  "unique_contexts": 1290623,
-  "total_transitions": 1929076633
+  "unique_contexts": 751177,
+  "total_transitions": 570348046
 }

models/subword_markov/ar_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f7ee34b80789c6d5441f10ed5482fffa62a5db280978fce263f369b0031fc1f2
-size 165497966
+oid sha256:07694bbe65c76bc4b5517dfd4f4d441fb1e34a96b51bc8787e5cef7f8137044a
+size 91923589

models/subword_markov/ar_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "ar",
-  "unique_contexts": 5343485,
-  "total_transitions": 1927811249
+  "unique_contexts": 3113652,
+  "total_transitions": 570148046
 }

models/subword_ngram/ar_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:051d0c469791c68d0aab623ee6f2f4956d7da4e670aec6403120a79e8734828f
-size 1010043
+oid sha256:85c8e81a237670dd83f41fca1f4a1cde191e26f9437ee143db0965c7b6d22525
+size 641342

models/subword_ngram/ar_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "ar",
-  "unique_ngrams": 70700,
-  "total_ngrams": 1931607401
+  "unique_ngrams": 44225,
+  "total_ngrams": 570748046
 }

models/subword_ngram/ar_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:11567f60179d7d679ac0fe53473a44a51239bc6838cc731298039e543e3539ec
-size 7513640
+oid sha256:5f0bc18aeb185668ed8330b530f401a76e72574597f9a070dff414446229d05e
+size 4404735

models/subword_ngram/ar_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "ar",
-  "unique_ngrams": 528264,
-  "total_ngrams": 1930342017
+  "unique_ngrams": 321654,
+  "total_ngrams": 570548046
 }

models/subword_ngram/ar_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cf2ba8cd29e1c4e91b5db8c9a037dd559bec62bb1a44450fe5ffc882168bea52
-size 40025731
+oid sha256:2a9da743ee22b7d638bc93633862fccf3d26e98fb0961bdebdb14c96c3e0b076
+size 23078539

models/subword_ngram/ar_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "ar",
-  "unique_ngrams": 2851427,
-  "total_ngrams": 1929076633
+  "unique_ngrams": 1666030,
+  "total_ngrams": 570348046
 }

models/subword_ngram/ar_5gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4d827290f0e4a08a5ac6ad25f1e75c878bc6b8a34c0c2c90d128f1fea76e05bd
-size 139944679
+oid sha256:c2e38dae79bd188be50f4564c8424a813005cafdffd7909810d31a64ed8fda5f
+size 76060564

models/subword_ngram/ar_5gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 5,
   "variant": "subword",
   "language": "ar",
-  "unique_ngrams": 9618770,
-  "total_ngrams": 1927811249
+  "unique_ngrams": 5324551,
+  "total_ngrams": 570148046
 }

models/tokenizer/ar_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:762cad48886cec152e6e3f7ed18568586bc3c9f4e9953b3185639b108ba8aac0
-size 560431
+oid sha256:c2559917c0122732cb2a776d370d5c7e5979c181a37023fd8b41e2c531c89f94
+size 560444

models/tokenizer/ar_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c76c24bcacf6d2321aead7205693601874ae17f908952dc1038e02c3176e6192
-size 898238
+oid sha256:f89d8be50621d034e2113f1db13c6a069ddc5597b020e92abd25bc9eb70abaa9
+size 898267

models/tokenizer/ar_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f086baad9b6b110330fd920dd11031d9f3ef7f5660ab6ac1c35134bb867b9124
-size 1589613
+oid sha256:fb8462b65f3bafaca92b2c52ab2eb99e0e8559caa8a49964b92ada753bb1c34c
+size 1588432

models/tokenizer/ar_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:77f3c2a9fceaccd03fdcfabdbaec3cf8ededb0885e5142dd55eb2decced61240
-size 396806
+oid sha256:39e7834c341171505c74bc358974bcbf01abd8fe6edb43d74f6b93448701eb75
+size 396789

models/vocabulary/ar_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b31772fd6a7b4c36fa172ae0fd4a4d5dabd71f58e9966f89680cfe83fda2db9c
-size 30017440
+oid sha256:afd83458fca2ab7172dd960492c27e62334d37aeea40ba7d9310eec0710d83e9
+size 14934604

models/vocabulary/ar_vocabulary_metadata.json

@@ -1,17 +1,17 @@
 {
   "language": "ar",
-  "vocabulary_size": 1950572,
+  "vocabulary_size": 986324,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.01377005207307626,
+    "type_token_ratio": 0.022798498576049157,
     "coverage": {
-      "top_100": 0.22964907469610985,
-      "top_1000": 0.4549956533720101,
-      "top_5000": 0.6553878216380935,
-      "top_10000": 0.7363291209271269
+      "top_100": 0.21751010819733138,
+      "top_1000": 0.4283766142532598,
+      "top_5000": 0.6275286231012406,
+      "top_10000": 0.7122672223562223
     },
-    "hapax_count": 2521609,
-    "hapax_ratio": 0.5638432344308068,
-    "total_documents": 1265384
+    "hapax_count": 1204769,
+    "hapax_ratio": 0.5498484089903989,
+    "total_documents": 200000
   }
 }

models/word_markov/ar_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a9fddcbf1a23594543219647432ec4ead2edb2b2e1de36c8a8f47889080afef7
-size 765307915
+oid sha256:05c13eef003a0e333d4228ebe6db3df097686d981e0730b7e9e17f85ac68e5b4
+size 320610853

models/word_markov/ar_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "ar",
-  "unique_contexts": 4471621,
-  "total_transitions": 323510512
+  "unique_contexts": 2190668,
+  "total_transitions": 95906899
 }

models/word_markov/ar_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ebcb89d9a7b0b49e30c58921c6880b16c819b128e34c72ab88840f17104834cc
-size 3214329856
+oid sha256:751d17526f4fb8ef94a12a48dac950e5b876ed04957df675fd3c10e0afcb3e65
+size 1237352806

models/word_markov/ar_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "ar",
-  "unique_contexts": 78540786,
-  "total_transitions": 322245128
+  "unique_contexts": 33010787,
+  "total_transitions": 95706899
 }

models/word_markov/ar_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:65e1dcc162857b9cb13d2e4b4927f8459f5f7a5a71cad4a8c86909b717912497
-size 5738037502
+oid sha256:866b6fdcbeaf5bbc8ba9ef70c09a4e350967e03711364e04b9306269f150ef0f
+size 2023586391

models/word_markov/ar_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "ar",
-  "unique_contexts": 181002468,
-  "total_transitions": 320979744
+  "unique_contexts": 67054969,
+  "total_transitions": 95506899
 }