Upload all models and assets for awa (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.121
+    value: 3.897
   - name: best_isotropy
     type: isotropy
-    value: 0.7531
+    value: 0.7129
   - name: vocabulary_size
     type: vocab
-    value: 6221
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # AWA - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,51 +70,53 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.216x | 3.14 | 0.1062% | 117,684 |
-| **16k** | 3.545x | 3.46 | 0.1171% | 106,780 |
-| **32k** | 3.841x | 3.75 | 0.1268% | 98,555 |
-| **64k** | 4.121x 🏆 | 4.02 | 0.1361% | 91,849 |
+| **8k** | 3.327x | 3.33 | 0.1233% | 131,409 |
+| **16k** | 3.614x | 3.62 | 0.1339% | 120,950 |
+| **32k** | 3.897x 🏆 | 3.91 | 0.1444% | 112,178 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `हिंदीनेस्ट डॉट कॉम हिन्दी भाषा कय एक्ठु पत्रिका होय ।`
+**Sample 1:** `मानवशास्त्र या नृविज्ञान (:en:Anthropology) मनईन, वनकय जेनेटिक्स, संस्कृति अउर स...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁हिंदी ने स्ट ▁डॉ ट ▁कॉ म ▁हिन्दी ▁भाषा ▁कय ... (+4 more)` | 14 |
-| 16k | `▁हिंदी नेस्ट ▁डॉ ट ▁कॉम ▁हिन्दी ▁भाषा ▁कय ▁एक्ठु ▁पत्रिका ... (+2 more)` | 12 |
-| 32k | `▁हिंदी नेस्ट ▁डॉ ट ▁कॉम ▁हिन्दी ▁भाषा ▁कय ▁एक्ठु ▁पत्रिका ... (+2 more)` | 12 |
-| 64k | `▁हिंदीनेस्ट ▁डॉट ▁कॉम ▁हिन्दी ▁भाषा ▁कय ▁एक्ठु ▁पत्रिका ▁होय ▁।` | 10 |
+| 8k | `▁मानव शास्त्र ▁या ▁न ृ विज्ञान ▁(: en : an ... (+25 more)` | 35 |
+| 16k | `▁मानवशास्त्र ▁या ▁नृ विज्ञान ▁(: en : an throp ology ... (+23 more)` | 33 |
+| 32k | `▁मानवशास्त्र ▁या ▁नृविज्ञान ▁(: en : anthropology ) ▁मनईन , ... (+16 more)` | 26 |
 
-**Sample 2:** `इशांत शर्मा भारतीय क्रिकेट खिलाड़ी होयँ। इशांत  शर्मा कय जनम २ सितम्बर १९८८ को द...`
+**Sample 2:** `सिरसा, भारत देश के हरियाणा राज्य कय एक्ठु जिला अव नगर परिषद होय । कय नगर परिषद म...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁इ श ांत ▁शर्मा ▁भारतीय ▁क्रिकेट ▁खिलाड़ी ▁होयँ । ▁इ ... (+15 more)` | 25 |
-| 16k | `▁इश ांत ▁शर्मा ▁भारतीय ▁क्रिकेट ▁खिलाड़ी ▁होयँ । ▁इश ांत ... (+12 more)` | 22 |
-| 32k | `▁इशांत ▁शर्मा ▁भारतीय ▁क्रिकेट ▁खिलाड़ी ▁होयँ । ▁इशांत ▁शर्मा ▁कय ... (+10 more)` | 20 |
-| 64k | `▁इशांत ▁शर्मा ▁भारतीय ▁क्रिकेट ▁खिलाड़ी ▁होयँ । ▁इशांत ▁शर्मा ▁कय ... (+10 more)` | 20 |
+| 8k | `▁सिरसा , ▁भारत ▁देश ▁के ▁हरियाणा ▁राज्य ▁कय ▁एक्ठु ▁जिला ... (+11 more)` | 21 |
+| 16k | `▁सिरसा , ▁भारत ▁देश ▁के ▁हरियाणा ▁राज्य ▁कय ▁एक्ठु ▁जिला ... (+11 more)` | 21 |
+| 32k | `▁सिरसा , ▁भारत ▁देश ▁के ▁हरियाणा ▁राज्य ▁कय ▁एक्ठु ▁जिला ... (+11 more)` | 21 |
 
-**Sample 3:** `बीरबल साहनी (नवंबर, 1891 - 10 अप्रैल, 1949) पुरावनस्पति वैज्ञानिक रहे।`
+**Sample 3:** `अनूपशहर, भारत देश के उत्तर प्रदेश प्रान्त के बुलंदशहर जिला कय एक्ठु नगर पालिका प...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁बी र बल ▁साह नी ▁( न वंबर , ▁ ... (+22 more)` | 32 |
-| 16k | `▁बीर बल ▁साहनी ▁( न वंबर , ▁ 1 8 ... (+20 more)` | 30 |
-| 32k | `▁बीर बल ▁साहनी ▁( नवंबर , ▁ 1 8 9 ... (+18 more)` | 28 |
-| 64k | `▁बीरबल ▁साहनी ▁( नवंबर , ▁ 1 8 9 1 ... (+16 more)` | 26 |
+| 8k | `▁अन ूप श हर , ▁भारत ▁देश ▁के ▁उत्तर ▁प्रदेश ... (+21 more)` | 31 |
+| 16k | `▁अनूपश हर , ▁भारत ▁देश ▁के ▁उत्तर ▁प्रदेश ▁प्रान्त ▁के ... (+19 more)` | 29 |
+| 32k | `▁अनूपशहर , ▁भारत ▁देश ▁के ▁उत्तर ▁प्रदेश ▁प्रान्त ▁के ▁बुलंदशहर ... (+18 more)` | 28 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.121x compression
-- **Lowest UNK Rate:** 8k with 0.1062% unknown tokens
+- **Best Compression:** 32k achieves 3.897x compression
+- **Lowest UNK Rate:** 8k with 0.1233% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -121,57 +125,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 1,385 🏆 | 10.44 | 12,281 | 39.7% | 79.7% |
-| **2-gram** | 675 🏆 | 9.40 | 4,587 | 48.8% | 91.0% |
-| **3-gram** | 8,332 | 13.02 | 41,199 | 17.5% | 46.9% |
-| **3-gram** | 5,112 | 12.32 | 31,251 | 20.4% | 54.9% |
-| **4-gram** | 26,093 | 14.67 | 106,862 | 12.1% | 31.7% |
-| **4-gram** | 20,452 | 14.32 | 106,978 | 12.4% | 33.6% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 2,211 | 11.11 | 5,376 | 29.5% | 59.6% |
+| **2-gram** | Subword | 1,584 | 10.63 | 11,871 | 40.0% | 73.5% |
+| **3-gram** | Word | 1,558 🏆 | 10.61 | 4,851 | 36.7% | 66.9% |
+| **3-gram** | Subword | 9,994 | 13.29 | 42,588 | 17.4% | 41.6% |
+| **4-gram** | Word | 3,905 | 11.93 | 12,076 | 28.3% | 51.2% |
+| **4-gram** | Subword | 29,097 | 14.83 | 105,286 | 12.1% | 28.9% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `प्रदेश कय` | 1,241 |
+| 2 | `कय एक्ठु` | 1,217 |
+| 3 | `नगर पंचायत` | 932 |
+| 4 | `शहरी निकाय` | 837 |
+| 5 | `उत्तर प्रदेश` | 773 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `कय एक्ठु नगर` | 700 |
+| 2 | `भारत देश के` | 696 |
+| 3 | `जिला कय एक्ठु` | 680 |
+| 4 | `कय शहरी निकाय` | 667 |
+| 5 | `के उत्तर प्रदेश` | 586 |
+
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `् र` | 15,730 |
-| 2 | `क ा` | 10,109 |
-| 3 | `र ा` | 9,823 |
-| 4 | `क े` | 9,329 |
-| 5 | `ा र` | 9,129 |
+| 1 | `जिला कय एक्ठु नगर` | 661 |
+| 2 | `के उत्तर प्रदेश प्रान्त` | 582 |
+| 3 | `निकाय प��रदेश कय नगर` | 581 |
+| 4 | `शहरी निकाय प्रदेश कय` | 581 |
+| 5 | `कय शहरी निकाय प्रदेश` | 581 |
 
-**3-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `् र े` | 5,246 |
-| 2 | `श ् र` | 4,638 |
-| 3 | `म े ं` | 4,298 |
-| 4 | `र े ण` | 4,146 |
-| 5 | `े ण ी` | 4,113 |
+| 1 | `र _` | 18,112 |
+| 2 | `य _` | 17,719 |
+| 3 | `_ क` | 16,272 |
+| 4 | `न _` | 12,852 |
+| 5 | `। _` | 11,559 |
 
-**4-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `श ् र े` | 4,151 |
-| 2 | `् र े ण` | 4,138 |
-| 3 | `र े ण ी` | 4,112 |
-| 4 | `े ण ी :` | 3,973 |
-| 5 | `ह ो य ।` | 2,934 |
+| 1 | `क य _` | 10,797 |
+| 2 | `_ क य` | 10,549 |
+| 3 | `_ के _` | 6,719 |
+| 4 | `_ से _` | 3,956 |
+| 5 | `_ में _` | 3,886 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ क य _` | 10,506 |
+| 2 | `_ प्र दे श` | 2,241 |
+| 3 | `प्र दे श _` | 2,190 |
+| 4 | `_ है । _` | 2,071 |
+| 5 | `भा र त _` | 2,019 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 675
+- **Best Perplexity:** 3-gram (word) with 1,558
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~34% of corpus
+- **Coverage:** Top-1000 patterns cover ~29% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -179,55 +215,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.5922 | 1.508 | 5.79 | 13,918 | 40.8% |
-| **1** | 1.5439 | 2.916 | 14.22 | 676 | 0.0% |
-| **2** | 0.4119 | 1.330 | 2.61 | 80,553 | 58.8% |
-| **2** | 1.1858 | 2.275 | 6.79 | 9,605 | 0.0% |
-| **3** | 0.3135 | 1.243 | 1.85 | 210,504 | 68.7% |
-| **3** | 0.7675 | 1.702 | 3.28 | 65,205 | 23.2% |
-| **4** | 0.2012 🏆 | 1.150 | 1.41 | 389,838 | 79.9% |
-| **4** | 0.4559 🏆 | 1.372 | 2.00 | 214,041 | 54.4% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.7301 | 1.659 | 4.17 | 37,356 | 27.0% |
+| **1** | Subword | 1.0434 | 2.061 | 10.70 | 3,632 | 0.0% |
+| **2** | Word | 0.1929 | 1.143 | 1.36 | 155,195 | 80.7% |
+| **2** | Subword | 0.5412 | 1.455 | 3.46 | 38,845 | 45.9% |
+| **3** | Word | 0.0474 | 1.033 | 1.07 | 209,159 | 95.3% |
+| **3** | Subword | 0.4514 | 1.367 | 2.30 | 134,413 | 54.9% |
+| **4** | Word | 0.0142 🏆 | 1.010 | 1.02 | 221,759 | 98.6% |
+| **4** | Subword | 0.2387 | 1.180 | 1.51 | 308,778 | 76.1% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `कय एक्ठु राजनीतिक पार्टी पाकिस्तान कय एक्ठु जिला चुराचांदपुर जिला कय एक्ठो नगरपालिका सप्तरी जिला कय`
+2. `के नाम से गुवाहाटी सिलचर एन यू कि सिक्‍ख गुरू योगी आदित्यनाथ होइ सन्दर्भ कय शहरी`
+3. `में मौसम रहत है शरीर का अड्डा bho इहो देखा जाय रहा साथ जोश और निम्नतम`
+
+**Context Size 2:**
+
+1. `प्रदेश कय मंडल होय एहमा 05 जिला आवत हँय फतेहाबाद जींद हिसार महेंद्रगढ़ गुड़गांव रोहतक और हिसार`
+2. `कय एक्ठु नगर पंचायत के पार्षद चुनाव में ६ राष्ट्रीय दल चुनाव लड़ रहे हैं भारतीय जनता`
+3. `उत्तर प्रदेश प्रान्त के बिजनौर जिला कय मुख्यालय अहै एह समाज मा खुदै आंतरिक सुधार कइके आपन`
+
+**Context Size 3:**
+
+1. `कय एक्ठु नगर पंचायत होय संदर्भ प्रदेश कय शहरी निकाय प्रदेश कय नगर पंचायत पंचायत कय शहरी निकाय`
+2. `भारत देश के उत्तर प्र��ेश प्रान्त के आजमगढ़ जिला कय एक्ठु नगर पालिका होय संदर्भ कय नगर पालिका`
+3. `जिला कय एक्ठु नगर पालिका परिषद पालिका परिषद कय शहरी निकाय प्रदेश कय नगर पंचायत नगर`
+
+**Context Size 4:**
+
+1. `जिला कय एक्ठु नगर पालिका परिषद होय संदर्भ प्रदेश कय शहरी निकाय प्रदेश कय नगर पालिका परिषद खीरी`
+2. `के उत्तर प्रदेश प्रान्त के आजमगढ़ जिला कय एक्ठु नगर पालिका परिषद होय संदर्भ प्रदेश कय शहरी निकाय प्र...`
+3. `शहरी निकाय प्रदेश कय नगर पालिका परिषद पालिका परिषद कय शहरी निकाय`
+
 
-### Generated Text Samples
+### Generated Text Samples (Subword-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `ा । स े ं : उत ् व ा ब म ा कहत ि स`
-2. `् ग ृ ष ् र ा गय ा न ौ क े ं द ्`
-3. `े ट ा ल और 17 फ ा थ / localbodies . htm श ् ट`
+1. `_की_संग_कब्ज़ा.दिनसंलतः_इ`
+2. `र_५_के_कार_एथिरत_न,`
+3. `क_सिद्ध_भा_ए_आत्रेयत_प्रदे`
 
 **Context Size 2:**
 
-1. `् र ि श ी व ् द मह ा द े श क े अरब प`
-2. `क ा स ् त ा थ खतम करय अउर सब ध ा न कय च ौ`
-3. `र ा जन ै त ू र ् णय क ि ह ् मणन कय सबस े`
+1. `र_पता_है।_विश्व_प्रथम_ई_`
+2. `य_छात्र_में_बने_मा_स्पेस),_`
+3. `_कय_राष्ट्रीय_है_सें._पुर_मा`
 
 **Context Size 3:**
 
-1. `् र े ण ी : उत ् तर प ् रद े श कय नगर प ं`
-2. `श ् र े ण ी : र ा ष ् ट ् र े ज ़ ी`
-3. `म े ं प ं ज ा ब , स ि खय व ा ल ि आर ि`
+1. `कय_हाइड्रोकार्बन_कलायत_राज्य_`
+2. `_कय_लेकिन_वाली_एक_लोचनो_`
+3. `_के_प्रयाग,भारत_कय_क्रिस_मॉ`
 
 **Context Size 4:**
 
-1. `श ् र े ण ी : नगर प ं च ा यत श ् र े ण ी`
-2. `् र े ण ी : र ा जन ी त ि क दल ह ो य । स`
-3. `र े ण ी : उत ् तर प ् रद े श स ं दर ् भ http`
+1. `_कय_शहरी_निकाय_प्रदेश_कय_`
+2. `_प्रदेश_प्रान्त_के_झांसी_ललितपुर_`
+3. `प्रदेश_कय_लिए_रखे_गये_और_`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 79.9% predictability
+- **Best Predictability:** Context-4 (word) with 98.6% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (214,041 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (308,778 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -243,64 +310,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 6,221 |
-| Total Tokens | 624,335 |
-| Mean Frequency | 100.36 |
-| Median Frequency | 4 |
-| Frequency Std Dev | 1247.50 |
+| Vocabulary Size | 15,883 |
+| Total Tokens | 248,637 |
+| Mean Frequency | 15.65 |
+| Median Frequency | 3 |
+| Frequency Std Dev | 134.68 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | र | 42,910 |
-| 2 | क | 40,728 |
-| 3 | स | 27,096 |
-| 4 | न | 26,749 |
-| 5 | म | 24,046 |
-| 6 | ल | 23,543 |
-| 7 | य | 23,175 |
-| 8 | त | 22,215 |
-| 9 | प | 20,708 |
-| 10 | ह | 20,269 |
+| 1 | कय | 10,552 |
+| 2 | के | 6,740 |
+| 3 | में | 4,036 |
+| 4 | से | 4,015 |
+| 5 | है | 3,785 |
+| 6 | मा | 3,358 |
+| 7 | होय | 2,646 |
+| 8 | का | 2,496 |
+| 9 | प्रदेश | 2,219 |
+| 10 | भारत | 1,992 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | dish | 2 |
-| 2 | uzbekistan | 2 |
-| 3 | travel | 2 |
-| 4 | lagman | 2 |
-| 5 | उबलत | 2 |
-| 6 | रगमन | 2 |
-| 7 | एमई | 2 |
-| 8 | उचक | 2 |
-| 9 | शबरक | 2 |
-| 10 | एसएन | 2 |
+| 1 | दृश्यता | 2 |
+| 2 | दुर्घटनाग्रस्त | 2 |
+| 3 | परिवारन | 2 |
+| 4 | फेडरल | 2 |
+| 5 | टेरिटरी | 2 |
+| 6 | कुआला | 2 |
+| 7 | लुंपुर | 2 |
+| 8 | सेतापाक | 2 |
+| 9 | पेटलिंग | 2 |
+| 10 | ब्रुनेई | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.4504 |
-| R² (Goodness of Fit) | 0.991083 |
+| Zipf Coefficient | 1.0489 |
+| R² (Goodness of Fit) | 0.990725 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 81.6% |
-| Top 1,000 | 95.9% |
-| Top 5,000 | 99.6% |
-| Top 10,000 | 0.0% |
+| Top 100 | 38.4% |
+| Top 1,000 | 66.7% |
+| Top 5,000 | 87.7% |
+| Top 10,000 | 95.3% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9911 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 81.6% of corpus
-- **Long Tail:** -3,779 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9907 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 38.4% of corpus
+- **Long Tail:** 5,883 words needed for remaining 4.7% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -313,24 +380,76 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 6,567 | 32 | 3.456 | 0.807 | 0.7531 🏆 |
-| **mono_64d** | 6,567 | 64 | 3.529 | 0.788 | 0.3641 |
-| **mono_128d** | 6,567 | 128 | 3.540 | 0.789 | 0.0857 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.7129 🏆 | 0.3782 | N/A | N/A |
+| **mono_64d** | 64 | 0.3226 | 0.3543 | N/A | N/A |
+| **mono_128d** | 128 | 0.0790 | 0.3513 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.7531 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 6,567 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.7129 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3612. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+*No productive affixes detected.*
+
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+*No significant bound stems detected.*
+
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+*No significant affix co-occurrences detected.*
+
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+*Insufficient data for recursive segmentation.*
+
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language AWA appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -338,11 +457,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.12x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (675) |
-| Markov | **Context-4** | Highest predictability (79.9%) |
+| Tokenizer | **32k BPE** | Best compression (3.90x) |
+| N-gram | **3-gram** | Lowest perplexity (1,558) |
+| Markov | **Context-4** | Highest predictability (98.6%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -532,7 +652,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -548,7 +669,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 20:46:37*
+*Report Date: 2026-01-03 05:27:10*

models/embeddings/monolingual/awa_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2134a3d39707a0e61b22d795679798fa20b422fdbb70c34dcdbafce442c55188
-size 1030890467
+oid sha256:d8d0ee1e312f5e3791814211acf276b13b65aa885e4e5a95c71cac289f30625f
+size 1030147731

models/embeddings/monolingual/awa_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 6567
+  "vocab_size": 5860
 }

models/embeddings/monolingual/awa_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:03b687fb388d3d1b50f93a70036dcb1f05b12ed00918b4e3cdaedd42b14ce46f
-size 257847011
+oid sha256:d216891b957241c5e5afaef2d177855d2cb1d66c5343414a5fc2f4199c807971
+size 257647251

models/embeddings/monolingual/awa_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 6567
+  "vocab_size": 5860
 }

models/embeddings/monolingual/awa_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3760ee4214954055389216fc2c5c800072a4c4bd87d656c91aee6cf8a781448c
-size 515528163
+oid sha256:5151750f712ffaa4369a74c3e8f893fe2f8fadabed61f754c75e3773fba76476
+size 515147411

models/embeddings/monolingual/awa_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 6567
+  "vocab_size": 5860
 }

models/subword_markov/awa_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:71e90ab50ff08b4a4acc795b7fe73231a08bab2bd54daf823bbd44866515fa80
-size 75933
+oid sha256:271a40da9bb401252516694f1ddd0ed6e63fe5157677cf157cc751ab43de7160
+size 249317

models/subword_markov/awa_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "awa",
-  "unique_contexts": 676,
-  "total_transitions": 1804933
+  "unique_contexts": 3632,
+  "total_transitions": 1016594
 }

models/subword_markov/awa_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:67bd149dfcaf7d3495dc092a09462870ca5d0acdf88fed781138c910ca04903a
-size 448892
+oid sha256:98dd04397a244bec649c1467886404517928646d46bd56616366077081306470
+size 1046154

models/subword_markov/awa_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "awa",
-  "unique_contexts": 9605,
-  "total_transitions": 1801177
+  "unique_contexts": 38845,
+  "total_transitions": 1013808
 }

models/subword_markov/awa_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:761708f5e394a72fdac5044131d7b0e80264ee9745dd751ff90dfabafbd77a70
-size 1500356
+oid sha256:e59e71dcd671f534598174056219214277dba9269677f32b105b8f2259fd6891
+size 2860240

models/subword_markov/awa_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "awa",
-  "unique_contexts": 65205,
-  "total_transitions": 1797421
+  "unique_contexts": 134413,
+  "total_transitions": 1011022
 }

models/subword_markov/awa_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:622ff6876bbeef23f56e6323ef2265f4c833469295162d61d2bab7972e7c25f6
-size 3938365
+oid sha256:e144785343170a9509d1043785f2dd7bbb7df7fe73020aa276c66c93967a4d02
+size 5625952

models/subword_markov/awa_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "awa",
-  "unique_contexts": 214041,
-  "total_transitions": 1793665
+  "unique_contexts": 308778,
+  "total_transitions": 1008236
 }

models/subword_ngram/awa_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9261e6dccd50b7a4d81414b6f34f22dd3cc9f7a9ac446f9bf930c93c8e87cc48
-size 57668
+oid sha256:989cbc777e82e6057dffa1dead21d90a773e1d56a5764b6000da9a4e3b842dd1
+size 167016

models/subword_ngram/awa_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "awa",
-  "unique_ngrams": 4587,
-  "total_ngrams": 1804933
+  "unique_ngrams": 11871,
+  "total_ngrams": 1016594
 }

models/subword_ngram/awa_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8f766ae156ebafb96c712347ee1e7482db9e1521b9ba78cb3b31186aa751a422
-size 401856
+oid sha256:665623a662a43f315f72ad68154c03493d7df87f55876e69e0b3f72a6d291f5b
+size 629586

models/subword_ngram/awa_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "awa",
-  "unique_ngrams": 31251,
-  "total_ngrams": 1801177
+  "unique_ngrams": 42588,
+  "total_ngrams": 1013808
 }

models/subword_ngram/awa_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c54094aa0dc468f8b2ca51287495a31a126e91650ed7c84d3b1eab6859776be4
-size 1329394
+oid sha256:bec67a4e37346930b803666f70ab5257ee999e9bd39016ba78195b60ab06622a
+size 1555001

models/subword_ngram/awa_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "awa",
-  "unique_ngrams": 106978,
-  "total_ngrams": 1797421
+  "unique_ngrams": 105286,
+  "total_ngrams": 1011022
 }

models/tokenizer/awa_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:549bf3a2260ee062c6eecdc391eb803651a68253dfcc2f2f71e68654141faf5e
-size 612493
+oid sha256:864217b8d0193812cc5aee4fe1cab467d78e99fcae9dacce41a60305c3571ac8
+size 614761

models/tokenizer/awa_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c75db6c318fa63d3b1714875e57903d644a0773f7d91be210b5661b08ef4af33
-size 1019677
+oid sha256:62f7c3149d9e99f10f8cb52328bf87eac31e23ec3a63f7d084f6baeadff4767b
+size 1049188

models/tokenizer/awa_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0ecfd9f4200c7454b5e2a5f293587bb0d8938b384947708ba46f4a2771e724b4
-size 419840
+oid sha256:20d7fcc006f3a667e8dd54ecedcd7aa6ed9e5eafd6c3a512d492148c0bd08496
+size 423049

models/vocabulary/awa_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4b185a179a70f7ce1d5211d042ffa3ecf2ac042388e23ce966a8ac6d8a5c8007
-size 98312
+oid sha256:f5554520a8312b47703ae8b6767875831aa218ec61d7d682949e0bb40e61df61
+size 285293

models/vocabulary/awa_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "awa",
-  "vocabulary_size": 6221,
+  "vocabulary_size": 15883,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.02175327090379341,
+    "type_token_ratio": 0.13851608939291873,
     "coverage": {
-      "top_100": 0.8062510781677558,
-      "top_1000": 0.9471891672034426,
-      "top_5000": 0.9842274937921277,
-      "top_10000": 0.994073044777395
+      "top_100": 0.3530968472636558,
+      "top_1000": 0.6138212585776784,
+      "top_5000": 0.806893973602588,
+      "top_10000": 0.8767220128952025
     },
-    "hapax_count": 7524,
-    "hapax_ratio": 0.5473990542015278,
-    "total_documents": 3756
+    "hapax_count": 21541,
+    "hapax_ratio": 0.5755932022231723,
+    "total_documents": 2786
   }
 }

models/word_markov/awa_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:547a1d0bc04c5bdc57b9da50b12b1314c9e27ab160055111852a3e9b5f86815d
-size 568986
+oid sha256:73e2ba019d301bda283de84c77aff4fd0b8a4ed584b5f2007f9f1446f3ef1ea5
+size 1486796

models/word_markov/awa_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "awa",
-  "unique_contexts": 13918,
-  "total_transitions": 1156757
+  "unique_contexts": 37356,
+  "total_transitions": 267392
 }

models/word_markov/awa_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9f4ecc3b5dd1ec25a7362262b1112c268585d46537378074b220fffb50274064
-size 1749043
+oid sha256:e865dc423de365ded66b4a30914867dc27d054b14d58587d4a8bc7cb878c3b27
+size 3675021

models/word_markov/awa_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "awa",
-  "unique_contexts": 80553,
-  "total_transitions": 1153001
+  "unique_contexts": 155195,
+  "total_transitions": 264606
 }

models/word_markov/awa_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e8548a6290b983ef65f7e70de8617349551f2d343de81987a732f9aa2ffdfebc
-size 3946373
+oid sha256:731b656111acaf4fe5a35a1088862f6130bc2ed9424e8943b4dd5a97d7d29f6a
+size 4965463

models/word_markov/awa_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "awa",
-  "unique_contexts": 210504,
-  "total_transitions": 1149246
+  "unique_contexts": 209159,
+  "total_transitions": 261820
 }

models/word_markov/awa_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:29431e78a3780d03cfd535f409c832282511511ed9034a9a7724e9b83dee3c4d
-size 6740784
+oid sha256:e7bb59c8174451a6504aaa814138aecc464627ce55296f151295e8b7ee27816b
+size 5813922

models/word_markov/awa_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "awa",
-  "unique_contexts": 389838,
-  "total_transitions": 1145491
+  "unique_contexts": 221759,
+  "total_transitions": 259034
 }

models/word_ngram/awa_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7205d14fde081500a186427394bfbbaedc11bd03fbf38f7ae2709da963be8a62
-size 163970
+oid sha256:a54a18e0f5247b5c0bf4e50fe9f34313cecca88bc41ee7da5ddfdaf474e83d4c
+size 107508

models/word_ngram/awa_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "awa",
-  "unique_ngrams": 12281,
-  "total_ngrams": 1156757
+  "unique_ngrams": 5376,
+  "total_ngrams": 267392
 }

models/word_ngram/awa_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4d7fb7e5a2eb6cde2fc5e39665a97baa8467228e79b3fd1c7b9c908c0bcfa508
-size 558805
+oid sha256:822c838751c508caa1e7185abe59e79cb3e1b95ef96676dc3b46acc2f1c3d855
+size 120081

models/word_ngram/awa_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "awa",
-  "unique_ngrams": 41199,
-  "total_ngrams": 1153001
+  "unique_ngrams": 4851,
+  "total_ngrams": 264606
 }

models/word_ngram/awa_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e8b7a81bf9eba4f02d1f34643df98122eecc61f76c303f73ae4798386433fc7c
-size 1432471
+oid sha256:f925ca031e547cd749d95ce0076ff8c6e7a5d5cd632ba9e9345cea5d6377044f
+size 316670

models/word_ngram/awa_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "awa",
-  "unique_ngrams": 106862,
-  "total_ngrams": 1149246
+  "unique_ngrams": 12076,
+  "total_ngrams": 261820
 }