Upload all models and assets for awa (latest)

.gitattributes

@@ -38,3 +38,4 @@ visualizations/performance_dashboard.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: awa
-language_name: AWA
+language_name: Awadhi
 language_family: indoaryan_central
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-indoaryan_central
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 3.897
+    value: 3.892
   - name: best_isotropy
     type: isotropy
-    value: 0.7129
+    value: 0.7358
   - name: vocabulary_size
     type: vocab
     value: 0
 generated: 2026-01-03
 ---
 
-# AWA - Wikilangs Models
+# Awadhi - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **AWA** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Awadhi** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -60,7 +70,7 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
 - [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
@@ -80,43 +90,43 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.327x | 3.33 | 0.1233% | 131,409 |
-| **16k** | 3.614x | 3.62 | 0.1339% | 120,950 |
-| **32k** | 3.897x 🏆 | 3.91 | 0.1444% | 112,178 |
+| **8k** | 3.327x | 3.33 | 0.1230% | 131,731 |
+| **16k** | 3.618x | 3.63 | 0.1337% | 121,145 |
+| **32k** | 3.892x 🏆 | 3.90 | 0.1439% | 112,611 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `मानवशास्त्र या नृविज्ञान (:en:Anthropology) मनईन, वनकय जेनेटिक्स, संस्कृति अउर स...`
+**Sample 1:** `नीलम संजीव रेड्डी (२७ अक्तूबर - ९ नवंबर भारत कय छठवा राष्ट्रपति रहे। वनकय कार्यक...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁मानव शास्त्र ▁या ▁न ृ विज्ञान ▁(: en : an ... (+25 more)` | 35 |
-| 16k | `▁मानवशास्त्र ▁या ▁नृ विज्ञान ▁(: en : an throp ology ... (+23 more)` | 33 |
-| 32k | `▁मानवशास्त्र ▁या ▁नृविज्ञान ▁(: en : anthropology ) ▁मनईन , ... (+16 more)` | 26 |
+| 8k | `▁नीलम ▁सं जीव ▁रेड्डी ▁( २७ ▁अक्तूबर ▁- ▁९ ▁नवंबर ... (+16 more)` | 26 |
+| 16k | `▁नीलम ▁संजीव ▁रेड्डी ▁( २७ ▁अक्तूबर ▁- ▁९ ▁नवंबर ▁भारत ... (+15 more)` | 25 |
+| 32k | `▁नीलम ▁संजीव ▁रेड्डी ▁( २७ ▁अक्तूबर ▁- ▁९ ▁नवंबर ▁भारत ... (+15 more)` | 25 |
 
-**Sample 2:** `सिरसा, भारत देश के हरियाणा राज्य कय एक्ठु जिला अव नगर परिषद होय । कय नगर परिषद म...`
+**Sample 2:** `नकुड, भारत देश के उत्तर प्रदेश प्रान्त के सहारनपुर जिला कय एक्ठु नगर पालिका परिष...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁सिरसा , ▁भारत ▁देश ▁के ▁हरियाणा ▁राज्य ▁कय ▁एक्ठु ▁जिला ... (+11 more)` | 21 |
-| 16k | `▁सिरसा , ▁भारत ▁देश ▁के ▁हरियाणा ▁राज्य ▁कय ▁एक्ठु ▁जिला ... (+11 more)` | 21 |
-| 32k | `▁सिरसा , ▁भारत ▁देश ▁के ▁हरियाणा ▁राज्य ▁कय ▁एक्ठु ▁जिला ... (+11 more)` | 21 |
+| 8k | `���न कु ड , ▁भारत ▁देश ▁के ▁उत्तर ▁प्रदेश ▁प्रान्त ... (+20 more)` | 30 |
+| 16k | `▁न कु ड , ▁भारत ▁देश ▁के ▁उत्तर ▁प्रदेश ▁प्रान्त ... (+20 more)` | 30 |
+| 32k | `▁नकुड , ▁भारत ▁देश ▁के ▁उत्तर ▁प्रदेश ▁प्रान्त ▁के ▁सहारनपुर ... (+18 more)` | 28 |
 
-**Sample 3:** `अनूपशहर, भारत देश के उत्तर प्रदेश प्रान्त के बुलंदशहर जिला कय एक्ठु नगर पालिका प...`
+**Sample 3:** `नसीराबाद, भारत देश के उत्तर प्रदेश प्रान्त के रायबरेली जिला कय एक्ठु नगर पंचायत ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁अन ूप श हर , ▁भारत ▁देश ▁के ▁उत्तर ▁प्रदेश ... (+21 more)` | 31 |
-| 16k | `▁अनूपश हर , ▁भारत ▁देश ▁के ▁उत्तर ▁प्रदेश ▁प्रान्त ▁के ... (+19 more)` | 29 |
-| 32k | `▁अनूपशहर , ▁भारत ▁देश ▁के ▁उत्तर ▁प्रदेश ▁प्रान्त ▁के ▁बुलंदशहर ... (+18 more)` | 28 |
+| 8k | `▁न सी राबाद , ▁भारत ▁देश ▁के ▁उत्तर ▁प्रदेश ▁प्रान्त ... (+18 more)` | 28 |
+| 16k | `▁न सी राबाद , ▁भारत ▁देश ▁के ▁उत्तर ▁प्रदेश ▁प्रान्त ... (+18 more)` | 28 |
+| 32k | `▁नसीराबाद , ▁भारत ▁देश ▁के ▁उत्तर ▁प्रदेश ▁प्रान्त ▁के ▁रायबरेली ... (+16 more)` | 26 |
 
 
 ### Key Findings
 
-- **Best Compression:** 32k achieves 3.897x compression
-- **Lowest UNK Rate:** 8k with 0.1233% unknown tokens
+- **Best Compression:** 32k achieves 3.892x compression
+- **Lowest UNK Rate:** 8k with 0.1230% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -133,12 +143,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
 |--------|---------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | Word | 2,211 | 11.11 | 5,376 | 29.5% | 59.6% |
-| **2-gram** | Subword | 1,584 | 10.63 | 11,871 | 40.0% | 73.5% |
-| **3-gram** | Word | 1,558 🏆 | 10.61 | 4,851 | 36.7% | 66.9% |
-| **3-gram** | Subword | 9,994 | 13.29 | 42,588 | 17.4% | 41.6% |
-| **4-gram** | Word | 3,905 | 11.93 | 12,076 | 28.3% | 51.2% |
-| **4-gram** | Subword | 29,097 | 14.83 | 105,286 | 12.1% | 28.9% |
+| **2-gram** | Word | 2,396 | 11.23 | 5,750 | 28.4% | 58.2% |
+| **2-gram** | Subword | 1,608 🏆 | 10.65 | 12,278 | 39.9% | 73.3% |
+| **3-gram** | Word | 1,666 | 10.70 | 5,103 | 35.8% | 65.6% |
+| **3-gram** | Subword | 10,335 | 13.34 | 44,364 | 17.1% | 41.3% |
+| **4-gram** | Word | 4,269 | 12.06 | 12,850 | 27.4% | 49.6% |
+| **4-gram** | Subword | 30,718 | 14.91 | 110,971 | 11.6% | 28.3% |
+| **5-gram** | Word | 3,586 | 11.81 | 10,699 | 28.4% | 52.8% |
+| **5-gram** | Subword | 44,082 | 15.43 | 123,963 | 10.3% | 23.7% |
 
 ### Top 5 N-grams by Size
 
@@ -146,11 +158,11 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `प्रदेश कय` | 1,241 |
+| 1 | `प्रदेश कय` | 1,242 |
 | 2 | `कय एक्ठु` | 1,217 |
 | 3 | `नगर पंचायत` | 932 |
 | 4 | `शहरी निकाय` | 837 |
-| 5 | `उत्तर प्रदेश` | 773 |
+| 5 | `उत्तर प्रदेश` | 774 |
 
 **3-grams (Word):**
 
@@ -168,46 +180,66 @@ Below are sample sentences tokenized with each vocabulary size:
 |------|--------|-------|
 | 1 | `जिला कय एक्ठु नगर` | 661 |
 | 2 | `के उत्तर प्रदेश प्रान्त` | 582 |
-| 3 | `निकाय प्रदेश कय नगर` | 581 |
-| 4 | `शहरी निकाय प्रदेश कय` | 581 |
-| 5 | `कय शहरी निकाय प्रदेश` | 581 |
+| 3 | `प्रदेश कय शहरी निकाय` | 581 |
+| 4 | `कय शहरी निकाय प्रदेश` | 581 |
+| 5 | `शहरी निकाय प्रदेश कय` | 581 |
+
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `शहरी निकाय प्रदेश कय नगर` | 581 |
+| 2 | `कय शहर�� निकाय प्रदेश कय` | 581 |
+| 3 | `प्रदेश कय शहरी निकाय प्रदेश` | 581 |
+| 4 | `देश के उत्तर प्रदेश प्रान्त` | 580 |
+| 5 | `भारत देश के उत्तर प्रदेश` | 580 |
 
 **2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `र _` | 18,112 |
-| 2 | `य _` | 17,719 |
-| 3 | `_ क` | 16,272 |
-| 4 | `न _` | 12,852 |
-| 5 | `। _` | 11,559 |
+| 1 | `र _` | 19,312 |
+| 2 | `य _` | 17,947 |
+| 3 | `_ क` | 16,677 |
+| 4 | `न _` | 14,033 |
+| 5 | `। _` | 12,197 |
 
 **3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `क य _` | 10,797 |
-| 2 | `_ क य` | 10,549 |
-| 3 | `_ के _` | 6,719 |
-| 4 | `_ से _` | 3,956 |
-| 5 | `_ में _` | 3,886 |
+| 1 | `क य _` | 10,878 |
+| 2 | `_ क य` | 10,634 |
+| 3 | `_ के _` | 7,599 |
+| 4 | `_ से _` | 4,267 |
+| 5 | `_ में _` | 4,065 |
 
 **4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `_ क य _` | 10,506 |
-| 2 | `_ प्र दे श` | 2,241 |
-| 3 | `प्र दे श _` | 2,190 |
-| 4 | `_ है । _` | 2,071 |
-| 5 | `भा र त _` | 2,019 |
+| 1 | `_ क य _` | 10,589 |
+| 2 | `_ प्र दे श` | 2,239 |
+| 3 | `प्र दे श _` | 2,188 |
+| 4 | `_ है । _` | 2,147 |
+| 5 | `भा र त _` | 2,022 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ प्र दे श _` | 2,171 |
+| 2 | `_ भा र त _` | 1,826 |
+| 3 | `_ न ग र _` | 1,779 |
+| 4 | `_ क य _ ए` | 1,494 |
+| 5 | `_ अ उ र _` | 1,449 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 3-gram (word) with 1,558
+- **Best Perplexity:** 2-gram (subword) with 1,608
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~29% of corpus
+- **Coverage:** Top-1000 patterns cover ~24% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -223,14 +255,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
 |---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.7301 | 1.659 | 4.17 | 37,356 | 27.0% |
-| **1** | Subword | 1.0434 | 2.061 | 10.70 | 3,632 | 0.0% |
-| **2** | Word | 0.1929 | 1.143 | 1.36 | 155,195 | 80.7% |
-| **2** | Subword | 0.5412 | 1.455 | 3.46 | 38,845 | 45.9% |
-| **3** | Word | 0.0474 | 1.033 | 1.07 | 209,159 | 95.3% |
-| **3** | Subword | 0.4514 | 1.367 | 2.30 | 134,413 | 54.9% |
-| **4** | Word | 0.0142 🏆 | 1.010 | 1.02 | 221,759 | 98.6% |
-| **4** | Subword | 0.2387 | 1.180 | 1.51 | 308,778 | 76.1% |
+| **1** | Word | 0.7360 | 1.666 | 4.24 | 38,944 | 26.4% |
+| **1** | Subword | 1.0397 | 2.056 | 10.73 | 3,744 | 0.0% |
+| **2** | Word | 0.1950 | 1.145 | 1.36 | 164,372 | 80.5% |
+| **2** | Subword | 0.5443 | 1.458 | 3.48 | 40,149 | 45.6% |
+| **3** | Word | 0.0479 | 1.034 | 1.07 | 222,536 | 95.2% |
+| **3** | Subword | 0.4540 | 1.370 | 2.32 | 139,753 | 54.6% |
+| **4** | Word | 0.0142 🏆 | 1.010 | 1.02 | 236,208 | 98.6% |
+| **4** | Subword | 0.2417 | 1.182 | 1.52 | 323,693 | 75.8% |
 
 ### Generated Text Samples (Word-based)
 
@@ -238,27 +270,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `कय एक्ठु राजनीतिक पार्टी पाकिस्तान कय एक्ठु जिला चुराचांदपुर जिला कय एक्ठो नगरपालिका सप्तरी जिला कय`
-2. `के नाम से गुवाहाटी सिलचर एन यू कि सिक्‍ख गुरू योगी आदित्यनाथ होइ सन्दर्भ कय शहरी`
-3. `में मौसम रहत है शरीर का अड्डा bho इहो देखा जाय रहा साथ जोश और निम्नतम`
+1. `कय सुविधाजनक बनावेक अन्तर्राष्ट्रीय हवाईगिरान फाप्लु भोजपुर फर्रुखाबाद 195 कासगंज जिला आवत हैं मेघाल...`
+2. `के उत्तर भारतीय रुपया लेख आसानी से खेले रहें घरेलू क्रिकेट रहें आदित्यनाथ कय राजनीति में`
+3. `से दक्षिण दिल्ली मेट्रो फ़िल्मफ़ेयर सर्वश्रेष्ठ तमिल तेलुगू వికారాబాదు జిల్లా अंग्रेज़ी में गंगा नदी...`
 
 **Context Size 2:**
 
-1. `प्रद���श कय मंडल होय एहमा 05 जिला आवत हँय फतेहाबाद जींद हिसार महेंद्रगढ़ गुड़गांव रोहतक और हिसार`
-2. `कय एक्ठु नगर पंचायत के पार्षद चुनाव में ६ राष्ट्रीय दल चुनाव लड़ रहे हैं भारतीय जनता`
-3. `उत्तर प्रदेश प्रान्त के बिजनौर जिला कय मुख्यालय अहै एह समाज मा खुदै आंतरिक सुधार कइके आपन`
+1. `प्रदेश कय शहरी निकाय प्रदेश कय नगर पंचायत होय संदर्भ प्रदेश कय शहरी निकाय प्रदेश कय नगर`
+2. `कय एक्ठु भाषा होय ई ईलेक्ट्रोन प्रोटोन अव न्युट्रोन से बना है हिमालय क्षेत्र में मनुष्यों का`
+3. `उत्तर प्रदेश प्रान्त के शामली जिला कय एक्ठु नगर पालिका परिषद कय शहरी निकाय प्रदेश कय नगर`
 
 **Context Size 3:**
 
 1. `कय एक्ठु नगर पंचायत होय संदर्भ प्रदेश कय शहरी निकाय प्रदेश कय नगर पंचायत पंचायत कय शहरी निकाय`
-2. `भारत देश के उत्तर प्रदेश प्रान्त के आजमगढ़ जिला कय एक्ठु नगर पालिका होय संदर्भ कय नगर पालिका`
-3. `जिला कय एक्ठु नगर पालिका परिषद पालिका परिषद कय शहरी निकाय प्रदेश कय नगर पंचायत नगर`
+2. `भारत देश के उत्तर प्रदेश प्रान्त कय एक्ठु जिला होय इहौ देखैं कामारेड्डी तेलंगाना तेलंगाना कय जिला सन...`
+3. `जिला कय एक्ठु नगर पालिका परिषद होय संदर्भ 1 उत्तराखंड के सगरौ शहरी निकाय कय सूची 2 उत्तराखंड`
 
 **Context Size 4:**
 
-1. `जिला कय एक्ठु नगर पालिका परिषद होय संदर्भ प्रदेश कय शहरी निकाय प्रदेश कय नगर पालिका परिषद खीरी`
-2. `के उत्तर प्रदेश प्रान्त के आजमगढ़ जिला कय एक्ठु नगर पालिका परिषद होय संदर्भ प्रदेश कय शहरी निकाय प्र...`
-3. `शहरी निकाय प्रदेश कय नगर पालिका परिषद पालिका परिषद कय शहरी निकाय`
+1. `जिला कय एक्ठु नगर पंचायत होय संदर्भ प्रदेश कय शहरी निकाय प्रदेश कय नगर पंचायत noinclude`
+2. `के उत्तर प्रदेश प्रान्त के सीतापुर जिला कय एक्ठु नगर पालिका परिषद होय संदर्भ प्रदेश कय शहरी निकाय प्...`
+3. `निकाय प्रदेश कय नगर पंचायत देहात`
 
 
 ### Generated Text Samples (Subword-based)
@@ -267,34 +299,34 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_की_संग_कब्ज़ा.दिनसंलतः_इ`
-2. `र_५_के_कार_एथिरत_न,`
-3. `क_सिद्ध_भा_ए_आत्रेयत_प्रदे`
+1. `_के_हंयन_ह_सहइ_पालव`
+2. `रतह_रें।_केर_प_10_प्रा`
+3. `कय_की।_के_इति_-atem`
 
 **Context Size 2:**
 
-1. `र_पता_है।_विश्व_प्रथम_ई_`
-2. `य_छात्र_में_बने_मा_स्पेस),_`
-3. `_कय_राष्ट्रीय_है_सें._पुर_मा`
+1. `र_हरा_गांव_परिषद_पार्टी_(`
+2. `य_संगीत-होल्सटीन,_आंध्रप्रदेश`
+3. `_कय_जन्म_३_मद्रास)_शिक्षा_`
 
 **Context Size 3:**
 
-1. `कय_हाइड्रोकार्बन_कलायत_राज��य_`
-2. `_कय_लेकिन_वाली_एक_लोचनो_`
-3. `_के_प्रयाग,भारत_कय_क्रिस_मॉ`
+1. `कय_निकोसिया_का_यश_चोपड़ा_आ`
+2. `_कय_शहर_सिरसा_16_44_`
+3. `_के_भेस_अनुवादित_तब_ओका_`
 
 **Context Size 4:**
 
-1. `_कय_शहरी_निकाय_प्रदेश_कय_`
-2. `_प्रदेश_प्रान्त_के_झांसी_ललितपुर_`
-3. `प्रदेश_कय_लिए_रखे_गये_और_`
+1. `_कय_१५वाँ_राष्ट्रपति_रहे।_यह`
+2. `_प्रदेश_कय_भी_अविवाहित_भाई_`
+3. `प्रदेश_प्रान्त_के_गाजियाबाद_जिला_क`
 
 
 ### Key Findings
 
 - **Best Predictability:** Context-4 (word) with 98.6% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (308,778 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (323,693 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -310,64 +342,64 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 15,883 |
-| Total Tokens | 248,637 |
-| Mean Frequency | 15.65 |
+| Vocabulary Size | 16,641 |
+| Total Tokens | 263,395 |
+| Mean Frequency | 15.83 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 134.68 |
+| Frequency Std Dev | 138.02 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | कय | 10,552 |
-| 2 | के | 6,740 |
-| 3 | में | 4,036 |
-| 4 | से | 4,015 |
-| 5 | है | 3,785 |
-| 6 | मा | 3,358 |
-| 7 | होय | 2,646 |
-| 8 | का | 2,496 |
-| 9 | प्रदेश | 2,219 |
-| 10 | भारत | 1,992 |
+| 1 | कय | 10,633 |
+| 2 | के | 7,622 |
+| 3 | से | 4,333 |
+| 4 | में | 4,224 |
+| 5 | है | 3,954 |
+| 6 | मा | 3,849 |
+| 7 | होय | 2,668 |
+| 8 | का | 2,628 |
+| 9 | प्रदेश | 2,217 |
+| 10 | भारत | 1,996 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | दृश्यता | 2 |
-| 2 | दुर्घटनाग्रस्त | 2 |
-| 3 | परिवारन | 2 |
-| 4 | फेडरल | 2 |
-| 5 | टेरिटरी | 2 |
-| 6 | कुआला | 2 |
-| 7 | लुंपुर | 2 |
-| 8 | सेतापाक | 2 |
-| 9 | पेटलिंग | 2 |
-| 10 | ब्रुनेई | 2 |
+| 1 | मोड़ा | 2 |
+| 2 | कीमा | 2 |
+| 3 | चौकोरन | 2 |
+| 4 | दर्रे | 2 |
+| 5 | गिजर | 2 |
+| 6 | तड़हुंग | 2 |
+| 7 | कलाकृति | 2 |
+| 8 | स्टेपी | 2 |
+| 9 | ओलेक्सान्ड्रोविच | 2 |
+| 10 | टीएसएन | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.0489 |
-| R² (Goodness of Fit) | 0.990725 |
+| Zipf Coefficient | 1.0518 |
+| R² (Goodness of Fit) | 0.990696 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 38.4% |
-| Top 1,000 | 66.7% |
-| Top 5,000 | 87.7% |
-| Top 10,000 | 95.3% |
+| Top 100 | 38.1% |
+| Top 1,000 | 66.2% |
+| Top 5,000 | 87.3% |
+| Top 10,000 | 94.8% |
 
 ### Key Findings
 
 - **Zipf Compliance:** R²=0.9907 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 38.4% of corpus
-- **Long Tail:** 5,883 words needed for remaining 4.7% coverage
+- **High Frequency Dominance:** Top 100 words cover 38.1% of corpus
+- **Long Tail:** 6,641 words needed for remaining 5.2% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -383,37 +415,40 @@ Below are text samples generated from each subword-based Markov chain model:
 
 ### 5.1 Cross-Lingual Alignment
 
-> *Note: Multilingual alignment visualization not available for this language.*
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
 
 
 ### 5.2 Model Comparison
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.7129 🏆 | 0.3782 | N/A | N/A |
-| **mono_64d** | 64 | 0.3226 | 0.3543 | N/A | N/A |
-| **mono_128d** | 128 | 0.0790 | 0.3513 | N/A | N/A |
+| **mono_32d** | 32 | 0.7358 | 0.3755 | N/A | N/A |
+| **mono_64d** | 64 | 0.3489 | 0.3581 | N/A | N/A |
+| **mono_128d** | 128 | 0.0808 | 0.3463 | N/A | N/A |
+| **aligned_32d** | 32 | 0.7358 🏆 | 0.3759 | 0.0299 | 0.1549 |
+| **aligned_64d** | 64 | 0.3489 | 0.3500 | 0.0245 | 0.1848 |
+| **aligned_128d** | 128 | 0.0808 | 0.3480 | 0.0571 | 0.2636 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.7129 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.3612. Lower values indicate better semantic separation.
-- **Alignment Quality:** No aligned models evaluated in this run.
+- **Best Isotropy:** aligned_32d with 0.7358 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3590. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 5.7% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
 ## 6.  Morphological Analysis (Experimental)
 
-> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
-
 This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
 
 ### 6.1 Productivity & Complexity
 
 | Metric | Value | Interpretation | Recommendation |
 |--------|-------|----------------|----------------|
-| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
-| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **1.225** | High formulaic/idiomatic content | - |
 
 ### 6.2 Affix Inventory (Productive Units)
 
@@ -446,7 +481,9 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 ### 6.6 Linguistic Interpretation
 
 > **Automated Insight:**
-The language AWA appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+The language Awadhi shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
 
 ---
 ## 7. Summary & Recommendations
@@ -457,8 +494,8 @@ The language AWA appears to be more isolating or has a highly fixed vocabulary.
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (3.90x) |
-| N-gram | **3-gram** | Lowest perplexity (1,558) |
+| Tokenizer | **32k BPE** | Best compression (3.89x) |
+| N-gram | **2-gram** | Lowest perplexity (1,608) |
 | Markov | **Context-4** | Highest predictability (98.6%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
@@ -673,4 +710,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 05:27:10*
+*Report Date: 2026-01-03 17:51:14*

models/embeddings/aligned/awa_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33423d1471f61bd0d2d2aff082d7e6ffeb7515e48273e7fc4805d25cf36b7f11
+size 1030479084

models/embeddings/aligned/awa_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "awa", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/awa_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55fc733f60ff1d12cf4454845b355c4c7baa4b10c87c1da05f1a06b05b80bb33
+size 65664

models/embeddings/aligned/awa_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "awa",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 368,
+  "vocab_size": 6176
+}

models/embeddings/aligned/awa_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67f44c513bc0af9e0f9003c103131c96f4dcc5c13fb3efd7adb32084338f6f53
+size 257735916

models/embeddings/aligned/awa_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "awa", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/awa_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de3d170455f7a78cf21e4c69c64be2249d54797f61ec5deabab6798630c129ef
+size 4224

models/embeddings/aligned/awa_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "awa",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 368,
+  "vocab_size": 6176
+}

models/embeddings/aligned/awa_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c368cd6c1b7b02e5cd7b623d5a63d731e44cc545b6346ec167e414c422193ef0
+size 515316972

models/embeddings/aligned/awa_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "awa", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/awa_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe1b61938cfb6fdcb7ba0e88e010272b1ac2f0a47f995d29f49b95e1f43d90f0
+size 16512

models/embeddings/aligned/awa_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "awa",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 368,
+  "vocab_size": 6176
+}

models/embeddings/monolingual/awa_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d8d0ee1e312f5e3791814211acf276b13b65aa885e4e5a95c71cac289f30625f
-size 1030147731
+oid sha256:33423d1471f61bd0d2d2aff082d7e6ffeb7515e48273e7fc4805d25cf36b7f11
+size 1030479084

models/embeddings/monolingual/awa_128d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 128
   },
-  "vocab_size": 5860
+  "vocab_size": 6176
 }

models/embeddings/monolingual/awa_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d216891b957241c5e5afaef2d177855d2cb1d66c5343414a5fc2f4199c807971
-size 257647251
+oid sha256:67f44c513bc0af9e0f9003c103131c96f4dcc5c13fb3efd7adb32084338f6f53
+size 257735916

models/embeddings/monolingual/awa_32d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 32
   },
-  "vocab_size": 5860
+  "vocab_size": 6176
 }

models/embeddings/monolingual/awa_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5151750f712ffaa4369a74c3e8f893fe2f8fadabed61f754c75e3773fba76476
-size 515147411
+oid sha256:c368cd6c1b7b02e5cd7b623d5a63d731e44cc545b6346ec167e414c422193ef0
+size 515316972

models/embeddings/monolingual/awa_64d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 64
   },
-  "vocab_size": 5860
+  "vocab_size": 6176
 }

models/subword_markov/awa_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:271a40da9bb401252516694f1ddd0ed6e63fe5157677cf157cc751ab43de7160
-size 249317
+oid sha256:a46668aab429bf3c5448e9e6f480224db3f3e69cceaedddb6cfb5152600d3be8
+size 255963

models/subword_markov/awa_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "awa",
-  "unique_contexts": 3632,
-  "total_transitions": 1016594
+  "unique_contexts": 3744,
+  "total_transitions": 1074109
 }

models/subword_markov/awa_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:98dd04397a244bec649c1467886404517928646d46bd56616366077081306470
-size 1046154
+oid sha256:3b16197cd21d0bbe28751a389682e0f1e2038e13b5c0d110e1d07cbb04a884c9
+size 1087125

models/subword_markov/awa_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "awa",
-  "unique_contexts": 38845,
-  "total_transitions": 1013808
+  "unique_contexts": 40149,
+  "total_transitions": 1071282
 }

models/subword_markov/awa_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e59e71dcd671f534598174056219214277dba9269677f32b105b8f2259fd6891
-size 2860240
+oid sha256:e4d4847e0615b0492b6bb218220974d4f9d22cea1dbaccecc3de6b48c16a933c
+size 2979933

models/subword_markov/awa_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "awa",
-  "unique_contexts": 134413,
-  "total_transitions": 1011022
+  "unique_contexts": 139753,
+  "total_transitions": 1068455
 }

models/subword_markov/awa_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e144785343170a9509d1043785f2dd7bbb7df7fe73020aa276c66c93967a4d02
-size 5625952
+oid sha256:0767d992a1455f50e490b2411ea747e19e19c97a09f7aadc83ec86f386508568
+size 5954745

models/subword_markov/awa_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "awa",
-  "unique_contexts": 308778,
-  "total_transitions": 1008236
+  "unique_contexts": 323693,
+  "total_transitions": 1065628
 }

models/subword_ngram/awa_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:989cbc777e82e6057dffa1dead21d90a773e1d56a5764b6000da9a4e3b842dd1
-size 167016
+oid sha256:80db58945ad0f585c68cfb5879186746592a3d966ace9a65fc8856153d8d31f0
+size 172513

models/subword_ngram/awa_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "awa",
-  "unique_ngrams": 11871,
-  "total_ngrams": 1016594
+  "unique_ngrams": 12278,
+  "total_ngrams": 1074109
 }

models/subword_ngram/awa_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:665623a662a43f315f72ad68154c03493d7df87f55876e69e0b3f72a6d291f5b
-size 629586
+oid sha256:1e1efd961ae2b44bcfdb7c8d28304852c692cc87c80103e4eb5481301e48ca34
+size 655914

models/subword_ngram/awa_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "awa",
-  "unique_ngrams": 42588,
-  "total_ngrams": 1013808
+  "unique_ngrams": 44364,
+  "total_ngrams": 1071282
 }

models/subword_ngram/awa_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bec67a4e37346930b803666f70ab5257ee999e9bd39016ba78195b60ab06622a
-size 1555001
+oid sha256:fad54fab43fd32dbec4dcfa1aee4777c3983ffbb32b34060588c1299a733e5b8
+size 1637616

models/subword_ngram/awa_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "awa",
-  "unique_ngrams": 105286,
-  "total_ngrams": 1011022
+  "unique_ngrams": 110971,
+  "total_ngrams": 1068455
 }

models/subword_ngram/awa_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65cef4a3f75a583b30737877f4c14591069eabd97fd58bd9b6449a7d07b3d1a3
+size 1953232

models/subword_ngram/awa_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "awa",
+  "unique_ngrams": 123963,
+  "total_ngrams": 1065628
+}

models/tokenizer/awa_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:864217b8d0193812cc5aee4fe1cab467d78e99fcae9dacce41a60305c3571ac8
-size 614761
+oid sha256:ea20d33d577e819754bd24f12bf4babfff9146b52def22c3b43a7f341995c825
+size 615502

models/tokenizer/awa_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:62f7c3149d9e99f10f8cb52328bf87eac31e23ec3a63f7d084f6baeadff4767b
-size 1049188
+oid sha256:f36820eb38e2a29bcf6c717c20b34fdad2d38a37f5d6782983c84d68fcbcbe70
+size 1040140

models/tokenizer/awa_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:20d7fcc006f3a667e8dd54ecedcd7aa6ed9e5eafd6c3a512d492148c0bd08496
-size 423049
+oid sha256:7e8a64f5d62c8ad1c769a474ec77f18950d50a3e7cb8cdfa2ddadcb9d86aea3f
+size 420670

models/vocabulary/awa_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f5554520a8312b47703ae8b6767875831aa218ec61d7d682949e0bb40e61df61
-size 285293
+oid sha256:96440604fee5f0277af7079e3002ed8850b473c7335c2dec38ce70fc62eac3ed
+size 304056

models/vocabulary/awa_vocabulary_metadata.json

@@ -1,17 +1,17 @@
 {
   "language": "awa",
-  "vocabulary_size": 15883,
+  "vocabulary_size": 16641,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.13851608939291873,
+    "type_token_ratio": 0.13652031200243556,
     "coverage": {
-      "top_100": 0.3530968472636558,
-      "top_1000": 0.6138212585776784,
-      "top_5000": 0.806893973602588,
-      "top_10000": 0.8767220128952025
+      "top_100": 0.35092925355271953,
+      "top_1000": 0.6103923826054093,
+      "top_5000": 0.8042671127177036,
+      "top_10000": 0.8737048014641299
     },
-    "hapax_count": 21541,
-    "hapax_ratio": 0.5755932022231723,
-    "total_documents": 2786
+    "hapax_count": 22372,
+    "hapax_ratio": 0.5734498756824649,
+    "total_documents": 2827
   }
 }

models/word_markov/awa_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:73e2ba019d301bda283de84c77aff4fd0b8a4ed584b5f2007f9f1446f3ef1ea5
-size 1486796
+oid sha256:4c8efabccb591961cfeb20c412e8b2e010f1b9c1804f65a7299f87f494bf4f34
+size 1562018

models/word_markov/awa_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "awa",
-  "unique_contexts": 37356,
-  "total_transitions": 267392
+  "unique_contexts": 38944,
+  "total_transitions": 282940
 }

models/word_markov/awa_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e865dc423de365ded66b4a30914867dc27d054b14d58587d4a8bc7cb878c3b27
-size 3675021
+oid sha256:c2887b6f1e7f62c6add56ed25c36fced6e8921baa8609bfc02d0d8e07eb3517d
+size 3894000

models/word_markov/awa_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "awa",
-  "unique_contexts": 155195,
-  "total_transitions": 264606
+  "unique_contexts": 164372,
+  "total_transitions": 280113
 }

models/word_markov/awa_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:731b656111acaf4fe5a35a1088862f6130bc2ed9424e8943b4dd5a97d7d29f6a
-size 4965463
+oid sha256:6e09cad8a7244b5ca4e3d44c60170b2d7b311704df78a9c3f7412380094becb0
+size 5272718

models/word_markov/awa_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "awa",
-  "unique_contexts": 209159,
-  "total_transitions": 261820
+  "unique_contexts": 222536,
+  "total_transitions": 277286
 }