Upload all models and assets for anp (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.233
+    value: 3.779
   - name: best_isotropy
     type: isotropy
-    value: 0.8322
+    value: 0.8284
   - name: vocabulary_size
     type: vocab
-    value: 9567
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # ANP - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,54 +70,53 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.431x | 3.37 | 0.0938% | 441,405 |
-| **16k** | 3.755x | 3.69 | 0.1026% | 403,338 |
-| **32k** | 4.015x | 3.95 | 0.1098% | 377,196 |
-| **64k** | 4.233x 🏆 | 4.16 | 0.1157% | 357,719 |
+| **8k** | 3.293x | 3.29 | 0.1162% | 454,392 |
+| **16k** | 3.578x | 3.58 | 0.1263% | 418,207 |
+| **32k** | 3.779x 🏆 | 3.78 | 0.1334% | 395,905 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `मंजूषा कला अंगप्रदेश के एक बहुचर्चित लोकगाथा बिहुला विषहरी पर आधारित छै।`
+**Sample 1:** `मई ग्रेगोरी कैलंडर क 5मां महीना छेकै। इ उ सात महीना मँ सँ एक छेकै जेकरौ दिन सिनी...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁म ंज ूष ा ▁कला ▁अंग प्र देश ▁के ▁एक ... (+16 more)` | 26 |
-| 16k | `▁मंज ूष ा ▁कला ▁अंग प्रदेश ▁के ▁एक ▁बहु च ... (+13 more)` | 23 |
-| 32k | `▁मंज ूष ा ▁कला ▁अंगप्रदेश ▁के ▁एक ▁बहु चर्चित ▁लोकगाथा ... (+7 more)` | 17 |
-| 64k | `▁मंजूषा ▁कला ▁अंगप्रदेश ▁के ▁एक ▁बहुचर्चित ▁लोकगाथा ▁बिहुला ▁विष हरी ... (+4 more)` | 14 |
+| 8k | `▁मई ▁ग्रेगोरी ▁कैलंडर ▁क ▁ 5 मां ▁महीना ▁छेकै । ... (+24 more)` | 34 |
+| 16k | `▁मई ▁ग्रेगोरी ▁कैलंडर ▁क ▁ 5 मां ▁महीना ▁छेकै । ... (+24 more)` | 34 |
+| 32k | `▁मई ▁ग्रेगोरी ▁कैलंडर ▁क ▁ 5 मां ▁महीना ▁छेकै । ... (+24 more)` | 34 |
 
-**Sample 2:** `कार्बन के रासायनिक तत्व छेकै। इ ठोस अवस्था मँ पैलौ जाय वाला अधातु छेकै।
-
-एकरो दे...`
+**Sample 2:** `राजा महेश ठाकुर – ई. तक मधुबनी जिला के भउर (भौर) गांव म॑ छेलै, जे मधुबनी स॑ करीब...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁कार्बन ▁के ▁रासायनिक ▁तत्व ▁छेकै । ▁इ ▁ठोस ▁अवस्था ▁मँ ... (+13 more)` | 23 |
-| 16k | `▁कार्बन ▁के ▁रासायनिक ▁तत्व ▁छेकै । ▁इ ▁ठोस ▁अवस्था ▁मँ ... (+11 more)` | 21 |
-| 32k | `▁कार्बन ▁के ▁रासायनिक ▁तत्�� ▁छेकै । ▁इ ▁ठोस ▁अवस्था ▁मँ ... (+11 more)` | 21 |
-| 64k | `▁कार्बन ▁के ▁रासायनिक ▁तत्व ▁छेकै । ▁इ ▁ठोस ▁अवस्था ▁मँ ... (+11 more)` | 21 |
+| 8k | `▁राजा ▁महेश ▁ठाकुर ▁– ▁ई . ▁तक ▁मधुबनी ▁जिला ▁के ... (+28 more)` | 38 |
+| 16k | `▁राजा ▁महेश ▁ठाकुर ▁– ▁ई . ▁तक ▁मधुबनी ▁जिला ▁के ... (+27 more)` | 37 |
+| 32k | `▁राजा ▁महेश ▁ठाकुर ▁– ▁ई . ▁तक ▁मधुबनी ▁जिला ▁के ... (+24 more)` | 34 |
 
-**Sample 3:** `ब्रह्मपुत्र 
-श्रेणी:नद्दी`
+**Sample 3:** `पति पत्नी नंदा केरऽ ई. मं॑ बनलऽ हिंदी फ़िल्म छेकै.`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ब्रह्म पुत्र ▁श्रेणी : न द्दी` | 6 |
-| 16k | `▁ब्रह्मपुत्र ▁श्रेणी : न द्दी` | 5 |
-| 32k | `▁ब्रह्मपुत्र ▁श्रेणी : नद्दी` | 4 |
-| 64k | `▁ब्रह्मपुत्र ▁श्रेणी : नद्दी` | 4 |
+| 8k | `▁पति ▁पत्नी ▁नंदा ▁केरऽ ▁ई . ▁मं॑ ▁बनलऽ ▁हिंदी ▁फ़िल्म ... (+2 more)` | 12 |
+| 16k | `▁पति ▁पत्नी ▁नंदा ▁केरऽ ▁ई . ▁मं॑ ▁बनलऽ ▁हिंदी ▁फ़िल्म ... (+2 more)` | 12 |
+| 32k | `▁पति ▁पत्नी ▁नंदा ▁केरऽ ▁ई . ▁मं॑ ▁बनलऽ ▁हिंदी ▁फ़िल्म ... (+2 more)` | 12 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.233x compression
-- **Lowest UNK Rate:** 8k with 0.0938% unknown tokens
+- **Best Compression:** 32k achieves 3.779x compression
+- **Lowest UNK Rate:** 8k with 0.1162% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -124,57 +125,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 1,473 🏆 | 10.52 | 19,190 | 39.0% | 79.4% |
-| **2-gram** | 611 🏆 | 9.26 | 5,088 | 50.3% | 92.6% |
-| **3-gram** | 10,511 | 13.36 | 71,690 | 14.3% | 44.9% |
-| **3-gram** | 4,774 | 12.22 | 39,446 | 20.6% | 57.0% |
-| **4-gram** | 37,995 | 15.21 | 200,757 | 8.2% | 28.1% |
-| **4-gram** | 20,752 | 14.34 | 155,528 | 10.6% | 34.2% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 5,052 | 12.30 | 15,168 | 20.6% | 52.3% |
+| **2-gram** | Subword | 1,738 🏆 | 10.76 | 17,876 | 38.2% | 73.8% |
+| **3-gram** | Word | 4,130 | 12.01 | 14,962 | 20.9% | 59.9% |
+| **3-gram** | Subword | 12,170 | 13.57 | 72,480 | 14.9% | 40.7% |
+| **4-gram** | Word | 6,457 | 12.66 | 28,266 | 18.2% | 56.2% |
+| **4-gram** | Subword | 41,486 | 15.34 | 205,918 | 8.4% | 27.1% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `के लिए` | 2,018 |
+| 2 | `के अनुसार` | 1,711 |
+| 3 | `छै जे` | 1,623 |
+| 4 | `छै जेकरा` | 1,477 |
+| 5 | `के औसत` | 1,421 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `छै जेकरा म` | 1,239 |
+| 2 | `जनगणना के अनुसार` | 1,231 |
+| 3 | `के रूप में` | 808 |
+| 4 | `परिवार रहै छै` | 789 |
+| 5 | `म स्थित ऐगो` | 690 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `छै जेकरा म कुल` | 638 |
+| 2 | `के औसत लिंग अनुपात` | 559 |
+| 3 | `छै जनगणना के अनुसार` | 535 |
+| 4 | `के जनगणना के अनुसार` | 498 |
+| 5 | `गाँव छै जेकरा म` | 479 |
+
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `क े` | 45,157 |
-| 2 | `ा र` | 27,669 |
-| 3 | `् र` | 26,110 |
-| 4 | `ि क` | 21,854 |
-| 5 | `य ा` | 21,115 |
+| 1 | `र _` | 44,044 |
+| 2 | `_ के` | 42,780 |
+| 3 | `के _` | 39,580 |
+| 4 | `, _` | 27,198 |
+| 5 | `। _` | 27,084 |
 
-**3-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `म े ं` | 15,079 |
-| 2 | `ि य ा` | 9,483 |
-| 3 | `छ ै ।` | 7,624 |
-| 4 | `् य ा` | 7,466 |
-| 5 | `ा क े` | 7,405 |
+| 1 | `_ के _` | 37,016 |
+| 2 | `_ में _` | 14,280 |
+| 3 | `_ की _` | 9,494 |
+| 4 | `_ औ र` | 9,303 |
+| 5 | `औ र _` | 9,298 |
 
-**4-grams:**
+**4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `भ ा ष ा` | 4,716 |
-| 2 | `प ् र ा` | 3,403 |
-| 3 | `क े र ौ` | 3,367 |
-| 4 | `ा ह ै ।` | 3,140 |
-| 5 | `ै , ज े` | 3,079 |
+| 1 | `_ औ र _` | 9,269 |
+| 2 | `_ है । _` | 6,536 |
+| 3 | `_ छै । _` | 5,833 |
+| 4 | `_ ए क _` | 4,768 |
+| 5 | `र _ के _` | 3,598 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 611
+- **Best Perplexity:** 2-gram (subword) with 1,738
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~34% of corpus
+- **Coverage:** Top-1000 patterns cover ~27% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -182,55 +215,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.6716 | 1.593 | 6.02 | 20,822 | 32.8% |
-| **1** | 1.3840 | 2.610 | 13.82 | 755 | 0.0% |
-| **2** | 0.3861 | 1.307 | 2.68 | 125,337 | 61.4% |
-| **2** | 1.1859 | 2.275 | 7.36 | 10,432 | 0.0% |
-| **3** | 0.3331 | 1.260 | 2.02 | 335,781 | 66.7% |
-| **3** | 0.8363 | 1.785 | 3.79 | 76,706 | 16.4% |
-| **4** | 0.2418 🏆 | 1.182 | 1.55 | 678,169 | 75.8% |
-| **4** | 0.5341 🏆 | 1.448 | 2.27 | 290,862 | 46.6% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.8691 | 1.827 | 5.82 | 57,434 | 13.1% |
+| **1** | Subword | 0.9723 | 1.962 | 11.43 | 4,617 | 2.8% |
+| **2** | Word | 0.2533 | 1.192 | 1.57 | 333,590 | 74.7% |
+| **2** | Subword | 0.5474 | 1.461 | 3.83 | 52,772 | 45.3% |
+| **3** | Word | 0.0719 | 1.051 | 1.12 | 522,356 | 92.8% |
+| **3** | Subword | 0.4946 | 1.409 | 2.66 | 202,187 | 50.5% |
+| **4** | Word | 0.0215 🏆 | 1.015 | 1.03 | 583,736 | 97.9% |
+| **4** | Subword | 0.2981 | 1.230 | 1.71 | 538,126 | 70.2% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `के बीच शासन और विज्ञापन और २०० फिल्में रिलीज़ हुआ है जिसके कारण ही मैन 2`
+2. `में पाकिस्तान श्रीलंका मे पुणे शहर आरू एकरऽ द्रव्यमान संरक्षण के स्थान पऽ एकाग्र करै लेली`
+3. `है कि विक्की ग्युरेरो के बच्चा के बच्चा के 61 80 मीटर 16 राज्यो मँ संकटग्रस्त`
+
+**Context Size 2:**
+
+1. `के लिए पौधों को जिन्हें फूलने से पहले उन्होंने डस्टी रोहड्स का भी समर्थन प्राप्त हो सकती`
+2. `के अनुसार मुंजथ गांव के कुल आबादी के साथ दो दुर्भाग्यपूर्ण मामलों के लिए लिख लेते थे`
+3. `छै जे बिहार राज्य मँ स्थित छै इ जिला पौराणिक काल म॑ विश्व भर में १० १४`
+
+**Context Size 3:**
+
+1. `छै जेकरा म कुल 22 परिवार रहै छै तुम्बापहाड़ गांव के जनसंख्या 188 छै जेकरा म पुरुष आरू`
+2. `जनगणना के अनुसार गोबिंदपुर गाँव के जनसंख्या छै जेकरा पुरुष आरु महिला छै गौरीपुर गांव के औसत लिंग`
+3. `के रूप में अरबी गोंद के साथ मिलाया जा सकता था इसी कारणवश बादशाह मुहम्‍मद बिन तुगलक ने`
 
-### Generated Text Samples
+**Context Size 4:**
+
+1. `छै जेकरा म कुल 545 परिवार रहै छै के जनगणना के अनुसार मुस्तफाबाद गाँव के जनसंख्या 291 छै जे`
+2. `के औसत लिंग अनुपात 782 छै जे बिहार राज्य के औसत 918 स कम छै जनगणना के अनुसार तेतरिया`
+3. `छै जनगणना के अनुसार सहनी खेड़ा के बाल लिंग अनुपात 836 छै जे बिहार राज्य के औसत 918 स`
+
+
+### Generated Text Samples (Subword-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `ा ण ी कम ी ] स ा ) घ ो क ृ त ् रभ`
-2. `् डल स ं ख ् ध ा य ु न े ण अक ा प`
-3. `े म ँ स ू प ि त च ा जन ा न ह ै ज`
+1. `_ला_का_अनुसाय_उसनता_शि`
+2. `रचरण-हा_जे_के_शिय-_में`
+3. `क_किंगर_मशिक्षा_बांकारलिंग_`
 
 **Context Size 2:**
 
-1. `क े प ु स ् थ अलग करत ी थ ी , ड ा इऑक ्`
-2. `ा र अध ि न े अध ि क ा ल ा म द ि य ा`
-3. `् र ो स ि मड े ग ा न , 2006 क ो खतर े क`
+1. `र_सम्पादक_छै_।_नीतिक_न॑_`
+2. `_के_कुल_रूप_पुर_सक्षमता_`
+3. `के_लिमिटेड_श्रेणी:_भास्कराचार_`
 
 **Context Size 3:**
 
-1. `म े ं क े द ु मक ा स ॑ ई बड ़ ऽ क ् ष`
-2. `ि य ा च ि त ह ै और यह ी ं भ ी उनक े प ा`
-3. `् य ा 1484 छ े ल ै । २८ नवम ् बर 1889 क ो और तन`
+1. `_के_रूप_से_एक_छै।_जनसंख्या`
+2. `_में_रूचि_रखै_वाला_नहीं_है।_`
+3. `_की_भी_हैं_दाग_डॉक्टर_के_रूप`
 
 **Context Size 4:**
 
-1. `भ ा ष ा स ि न ी क स ं स ् थ ा प ि त करन`
-2. `प ् र ा प ् त छ ै । शब ् द - स ा धन इत ि`
-3. `क े र ौ व ि भ ि न ् न - भ ि न ् न प ्`
+1. `_और_श्रुति_साहित्य,_दर्शन_हेतु_`
+2. `_है।_इसके_बजाय_व्याख्यान_क_ए`
+3. `_छै।_चूना_ऐगो_अंतर_छै,_अपि`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 75.8% predictability
+- **Best Predictability:** Context-4 (word) with 97.9% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (290,862 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (538,126 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -246,64 +310,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 9,567 |
-| Total Tokens | 1,448,534 |
-| Mean Frequency | 151.41 |
+| Vocabulary Size | 26,612 |
+| Total Tokens | 692,487 |
+| Mean Frequency | 26.02 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 2528.69 |
+| Frequency Std Dev | 316.81 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | क | 136,096 |
-| 2 | र | 92,258 |
-| 3 | स | 77,006 |
-| 4 | म | 59,103 |
-| 5 | त | 56,261 |
-| 6 | न | 54,426 |
-| 7 | य | 53,099 |
-| 8 | ल | 48,343 |
-| 9 | प | 45,425 |
-| 10 | ह | 44,325 |
+| 1 | के | 37,114 |
+| 2 | में | 15,064 |
+| 3 | छै | 12,685 |
+| 4 | है | 12,473 |
+| 5 | की | 9,887 |
+| 6 | और | 9,313 |
+| 7 | का | 7,757 |
+| 8 | से | 7,397 |
+| 9 | को | 5,594 |
+| 10 | हैं | 5,335 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | css | 2 |
-| 2 | zeros | 2 |
-| 3 | ignored | 2 |
-| 4 | dmy | 2 |
-| 5 | mdy | 2 |
-| 6 | paren | 2 |
-| 7 | breaking | 2 |
-| 8 | inserted | 2 |
-| 9 | values | 2 |
-| 10 | separator | 2 |
+| 1 | pmegp | 2 |
+| 2 | odop | 2 |
+| 3 | naps | 2 |
+| 4 | संवर्द्धन | 2 |
+| 5 | आईज़ | 2 |
+| 6 | रिटेल | 2 |
+| 7 | एक्सीलेंस | 2 |
+| 8 | इंस्टाग्राम | 2 |
+| 9 | कास्टिंग | 2 |
+| 10 | ईयर | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.4601 |
-| R² (Goodness of Fit) | 0.993188 |
+| Zipf Coefficient | 1.1238 |
+| R² (Goodness of Fit) | 0.994960 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 82.8% |
-| Top 1,000 | 96.2% |
-| Top 5,000 | 99.3% |
-| Top 10,000 | 0.0% |
+| Top 100 | 40.3% |
+| Top 1,000 | 69.8% |
+| Top 5,000 | 87.2% |
+| Top 10,000 | 93.2% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9932 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 82.8% of corpus
-- **Long Tail:** -433 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9950 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 40.3% of corpus
+- **Long Tail:** 16,612 words needed for remaining 6.8% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -316,24 +380,103 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 12,167 | 32 | 3.479 | 0.971 | 0.8322 🏆 |
-| **mono_64d** | 12,167 | 64 | 3.809 | 0.905 | 0.7196 |
-| **mono_128d** | 12,167 | 128 | 3.964 | 0.866 | 0.3647 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8284 🏆 | 0.3485 | N/A | N/A |
+| **mono_64d** | 64 | 0.6880 | 0.2899 | N/A | N/A |
+| **mono_128d** | 128 | 0.3275 | 0.2699 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8322 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 12,167 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8284 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3027. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
+
+---
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-प्` | प्रिंत्सीप, प्रतिअंकन, प्रभाग |
+| `-प्र` | प्रिंत्सीप, प्रतिअंकन, प्रभाग |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-ों` | साँपों, अनुभववादियों, रेलों |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `tion` | 2.57x | 12 contexts | motion, action, section |
+| `atio` | 2.57x | 12 contexts | station, nations, stations |
+| `stat` | 2.59x | 6 contexts | state, states, status |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-प्` | `-ों` | 20 words | प्रयासों, प्रकृतिवादियों |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| प्रवृत्ति | **`प्र-वृत्ति`** | 4.5 | `वृत्ति` |
+| अक्षांशों | **`अक्षांश-ों`** | 4.5 | `अक्षांश` |
+| व्यवसायों | **`व्यवसाय-ों`** | 4.5 | `व्यवसाय` |
+| चक्रवातों | **`चक्रवात-ों`** | 4.5 | `चक्रवात` |
+| भागीदारों | **`भागीदार-ों`** | 4.5 | `भागीदार` |
+| तीर्थंकरों | **`तीर्थंकर-ों`** | 4.5 | `तीर्थंकर` |
+| उद्दीपकों | **`उद्दीपक-ों`** | 4.5 | `उद्दीपक` |
+| काव्यतत्वों | **`काव्यतत्व-ों`** | 4.5 | `काव्यतत्व` |
+| संग्रहालयों | **`संग्रहालय-ों`** | 4.5 | `संग्रहालय` |
+| साहित्यकारों | **`साहित्यकार-ों`** | 4.5 | `साहित्यकार` |
+| चिकित्सकों | **`चिकित्सक-ों`** | 4.5 | `चिकित्सक` |
+| उद्देश्यों | **`उद्देश्य-ों`** | 4.5 | `उद्देश्य` |
+| विश्वकोशों | **`विश्वकोश-ों`** | 4.5 | `विश्वकोश` |
+| निष्कर्षों | **`निष्कर्ष-ों`** | 4.5 | `निष्कर्ष` |
+| प्रशंसकों | **`प्र-शंसक-ों`** | 3.0 | `शंसक` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language ANP appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
 
 ---
-## 6. Summary & Recommendations
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -341,11 +484,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.23x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (611) |
-| Markov | **Context-4** | Highest predictability (75.8%) |
+| Tokenizer | **32k BPE** | Best compression (3.78x) |
+| N-gram | **2-gram** | Lowest perplexity (1,738) |
+| Markov | **Context-4** | Highest predictability (97.9%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -535,7 +679,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -551,7 +696,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 06:08:15*
+*Report Date: 2026-01-03 05:15:16*

models/embeddings/monolingual/anp_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5e5037926f606a5261779717c539c9dbbd1ae5b5239b116508b41fe3616febae
-size 1036770452
+oid sha256:a9caacde775c5d9ed45854d4282f8461327a3fb82d634b04bde4c1a7010bfb01
+size 1036094760

models/embeddings/monolingual/anp_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 12167
+  "vocab_size": 11521
 }

models/embeddings/monolingual/anp_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dd35312939967d31d7d904a9ca1a76cbbdcc4c44fad15183039efe9bf32cbdd2
-size 259426196
+oid sha256:550b641eec37665cd8011ee7edf15fb97b5a67b4ea66eddae766e6ab6db5fa8b
+size 259246632

models/embeddings/monolingual/anp_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 12167
+  "vocab_size": 11521
 }

models/embeddings/monolingual/anp_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:95dc1f4810dddc464023dcea010decc1b484ac058bc2a8c9392076ea90c65544
-size 518540948
+oid sha256:5d2fb6809bbbfd3fdc21a99d5055c629f05387c2aad2b8c7bd73362067d6a1b8
+size 518196008

models/embeddings/monolingual/anp_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 12167
+  "vocab_size": 11521
 }

models/subword_markov/anp_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4297aac5947b461b82ded3981b487a038458065f66a84607b9d1d55bb71f1718
-size 83720
+oid sha256:95c04e6c8a5663b3755575bf1c4f4bd7a74ee1859643e6a15404e0aefd1820ec
+size 364915

models/subword_markov/anp_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "anp",
-  "unique_contexts": 755,
-  "total_transitions": 4171640
+  "unique_contexts": 4617,
+  "total_transitions": 2607475
 }

models/subword_markov/anp_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a0e556496b318b18ecfe9c97c36c2d23a696de1824af13d8e624e9d9e2c53bad
-size 561964
+oid sha256:daa9cbe0600238ddf33258710db221432206594aadb8cd97e7c233736df61bc1
+size 1615892

models/subword_markov/anp_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "anp",
-  "unique_contexts": 10432,
-  "total_transitions": 4168473
+  "unique_contexts": 52772,
+  "total_transitions": 2604446
 }

models/subword_markov/anp_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c16a8be91a6fe7953eef4ac6eccf82dca4b99b8816d61fea6fe3718b7600bb34
-size 2057035
+oid sha256:c285b9eb6f3174cff5126cc788535f54c2aac763cfabb71374694688d74bb8d7
+size 4795293

models/subword_markov/anp_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "anp",
-  "unique_contexts": 76706,
-  "total_transitions": 4165306
+  "unique_contexts": 202187,
+  "total_transitions": 2601417
 }

models/subword_markov/anp_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6e54d19d4e7dafad81e1f50c98d138e50a189da919f32e24193b667682fa96a8
-size 5952951
+oid sha256:cf8bec6d17e4fe549aa7177f537ef96f42090d3142c162b4f6ddcecc9a3ca3f9
+size 10831894

models/subword_markov/anp_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "anp",
-  "unique_contexts": 290862,
-  "total_transitions": 4162139
+  "unique_contexts": 538126,
+  "total_transitions": 2598388
 }

models/subword_ngram/anp_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:355f1e9ea12646e5552194569897d90f61b54d55a6596d37ad576bac53b4c18a
-size 66307
+oid sha256:775113617eee04863418a0e3d9184050647852be0f10c589b8fe1da6543c4c55
+size 264208

models/subword_ngram/anp_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "anp",
-  "unique_ngrams": 5088,
-  "total_ngrams": 4171640
+  "unique_ngrams": 17876,
+  "total_ngrams": 2607475
 }

models/subword_ngram/anp_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b14727440da7b6e77cb354d37b0c27476ed4374fb0258851694ef1e8116220f5
-size 516525
+oid sha256:d1beda611527797d1c53b1c09dc17f9ce7383569eb890ec2fb07d1f5209c0c22
+size 1044126

models/subword_ngram/anp_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "anp",
-  "unique_ngrams": 39446,
-  "total_ngrams": 4168473
+  "unique_ngrams": 72480,
+  "total_ngrams": 2604446
 }

models/subword_ngram/anp_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e2aa49d6568ec1b8bf85049449b1f7efb6e4ddd46688f5f01f50be945751e4cd
-size 1939235
+oid sha256:1fe6cf4b2975ea16eac745cd73317aa126e7bd7842670a9adabb558d579c736b
+size 3083912

models/subword_ngram/anp_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "anp",
-  "unique_ngrams": 155528,
-  "total_ngrams": 4165306
+  "unique_ngrams": 205918,
+  "total_ngrams": 2601417
 }

models/tokenizer/anp_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:efb5cb1039a55c7f7bb857ffb86e33f81c6dc4d0c89b00b0b2bf254abbf18eaa
-size 617718
+oid sha256:4dbad16cbb100bec382d11f61c37dc66eb44c51310eda04fcecebe9af7d4c358
+size 626731

models/tokenizer/anp_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2b0c4178d49ec2b0885bec0eaad33f6a5f7167c07accfe726ef9bfc8acb39865
-size 1020495
+oid sha256:307cc8ede7a5261bd978c0775a7d475811f92b91caf248ac449e45d8c1a3f03d
+size 1055594

models/tokenizer/anp_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:326640bb83ed996d173f81797fc5a30047a8b1b52c4a054dda0f50dacd58e6d7
-size 424492
+oid sha256:aa6e8b6fe63bd83b4d41987afef044fc696594969b6cbbbe9dc22aee174de945
+size 430161

models/vocabulary/anp_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7846de396d06f632490fe25bb62f459573bd28942c91c218251658557d166439
-size 154433
+oid sha256:23c24e34a334b98f6f430ae4bfb9547516edad531380c24b4d8922e550ce220d
+size 502668

models/vocabulary/anp_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "anp",
-  "vocabulary_size": 9567,
+  "vocabulary_size": 26612,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.014131336612329602,
+    "type_token_ratio": 0.07945542193325766,
     "coverage": {
-      "top_100": 0.8221627535895281,
-      "top_1000": 0.954931271936766,
-      "top_5000": 0.9849848553672154,
-      "top_10000": 0.992719888352438
+      "top_100": 0.3854363523002036,
+      "top_1000": 0.6679237822959595,
+      "top_5000": 0.8348141768358013,
+      "top_10000": 0.8920299882905762
     },
-    "hapax_count": 11059,
-    "hapax_ratio": 0.5361679433724426,
-    "total_documents": 3167
+    "hapax_count": 30862,
+    "hapax_ratio": 0.5369732400737725,
+    "total_documents": 3029
   }
 }

models/word_markov/anp_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:192ca60a300173cfabd8b1fb3660b244c3be732254c68e60bf1ee02dd07b598e
-size 886162
+oid sha256:d37e0230173ff329a06a1b9fde21c583b6b6fb6e94481f23a7a68e72460175c5
+size 2920307

models/word_markov/anp_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "anp",
-  "unique_contexts": 20822,
-  "total_transitions": 2733532
+  "unique_contexts": 57434,
+  "total_transitions": 720320
 }

models/word_markov/anp_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8a4a530af52e6c9f3eaab3f9243da477a38d9854b21682a25de5c7b9f6c8f144
-size 2833179
+oid sha256:fdf85ce95679a912c641b5ffb10719ad0f16e152363efc3c3922f250107b17b8
+size 8558224

models/word_markov/anp_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "anp",
-  "unique_contexts": 125337,
-  "total_transitions": 2730365
+  "unique_contexts": 333590,
+  "total_transitions": 717291
 }

models/word_markov/anp_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a7e40a5b87677577d689aa253f08740d9ad689107604462b8e1e4df9ad5a7cba
-size 6746500
+oid sha256:d46521c8f50238b1c6c19de18c1d6e31bb2ae9bb5e64373564f718f502ccb21a
+size 12610274

models/word_markov/anp_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "anp",
-  "unique_contexts": 335781,
-  "total_transitions": 2727198
+  "unique_contexts": 522356,
+  "total_transitions": 714262
 }

models/word_markov/anp_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:df1d380e4e3dbf318266422fddd74c8e3c3c5bbb43bcf44df36feb99255fc67d
-size 12362534
+oid sha256:2a6d0791af9d7c4cd60f49ebd2f3e62098fd20d5aa9181255874c2868e60584a
+size 15137005

models/word_markov/anp_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "anp",
-  "unique_contexts": 678169,
-  "total_transitions": 2724031
+  "unique_contexts": 583736,
+  "total_transitions": 711233
 }

models/word_ngram/anp_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:380fd982ea2b330e8e2918f8c25bfd63e9d5582ff2d103f96917453a9eec5bcd
-size 261069
+oid sha256:c89f5961a7a5db5f9e4f519a74d84544451a49614e2bf927884a2b6969b4d734
+size 309620

models/word_ngram/anp_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "anp",
-  "unique_ngrams": 19190,
-  "total_ngrams": 2733532
+  "unique_ngrams": 15168,
+  "total_ngrams": 720320
 }

models/word_ngram/anp_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6f8e4b8e3220800bfc9f062abf2dce74ef08616fa574c66f309c8d9d202644d1
-size 955234
+oid sha256:67658d2f94c99a0c6dcc25772c19631b533310f0ff80aaa0a3d844108d031bd1
+size 351736

models/word_ngram/anp_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "anp",
-  "unique_ngrams": 71690,
-  "total_ngrams": 2730365
+  "unique_ngrams": 14962,
+  "total_ngrams": 717291
 }

models/word_ngram/anp_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9711dcecd0f23b0bdaba6380c42f3dd115a1ded144cc810366c614694e28520a
-size 2713081
+oid sha256:1c170b7b16fc88c43f6a225f5978bb19aefa148a6d008bfb548a6416e19c1573
+size 702952

models/word_ngram/anp_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "anp",
-  "unique_ngrams": 200757,
-  "total_ngrams": 2727198
+  "unique_ngrams": 28266,
+  "total_ngrams": 714262
 }