Upload all models and assets for bh (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.138
+    value: 4.103
   - name: best_isotropy
     type: isotropy
-    value: 0.8624
+    value: 0.8668
   - name: vocabulary_size
     type: vocab
-    value: 16281
-generated: 2025-12-28
+    value: 0
+generated: 2026-01-03
 ---
 
 # BH - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,62 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.422x | 3.37 | 0.1263% | 371,376 |
-| **16k** | 3.721x | 3.67 | 0.1374% | 341,460 |
-| **32k** | 3.953x | 3.90 | 0.1459% | 321,424 |
-| **64k** | 4.138x 🏆 | 4.08 | 0.1527% | 307,072 |
+| **8k** | 3.436x | 3.44 | 0.1753% | 369,577 |
+| **16k** | 3.741x | 3.74 | 0.1909% | 339,439 |
+| **32k** | 3.960x | 3.96 | 0.2021% | 320,666 |
+| **64k** | 4.103x 🏆 | 4.11 | 0.2094% | 309,485 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `कड़ुआ तेल चाहे करूआ तेल एक परकार के तेल हऽ जेकरा सरसो के बीया के पिसि के बनावल ज...`
+**Sample 1:** `अँगारपाथर भारत के झारखंड राज्य में धनबाद शहर के एगो मोहल्ला बाटे। संदर्भ के शहर‏...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁कड़ ुआ ▁तेल ▁चाहे ▁कर ू आ ▁तेल ▁एक ▁परकार ... (+24 more)` | 34 |
-| 16k | `▁कड़ ुआ ▁तेल ▁चाहे ▁कर ू आ ▁तेल ▁एक ▁परकार ... (+21 more)` | 31 |
-| 32k | `▁कड़ ुआ ▁तेल ▁चाहे ▁कर ू आ ▁तेल ▁एक ▁परकार ... (+20 more)` | 30 |
-| 64k | `▁कड़ ुआ ▁तेल ▁चाहे ▁कर ूआ ▁तेल ▁एक ▁परकार ▁के ... (+19 more)` | 29 |
-
-**Sample 2:** `भगतडीह भारत के झारखंड राज्य में एक ठो कसबा बाटे।
+| 8k | `▁अँ गार पा थर ▁भारत ▁के ▁झारखंड ▁राज्य ▁में ▁धन ... (+12 more)` | 22 |
+| 16k | `▁अँ गार पा थर ▁भारत ▁के ▁झारखंड ▁राज्य ▁में ▁धनबाद ... (+11 more)` | 21 |
+| 32k | `▁अँ गार पाथर ▁भारत ▁के ▁झारखंड ▁राज्य ▁में ▁धनबाद ▁शहर ... (+10 more)` | 20 |
+| 64k | `▁अँगारपाथर ▁भारत ▁के ▁झारखंड ▁राज्य ▁में ▁धनबाद ▁शहर ▁के ▁एगो ... (+8 more)` | 18 |
 
-श्रेणी:झारखंड के शहर‏‎ आ कस्बा`
+**Sample 2:** `जून ग्रेगरियन कैलेंडर के छठवाँ महीना ह। घटना तिहुआर अउरी दूसर महत्व के दिन अउरी ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁भगत डी ह ▁भारत ▁के ▁झारखंड ▁राज्य ▁में ▁एक ▁ठो ... (+10 more)` | 20 |
-| 16k | `▁भगत डीह ▁भारत ▁के ▁झारखंड ▁राज्य ▁में ▁एक ▁ठो ▁कसबा ... (+9 more)` | 19 |
-| 32k | `▁भगत डीह ▁भारत ▁के ▁झारखंड ▁राज्य ▁में ▁एक ▁ठो ▁कसबा ... (+9 more)` | 19 |
-| 64k | `▁भगत डीह ▁भारत ▁के ▁झारखंड ▁राज्य ▁में ▁एक ▁ठो ▁कसबा ... (+9 more)` | 19 |
-
-**Sample 3:** `घटना
-
-जनम
-
-निधन
-
-तिहुआर, छुट्टी अउरी खास महत्व 
+| 8k | `▁जून ▁ग्रेगरियन ▁कैलेंडर ▁के ▁छठ वाँ ▁महीना ▁ह । ▁घटना ... (+14 more)` | 24 |
+| 16k | `▁जून ▁ग्रेगरियन ▁कैलेंडर ▁के ▁छठवाँ ▁महीना ▁ह । ▁घटना ▁तिहुआर ... (+13 more)` | 23 |
+| 32k | `▁जून ▁ग्रेगरियन ▁कैलेंडर ▁के ▁छठवाँ ▁महीना ▁ह । ▁घटना ▁तिहुआर ... (+13 more)` | 23 |
+| 64k | `▁जून ▁ग्रेगरियन ▁कैलेंडर ▁के ▁छठवाँ ▁महीना ▁ह । ▁घटना ▁तिहुआर ... (+13 more)` | 23 |
 
-श्रेणी:साल के दिन
-श्रेणी:अगस्त`
+**Sample 3:** `बदायूँ जिला उत्तर प्रदेश की बरेली मंडल में एगो जिला बाटे जौना के मुख्यालय बदायूँ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁घटना ▁जनम ▁निधन ▁तिहुआर , ▁छुट्टी ▁अउरी ▁खास ▁महत्व ▁श्रेणी ... (+7 more)` | 17 |
-| 16k | `▁घटना ▁जनम ▁निधन ▁तिहुआर , ▁छुट्टी ▁अउरी ▁खास ▁महत्व ▁श्रेणी ... (+7 more)` | 17 |
-| 32k | `▁घटना ▁जनम ▁निधन ▁तिहुआर , ▁छुट्टी ▁अउरी ▁खास ▁महत्व ▁श्रेणी ... (+7 more)` | 17 |
-| 64k | `▁घटना ▁जनम ▁निधन ▁तिहुआर , ▁छुट्टी ▁अउरी ▁खास ▁महत्व ▁श्रेणी ... (+7 more)` | 17 |
+| 8k | `▁ब दाय ूँ ▁जिला ▁उत्तर ▁प्रदेश ▁की ▁ब रेली ▁मंडल ... (+19 more)` | 29 |
+| 16k | `▁ब दाय ूँ ▁जिला ▁उत्तर ▁प्रदेश ▁की ▁बरेली ▁मंडल ▁में ... (+18 more)` | 28 |
+| 32k | `▁ब दायूँ ▁जिला ▁उत्तर ▁प्रदेश ▁की ▁बरेली ▁मंडल ▁में ▁एगो ... (+16 more)` | 26 |
+| 64k | `▁बदायूँ ▁जिला ▁उत्तर ▁प्रदेश ▁की ▁बरेली ▁मंडल ▁में ▁एगो ▁जिला ... (+14 more)` | 24 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.138x compression
-- **Lowest UNK Rate:** 8k with 0.1263% unknown tokens
+- **Best Compression:** 64k achieves 4.103x compression
+- **Lowest UNK Rate:** 8k with 0.1753% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -132,57 +129,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 1,553 🏆 | 10.60 | 29,844 | 40.4% | 78.0% |
-| **2-gram** | 643 🏆 | 9.33 | 6,318 | 48.7% | 91.9% |
-| **3-gram** | 11,076 | 13.44 | 104,826 | 16.6% | 45.2% |
-| **3-gram** | 4,981 | 12.28 | 50,779 | 21.0% | 55.3% |
-| **4-gram** | 42,879 | 15.39 | 294,598 | 9.9% | 28.5% |
-| **4-gram** | 22,284 | 14.44 | 220,351 | 12.6% | 34.3% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 9,163 | 13.16 | 29,857 | 16.5% | 43.3% |
+| **2-gram** | Subword | 1,500 🏆 | 10.55 | 21,796 | 39.5% | 76.5% |
+| **3-gram** | Word | 13,824 | 13.75 | 38,729 | 15.8% | 36.1% |
+| **3-gram** | Subword | 11,162 | 13.45 | 93,652 | 16.7% | 42.2% |
+| **4-gram** | Word | 17,666 | 14.11 | 53,247 | 17.6% | 35.4% |
+| **4-gram** | Subword | 44,915 | 15.45 | 295,453 | 9.1% | 27.7% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `सभ के` | 4,161 |
+| 2 | `भारत के` | 3,814 |
+| 3 | `रूप में` | 3,157 |
+| 4 | `के रूप` | 2,933 |
+| 5 | `देखल जाय` | 2,149 |
+
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `क े` | 114,130 |
-| 2 | `े ं` | 61,648 |
-| 3 | `म े` | 52,589 |
-| 4 | `् र` | 51,939 |
-| 5 | `ल ा` | 46,397 |
+| 1 | `के रूप में` | 2,740 |
+| 2 | `इहो देखल जाय` | 2,002 |
+| 3 | `के हिसाब से` | 1,423 |
+| 4 | `संदर्भ बाहरी कड़ी` | 1,392 |
+| 5 | `शहर आ कस्बा` | 1,209 |
 
-**3-grams:**
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `म े ं` | 46,997 |
-| 2 | `् र े` | 23,873 |
-| 3 | `ा क े` | 20,218 |
-| 4 | `श ् र` | 18,887 |
-| 5 | `र े ण` | 18,054 |
+| 1 | `के शहर आ कस्बा` | 1,206 |
+| 2 | `बाटे इहो देखल जाय` | 780 |
+| 3 | `राज्य में एक ठो` | 667 |
+| 4 | `के हिसाब से ई` | 539 |
+| 5 | `में एगो जिला बाटे` | 536 |
 
-**4-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `श ् र े` | 18,107 |
-| 2 | `् र े ण` | 18,022 |
-| 3 | `र े ण ी` | 18,015 |
-| 4 | `े ण ी :` | 17,670 |
-| 5 | `ब ा ट े` | 8,587 |
+| 1 | `के _` | 114,253 |
+| 2 | `_ के` | 110,824 |
+| 3 | `र _` | 75,001 |
+| 4 | `ल _` | 68,413 |
+| 5 | `न _` | 54,528 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ के _` | 109,027 |
+| 2 | `_ में _` | 44,490 |
+| 3 | `_ आ _` | 29,937 |
+| 4 | `_ से _` | 20,956 |
+| 5 | `ल _ जा` | 13,886 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `न _ के _` | 9,495 |
+| 2 | `_ स भ _` | 8,569 |
+| 3 | `_ ए गो _` | 8,113 |
+| 4 | `र _ के _` | 7,353 |
+| 5 | `ल _ जा ला` | 7,231 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 643
+- **Best Perplexity:** 2-gram (subword) with 1,500
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~34% of corpus
+- **Coverage:** Top-1000 patterns cover ~28% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -190,55 +219,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.5759 | 1.491 | 5.28 | 42,274 | 42.4% |
-| **1** | 1.2819 | 2.432 | 11.97 | 1,128 | 0.0% |
-| **2** | 0.3584 | 1.282 | 2.50 | 223,237 | 64.2% |
-| **2** | 1.1592 | 2.233 | 7.49 | 13,497 | 0.0% |
-| **3** | 0.2903 | 1.223 | 1.93 | 557,656 | 71.0% |
-| **3** | 0.8362 | 1.785 | 4.01 | 101,022 | 16.4% |
-| **4** | 0.2272 🏆 | 1.171 | 1.55 | 1,074,886 | 77.3% |
-| **4** | 0.5736 🏆 | 1.488 | 2.44 | 405,495 | 42.6% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.8766 | 1.836 | 6.16 | 84,482 | 12.3% |
+| **1** | Subword | 0.9997 | 2.000 | 12.30 | 4,952 | 0.0% |
+| **2** | Word | 0.2946 | 1.227 | 1.77 | 519,009 | 70.5% |
+| **2** | Subword | 0.5586 | 1.473 | 4.02 | 60,879 | 44.1% |
+| **3** | Word | 0.1069 | 1.077 | 1.19 | 917,743 | 89.3% |
+| **3** | Subword | 0.5222 | 1.436 | 2.95 | 244,880 | 47.8% |
+| **4** | Word | 0.0351 🏆 | 1.025 | 1.05 | 1,088,288 | 96.5% |
+| **4** | Subword | 0.3351 | 1.261 | 1.87 | 721,221 | 66.5% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `के कुछ अउरी पढ़े के इस्तेमाल होला आ नया विमानन क के दैविक घटना जनम 11`
+2. `में एकट्ठा क क्षमता में राजा दशरथ आ एकर नक़ल उतारे जा सके ला बाद के`
+3. `आ भोजपुरी में एक ठो इतिहासी भूबिज्ञान आ मिजोरम के नाँव ढेर बरफ के अबतक ले`
+
+**Context Size 2:**
+
+1. `सभ के कक्षा ऑर्बिट सुरुज के सभसे पबित्र मानल जाला एह दिन के मतलब मैदान के मैदान`
+2. `भारत के बारहवाँ कार्यकाल अनुसार 14वाँ आ वर्तमान में भारत के प्रतिनिधित्व यूरोपियन कमीशन द्वारा 12 मई`
+3. `रूप में अवधारणा के अंतर्राष्ट्रीय बॉर्डर के रूप में निरूपण नक्शा कौनों इलाका के मिला लिहल जाय`
+
+**Context Size 3:**
+
+1. `के रूप में बर्गीकरण कइल जाला कुछ दशा में घार्मिक कामकाज खातिर भी ई नगर के महतव बा`
+2. `इहो देखल जाय बिहार सरकार बिहार के बिकास के रूप में एमबीए कइलें स्टैनफोर्ड में पढ़त घरी इनके`
+3. `के हिसाब से दुनिया के छत्तीसवाँ देस हवे आ पूरबी हिस्सा में उत्तर से दक्खिन ओर फइलल बिसाल`
+
+**Context Size 4:**
+
+1. `बाटे इहो देखल जाय ओडिशा के जिला भारत के जिला सभ के लिस्ट संदर्भ बाहरी कड़ी जिला समन्वय समिति`
+2. `राज्य में एक ठो कसबा बाटे के शहर आ कस्बा जिला के शहर आ कस्बा बंगाल के शहर आ`
+3. `के हिसाब से ई भारत के 191वाँ शहर बाटे जनगणना आँकड़ा के मोताबिक एह शहर में लिंगानुपात 992 आ`
+
 
-### Generated Text Samples
+### Generated Text Samples (Subword-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `ा ब ि ं आ य स ा क ि ल गइल । 1847 म स`
-2. `े खक आ अब ा थ ा लगववल े । एकर ठ ो र े ं`
-3. `् र ा न क े क े वन ब ि र ि ड ़ क`
+1. `_क्रिके_की_दर्भ_रव_धन_हो`
+2. `रण_योना_ल_मिल_बिकरेडिट`
+3. `के_mainnaronamoxt`
 
 **Context Size 2:**
 
-1. `क े उपशहर ी इल ा क े ख ा त ा आ इनक े म ्`
-2. `े ं द ् र ि ल ा । एकर क े क ु म ा म`
-3. `म े ं व े ल ा । घटन ा सम ू ह ि मनद , मन`
+1. `के_पाँचवीं_सस्पेंसन_-टिप्पणी_के`
+2. `_के_दिल्ली_गेम_बस_ce_th`
+3. `र_स्थान_का_आउटवारा_से_सुन`
 
 **Context Size 3:**
 
-1. `म े ं अउर ी कल ा न ि यर र ी फ ि ल ् म बनल`
-2. `् र े ण ी : ग ा यब ह ो च ु कल ब ा ड ़`
-3. `ा क े औसत ढ ा ल क े भइल आ पह ि ल ा व े व`
+1. `_के_राजा_हवे,_एक_ठो_छोट_`
+2. `_में_इनहन_में_गलुन्गाम,_पीप`
+3. `_आ_इनकर_दि_बीच_के_पहिला`
 
 **Context Size 4:**
 
-1. `श ् र े ण ी : 1991 श ् र े ण ी : भ ा रत ी`
-2. `् र े ण ी : न े क ् टरन ि ड ा ई ( फ ू ल`
-3. `र े ण ी : र ो हत ा स - र ो हत ा स ज ि नक`
+1. `न_के_राजनर्तकी_अंबपाली_(आम्रपा`
+2. `_सभ_में_राज_कर_सके_ला,_ब`
+3. `_एगो_काल्पनिक_दुनिया_के_इस्तेमाल`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 77.3% predictability
+- **Best Predictability:** Context-4 (word) with 96.5% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (405,495 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (721,221 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -254,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 16,281 |
-| Total Tokens | 2,496,952 |
-| Mean Frequency | 153.37 |
+| Vocabulary Size | 38,858 |
+| Total Tokens | 1,245,419 |
+| Mean Frequency | 32.05 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 3206.88 |
+| Frequency Std Dev | 665.90 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | क | 211,186 |
-| 2 | र | 156,322 |
-| 3 | ल | 130,779 |
-| 4 | �� | 126,220 |
-| 5 | म | 115,896 |
-| 6 | न | 83,252 |
-| 7 | ब | 77,121 |
-| 8 | ज | 73,804 |
-| 9 | त | 63,272 |
-| 10 | प | 61,248 |
+| 1 | के | 109,634 |
+| 2 | में | 46,202 |
+| 3 | आ | 30,024 |
+| 4 | से | 21,308 |
+| 5 | बा | 11,775 |
+| 6 | ई | 10,655 |
+| 7 | सभ | 8,830 |
+| 8 | बाटे | 8,519 |
+| 9 | एगो | 8,159 |
+| 10 | जाला | 8,051 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | जदम | 2 |
-| 2 | जदब | 2 |
-| 3 | जदड | 2 |
-| 4 | ईजदय | 2 |
-| 5 | वजदय | 2 |
-| 6 | जदअन | 2 |
-| 7 | यरमतद | 2 |
-| 8 | धनमह | 2 |
-| 9 | नगद | 2 |
-| 10 | रचनन | 2 |
+| 1 | मापे | 2 |
+| 2 | संयुक्‍त | 2 |
+| 3 | व्युत्क्रमानुपात | 2 |
+| 4 | सीमाएँ | 2 |
+| 5 | एहन | 2 |
+| 6 | परिपथ | 2 |
+| 7 | voltage | 2 |
+| 8 | विभवांतर | 2 |
+| 9 | वोल्ट | 2 |
+| 10 | एम्पियर | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.4509 |
-| R² (Goodness of Fit) | 0.997542 |
+| Zipf Coefficient | 1.1200 |
+| R² (Goodness of Fit) | 0.994371 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 80.9% |
-| Top 1,000 | 94.9% |
-| Top 5,000 | 98.5% |
-| Top 10,000 | 99.5% |
+| Top 100 | 43.0% |
+| Top 1,000 | 69.5% |
+| Top 5,000 | 86.1% |
+| Top 10,000 | 91.7% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9975 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 80.9% of corpus
-- **Long Tail:** 6,281 words needed for remaining 0.5% coverage
+- **Zipf Compliance:** R²=0.9944 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 43.0% of corpus
+- **Long Tail:** 28,858 words needed for remaining 8.3% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -324,24 +384,88 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 18,690 | 32 | 3.381 | 0.836 | 0.8624 🏆 |
-| **mono_64d** | 18,690 | 64 | 3.821 | 0.782 | 0.8215 |
-| **mono_128d** | 18,690 | 128 | 4.172 | 0.721 | 0.6400 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8668 🏆 | 0.3638 | N/A | N/A |
+| **mono_64d** | 64 | 0.8282 | 0.2819 | N/A | N/A |
+| **mono_128d** | 128 | 0.6394 | 0.2329 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8624 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 18,690 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8668 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2929. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+*No productive affixes detected.*
+
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `ther` | 2.68x | 27 contexts | other, there, rather |
+| `ight` | 2.68x | 21 contexts | fight, right, light |
+| `tion` | 2.59x | 21 contexts | action, nation, motion |
+| `ount` | 2.65x | 15 contexts | count, mount, amount |
+| `atio` | 2.62x | 15 contexts | ratio, nation, nations |
+| `ctio` | 2.61x | 14 contexts | action, fiction, auction |
+| `ater` | 2.67x | 11 contexts | eater, water, later |
+| `stat` | 2.63x | 10 contexts | stato, state, stats |
+| `vers` | 2.52x | 11 contexts | verse, covers, rivers |
+| `rati` | 2.58x | 9 contexts | ratio, rating, bharati |
+| `ment` | 2.50x | 9 contexts | cement, ferment, element |
+| `ical` | 2.57x | 8 contexts | radical, musical, typical |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+*No significant affix co-occurrences detected.*
+
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+*Insufficient data for recursive segmentation.*
+
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language BH appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -349,11 +473,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.14x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (643) |
-| Markov | **Context-4** | Highest predictability (77.3%) |
+| Tokenizer | **64k BPE** | Best compression (4.10x) |
+| N-gram | **2-gram** | Lowest perplexity (1,500) |
+| Markov | **Context-4** | Highest predictability (96.5%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -543,7 +668,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -559,7 +685,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-28 05:13:57*
+*Report Date: 2026-01-03 07:15:04*

models/embeddings/monolingual/bh_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0b1ff2a50937859b5360046ab048515a5ac046170ccc776fa29af74552a1947e
-size 1043608397
+oid sha256:b54269438aa2440d60ff4d391e0d7d5fe6f82e878ecccb3b40c6d625a663fdac
+size 1042368151

models/embeddings/monolingual/bh_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 18690
+  "vocab_size": 17513
 }

models/embeddings/monolingual/bh_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ce3ef16ac87b649feee451bd2f7b16b3f7ce19355ed064d1e46186107cd4b881
-size 261254477
+oid sha256:8fc27b77183033d6f955d2050d7664babad88301230b75a3837360a073560355
+size 260918167

models/embeddings/monolingual/bh_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 18690
+  "vocab_size": 17513
 }

models/embeddings/monolingual/bh_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:486a99b2d2d495f6861e5d7e3f58ad9b6ce636c0d60745f18dca97df31fcd2b4
-size 522039117
+oid sha256:3e537a56a3e60b9d66cfebd9427db3b6086b87ba6d254b7115af9f4351f51ee9
+size 521401495

models/embeddings/monolingual/bh_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 18690
+  "vocab_size": 17513
 }

models/subword_markov/bh_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:69cdf549946ec27557811610ffcb016a41a77c1f7890f2e6bfc79de1f616c82f
-size 106588
+oid sha256:8a66696e8530ad494e5751234f5b92a3bd0818c8159bfbb9706dcb5b0b509e8c
+size 417541

models/subword_markov/bh_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "bh",
-  "unique_contexts": 1128,
-  "total_transitions": 7494326
+  "unique_contexts": 4952,
+  "total_transitions": 4812401
 }

models/subword_markov/bh_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:615ad0f901cc2ca48f906d9d728389af05ad0d2d90163a1ef103e58ae6cd4d1d
-size 759236
+oid sha256:9c671b706107b700c31f8eb2af638805a0914ba3b55f1ce6cbdde36b234bc3e5
+size 1958764

models/subword_markov/bh_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "bh",
-  "unique_contexts": 13497,
-  "total_transitions": 7485436
+  "unique_contexts": 60879,
+  "total_transitions": 4803696
 }

models/subword_markov/bh_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f04fbf31f7e2685a99636ad40560cbf1ef9b299b1cf2c8896fc34bddef7449cc
-size 2846117
+oid sha256:d82bb62a4fc528dad712c5f681a9bc4d24b2fb4660975eb2b9fd5687311890b9
+size 6336160

models/subword_markov/bh_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "bh",
-  "unique_contexts": 101022,
-  "total_transitions": 7476546
+  "unique_contexts": 244880,
+  "total_transitions": 4794991
 }

models/subword_markov/bh_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8df27eacc02f78ff6597ef2754ad51dee35853fba207531251e99ed64431c7a6
-size 8727174
+oid sha256:2311c03179b14bfb2afa166ca250ff4c7d2dea7a46e1bf00cd42ce46698deb50
+size 15120750

models/subword_markov/bh_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "bh",
-  "unique_contexts": 405495,
-  "total_transitions": 7467656
+  "unique_contexts": 721221,
+  "total_transitions": 4786286
 }

models/subword_ngram/bh_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d4a7ae290794b22de21cc6a4e4d6b1e711f11b604ed51f2c18bb80ee312c9ec3
-size 84812
+oid sha256:8ccdb8f666b779178ca817f13ce58935b027cf7910ea8685631ae2bed535b525
+size 327904

models/subword_ngram/bh_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "bh",
-  "unique_ngrams": 6318,
-  "total_ngrams": 7494326
+  "unique_ngrams": 21796,
+  "total_ngrams": 4812401
 }

models/subword_ngram/bh_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ad55bef17a77bd5c4d71c6ad5f5002a0520dd3d2e47998884e722f97608269dc
-size 650077
+oid sha256:4a7163268a6eacd9599d89bf5194ce294c4818e15be4eee3b8f7e92ec8dbbc7d
+size 1357005

models/subword_ngram/bh_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "bh",
-  "unique_ngrams": 50779,
-  "total_ngrams": 7485436
+  "unique_ngrams": 93652,
+  "total_ngrams": 4803696
 }

models/subword_ngram/bh_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3723adc967a3e75e97a7cb5f8f8e97fb0c9ea2c0bed0ac37110ee5d6b24248d3
-size 2759247
+oid sha256:55ff05f51da64f27b2db568ecf99adc094808b5963aba504d817728e105ceaf2
+size 4398449

models/subword_ngram/bh_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "bh",
-  "unique_ngrams": 220351,
-  "total_ngrams": 7476546
+  "unique_ngrams": 295453,
+  "total_ngrams": 4794991
 }

models/tokenizer/bh_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dc89a93a2834b601d762408fa6c5c1b0e2d93181406641693ab423aa525a5d85
-size 604607
+oid sha256:8e2c6b69efb0696c2593a4e674d531da6cc186970c006b009a5da905fa2c510c
+size 609055

models/tokenizer/bh_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e403fe31557505ddad30a39ed01cff1c6aa60370ef5283dad079679f82639115
-size 993131
+oid sha256:a0da439439d6270f2063db9c16c1f281095ac0ed4458b76695c52f7cde9774aa
+size 994788

models/tokenizer/bh_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:85cd6ebc734cc4ea97dd8f8495efa400e2d1517a8388c0513880653d6d06499c
-size 1784572
+oid sha256:7e2b6202b97f515f97c09ecfc32ba51664cabebf6f035981a41fc7b7dd5f2814
+size 1811363

models/tokenizer/bh_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:adcaedcc6c154db25766a576018c150c327b28e69509230d48e109e8444b4e9c
-size 418066
+oid sha256:628a8222f74f579cd406af45e334c28746fbd9325f4a9fcd5c6793ab2534c437
+size 420480

models/vocabulary/bh_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:77d40dde0ef6e5b66ef83c70b1d41bd20dc473cb13e9ced47128fcf6bfb78f43
-size 266114
+oid sha256:ebd185f91b3ddec379354027ad53da261b000d0e2dccfa4cbfda1efe36a7354d
+size 740324

models/vocabulary/bh_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "bh",
-  "vocabulary_size": 16281,
+  "vocabulary_size": 38858,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.016656816074580676,
+    "type_token_ratio": 0.06547537905779276,
     "coverage": {
-      "top_100": 0.8005899255992907,
-      "top_1000": 0.9395661220498269,
-      "top_5000": 0.9751138764121329,
-      "top_10000": 0.9843766041897323
+      "top_100": 0.41446182158414246,
+      "top_1000": 0.6701453648682979,
+      "top_5000": 0.8302279613599608,
+      "top_10000": 0.8843223122060637
     },
-    "hapax_count": 25739,
-    "hapax_ratio": 0.6125416468348406,
-    "total_documents": 8890
+    "hapax_count": 45677,
+    "hapax_ratio": 0.5403324066954516,
+    "total_documents": 8705
   }
 }

models/word_markov/bh_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:252e6e9814b33628c12ecc02738f2fd113b1a3736388560c48bb91fe13490fdf
-size 1670612
+oid sha256:073f45c70364978f33599e10f7fceb20f9022efb2b123ccffca4352c3c559f2b
+size 5186664

models/word_markov/bh_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "bh",
-  "unique_contexts": 42274,
-  "total_transitions": 4613638
+  "unique_contexts": 84482,
+  "total_transitions": 1282391
 }

models/word_markov/bh_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:77d0a4ffa8e786438a321a5920beb57b3bdc420e63d551feb3dd0f8948ee3362
-size 4930135
+oid sha256:a451816cb173e2f9ea644994e76882f796bf7065d5fff1d37c89f78c1ae41b17
+size 14774364

models/word_markov/bh_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "bh",
-  "unique_contexts": 223237,
-  "total_transitions": 4604748
+  "unique_contexts": 519009,
+  "total_transitions": 1273686
 }

models/word_markov/bh_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:94881ee1b2c62d04c10a93deef131652cd66362bd5114616ecc62b18e7d69c3b
-size 11167272
+oid sha256:680ced9c1e8b69912207d0b18b1a212447d99ccf23ff1e2566888889e05ce256
+size 23197527

models/word_markov/bh_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "bh",
-  "unique_contexts": 557656,
-  "total_transitions": 4595859
+  "unique_contexts": 917743,
+  "total_transitions": 1264981
 }

models/word_markov/bh_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9d5c3651e210a6d2bddfa7d0bb7d0dd360a6eb40048135aba4bcb0a73a3c2a56
-size 20308057
+oid sha256:c8250fe64650886f54c3e5c9e5f460ea7418b226c2812cbd94d0837e470a92eb
+size 28187056

models/word_markov/bh_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "bh",
-  "unique_contexts": 1074886,
-  "total_transitions": 4586970
+  "unique_contexts": 1088288,
+  "total_transitions": 1256276
 }

models/word_ngram/bh_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b0c4dd9d3af349404f11790e8ef5a5c05d0d144df6541a2c72b9e8c7fa528032
-size 422945
+oid sha256:4141ecad38d9f7774f3dd9904b1a6863de3cbda29a595c2b1fbe7f0db5d83941
+size 594257

models/word_ngram/bh_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "bh",
-  "unique_ngrams": 29844,
-  "total_ngrams": 4613638
+  "unique_ngrams": 29857,
+  "total_ngrams": 1282391
 }

models/word_ngram/bh_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:41f8def9633bec7bd266d75861a561a98338565712553cd7c9053ca4c03c0278
-size 1430098
+oid sha256:f052da2c233a2332f4efd34c54fe1a284f577aad87670fed9cae61c8380ec406
+size 898283

models/word_ngram/bh_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "bh",
-  "unique_ngrams": 104826,
-  "total_ngrams": 4604748
+  "unique_ngrams": 38729,
+  "total_ngrams": 1273686
 }

models/word_ngram/bh_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ea9eb1016246ebf30425a2e109fd56a6c149bda0fe7ad0461813f8f29c1abb0c
-size 4130422
+oid sha256:4cbd4bdf202e522296fc743ecbedff3f3c2de48df39c84775dc26336cb11af98
+size 1340231

models/word_ngram/bh_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "bh",
-  "unique_ngrams": 294598,
-  "total_ngrams": 4595859
+  "unique_ngrams": 53247,
+  "total_ngrams": 1264981
 }