Upload all models and assets for ann (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.268
+    value: 4.351
   - name: best_isotropy
     type: isotropy
-    value: 0.1631
+    value: 0.1947
   - name: vocabulary_size
     type: vocab
-    value: 4397
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # ANN - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,51 +70,49 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 4.015x | 4.00 | 0.1279% | 136,774 |
-| **16k** | 4.268x 🏆 | 4.25 | 0.1360% | 128,655 |
+| **8k** | 4.112x | 4.12 | 0.1464% | 133,892 |
+| **16k** | 4.351x 🏆 | 4.36 | 0.1549% | 126,546 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Nọwè ìre ido me Yurop.
-
-150px|thumb|Iman̄-ido Nọwè
-
-Ọgbọn̄:Ido
-Ọgbọn̄:Yurop`
+**Sample 1:** `Luwis òso 14 ìre ogwu ubọọn̄ me Furans bene me acha abayaage ire usen mkpa kan̄ ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁nọwè ▁ìre ▁ido ▁me ▁yurop . ▁ 1 5 0 ... (+14 more)` | 24 |
-| 16k | `▁nọwè ▁ìre ▁ido ▁me ▁yurop . ▁ 1 5 0 ... (+14 more)` | 24 |
+| 8k | `▁lu wis ▁òso ▁ 1 4 ▁ìre ▁ogwu ▁ubọọn̄ ▁me ... (+23 more)` | 33 |
+| 16k | `▁luwis ▁òso ▁ 1 4 ▁ìre ▁ogwu ▁ubọọn̄ ▁me ▁furans ... (+21 more)` | 31 |
 
-**Sample 2:** `Ọrọn ìre mkpulu-ija ge òkjp me Agan̄ Mkpulu Akwa Ibom me ido Naijiria. Mkpulu-ij...`
+**Sample 2:** `Mọlita ìre ido me Yurop. thumb|Egop Ido Mọlita thumb|Iman̄-ido Mọlita thumb|Okwa...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ọrọn ▁ìre ▁mkpulu - ija ▁ge ▁òk j p ▁me ... (+15 more)` | 25 |
-| 16k | `▁ọrọn ▁ìre ▁mkpulu - ija ▁ge ▁òkjp ▁me ▁agan̄ ▁mkpulu ... (+13 more)` | 23 |
+| 8k | `▁mọlita ▁ìre ▁ido ▁me ▁yurop . ▁thumb | egop ▁ido ... (+19 more)` | 29 |
+| 16k | `▁mọlita ▁ìre ▁ido ▁me ▁yurop . ▁thumb | egop ▁ido ... (+19 more)` | 29 |
 
-**Sample 3:** `Ikpọ̀n̄ ìre mfut uko òkitibi mè ito lek me emen ijọn̄. Îre mfut eyi acha ge.
-
-
-Ọ...`
+**Sample 3:** `Saint Marino ìre ido me Yurop. thumb|Egop Ido Saint Marino thumb|Iman̄-ido Saint...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ikpọ̀n̄ ▁ìre ▁mfut ▁uko ▁òkitibi ▁mè ▁ito ▁lek ▁me ▁emen ... (+13 more)` | 23 |
-| 16k | `▁ikpọ̀n̄ ▁ìre ▁mfut ▁uko ▁òkitibi ▁mè ▁ito ▁lek ▁me ▁emen ... (+13 more)` | 23 |
+| 8k | `▁saint ▁marino ▁ìre ▁ido ▁me ▁yurop . ▁thumb | egop ... (+17 more)` | 27 |
+| 16k | `▁saint ▁marino ▁ìre ▁ido ▁me ▁yurop . ▁thumb | egop ... (+17 more)` | 27 |
 
 
 ### Key Findings
 
-- **Best Compression:** 16k achieves 4.268x compression
-- **Lowest UNK Rate:** 8k with 0.1279% unknown tokens
+- **Best Compression:** 16k achieves 4.351x compression
+- **Lowest UNK Rate:** 8k with 0.1464% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -121,55 +121,87 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 1,256 🏆 | 10.29 | 3,472 | 37.4% | 75.8% |
-| **2-gram** | 253 🏆 | 7.98 | 1,421 | 67.5% | 99.4% |
-| **3-gram** | 2,655 | 11.37 | 5,764 | 25.2% | 58.7% |
-| **3-gram** | 1,454 | 10.51 | 7,986 | 32.8% | 79.6% |
-| **4-gram** | 4,959 | 12.28 | 9,244 | 18.4% | 44.6% |
-| **4-gram** | 4,906 | 12.26 | 25,888 | 20.8% | 56.1% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 1,111 | 10.12 | 2,498 | 36.3% | 78.1% |
+| **2-gram** | Subword | 241 🏆 | 7.91 | 1,230 | 67.8% | 99.7% |
+| **3-gram** | Word | 1,927 | 10.91 | 3,289 | 25.2% | 65.4% |
+| **3-gram** | Subword | 1,414 | 10.47 | 7,165 | 32.1% | 80.6% |
+| **4-gram** | Word | 3,376 | 11.72 | 4,802 | 16.9% | 48.5% |
+| **4-gram** | Subword | 4,883 | 12.25 | 24,184 | 20.0% | 55.6% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `me lek` | 1,089 |
+| 2 | `me agan̄` | 844 |
+| 3 | `me emen` | 801 |
+| 4 | `ido ya` | 458 |
+| 5 | `ichit me` | 381 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `agan̄ ichep ura` | 217 |
+| 2 | `me ido ya` | 190 |
+| 3 | `me agan̄ osiki` | 183 |
+| 4 | `agan̄ mbum ura` | 176 |
+| 5 | `me agan̄ inyọn̄` | 172 |
+
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `agan ̄` | 1,895 |
-| 2 | `me lek` | 1,071 |
-| 3 | `me agan` | 841 |
-| 4 | `me emen` | 793 |
-| 5 | `ijọn ̄` | 694 |
+| 1 | `me agan̄ mbum ura` | 103 |
+| 2 | `me agan̄ ichep ura` | 96 |
+| 3 | `me ido ya ìre` | 62 |
+| 4 | `agan̄ inyọn̄ mbum ura` | 56 |
+| 5 | `ewabe ichit me emen` | 50 |
 
-**3-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `me agan ̄` | 831 |
-| 2 | `̄ - mkpulu` | 373 |
-| 3 | `agan ̄ -` | 372 |
-| 4 | `̄ a me` | 371 |
-| 5 | `ọgbọn ̄ :` | 339 |
+| 1 | `e _` | 19,443 |
+| 2 | `_ i` | 16,978 |
+| 3 | `_ m` | 15,100 |
+| 4 | `_ e` | 11,773 |
+| 5 | `a _` | 9,778 |
 
-**4-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `agan ̄ - mkpulu` | 364 |
-| 2 | `agan ̄ inyọn ̄` | 302 |
-| 3 | `̄ ichep - ura` | 255 |
-| 4 | `̄ mbum - ura` | 235 |
-| 5 | `. ọgbọn ̄ :` | 233 |
+| 1 | `_ m e` | 7,822 |
+| 2 | `m e _` | 7,755 |
+| 3 | `a n̄ _` | 4,098 |
+| 4 | `r e _` | 4,084 |
+| 5 | `e _ i` | 3,290 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ m e _` | 7,635 |
+| 2 | `_ m è _` | 2,895 |
+| 3 | `l e k _` | 2,350 |
+| 4 | `_ a g a` | 1,914 |
+| 5 | `a g a n̄` | 1,906 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 253
+- **Best Perplexity:** 2-gram (subword) with 241
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
 - **Coverage:** Top-1000 patterns cover ~56% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
@@ -179,55 +211,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.6614 | 1.582 | 4.53 | 10,250 | 33.9% |
-| **1** | 1.2658 | 2.405 | 9.65 | 293 | 0.0% |
-| **2** | 0.3064 | 1.237 | 1.78 | 46,339 | 69.4% |
-| **2** | 1.1141 | 2.165 | 5.65 | 2,827 | 0.0% |
-| **3** | 0.1431 | 1.104 | 1.26 | 82,397 | 85.7% |
-| **3** | 0.7678 | 1.703 | 3.09 | 15,977 | 23.2% |
-| **4** | 0.0688 🏆 | 1.049 | 1.11 | 104,013 | 93.1% |
-| **4** | 0.4695 🏆 | 1.385 | 1.96 | 49,379 | 53.0% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.7773 | 1.714 | 4.68 | 9,818 | 22.3% |
+| **1** | Subword | 1.1325 | 2.192 | 8.75 | 290 | 0.0% |
+| **2** | Word | 0.2751 | 1.210 | 1.61 | 45,714 | 72.5% |
+| **2** | Subword | 1.0635 | 2.090 | 5.50 | 2,537 | 0.0% |
+| **3** | Word | 0.1069 | 1.077 | 1.18 | 73,222 | 89.3% |
+| **3** | Subword | 0.7725 | 1.708 | 3.21 | 13,932 | 22.7% |
+| **4** | Word | 0.0450 🏆 | 1.032 | 1.07 | 86,066 | 95.5% |
+| **4** | Subword | 0.4908 | 1.405 | 2.04 | 44,651 | 50.9% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `̄ mbum - paeonia mè ebot mè ìre anam oron - enenwaan ̄ sa me acha`
-2. `me lek keya serum ; mè ikaan ̄ inu ike ikpa ebi kè ido ya me`
-3. `. erieen ̄ igege inu kire ama ebilene ; ọmọ ìluk me < emirate ] eyi`
+1. `me akparalek ijọn̄ ikọkọp uji ijọn̄ ìjeen̄ ibe ekọp esaba èwê alt left thumb iman̄ ido`
+2. `mè owuwa ebi barazilu thumb iman̄ kan̄ sà belgiọm burazil pọtugalu thumb egop ubọọn̄ me zambia`
+3. `agan̄ mkpulu ubọọn̄ yi ìre siera leyon togo me yurop ìniluk me agan̄ ichepura eyi india`
 
 **Context Size 2:**
 
-1. `agan ̄ osiki mè ichep - ura kan ̄ , ike ekpabe me ibebene emen 1990 cha`
-2. `me lek otu - ifuk ebi ìluk me lek èwê sayara me afirika agan ̄ osiki mbum`
-3. `me agan ̄ - mkpulu 36 cha ọmọ ore ama ibot irân ire bagidadi . ọgbọn ̄`
+1. `me lek ike uti ìkatibi me èwê dubai`
+2. `me agan̄ osiki ire ebi kè ofifi èwê ere òla ijọn̄ eba ìkup ewuuk ewuuk me mgbọ`
+3. `me emen senturi akọp mè gweregwen ene ewabe me emen mîwa iraka efie ita thumb egop agan̄`
 
 **Context Size 3:**
 
-1. `me agan ̄ òsiki . emen - awaji atilantik , emen - awaji eyi india , emen -`
-2. `̄ - mkpulu yi ìnan ̄ a me ujit fo ulom , mè ìyaka ìbọkọ si mkpukpe òmimin`
-3. `agan ̄ - mkpulu yi obenbe ìre 9 , 251 km² , sà otu - ifuk ifit ya`
+1. `agan̄ ichep ura me agan̄ ichep ura ruwanda me agan̄ osiki me ido naijiria achubọk inyinyi òrom òkuku...`
+2. `me ido ya bene me senturi 16 re 19 emen awaji atilantik ore achubọk ebon ere ewe inyam`
+3. `me agan̄ osiki naijiria ama mkpulu ìtatap ikana ọmọ ìre ginì ikwetọ me agan̄ inyọn̄ mbum ura ido`
 
 **Context Size 4:**
 
-1. `agan ̄ - mkpulu yi ìre okwaan ̄ imo , òkilibi iraka me okike ijọn ̄ kan ̄ .`
-2. `agan ̄ inyọn ̄ mè ọfọkọ agan ̄ osiki . iman ̄ yi ìlibi inan ̄ a me ofifi`
-3. `̄ ichep - ura , bawuchi mè gombe me agan ̄ osiki , mè kogi me agan ̄ inyọn`
+1. `me agan̄ mbum ura naija me agan̄ inyọn̄ mbum ura mè silovakia me agan̄ osiki mbum ura me lek`
+2. `me agan̄ ichep ura eyi amerika agan̄ inyọn̄ thumb ọrọsi thumb ọrọsi môkọt ikaan̄ esese mbet unwen mè...`
+3. `me ido ya ìre eyi ebọkọbe itap me 17 akọp mè onyan̄ ge otu ifuk ene ìluk me ido`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_ma_mè_mè_erirup`
+2. `e_ògan̄_chikilukp`
+3. `ituwupanwebọte_m`
+
+**Context Size 2:**
+
+1. `e_lek_mè_ìkuria_m`
+2. `_ififuuke_si_ichọ`
+3. `_me_òkuk_use_agan̄`
+
+**Context Size 3:**
+
+1. `_me_lek_ebi_kibert`
+2. `me_levan_obolo_pas`
+3. `an̄_echieen̄_ya_orọr`
+
+**Context Size 4:**
+
+1. `_me_lek_ìmọnọ_ire_o`
+2. `_mè_ebi_kè_ama-ile_`
+3. `lek_<raw_mate>_igba`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 93.1% predictability
+- **Best Predictability:** Context-4 (word) with 95.5% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (49,379 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (44,651 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -243,38 +306,38 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 4,397 |
-| Total Tokens | 93,822 |
-| Mean Frequency | 21.34 |
+| Vocabulary Size | 4,243 |
+| Total Tokens | 93,606 |
+| Mean Frequency | 22.06 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 149.48 |
+| Frequency Std Dev | 154.88 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | me | 7,521 |
-| 2 | mè | 2,898 |
-| 3 | agan | 1,911 |
-| 4 | ebi | 1,731 |
-| 5 | ido | 1,637 |
-| 6 | ìre | 1,613 |
-| 7 | lek | 1,578 |
-| 8 | eyi | 1,242 |
-| 9 | ya | 1,175 |
-| 10 | emen | 1,068 |
+| 1 | me | 7,683 |
+| 2 | mè | 2,927 |
+| 3 | agan̄ | 1,906 |
+| 4 | ido | 1,757 |
+| 5 | ebi | 1,749 |
+| 6 | ìre | 1,621 |
+| 7 | lek | 1,606 |
+| 8 | eyi | 1,291 |
+| 9 | ya | 1,169 |
+| 10 | emen | 1,082 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | 3500 | 2 |
-| 2 | iyaak | 2 |
-| 3 | medvedev | 2 |
-| 4 | race | 2 |
-| 5 | lenin | 2 |
+| 1 | iyaak | 2 |
+| 2 | medvedev | 2 |
+| 3 | race | 2 |
+| 4 | lenin | 2 |
+| 5 | walvis | 2 |
 | 6 | ọkọlọba | 2 |
-| 7 | ǹkọọn | 2 |
+| 7 | ǹkọọn̄ | 2 |
 | 8 | edeh | 2 |
 | 9 | ogwuile | 2 |
 | 10 | bruxelles | 2 |
@@ -283,24 +346,24 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.1589 |
-| R² (Goodness of Fit) | 0.990794 |
+| Zipf Coefficient | 1.1690 |
+| R² (Goodness of Fit) | 0.990906 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 58.8% |
-| Top 1,000 | 87.2% |
+| Top 100 | 59.7% |
+| Top 1,000 | 87.8% |
 | Top 5,000 | 0.0% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9908 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 58.8% of corpus
-- **Long Tail:** -5,603 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9909 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 59.7% of corpus
+- **Long Tail:** -5,757 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -313,24 +376,115 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 1,947 | 32 | 2.277 | 0.449 | 0.1631 🏆 |
-| **mono_64d** | 1,947 | 64 | 2.234 | 0.444 | 0.0371 |
-| **mono_128d** | 1,947 | 128 | 2.248 | 0.444 | 0.0066 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.1947 🏆 | 0.5302 | N/A | N/A |
+| **mono_64d** | 64 | 0.0325 | 0.5569 | N/A | N/A |
+| **mono_128d** | 128 | 0.0071 | 0.5825 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.1631 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 1,947 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.1947 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.5565. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ek` | ekefuk, ekwut, ekpọkbe |
+| `-ik` | ikike, ikwuk, ikisip |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-n̄` | mun̄, òrọriọọn̄, ijejeen̄ |
+| `-be` | îgebe, olobobe, eweekbe |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `gọọk` | 1.55x | 19 contexts | agọọk, igọọk, îgọọk |
+| `tumu` | 1.48x | 21 contexts | ìtumu, òtumu, etumu |
+| `kpul` | 1.58x | 16 contexts | ikpulu, ìkpulu, îkpulu |
+| `sibi` | 1.52x | 18 contexts | ìsibi, osibi, îsibi |
+| `kana` | 1.46x | 20 contexts | ikana, okana, ìkana |
+| `kikp` | 1.44x | 19 contexts | ikikpa, ikikpọ, ìkikpa |
+| `kisa` | 1.46x | 18 contexts | okisa, îkisa, ekisa |
+| `chie` | 1.55x | 14 contexts | chief, echieek, ìchieek |
+| `kpọk` | 1.42x | 17 contexts | okpọk, akpọk, ọkpọk |
+| `gbaa` | 1.46x | 15 contexts | ogbaan̄, egbaan̄, îgbaan̄ |
+| `ikaa` | 1.60x | 11 contexts | ikaan̄, ikikaan̄, ekikaan̄ |
+| `riọọ` | 1.54x | 12 contexts | nriọọk, riọọn̄, oriọọn̄ |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-ek` | `-be` | 15 words | ekpọkbe, ekifukbe |
+| `-ik` | `-n̄` | 15 words | ikwaan̄, ikikaan̄ |
+| `-ek` | `-n̄` | 10 words | ekimọọn̄, ekekaan̄ |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| ekinyambe | **`ek-inyam-be`** | 6.0 | `inyam` |
+| ekitumube | **`ek-itumu-be`** | 6.0 | `itumu` |
+| ekigwenbe | **`ek-igwen-be`** | 6.0 | `igwen` |
+| echichinibe | **`echichini-be`** | 4.5 | `echichini` |
+| echieekbe | **`echieek-be`** | 4.5 | `echieek` |
+| ekikpulube | **`ek-ik-pulu-be`** | 4.5 | `pulu` |
+| ikichieek | **`ik-ichieek`** | 4.5 | `ichieek` |
+| ekichichini | **`ek-ichichini`** | 4.5 | `ichichini` |
+| ekekikpulu | **`ek-ek-ik-pulu`** | 4.5 | `pulu` |
+| ekiweweek | **`ek-iweweek`** | 4.5 | `iweweek` |
+| ikibieen̄ | **`ik-ibiee-n̄`** | 3.0 | `ibiee` |
+| ekititiin̄ | **`ek-ititii-n̄`** | 3.0 | `ititii` |
+| etitiin̄be | **`etitii-n̄-be`** | 3.0 | `etitii` |
+| îriọọn̄be | **`îriọọ-n̄-be`** | 3.0 | `îriọọ` |
+| ekikpukpo | **`ek-ik-pukpo`** | 3.0 | `pukpo` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language ANN appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -338,11 +492,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.27x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (253) |
-| Markov | **Context-4** | Highest predictability (93.1%) |
+| Tokenizer | **16k BPE** | Best compression (4.35x) |
+| N-gram | **2-gram** | Lowest perplexity (241) |
+| Markov | **Context-4** | Highest predictability (95.5%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -532,7 +687,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -548,7 +704,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 06:05:57*
+*Report Date: 2026-01-03 05:13:43*

models/embeddings/monolingual/ann_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6c8ac147d21bde736ae8d0674b799ef6f26441430d0e7754d09ab5da9be7f851
-size 1026025890
+oid sha256:ce6196c912d733cb4ecafe0bf16e0d4c89052442dc36f0dbeb46533cf966be8c
+size 1026069585

models/embeddings/monolingual/ann_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 1947
+  "vocab_size": 1989
 }

models/embeddings/monolingual/ann_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:89d9bc50c043837d5b812b9600be7dba0384f9a55413156e947a0bcbc30ca5c1
-size 256530594
+oid sha256:a08097b63d64495bc93a7d4418641dbfe3c922fe0270d5f7d226318ad60d9b5c
+size 256542033

models/embeddings/monolingual/ann_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 1947
+  "vocab_size": 1989
 }

models/embeddings/monolingual/ann_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0da62b6af0039fbc61e23dffa4254301632136ee334f5519f04819ab39be4a25
-size 513029026
+oid sha256:2afa2ec65fb8007208da56111bface363437e9bec87e59314877a23345b9ad5e
+size 513051217

models/embeddings/monolingual/ann_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 1947
+  "vocab_size": 1989
 }

models/subword_markov/ann_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ecf547e06e5712d4b04214294e5fe958b9bf03373aa50626175860065f20428e
-size 26355
+oid sha256:133284a0112df666902ff1c8b3f87c98d928ca5e21cc040ac90519fc9b835b57
+size 24522

models/subword_markov/ann_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "ann",
-  "unique_contexts": 293,
-  "total_transitions": 548630
+  "unique_contexts": 290,
+  "total_transitions": 538876
 }

models/subword_markov/ann_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f1269622d9cb0765bc517988d1e40192230b066ef5a1629ea9cff8a1101e8a24
-size 117957
+oid sha256:e64bce1c8aaafc988a5935e2ed628caf16c7fa8930527c46f194ce12fe4dc84a
+size 114074

models/subword_markov/ann_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "ann",
-  "unique_contexts": 2827,
-  "total_transitions": 548137
+  "unique_contexts": 2537,
+  "total_transitions": 538383
 }

models/subword_markov/ann_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:81bc0f939df0f38a1f5ec9338173240ea58f4d86fc0a431cf7504471584893c3
-size 363748
+oid sha256:72b96d3df529060de51aa1fc4dcb9ab588af0929fb1e0eb2b958bff6d7a6b633
+size 338702

models/subword_markov/ann_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "ann",
-  "unique_contexts": 15977,
-  "total_transitions": 547644
+  "unique_contexts": 13932,
+  "total_transitions": 537890
 }

models/subword_markov/ann_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:267ef9b402c45a76709f66b10ed72c3504d3ae9ca221e297e7c4c603c2d7470c
-size 840617
+oid sha256:e09659d50aeeca9954a013b1ca95d57253b9f038aa3a3b0c4df7c0d2fc0caa1e
+size 785453

models/subword_markov/ann_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "ann",
-  "unique_contexts": 49379,
-  "total_transitions": 547151
+  "unique_contexts": 44651,
+  "total_transitions": 537397
 }

models/subword_ngram/ann_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:69538475aea5f87aef1184268f8bfc1a0b8dbb16c1a141680da52ca2adf5b059
-size 18665
+oid sha256:7b7879bdc16bf8a3f29b779a338720f6ce381e91fa8ba5e247518dc7ca628446
+size 17029

models/subword_ngram/ann_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "ann",
-  "unique_ngrams": 1421,
-  "total_ngrams": 548630
+  "unique_ngrams": 1230,
+  "total_ngrams": 538876
 }

models/subword_ngram/ann_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1a148b3112af41ea4f39af82bc1c4be69f72c67b657e0eeb480f61328928eff9
-size 90871
+oid sha256:a22334c4abf593637f95f14f3195c7e9b977cab70ffe6f1ebd3ac954408c1e58
+size 82404

models/subword_ngram/ann_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "ann",
-  "unique_ngrams": 7986,
-  "total_ngrams": 548137
+  "unique_ngrams": 7165,
+  "total_ngrams": 538383
 }

models/subword_ngram/ann_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:57b1e35698e957b0f60223159efebdd60b66929722e830d1399e6b4612b37666
-size 314192
+oid sha256:edff385989d8cb6ddba4a7096924e12c5f750a23c337bf178ed91664b6d637f8
+size 292528

models/subword_ngram/ann_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "ann",
-  "unique_ngrams": 25888,
-  "total_ngrams": 547644
+  "unique_ngrams": 24184,
+  "total_ngrams": 537890
 }

models/tokenizer/ann_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:abf621f3a63c6598066de3019333d0af711209d13793bd1949e9c2d8369cf043
-size 511896
+oid sha256:685a0e9ed7c0107b85e07b0286c03eb8b5cd6fc5ada26786ea117b602d0ca131
+size 511767

models/tokenizer/ann_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b73ce8c5900da87366023f15d65ca67944aa8743a56de8103a1ad2424ebe8ada
-size 375004
+oid sha256:13e4e094459602ef55a87f4cad2bf9b7eaaeb3ca6f3d5f2b6a36760cf76610a2
+size 374711

models/vocabulary/ann_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:58d080164da488d03c299a8e9a02ddb54cd4b2da9b5cb12db34d81e88b321982
-size 70840
+oid sha256:8145b130fe314515c8d0e309cb26039c412cf86c9b7eeafd8670fe99f9cdda88
+size 69333

models/vocabulary/ann_vocabulary_metadata.json

@@ -1,16 +1,16 @@
 {
   "language": "ann",
-  "vocabulary_size": 4397,
+  "vocabulary_size": 4243,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.10227580737453947,
+    "type_token_ratio": 0.09944472997349618,
     "coverage": {
-      "top_100": 0.5540742674148956,
-      "top_1000": 0.8211076867477136,
-      "top_5000": 0.9479184443797496,
-      "top_10000": 0.9981126961340387
+      "top_100": 0.562737450998176,
+      "top_1000": 0.8278562142878737,
+      "top_5000": 0.9509427497455433
     },
-    "hapax_count": 5791,
-    "hapax_ratio": 0.5684138201806046,
+    "hapax_count": 5625,
+    "hapax_ratio": 0.5700243210376976,
     "total_documents": 493
   }
 }

models/word_markov/ann_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d2e556d1fd7d9a59f1dffeb5b564dd904143106170ef739f02c6f80b56b5172c
-size 341591
+oid sha256:0ba0963e3a66739914b91e29d2fbde25bf375a4973d157084f09f818b169e344
+size 334500

models/word_markov/ann_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "ann",
-  "unique_contexts": 10250,
-  "total_transitions": 133901
+  "unique_contexts": 9818,
+  "total_transitions": 98738
 }

models/word_markov/ann_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8281341dbefc67d701918a39a9e93bc92ab9c2e2461b6a982a5c37d16ed5ab04
-size 866375
+oid sha256:32de02b53a9ca990e598e15d5ee6ad630df14d4ec653fb740b4a7d42bdb26a5f
+size 845652

models/word_markov/ann_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "ann",
-  "unique_contexts": 46339,
-  "total_transitions": 133408
+  "unique_contexts": 45714,
+  "total_transitions": 98245
 }

models/word_markov/ann_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:93e13f3f7c176e0031e47e25b899f5aed4bbd76fe9a9ac1d9c2f76379c8b8fd4
-size 1357580
+oid sha256:ed08c5a00857486794ad9d25afd118b795f050f0abf1cd60420d0f37b096aa05
+size 1239826

models/word_markov/ann_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "ann",
-  "unique_contexts": 82397,
-  "total_transitions": 132915
+  "unique_contexts": 73222,
+  "total_transitions": 97752
 }

models/word_markov/ann_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2cbd65a9ec2496a97b6bb243bb584b5dd545fdfd674798917387d2da66487463
-size 1704982
+oid sha256:9952b21c610e474602739d4608bcdf47886b1e13139eb17d44e2d73ff21aff7f
+size 1490446

models/word_markov/ann_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "ann",
-  "unique_contexts": 104013,
-  "total_transitions": 132423
+  "unique_contexts": 86066,
+  "total_transitions": 97259
 }

models/word_ngram/ann_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:14fbde219525c990b739a22b458dc11166917bc6789d7b73d11e5231eb380678
-size 45646
+oid sha256:2c61da640686bd6bfeec4d6576ef9e29d217c1fa800dd2af2b9880484a2384b7
+size 34828

models/word_ngram/ann_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "ann",
-  "unique_ngrams": 3472,
-  "total_ngrams": 133901
+  "unique_ngrams": 2498,
+  "total_ngrams": 98738
 }

models/word_ngram/ann_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c41b88becd4a74c9903a966352910559008a2ae297ba677e163bdd3c91de9298
-size 84226
+oid sha256:e5769309a279012b1cb2f9f14506c918160ad7147e5ffe7fbb92c676234d85cc
+size 51996

models/word_ngram/ann_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "ann",
-  "unique_ngrams": 5764,
-  "total_ngrams": 133408
+  "unique_ngrams": 3289,
+  "total_ngrams": 98245
 }

models/word_ngram/ann_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c947edc75b3e415ed42a680d46a1cbe7325d564a49d11895f12e31dba3adef32
-size 146597
+oid sha256:cf708333454bbbf754ad5ccdfbe88eb025359ee3adeafccce332299eea699ef1
+size 82752

models/word_ngram/ann_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "ann",
-  "unique_ngrams": 9244,
-  "total_ngrams": 132915
+  "unique_ngrams": 4802,
+  "total_ngrams": 97752
 }