Upload all models and assets for avk (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.125
+    value: 4.690
   - name: best_isotropy
     type: isotropy
-    value: 0.8585
+    value: 0.8793
   - name: vocabulary_size
     type: vocab
-    value: 60886
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # AVK - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,81 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.392x | 3.33 | 0.1692% | 332,130 |
-| **16k** | 3.661x | 3.60 | 0.1827% | 307,678 |
-| **32k** | 3.908x | 3.84 | 0.1950% | 288,266 |
-| **64k** | 4.125x 🏆 | 4.06 | 0.2058% | 273,051 |
+| **8k** | 3.687x | 3.69 | 0.2370% | 256,066 |
+| **16k** | 4.050x | 4.05 | 0.2604% | 233,136 |
+| **32k** | 4.380x | 4.38 | 0.2816% | 215,576 |
+| **64k** | 4.690x 🏆 | 4.69 | 0.3015% | 201,338 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Loma:TezaLoma:Rovulegan liwot`
+**Sample 1:** `Bifa Afrika Amerika Asia Europa Oceania Koblira Awalkera sanda`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁loma : te zal oma : rov ul eg an ... (+1 more)` | 11 |
-| 16k | `▁loma : tezaloma : rov ul eg an ▁liwot` | 9 |
-| 32k | `▁loma : tezaloma : rovulegan ▁liwot` | 6 |
-| 64k | `▁loma : tezaloma : rovulegan ▁liwot` | 6 |
-
-**Sample 2:** `Bifa 
- Afrika 
-
- Amerika 
-
- Asia 
-
- Europa 
-
- Oceania 
-
- Koblira 
+| 8k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁sanda` | 9 |
+| 16k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁sanda` | 9 |
+| 32k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁sanda` | 9 |
+| 64k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁sanda` | 9 |
 
- Awalkera 
-
-L...`
+**Sample 2:** `Bifa Afrika Amerika Asia Europa Oceania Koblira Awalkera sanda`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁loma : ... (+7 more)` | 17 |
-| 16k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁loma : ... (+7 more)` | 17 |
-| 32k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁loma : ... (+7 more)` | 17 |
-| 64k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁loma : ... (+7 more)` | 17 |
-
-**Sample 3:** `Bifa 
- Afrika 
-
- Amerika 
-
- Asia 
-
- Europa 
+| 8k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁sanda` | 9 |
+| 16k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁sanda` | 9 |
+| 32k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁sanda` | 9 |
+| 64k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁sanda` | 9 |
 
- Oceania 
-
- Koblira 
-
- Awalkera 
-
-L...`
+**Sample 3:** `Bifa Afrika Amerika Asia Europa Oceania Koblira Awalkera sanda`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁loma : ... (+6 more)` | 16 |
-| 16k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁loma : ... (+6 more)` | 16 |
-| 32k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁loma : ... (+6 more)` | 16 |
-| 64k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁loma : ... (+6 more)` | 16 |
+| 8k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁sanda` | 9 |
+| 16k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁sanda` | 9 |
+| 32k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁sanda` | 9 |
+| 64k | `▁bifa ▁afrika ▁amerika ▁asia ▁europa ▁oceania ▁koblira ▁awalkera ▁sanda` | 9 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.125x compression
-- **Lowest UNK Rate:** 8k with 0.1692% unknown tokens
+- **Best Compression:** 64k achieves 4.690x compression
+- **Lowest UNK Rate:** 8k with 0.2370% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -151,57 +129,89 @@ L...`
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 3,356 🏆 | 11.71 | 88,920 | 38.8% | 64.9% |
-| **2-gram** | 346 🏆 | 8.43 | 4,094 | 58.9% | 99.3% |
-| **3-gram** | 6,742 | 12.72 | 180,799 | 34.6% | 57.5% |
-| **3-gram** | 2,400 | 11.23 | 30,298 | 24.5% | 70.5% |
-| **4-gram** | 13,085 | 13.68 | 334,115 | 31.4% | 50.6% |
-| **4-gram** | 8,768 | 13.10 | 165,638 | 16.2% | 48.8% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 4,363 | 12.09 | 65,566 | 38.8% | 59.5% |
+| **2-gram** | Subword | 284 🏆 | 8.15 | 3,342 | 63.4% | 99.6% |
+| **3-gram** | Word | 9,089 | 13.15 | 131,822 | 34.4% | 51.4% |
+| **3-gram** | Subword | 1,998 | 10.96 | 24,557 | 26.3% | 74.1% |
+| **4-gram** | Word | 16,964 | 14.05 | 222,393 | 30.3% | 44.4% |
+| **4-gram** | Subword | 7,482 | 12.87 | 124,823 | 17.3% | 50.8% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `vuest -` | 137,464 |
-| 2 | `) vuest` | 137,409 |
-| 3 | `- :` | 137,400 |
-| 4 | `( en` | 136,694 |
-| 5 | `en )` | 126,393 |
+| 1 | `en vuest` | 113,005 |
+| 2 | `of life` | 25,896 |
+| 3 | `of the` | 24,998 |
+| 4 | `the world` | 24,670 |
+| 5 | `species of` | 24,652 |
 
-**3-grams:**
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `) vuest -` | 137,409 |
-| 2 | `vuest - :` | 137,400 |
-| 3 | `en ) vuest` | 124,379 |
-| 4 | `( en )` | 123,735 |
-| 5 | `) ( en` | 67,196 |
+| 1 | `of the world` | 24,657 |
+| 2 | `mammal species of` | 24,652 |
+| 3 | `species of the` | 24,652 |
+| 4 | `taneon zo pimtayar` | 15,544 |
+| 5 | `bak taneon zo` | 15,311 |
 
-**4-grams:**
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `) vuest - :` | 137,400 |
-| 2 | `en ) vuest -` | 124,379 |
-| 3 | `( en ) vuest` | 121,727 |
-| 4 | `) ( en )` | 54,244 |
-| 5 | `species of the world` | 25,857 |
+| 1 | `species of the world` | 24,652 |
+| 2 | `mammal species of the` | 24,652 |
+| 3 | `bak taneon zo pimtayar` | 15,309 |
+| 4 | `zo pimtayar vexala dem` | 15,224 |
+| 5 | `taneon zo pimtayar vexala` | 15,223 |
+
+**2-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `a _` | 684,240 |
+| 2 | `s _` | 476,785 |
+| 3 | `_ (` | 458,313 |
+| 4 | `e _` | 387,458 |
+| 5 | `_ v` | 360,515 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ : _` | 268,667 |
+| 2 | `u s _` | 176,828 |
+| 3 | `e s t` | 175,968 |
+| 4 | `_ v u` | 167,656 |
+| 5 | `u e s` | 166,553 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `u e s t` | 164,190 |
+| 2 | `_ v u e` | 163,879 |
+| 3 | `v u e s` | 163,702 |
+| 4 | `) _ v u` | 124,953 |
+| 5 | `e s t -` | 124,892 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 346
+- **Best Perplexity:** 2-gram (subword) with 284
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~49% of corpus
+- **Coverage:** Top-1000 patterns cover ~51% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -209,55 +219,86 @@ L...`
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.7348 | 1.664 | 4.95 | 126,229 | 26.5% |
-| **1** | 1.1169 | 2.169 | 9.23 | 906 | 0.0% |
-| **2** | 0.3179 | 1.247 | 1.83 | 623,886 | 68.2% |
-| **2** | 1.0082 | 2.011 | 6.31 | 8,363 | 0.0% |
-| **3** | 0.1523 | 1.111 | 1.34 | 1,136,410 | 84.8% |
-| **3** | 0.8558 | 1.810 | 4.58 | 52,772 | 14.4% |
-| **4** | 0.1022 🏆 | 1.073 | 1.23 | 1,523,485 | 89.8% |
-| **4** | 0.7129 🏆 | 1.639 | 3.05 | 241,547 | 28.7% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.9061 | 1.874 | 5.53 | 115,135 | 9.4% |
+| **1** | Subword | 1.0381 | 2.053 | 7.87 | 900 | 0.0% |
+| **2** | Word | 0.2494 | 1.189 | 1.63 | 635,326 | 75.1% |
+| **2** | Subword | 0.9486 | 1.930 | 5.95 | 7,086 | 5.1% |
+| **3** | Word | 0.1397 | 1.102 | 1.31 | 1,030,372 | 86.0% |
+| **3** | Subword | 0.7945 | 1.734 | 4.30 | 42,171 | 20.6% |
+| **4** | Word | 0.1004 🏆 | 1.072 | 1.21 | 1,346,672 | 90.0% |
+| **4** | Subword | 0.6921 | 1.616 | 3.14 | 181,330 | 30.8% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `en forsythe david friedrich germanaf suterotik kiren sini va kulak ke patecta divatcewer kazaxo koe ...`
+2. `vuest paleobiology database lagidium viscacia katca ctenomys johannis dene internet ok ino va jontik...`
+3. `ke zosteropidae yasa ke capensis philippsi hinton en vuest walvedeyafa zveriopafa aba leptotrygon ti...`
 
-### Generated Text Samples
+**Context Size 2:**
+
+1. `en vuest paleobiology database pipistrellus kuhlii lepidus blyth en vuest animal diversity web leopa...`
+2. `of life procyon lotor grinnelli nelson and goldman tovumol thomomys umbrinus nelsoni merriam en vues...`
+3. `of the world siatos ke bata katca vas 17 oxi zo torigir ise va volkeafi is kategisafi`
+
+**Context Size 3:**
+
+1. `of the world siatos ke konakara apta dere tid ke mila veyafa katca putcuxol cephalophus ogilbyi putc...`
+2. `mammal species of the world v 3 leptailurus serval lonnbergi cabrera abrugol leptailurus serval beir...`
+3. `species of the world v 3 solenodontidae gill en vuest animal diversity web corythopis en vuest anima...`
+
+**Context Size 4:**
+
+1. `species of the world siatos ke bata katca tir aptiskafa pulasa vuestexa is xantaza en vuest mammal s...`
+2. `mammal species of the world v 3 sminthopsis griseoventer kitchener stoddart en fr vuest itis glaucom...`
+3. `bak taneon zo pimtayar vexala dem apteem sedme mammal species of the world siatos ke konakara apta d...`
+
+
+### Generated Text Samples (Subword-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `( kishida , 1814 taneon zo pimtayar . vincent van gogh nederlandaf lingesik bak muvugal ,`
-2. `) vuest - : prionailurus bengalensis ( ukraina gan wagner , fr ) ( kotava winugaf`
-3. `: pteropus temminckii temminckii ) vuest - : crocidura mariquensis shortridgei ) vuest - align :`
+1. `_(_denata_(erus_`
+2. `asa_(_sideven_t_`
+3. `e_worva_(mi_rota`
 
 **Context Size 2:**
 
-1. `vuest - : catalogue of life web project : macrotus waterhousii waterhousii ( gray , 1863 )`
-2. `) vuest - : uicn : katca dymecodon pilirostris ) dene wikispecies kotavafa vuestesa xantaza kotava w...`
-3. `- : itis : caracal caracal ) ( en , fr ) vuest - : alan p`
+1. `a_dem_sinzo_pala_`
+2. `s_rimallifa_jechy`
+3. `_(_puldaegan_baka`
 
 **Context Size 3:**
 
-1. `) vuest - : tree of life web project : rattus palmarum ( zelebor , 1869 ) (`
-2. `vuest - : uicn : katca penthetor lucasi ( dobson , 1880 ) elmol ( dymecodon pilirostris )`
-3. `en ) vuest - : paleobiology database : thomomys bottae nanus ( hall , 1941 ) ratsikisol (`
+1. `_:_citesa_)_ke_cou`
+2. `us_flowasinafa._13`
+3. `est-_:_pert_ke_cou`
 
 **Context Size 4:**
 
-1. `) vuest - : ncbi : simias ara vuestexa tekudol ( nasalis larvatus ) dene wikispecies kotavafa vueste...`
-2. `en ) vuest - : tree of life web project : sorex trowbridgii ( en ) vuest - :`
-3. `( en ) vuest - : tree of life web project : aepyceros melampus rendilis ( lönnberg , 1912`
+1. `uestexa_is_katceem_`
+2. `_vuest-_:_cites_zib`
+3. `vuestexa_iku_hulske`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 89.8% predictability
+- **Best Predictability:** Context-4 (word) with 90.0% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (241,547 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (181,330 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -273,39 +314,39 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 60,886 |
-| Total Tokens | 4,116,931 |
-| Mean Frequency | 67.62 |
-| Median Frequency | 6 |
-| Frequency Std Dev | 1162.11 |
+| Vocabulary Size | 58,132 |
+| Total Tokens | 3,516,474 |
+| Mean Frequency | 60.49 |
+| Median Frequency | 5 |
+| Frequency Std Dev | 1081.22 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | en | 140,161 |
-| 2 | vuest | 137,464 |
-| 3 | ke | 96,689 |
-| 4 | of | 56,680 |
-| 5 | tir | 40,544 |
-| 6 | is | 40,292 |
-| 7 | va | 36,873 |
-| 8 | katca | 36,170 |
-| 9 | koe | 30,224 |
-| 10 | bak | 28,949 |
+| 1 | en | 127,536 |
+| 2 | vuest | 124,885 |
+| 3 | ke | 85,674 |
+| 4 | of | 52,510 |
+| 5 | tir | 40,501 |
+| 6 | is | 37,526 |
+| 7 | katca | 36,175 |
+| 8 | va | 35,605 |
+| 9 | bak | 28,769 |
+| 10 | koe | 28,642 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | tageltaf | 2 |
-| 2 | l4 | 2 |
-| 3 | l5 | 2 |
-| 4 | l6 | 2 |
-| 5 | l8 | 2 |
-| 6 | fakaf | 2 |
-| 7 | docs | 2 |
-| 8 | 814359978 | 2 |
+| 1 | medotegalaf | 2 |
+| 2 | j1 | 2 |
+| 3 | tageltaf | 2 |
+| 4 | l4 | 2 |
+| 5 | l5 | 2 |
+| 6 | l6 | 2 |
+| 7 | l8 | 2 |
+| 8 | fakaf | 2 |
 | 9 | rozuxa | 2 |
 | 10 | eaksat | 2 |
 
@@ -313,24 +354,24 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.1598 |
-| R² (Goodness of Fit) | 0.995097 |
+| Zipf Coefficient | 1.1323 |
+| R² (Goodness of Fit) | 0.996890 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 46.2% |
-| Top 1,000 | 71.1% |
-| Top 5,000 | 86.2% |
+| Top 100 | 48.8% |
+| Top 1,000 | 72.1% |
+| Top 5,000 | 86.1% |
 | Top 10,000 | 91.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9951 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 46.2% of corpus
-- **Long Tail:** 50,886 words needed for remaining 9.0% coverage
+- **Zipf Compliance:** R²=0.9969 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 48.8% of corpus
+- **Long Tail:** 48,132 words needed for remaining 9.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -343,24 +384,116 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 50,177 | 32 | 5.680 | 1.138 | 0.8585 🏆 |
-| **mono_64d** | 50,177 | 64 | 6.303 | 1.051 | 0.8386 |
-| **mono_128d** | 50,177 | 128 | 6.840 | 0.972 | 0.7221 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8793 🏆 | 0.3481 | N/A | N/A |
+| **mono_64d** | 64 | 0.8305 | 0.2964 | N/A | N/A |
+| **mono_128d** | 128 | 0.6711 | 0.2516 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8585 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 50,177 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8793 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2987. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-a` | engada, winkapa, riftakola |
+| `-s` | latimanus, hyladelphys, adocetus |
+| `-us` | latimanus, adocetus, eptesicus |
+| `-ra` | rupera, remtrakura, prosthemadera |
+| `-on` | daemon, lavion, prostelayon |
+| `-fa` | altokafa, kalkafa, ronepafa |
+| `-afa` | altokafa, kalkafa, ronepafa |
+| `-is` | africaeaustralis, variabilis, louis |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `ensi` | 2.40x | 45 contexts | owensi, pensil, ozensis |
+| `ayar` | 1.87x | 93 contexts | gayar, vayar, iayar |
+| `urus` | 2.16x | 23 contexts | purus, urusí, gaurus |
+| `anta` | 1.52x | 73 contexts | yanta, danta, canta |
+| `imta` | 2.04x | 22 contexts | pimtas, kimtaf, krimta |
+| `tava` | 1.80x | 25 contexts | stava, kotava, yultava |
+| `atca` | 1.63x | 31 contexts | zatca, datca, catca |
+| `pimt` | 2.38x | 8 contexts | pimtas, pimtar, pimtan |
+| `stes` | 1.74x | 16 contexts | restes, lestes, estesa |
+| `neon` | 2.09x | 8 contexts | roneon, taneon, deneon |
+| `xant` | 1.53x | 19 contexts | xanta, xanto, xantik |
+| `katc` | 1.55x | 14 contexts | katca, katcaf, katcaal |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+*No significant affix co-occurrences detected.*
+
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| rumeikafa | **`rumeik-afa`** | 4.5 | `rumeik` |
+| dolekikafa | **`dolekik-afa`** | 4.5 | `dolekik` |
+| unenikafa | **`unenik-afa`** | 4.5 | `unenik` |
+| tetschener | **`tetschen-er`** | 4.5 | `tetschen` |
+| jotugalafa | **`jotugal-afa`** | 4.5 | `jotugal` |
+| gogolason | **`gogolas-on`** | 4.5 | `gogolas` |
+| rontagentimafa | **`rontagentim-afa`** | 4.5 | `rontagentim` |
+| getalteon | **`getalte-on`** | 4.5 | `getalte` |
+| azilnyofara | **`azilnyo-fa-ra`** | 3.0 | `azilnyo` |
+| tunotrara | **`tunot-ra-ra`** | 3.0 | `tunot` |
+| dimpiyison | **`dimpiy-is-on`** | 3.0 | `dimpiy` |
+| otonycteris | **`otonyct-er-is`** | 3.0 | `otonyct` |
+| rhinonicteris | **`rhinonict-er-is`** | 3.0 | `rhinonict` |
+| chrotopterus | **`chrotopt-er-us`** | 3.0 | `chrotopt` |
+| talturonon | **`taltur-on-on`** | 3.0 | `taltur` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language AVK appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -368,11 +501,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.13x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (346) |
-| Markov | **Context-4** | Highest predictability (89.8%) |
+| Tokenizer | **64k BPE** | Best compression (4.69x) |
+| N-gram | **2-gram** | Lowest perplexity (284) |
+| Markov | **Context-4** | Highest predictability (90.0%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -562,7 +696,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -578,7 +713,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 20:44:59*
+*Report Date: 2026-01-03 05:31:10*

models/embeddings/monolingual/avk_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c77e9b8ba1437da33f48cdca5b862658b27db7a2b928bd457484eca03247babd
-size 1076298214
+oid sha256:c0357f4c0fa47381c2d92e0415911327b53c389a3d486edc400fcce551ae310d
+size 1070325438

models/embeddings/monolingual/avk_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 50177
+  "vocab_size": 44454
 }

models/embeddings/monolingual/avk_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5694b8179c04c88002b40e6c2183367753202e5fa90e012bef7f5d39cfc1b2cf
-size 269762278
+oid sha256:829d093f87f92bee34bc2d9b3ee3c6e53006f7208dba8cd714618f0305b957af
+size 268184766

models/embeddings/monolingual/avk_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 50177
+  "vocab_size": 44454
 }

models/embeddings/monolingual/avk_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d7dc9c61284134af30b4d1b2aab6224be3afd4e9f49b2a64ed0c40abc179bcfa
-size 538607590
+oid sha256:7d210dae793291e501a5a742665aafce5364647858baa5dd956782a0c6d80a70
+size 535564990

models/embeddings/monolingual/avk_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 50177
+  "vocab_size": 44454
 }

models/subword_markov/avk_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9c87d050968fb890ac14fd71d59704c1f20be0cade22594a02304808593949be
-size 72153
+oid sha256:21791c83470a53c161c68f122259b7897a3a907d75078d307676c0177d2c36bb
+size 63469

models/subword_markov/avk_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "avk",
-  "unique_contexts": 906,
-  "total_transitions": 31142755
+  "unique_contexts": 900,
+  "total_transitions": 26140715
 }

models/subword_markov/avk_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f8f446b4ef71a4c8448a34fa74f0f82934b9a8860a62763f7de0514ccfd67de8
-size 477700
+oid sha256:698bcc0cfe9a7f9b86fa6f44a6777b8a6a41971070d8ab3b73f98b09f24b3b36
+size 379498

models/subword_markov/avk_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "avk",
-  "unique_contexts": 8363,
-  "total_transitions": 31112980
+  "unique_contexts": 7086,
+  "total_transitions": 26111423
 }

models/subword_markov/avk_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:30469dc41cdc5cdb5c2b8b6b30f354558cbbf809a10ce1a41cb209d02349b970
-size 1961477
+oid sha256:e97b0774917adc7e1e8666a420b5a88cece271cf99ea1b5f68cd532f0018a92b
+size 1583547

models/subword_markov/avk_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "avk",
-  "unique_contexts": 52772,
-  "total_transitions": 31083205
+  "unique_contexts": 42171,
+  "total_transitions": 26082131
 }

models/subword_markov/avk_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6d2a2dc2ccd788db6d4fe7dc4b501e476474e463f9b5daf894e7bc60a3807863
-size 6015167
+oid sha256:aecee602f0b44cb0f438169dfcceb1a8a89110470f4c323a12fc42d4af2e25ae
+size 4497423

models/subword_markov/avk_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "avk",
-  "unique_contexts": 241547,
-  "total_transitions": 31053430
+  "unique_contexts": 181330,
+  "total_transitions": 26052839
 }

models/subword_ngram/avk_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:57623fd2d7680170fb70b8a54a85080d046adabcb09f6ba248e51ab954c23704
-size 58053
+oid sha256:542c6d1e24d5e6dfba739d8aa22907d19ad252430bc7d0ccd256a501d0531f72
+size 47284

models/subword_ngram/avk_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "avk",
-  "unique_ngrams": 4094,
-  "total_ngrams": 31142755
+  "unique_ngrams": 3342,
+  "total_ngrams": 26140715
 }

models/subword_ngram/avk_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:19dd489abd0754a69d92d86b3d70f88fb08a90407bbf8d44bfe8e11b8162b8a8
-size 400583
+oid sha256:a059662c6d88d086a3fb221455feeb7c778b83b67df81086daea2aa0cddec23c
+size 326709

models/subword_ngram/avk_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "avk",
-  "unique_ngrams": 30298,
-  "total_ngrams": 31112980
+  "unique_ngrams": 24557,
+  "total_ngrams": 26111423
 }

models/subword_ngram/avk_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e008f4e015b488a0806b82114bc302e1ba6408d9dba61c412e1b9c744997da9e
-size 1909769
+oid sha256:828f99a99e0050489b41c4a9df2305254560e588c8d8d5ec02dd0d2eb1cd3530
+size 1449644

models/subword_ngram/avk_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "avk",
-  "unique_ngrams": 165638,
-  "total_ngrams": 31083205
+  "unique_ngrams": 124823,
+  "total_ngrams": 26082131
 }

models/tokenizer/avk_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f9c55784109e43a9432c17273e8dd91088994ff7e64a334adde79a110fa50a78
-size 508877
+oid sha256:dc42b2323e7bee7429a820f2ea7e2009b2fb74e192b3fe3a17923335350d22c5
+size 510898

models/tokenizer/avk_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f71b9f51b6fde0ce3af71bf329e3a16fab1dbe74661cb0d5e5795706d4346ef3
-size 787205
+oid sha256:91bac420b4e318b698464d40a4467873b19d48b58df745bb29a08985b8c357b4
+size 793808

models/tokenizer/avk_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:592345fe0811d55c932390e43251d8f96797fc431da7b02804fa98e5b8b37b49
-size 1365457
+oid sha256:f27f3f01c3515fa7385b2952b4f6df13e378c5cd044085d3bbce4d2ad95a544a
+size 1369364

models/tokenizer/avk_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:40ef85dbb33d9d8d185ae20f51d9e408ef65f997f3f54be0cc29a91476ae2b6b
-size 372682
+oid sha256:6cee4ebbae3d1c7add217a4a460590ce0d0dd21f13b527e6db5ed0db19b09351
+size 374235

models/vocabulary/avk_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9badd3be683b3de4d818a068b65f846f74b9b2c821667b9ee89e8ff9dfd1c380
-size 1067310
+oid sha256:3d2f858e168566644dadfc0198deb685658a860427034fac7e9621ba6c6c331f
+size 1016629

models/vocabulary/avk_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "avk",
-  "vocabulary_size": 60886,
+  "vocabulary_size": 58132,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.03016752857100501,
+    "type_token_ratio": 0.032232319628378144,
     "coverage": {
-      "top_100": 0.45497789208198913,
-      "top_1000": 0.6999908182559851,
-      "top_5000": 0.8483328917807131,
-      "top_10000": 0.8961334336948964
+      "top_100": 0.48045109520711343,
+      "top_1000": 0.7096278884294919,
+      "top_5000": 0.8476448324833323,
+      "top_10000": 0.8952395743698449
     },
-    "hapax_count": 65281,
-    "hapax_ratio": 0.5174173912354261,
-    "total_documents": 29775
+    "hapax_count": 57051,
+    "hapax_ratio": 0.4953074672477709,
+    "total_documents": 29292
   }
 }

models/word_markov/avk_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:45daf4d90b1c151ceeaf2eafabe0c4eb0fa4f302b0586de0d0c7c9828132282f
-size 5408930
+oid sha256:49390c536dd83430c0715d46b765fc2890bf4f4d31c5442852be6a75add7a744
+size 5111239

models/word_markov/avk_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "avk",
-  "unique_contexts": 126229,
-  "total_transitions": 6480020
+  "unique_contexts": 115135,
+  "total_transitions": 3544233
 }

models/word_markov/avk_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:490cc4a34315fffeeafc0bb6770bd3f6596968d38e894a020c7de5de669ad396
-size 12552041
+oid sha256:08a9aef65566043a63c8658765edb2f951063117cc9f1704b722bd3a5d81e0ca
+size 11858936

models/word_markov/avk_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "avk",
-  "unique_contexts": 623886,
-  "total_transitions": 6450251
+  "unique_contexts": 635326,
+  "total_transitions": 3514941
 }

models/word_markov/avk_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:65c7f7d50c4f9d2ab9884a6416fce5a63b12d44a814648773af5258e5bdca88c
-size 18746428
+oid sha256:e2e6a6857df1e25e0460139d92c8f9bcb539dd1655bc6e5082d9041f991d382e
+size 17254031

models/word_markov/avk_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "avk",
-  "unique_contexts": 1136410,
-  "total_transitions": 6420500
+  "unique_contexts": 1030372,
+  "total_transitions": 3485649
 }

models/word_markov/avk_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cb6d9225de08c55bad0b5b1c52937c5a7af6b67625fbe4a58d094126e4da419d
-size 24304336
+oid sha256:e5184dd4650abc3b773720e48a95afd66db6b4b1d6b78b6c2e54b8b6893ef521
+size 22000561

models/word_markov/avk_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "avk",
-  "unique_contexts": 1523485,
-  "total_transitions": 6390750
+  "unique_contexts": 1346672,
+  "total_transitions": 3456357
 }

models/word_ngram/avk_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0be3cc4f49219f8041292992a9db6978b8873a52d64b4d15b2db6d55a0194adb
-size 1119286
+oid sha256:ebd625f3d160fcc752c3eb1214deddbd9ec1e26c8d5b3228ab9a0bb4750e7921
+size 894389

models/word_ngram/avk_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "avk",
-  "unique_ngrams": 88920,
-  "total_ngrams": 6480020
+  "unique_ngrams": 65566,
+  "total_ngrams": 3544233
 }

models/word_ngram/avk_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2399a2f98604cf3b9e3bceaee6ac55b8f98eedfafb4ece0a8e26bf3ee2a0fb04
-size 2239472
+oid sha256:fa6659f593d23add7532113c38a73745568b1d4569187b0a4f675162d5edf125
+size 1677310

models/word_ngram/avk_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "avk",
-  "unique_ngrams": 180799,
-  "total_ngrams": 6450251
+  "unique_ngrams": 131822,
+  "total_ngrams": 3514941
 }

models/word_ngram/avk_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cddb3b457d830002d6f39a4845d76b1095745537a700a6adf4d1c7fe8365ea85
-size 4190267
+oid sha256:14e71556139c544ae7786e07e17777a6d13f770333441e2995879912fc5cd48b
+size 3019124

models/word_ngram/avk_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "avk",
-  "unique_ngrams": 334115,
-  "total_ngrams": 6420500
+  "unique_ngrams": 222393,
+  "total_ngrams": 3485649
 }