Upload all models and assets for ann (latest)

.gitattributes

@@ -39,3 +39,5 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text
+visualizations/ngram_coverage.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: ann
-language_name: ANN
+language_name: Obolo
 language_family: atlantic_other
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-atlantic_other
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.351
+    value: 4.353
   - name: best_isotropy
     type: isotropy
-    value: 0.1947
+    value: 0.1710
   - name: vocabulary_size
     type: vocab
     value: 0
 generated: 2026-01-03
 ---
 
-# ANN - Wikilangs Models
+# Obolo - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **ANN** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Obolo** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -60,7 +70,7 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
 - [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
@@ -80,39 +90,39 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 4.112x | 4.12 | 0.1464% | 133,892 |
-| **16k** | 4.351x 🏆 | 4.36 | 0.1549% | 126,546 |
+| **8k** | 4.116x | 4.12 | 0.1487% | 128,471 |
+| **16k** | 4.353x 🏆 | 4.36 | 0.1572% | 121,476 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Luwis òso 14 ìre ogwu ubọọn̄ me Furans bene me acha abayaage ire usen mkpa kan̄ ...`
+**Sample 1:** `Ọrọn môkọt ire: Ebi Ọrọn (ife) Ido Ọrọn (ama mè ere) Ọrọn (Mkpulu-ija) Usem Ọrọn...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁lu wis ▁òso ▁ 1 4 ▁ìre ▁ogwu ▁ubọọn̄ ▁me ... (+23 more)` | 33 |
-| 16k | `▁luwis ▁òso ▁ 1 4 ▁ìre ▁ogwu ▁ubọọn̄ ▁me ▁furans ... (+21 more)` | 31 |
+| 8k | `▁ọrọn ▁môkọt ▁ire : ▁ebi ▁ọrọn ▁( ife ) ▁ido ... (+17 more)` | 27 |
+| 16k | `▁ọrọn ▁môkọt ▁ire : ▁ebi ▁ọrọn ▁( ife ) ▁ido ... (+17 more)` | 27 |
 
-**Sample 2:** `Mọlita ìre ido me Yurop. thumb|Egop Ido Mọlita thumb|Iman̄-ido Mọlita thumb|Okwa...`
+**Sample 2:** `Nde ìre oke mgbọ òsoso usen jaaba. Ekisa nde ifuk onyan̄ mè isa ifuk acha si. Nd...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁mọlita ▁ìre ▁ido ▁me ▁yurop . ▁thumb | egop ▁ido ... (+19 more)` | 29 |
-| 16k | `▁mọlita ▁ìre ▁ido ▁me ▁yurop . ▁thumb | egop ▁ido ... (+19 more)` | 29 |
+| 8k | `▁nde ▁ìre ▁oke ▁mgbọ ▁òsoso ▁usen ▁jaaba . ▁ekisa ▁nde ... (+27 more)` | 37 |
+| 16k | `▁nde ▁ìre ▁oke ▁mgbọ ▁òsoso ▁usen ▁jaaba . ▁ekisa ▁nde ... (+26 more)` | 36 |
 
-**Sample 3:** `Saint Marino ìre ido me Yurop. thumb|Egop Ido Saint Marino thumb|Iman̄-ido Saint...`
+**Sample 3:** `Ọngari (òrere Hungary me usem Ebeke, mè ire Magyarország me usem Ọn̄gari) ìre id...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁saint ▁marino ▁ìre ▁ido ▁me ▁yurop . ▁thumb | egop ... (+17 more)` | 27 |
-| 16k | `▁saint ▁marino ▁ìre ▁ido ▁me ▁yurop . ▁thumb | egop ... (+17 more)` | 27 |
+| 8k | `▁ọ n gari ▁( òrere ▁h ungary ▁me ▁usem ▁ebeke ... (+28 more)` | 38 |
+| 16k | `▁ọngari ▁( òrere ▁hungary ▁me ▁usem ▁ebeke , ▁mè ▁ire ... (+19 more)` | 29 |
 
 
 ### Key Findings
 
-- **Best Compression:** 16k achieves 4.351x compression
-- **Lowest UNK Rate:** 8k with 0.1464% unknown tokens
+- **Best Compression:** 16k achieves 4.353x compression
+- **Lowest UNK Rate:** 8k with 0.1487% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -129,12 +139,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
 |--------|---------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | Word | 1,111 | 10.12 | 2,498 | 36.3% | 78.1% |
-| **2-gram** | Subword | 241 🏆 | 7.91 | 1,230 | 67.8% | 99.7% |
-| **3-gram** | Word | 1,927 | 10.91 | 3,289 | 25.2% | 65.4% |
-| **3-gram** | Subword | 1,414 | 10.47 | 7,165 | 32.1% | 80.6% |
-| **4-gram** | Word | 3,376 | 11.72 | 4,802 | 16.9% | 48.5% |
-| **4-gram** | Subword | 4,883 | 12.25 | 24,184 | 20.0% | 55.6% |
+| **2-gram** | Word | 1,077 | 10.07 | 2,406 | 36.6% | 78.7% |
+| **2-gram** | Subword | 236 🏆 | 7.88 | 1,214 | 68.6% | 99.7% |
+| **3-gram** | Word | 1,871 | 10.87 | 3,155 | 25.1% | 66.0% |
+| **3-gram** | Subword | 1,382 | 10.43 | 7,013 | 32.7% | 80.9% |
+| **4-gram** | Word | 3,277 | 11.68 | 4,661 | 17.3% | 49.0% |
+| **4-gram** | Subword | 4,770 | 12.22 | 23,560 | 20.3% | 56.1% |
+| **5-gram** | Word | 2,207 | 11.11 | 2,786 | 18.0% | 56.1% |
+| **5-gram** | Subword | 9,921 | 13.28 | 38,424 | 14.8% | 42.7% |
 
 ### Top 5 N-grams by Size
 
@@ -142,21 +154,21 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `me lek` | 1,089 |
-| 2 | `me agan̄` | 844 |
-| 3 | `me emen` | 801 |
+| 1 | `me lek` | 1,069 |
+| 2 | `me agan̄` | 831 |
+| 3 | `me emen` | 791 |
 | 4 | `ido ya` | 458 |
-| 5 | `ichit me` | 381 |
+| 5 | `ichit me` | 380 |
 
 **3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `agan̄ ichep ura` | 217 |
+| 1 | `agan̄ ichep ura` | 215 |
 | 2 | `me ido ya` | 190 |
-| 3 | `me agan̄ osiki` | 183 |
-| 4 | `agan̄ mbum ura` | 176 |
-| 5 | `me agan̄ inyọn̄` | 172 |
+| 3 | `me agan̄ osiki` | 182 |
+| 4 | `agan̄ mbum ura` | 174 |
+| 5 | `me agan̄ inyọn̄` | 171 |
 
 **4-grams (Word):**
 
@@ -165,45 +177,65 @@ Below are sample sentences tokenized with each vocabulary size:
 | 1 | `me agan̄ mbum ura` | 103 |
 | 2 | `me agan̄ ichep ura` | 96 |
 | 3 | `me ido ya ìre` | 62 |
-| 4 | `agan̄ inyọn̄ mbum ura` | 56 |
-| 5 | `ewabe ichit me emen` | 50 |
+| 4 | `agan̄ inyọn̄ mbum ura` | 55 |
+| 5 | `me usem uket chieen̄` | 50 |
+
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ene ewabe ichit me emen` | 48 |
+| 2 | `me agan̄ inyọn̄ mbum ura` | 38 |
+| 3 | `me agan̄ osiki mbum ura` | 37 |
+| 4 | `me agan̄ osiki ichep ura` | 36 |
+| 5 | `otu ifuk ebi ìluk me` | 33 |
 
 **2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `e _` | 19,443 |
-| 2 | `_ i` | 16,978 |
-| 3 | `_ m` | 15,100 |
-| 4 | `_ e` | 11,773 |
-| 5 | `a _` | 9,778 |
+| 1 | `e _` | 19,047 |
+| 2 | `_ i` | 16,640 |
+| 3 | `_ m` | 14,795 |
+| 4 | `_ e` | 11,553 |
+| 5 | `a _` | 9,463 |
 
 **3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `_ m e` | 7,822 |
-| 2 | `m e _` | 7,755 |
-| 3 | `a n̄ _` | 4,098 |
-| 4 | `r e _` | 4,084 |
-| 5 | `e _ i` | 3,290 |
+| 1 | `_ m e` | 7,633 |
+| 2 | `m e _` | 7,573 |
+| 3 | `r e _` | 4,030 |
+| 4 | `a n̄ _` | 3,973 |
+| 5 | `e _ i` | 3,231 |
 
 **4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `_ m e _` | 7,635 |
-| 2 | `_ m è _` | 2,895 |
-| 3 | `l e k _` | 2,350 |
-| 4 | `_ a g a` | 1,914 |
-| 5 | `a g a n̄` | 1,906 |
+| 1 | `_ m e _` | 7,454 |
+| 2 | `_ m è _` | 2,866 |
+| 3 | `l e k _` | 2,314 |
+| 4 | `_ a g a` | 1,867 |
+| 5 | `_ e b i` | 1,856 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ a g a n̄` | 1,844 |
+| 2 | `_ e b i _` | 1,713 |
+| 3 | `_ m e _ a` | 1,652 |
+| 4 | `_ ì r e _` | 1,547 |
+| 5 | `a g a n̄ _` | 1,513 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram (subword) with 241
+- **Best Perplexity:** 2-gram (subword) with 236
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~56% of corpus
+- **Coverage:** Top-1000 patterns cover ~43% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -219,14 +251,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
 |---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.7773 | 1.714 | 4.68 | 9,818 | 22.3% |
-| **1** | Subword | 1.1325 | 2.192 | 8.75 | 290 | 0.0% |
-| **2** | Word | 0.2751 | 1.210 | 1.61 | 45,714 | 72.5% |
-| **2** | Subword | 1.0635 | 2.090 | 5.50 | 2,537 | 0.0% |
-| **3** | Word | 0.1069 | 1.077 | 1.18 | 73,222 | 89.3% |
-| **3** | Subword | 0.7725 | 1.708 | 3.21 | 13,932 | 22.7% |
-| **4** | Word | 0.0450 🏆 | 1.032 | 1.07 | 86,066 | 95.5% |
-| **4** | Subword | 0.4908 | 1.405 | 2.04 | 44,651 | 50.9% |
+| **1** | Word | 0.7698 | 1.705 | 4.61 | 9,664 | 23.0% |
+| **1** | Subword | 1.1244 | 2.180 | 8.63 | 290 | 0.0% |
+| **2** | Word | 0.2727 | 1.208 | 1.60 | 44,320 | 72.7% |
+| **2** | Subword | 1.0645 | 2.091 | 5.47 | 2,502 | 0.0% |
+| **3** | Word | 0.1063 | 1.076 | 1.18 | 70,635 | 89.4% |
+| **3** | Subword | 0.7756 | 1.712 | 3.20 | 13,671 | 22.4% |
+| **4** | Word | 0.0452 🏆 | 1.032 | 1.07 | 82,762 | 95.5% |
+| **4** | Subword | 0.4895 | 1.404 | 2.03 | 43,657 | 51.0% |
 
 ### Generated Text Samples (Word-based)
 
@@ -234,27 +266,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `me akparalek ijọn̄ ikọkọp uji ijọn̄ ìjeen̄ ibe ekọp esaba èwê alt left thumb iman̄ ido`
-2. `mè owuwa ebi barazilu thumb iman̄ kan̄ sà belgiọm burazil pọtugalu thumb egop ubọọn̄ me zambia`
-3. `agan̄ mkpulu ubọọn̄ yi ìre siera leyon togo me yurop ìniluk me agan̄ ichepura eyi india`
+1. `me lek kiban̄ ekitimbe akọn̄ kè ofirikosok gbọgbọ otu ifuk mè ikpọk ya ìre isilam inu`
+2. `mè ijọn̄ ido ya me naijiria agan̄ ichep ura sà ogwu òkitaak chieen̄ ikpọ mè mgbọ`
+3. `agan̄ mkpulu uwu usọ ifuk ene ewabe ichit me emen mgbọ etiopia ìkup me agọọk nkween̄`
 
 **Context Size 2:**
 
-1. `me lek ike uti ìkatibi me èwê dubai`
-2. `me agan̄ osiki ire ebi kè ofifi èwê ere òla ijọn̄ eba ìkup ewuuk ewuuk me mgbọ`
-3. `me emen senturi akọp mè gweregwen ene ewabe me emen mîwa iraka efie ita thumb egop agan̄`
+1. `me lek adasi nkwukwuuk cha isa ikije mè isa me ikeya lesoto ìre ge me lek ijọn̄`
+2. `me agan̄ inyọn̄ mbum ura mè emen awaji atik me agan̄ inyọn̄ ichep ura mè agan̄ ichep`
+3. `me emen wire môkọtbe irọ inu due process odobe`
 
 **Context Size 3:**
 
-1. `agan̄ ichep ura me agan̄ ichep ura ruwanda me agan̄ osiki me ido naijiria achubọk inyinyi òrom òkuku...`
-2. `me ido ya bene me senturi 16 re 19 emen awaji atilantik ore achubọk ebon ere ewe inyam`
-3. `me agan̄ osiki naijiria ama mkpulu ìtatap ikana ọmọ ìre ginì ikwetọ me agan̄ inyọn̄ mbum ura ido`
+1. `agan̄ ichep ura oniin̄ mè ikpọkpọk ikire ibot ikọ me ukpatu ebi uga ifuk ibot chereyi kperiọọn̄ owuw...`
+2. `me ido ya ìnire usem furenchi mèlek usem wolof sa me ebi otoko wolof erebe ebi ìwawa ichit`
+3. `me agan̄ osiki sà ruwanda burundi mè kongo kinshasa ekup me agan̄ inyọn̄ abia mè akwa ibom me`
 
 **Context Size 4:**
 
-1. `me agan̄ mbum ura naija me agan̄ inyọn̄ mbum ura mè silovakia me agan̄ osiki mbum ura me lek`
-2. `me agan̄ ichep ura eyi amerika agan̄ inyọn̄ thumb ọrọsi thumb ọrọsi môkọt ikaan̄ esese mbet unwen mè...`
-3. `me ido ya ìre eyi ebọkọbe itap me 17 akọp mè onyan̄ ge otu ifuk ene ìluk me ido`
+1. `me agan̄ mbum ura kan̄ sà emen awaji atilantik otap ikana ọmọ me agan̄ inyọn̄ afirika agan̄ inyọn̄ ò...`
+2. `me agan̄ ichep ura isi ire iteke indus me agan̄ mbum ura me afirika mè ire si òso 20`
+3. `me ido ya ìre furench ire îre akọp irek me efit si re akọp mè irek go me efit`
 
 
 ### Generated Text Samples (Subword-based)
@@ -263,34 +295,34 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_ma_mè_mè_erirup`
-2. `e_ògan̄_chikilukp`
-3. `ituwupanwebọte_m`
+1. `_idem_ijan̄_masee`
+2. `e_erelukp_mi_lup`
+3. `irit_enyikp_n_si`
 
 **Context Size 2:**
 
-1. `e_lek_mè_ìkuria_m`
-2. `_ififuuke_si_ichọ`
-3. `_me_òkuk_use_agan̄`
+1. `e_obageeleki_[cor`
+2. `_ike_ubọ_erere_ik`
+3. `_me_ebi_mè_mem_ya`
 
 **Context Size 3:**
 
-1. `_me_lek_ebi_kibert`
-2. `me_levan_obolo_pas`
-3. `an̄_echieen̄_ya_orọr`
+1. `_me_emire_ge,_mè_d`
+2. `me_jodan_ichechich`
+3. `re_ge,_ìkigwook,_ò`
 
 **Context Size 4:**
 
-1. `_me_lek_ìmọnọ_ire_o`
-2. `_mè_ebi_kè_ama-ile_`
-3. `lek_<raw_mate>_igba`
+1. `_me_si_inwàn_ikwaan̄`
+2. `_mè_ikikaan̄ge;_me_<`
+3. `lek_ebi_ìkike_eriọọ`
 
 
 ### Key Findings
 
 - **Best Predictability:** Context-4 (word) with 95.5% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (44,651 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (43,657 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -306,36 +338,36 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 4,243 |
-| Total Tokens | 93,606 |
-| Mean Frequency | 22.06 |
+| Vocabulary Size | 4,154 |
+| Total Tokens | 89,919 |
+| Mean Frequency | 21.65 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 154.88 |
+| Frequency Std Dev | 152.24 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | me | 7,683 |
-| 2 | mè | 2,927 |
-| 3 | agan̄ | 1,906 |
-| 4 | ido | 1,757 |
-| 5 | ebi | 1,749 |
-| 6 | ìre | 1,621 |
-| 7 | lek | 1,606 |
-| 8 | eyi | 1,291 |
-| 9 | ya | 1,169 |
-| 10 | emen | 1,082 |
+| 1 | me | 7,502 |
+| 2 | mè | 2,898 |
+| 3 | agan̄ | 1,854 |
+| 4 | ebi | 1,728 |
+| 5 | ìre | 1,597 |
+| 6 | lek | 1,576 |
+| 7 | ido | 1,514 |
+| 8 | eyi | 1,242 |
+| 9 | ya | 1,165 |
+| 10 | emen | 1,065 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | iyaak | 2 |
-| 2 | medvedev | 2 |
-| 3 | race | 2 |
-| 4 | lenin | 2 |
-| 5 | walvis | 2 |
+| 1 | lanzarote | 2 |
+| 2 | iyaak | 2 |
+| 3 | medvedev | 2 |
+| 4 | race | 2 |
+| 5 | lenin | 2 |
 | 6 | ọkọlọba | 2 |
 | 7 | ǹkọọn̄ | 2 |
 | 8 | edeh | 2 |
@@ -346,24 +378,24 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.1690 |
-| R² (Goodness of Fit) | 0.990906 |
+| Zipf Coefficient | 1.1652 |
+| R² (Goodness of Fit) | 0.990704 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 59.7% |
-| Top 1,000 | 87.8% |
+| Top 100 | 59.9% |
+| Top 1,000 | 87.9% |
 | Top 5,000 | 0.0% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9909 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 59.7% of corpus
-- **Long Tail:** -5,757 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9907 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 59.9% of corpus
+- **Long Tail:** -5,846 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -379,37 +411,40 @@ Below are text samples generated from each subword-based Markov chain model:
 
 ### 5.1 Cross-Lingual Alignment
 
-> *Note: Multilingual alignment visualization not available for this language.*
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
 
 
 ### 5.2 Model Comparison
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.1947 🏆 | 0.5302 | N/A | N/A |
-| **mono_64d** | 64 | 0.0325 | 0.5569 | N/A | N/A |
-| **mono_128d** | 128 | 0.0071 | 0.5825 | N/A | N/A |
+| **mono_32d** | 32 | 0.1710 🏆 | 0.5365 | N/A | N/A |
+| **mono_64d** | 64 | 0.0323 | 0.5579 | N/A | N/A |
+| **mono_128d** | 128 | 0.0059 | 0.5505 | N/A | N/A |
+| **aligned_32d** | 32 | 0.1710 | 0.5499 | 0.0111 | 0.1274 |
+| **aligned_64d** | 64 | 0.0323 | 0.5740 | 0.0222 | 0.1717 |
+| **aligned_128d** | 128 | 0.0059 | 0.5600 | 0.0139 | 0.1634 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.1947 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.5565. Lower values indicate better semantic separation.
-- **Alignment Quality:** No aligned models evaluated in this run.
+- **Best Isotropy:** mono_32d with 0.1710 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.5548. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 2.2% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
 ## 6.  Morphological Analysis (Experimental)
 
-> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
-
 This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
 
 ### 6.1 Productivity & Complexity
 
 | Metric | Value | Interpretation | Recommendation |
 |--------|-------|----------------|----------------|
-| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
-| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **0.314** | High formulaic/idiomatic content | - |
 
 ### 6.2 Affix Inventory (Productive Units)
 
@@ -418,14 +453,14 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Prefixes
 | Prefix | Examples |
 |--------|----------|
-| `-ek` | ekefuk, ekwut, ekpọkbe |
-| `-ik` | ikike, ikwuk, ikisip |
+| `-ek` | eket, ekpabe, ekibene |
+| `-ik` | ikpele, ikinen̄e, ikinyam |
 
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-n̄` | mun̄, òrọriọọn̄, ijejeen̄ |
-| `-be` | îgebe, olobobe, eweekbe |
+| `-n̄` | ugwun̄, akwaan̄, esun̄ |
+| `-be` | ekpabe, îkwube, ejibibe |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -433,18 +468,18 @@ Bound stems are high-frequency subword units that are semantically cohesive but
 
 | Stem | Cohesion | Substitutability | Examples |
 |------|----------|------------------|----------|
-| `gọọk` | 1.55x | 19 contexts | agọọk, igọọk, îgọọk |
-| `tumu` | 1.48x | 21 contexts | ìtumu, òtumu, etumu |
-| `kpul` | 1.58x | 16 contexts | ikpulu, ìkpulu, îkpulu |
-| `sibi` | 1.52x | 18 contexts | ìsibi, osibi, îsibi |
-| `kana` | 1.46x | 20 contexts | ikana, okana, ìkana |
-| `kikp` | 1.44x | 19 contexts | ikikpa, ikikpọ, ìkikpa |
-| `kisa` | 1.46x | 18 contexts | okisa, îkisa, ekisa |
-| `chie` | 1.55x | 14 contexts | chief, echieek, ìchieek |
-| `kpọk` | 1.42x | 17 contexts | okpọk, akpọk, ọkpọk |
-| `gbaa` | 1.46x | 15 contexts | ogbaan̄, egbaan̄, îgbaan̄ |
-| `ikaa` | 1.60x | 11 contexts | ikaan̄, ikikaan̄, ekikaan̄ |
-| `riọọ` | 1.54x | 12 contexts | nriọọk, riọọn̄, oriọọn̄ |
+| `gọọk` | 1.51x | 19 contexts | igọọk, agọọk, ogọọk |
+| `tumu` | 1.46x | 21 contexts | ntumu, etumu, itumu |
+| `kpul` | 1.58x | 16 contexts | ikpulu, îkpulu, òkpulu |
+| `sibi` | 1.51x | 18 contexts | nsibi, ìsibi, îsibi |
+| `kikp` | 1.44x | 19 contexts | òkikpa, òkikpọ, ekikpọ |
+| `kana` | 1.41x | 20 contexts | ekana, ìkana, nkana |
+| `chie` | 1.55x | 14 contexts | chief, ichieek, ìchieek |
+| `riọọ` | 1.63x | 12 contexts | nriọọk, riọọn̄, nriọọn̄ |
+| `kisa` | 1.43x | 17 contexts | òkisa, ìkisa, îkisa |
+| `gbaa` | 1.46x | 15 contexts | ògbaan̄, egbaan̄, ìgbaan̄ |
+| `ikaa` | 1.61x | 11 contexts | ikaan̄, enikaan̄, ebikaan̄ |
+| `kpọk` | 1.39x | 16 contexts | ukpọk, okpọk, ọkpọk |
 
 ### 6.4 Affix Compatibility (Co-occurrence)
 
@@ -452,9 +487,9 @@ This table shows which prefixes and suffixes most frequently co-occur on the sam
 
 | Prefix | Suffix | Frequency | Examples |
 |--------|--------|-----------|----------|
-| `-ek` | `-be` | 15 words | ekpọkbe, ekifukbe |
-| `-ik` | `-n̄` | 15 words | ikwaan̄, ikikaan̄ |
-| `-ek` | `-n̄` | 10 words | ekimọọn̄, ekekaan̄ |
+| `-ik` | `-n̄` | 15 words | ikikpan̄, ikwaan̄ |
+| `-ek` | `-be` | 13 words | ekpabe, ekaan̄be |
+| `-ek` | `-n̄` | 10 words | ekijeen̄, ekitoon̄ |
 
 ### 6.5 Recursive Morpheme Segmentation
 
@@ -465,23 +500,25 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 | ekinyambe | **`ek-inyam-be`** | 6.0 | `inyam` |
 | ekitumube | **`ek-itumu-be`** | 6.0 | `itumu` |
 | ekigwenbe | **`ek-igwen-be`** | 6.0 | `igwen` |
-| echichinibe | **`echichini-be`** | 4.5 | `echichini` |
-| echieekbe | **`echieek-be`** | 4.5 | `echieek` |
-| ekikpulube | **`ek-ik-pulu-be`** | 4.5 | `pulu` |
 | ikichieek | **`ik-ichieek`** | 4.5 | `ichieek` |
 | ekichichini | **`ek-ichichini`** | 4.5 | `ichichini` |
-| ekekikpulu | **`ek-ek-ik-pulu`** | 4.5 | `pulu` |
+| echichinibe | **`echichini-be`** | 4.5 | `echichini` |
 | ekiweweek | **`ek-iweweek`** | 4.5 | `iweweek` |
-| ikibieen̄ | **`ik-ibiee-n̄`** | 3.0 | `ibiee` |
-| ekititiin̄ | **`ek-ititii-n̄`** | 3.0 | `ititii` |
-| etitiin̄be | **`etitii-n̄-be`** | 3.0 | `etitii` |
-| îriọọn̄be | **`îriọọ-n̄-be`** | 3.0 | `îriọọ` |
+| ekikpulube | **`ek-ik-pulu-be`** | 4.5 | `pulu` |
+| ekekikpulu | **`ek-ek-ik-pulu`** | 4.5 | `pulu` |
+| echieekbe | **`echieek-be`** | 4.5 | `echieek` |
 | ekikpukpo | **`ek-ik-pukpo`** | 3.0 | `pukpo` |
+| ikpọchieen̄ | **`ik-pọchiee-n̄`** | 3.0 | `pọchiee` |
+| ikibieen̄ | **`ik-ibiee-n̄`** | 3.0 | `ibiee` |
+| eriọọn̄be | **`eriọọ-n̄-be`** | 3.0 | `eriọọ` |
+| egbaan̄be | **`egbaa-n̄-be`** | 3.0 | `egbaa` |
 
 ### 6.6 Linguistic Interpretation
 
 > **Automated Insight:**
-The language ANN appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+The language Obolo shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
 
 ---
 ## 7. Summary & Recommendations
@@ -493,7 +530,7 @@ The language ANN appears to be more isolating or has a highly fixed vocabulary.
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
 | Tokenizer | **16k BPE** | Best compression (4.35x) |
-| N-gram | **2-gram** | Lowest perplexity (241) |
+| N-gram | **2-gram** | Lowest perplexity (236) |
 | Markov | **Context-4** | Highest predictability (95.5%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
@@ -708,4 +745,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 05:13:43*
+*Report Date: 2026-01-03 14:12:13*

models/embeddings/aligned/ann_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc209d93d13b120c8c95cba7ed8ea3511e7f3a3be58c71d6b52f823aba7e3a58
+size 1025972707

models/embeddings/aligned/ann_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "ann", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/ann_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ecbf9194a3110e9db3f94be28e4d970c89d4ae602cd76c5e4114d930138c369
+size 65664

models/embeddings/aligned/ann_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "ann",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 361,
+  "vocab_size": 1896
+}

models/embeddings/aligned/ann_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:046109a4422a9a603a48c2abd0d192b758882631e72ac429557dc6f857e7b375
+size 256516579

models/embeddings/aligned/ann_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "ann", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/ann_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ac69b84943611045a295f4bb05aa98d313c747e3e3ff20df0a86d90596c100c
+size 4224

models/embeddings/aligned/ann_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "ann",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 361,
+  "vocab_size": 1896
+}

models/embeddings/aligned/ann_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b00d28e874e41679c615dce915575ad329279f02507e33677323ebb64d12a95e
+size 513001955

models/embeddings/aligned/ann_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "ann", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/ann_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8417b28498d4ded38028edb9ebd5965b99df3b52961d1191e017b63208828a47
+size 16512

models/embeddings/aligned/ann_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "ann",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 361,
+  "vocab_size": 1896
+}

models/embeddings/monolingual/ann_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ce6196c912d733cb4ecafe0bf16e0d4c89052442dc36f0dbeb46533cf966be8c
-size 1026069585
+oid sha256:dc209d93d13b120c8c95cba7ed8ea3511e7f3a3be58c71d6b52f823aba7e3a58
+size 1025972707

models/embeddings/monolingual/ann_128d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 128
   },
-  "vocab_size": 1989
+  "vocab_size": 1896
 }

models/embeddings/monolingual/ann_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a08097b63d64495bc93a7d4418641dbfe3c922fe0270d5f7d226318ad60d9b5c
-size 256542033
+oid sha256:046109a4422a9a603a48c2abd0d192b758882631e72ac429557dc6f857e7b375
+size 256516579

models/embeddings/monolingual/ann_32d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 32
   },
-  "vocab_size": 1989
+  "vocab_size": 1896
 }

models/embeddings/monolingual/ann_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2afa2ec65fb8007208da56111bface363437e9bec87e59314877a23345b9ad5e
-size 513051217
+oid sha256:b00d28e874e41679c615dce915575ad329279f02507e33677323ebb64d12a95e
+size 513001955

models/embeddings/monolingual/ann_64d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 64
   },
-  "vocab_size": 1989
+  "vocab_size": 1896
 }

models/subword_markov/ann_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:133284a0112df666902ff1c8b3f87c98d928ca5e21cc040ac90519fc9b835b57
-size 24522
+oid sha256:14f1e4d2b16ca83acf99005e1d8f2ef8ddd448d536aa3453c2aed6989f746a93
+size 24116

models/subword_markov/ann_markov_ctx1_subword_metadata.json

@@ -3,5 +3,5 @@
   "variant": "subword",
   "language": "ann",
   "unique_contexts": 290,
-  "total_transitions": 538876
+  "total_transitions": 517578
 }

models/subword_markov/ann_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e64bce1c8aaafc988a5935e2ed628caf16c7fa8930527c46f194ce12fe4dc84a
-size 114074
+oid sha256:7c0bec0998b02a5eeadf5a6d286401974bf066faefc1a30b5c0c82d32484aab0
+size 106684

models/subword_markov/ann_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "ann",
-  "unique_contexts": 2537,
-  "total_transitions": 538383
+  "unique_contexts": 2502,
+  "total_transitions": 517106
 }

models/subword_markov/ann_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:72b96d3df529060de51aa1fc4dcb9ab588af0929fb1e0eb2b958bff6d7a6b633
-size 338702
+oid sha256:969d810e955fb0ffe5cc55f4bb726f3ed8d55b5683f15470101b94fd4aa037d2
+size 339360

models/subword_markov/ann_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "ann",
-  "unique_contexts": 13932,
-  "total_transitions": 537890
+  "unique_contexts": 13671,
+  "total_transitions": 516634
 }

models/subword_markov/ann_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e09659d50aeeca9954a013b1ca95d57253b9f038aa3a3b0c4df7c0d2fc0caa1e
-size 785453
+oid sha256:c7dcda352ac96f30b9157dcea88dff5aabf8fc2a3e539499978e16442601f454
+size 772993

models/subword_markov/ann_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "ann",
-  "unique_contexts": 44651,
-  "total_transitions": 537397
+  "unique_contexts": 43657,
+  "total_transitions": 516162
 }

models/subword_ngram/ann_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7b7879bdc16bf8a3f29b779a338720f6ce381e91fa8ba5e247518dc7ca628446
-size 17029
+oid sha256:54f121852b71196304a2a8986075bbf0fff2db6ac1109f14ff830a4533d19c57
+size 16779

models/subword_ngram/ann_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "ann",
-  "unique_ngrams": 1230,
-  "total_ngrams": 538876
+  "unique_ngrams": 1214,
+  "total_ngrams": 517578
 }

models/subword_ngram/ann_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a22334c4abf593637f95f14f3195c7e9b977cab70ffe6f1ebd3ac954408c1e58
-size 82404
+oid sha256:207a1b64805bd946990386c767a8f886d3619c8a8403b02522c13f431e351ae3
+size 80691

models/subword_ngram/ann_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "ann",
-  "unique_ngrams": 7165,
-  "total_ngrams": 538383
+  "unique_ngrams": 7013,
+  "total_ngrams": 517106
 }

models/subword_ngram/ann_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:edff385989d8cb6ddba4a7096924e12c5f750a23c337bf178ed91664b6d637f8
-size 292528
+oid sha256:b1033bd3465d84b199fe00a7423d9aea18725a1aedee84e66a6041c26eb24017
+size 289755

models/subword_ngram/ann_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "ann",
-  "unique_ngrams": 24184,
-  "total_ngrams": 537890
+  "unique_ngrams": 23560,
+  "total_ngrams": 516634
 }

models/subword_ngram/ann_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:afb932726cd84df3cca7c67ff8301e1bf81a11bdf5dd6200fb84663dfb585e6c
+size 498830

models/subword_ngram/ann_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "ann",
+  "unique_ngrams": 38424,
+  "total_ngrams": 516162
+}

models/tokenizer/ann_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:685a0e9ed7c0107b85e07b0286c03eb8b5cd6fc5ada26786ea117b602d0ca131
-size 511767
+oid sha256:b74f59549b94c9bbb91cffef6bc44cacadb4f440a30004c5287afd482a9adcf2
+size 510998

models/tokenizer/ann_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:13e4e094459602ef55a87f4cad2bf9b7eaaeb3ca6f3d5f2b6a36760cf76610a2
-size 374711
+oid sha256:89d553eae70a0c59b46d72bb3b7a2e84791949a1b7ea431452fca72f093214e2
+size 374759

models/vocabulary/ann_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8145b130fe314515c8d0e309cb26039c412cf86c9b7eeafd8670fe99f9cdda88
-size 69333
+oid sha256:9005473acebe3bdbe8c244c492a69eb7e7aad2dd59e7a35a9a4ca06fbaa2f2e5
+size 67918

models/vocabulary/ann_vocabulary_metadata.json

@@ -1,16 +1,16 @@
 {
   "language": "ann",
-  "vocabulary_size": 4243,
+  "vocabulary_size": 4154,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.09944472997349618,
+    "type_token_ratio": 0.10174905739421869,
     "coverage": {
-      "top_100": 0.562737450998176,
-      "top_1000": 0.8278562142878737,
-      "top_5000": 0.9509427497455433
+      "top_100": 0.564390448261416,
+      "top_1000": 0.8281210724759112,
+      "top_5000": 0.9506179304566401
     },
-    "hapax_count": 5625,
-    "hapax_ratio": 0.5700243210376976,
-    "total_documents": 493
+    "hapax_count": 5561,
+    "hapax_ratio": 0.5724137931034483,
+    "total_documents": 472
   }
 }

models/word_markov/ann_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0ba0963e3a66739914b91e29d2fbde25bf375a4973d157084f09f818b169e344
-size 334500
+oid sha256:e2c2df7df4f25457b6030282137cac1a9855ddf787d0f90b75ec2442c37970bc
+size 326225

models/word_markov/ann_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "ann",
-  "unique_contexts": 9818,
-  "total_transitions": 98738
+  "unique_contexts": 9664,
+  "total_transitions": 95008
 }

models/word_markov/ann_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:32de02b53a9ca990e598e15d5ee6ad630df14d4ec653fb740b4a7d42bdb26a5f
-size 845652
+oid sha256:1421fc6795e12b35a51213ce822b2be78a7f4e7d0d13d3be77f09b266e60c0cd
+size 823564

models/word_markov/ann_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "ann",
-  "unique_contexts": 45714,
-  "total_transitions": 98245
+  "unique_contexts": 44320,
+  "total_transitions": 94536
 }

models/word_markov/ann_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ed08c5a00857486794ad9d25afd118b795f050f0abf1cd60420d0f37b096aa05
-size 1239826
+oid sha256:2e1cc33784190581e139822dc45a48a345ed714d3a644a8855faf21ee3b75546
+size 1200817

models/word_markov/ann_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "ann",
-  "unique_contexts": 73222,
-  "total_transitions": 97752
+  "unique_contexts": 70635,
+  "total_transitions": 94064
 }

models/word_markov/ann_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9952b21c610e474602739d4608bcdf47886b1e13139eb17d44e2d73ff21aff7f
-size 1490446
+oid sha256:3282749b62f9354b96f20943251ce901348cdeac9a17c0d4a1a76554e03042c7
+size 1439251

models/word_markov/ann_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "ann",
-  "unique_contexts": 86066,
-  "total_transitions": 97259
+  "unique_contexts": 82762,
+  "total_transitions": 93592
 }