Upload all models and assets for ami (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 3.626
+    value: 3.608
   - name: best_isotropy
     type: isotropy
-    value: 0.8477
+    value: 0.8374
   - name: vocabulary_size
     type: vocab
-    value: 31948
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # AMI - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,56 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.080x | 3.05 | 0.1304% | 771,216 |
-| **16k** | 3.309x | 3.28 | 0.1401% | 717,947 |
-| **32k** | 3.478x | 3.45 | 0.1473% | 682,898 |
-| **64k** | 3.626x 🏆 | 3.59 | 0.1536% | 655,126 |
+| **8k** | 3.161x | 3.16 | 0.4493% | 709,382 |
+| **16k** | 3.338x | 3.34 | 0.4744% | 671,788 |
+| **32k** | 3.486x | 3.49 | 0.4954% | 643,295 |
+| **64k** | 3.608x 🏆 | 3.61 | 0.5128% | 621,527 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `sapal(幼苗)
-
-Maripa' no mako ko sapal no panay.
-
-
-Kasasiwasiw:Siwkulang 'Amis`
+**Sample 1:** `makomod(統治) I a mihecaan, misatapang a makomod ko Ripon to Taywan tangasa i a mi...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁sapal ( 幼 苗 ) ▁mar ipa ' ▁no ▁mako ... (+10 more)` | 20 |
-| 16k | `▁sapal ( 幼 苗 ) ▁mar ipa ' ▁no ▁mako ... (+10 more)` | 20 |
-| 32k | `▁sapal ( 幼 苗 ) ▁mar ipa ' ▁no ▁mako ... (+10 more)` | 20 |
-| 64k | `▁sapal ( 幼 苗 ) ▁maripa ' ▁no ▁mako ▁ko ... (+9 more)` | 19 |
+| 8k | `▁makomod ( 統 治 ) ▁i ▁a ▁mihecaan , ▁misatapang ... (+11 more)` | 21 |
+| 16k | `▁makomod ( 統治 ) ▁i ▁a ▁mihecaan , ▁misatapang ▁a ... (+10 more)` | 20 |
+| 32k | `▁makomod ( 統治 ) ▁i ▁a ▁mihecaan , ▁misatapang ▁a ... (+10 more)` | 20 |
+| 64k | `▁makomod ( 統治 ) ▁i ▁a ▁mihecaan , ▁misatapang ▁a ... (+10 more)` | 20 |
 
-**Sample 2:** `Talip(Sekato;Hakama^;Sukun;Sokato;Tarip、kuwaping a sowal: 裙)`
+**Sample 2:** `malitengay(老人家) Romadiw ci malitengay. (老人家在唱歌) 'Amis`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁tal ip ( s ek ato ; hak ama ^ ... (+19 more)` | 29 |
-| 16k | `▁tal ip ( s ek ato ; hak ama ^ ... (+19 more)` | 29 |
-| 32k | `▁talip ( sek ato ; hak ama ^ ; s ... (+15 more)` | 25 |
-| 64k | `▁talip ( sek ato ; hak ama ^ ; s ... (+14 more)` | 24 |
+| 8k | `▁malitengay ( 老 人 家 ) ▁romadiw ▁ci ▁malitengay . ... (+10 more)` | 20 |
+| 16k | `▁malitengay ( 老人家 ) ▁romadiw ▁ci ▁malitengay . ▁( 老人家 ... (+6 more)` | 16 |
+| 32k | `▁malitengay ( 老人家 ) ▁romadiw ▁ci ▁malitengay . ▁( 老人家在 ... (+5 more)` | 15 |
+| 64k | `▁malitengay ( 老人家 ) ▁romadiw ▁ci ▁malitengay . ▁( 老人家在 ... (+5 more)` | 15 |
 
-**Sample 3:** `taylin o pacaliwen no paylang a caciyaw, o Amilika no sowal i, plice mahaenay！`
+**Sample 3:** `Sokoy 木鱉果 縮圖|sokoy Caay to ka'aloman ko mipaloma'ay to matiniay a sokay, carekah...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁tay lin ▁o ▁pac aliw en ▁no ▁pay lang ▁a ... (+12 more)` | 22 |
-| 16k | `▁tay lin ▁o ▁pacaliw en ▁no ▁pay lang ▁a ▁caciyaw ... (+11 more)` | 21 |
-| 32k | `▁tay lin ▁o ▁pacaliwen ▁no ▁paylang ▁a ▁caciyaw , ▁o ... (+9 more)` | 19 |
-| 64k | `▁taylin ▁o ▁pacaliwen ▁no ▁paylang ▁a ▁caciyaw , ▁o ▁amilika ... (+8 more)` | 18 |
+| 8k | `▁so koy ▁ 木 鱉 果 ▁縮圖 | so koy ... (+21 more)` | 31 |
+| 16k | `▁sokoy ▁ 木 鱉 果 ▁縮圖 | so koy ▁caay ... (+19 more)` | 29 |
+| 32k | `▁sokoy ▁木 鱉 果 ▁縮圖 | so koy ▁caay ▁to ... (+17 more)` | 27 |
+| 64k | `▁sokoy ▁木 鱉 果 ▁縮圖 | sokoy ▁caay ▁to ▁ka ... (+16 more)` | 26 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 3.626x compression
-- **Lowest UNK Rate:** 8k with 0.1304% unknown tokens
+- **Best Compression:** 64k achieves 3.608x compression
+- **Lowest UNK Rate:** 8k with 0.4493% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -126,57 +129,89 @@ Kasasiwasiw:Siwkulang 'Amis`
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 6,797 🏆 | 12.73 | 30,025 | 21.0% | 48.9% |
-| **2-gram** | 185 🏆 | 7.54 | 8,598 | 77.5% | 97.7% |
-| **3-gram** | 17,575 | 14.10 | 58,959 | 14.4% | 34.9% |
-| **3-gram** | 982 | 9.94 | 32,049 | 42.6% | 80.9% |
-| **4-gram** | 43,585 | 15.41 | 125,762 | 12.4% | 26.1% |
-| **4-gram** | 3,737 | 11.87 | 117,566 | 27.2% | 56.9% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 6,678 | 12.71 | 22,550 | 20.3% | 47.3% |
+| **2-gram** | Subword | 207 🏆 | 7.69 | 6,731 | 78.5% | 98.2% |
+| **3-gram** | Word | 12,757 | 13.64 | 35,948 | 17.2% | 36.4% |
+| **3-gram** | Subword | 1,373 | 10.42 | 25,440 | 36.9% | 81.6% |
+| **4-gram** | Word | 30,756 | 14.91 | 77,159 | 15.4% | 26.9% |
+| **4-gram** | Subword | 6,401 | 12.64 | 95,881 | 18.2% | 53.7% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ira ko` | 5,064 |
+| 2 | `romi ad` | 4,019 |
+| 3 | `i miheca` | 2,827 |
+| 4 | `a tamdaw` | 2,806 |
+| 5 | `a sowal` | 2,768 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ka aloman no` | 2,123 |
+| 2 | `a romi ad` | 1,671 |
+| 3 | `ko tamdaw o` | 1,565 |
+| 4 | `sa osi no` | 1,535 |
+| 5 | `ko ka aloman` | 1,534 |
+
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `, o` | 10,654 |
-| 2 | `i ,` | 9,935 |
-| 3 | `. o` | 6,491 |
-| 4 | `ira ko` | 5,079 |
-| 5 | `’ ad` | 4,361 |
+| 1 | `ko sa osi no` | 1,482 |
+| 2 | `ko ka aloman no` | 1,395 |
+| 3 | `nina angan tilid i` | 853 |
+| 4 | `nano nina angan tilid` | 845 |
+| 5 | `o roma sato i` | 766 |
 
-**3-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `romi ’ ad` | 4,026 |
-| 2 | `ka ’ aloman` | 2,293 |
-| 3 | `’ aloman no` | 2,134 |
-| 4 | `] . （` | 2,065 |
-| 5 | `sa ’ osi` | 1,873 |
+| 1 | `o _` | 200,857 |
+| 2 | `a _` | 143,109 |
+| 3 | `a n` | 139,584 |
+| 4 | `_ k` | 106,296 |
+| 5 | `a y` | 96,390 |
 
-**4-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ka ’ aloman no` | 2,121 |
-| 2 | `a romi ’ ad` | 1,630 |
-| 3 | `ko ka ’ aloman` | 1,534 |
-| 4 | `sa ’ osi no` | 1,530 |
-| 5 | `ko sa ’ osi` | 1,509 |
+| 1 | `a y _` | 60,395 |
+| 2 | `_ a _` | 58,815 |
+| 3 | `a n _` | 54,544 |
+| 4 | `n o _` | 54,458 |
+| 5 | `t o _` | 53,668 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ n o _` | 47,644 |
+| 2 | `_ k o _` | 44,141 |
+| 3 | `_ t o _` | 37,131 |
+| 4 | `o _ k a` | 18,566 |
+| 5 | `a y _ a` | 15,366 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 185
+- **Best Perplexity:** 2-gram (subword) with 207
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~57% of corpus
+- **Coverage:** Top-1000 patterns cover ~54% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -184,55 +219,86 @@ Kasasiwasiw:Siwkulang 'Amis`
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.5042 | 1.418 | 4.24 | 80,205 | 49.6% |
-| **1** | 1.6851 | 3.216 | 12.15 | 4,318 | 0.0% |
-| **2** | 0.3377 | 1.264 | 2.05 | 339,828 | 66.2% |
-| **2** | 0.4256 | 1.343 | 2.40 | 52,454 | 57.4% |
-| **3** | 0.1611 | 1.118 | 1.34 | 695,243 | 83.9% |
-| **3** | 0.3826 | 1.304 | 2.20 | 125,822 | 61.7% |
-| **4** | 0.0722 🏆 | 1.051 | 1.13 | 934,421 | 92.8% |
-| **4** | 0.3723 🏆 | 1.294 | 1.92 | 276,161 | 62.8% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.6170 | 1.534 | 4.54 | 72,606 | 38.3% |
+| **1** | Subword | 1.5133 | 2.855 | 9.96 | 4,060 | 0.0% |
+| **2** | Word | 0.3021 | 1.233 | 1.87 | 329,508 | 69.8% |
+| **2** | Subword | 0.4126 | 1.331 | 2.39 | 40,428 | 58.7% |
+| **3** | Word | 0.1213 | 1.088 | 1.23 | 614,195 | 87.9% |
+| **3** | Subword | 0.3832 | 1.304 | 2.24 | 96,490 | 61.7% |
+| **4** | Word | 0.0415 🏆 | 1.029 | 1.07 | 756,623 | 95.8% |
+| **4** | Subword | 0.3927 | 1.313 | 2.01 | 215,619 | 60.7% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `, 75 % , kimolmolay dadingo sinpon i sra apong , nikawrira , - tinsikiw ,`
-2. `’ atomo , senpitopaw si misaakoako misatapang ko 1 , likakawa haw i singko saadihay sato`
-3. `a sofitay no harana ’ asay amipa ’ iked misingkiwan tamdaw mangalefay ko pakayraan ko roma`
+1. `a niyaro ira itiya mitahidang ko nani sera ira ko tamdaw no i likisi kingkiwso a`
+2. `no kalingko posong kowan 395 satoko cilafas to sapitoripes paysin hananay a tayni i lalan matengil`
+3. `ko sowal 波札那共和國 o no switzerland 瑞士 anini a tapolo malowid no sici misatapang romakat cira`
 
 **Context Size 2:**
 
-1. `, o congli tapang no naci - toic to sapifaolawaw to yotaya tamdaw mikapot to amilika sifo`
-2. `i , pakawas . onini ko sakasaan no tiawcaci konini a “ satefoc 100 liyad pisalofan i`
-3. `. o so ’ elinay mafalic , halo tamdaw sato cangra a miharateng to nga ’ ay`
+1. `ira ko piawniya taerniya maciton ato seroys etal a cakoma tamdaw chakma o sangco fociyaw theravāda k...`
+2. `romi ad pi arawan a patefoc ano ca i kiwkay ato mimokongay foksi ci cang congmin zhang`
+3. `i miheca saka 4 folad 22 romi ad no papotalay a kakafit list of current heads of`
 
 **Context Size 3:**
 
-1. `romi ’ ad nai inkiris misiiked . o iraq mihayda to nai “ 1913 mihecaan a “ misatatad`
-2. `ka ’ aloman no yincomin ( 原住民 ) , polong han i , 274 ko tamdaw . o`
-3. `’ aloman no tamdaw no kasafinacadan ( 族群 ) i , ko bunun ( 布農族 ) 1 %`
+1. `ka aloman no tamdaw no kasafinacadan i ko ira ko picodadan 台東專科 原住民族部落大學 空中大學 i niyaro ira ko`
+2. `a romi ad pawsa sato kiya wina niya wawa a pasowal jiya wina ningra ya saan ya wina`
+3. `ko tamdaw o roma sato i 31 ko tamdaw o pasinto no ka aloman no roma a finacadan`
 
 **Context Size 4:**
 
-1. `ka ’ aloman no roma a finacadan , polong han i , 71 ko tamdaw . o pa -`
-2. `a romi ’ ad . no papotalay a kakafit list of current heads of state and government kasasiwasiw :`
-3. `ko ka ’ aloman no yincomin ( 原住民 ) , polong han i , 838 ko tamdaw . o`
+1. `ko sa osi no parod no loma 921 ko sa osi no tamdaw 98 ko ka aloman no roma`
+2. `ko ka aloman no yincomin polong han i 97 ko tamdaw o roma sato i 9 ko ka aloman`
+3. `nina angan tilid i 18 南アフリカ共和国 日本外務省 nano nina angan tilid pdf i 24 7 government of ireland article`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `ayayah_n_n_ka._k`
+2. `_no_(池上田部—“f_nc_`
+3. `omicecakoli_para`
+
+**Context Size 2:**
+
+1. `o_lay_tan_a_kitay`
+2. `a_ko_cininay_to_a`
+3. `analay_tok_atoker`
+
+**Context Size 3:**
+
+1. `ay_a_honti”_ni_kit`
+2. `_a_roman_no_maka,_`
+3. `an_of_stas_no_paka`
+
+**Context Size 4:**
+
+1. `_no_opi_lilay._sa’o`
+2. `_ko_pikinko-’aloma’`
+3. `_to_tasiya_finaca_a`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 92.8% predictability
+- **Best Predictability:** Context-4 (word) with 95.8% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (276,161 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (215,619 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -248,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 31,948 |
-| Total Tokens | 962,770 |
-| Mean Frequency | 30.14 |
+| Vocabulary Size | 29,996 |
+| Total Tokens | 911,467 |
+| Mean Frequency | 30.39 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 634.99 |
+| Frequency Std Dev | 650.13 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | a | 59,912 |
-| 2 | no | 48,183 |
-| 3 | ko | 44,638 |
-| 4 | to | 40,008 |
-| 5 | i | 38,103 |
-| 6 | o | 30,368 |
-| 7 | ato | 10,842 |
-| 8 | tamdaw | 10,833 |
-| 9 | miheca | 6,862 |
-| 10 | sa | 6,789 |
+| 1 | a | 59,636 |
+| 2 | no | 47,923 |
+| 3 | ko | 44,308 |
+| 4 | to | 39,595 |
+| 5 | i | 37,830 |
+| 6 | o | 30,176 |
+| 7 | ato | 10,792 |
+| 8 | tamdaw | 10,688 |
+| 9 | miheca | 6,765 |
+| 10 | sa | 6,716 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | hahihay | 2 |
-| 2 | hiay | 2 |
-| 3 | pasitenokay | 2 |
-| 4 | satsuma | 2 |
-| 5 | pisamawmaw | 2 |
-| 6 | saigo | 2 |
-| 7 | tsumoru | 2 |
-| 8 | vetoma | 2 |
-| 9 | mitingting | 2 |
-| 10 | kalosaasik | 2 |
+| 1 | paiyo | 2 |
+| 2 | parangalan | 2 |
+| 3 | 對豐年祭的一些看法 | 2 |
+| 4 | kalikowatan | 2 |
+| 5 | pisifat | 2 |
+| 6 | suise | 2 |
+| 7 | pililafangan | 2 |
+| 8 | sapikomod | 2 |
+| 9 | piselong | 2 |
+| 10 | ekelay | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.1668 |
-| R² (Goodness of Fit) | 0.995322 |
+| Zipf Coefficient | 1.1663 |
+| R² (Goodness of Fit) | 0.995345 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 51.0% |
-| Top 1,000 | 75.7% |
-| Top 5,000 | 89.3% |
-| Top 10,000 | 93.7% |
+| Top 100 | 52.9% |
+| Top 1,000 | 76.5% |
+| Top 5,000 | 89.8% |
+| Top 10,000 | 94.1% |
 
 ### Key Findings
 
 - **Zipf Compliance:** R²=0.9953 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 51.0% of corpus
-- **Long Tail:** 21,948 words needed for remaining 6.3% coverage
+- **High Frequency Dominance:** Top 100 words cover 52.9% of corpus
+- **Long Tail:** 19,996 words needed for remaining 5.9% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -318,24 +384,132 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 12,970 | 32 | 3.455 | 0.855 | 0.8477 🏆 |
-| **mono_64d** | 12,970 | 64 | 3.941 | 0.764 | 0.8135 |
-| **mono_128d** | 12,970 | 128 | 4.314 | 0.719 | 0.5720 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8374 🏆 | 0.3299 | N/A | N/A |
+| **mono_64d** | 64 | 0.7849 | 0.2563 | N/A | N/A |
+| **mono_128d** | 128 | 0.4896 | 0.2197 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8477 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 12,970 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8374 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2686. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ma` | mapateli, matoasto, macekelay |
+| `-mi` | milika, misahiraterateng, milafo |
+| `-ka` | kariponan, kaorira, kalapaliw |
+| `-pa` | pafatisay, paherekan, palecapu |
+| `-sa` | safaniyotan, sarosaros, sakiikoray |
+| `-pi` | piaw, pidemak, pirayray |
+| `-ta` | tahaf, tanetekay, tadamarorayay |
+| `-mal` | malamisiieday, malikiday, malatoloay |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-n` | balkan, iskawalian, otoman |
+| `-y` | elay, qehuy, macekelay |
+| `-ay` | elay, macekelay, tanetekay |
+| `-an` | balkan, iskawalian, otoman |
+| `-ng` | jinfeng, kopitahidang, arawang |
+| `-en` | haratengen, adihayen, tatayalen |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `emak` | 2.36x | 36 contexts | demak, hemak, ademak |
+| `alom` | 1.99x | 51 contexts | alomi, aloma, paloma |
+| `ilid` | 2.22x | 32 contexts | tilid, atilid, pitilid |
+| `dema` | 2.18x | 33 contexts | demak, ademak, odemak |
+| `olon` | 1.98x | 47 contexts | kolon, tolon, polon |
+| `iren` | 2.34x | 25 contexts | ireng, yiren, sairen |
+| `ihec` | 2.13x | 28 contexts | niheca, kiheca, ciheci |
+| `taki` | 2.23x | 15 contexts | takid, takimi, kitaki |
+| `ngra` | 2.05x | 19 contexts | ingra, angra, cngra |
+| `onga` | 1.49x | 55 contexts | fonga, ongay, tonga |
+| `mihe` | 2.10x | 14 contexts | mihea, miheca, mihemek |
+| `itak` | 1.81x | 22 contexts | kitakt, mitaka, kitaki |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-ma` | `-y` | 239 words | masamaanay, malekoay |
+| `-ma` | `-ay` | 238 words | masamaanay, malekoay |
+| `-ka` | `-n` | 174 words | kalamkamen, kasopedan |
+| `-mi` | `-y` | 173 words | misamoraday, mipelengay |
+| `-mi` | `-ay` | 169 words | misamoraday, mipelengay |
+| `-ka` | `-an` | 154 words | kasopedan, kacitiyadan |
+| `-pa` | `-n` | 122 words | paecasan, pahapingan |
+| `-pi` | `-n` | 122 words | pitokadan, pitengilan |
+| `-pi` | `-an` | 117 words | pitokadan, pitengilan |
+| `-pa` | `-y` | 81 words | papaysoay, pakaenay |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| masataporoay | **`ma-sa-ta-poro-ay`** | 9.0 | `poro` |
+| pipafilongan | **`pi-pa-filo-ng-an`** | 9.0 | `filo` |
+| masamaamaanay | **`ma-sa-ma-amaan-ay`** | 9.0 | `amaan` |
+| papisatoronen | **`pa-pi-sa-toron-en`** | 9.0 | `toron` |
+| milinganganay | **`mi-linga-ng-an-ay`** | 9.0 | `linga` |
+| mikapolongan | **`mi-ka-polo-ng-an`** | 9.0 | `polo` |
+| kasakakitaan | **`ka-sa-ka-kita-an`** | 9.0 | `kita` |
+| mapanganganay | **`ma-pa-ngang-an-ay`** | 9.0 | `ngang` |
+| talolongay | **`ta-lolo-ng-ay`** | 7.5 | `lolo` |
+| mipalawacoay | **`mi-pa-lawaco-ay`** | 7.5 | `lawaco` |
+| sakalaloodan | **`sa-ka-lalood-an`** | 7.5 | `lalood` |
+| masakapahay | **`ma-sa-ka-pahay`** | 7.5 | `pahay` |
+| pisaomahan | **`pi-sa-omah-an`** | 7.5 | `omah` |
+| pakapatayay | **`pa-ka-pa-tayay`** | 7.5 | `tayay` |
+| mamipadoedo | **`ma-mi-pa-doedo`** | 7.5 | `doedo` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language AMI appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -343,11 +517,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (3.63x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (185) |
-| Markov | **Context-4** | Highest predictability (92.8%) |
+| Tokenizer | **64k BPE** | Best compression (3.61x) |
+| N-gram | **2-gram** | Lowest perplexity (207) |
+| Markov | **Context-4** | Highest predictability (95.8%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -537,7 +712,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -553,7 +729,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 05:44:49*
+*Report Date: 2026-01-03 05:06:08*

models/embeddings/monolingual/ami_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:996aff94cf3bb9c728be627b571a1edeead0f09cda7df9bb07117bf42ce1e8f6
-size 1037511856
+oid sha256:15905bdf3ef2a2c33080d911e43499747f02a98b908fe2fd8f60829e51b06e3b
+size 1036991498

models/embeddings/monolingual/ami_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 12970
+  "vocab_size": 12471
 }

models/embeddings/monolingual/ami_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5d60b5f427ae174c4924ceeb6752f7bab8acecf6c833dd0975f135d607bb103a
-size 259550896
+oid sha256:1637a213277140817cb0abdc86b03667570ab2c62b90fadaab2d4f67dd7d34c1
+size 259413770

models/embeddings/monolingual/ami_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 12970
+  "vocab_size": 12471
 }

models/embeddings/monolingual/ami_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f6d0645c29630c896eba9768226cbbdb875b22ef317a5545e1a401c3e2661be0
-size 518871216
+oid sha256:d86b494753732d38bf3c59ecf55e84334333d1b828be2bb495f8f20346d41fe7
+size 518606346

models/embeddings/monolingual/ami_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 12970
+  "vocab_size": 12471
 }

models/subword_markov/ami_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:71450850ae7c3385727c3b3747d4c83449ea0f95fc516e229eaa814f5c4fcccd
-size 294376
+oid sha256:bd6c5414c5d99f255c9a24fd799f5715c61d98151829429da1c891c950c71b6c
+size 237122

models/subword_markov/ami_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "ami",
-  "unique_contexts": 4318,
-  "total_transitions": 6575362
+  "unique_contexts": 4060,
+  "total_transitions": 5416535
 }

models/subword_markov/ami_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ee631c071b1253baacaf7694f59e48c9b7771120278456a30f46ba270682f4a7
-size 1143059
+oid sha256:c95bebddb17ca6bb55229dc5f814136974b0cdf0bec76337ee41e0101f6d951d
+size 896649

models/subword_markov/ami_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "ami",
-  "unique_contexts": 52454,
-  "total_transitions": 6573527
+  "unique_contexts": 40428,
+  "total_transitions": 5414744
 }

models/subword_markov/ami_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:db3b0de53581d8bf1696e13a9088efb66d3b9928df878c5ad2b30d560b51efb9
-size 2579415
+oid sha256:3bdee46d76e034d877f43a9063eb7dced0030fd717bea264027433f451c9d30b
+size 2005109

models/subword_markov/ami_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "ami",
-  "unique_contexts": 125822,
-  "total_transitions": 6571692
+  "unique_contexts": 96490,
+  "total_transitions": 5412953
 }

models/subword_markov/ami_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7c988764c01283eb4b190b72c06246b2298f1ef22bf8c14d81f60b7b568a0393
-size 5188374
+oid sha256:5583564cfdde5d8d793971d0b83f3491b51f34500300b4550db8c3f95b407fdc
+size 4098231

models/subword_markov/ami_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "ami",
-  "unique_contexts": 276161,
-  "total_transitions": 6569857
+  "unique_contexts": 215619,
+  "total_transitions": 5411162
 }

models/subword_ngram/ami_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:869792fa908529a263bfe81ecdcfe359b7706221509eaab21f94c7873a71c428
-size 113262
+oid sha256:e1026db4cb824d7fb2d17207e6726e192995fccc42121a6deb342c7b4330714a
+size 88989

models/subword_ngram/ami_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "ami",
-  "unique_ngrams": 8598,
-  "total_ngrams": 6575362
+  "unique_ngrams": 6731,
+  "total_ngrams": 5416535
 }

models/subword_ngram/ami_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3a3549ed31695504b9278dc147e5534a52e6d5fc06f9304ad730ba717a3f50a3
-size 422836
+oid sha256:75fd09f81b66ca76353d7986bbb6cbcf010ea89e9f05da2205a5da78535f5399
+size 336416

models/subword_ngram/ami_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "ami",
-  "unique_ngrams": 32049,
-  "total_ngrams": 6573527
+  "unique_ngrams": 25440,
+  "total_ngrams": 5414744
 }

models/subword_ngram/ami_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:40f67c05d8d68f78e02e07923b387ce8896954839d12c21b3f122450c3feb7b0
-size 1419899
+oid sha256:2dba14c53ee7a60b255aeac3e25837701b86d507511cedb46513b241b52709d2
+size 1158460

models/subword_ngram/ami_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "ami",
-  "unique_ngrams": 117566,
-  "total_ngrams": 6571692
+  "unique_ngrams": 95881,
+  "total_ngrams": 5412953
 }

models/tokenizer/ami_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:92cc2985510da9d8ea3f08c9c2fe0987e1317dd39fdb442e977d73465247ddc8
-size 497352
+oid sha256:717608566d5035e11e0b6a237c9dc8de306a618502b3c9c2d163cff3d89a1fd3
+size 504074

models/tokenizer/ami_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:84774ac22437cd71ddffff17b52c20adc0fc83af168f378c75ec102a479a4062
-size 772651
+oid sha256:65580b9ce13917e013a326ec049db42b294a2d5b301ede6cece658a66bf01a32
+size 812779

models/tokenizer/ami_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cf4b9eb22cd684b3c74f359de1d2da1c756857c17d5ae812e601cbcd70722174
-size 1376238
+oid sha256:cc91af1f68cf46d19a5cf0946b2630e7d893f6743a951a7b586d0bab9b304a41
+size 1350707

models/tokenizer/ami_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:62a8450b8350a9a3296893cca86f7be81850b877d639aa20e246afe3827fce8f
-size 363501
+oid sha256:299585e3fae7c7c16e32939fafacf44061bf52adbb735a7cc3118845a7421988
+size 367877

models/vocabulary/ami_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:56d1101428b77726e336991569026e53477c19502cf4df6ee4b3ef0446ea2fd6
-size 556457
+oid sha256:5b2554bb753aa34783bd74f1d3bc80f61a566f9e2273bab9bfdd7d5526d3aa32
+size 519354

models/vocabulary/ami_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "ami",
-  "vocabulary_size": 31948,
+  "vocabulary_size": 29996,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.07917903648473924,
+    "type_token_ratio": 0.07616371567334841,
     "coverage": {
-      "top_100": 0.4856909110154611,
-      "top_1000": 0.7209151406573209,
-      "top_5000": 0.8507747355966844,
-      "top_10000": 0.8927231340411788
+      "top_100": 0.5051533209941497,
+      "top_1000": 0.7309960152681676,
+      "top_5000": 0.8578964137413508,
+      "top_10000": 0.8988609661874228
     },
-    "hapax_count": 48091,
-    "hapax_ratio": 0.6008445882632216,
-    "total_documents": 1835
+    "hapax_count": 42675,
+    "hapax_ratio": 0.5872356235637325,
+    "total_documents": 1791
   }
 }

models/word_markov/ami_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:497e8c15f7f0df3064b06bc6079d7fe2457f423038e4205aa304880db1c508cd
-size 3074954
+oid sha256:82994f99e1f273b3e05d36bc2663a6668f77084bae001566586edc6fa521ceac
+size 2810453

models/word_markov/ami_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "ami",
-  "unique_contexts": 80205,
-  "total_transitions": 1354574
+  "unique_contexts": 72606,
+  "total_transitions": 952351
 }

models/word_markov/ami_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bf92aafa3f3d2a40c2d9ca33400c1d42bd30f230689e2bd061dff62c9a9b7c3c
-size 7158620
+oid sha256:8c3d060cf7c94ddcc5b80137ae2381c825554392a922b3f095d2bc6335335c03
+size 6568432

models/word_markov/ami_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "ami",
-  "unique_contexts": 339828,
-  "total_transitions": 1352740
+  "unique_contexts": 329508,
+  "total_transitions": 950560
 }

models/word_markov/ami_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:692511fad35e939d0e112742c632a842165e2fe7c3b2a8b5216f6a86f7f9dc7c
-size 11871433
+oid sha256:7c36780ba30648979c6f36aa779a8ceda3c56142eba2cb087811e6d38efb2225
+size 10206751

models/word_markov/ami_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "ami",
-  "unique_contexts": 695243,
-  "total_transitions": 1350906
+  "unique_contexts": 614195,
+  "total_transitions": 948769
 }

models/word_markov/ami_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:caa72e38f0bd63ad61b98e3fa42473f895753d6215e73dc0853470cde5ad829d
-size 15253116
+oid sha256:649cf9d20bbed0ad2be2491e00a9bbbf30767d7fa1d44c83bfa1d10c47e620e5
+size 12409860

models/word_markov/ami_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "ami",
-  "unique_contexts": 934421,
-  "total_transitions": 1349074
+  "unique_contexts": 756623,
+  "total_transitions": 946978
 }

models/word_ngram/ami_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a6f3d72babd69a9005eb7c3f9ad5ca842fbc6123aa093f970b7cd777994d73f3
-size 402572
+oid sha256:34a3cb7ef204727a0d03453df07384a8f62a18d7f3dba653e2b66ce2ca76ea57
+size 304726

models/word_ngram/ami_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "ami",
-  "unique_ngrams": 30025,
-  "total_ngrams": 1354574
+  "unique_ngrams": 22550,
+  "total_ngrams": 952351
 }

models/word_ngram/ami_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cf66cc0eda4cc22ffc8385a31de3e3e3052521973994d2e6f28427203ab4a9f1
-size 835271
+oid sha256:c0d4d3ce7e45b0c943854efac16203bc64aa247604e0b004c3f885c1d84b995a
+size 531269

models/word_ngram/ami_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "ami",
-  "unique_ngrams": 58959,
-  "total_ngrams": 1352740
+  "unique_ngrams": 35948,
+  "total_ngrams": 950560
 }

models/word_ngram/ami_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:60863da19abbe4a1ebb1e300b0413297c9f13f369d4db484bbb83d656705dac2
-size 1759231
+oid sha256:95d6030bf37507ad465d99f9e8f7515ad90e1dbd38c9548f4093150077b6b398
+size 1107511

models/word_ngram/ami_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "ami",
-  "unique_ngrams": 125762,
-  "total_ngrams": 1350906
+  "unique_ngrams": 77159,
+  "total_ngrams": 948769
 }