Upload all models and assets for btm (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.785
+    value: 5.210
   - name: best_isotropy
     type: isotropy
-    value: 0.4253
+    value: 0.3926
   - name: vocabulary_size
     type: vocab
-    value: 12175
-generated: 2025-12-28
+    value: 0
+generated: 2026-01-03
 ---
 
 # BTM - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,51 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.846x | 3.82 | 0.0837% | 246,208 |
-| **16k** | 4.225x | 4.20 | 0.0919% | 224,097 |
-| **32k** | 4.544x | 4.52 | 0.0989% | 208,361 |
-| **64k** | 4.785x 🏆 | 4.76 | 0.1041% | 197,875 |
+| **8k** | 4.162x | 4.17 | 0.0869% | 217,411 |
+| **16k** | 4.607x | 4.61 | 0.0962% | 196,367 |
+| **32k** | 5.005x | 5.01 | 0.1045% | 180,776 |
+| **64k** | 5.210x 🏆 | 5.22 | 0.1088% | 173,672 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `23 Januari ima ari pa-23 i kalender Gregorian dohot 361 ari (sanga 362 ari i tao...`
+**Sample 1:** `Natal ima sada kecamatan di Kabupaten Mandailing Natal, Sumatera Utara, Indonesi...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ 2 3 ▁januari ▁ima ▁ari ▁pa - 2 3 ... (+24 more)` | 34 |
-| 16k | `▁ 2 3 ▁januari ▁ima ▁ari ▁pa - 2 3 ... (+24 more)` | 34 |
-| 32k | `▁ 2 3 ▁januari ▁ima ▁ari ▁pa - 2 3 ... (+24 more)` | 34 |
-| 64k | `▁ 2 3 ▁januari ▁ima ▁ari ▁pa - 2 3 ... (+24 more)` | 34 |
+| 8k | `▁natal ▁ima ▁sada ▁kecamatan ▁di ▁kabupaten ▁mandailing ▁natal , ▁sumatera ... (+4 more)` | 14 |
+| 16k | `▁natal ▁ima ▁sada ▁kecamatan ▁di ▁kabupaten ▁mandailing ▁natal , ▁sumatera ... (+4 more)` | 14 |
+| 32k | `▁natal ▁ima ▁sada ▁kecamatan ▁di ▁kabupaten ▁mandailing ▁natal , ▁sumatera ... (+4 more)` | 14 |
+| 64k | `▁natal ▁ima ▁sada ▁kecamatan ▁di ▁kabupaten ▁mandailing ▁natal , ▁sumatera ... (+4 more)` | 14 |
 
-**Sample 2:** `Gunung Tua Ms ima salah sada huta na adong i kecamatan Kotanopan, kabupaten Mand...`
+**Sample 2:** `Luak Kakuasoan ima luak karejo perangkat pamarentah pusat na mandalankon karejo ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁gunung ▁tua ▁ms ▁ima ▁salah ▁sada ▁huta ▁na ▁adong ▁i ... (+13 more)` | 23 |
-| 16k | `▁gunung ▁tua ▁ms ▁ima ▁salah ▁sada ▁huta ▁na ▁adong ▁i ... (+13 more)` | 23 |
-| 32k | `▁gunung ▁tua ▁ms ▁ima ▁salah ▁sada ▁huta ▁na ▁adong ▁i ... (+13 more)` | 23 |
-| 64k | `▁gunung ▁tua ▁ms ▁ima ▁salah ▁sada ▁huta ▁na ▁adong ▁i ... (+13 more)` | 23 |
+| 8k | `▁luak ▁kakuasoan ▁ima ▁luak ▁karejo ▁perangkat ▁pamarentah ▁pusat ▁na ▁mandalankon ... (+9 more)` | 19 |
+| 16k | `▁luak ▁kakuasoan ▁ima ▁luak ▁karejo ▁perangkat ▁pamarentah ▁pusat ▁na ▁mandalankon ... (+9 more)` | 19 |
+| 32k | `▁luak ▁kakuasoan ▁ima ▁luak ▁karejo ▁perangkat ▁pamarentah ▁pusat ▁na ▁mandalankon ... (+9 more)` | 19 |
+| 64k | `▁luak ▁kakuasoan ▁ima ▁luak ▁karejo ▁perangkat ▁pamarentah ▁pusat ▁na ▁mandalankon ... (+9 more)` | 19 |
 
-**Sample 3:** `Mangalap boru ima sada karejo na ibaen i Mandailing Natal, karejo on pema jadi s...`
+**Sample 3:** `17 Juni' ima ari pa-169 (ari pa-170 i taon kabisat) i kalender Gregorian.`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁mangalap ▁boru ▁ima ▁sada ▁karejo ▁na ▁ibaen ▁i ▁mandailing ▁natal ... (+13 more)` | 23 |
-| 16k | `▁mangalap ▁boru ▁ima ▁sada ▁karejo ▁na ▁ibaen ▁i ▁mandailing ▁natal ... (+13 more)` | 23 |
-| 32k | `▁mangalap ▁boru ▁ima ▁sada ▁karejo ▁na ▁ibaen ▁i ▁mandailing ▁natal ... (+12 more)` | 22 |
-| 64k | `▁mangalap ▁boru ▁ima ▁sada ▁karejo ▁na ▁ibaen ▁i ▁mandailing ▁natal ... (+12 more)` | 22 |
+| 8k | `▁ 1 7 ▁juni ' ▁ima ▁ari ▁pa - 1 ... (+17 more)` | 27 |
+| 16k | `▁ 1 7 ▁juni ' ▁ima ▁ari ▁pa - 1 ... (+17 more)` | 27 |
+| 32k | `▁ 1 7 ▁juni ' ▁ima ▁ari ▁pa - 1 ... (+17 more)` | 27 |
+| 64k | `▁ 1 7 ▁juni ' ▁ima ▁ari ▁pa - 1 ... (+17 more)` | 27 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.785x compression
-- **Lowest UNK Rate:** 8k with 0.0837% unknown tokens
+- **Best Compression:** 64k achieves 5.210x compression
+- **Lowest UNK Rate:** 8k with 0.0869% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -121,57 +129,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 2,793 🏆 | 11.45 | 5,965 | 25.6% | 57.6% |
-| **2-gram** | 237 🏆 | 7.89 | 1,918 | 71.1% | 99.2% |
-| **3-gram** | 2,655 | 11.37 | 5,700 | 27.3% | 55.1% |
-| **3-gram** | 1,865 | 10.87 | 12,988 | 30.0% | 75.1% |
-| **4-gram** | 3,977 | 11.96 | 8,130 | 23.7% | 46.0% |
-| **4-gram** | 8,932 | 13.12 | 49,385 | 14.7% | 44.5% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 2,126 | 11.05 | 3,791 | 25.0% | 62.5% |
+| **2-gram** | Subword | 193 🏆 | 7.60 | 1,424 | 75.4% | 99.7% |
+| **3-gram** | Word | 1,572 | 10.62 | 2,726 | 28.6% | 65.6% |
+| **3-gram** | Subword | 1,481 | 10.53 | 9,264 | 32.5% | 79.4% |
+| **4-gram** | Word | 1,863 | 10.86 | 3,343 | 28.5% | 56.4% |
+| **4-gram** | Subword | 7,317 | 12.84 | 38,756 | 16.0% | 47.2% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ima sada` | 613 |
+| 2 | `on pe` | 485 |
+| 3 | `na adong` | 408 |
+| 4 | `sian on` | 359 |
+| 5 | `i taon` | 350 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `na adong i` | 259 |
+| 2 | `kabupaten mandailing natal` | 176 |
+| 3 | `i kalender gregorian` | 169 |
+| 4 | `ima ari pa` | 156 |
+| 5 | `sumatera utara indonesia` | 156 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `provinsi sumatera utara indonesia` | 130 |
+| 2 | `kabupaten mandailing natal provinsi` | 127 |
+| 3 | `natal provinsi sumatera utara` | 126 |
+| 4 | `mandailing natal provinsi sumatera` | 126 |
+| 5 | `taon kabisat i kalender` | 125 |
+
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `kategori :` | 781 |
-| 2 | `/ /` | 707 |
-| 3 | `: /` | 703 |
-| 4 | `ima sada` | 627 |
-| 5 | `, dot` | 599 |
+| 1 | `a n` | 41,122 |
+| 2 | `a _` | 36,766 |
+| 3 | `n _` | 28,003 |
+| 4 | `m a` | 25,432 |
+| 5 | `i _` | 24,703 |
 
-**3-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `: / /` | 702 |
-| 2 | `https : /` | 461 |
-| 3 | `. https :` | 384 |
-| 4 | `sumberna kategori :` | 348 |
-| 5 | `ari pa -` | 308 |
+| 1 | `_ m a` | 15,316 |
+| 2 | `a n _` | 13,300 |
+| 3 | `a n g` | 11,520 |
+| 4 | `_ n a` | 11,505 |
+| 5 | `n a _` | 10,547 |
 
-**4-grams:**
+**4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `https : / /` | 461 |
-| 2 | `. https : /` | 384 |
-| 3 | `: / / www` | 221 |
-| 4 | `/ / www .` | 221 |
-| 5 | `ima ari pa -` | 183 |
+| 1 | `_ n a _` | 6,885 |
+| 2 | `_ m a n` | 5,972 |
+| 3 | `a _ m a` | 4,367 |
+| 4 | `i m a _` | 4,073 |
+| 5 | `_ i m a` | 4,072 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 237
+- **Best Perplexity:** 2-gram (subword) with 193
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~45% of corpus
+- **Coverage:** Top-1000 patterns cover ~47% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -179,55 +219,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.7327 | 1.662 | 4.59 | 29,952 | 26.7% |
-| **1** | 1.0526 | 2.074 | 7.13 | 776 | 0.0% |
-| **2** | 0.2488 | 1.188 | 1.51 | 137,080 | 75.1% |
-| **2** | 0.9494 | 1.931 | 5.09 | 5,530 | 5.1% |
-| **3** | 0.0695 | 1.049 | 1.11 | 206,158 | 93.1% |
-| **3** | 0.8132 | 1.757 | 3.45 | 28,161 | 18.7% |
-| **4** | 0.0218 🏆 | 1.015 | 1.03 | 227,434 | 97.8% |
-| **4** | 0.5358 🏆 | 1.450 | 2.24 | 97,224 | 46.4% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.8051 | 1.747 | 4.52 | 26,321 | 19.5% |
+| **1** | Subword | 0.8855 | 1.847 | 5.45 | 845 | 11.4% |
+| **2** | Word | 0.2150 | 1.161 | 1.41 | 118,363 | 78.5% |
+| **2** | Subword | 0.7881 | 1.727 | 4.37 | 4,600 | 21.2% |
+| **3** | Word | 0.0511 | 1.036 | 1.07 | 165,958 | 94.9% |
+| **3** | Subword | 0.7696 | 1.705 | 3.50 | 20,094 | 23.0% |
+| **4** | Word | 0.0119 🏆 | 1.008 | 1.01 | 176,808 | 98.8% |
+| **4** | Subword | 0.5810 | 1.496 | 2.41 | 70,348 | 41.9% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
 
-### Generated Text Samples
+**Context Size 1:**
+
+1. `i etika deskriptif tanpa jejak ni ibana pautan luar angkasa internasional dot filsafat tarbonggal im...`
+2. `na ma tu kadiua pengantin adaboru i kota nagodangna ima kalimat frasa arab ا alif alif`
+3. `ima simfoni dagdanak carito na adong i kamajuan sosial yang ditunjukkan dalam menjelaskan proses pal...`
+
+**Context Size 2:**
+
+1. `ima sada pamikir paling ponting ison ima bagain ni alak etika 24 25 manjadi cabang ni elmu`
+2. `on pe i artion ima panasehat mara boru na tobang tingon saro perancis partongaan dot pangujung abad`
+3. `na adong i harana suden aon bisa di turuti dungi anggon na idalani satiop get manyuan anso`
+
+**Context Size 3:**
+
+1. `na adong i mandailing ima ibagain jolo ni bagas on samuloi on toru sampe tu ginjang i jepang`
+2. `kabupaten mandailing natal sumatera utara indonesia baru koordinat nai ima na adong tingon simatoban...`
+3. `ima ari pa 105 ari pa 106 i taon kabisat i kalender gregorian dohot 361 ari sanga 362`
+
+**Context Size 4:**
 
-Below are text samples generated from each Markov chain model:
+1. `kabupaten mandailing natal provinsi sumatera utara indonesia sumberna`
+2. `natal provinsi sumatera utara indonesia huta on pe adong na ima sacara alami do on inda na ibaen bae...`
+3. `mandailing natal provinsi sumatera utara indonesia i batahan`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. hami sanga kamp parkarejo jerman . biasona aong ibaen anso mambuat sada tim dohot mamake`
-2. `, uhum kodrat - ira satonga tarbuka i provinsi opolskie provinsi i diafragma mambatasi bagian sian`
-3. `i kalender gregorian dohot tema na ilukis nia hagoluan nalai ima . sacara enskapitulasi tingon satio...`
+1. `a_hur:_ig_bumani`
+2. `_i_a_0_a_u_agong`
+3. `numayalleri_dusi`
 
 **Context Size 2:**
 
-1. `kategori : ilmuwan yahudi kategori : politik kategori : saro inggeris : engineering ) ima sada penya...`
-2. `/ / download / 18320 / 13475 baru nung i presepsi na padua bolasna na dua kota`
-3. `: / / www . rusdionoconsulting . com / tag / pemandian - air - terjun -`
+1. `ang_rovskithe_tin`
+2. `a_tikabindot_puna`
+3. `n_nak,_ina_dohorc`
 
 **Context Size 3:**
 
-1. `: / / onepiece . fandom . com / id / tourism / gunung - sorik - marapi`
-2. `https : / / startfmmadina . com / 40 - daftar - kumpulan - terjemahan - bahasa -`
-3. `. https : / / madina . go . id / sejarah - bulu - tangkis / tarsarupo`
+1. `_man_baru_najo._am`
+2. `an_reicht_ditasali`
+3. `ang_i_the_pada_raj`
 
 **Context Size 4:**
 
-1. `https : / / www . pmimedan . or . id / sahabat - nabi / songkoni juo ma`
-2. `. https : / / translate . google . com / translate ? u = https : / /`
-3. `: / / www . harianhaluan . com / news / world - middle - east - 14703476 "`
+1. `_na_mander_gregoria`
+2. `_manurutnia_iangir_`
+3. `a_mang,_31_taon_ima`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 97.8% predictability
+- **Best Predictability:** Context-4 (word) with 98.8% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (97,224 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (70,348 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -243,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 12,175 |
-| Total Tokens | 189,541 |
-| Mean Frequency | 15.57 |
-| Median Frequency | 3 |
-| Frequency Std Dev | 125.93 |
+| Vocabulary Size | 11,024 |
+| Total Tokens | 173,772 |
+| Mean Frequency | 15.76 |
+| Median Frequency | 4 |
+| Frequency Std Dev | 129.04 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | i | 7,254 |
-| 2 | na | 7,126 |
-| 3 | ima | 4,033 |
-| 4 | on | 3,983 |
-| 5 | dohot | 2,990 |
-| 6 | ni | 2,684 |
-| 7 | dot | 2,484 |
-| 8 | sada | 1,835 |
-| 9 | tu | 1,710 |
-| 10 | ma | 1,482 |
+| 1 | i | 7,102 |
+| 2 | na | 6,996 |
+| 3 | ima | 3,950 |
+| 4 | on | 3,907 |
+| 5 | dohot | 2,932 |
+| 6 | ni | 2,627 |
+| 7 | dot | 2,463 |
+| 8 | sada | 1,805 |
+| 9 | tu | 1,679 |
+| 10 | ma | 1,474 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | lil | 2 |
-| 2 | imah | 2 |
-| 3 | nasida | 2 |
-| 4 | sunusi | 2 |
-| 5 | nunga | 2 |
-| 6 | majmu | 2 |
-| 7 | fatawa | 2 |
-| 8 | fiqhi | 2 |
-| 9 | panjalakian | 2 |
-| 10 | martoba | 2 |
+| 1 | harvard | 2 |
+| 2 | syahadat | 2 |
+| 3 | dans | 2 |
+| 4 | philosophie | 2 |
+| 5 | évasion | 2 |
+| 6 | bénézé | 2 |
+| 7 | infini | 2 |
+| 8 | delà | 2 |
+| 9 | telos | 2 |
+| 10 | apganistan | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.0695 |
-| R² (Goodness of Fit) | 0.987847 |
+| Zipf Coefficient | 1.0692 |
+| R² (Goodness of Fit) | 0.988968 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 40.1% |
-| Top 1,000 | 69.3% |
-| Top 5,000 | 90.3% |
-| Top 10,000 | 97.7% |
+| Top 100 | 41.7% |
+| Top 1,000 | 71.1% |
+| Top 5,000 | 91.5% |
+| Top 10,000 | 98.8% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9878 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 40.1% of corpus
-- **Long Tail:** 2,175 words needed for remaining 2.3% coverage
+- **Zipf Compliance:** R²=0.9890 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 41.7% of corpus
+- **Long Tail:** 1,024 words needed for remaining 1.2% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -313,24 +384,133 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 4,643 | 32 | 2.986 | 0.583 | 0.4253 🏆 |
-| **mono_64d** | 4,643 | 64 | 3.026 | 0.583 | 0.1288 |
-| **mono_128d** | 4,643 | 128 | 3.050 | 0.582 | 0.0237 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.3926 🏆 | 0.4276 | N/A | N/A |
+| **mono_64d** | 64 | 0.1169 | 0.4242 | N/A | N/A |
+| **mono_128d** | 128 | 0.0230 | 0.4239 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.4253 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 4,643 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.3926 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.4252. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ma` | marmangan, manjadion, manembak |
+| `-pa` | palo, pasiap, panyalahgunaan |
+| `-man` | manjadion, manembak, mangargai |
+| `-mar` | marmangan, mardikir, markombang |
+| `-sa` | salama, samo, sasabagas |
+| `-ta` | tagalog, tas, targinjang |
+| `-ka` | karang, kadua, kamis |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-n` | asisten, tolongan, proclamation |
+| `-an` | tolongan, panyalahgunaan, marmangan |
+| `-a` | tionghua, natarida, moskwa |
+| `-ng` | karang, gedung, targinjang |
+| `-on` | proclamation, idasorkon, manjadion |
+| `-na` | pascasarjana, nalainna, paduana |
+| `-ang` | karang, targinjang, uwang |
+| `-kon` | idasorkon, ilaporkon, namangobankon |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `anga` | 1.50x | 76 contexts | nanga, angan, sanga |
+| `angk` | 1.52x | 58 contexts | angke, angka, angko |
+| `mang` | 1.68x | 31 contexts | amang, mango, tamang |
+| `anda` | 1.40x | 53 contexts | tanda, banda, ganda |
+| `dang` | 1.48x | 41 contexts | udang, undang, sedang |
+| `amba` | 1.48x | 39 contexts | hamba, tamba, gambar |
+| `aran` | 1.38x | 47 contexts | arani, arang, arana |
+| `ngka` | 1.41x | 39 contexts | angka, dangka, angkat |
+| `ngan` | 1.32x | 43 contexts | angan, tangan, lengan |
+| `anja` | 1.38x | 33 contexts | hanja, banjar, anjadi |
+| `angg` | 1.31x | 39 contexts | anggi, anggo, anggap |
+| `tang` | 1.35x | 29 contexts | utang, otang, tangan |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-pa` | `-n` | 297 words | parsiajaran, paridian |
+| `-pa` | `-an` | 266 words | parsiajaran, paridian |
+| `-ma` | `-n` | 232 words | malainkon, malahirkon |
+| `-ma` | `-on` | 148 words | malainkon, malahirkon |
+| `-ka` | `-n` | 111 words | kabupaten, kahangatan |
+| `-ka` | `-an` | 108 words | kahangatan, kapastian |
+| `-ma` | `-kon` | 96 words | malainkon, malahirkon |
+| `-ma` | `-a` | 96 words | marga, maninggalnaia |
+| `-ma` | `-ng` | 67 words | markombang, margelombang |
+| `-ma` | `-an` | 60 words | marsegaan, masakan |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| markalanjutan | **`mar-ka-lanjut-an`** | 7.5 | `lanjut` |
+| malambangkon | **`ma-lamb-ang-kon`** | 7.5 | `lamb` |
+| kabolakangan | **`ka-bolak-ang-an`** | 7.5 | `bolak` |
+| kamanusiaan | **`ka-man-usia-an`** | 7.5 | `usia` |
+| kakuasoanna | **`ka-kuaso-an-na`** | 7.5 | `kuaso` |
+| markabangsoan | **`mar-ka-bangso-an`** | 7.5 | `bangso` |
+| sakaturunan | **`sa-ka-turun-an`** | 7.5 | `turun` |
+| pamabangan | **`pa-ma-bang-an`** | 7.5 | `bang` |
+| paporangan | **`pa-pora-ng-an`** | 7.5 | `pora` |
+| kaputusan | **`ka-putus-an`** | 6.0 | `putus` |
+| martibalna | **`mar-tibal-na`** | 6.0 | `tibal` |
+| mandapatkon | **`man-dapat-kon`** | 6.0 | `dapat` |
+| kaseharian | **`ka-sehari-an`** | 6.0 | `sehari` |
+| malahirkon | **`ma-lahir-kon`** | 6.0 | `lahir` |
+| kayakinan | **`ka-yakin-an`** | 6.0 | `yakin` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language BTM appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -338,11 +518,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.78x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (237) |
-| Markov | **Context-4** | Highest predictability (97.8%) |
+| Tokenizer | **64k BPE** | Best compression (5.21x) |
+| N-gram | **2-gram** | Lowest perplexity (193) |
+| Markov | **Context-4** | Highest predictability (98.8%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -532,7 +713,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -548,7 +730,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-28 09:18:23*
+*Report Date: 2026-01-03 08:51:39*

models/embeddings/monolingual/btm_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:de188f03b192c2ca078f88738ce73449fcbf4320cd395bd332ecef48590d80a8
-size 1028830510
+oid sha256:eab8193f169ed951260bc5e2a8335421cc96bdc9afa7a4fbc2e5f19107d9f18c
+size 1028667993

models/embeddings/monolingual/btm_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 4643
+  "vocab_size": 4487
 }

models/embeddings/monolingual/btm_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:13216c4fed3a766eb9ae09c6f6a9636ab84897b0ed1d8023ee3f84bfc09c60fe
-size 257264686
+oid sha256:8c0f87796e30b250180cd4e0a21e5c53df57dbf3ac48c6ece59a24ce9ea2a2f9
+size 257221977

models/embeddings/monolingual/btm_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 4643
+  "vocab_size": 4487
 }

models/embeddings/monolingual/btm_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:51848e61c1fca01ab96c42576eb6e62eefed815eb2acc39e03fea244cb0d7a55
-size 514453294
+oid sha256:06a10d115cb9f5acc6300b8c1370ad70ca5bbbf741198c826bb6b2380bd77f72
+size 514370649

models/embeddings/monolingual/btm_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 4643
+  "vocab_size": 4487
 }

models/subword_markov/btm_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d9fb8732e5d0b088e8cb27dcdd5b2f86c532d02ef8d60b6a276973da326e2489
-size 44542
+oid sha256:4a5a6ceefeddffa92ade2020283b0670a56c9de4e981d4c37d567af5d7edf00c
+size 40549

models/subword_markov/btm_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "btm",
-  "unique_contexts": 776,
-  "total_transitions": 1324161
+  "unique_contexts": 845,
+  "total_transitions": 1194771
 }

models/subword_markov/btm_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0b32d6be400f0fe5ba28845e329de73b2eca4c57b1ecb34e7b271b0a81000cd5
-size 221609
+oid sha256:f8e63c72b397369614152fea75601c14782ddf5f412d72b019eb11cd857384b5
+size 173900

models/subword_markov/btm_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "btm",
-  "unique_contexts": 5530,
-  "total_transitions": 1322799
+  "unique_contexts": 4600,
+  "total_transitions": 1193471
 }

models/subword_markov/btm_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:baf72e1c86d729f7636eb3dd63f352dcd015fc51d995b57fe2a4a2ae73643c47
-size 752323
+oid sha256:b7597ab768a07a1247aa2becf4d6d80afbba74e537d60f41be994856978e37bc
+size 548580

models/subword_markov/btm_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "btm",
-  "unique_contexts": 28161,
-  "total_transitions": 1321437
+  "unique_contexts": 20094,
+  "total_transitions": 1192171
 }

models/subword_markov/btm_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f0a69fd846d53a6e55464bb04597ad44e2cd78771e025aad882c45ac3903096d
-size 1709839
+oid sha256:b4520c606c01af7c8dd7bdc888cc9cc1db3d4aa76abe56123d646a75bf737e6d
+size 1327069

models/subword_markov/btm_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "btm",
-  "unique_contexts": 97224,
-  "total_transitions": 1320075
+  "unique_contexts": 70348,
+  "total_transitions": 1190871
 }

models/subword_ngram/btm_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:86dfdf6bd3e46f0af84e22f9dfc755a0a319afbfb427a853d7447f54aa36b813
-size 25120
+oid sha256:2df596ff923d644e93e8a0534100541ef70c30a47243fc6d946a28b41b0b391f
+size 19810

models/subword_ngram/btm_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "btm",
-  "unique_ngrams": 1918,
-  "total_ngrams": 1324161
+  "unique_ngrams": 1424,
+  "total_ngrams": 1194771
 }

models/subword_ngram/btm_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6ac8d488fb34cbcad5cee074bab2aadc311377c61f106fdc7d15893ac59de7ac
-size 146419
+oid sha256:8c67b8a177fdee1d2cae2461ece233e72cc6271f2ec127adb17efbab362cb96f
+size 106854

models/subword_ngram/btm_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "btm",
-  "unique_ngrams": 12988,
-  "total_ngrams": 1322799
+  "unique_ngrams": 9264,
+  "total_ngrams": 1193471
 }

models/subword_ngram/btm_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a86ef1d182c41d4ec45dc51eae1271722efec2f5a35604893699f66328234dfd
-size 594616
+oid sha256:685a19fd481b30cfe583384638a09926eca754668b4379cc350384ca3d6ec64d
+size 469711

models/subword_ngram/btm_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "btm",
-  "unique_ngrams": 49385,
-  "total_ngrams": 1321437
+  "unique_ngrams": 38756,
+  "total_ngrams": 1192171
 }

models/tokenizer/btm_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ddd3c2f8af55f1ab6fddfd062122811cc700509a098060ad39d467f214f86732
-size 505236
+oid sha256:1c56b4bf3a440a54cda19a21cb12647b4eda168f5ed41c73de87c5dab1bb403c
+size 504094

models/tokenizer/btm_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c2a3d5dac68f0879797fe727d4d2b0cbc7840fdedeee4e86c6bb03f5adc7f0da
-size 788854
+oid sha256:6f9e24ab010b800e8b37dbe7ba7c3683db68cd1f831fc7c1773820eddf148959
+size 792383

models/tokenizer/btm_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:07075eace5ffa2fc3001cd72bdd62fa08486ac53b2331193d7cd888c21fbbbf1
-size 1322499
+oid sha256:e66c10c510f5dcaba827e3e9710ee5e2bf9168b1de095a4d7f971bdd4106bdae
+size 1307068

models/tokenizer/btm_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:842451ea10488cf3aa06059e02fdc805910e7ac01548035e26320a5ec1c675ff
-size 372071
+oid sha256:d9a7e75986934266a99f2c51f3d64b3ab75a0c504b45e45c89b6718cdc0aa876
+size 372323

models/vocabulary/btm_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c2e1669d9cb891748049f7b0f79cddae67d9c1d88f9aacfd042c76bc0023e258
-size 194790
+oid sha256:bbd9b813ffcb4592d15089be87b44c35ccb530113e5f1cdc05f8deb485295f2d
+size 177391

models/vocabulary/btm_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "btm",
-  "vocabulary_size": 12175,
+  "vocabulary_size": 11024,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.1439864864864865,
+    "type_token_ratio": 0.1393775944581053,
     "coverage": {
-      "top_100": 0.36650096525096526,
-      "top_1000": 0.6339671814671815,
-      "top_5000": 0.825945945945946,
-      "top_10000": 0.8937789575289575
+      "top_100": 0.3835752624203485,
+      "top_1000": 0.6533618888977023,
+      "top_5000": 0.8406599508209724,
+      "top_10000": 0.9080880992041458
     },
-    "hapax_count": 17659,
-    "hapax_ratio": 0.5919085607025542,
-    "total_documents": 1362
+    "hapax_count": 15333,
+    "hapax_ratio": 0.5817429904769131,
+    "total_documents": 1300
   }
 }

models/word_markov/btm_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5c217e1a408565da0c01ce1be6c3a82b8d05ba2eb243a350de04d9a1d0c26215
-size 1027194
+oid sha256:d7836709a62d3e6d6f77608b52ac844f6dc543e833b71cee1fecb9081e7920a1
+size 896043

models/word_markov/btm_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "btm",
-  "unique_contexts": 29952,
-  "total_transitions": 252626
+  "unique_contexts": 26321,
+  "total_transitions": 187805
 }

models/word_markov/btm_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:669e28490302d21e70f64127f665d428c1484dca09fc3c62a516bd2cce61c8e0
-size 2302244
+oid sha256:e923879187734024aa16a8259d19e5aa01d660c5479ca45138d2389819ed17a8
+size 2002914

models/word_markov/btm_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "btm",
-  "unique_contexts": 137080,
-  "total_transitions": 251264
+  "unique_contexts": 118363,
+  "total_transitions": 186505
 }

models/word_markov/btm_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5ca7a37098cceada0a3b495bf357855b55ae3c07593991310fc65a3987345d93
-size 3187577
+oid sha256:ae61b67145def4d85b3168d5df32d9a6ca772fee66ec6fc2c9b332518d820b52
+size 2632406

models/word_markov/btm_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "btm",
-  "unique_contexts": 206158,
-  "total_transitions": 249909
+  "unique_contexts": 165958,
+  "total_transitions": 185205
 }

models/word_markov/btm_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3644395ecf865c63d7aad1a53f542d746ec0774d12d6dae591b4e5db701c596a
-size 3651246
+oid sha256:564eaae60a3d3c768d5fb4a640a20c896748155dee9180b6baa3b03d48dddffc
+size 2962570

models/word_markov/btm_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "btm",
-  "unique_contexts": 227434,
-  "total_transitions": 248555
+  "unique_contexts": 176808,
+  "total_transitions": 183905
 }

models/word_ngram/btm_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:45947c8329577e1dfdfbb73f89b03e03aff959f4f2a1ebbc65fabc83ca7fc373
-size 76262
+oid sha256:b28566349442bdf587231d3e120350b8ce5af5eaaf1de6491b36baf6074721c6
+size 50150

models/word_ngram/btm_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "btm",
-  "unique_ngrams": 5965,
-  "total_ngrams": 252626
+  "unique_ngrams": 3791,
+  "total_ngrams": 187805
 }

models/word_ngram/btm_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:71ff57ab29c1ba5d6f5a7953a92d53d02c269b0162697be67aee03a490046ccb
-size 84350
+oid sha256:ba3f76c75ec581b71593b1ea9ac1e71af0a81dd5e2cb085a3ba21e79a2495c4d
+size 42795

models/word_ngram/btm_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "btm",
-  "unique_ngrams": 5700,
-  "total_ngrams": 251264
+  "unique_ngrams": 2726,
+  "total_ngrams": 186505
 }

models/word_ngram/btm_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:80d28cd8c28607e55ee503e11b2b708cd6502cdec2a7c34455ba7d5777b38646
-size 129368
+oid sha256:8332dcec08aee0b721f865e4dc871f229499960b01bd397769f75ecd597eeb12
+size 58180

models/word_ngram/btm_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "btm",
-  "unique_ngrams": 8130,
-  "total_ngrams": 249909
+  "unique_ngrams": 3343,
+  "total_ngrams": 185205
 }