Upload all models and assets for ban (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.782
+    value: 5.077
   - name: best_isotropy
     type: isotropy
-    value: 0.8612
+    value: 0.8530
   - name: vocabulary_size
     type: vocab
-    value: 109825
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # BAN - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,81 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.889x | 3.84 | 0.1469% | 269,485 |
-| **16k** | 4.255x | 4.21 | 0.1608% | 246,312 |
-| **32k** | 4.547x | 4.49 | 0.1718% | 230,479 |
-| **64k** | 4.782x 🏆 | 4.73 | 0.1807% | 219,125 |
+| **8k** | 4.073x | 4.08 | 0.1890% | 240,149 |
+| **16k** | 4.474x | 4.48 | 0.2076% | 218,639 |
+| **32k** | 4.813x | 4.82 | 0.2234% | 203,246 |
+| **64k** | 5.077x 🏆 | 5.08 | 0.2356% | 192,667 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `1020
-
-1021
-
-1022
-
-1023
-
-1024
-
-1025
-
-1026
-
-1027
-
-1028
-
-1029
-
-Jadma
-
-Embas
-
-Seda
-
-...`
+**Sample 1:** `Hamm (, Latin: Hammona) inggih punika kota ring Rhine-Westphalia Kalér, Jerman.`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ 1 0 2 0 ▁ 1 0 2 1 ... (+53 more)` | 63 |
-| 16k | `▁ 1 0 2 0 ▁ 1 0 2 1 ... (+53 more)` | 63 |
-| 32k | `▁ 1 0 2 0 ▁ 1 0 2 1 ... (+53 more)` | 63 |
-| 64k | `▁ 1 0 2 0 ▁ 1 0 2 1 ... (+53 more)` | 63 |
-
-**Sample 2:** `Pustaka
+| 8k | `▁ham m ▁(, ▁latin : ▁ham m ona ) ▁inggih ... (+12 more)` | 22 |
+| 16k | `▁ham m ▁(, ▁latin : ▁ham m ona ) ▁inggih ... (+10 more)` | 20 |
+| 32k | `▁ham m ▁(, ▁latin : ▁ham m ona ) ▁inggih ... (+10 more)` | 20 |
+| 64k | `▁hamm ▁(, ▁latin : ▁hamm ona ) ▁inggih ▁punika ▁kota ... (+8 more)` | 18 |
 
-Pranala liyané
-
-Kategori:Abad ka-17`
+**Sample 2:** `Kharkiv (), utawi Kharkov () inggih punika kota pinih ageng kakalih ring Ukraina...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁pustaka ▁pranala ▁liyané ▁kategori : abad ▁ka - 1 7` | 10 |
-| 16k | `▁pustaka ▁pranala ▁liyané ▁kategori : abad ▁ka - 1 7` | 10 |
-| 32k | `▁pustaka ▁pranala ▁liyané ▁kategori : abad ▁ka - 1 7` | 10 |
-| 64k | `▁pustaka ▁pranala ▁liyané ▁kategori : abad ▁ka - 1 7` | 10 |
+| 8k | `▁kh ark iv ▁(), ▁utawi ▁kh ark ov ▁() ▁inggih ... (+24 more)` | 34 |
+| 16k | `▁kh ark iv ▁(), ▁utawi ▁kh ark ov ▁() ▁inggih ... (+22 more)` | 32 |
+| 32k | `▁kh ark iv ▁(), ▁utawi ▁kh ark ov ▁() ▁inggih ... (+22 more)` | 32 |
+| 64k | `▁kharkiv ▁(), ▁utawi ▁khark ov ▁() ▁inggih ▁punika ▁kota ▁pinih ... (+15 more)` | 25 |
 
-**Sample 3:** `Siung Sri Lanka (Gracula ptilogenys), inggih punika satunggil curik, anggota kul...`
+**Sample 3:** `Brasília (;"Brasilia" (US) tur ) inggih punika ibu kota saking Brasil. Pustaka`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁si ung ▁sri ▁lan ka ▁( gr ac ula ▁p ... (+24 more)` | 34 |
-| 16k | `▁si ung ▁sri ▁lanka ▁( gr ac ula ▁p til ... (+21 more)` | 31 |
-| 32k | `▁siung ▁sri ▁lanka ▁( gr ac ula ▁p til ogen ... (+19 more)` | 29 |
-| 64k | `▁siung ▁sri ▁lanka ▁( gracula ▁ptil ogen ys ), ▁inggih ... (+15 more)` | 25 |
+| 8k | `▁br as í l ia ▁(; " br asil ia ... (+14 more)` | 24 |
+| 16k | `▁br as í lia ▁(; " br asil ia " ... (+13 more)` | 23 |
+| 32k | `▁brasília ▁(; " br asil ia " ▁( us ) ... (+10 more)` | 20 |
+| 64k | `▁brasília ▁(;" brasil ia " ▁( us ) ▁tur ▁) ... (+8 more)` | 18 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.782x compression
-- **Lowest UNK Rate:** 8k with 0.1469% unknown tokens
+- **Best Compression:** 64k achieves 5.077x compression
+- **Lowest UNK Rate:** 8k with 0.1890% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -151,57 +129,89 @@ Kategori:Abad ka-17`
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 6,772 🏆 | 12.73 | 86,017 | 32.0% | 53.5% |
-| **2-gram** | 287 🏆 | 8.17 | 6,739 | 67.5% | 98.5% |
-| **3-gram** | 9,433 | 13.20 | 132,180 | 30.5% | 50.3% |
-| **3-gram** | 2,255 | 11.14 | 56,338 | 28.2% | 73.8% |
-| **4-gram** | 14,846 | 13.86 | 212,984 | 26.8% | 45.5% |
-| **4-gram** | 10,513 | 13.36 | 295,874 | 17.0% | 50.1% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 4,798 | 12.23 | 59,688 | 35.6% | 57.3% |
+| **2-gram** | Subword | 225 🏆 | 7.81 | 7,788 | 73.4% | 99.2% |
+| **3-gram** | Word | 5,769 | 12.49 | 77,113 | 33.5% | 55.7% |
+| **3-gram** | Subword | 1,669 | 10.70 | 42,522 | 31.2% | 79.1% |
+| **4-gram** | Word | 8,680 | 13.08 | 116,715 | 28.6% | 51.0% |
+| **4-gram** | Subword | 7,684 | 12.91 | 208,144 | 18.1% | 53.6% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `situs resmi` | 41,099 |
+| 2 | `inggih punika` | 37,495 |
+| 3 | `silih tunggil` | 22,082 |
+| 4 | `pranala jaba` | 21,960 |
+| 5 | `pusat statistik` | 21,725 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `badan pusat statistik` | 21,708 |
+| 2 | `pustaka pranala jaba` | 20,507 |
+| 3 | `inggih punika silih` | 19,377 |
+| 4 | `punika silih tunggil` | 19,020 |
+| 5 | `pranala jaba situs` | 17,860 |
+
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `kategori :` | 56,343 |
-| 2 | `situs resmi` | 43,670 |
-| 3 | `inggih punika` | 39,156 |
-| 4 | `pusat statistik` | 24,773 |
-| 5 | `badan pusat` | 24,763 |
+| 1 | `inggih punika silih tunggil` | 18,913 |
+| 2 | `pranala jaba situs resmi` | 17,672 |
+| 3 | `pustaka pranala jaba situs` | 17,290 |
+| 4 | `dados kauahin ilang yening` | 14,166 |
+| 5 | `kauahin ilang yening url` | 13,881 |
 
-**3-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `badan pusat statistik` | 24,761 |
-| 2 | `pustaka pranala jaba` | 21,699 |
-| 3 | `) inggih punika` | 21,548 |
-| 4 | `inggih punika silih` | 20,523 |
-| 5 | `punika silih tunggil` | 20,157 |
+| 1 | `a n` | 880,577 |
+| 2 | `n g` | 735,053 |
+| 3 | `a _` | 536,413 |
+| 4 | `i n` | 523,219 |
+| 5 | `n _` | 516,092 |
 
-**4-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `inggih punika silih tunggil` | 20,047 |
-| 2 | `pranala jaba situs resmi` | 19,038 |
-| 3 | `pustaka pranala jaba situs` | 18,670 |
-| 4 | `) inggih punika silih` | 18,246 |
-| 5 | `( aksara bali :` | 17,893 |
+| 1 | `n g _` | 361,156 |
+| 2 | `a n _` | 287,413 |
+| 3 | `i n g` | 287,067 |
+| 4 | `a n g` | 219,608 |
+| 5 | `_ k a` | 213,760 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `i n g _` | 219,518 |
+| 2 | `r i n g` | 145,165 |
+| 3 | `_ r i n` | 128,090 |
+| 4 | `a n g _` | 86,655 |
+| 5 | `u n i k` | 72,566 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 287
+- **Best Perplexity:** 2-gram (subword) with 225
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~50% of corpus
+- **Coverage:** Top-1000 patterns cover ~54% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -209,55 +219,86 @@ Kategori:Abad ka-17`
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.5448 | 1.459 | 4.25 | 354,734 | 45.5% |
-| **1** | 1.0973 | 2.140 | 6.84 | 3,094 | 0.0% |
-| **2** | 0.2539 | 1.192 | 1.68 | 1,504,653 | 74.6% |
-| **2** | 0.8467 | 1.798 | 5.47 | 21,162 | 15.3% |
-| **3** | 0.0992 | 1.071 | 1.20 | 2,520,521 | 90.1% |
-| **3** | 0.8884 | 1.851 | 4.45 | 115,750 | 11.2% |
-| **4** | 0.0443 🏆 | 1.031 | 1.08 | 3,008,321 | 95.6% |
-| **4** | 0.7379 🏆 | 1.668 | 3.16 | 515,141 | 26.2% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.7212 | 1.649 | 5.13 | 253,714 | 27.9% |
+| **1** | Subword | 0.9714 | 1.961 | 7.03 | 4,633 | 2.9% |
+| **2** | Word | 0.2297 | 1.173 | 1.53 | 1,298,868 | 77.0% |
+| **2** | Subword | 0.6107 | 1.527 | 3.55 | 32,560 | 38.9% |
+| **3** | Word | 0.0749 | 1.053 | 1.14 | 1,983,308 | 92.5% |
+| **3** | Subword | 0.5954 | 1.511 | 3.32 | 115,474 | 40.5% |
+| **4** | Word | 0.0289 🏆 | 1.020 | 1.05 | 2,240,261 | 97.1% |
+| **4** | Subword | 0.6610 | 1.581 | 2.96 | 383,801 | 33.9% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `, seperti kota binjai kategori : désa dinas sané mangkin madué 10 désa pakraman buléléng .`
-2. `. iklan di pulo kyushu . akéhnyané 1 . kategori : ᬓ ᭂ ᬕᬮ ᭄ ᬤ`
-3. `ring warsa 2019 , definisi definisi asli riantara 24 / ilang . there is defined hypnosis`
+1. `ring warsa puniki dados kauahin ilang yening url dados kaapus saking sistem ekologi dan bedah langsu...`
+2. `kabupatén kediri jawa timur pustaka pranala jaba situs resmi pamréntahan wali ngancan ngamokohang ba...`
+3. `punika silih tunggil désa ring thailand punika wenten ring sérial mabasis ring wewidangan kecamatan ...`
 
 **Context Size 2:**
 
-1. `kategori : kota kendari wali kota ngawit jabatan saking pinanggal 22 pébruari 1857 – 1 al -`
-2. `situs resmi pamréntahan kabupatén tuban cutetan : pranala dados kauahin / ilang . yening url nenten ...`
-3. `inggih punika silih tunggil sanganan sané nénten pastika sakéwanten sumber akéh saking cina , itsĕrl...`
+1. `situs resmi pamréntahan kabupatén bima badan pusat statistik kota bengkulu badan pusat statistik pro...`
+2. `inggih punika silih sinunggil gendingan tradisional thailand sane pinih sering kacingak pinaka gerha...`
+3. `silih tunggil pagending tur ngamedalang surat kaputusan nomor sadurugnyane ring warsa akéh kramanyan...`
 
 **Context Size 3:**
 
-1. `badan pusat statistik kepulauan bangka belitung badan pusat statistik kota surabaya cutetan : url da...`
-2. `pustaka pranala jaba taman pahlawan margarana , ring pamahbah nyané , kain sasirangan kapercaya pras...`
-3. `) inggih punika silih tunggil kecamatan ring kabupatén bungo , propinsi jambi , ring panegara indoné...`
+1. `badan pusat statistik propinsi jawa tengah indonésia mawit saking pérméndagri nomor 137 warsa indik ...`
+2. `pustaka pranala jaba situs resmi propinsi bali badan pusat statistik propinsi kalimantan selatan bad...`
+3. `inggih punika silih tunggil kecamatan ring kabupatén timor tengah utara ring nusa tenggara timur bad...`
 
 **Context Size 4:**
 
-1. `inggih punika silih tunggil kecamatan ring kabupatén gowa , propinsi sulawesi selatan tanjung batu ,...`
-2. `pranala jaba situs resmi pemerintah kota tangerang situs resmi bps kota tangerang cutetan : url dado...`
-3. `pustaka pranala jaba situs resmi pamréntahan nusa tenggara barat badan pusat statistik sumatra utara...`
+1. `inggih punika silih tunggil désa ring kecamatan pulau pulau kur tual propinsi maluku indonésia pusta...`
+2. `pranala jaba situs resmi pamrentahan provinsi kepulauan bangka belitung badan pusat statistik kabupa...`
+3. `pustaka pranala jaba situs resmi pamrentahan provinsi kepulauan bangka belitung badan pusat statisti...`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `a._gané_l_i,_dan`
+2. `_ptrandopi_mi_ba`
+3. `n_107_sika,_dika`
+
+**Context Size 2:**
+
+1. `angaing_wawewidué`
+2. `ng_gu_kin_éman_no`
+3. `a_]_garingang_lat`
+
+**Context Size 3:**
+
+1. `ng_kabupatén_sané_`
+2. `an_kaapustaka_miwa`
+3. `inggih_tunggih_pas`
+
+**Context Size 4:**
+
+1. `ing_basa_badan_pran`
+2. `ring_kaapus_sané_ri`
+3. `_ring_kabupatén_kah`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 95.6% predictability
+- **Best Predictability:** Context-4 (word) with 97.1% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (515,141 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (383,801 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -273,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 109,825 |
-| Total Tokens | 4,059,826 |
-| Mean Frequency | 36.97 |
+| Vocabulary Size | 96,177 |
+| Total Tokens | 3,540,495 |
+| Mean Frequency | 36.81 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 763.95 |
+| Frequency Std Dev | 739.04 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ring | 133,380 |
-| 2 | kabupatén | 67,955 |
-| 3 | kategori | 56,442 |
-| 4 | punika | 52,655 |
-| 5 | situs | 48,035 |
-| 6 | sané | 47,128 |
-| 7 | resmi | 44,824 |
-| 8 | kecamatan | 42,212 |
-| 9 | inggih | 39,593 |
-| 10 | saking | 39,394 |
+| 1 | ring | 127,899 |
+| 2 | kabupatén | 58,514 |
+| 3 | punika | 50,657 |
+| 4 | sané | 45,835 |
+| 5 | situs | 44,988 |
+| 6 | resmi | 42,224 |
+| 7 | inggih | 37,927 |
+| 8 | saking | 37,341 |
+| 9 | url | 32,061 |
+| 10 | miwah | 31,507 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | padaido | 2 |
-| 2 | inswambesi | 2 |
-| 3 | asaryendi | 2 |
-| 4 | sopendo | 2 |
-| 5 | pomdori | 2 |
-| 6 | yawosi | 2 |
-| 7 | ᬧᬓᬓ | 2 |
-| 8 | potrekwastanngawit | 2 |
-| 9 | patonangi | 2 |
-| 10 | ᬢᬢᬓᬦ | 2 |
+| 1 | kitou | 2 |
+| 2 | sialet | 2 |
+| 3 | dibanda | 2 |
+| 4 | ᬦᬶᬲᬫ᭄ | 2 |
+| 5 | reuba | 2 |
+| 6 | reuleut | 2 |
+| 7 | rheue | 2 |
+| 8 | uleue | 2 |
+| 9 | muling | 2 |
+| 10 | sanderling | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.1336 |
-| R² (Goodness of Fit) | 0.997567 |
+| Zipf Coefficient | 1.1306 |
+| R² (Goodness of Fit) | 0.997983 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 43.3% |
-| Top 1,000 | 67.9% |
-| Top 5,000 | 82.1% |
-| Top 10,000 | 87.0% |
+| Top 100 | 44.6% |
+| Top 1,000 | 68.9% |
+| Top 5,000 | 82.9% |
+| Top 10,000 | 87.9% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9976 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 43.3% of corpus
-- **Long Tail:** 99,825 words needed for remaining 13.0% coverage
+- **Zipf Compliance:** R²=0.9980 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 44.6% of corpus
+- **Long Tail:** 86,177 words needed for remaining 12.1% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -343,24 +384,127 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 50,333 | 32 | 4.290 | 1.041 | 0.8612 🏆 |
-| **mono_64d** | 50,333 | 64 | 4.879 | 1.017 | 0.8485 |
-| **mono_128d** | 50,333 | 128 | 5.532 | 0.920 | 0.8053 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8530 🏆 | 0.3516 | N/A | N/A |
+| **mono_64d** | 64 | 0.8495 | 0.2832 | N/A | N/A |
+| **mono_128d** | 128 | 0.8092 | 0.2232 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8612 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 50,333 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8530 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2860. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ma` | martins, masduki, maffin |
+| `-ka` | kaméloh, kaaranin, kasum |
+| `-pa` | palopat, panandatanganan, pail |
+| `-pe` | peting, pencok, pemantauan |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-n` | baharuddin, setyawan, roussillon |
+| `-an` | setyawan, panandatanganan, mengupayakan |
+| `-ng` | peting, speaking, sanderling |
+| `-ang` | tenggarang, lendang, nguwahang |
+| `-né` | leluhurnyané, putranidané, bébékné |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `anga` | 1.47x | 361 contexts | nanga, sanga, hanga |
+| `ngan` | 1.54x | 182 contexts | angan, ingan, tengan |
+| `nten` | 1.71x | 86 contexts | inten, enten, wnten |
+| `atan` | 1.52x | 149 contexts | vatan, gatan, matan |
+| `ungg` | 1.55x | 117 contexts | tungg, ungga, unggun |
+| `akin` | 1.88x | 41 contexts | aking, yakin, dakin |
+| `nggi` | 1.58x | 73 contexts | anggi, nggih, senggi |
+| `taha` | 1.90x | 32 contexts | tahai, tahap, tahan |
+| `ggih` | 2.03x | 22 contexts | nggih, lnggih, inggih |
+| `ados` | 2.01x | 22 contexts | dados, sados, padosa |
+| `isti` | 1.61x | 36 contexts | bistik, sistim, pistia |
+| `cama` | 1.87x | 19 contexts | camat, camas, camah |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-pa` | `-n` | 112 words | palimunan, pawacanan |
+| `-ka` | `-n` | 112 words | kamerdékaan, kagenahin |
+| `-pa` | `-an` | 96 words | palimunan, pawacanan |
+| `-pe` | `-n` | 92 words | perhubungan, penyaringan |
+| `-pe` | `-an` | 81 words | perhubungan, penyaringan |
+| `-ka` | `-ng` | 77 words | kaidipang, kawedharang |
+| `-ka` | `-ang` | 61 words | kaidipang, kawedharang |
+| `-ka` | `-an` | 56 words | kamerdékaan, kamaharajan |
+| `-ma` | `-n` | 55 words | marepan, mapitungan |
+| `-ma` | `-an` | 39 words | marepan, mapitungan |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| kauningan | **`ka-uning-an`** | 6.0 | `uning` |
+| kaorganisasiang | **`ka-organisasi-ang`** | 6.0 | `organisasi` |
+| kakaonang | **`ka-ka-onang`** | 6.0 | `onang` |
+| pasilihan | **`pa-silih-an`** | 6.0 | `silih` |
+| kajahatan | **`ka-jahat-an`** | 6.0 | `jahat` |
+| kasedukan | **`ka-seduk-an`** | 6.0 | `seduk` |
+| kalaporang | **`ka-lapor-ang`** | 6.0 | `lapor` |
+| kakuasaan | **`ka-kuasa-an`** | 6.0 | `kuasa` |
+| padruwénan | **`pa-druwén-an`** | 6.0 | `druwén` |
+| palekadan | **`pa-lekad-an`** | 6.0 | `lekad` |
+| mategakan | **`ma-tegak-an`** | 6.0 | `tegak` |
+| kaungkabang | **`ka-ungkab-ang`** | 6.0 | `ungkab` |
+| kauwugang | **`ka-uwug-ang`** | 6.0 | `uwug` |
+| panyambung | **`pa-nyambu-ng`** | 6.0 | `nyambu` |
+| panularan | **`pa-nular-an`** | 6.0 | `nular` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language BAN appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -368,11 +512,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.78x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (287) |
-| Markov | **Context-4** | Highest predictability (95.6%) |
+| Tokenizer | **64k BPE** | Best compression (5.08x) |
+| N-gram | **2-gram** | Lowest perplexity (225) |
+| Markov | **Context-4** | Highest predictability (97.1%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -562,7 +707,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -578,7 +724,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 23:53:08*
+*Report Date: 2026-01-03 06:12:56*

models/embeddings/monolingual/ban_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ea87c1477ff7362a05ee5aecc04514daeeb0751d9258d36096220826e2071195
-size 1076400925
+oid sha256:bdf8bd41f143ec229b2b83e523def59ca3894f5e32c88dc9ad1d5777bb3cdbc1
+size 1069224476

models/embeddings/monolingual/ban_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 50333
+  "vocab_size": 43447
 }

models/embeddings/monolingual/ban_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b78c8beda9adf946cfb61ebe5d7345f1a46cfbada20e008894b25148ee2b7fb6
-size 269745181
+oid sha256:3c7b5b04af24d58eb2352605fd5a6a3884aa17bd5211923840f1d2399995b3dc
+size 267857180

models/embeddings/monolingual/ban_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 50333
+  "vocab_size": 43447
 }

models/embeddings/monolingual/ban_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ac247ae04637df7c57b93803868d52ed47fb91b34dd674de9b86612a1856543d
-size 538630429
+oid sha256:45be3ee34bd2a796319b97a63a3801529c7d2b0804d891a90aa7697ac777fcda
+size 534979612

models/embeddings/monolingual/ban_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 50333
+  "vocab_size": 43447
 }

models/subword_markov/ban_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:07d34933ba1ea350752ae85b877a8c59815d819177bf4c06cc1c33c52a23a4ab
-size 164464
+oid sha256:b557a43cbf89ec6e5b8fe521107f375a803257117295fa8f2f26d54017e944ef
+size 238064

models/subword_markov/ban_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "ban",
-  "unique_contexts": 3094,
-  "total_transitions": 31281701
+  "unique_contexts": 4633,
+  "total_transitions": 26072638
 }

models/subword_markov/ban_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7a094b1a8d92237ba3ee5d4bb02c77daf556d4e57c92818f8594ee12d0d9dba7
-size 930375
+oid sha256:bc162e729fc9f1b78bdc516d217698eb4c1e8af546cf0d4b64801623d1296b5e
+size 1052622

models/subword_markov/ban_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "ban",
-  "unique_contexts": 21162,
-  "total_transitions": 31247141
+  "unique_contexts": 32560,
+  "total_transitions": 26039719
 }

models/subword_markov/ban_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c14ef4a97de8adc59b7a39da2f9d5cdcbdc15c1339b56d4fe8f54919ee7676cc
-size 3747847
+oid sha256:62c18ce24bb55f6d1333346623229b9025cc0f78b411f131da48dbb5f6b5bfaf
+size 3150588

models/subword_markov/ban_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "ban",
-  "unique_contexts": 115750,
-  "total_transitions": 31212581
+  "unique_contexts": 115474,
+  "total_transitions": 26006800
 }

models/subword_markov/ban_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:83b89243df0423d04c3f14b54a3854b87b3df8860b6528722a527ca63138bedf
-size 12242821
+oid sha256:beb82bbbccc1e498763133c2ca43a33d34399701a3feadc33b33d7e28db10c2d
+size 9163427

models/subword_markov/ban_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "ban",
-  "unique_contexts": 515141,
-  "total_transitions": 31178021
+  "unique_contexts": 383801,
+  "total_transitions": 25973881
 }

models/subword_ngram/ban_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1d331dea0ac3863010c7f44bc8af6db64163208b39797a90da3a3db2cb5068c2
-size 89432
+oid sha256:f8d06b7f0eea6caff12eb48816bcfb2e270b3b2491f26727e3f0514ff35fc746
+size 106802

models/subword_ngram/ban_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "ban",
-  "unique_ngrams": 6739,
-  "total_ngrams": 31281701
+  "unique_ngrams": 7788,
+  "total_ngrams": 26072638
 }

models/subword_ngram/ban_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e87314f3eb227a47badf79f1a2964e00acef96da0cc33f99259523ddadbe0267
-size 689305
+oid sha256:9343f458518ed0d1f0a407a342e9b5cecbee42b529784e0dcb0b72918b79d1d3
+size 564805

models/subword_ngram/ban_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "ban",
-  "unique_ngrams": 56338,
-  "total_ngrams": 31247141
+  "unique_ngrams": 42522,
+  "total_ngrams": 26039719
 }

models/subword_ngram/ban_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8fe02f497267ca92b94af3687493cf750310e35ab85b736752b26ff56e51a295
-size 3233720
+oid sha256:b5c1474de28fe82c885eb32b731cad7f32987a3ef6b82a9ab47b3b48c9e2ce1a
+size 2348974

models/subword_ngram/ban_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "ban",
-  "unique_ngrams": 295874,
-  "total_ngrams": 31212581
+  "unique_ngrams": 208144,
+  "total_ngrams": 26006800
 }

models/tokenizer/ban_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bedf8563a1afe2051346993a2e31dc66fa70f8ddf79e172628e2963e0a99dd47
-size 503307
+oid sha256:0194959437d4a8b04b5d3f37d4b95caf926ecb7c4136e157e06f185417710348
+size 507366

models/tokenizer/ban_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c710ee000231372d734483b9d28128e8359cb0283ee8585cfee7e121ff3e088c
-size 776699
+oid sha256:8d9e73114a8bfb09cc05cf62d77156539332a4b993d735d59704b21d2055bd5b
+size 785247

models/tokenizer/ban_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a7f6893ba9c5195f098265006c68b68d4f854f4f3aef663fccea594bd3ecddbb
-size 1338022
+oid sha256:0c4163cfc92fedee34979f21e46b3960651c5a1aea63e481f06e935586c4982c
+size 1355653

models/tokenizer/ban_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e665b6cb1dccbf5b7fa71b5c8e3e9ea156603fcf9f94aeeaa2d783a6631da368
-size 371066
+oid sha256:79dc41677c1fda74a9243a5e8e6b256d3a8d6e89e202f72fe68235e2c50ce0df
+size 372914

models/vocabulary/ban_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:33bac4c04f637c4a7c7b840996ea84729668359a0bba018a9c721c27ccd525bc
-size 1847029
+oid sha256:8e4e8f2a999ef9ad888861be852297181f3f88c4f902b2c2d3a001b3f88a5eeb
+size 1669252

models/vocabulary/ban_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "ban",
-  "vocabulary_size": 109825,
+  "vocabulary_size": 96177,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.08234307722115898,
+    "type_token_ratio": 0.06865216883420294,
     "coverage": {
-      "top_100": 0.4083867316599922,
-      "top_1000": 0.6399472916582452,
-      "top_5000": 0.7739293501921968,
-      "top_10000": 0.8207767696718877
+      "top_100": 0.4271482296290528,
+      "top_1000": 0.6597757616661908,
+      "top_5000": 0.7939794084053682,
+      "top_10000": 0.8415540715935934
     },
-    "hapax_count": 244616,
-    "hapax_ratio": 0.690145891699888,
-    "total_documents": 34560
+    "hapax_count": 157713,
+    "hapax_ratio": 0.6211863405411793,
+    "total_documents": 32919
   }
 }

models/word_markov/ban_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:490f92d81b31f52d83fd0f2cd5b002bd546e2be9ec34f436bbd4b503863a38fd
-size 14573406
+oid sha256:9b32e21376297d35ef6bbd5d4573db01b79bce03f68060112590a5eccf22d3f2
+size 12122512

models/word_markov/ban_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "ban",
-  "unique_contexts": 354734,
-  "total_transitions": 5300849
+  "unique_contexts": 253714,
+  "total_transitions": 3665289
 }

models/word_markov/ban_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7933f21fe3dc7737031daf59c6f0b75d5f0ec0760ea5fbe33e643b28f9377775
-size 30285330
+oid sha256:c0cc40598890a3afe8525cbf4abca4b849a1276159d24aed089a152527e3bc5a
+size 26358283

models/word_markov/ban_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "ban",
-  "unique_contexts": 1504653,
-  "total_transitions": 5266289
+  "unique_contexts": 1298868,
+  "total_transitions": 3632370
 }

models/word_markov/ban_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b7d248fbf95bb7c4c07d3ad6a40e7491d998158dd768f5df49fb1a1fb786ad4d
-size 44547695
+oid sha256:fb834be2557bcc40bfaf6bbd8ef0594de27818bffcc89cc2b4522ed98fb5eb69
+size 35570394

models/word_markov/ban_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "ban",
-  "unique_contexts": 2520521,
-  "total_transitions": 5231730
+  "unique_contexts": 1983308,
+  "total_transitions": 3599451
 }

models/word_markov/ban_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:af85b8acafe95f6c6598d6c3d9f95ab7ddbfe52fec36eb64d4e73ed213f6ba54
-size 53030226
+oid sha256:b5519a7b2bd7e347aaad32b2cb5de479a42d7d09c0c4a8a797a857d5a6c9d26d
+size 40846702

models/word_markov/ban_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "ban",
-  "unique_contexts": 3008321,
-  "total_transitions": 5197173
+  "unique_contexts": 2240261,
+  "total_transitions": 3566533
 }

models/word_ngram/ban_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3a7e05025863931724c16c54d53b71d59185644881045f57e68714b7edff8d95
-size 1171748
+oid sha256:aa43961bd9c5b0bc97c6e58a2e5768d2613949b7a4ca2a54797209038ce2a44e
+size 878363

models/word_ngram/ban_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "ban",
-  "unique_ngrams": 86017,
-  "total_ngrams": 5300849
+  "unique_ngrams": 59688,
+  "total_ngrams": 3665289
 }

models/word_ngram/ban_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6382c65c522ccb8d948baed426584d1da5bc0135ba2f1a160c62c3a280a15c16
-size 1910455
+oid sha256:ec8256fd7645e4ba79178c17a3abd2abc970f54a61627bd98d6add5ed6abd22f
+size 1209013

models/word_ngram/ban_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "ban",
-  "unique_ngrams": 132180,
-  "total_ngrams": 5266289
+  "unique_ngrams": 77113,
+  "total_ngrams": 3632370
 }

models/word_ngram/ban_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4076c7de92dd17498060b87078339b4ccf91574d719a5e0d84560c92179b24b1
-size 3227946
+oid sha256:08d1fb1d14940fae2e02caf50543e44b49547bf66e935ba7c9d5f912cb36ee4f
+size 1928264

models/word_ngram/ban_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "ban",
-  "unique_ngrams": 212984,
-  "total_ngrams": 5231730
+  "unique_ngrams": 116715,
+  "total_ngrams": 3599451
 }