Upload all models and assets for ce (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 3.716
+    value: 3.783
   - name: best_isotropy
     type: isotropy
-    value: 0.8750
+    value: 0.8761
   - name: vocabulary_size
     type: vocab
-    value: 267119
-generated: 2025-12-28
+    value: 0
+generated: 2026-01-03
 ---
 
 # CE - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,59 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 2.743x | 2.70 | 1.0676% | 595,703 |
-| **16k** | 3.096x | 3.04 | 1.2050% | 527,806 |
-| **32k** | 3.417x | 3.36 | 1.3298% | 478,250 |
-| **64k** | 3.716x 🏆 | 3.65 | 1.4461% | 439,790 |
+| **8k** | 2.792x | 2.80 | 0.9604% | 543,837 |
+| **16k** | 3.140x | 3.15 | 1.0803% | 483,478 |
+| **32k** | 3.480x | 3.49 | 1.1970% | 436,328 |
+| **64k** | 3.783x 🏆 | 3.79 | 1.3016% | 401,281 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `ДагӀйурд () — Азербайджанан Ходжалин кӀоштара эвла.
-
- Бахархой 
-
- Билгалдахарш 
-...`
+**Sample 1:** `Жаныспай (Акмолан область) Жаныспай (Костанайн область)`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁да гӏ й ур д ▁() ▁— ▁азербайджанан ▁х од ... (+17 more)` | 27 |
-| 16k | `▁дагӏ йур д ▁() ▁— ▁азербайджанан ▁ход ж алин ▁кӏоштара ... (+13 more)` | 23 |
-| 32k | `▁дагӏ йур д ▁() ▁— ▁азербайджанан ▁ходжалин ▁кӏоштара ▁эвла . ... (+9 more)` | 19 |
-| 64k | `▁дагӏ йур д ▁() ▁— ▁азербайджанан ▁ходжалин ▁кӏоштара ▁эвла . ... (+9 more)` | 19 |
+| 8k | `▁жан ыс п ай ▁( ак молан ▁область ) ▁жан ... (+8 more)` | 18 |
+| 16k | `▁жан ыс пай ▁( акмолан ▁область ) ▁жан ыс пай ... (+5 more)` | 15 |
+| 32k | `▁жан ыс пай ▁( акмолан ▁область ) ▁жан ыс пай ... (+4 more)` | 14 |
+| 64k | `▁жан ыс пай ▁( акмолан ▁область ) ▁жан ыс пай ... (+4 more)` | 14 |
 
-**Sample 2:** `Перселл (Миссури)
- Перселл (Оклахома)`
+**Sample 2:** `Антиго (Висконсин) Антиго (Маса-Карара) Антиго (гӀала, Висконсин)`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁пер сел л ▁( миссури ) ▁пер сел л ▁( ... (+2 more)` | 12 |
-| 16k | `▁пер сел л ▁( миссури ) ▁пер сел л ▁( ... (+2 more)` | 12 |
-| 32k | `▁пер сел л ▁( миссури ) ▁пер сел л ▁( ... (+2 more)` | 12 |
-| 64k | `▁пер селл ▁( миссури ) ▁пер селл ▁( оклахома )` | 10 |
+| 8k | `▁анти го ▁( ви сконсин ) ▁анти го ▁( ма ... (+12 more)` | 22 |
+| 16k | `▁анти го ▁( висконсин ) ▁анти го ▁( ма са ... (+11 more)` | 21 |
+| 32k | `▁анти го ▁( висконсин ) ▁анти го ▁( маса - ... (+9 more)` | 19 |
+| 64k | `▁анти го ▁( висконсин ) ▁анти го ▁( маса - ... (+9 more)` | 19 |
 
-**Sample 3:** `Эль Баро (Мочитлан)
- Эль Ба��о (Сан Мигел Тотолапан)
- Эль Баро (Хенерал Элиодоро ...`
+**Sample 3:** `Барда (Иркутскан область) Барда (Пермийн мохк) Барда (гӀала)`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁эль ▁баро ▁( м оч ит лан ) ▁эль ▁баро ... (+21 more)` | 31 |
-| 16k | `▁эль ▁баро ▁( м оч итлан ) ▁эль ▁баро ▁( ... (+18 more)` | 28 |
-| 32k | `▁эль ▁баро ▁( м оч итлан ) ▁эль ▁баро ▁( ... (+11 more)` | 21 |
-| 64k | `▁эль ▁баро ▁( моч итлан ) ▁эль ▁баро ▁( сан ... (+10 more)` | 20 |
+| 8k | `▁бар да ▁( иркутскан ▁область ) ▁бар да ▁( пермийн ... (+7 more)` | 17 |
+| 16k | `▁бар да ▁( иркутскан ▁область ) ▁бар да ▁( пермийн ... (+7 more)` | 17 |
+| 32k | `▁барда ▁( иркутскан ▁область ) ▁барда ▁( пермийн ▁мохк ) ... (+4 more)` | 14 |
+| 64k | `▁барда ▁( иркутскан ▁область ) ▁барда ▁( пермийн ▁мохк ) ... (+4 more)` | 14 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 3.716x compression
-- **Lowest UNK Rate:** 8k with 1.0676% unknown tokens
+- **Best Compression:** 64k achieves 3.783x compression
+- **Lowest UNK Rate:** 8k with 0.9604% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -129,57 +129,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 3,434 🏆 | 11.75 | 180,710 | 25.5% | 63.0% |
-| **2-gram** | 484 🏆 | 8.92 | 7,755 | 52.4% | 97.1% |
-| **3-gram** | 5,932 | 12.53 | 322,719 | 16.0% | 53.7% |
-| **3-gram** | 2,779 | 11.44 | 72,318 | 22.8% | 66.2% |
-| **4-gram** | 7,779 | 12.93 | 709,202 | 12.7% | 49.4% |
-| **4-gram** | 7,269 | 12.83 | 422,662 | 15.4% | 47.3% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 2,545 | 11.31 | 100,140 | 25.5% | 70.0% |
+| **2-gram** | Subword | 423 🏆 | 8.72 | 6,176 | 55.1% | 98.2% |
+| **3-gram** | Word | 3,286 | 11.68 | 157,541 | 21.2% | 65.9% |
+| **3-gram** | Subword | 2,337 | 11.19 | 58,954 | 23.8% | 69.8% |
+| **4-gram** | Word | 4,089 | 12.00 | 330,019 | 18.2% | 63.2% |
+| **4-gram** | Subword | 5,832 | 12.51 | 337,533 | 16.4% | 50.9% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `нах беха` | 927,008 |
+| 2 | `беха меттигаш` | 876,464 |
+| 3 | `билгалдахарш хьажоргаш` | 387,483 |
+| 4 | `климат кхузахь` | 294,017 |
+| 5 | `сахьтан аса` | 272,866 |
+
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `. —` | 1,635,703 |
-| 2 | `категори :` | 1,346,163 |
-| 3 | `нах беха` | 1,039,301 |
-| 4 | `беха меттигаш` | 953,016 |
-| 5 | `м .` | 797,532 |
+| 1 | `нах беха меттигаш` | 876,426 |
+| 2 | `кӏоштан нах беха` | 256,950 |
+| 3 | `климат кхузахь климат` | 254,686 |
+| 4 | `бахархой билгалдахарш хьажоргаш` | 156,558 |
+| 5 | `сахьтан аса йу` | 135,690 |
 
-**3-grams:**
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `нах беха меттигаш` | 952,979 |
-| 2 | `( ) —` | 477,700 |
-| 3 | `меттигаш категори :` | 455,946 |
-| 4 | `беха меттигаш категори` | 448,323 |
-| 5 | `. а .` | 416,844 |
+| 1 | `кӏоштан нах беха меттигаш` | 256,946 |
+| 2 | `лелаш ду сахьтан аса` | 134,397 |
+| 3 | `нийса лелаш ду сахьтан` | 134,397 |
+| 4 | `сахьтан аса йу utc` | 133,768 |
+| 5 | `ду сахьтан аса йу` | 133,768 |
 
-**4-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `беха меттигаш категори :` | 448,323 |
-| 2 | `нах беха меттигаш категори` | 448,323 |
-| 3 | `. — м .` | 345,745 |
-| 4 | `— м . :` | 345,423 |
-| 5 | `кӏоштан нах беха меттигаш` | 256,924 |
+| 1 | `а _` | 8,696,976 |
+| 2 | `. _` | 8,337,924 |
+| 3 | `н _` | 7,066,559 |
+| 4 | `а н` | 6,445,422 |
+| 5 | `р а` | 5,305,199 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `а н _` | 4,127,441 |
+| 2 | `_ — _` | 2,719,160 |
+| 3 | `а ш _` | 1,910,774 |
+| 4 | `и н _` | 1,668,837 |
+| 5 | `а р а` | 1,610,648 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `т а н _` | 1,416,987 |
+| 2 | `а х а р` | 1,374,119 |
+| 3 | `. _ — _` | 1,045,081 |
+| 4 | `а _ м е` | 1,006,220 |
+| 5 | `_ м е т` | 999,858 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 484
+- **Best Perplexity:** 2-gram (subword) with 423
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~47% of corpus
+- **Coverage:** Top-1000 patterns cover ~51% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -187,55 +219,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.4898 | 1.404 | 3.61 | 596,401 | 51.0% |
-| **1** | 1.0551 | 2.078 | 11.42 | 1,510 | 0.0% |
-| **2** | 0.2471 | 1.187 | 1.75 | 2,141,469 | 75.3% |
-| **2** | 1.0286 | 2.040 | 7.78 | 17,227 | 0.0% |
-| **3** | 0.1096 | 1.079 | 1.30 | 3,726,034 | 89.0% |
-| **3** | 0.8548 | 1.809 | 4.95 | 133,970 | 14.5% |
-| **4** | 0.0635 🏆 | 1.045 | 1.17 | 4,825,259 | 93.6% |
-| **4** | 0.7262 🏆 | 1.654 | 3.28 | 662,768 | 27.4% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.6226 | 1.540 | 3.90 | 520,111 | 37.7% |
+| **1** | Subword | 0.9426 | 1.922 | 9.07 | 1,553 | 5.7% |
+| **2** | Word | 0.1849 | 1.137 | 1.44 | 2,019,671 | 81.5% |
+| **2** | Subword | 0.9737 | 1.964 | 7.37 | 14,069 | 2.6% |
+| **3** | Word | 0.0632 | 1.045 | 1.13 | 2,889,994 | 93.7% |
+| **3** | Subword | 0.8560 | 1.810 | 4.97 | 103,627 | 14.4% |
+| **4** | Word | 0.0320 🏆 | 1.022 | 1.08 | 3,246,178 | 96.8% |
+| **4** | Subword | 0.7168 | 1.643 | 3.27 | 515,118 | 28.3% |
+
+### Generated Text Samples (Word-based)
 
-### Generated Text Samples
+Below are text samples generated from each word-based Markov chain model:
 
-Below are text samples generated from each Markov chain model:
+**Context Size 1:**
+
+1. `а ду йалташ хастоьмаш малхбален кӏошташкара пачхьалкхан европин дехьайолуш алсама гӏийлачу мехца бек...`
+2. `нах беха меттигаш нах беха меттигаш провинцин нах беха меттигаш кӏоштан нах беха меттигаш воеводалли...`
+3. `беха меттигаш штатан йукъахь квинс университет им м в пономарёва м прохоров т 82 т и`
+
+**Context Size 2:**
+
+1. `нах беха меттигаш нах беха меттигаш нисйина нах беха меттигаш кӏоштан нах беха меттигаш кӏоштан нах ...`
+2. `беха меттигаш кӏоштан нах беха меттигаш кӏоштан нах беха меттигаш воеводаллин нах беха меттигаш нах ...`
+3. `билгалдахарш хьажоргаш черкассин областан индексаш кӏоштан нах беха меттигаш микрокӏошташ нах беха м...`
+
+**Context Size 3:**
+
+1. `нах беха меттигаш микрокӏошташ нах беха меттигаш нисйина нах беха меттигаш нах беха меттигаш микрокӏ...`
+2. `кӏоштан нах беха меттигаш нисйина нах беха меттигаш нах беха меттигаш кӏоштан нах беха меттигаш нах ...`
+3. `климат кхузахь климат барамехь континенталан йу аьхка йовха хуьлу ткъа ӏа барамехь шийла хуьлу шаран...`
+
+**Context Size 4:**
+
+1. `нийса лелаш ду сахьтан аса йу utc 3 билгалдахарш хьажоргаш неклиновскан кӏоштан индексаш кӏоштан нах...`
+2. `лелаш ду сахьтан аса йу utc 3 билгалдахарш хьажоргаш селижарован кӏоштан индексаш кӏоштан нах беха м...`
+3. `ду сахьтан аса йу utc 3 билгалдахарш хьажоргаш максатихан кӏоштан индексаш кӏоштан нах беха меттигаш...`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. геогр . — м . а беттанашкахь , цуьнан гуш болу седин атмосфера конвекцина дикка`
-2. `, йуккъера барам 2000 . изд . catherine b . surface properties of chicle extraction in`
-3. `— июль ( по зарубежным странам ) кӏеззиг къилбаседехьа хокана мотт буьйцуш долу ӏаьнцаклимат калужск...`
+1. `_/_циллан_olevia`
+2. `а_перерхашес._ба`
+3. `нилию._7959-со_к`
 
 **Context Size 2:**
 
-1. `. — екатеринбург : у - фактория , 2006 . альперович м . родригеса , м .`
-2. `категори : сербин нах беха меттигаш категори : мексикин нах беха меттигаш категори : витебскан облас...`
-3. `нах беха меттигаш категори : мексикин нах беха меттиг . географи . бахархойн дукхалла бахархойн дукх...`
+1. `а_койн_сахар_тӏуь`
+2. `._ре_нашкая_:_спу`
+3. `н_схойн_стр_штаме`
 
 **Context Size 3:**
 
-1. `( ) — российн федерацин вологдин областан междуреченскан кӏоштара дӏатесна эвла . бахархойн дукхалла...`
-2. `нах беха меттигаш категори : молдавин нах беха меттигаш категори : ацш гуонаш ru : калмен ( округ`
-3. `меттигаш категори : идальго штатан нах беха меттигаш категори : белхан категори категори : новосибир...`
+1. `ан_аркатерия_исти_`
+2. `_—_итан_новгорокӏо`
+3. `аш_беха_местник_гу`
 
 **Context Size 4:**
 
-1. `беха меттигаш категори : подлясьен воеводаллин нах беха меттигаш категори : абатца нисйина нах беха ...`
-2. `нах беха меттигаш категори : лаха калифорни штатан нах беха меттигаш категори : вилча жудецан коммун...`
-3. `. — м . : высшая школа , 2005 . — 463 с . — isbn 5060045196 . новая`
+1. `тан_кӏоштан_воеводс`
+2. `ахарш_хьажоргаш_нах`
+3. `._—_b.,_heidelberg,`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 93.6% predictability
+- **Best Predictability:** Context-4 (word) with 96.8% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (662,768 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (515,118 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -251,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 267,119 |
-| Total Tokens | 73,448,738 |
-| Mean Frequency | 274.97 |
+| Vocabulary Size | 230,774 |
+| Total Tokens | 54,539,322 |
+| Mean Frequency | 236.33 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 8220.95 |
+| Frequency Std Dev | 7087.98 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | а | 1,816,439 |
-| 2 | категори | 1,354,932 |
-| 3 | нах | 1,049,211 |
-| 4 | беха | 1,039,698 |
-| 5 | меттигаш | 968,759 |
-| 6 | йу | 814,168 |
-| 7 | м | 798,682 |
-| 8 | климат | 741,279 |
-| 9 | в | 737,093 |
-| 10 | билгалдахарш | 631,115 |
+| 1 | а | 1,429,788 |
+| 2 | нах | 929,389 |
+| 3 | беха | 927,412 |
+| 4 | меттигаш | 892,206 |
+| 5 | в | 665,820 |
+| 6 | климат | 663,481 |
+| 7 | м | 649,926 |
+| 8 | йу | 631,461 |
+| 9 | билгалдахарш | 595,304 |
+| 10 | с | 497,975 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | эмпачадо | 2 |
-| 2 | энано | 2 |
-| 3 | эскопетал | 2 |
-| 4 | эскриторио | 2 |
-| 5 | макариос | 2 |
-| 6 | эроика | 2 |
-| 7 | скирринг | 2 |
-| 8 | зигуинчор | 2 |
-| 9 | зигуиншор | 2 |
-| 10 | люксембургхо | 2 |
+| 1 | горушкинскан | 2 |
+| 2 | тулинскан | 2 |
+| 3 | долгопольскан | 2 |
+| 4 | погостищенскан | 2 |
+| 5 | кохановскан | 2 |
+| 6 | морховскан | 2 |
+| 7 | нежадовскан | 2 |
+| 8 | липиницкан | 2 |
+| 9 | зачепичи | 2 |
+| 10 | меетиг | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.8071 |
-| R² (Goodness of Fit) | 0.946340 |
+| Zipf Coefficient | 1.8318 |
+| R² (Goodness of Fit) | 0.964473 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 40.3% |
-| Top 1,000 | 81.6% |
-| Top 5,000 | 96.4% |
-| Top 10,000 | 97.6% |
+| Top 100 | 44.4% |
+| Top 1,000 | 86.7% |
+| Top 5,000 | 96.7% |
+| Top 10,000 | 97.7% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9463 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 40.3% of corpus
-- **Long Tail:** 257,119 words needed for remaining 2.4% coverage
+- **Zipf Compliance:** R²=0.9645 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 44.4% of corpus
+- **Long Tail:** 220,774 words needed for remaining 2.3% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -321,24 +384,129 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 105,624 | 32 | 6.269 | 1.405 | 0.8750 🏆 |
-| **mono_64d** | 105,624 | 64 | 6.426 | 0.986 | 0.8540 |
-| **mono_128d** | 105,624 | 128 | 6.612 | 0.771 | 0.7972 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8761 🏆 | 0.3710 | N/A | N/A |
+| **mono_64d** | 64 | 0.8520 | 0.3045 | N/A | N/A |
+| **mono_128d** | 128 | 0.7849 | 0.2825 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8750 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 105,624 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8761 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3193. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ка` | картографии, карайора, карпат |
+| `-ко` | количество, кочаны, кошехаблан |
+| `-ма` | майкен, маршаллвилл, машано |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-а` | ривица, валенсуэла, карайора |
+| `-о` | монтеморо, количество, мятнево |
+| `-н` | расистийн, майкен, тефран |
+| `-ан` | тефран, дмитрован, кертан |
+| `-во` | количество, мятнево, крайково |
+| `-ки` | исаковски, юридически, перлавки |
+| `-ово` | крайково, перегудово, дубново |
+| `-ка` | узника, кукушка, тлаика |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `архо` | 2.04x | 122 contexts | архон, тархо, лархо |
+| `галд` | 2.73x | 16 contexts | галдо, галда, угалде |
+| `ргаш` | 2.16x | 34 contexts | ургаш, бергаш, цер��аш |
+| `лгал` | 2.58x | 17 contexts | билгал, билгало, билгала |
+| `етти` | 1.89x | 42 contexts | бетти, нетти, меттин |
+| `харх` | 1.88x | 41 contexts | ахархо, вахарх, мухарх |
+| `халл` | 1.51x | 92 contexts | халла, халле, халль |
+| `ийла` | 1.86x | 35 contexts | кийла, шийла, мийла |
+| `игаш` | 2.25x | 18 contexts | бигаш, цигаш, книгаш |
+| `рхой` | 2.21x | 19 contexts | лархой, сурхой, сурхойн |
+| `ласт` | 1.59x | 60 contexts | пласт, ласта, селаст |
+| `ттиг` | 1.99x | 25 contexts | меттиг, гаттиг, ме́ттиг |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-ко` | `-а` | 51 words | королиха, кокориха |
+| `-ка` | `-а` | 43 words | карпеевка, камила |
+| `-ка` | `-о` | 38 words | картелево, катюшино |
+| `-ма` | `-а` | 35 words | машакепара, малакода |
+| `-ко` | `-о` | 33 words | косогорово, косяково |
+| `-ка` | `-н` | 31 words | калустовгӏеран, камблен |
+| `-ма` | `-н` | 24 words | малоярославецан, марьинкан |
+| `-ко` | `-н` | 23 words | коритен, койдин |
+| `-ма` | `-о` | 22 words | маторо, манкузо |
+| `-ко` | `-во` | 18 words | косогорово, косяково |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| полканово | **`полк-ан-ово`** | 6.0 | `полк` |
+| андрюшкино | **`андрюш-ки-но`** | 6.0 | `андрюш` |
+| зимовники | **`зимовни-ки`** | 4.5 | `зимовни` |
+| гринвичан | **`гринвич-ан`** | 4.5 | `гринвич` |
+| гуннбьёрнан | **`гуннбьёрн-ан`** | 4.5 | `гуннбьёрн` |
+| хромосоман | **`хромосом-ан`** | 4.5 | `хромосом` |
+| боьлкъазаран | **`боьлкъазар-ан`** | 4.5 | `боьлкъазар` |
+| хӏуманашна | **`хӏуманаш-на`** | 4.5 | `хӏуманаш` |
+| ынтымакан | **`ынтымак-ан`** | 4.5 | `ынтымак` |
+| бартолина | **`бартоли-на`** | 4.5 | `бартоли` |
+| судженскан | **`судженск-ан`** | 4.5 | `судженск` |
+| бузиновка | **`бузинов-ка`** | 4.5 | `бузинов` |
+| тракторашна | **`трактораш-на`** | 4.5 | `трактораш` |
+| пайхӏамаран | **`пайхӏамар-ан`** | 4.5 | `пайхӏамар` |
+| нуьрнберган | **`нуьрнберг-ан`** | 4.5 | `нуьрнберг` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language CE appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -346,11 +514,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (3.72x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (484) |
-| Markov | **Context-4** | Highest predictability (93.6%) |
+| Tokenizer | **64k BPE** | Best compression (3.78x) |
+| N-gram | **2-gram** | Lowest perplexity (423) |
+| Markov | **Context-4** | Highest predictability (96.8%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -540,7 +709,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -556,7 +726,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-28 17:05:28*
+*Report Date: 2026-01-03 10:17:57*

models/embeddings/monolingual/ce_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:18a8bbd0641b8970aab2c0ccf604502eba74b97f35c3606ad01acb009b142f02
-size 1134808955
+oid sha256:552e31d70a010dcf9ef87e857ff88199b6929bbb6fb3bcdccca9386585c7aa73
+size 1106869199

models/embeddings/monolingual/ce_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 105624
+  "vocab_size": 79041
 }

models/embeddings/monolingual/ce_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:44c015065e910726d15abdf60c4dd0e4462950b3f9c99e40c82a6bec99b9cec2
-size 285689723
+oid sha256:5102b3e58f419aba998033902e9953b5363500654754d4e63832e110651c49fa
+size 278165711

models/embeddings/monolingual/ce_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 105624
+  "vocab_size": 79041
 }

models/embeddings/monolingual/ce_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:160402c76b1bde05f4bbde11003b39939ec6f0e7a3c4e8d2d20b4d20e5810eed
-size 568729467
+oid sha256:78021fa809a68d2e2ee1a5da53c5c92c25afcb13d930f8165b5ca92116725dfd
+size 554400207

models/embeddings/monolingual/ce_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 105624
+  "vocab_size": 79041
 }

models/subword_markov/ce_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9cf69dd5043773e4a6ec0dd7fca8fb532d1def26240aa6214a05619b9f7a0320
-size 134168
+oid sha256:9b662cf799690d78b190708472b117520cd7cbcad0ac633b1286d43e5c79ae3a
+size 117929

models/subword_markov/ce_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "ce",
-  "unique_contexts": 1510,
-  "total_transitions": 552073024
+  "unique_contexts": 1553,
+  "total_transitions": 402142071
 }

models/subword_markov/ce_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9a6187b0fabb36d6b719c6ec916b358a15d63a341364e7852e2f3a23c5586073
-size 1098619
+oid sha256:c4dfc6fd0f57c29113d3b3fec461c0764a0522c46a13186880ed52b9d6958571
+size 872325

models/subword_markov/ce_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "ce",
-  "unique_contexts": 17227,
-  "total_transitions": 551384673
+  "unique_contexts": 14069,
+  "total_transitions": 401528698
 }

models/subword_markov/ce_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d252c42256fc6c0b6fce0d261303ba6de8609a4c175ccf6885164947f96bdded
-size 5724342
+oid sha256:c3414e9127a5344c044f38d2218042344b271b5e8396bc6d4d72233e2b2d1118
+size 4229687

models/subword_markov/ce_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "ce",
-  "unique_contexts": 133970,
-  "total_transitions": 550696322
+  "unique_contexts": 103627,
+  "total_transitions": 400915325
 }

models/subword_markov/ce_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e1e81c1dd821aba41445dadb821e7e3bcb207d9eb6e163013e5c9212255e4ccc
-size 19564690
+oid sha256:f130a9685a8aed9c6b457e355c462d4bb6bc97cd44b8ef634c042ea0133165d1
+size 15358547

models/subword_markov/ce_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "ce",
-  "unique_contexts": 662768,
-  "total_transitions": 550007971
+  "unique_contexts": 515118,
+  "total_transitions": 400301952
 }

models/subword_ngram/ce_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4a1697b51648dc86c9e845aafa0afbfc74dc00aa7028ba2b5d395142bf6f3640
-size 118388
+oid sha256:461d6692fb3cde3c1a7fd56b30b64912a0c140583d19fdeca1497422c4effcb1
+size 97261

models/subword_ngram/ce_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "ce",
-  "unique_ngrams": 7755,
-  "total_ngrams": 552073024
+  "unique_ngrams": 6176,
+  "total_ngrams": 402142071
 }

models/subword_ngram/ce_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4acca9406308437ca11a05cfcb287cb1fec43d90ddfdd2e54583391e67b0dcfe
-size 1012357
+oid sha256:58a82a7f67b113302603309f686885d9e1c8883bfbb49e51f83b18800eabd3e3
+size 816823

models/subword_ngram/ce_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "ce",
-  "unique_ngrams": 72318,
-  "total_ngrams": 551384673
+  "unique_ngrams": 58954,
+  "total_ngrams": 401528698
 }

models/subword_ngram/ce_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cde1675327291babdadf64b31e49b9f88ccfec8517d7a1de3c798508559e9140
-size 5464388
+oid sha256:47cefb2349ec2159ead2601f3247b831fb17076c78f2ad1df98bfeca85c04804
+size 4353133

models/subword_ngram/ce_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "ce",
-  "unique_ngrams": 422662,
-  "total_ngrams": 550696322
+  "unique_ngrams": 337533,
+  "total_ngrams": 400915325
 }

models/tokenizer/ce_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:693fef07fd91c952bd6dc391258ac00b72412b245cccf2273b8da2e74f98a918
-size 580690
+oid sha256:47044bfafe2471fb7dd149ae56b1ee71a3dc0dae2187dde2da97d70536d94302
+size 583986

models/tokenizer/ce_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7641da1e8ba9bb6ac932a554231606e62d54ac9ed4acdfcc75c032e584b726cf
-size 947341
+oid sha256:4dc3aa08231d203aaef76058c47ffd13abd5516006e009bda38a089e8f521043
+size 941717

models/tokenizer/ce_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dd5f0f5cb1ac9ddae9543ca026331e7095bfb49e92713ef6d2d1f2b1af81c700
-size 1695288
+oid sha256:b0ca3910cc123379fbdb7db536fdc659e073f98693c0cdc2028daa98f77221fe
+size 1671632

models/tokenizer/ce_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:61b5f6dfdef5fedd1877009ce07f66b91e1f97f11807dd468dcdb16a014824e2
-size 406333
+oid sha256:6f2a3488af9a58357efbe0e4a03cc1777791f582cd49bc1b578b466f3c6fe09e
+size 409035

models/vocabulary/ce_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:12b88a5312e7a67ba957dffeff9943c20553b754f8d99ae3bab9bc7925021712
-size 4165372
+oid sha256:53491c1be3ddba1539a45ab8004c1a5fe96048b27ea345ef740d3827b2c5eb4a
+size 3729004

models/vocabulary/ce_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "ce",
-  "vocabulary_size": 267119,
+  "vocabulary_size": 230774,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.00808419251496939,
+    "type_token_ratio": 0.009492890208497928,
     "coverage": {
-      "top_100": 0.4013568533353177,
-      "top_1000": 0.8125705130068827,
-      "top_5000": 0.9597777008352958,
-      "top_10000": 0.9715211792991832
+      "top_100": 0.4413509820362693,
+      "top_1000": 0.8620556765599773,
+      "top_5000": 0.9621095823063379,
+      "top_10000": 0.9714254859934246
     },
-    "hapax_count": 329317,
-    "hapax_ratio": 0.5521413865024914,
-    "total_documents": 688351
+    "hapax_count": 289712,
+    "hapax_ratio": 0.5566182375702708,
+    "total_documents": 613373
   }
 }

models/word_markov/ce_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:599dfc01d9395014b5691fd2f74145fcc4405443ba6655b4078f8756b54fe14c
-size 26132254
+oid sha256:c3bb6f9d89f6d5115c2926090a43cce4d79d1f587ec3a35363d32b6e7597bef4
+size 26676852

models/word_markov/ce_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "ce",
-  "unique_contexts": 596401,
-  "total_transitions": 106354951
+  "unique_contexts": 520111,
+  "total_transitions": 54215661
 }

models/word_markov/ce_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fb42589d1cb2a6e1ec32d0fb09f361272a03b0d53a6ddf4b788ff111a967fbd8
-size 63812094
+oid sha256:e50bac15d55d8209d7a94c5b47b626d9ec9efcfbc87ea034378579c8baae771a
+size 61192411

models/word_markov/ce_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "ce",
-  "unique_contexts": 2141469,
-  "total_transitions": 105666601
+  "unique_contexts": 2019671,
+  "total_transitions": 53602288
 }

models/word_markov/ce_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f8e2020efe22d041025528c0a6f3588a337e4ff44e09ea096401c37fdf145a05
-size 98196773
+oid sha256:d7731dcdedfda586ce23049d63d1cf35f363b73a5694fc5959c9acc5d96708db
+size 83617046

models/word_markov/ce_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "ce",
-  "unique_contexts": 3726034,
-  "total_transitions": 104978251
+  "unique_contexts": 2889994,
+  "total_transitions": 52988915
 }

models/word_markov/ce_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9d606c8d8d7711381abb87ee93407843ba0f735bcc2358cc37ee6bb64dc89a91
-size 127811549
+oid sha256:b7a571da9e93c548265c09ac2fa741dd64799c4eb59edfd177a5c2d1d6d9b405
+size 103083705

models/word_markov/ce_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "ce",
-  "unique_contexts": 4825259,
-  "total_transitions": 104289907
+  "unique_contexts": 3246178,
+  "total_transitions": 52375542
 }

models/word_ngram/ce_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5f013fabd46773d7120af2601c77cf0ab74f745ad856acbd7e9eee74a785ae73
-size 3471073
+oid sha256:7eec9750c3913795005af6c38ea29300fed66c9a6bf75815c3f9610aa60530e9
+size 2175399

models/word_ngram/ce_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "ce",
-  "unique_ngrams": 180710,
-  "total_ngrams": 106354951
+  "unique_ngrams": 100140,
+  "total_ngrams": 54215661
 }

models/word_ngram/ce_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:38e66219baeec769e5c5ad382c8dd648581653177e8a32ad53b8b8ba808b9c3c
-size 6715161
+oid sha256:237f7c39561c5eecec8a4b4fe0f5d823821231402041fe6b873a1bdb29e2ab83
+size 3958635

models/word_ngram/ce_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "ce",
-  "unique_ngrams": 322719,
-  "total_ngrams": 105666601
+  "unique_ngrams": 157541,
+  "total_ngrams": 53602288
 }

models/word_ngram/ce_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:918e1068a19861c1df2e2c44e0f5e49bde38b0e6eb1e0cdf4f44d0350bbee9a0
-size 15779066
+oid sha256:66b57f5089c75aef2f67e25c1358b26b0ace9b73aac63422694c1db35df26046
+size 9113237

models/word_ngram/ce_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "ce",
-  "unique_ngrams": 709202,
-  "total_ngrams": 104978251
+  "unique_ngrams": 330019,
+  "total_ngrams": 52988915
 }