Upload all models and assets for cv (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 2.848
+    value: 3.792
   - name: best_isotropy
     type: isotropy
-    value: 0.8113
+    value: 0.8332
   - name: vocabulary_size
     type: vocab
-    value: 199715
-generated: 2025-12-29
+    value: 0
+generated: 2026-01-03
 ---
 
 # CV - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,67 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 2.543x | 2.48 | 0.2499% | 342,143 |
-| **16k** | 2.668x | 2.60 | 0.2621% | 326,196 |
-| **32k** | 2.769x | 2.70 | 0.2721% | 314,244 |
-| **64k** | 2.848x 🏆 | 2.77 | 0.2798% | 305,594 |
+| **8k** | 3.084x | 3.09 | 0.2455% | 244,836 |
+| **16k** | 3.356x | 3.36 | 0.2672% | 224,964 |
+| **32k** | 3.590x | 3.60 | 0.2857% | 210,324 |
+| **64k** | 3.792x 🏆 | 3.80 | 0.3018% | 199,115 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Менантăр — ят:
- Менантăр гностик
- Менантăр грек камит çыравçи — авалхи грек драм...`
+**Sample 1:** `— Ăна хуплашкасене, тата пекех япаласене каясран сыхлама усă тата шалти Ĕç тата ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁мен ант ă р ▁— ▁ят : ▁мен ант ă ... (+35 more)` | 45 |
-| 16k | `▁мен ант ă р ▁— ▁ят : ▁мен ант ă ... (+34 more)` | 44 |
-| 32k | `▁мен ант ă р ▁— ▁ят : ▁мен ант ă ... (+34 more)` | 44 |
-| 64k | `▁мен ант ă р ▁— ▁ят : ▁мен ант ă ... (+32 more)` | 42 |
-
-**Sample 2:** `Пулса иртнĕ 
+| 8k | `▁— ▁ă на ▁хуп лаш ка сене , ▁тата ▁пекех ... (+24 more)` | 34 |
+| 16k | `▁— ▁ă на ▁хуп лаш ка сене , ▁тата ▁пекех ... (+24 more)` | 34 |
+| 32k | `▁— ▁ă на ▁хуп лашка сене , ▁тата ▁пекех ▁япаласене ... (+21 more)` | 31 |
+| 64k | `▁— ▁ă на ▁хуп лашка сене , ▁тата ▁пекех ▁япаласене ... (+20 more)` | 30 |
 
- Çуралнă 
-
- Вилнĕ 
-
-
-Категори:Çулсем`
+**Sample 2:** `Шатдорф — () — коммуна, Швейцарири варринче Ури кантонне çын код — сайт хулисем ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁пулса ▁иртн ĕ ▁ç уралн ă ▁вилн ĕ ▁категори : ... (+2 more)` | 12 |
-| 16k | `▁пулса ▁иртн ĕ ▁ç уралн ă ▁вилн ĕ ▁категори : ... (+2 more)` | 12 |
-| 32k | `▁пулса ▁иртн ĕ ▁ç уралн ă ▁вилн ĕ ▁категори : ... (+2 more)` | 12 |
-| 64k | `▁пулса ▁иртн ĕ ▁ç уралн ă ▁вилн ĕ ▁категори : ... (+2 more)` | 12 |
-
-**Sample 3:** `Пулса иртнĕ 
+| 8k | `▁ш ат дорф ▁— ▁() ▁— ▁коммуна , ▁швейцарири ▁варринче ... (+9 more)` | 19 |
+| 16k | `▁шат дорф ▁— ▁() ▁— ▁коммуна , ▁швейцарири ▁варринче ▁ури ... (+8 more)` | 18 |
+| 32k | `▁шат дорф ▁— ▁() ▁— ▁коммуна , ▁швейцарири ▁варринче ▁ури ... (+8 more)` | 18 |
+| 64k | `▁шат дорф ▁— ▁() ▁— ▁коммуна , ▁швейцарири ▁варринче ▁ури ... (+8 more)` | 18 |
 
- Çуралнă 
-
- Вилнĕ 
-
-
-Категори:Çулсем`
+**Sample 3:** `Çĕр йышшисем – пуклак Малти урисем кайри урисем ытларах çĕр иртет. тата пурӑнать...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁пулса ▁иртн ĕ ▁ç уралн ă ▁вилн ĕ ▁категори : ... (+2 more)` | 12 |
-| 16k | `▁пулса ▁иртн ĕ ▁ç уралн ă ▁вилн ĕ ▁категори : ... (+2 more)` | 12 |
-| 32k | `▁пулса ▁иртн ĕ ▁ç уралн ă ▁вилн ĕ ▁категори : ... (+2 more)` | 12 |
-| 64k | `▁пулса ▁иртн ĕ ▁ç уралн ă ▁вилн ĕ ▁категори : ... (+2 more)` | 12 |
+| 8k | `▁çĕ р ▁йышшисем ▁– ▁пу к лак ▁малти ▁у рисем ... (+30 more)` | 40 |
+| 16k | `▁çĕ р ▁йышшисем ▁– ▁пу к лак ▁малти ▁урисем ▁кайри ... (+27 more)` | 37 |
+| 32k | `▁çĕ р ▁йышшисем ▁– ▁пук лак ▁малти ▁урисем ▁кайри ▁урисем ... (+26 more)` | 36 |
+| 64k | `▁çĕ р ▁йышшисем ▁– ▁пуклак ▁малти ▁урисем ▁кайри ▁урисем ▁ытларах ... (+25 more)` | 35 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 2.848x compression
-- **Lowest UNK Rate:** 8k with 0.2499% unknown tokens
+- **Best Compression:** 64k achieves 3.792x compression
+- **Lowest UNK Rate:** 8k with 0.2455% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -137,57 +129,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 12,528 🏆 | 13.61 | 161,660 | 22.3% | 48.6% |
-| **2-gram** | 595 🏆 | 9.22 | 10,155 | 49.1% | 94.6% |
-| **3-gram** | 25,302 | 14.63 | 291,182 | 17.4% | 41.8% |
-| **3-gram** | 5,176 | 12.34 | 93,120 | 17.8% | 55.6% |
-| **4-gram** | 46,365 | 15.50 | 552,862 | 15.5% | 36.9% |
-| **4-gram** | 24,822 | 14.60 | 514,686 | 11.2% | 33.8% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 9,368 | 13.19 | 70,721 | 26.8% | 48.1% |
+| **2-gram** | Subword | 533 🏆 | 9.06 | 7,915 | 52.7% | 95.2% |
+| **3-gram** | Word | 8,221 | 13.01 | 88,916 | 30.4% | 52.4% |
+| **3-gram** | Subword | 4,932 | 12.27 | 69,443 | 17.2% | 56.3% |
+| **4-gram** | Word | 14,425 | 13.82 | 168,587 | 26.5% | 47.7% |
+| **4-gram** | Subword | 26,358 | 14.69 | 379,120 | 10.1% | 32.1% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `шыв шыв` | 22,909 |
+| 2 | `территоринчи юханшыв` | 14,353 |
+| 3 | `территорипе юхать` | 13,579 |
+| 4 | `юхса юханшыв` | 13,517 |
+| 5 | `экологи министерстви` | 11,703 |
+
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `категори :` | 123,149 |
-| 2 | `. —` | 57,640 |
-| 3 | `) —` | 36,291 |
-| 4 | `- мĕш` | 36,084 |
-| 5 | `асăрхавсем каçăсем` | 32,656 |
+| 1 | `рф экологи министерстви` | 11,700 |
+| 2 | `территорин шыв геоинформаци` | 11,389 |
+| 3 | `агентстви рф территорин` | 11,389 |
+| 4 | `рф территорин шыв` | 11,389 |
+| 5 | `федераци агентстви рф` | 11,389 |
 
-**3-grams:**
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `юхса кĕрет .` | 13,956 |
-| 2 | `территоринчи юханшыв .` | 13,818 |
-| 3 | `территорипе юхать .` | 13,577 |
-| 4 | `. юханшыв тăршшĕ` | 13,556 |
-| 5 | `, гт —` | 13,538 |
+| 1 | `шыв геоинформаци системин шыв` | 11,389 |
+| 2 | `федераци агентстви рф территорин` | 11,389 |
+| 3 | `агентстви рф территорин шыв` | 11,389 |
+| 4 | `территорин шыв геоинформаци системин` | 11,389 |
+| 5 | `рф территорин шыв геоинформаци` | 11,389 |
 
-**4-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `юхса кĕрет . юх��ншыв` | 13,500 |
-| 2 | `кĕрет . юханшыв тăршшĕ` | 13,498 |
-| 3 | `раççей территоринчи юханшыв .` | 11,932 |
-| 4 | `— раççей территоринчи юханшыв` | 11,928 |
-| 5 | `экологи министерстви категори :` | 11,420 |
+| 1 | `. _` | 462,300 |
+| 2 | `а _` | 400,655 |
+| 3 | `и _` | 362,557 |
+| 4 | `— _` | 343,684 |
+| 5 | `_ —` | 341,328 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ — _` | 340,403 |
+| 2 | `ш ы в` | 149,607 |
+| 3 | `ы в _` | 121,960 |
+| 4 | `_ ю х` | 94,722 |
+| 5 | `т е р` | 86,265 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ш ы в _` | 121,866 |
+| 2 | `_ ш ы в` | 85,504 |
+| 3 | `_ ю х а` | 76,923 |
+| 4 | `ю х а н` | 63,389 |
+| 5 | `х а н ш` | 63,293 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 595
+- **Best Perplexity:** 2-gram (subword) with 533
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~34% of corpus
+- **Coverage:** Top-1000 patterns cover ~32% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -195,55 +219,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.6552 | 1.575 | 4.99 | 506,656 | 34.5% |
-| **1** | 0.7089 | 1.635 | 7.09 | 3,778 | 29.1% |
-| **2** | 0.2525 | 1.191 | 1.75 | 2,525,380 | 74.8% |
-| **2** | 1.0046 | 2.006 | 7.14 | 26,748 | 0.0% |
-| **3** | 0.1067 | 1.077 | 1.24 | 4,399,352 | 89.3% |
-| **3** | 0.8645 | 1.821 | 4.65 | 190,734 | 13.5% |
-| **4** | 0.0530 🏆 | 1.037 | 1.11 | 5,463,793 | 94.7% |
-| **4** | 0.7014 🏆 | 1.626 | 3.12 | 886,786 | 29.9% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.7797 | 1.717 | 5.33 | 352,008 | 22.0% |
+| **1** | Subword | 0.6180 | 1.535 | 6.05 | 3,627 | 38.2% |
+| **2** | Word | 0.1825 | 1.135 | 1.40 | 1,864,001 | 81.8% |
+| **2** | Subword | 0.9070 | 1.875 | 6.21 | 21,896 | 9.3% |
+| **3** | Word | 0.0523 | 1.037 | 1.09 | 2,580,848 | 94.8% |
+| **3** | Subword | 0.8725 | 1.831 | 4.70 | 135,774 | 12.7% |
+| **4** | Word | 0.0222 🏆 | 1.016 | 1.04 | 2,780,564 | 97.8% |
+| **4** | Subword | 0.7090 | 1.635 | 3.14 | 637,656 | 29.1% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `шыв бассейн км2 экономика культура ял акимов анатолий тимофеевич разин остин пауэрс голдмембер найдж...`
+2. `юханшыв хура kara online куликовский п в сталин рузвельт черчилль в в карьере сампраса на экраны`
+3. `в о герард доу джонс в иван карабиц валерий валерьевич милици ричард iii вăл 15 pp`
+
+**Context Size 2:**
+
+1. `шыв шыв иртыш юханшыв бассейн шыв кара таз чиккинчен енисей чиккичен бассейн çук юханшыв кара обь ху...`
+2. `территоринчи юханшыв республики ростов ставрополь ен территорипе юхать юханшыва юхса юханшыв 33 км ю...`
+3. `территорипе юхать лелен еган юханшыва юхса юханшыв 7 500 км ытла b 15 калса çу çур çу`
 
-### Generated Text Samples
+**Context Size 3:**
+
+1. `федераци агентстви рф территорин шыв геоинформаци системин шыв шыв гидрологи гт бассейн том гт 08 гт...`
+2. `шыв шыв гидрологи гт бассейн том гт 11 гт 1 рф экологи министерстви республикин ао юпписем`
+3. `шыв федераци агентстви рф территорин шыв геоинформаци системин шыв шыв гидрологи бассейн том гт 15 г...`
+
+**Context Size 4:**
+
+1. `шыв геоинформаци системин шыв шыв гидрологи бассейн том гт 15 гт 3 рф экологи министерстви автономи ...`
+2. `рф территорин шыв геоинформаци системин шыв шыв гидрологи бассейн том 15 3 рф экологи министерстви а...`
+3. `федераци агентстви рф территорин шыв геоинформаци системин шыв шыв гидрологи гт бассейн том гт 03 гт...`
 
-Below are text samples generated from each Markov chain model:
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. мĕн пирки ăнлантаракан сăмахсем çынсен пурăнмаççĕ . джеймс гордон байрон / / / books ?`
-2. `, окато кочĕ ( 99 - мӗш ҫулсен повеҫӗн " кавказ тата катунь пĕрлешнинчен пуçланса хафель`
-3. `— сучава таран . версаль керменне йышӑнакан ен категори : патшалăх шыв ресурсĕсен федераци агентстви...`
+1. `_kruk=_neesmblot`
+2. `адов_ин_пчнфи_ое`
+3. `изра_,_ацин_—_je`
 
 **Context Size 2:**
 
-1. `категори : ирĕксĕрлевле ĕç`
-2. `. — м . : молодая гвардия » а . пугачёв ( главный редактор ) . —`
-3. `) — сибгоскино киностудире иван пырьев режиссёр 1949 çулта сĕрме купăс симфони оркестрĕнче чаплă сĕр...`
+1. `._кий,_фрах_(3_г.`
+2. `а_—_перрипе_влеор`
+3. `и_э.в._афизм_—_де`
 
 **Context Size 3:**
 
-1. `юхса кĕрет . юханшыв тăршшĕ 14 км . шыв реестрĕн хыпарĕсем раççей патшалăх шыв реестрĕн хыпарĕпе тур...`
-2. `территоринчи юханшыв . архангельск облаçĕ , вологда облаçĕ , киров облаçĕ , чулхула облаçĕ , мари эл...`
-3. `территорипе юхать . юханшыв шурă нил юханшыва юхса кĕрет . юханшыв тăршшĕ 14 км . шыв реестрĕн хыпар...`
+1. `_—_дев_анчах_тата_`
+2. `шыва_сайтра_нингсе`
+3. `ыв_5_-6_км._шыв_чу`
 
 **Context Size 4:**
 
-1. `юхса кĕрет . юханшыв тăршшĕ 12 км . шыв реестрĕн хыпарĕсем раççей патшалăх шыв реестрĕн хыпарĕпе тур...`
-2. `кĕрет . юханшыв тăршшĕ 10 км . шыв реестрĕн хыпарĕсем раççей патшалăх шыв реестрĕн хыпарĕпе чулман а...`
-3. `раççей территоринчи юханшыв . томск облаçĕ , омск облаçĕ , çĕн çĕпĕр облаçĕ территорипе юхать . юхан...`
+1. `шыв_федераци_систер`
+2. `_шыв_65_çын_—_--_гр`
+3. `_юханшыв_шыв_-_ката`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 94.7% predictability
+- **Best Predictability:** Context-4 (word) with 97.8% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (886,786 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (637,656 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -259,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 199,715 |
-| Total Tokens | 6,696,557 |
-| Mean Frequency | 33.53 |
+| Vocabulary Size | 148,629 |
+| Total Tokens | 3,880,417 |
+| Mean Frequency | 26.11 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 574.69 |
+| Frequency Std Dev | 438.75 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | категори | 123,245 |
-| 2 | шыв | 84,387 |
-| 3 | юханшыв | 53,784 |
-| 4 | в | 45,557 |
-| 5 | каçăсем | 43,777 |
-| 6 | асăрхавсем | 43,059 |
-| 7 | и | 41,660 |
-| 8 | с | 38,151 |
-| 9 | кочĕ | 36,802 |
-| 10 | мĕш | 36,148 |
+| 1 | шыв | 84,182 |
+| 2 | юханшыв | 53,747 |
+| 3 | в | 44,759 |
+| 4 | и | 40,900 |
+| 5 | с | 37,083 |
+| 6 | тата | 34,644 |
+| 7 | бассейн | 28,458 |
+| 8 | км | 25,026 |
+| 9 | м | 24,683 |
+| 10 | рф | 24,443 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | джизакской | 2 |
-| 2 | сардоба | 2 |
-| 3 | баяут | 2 |
-| 4 | хаваст | 2 |
-| 5 | сырдарьинской | 2 |
-| 6 | пайт | 2 |
-| 7 | клинов | 2 |
-| 8 | тавралăхсенчи | 2 |
-| 9 | савăтри | 2 |
-| 10 | кожевниковăн | 2 |
+| 1 | тӑтӑм | 2 |
+| 2 | мероприятире | 2 |
+| 3 | амфистрофпа | 2 |
+| 4 | ыйтрӗҫ | 2 |
+| 5 | поэмӑна | 2 |
+| 6 | хуламӑрта | 2 |
+| 7 | маршаленко | 2 |
+| 8 | шахмаметьев | 2 |
+| 9 | топилкин | 2 |
+| 10 | lupulella | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.0879 |
-| R² (Goodness of Fit) | 0.994444 |
+| Zipf Coefficient | 1.0392 |
+| R² (Goodness of Fit) | 0.997768 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 27.7% |
-| Top 1,000 | 56.9% |
-| Top 5,000 | 73.9% |
-| Top 10,000 | 80.2% |
+| Top 100 | 30.1% |
+| Top 1,000 | 56.1% |
+| Top 5,000 | 72.5% |
+| Top 10,000 | 79.1% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9944 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 27.7% of corpus
-- **Long Tail:** 189,715 words needed for remaining 19.8% coverage
+- **Zipf Compliance:** R²=0.9978 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 30.1% of corpus
+- **Long Tail:** 138,629 words needed for remaining 20.9% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -329,24 +384,114 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 100,175 | 32 | 4.637 | 1.327 | 0.8074 |
-| **mono_64d** | 100,175 | 64 | 5.201 | 1.153 | 0.8113 🏆 |
-| **mono_128d** | 100,175 | 128 | 5.955 | 0.960 | 0.8034 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8332 🏆 | 0.3551 | N/A | N/A |
+| **mono_64d** | 64 | 0.8307 | 0.2707 | N/A | N/A |
+| **mono_128d** | 128 | 0.8029 | 0.2175 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_64d with 0.8113 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 100,175 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8332 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2811. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
+
+---
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-а` | ыйтса, родства, мещерякова |
+| `-ен` | килнисен, центрӗнчен, теппермен |
+| `-ов` | камышлов, ярдов, горбов |
+| `-не` | мартинсоне, кёльне, характеристикине |
+| `-ем` | автомагистральсем, сооруженисем, алкалоидсем |
+| `-сем` | автомагистральсем, сооруженисем, алкалоидсем |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `олог` | 1.78x | 172 contexts | зоолог, эколог, геолог |
+| `сейн` | 2.68x | 24 contexts | хасейн, хусейн, басейн |
+| `рито` | 2.40x | 26 contexts | крито, ритон, барито |
+| `огра` | 1.72x | 94 contexts | богра, ногра��, ограды |
+| `ссей` | 2.68x | 17 contexts | эссей, ессей, рассей |
+| `ншыв` | 2.63x | 17 contexts | юханшыв, юшаншыв, юханшыве |
+| `ерри` | 2.35x | 22 contexts | черри, дерри, шерри |
+| `исте` | 1.74x | 58 contexts | листе, истеми, листер |
+| `орин` | 1.61x | 74 contexts | шорин, горин, борин |
+| `аншы` | 2.63x | 13 contexts | юханшыв, юшаншыв, юханшыве |
+| `блик` | 2.15x | 17 contexts | облик, облике, облика |
+| `нист` | 1.75x | 30 contexts | финист, горнист, хронист |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+*No significant affix co-occurrences detected.*
+
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| арӑслансем | **`арӑслан-сем`** | 4.5 | `арӑслан` |
+| минералов | **`минерал-ов`** | 4.5 | `минерал` |
+| эскимосов | **`эскимос-ов`** | 4.5 | `эскимос` |
+| динамикине | **`динамики-не`** | 4.5 | `динамики` |
+| учрежденийӗсем | **`учрежденийӗ-сем`** | 4.5 | `учрежденийӗ` |
+| председательне | **`председатель-не`** | 4.5 | `председатель` |
+| флоренцине | **`флоренци-не`** | 4.5 | `флоренци` |
+| вестготсем | **`вестгот-сем`** | 4.5 | `вестгот` |
+| королевине | **`королеви-не`** | 4.5 | `королеви` |
+| паракансем | **`паракан-сем`** | 4.5 | `паракан` |
+| приоритетне | **`приоритет-не`** | 4.5 | `приоритет` |
+| юханшывен | **`юханшыв-ен`** | 4.5 | `юханшыв` |
+| йӑмпӑлчӑксем | **`йӑмпӑлчӑк-сем`** | 4.5 | `йӑмпӑлчӑк` |
+| пайланусем | **`пайлану-сем`** | 4.5 | `пайлану` |
+| асапланнине | **`асапланни-не`** | 4.5 | `асапланни` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language CV appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
 
 ---
-## 6. Summary & Recommendations
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -354,11 +499,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (2.85x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (595) |
-| Markov | **Context-4** | Highest predictability (94.7%) |
+| Tokenizer | **64k BPE** | Best compression (3.79x) |
+| N-gram | **2-gram** | Lowest perplexity (533) |
+| Markov | **Context-4** | Highest predictability (97.8%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -548,7 +694,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -564,7 +711,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-29 05:54:56*
+*Report Date: 2026-01-03 11:00:29*

models/embeddings/monolingual/cv_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:122165bd998c13e515ab92279a98ff03e4cd43098e710bd14b31c8950d240e3a
-size 1128988019
+oid sha256:6cb5a0b37d9ab0f0662eb5bebfad6713c0f0648b4a42c5b0743b36462becfd73
+size 1096617351

models/embeddings/monolingual/cv_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 100175
+  "vocab_size": 69346
 }

models/embeddings/monolingual/cv_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d54eef8f2bca5d8f3651b8ea301648fca2a5e5ef21336502707042ba9ad12fc0
-size 284053619
+oid sha256:70007f295deadd64b95415e5970e57d83f6d92216140317dc2902789bf6da1de
+size 275359623

models/embeddings/monolingual/cv_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 100175
+  "vocab_size": 69346
 }

models/embeddings/monolingual/cv_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:30ff501911f1128dc0a166a8e6f0aa6432752eb6fbe532d9a03993fb7a065990
-size 565698419
+oid sha256:a5e77b5b9974a5a7347da624921ef4bd323fb01d62ffb62e50b9d2ca69a948e4
+size 549112199

models/embeddings/monolingual/cv_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 100175
+  "vocab_size": 69346
 }

models/subword_markov/cv_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5f262997702ae4fe69dddea8a2d0009fc3e028ed6514647cfd3f9daf65ee38f6
-size 202929
+oid sha256:e40043954b83a926cb69dce26f1f67caf4075d1babdc98a7a3543ca0afb6275b
+size 174799

models/subword_markov/cv_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "cv",
-  "unique_contexts": 3778,
-  "total_transitions": 52939202
+  "unique_contexts": 3627,
+  "total_transitions": 29032785
 }

models/subword_markov/cv_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:73e9f41b379aea703dc037b2b9c14f5fdb1e458ffe89e0c55c5b4898e746ea31
-size 1417424
+oid sha256:da927115de7638fb21f20a2ffe9e1afb50762a8897417d080a3fbfbb98fede29
+size 1108384

models/subword_markov/cv_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "cv",
-  "unique_contexts": 26748,
-  "total_transitions": 52880768
+  "unique_contexts": 21896,
+  "total_transitions": 28978297
 }

models/subword_markov/cv_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a37adb8b9dda333257f7c9c07d1325c750731940644ce454d0e2649b78d66ba0
-size 6647946
+oid sha256:8cc4344364830518084929051b4ae7fb28c1adcdaa163605965b74168fdbce0e
+size 4746572

models/subword_markov/cv_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "cv",
-  "unique_contexts": 190734,
-  "total_transitions": 52822334
+  "unique_contexts": 135774,
+  "total_transitions": 28923809
 }

models/subword_markov/cv_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:eacafd6dce111498b520e8cd4ff008e4fd51dfaf3af129bbd1a971fa2a4a2550
-size 23229636
+oid sha256:bbad63be2ef01e2c8ae1ec9ea81d1bd588420f175327541dc01a0c9c80d33097
+size 16746581

models/subword_markov/cv_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "cv",
-  "unique_contexts": 886786,
-  "total_transitions": 52763900
+  "unique_contexts": 637656,
+  "total_transitions": 28869321
 }

models/subword_ngram/cv_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f2eb7de10dcfd664e64fb2ed5a3f8e87857151d5db4a0906f5f07ae91e5e4d1b
-size 141995
+oid sha256:a6dc007ee19af6fa94d26edbb4ded18e4913986e603f56c0eb0a613928d49afa
+size 111624

models/subword_ngram/cv_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "cv",
-  "unique_ngrams": 10155,
-  "total_ngrams": 52939202
+  "unique_ngrams": 7915,
+  "total_ngrams": 29032785
 }

models/subword_ngram/cv_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:220228fd5844ae730c6423468f9bc01ad76fff9f93f689889460364d76e69c42
-size 1192485
+oid sha256:5a53062fcd868eabb3bedaef471787c15f34a1e4d7bd64541430cfad78fca8d3
+size 879358

models/subword_ngram/cv_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "cv",
-  "unique_ngrams": 93120,
-  "total_ngrams": 52880768
+  "unique_ngrams": 69443,
+  "total_ngrams": 28978297
 }

models/subword_ngram/cv_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a6f0d46671324cae8d4f25dab656dea1037a443bb92908b89b7ecb396d909247
-size 6662741
+oid sha256:8ec5a505fdfed8bff6da064dc01c77eb56f326b342a958243b49281b2add80c1
+size 4733643

models/subword_ngram/cv_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "cv",
-  "unique_ngrams": 514686,
-  "total_ngrams": 52822334
+  "unique_ngrams": 379120,
+  "total_ngrams": 28923809
 }

models/tokenizer/cv_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7c005264fa8df37712f5401075ffb496fafd8f41c4855d46e13ac98f7e580c85
-size 563200
+oid sha256:47ce826b88b912a0ee22e1317e6f0258e7665b2a8c708987a2abda3b35abceff
+size 564983

models/tokenizer/cv_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ccabecff2a596028046eec63bfd451e84d08ef52693a8c732698bbe026159824
-size 911335
+oid sha256:4f2de0762f3ba8f96fbc31f699f83e9b5dc454f42975ac96462ea30d55c9456f
+size 912506

models/tokenizer/cv_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:545717ff4e55b01c9d5a2d2eadd4fbf3891b10da47cd9648e76bc8d764c636ba
-size 1632840
+oid sha256:5877d8c09381815bf1acb251e4535e07c5b0157e1b804e3dd591e8925c8268d9
+size 1629877

models/tokenizer/cv_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:91efcdde0155224f2fdd035b6766a9005069d52a0793aa505284a22739d52d70
-size 397143
+oid sha256:64845d052561c0fd324adfbc8fd0d900a63c92141c555143df24fa74a9d7b46e
+size 398474

models/vocabulary/cv_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:610d2c91cbfca21df72eff0e6aaeddfa8d5010aa7306e9e55a301a3f5d59e869
-size 3482994
+oid sha256:fc7aca4a1d1d56f053bd0d6a123e4ef044a92c56295fcfe1be7fd13c9ca9e750
+size 2661487

models/vocabulary/cv_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "cv",
-  "vocabulary_size": 199715,
+  "vocabulary_size": 148629,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.0723688409440108,
+    "type_token_ratio": 0.08645308234335648,
     "coverage": {
-      "top_100": 0.2646904325162261,
-      "top_1000": 0.5436663325095297,
-      "top_5000": 0.7069258195428667,
-      "top_10000": 0.7670165631236444
+      "top_100": 0.28553243324755495,
+      "top_1000": 0.5333145325860036,
+      "top_5000": 0.6888109114837315,
+      "top_10000": 0.7510137225871395
     },
-    "hapax_count": 307134,
-    "hapax_ratio": 0.6059674577635549,
-    "total_documents": 58434
+    "hapax_count": 204527,
+    "hapax_ratio": 0.5791406630497571,
+    "total_documents": 54488
   }
 }

models/word_markov/cv_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f3353e0ea87710b1342d1f22f9f144ae7edf0f47681b78994ccede547b267389
-size 31571704
+oid sha256:c12896d7bb4321044e1d15a1f2c739138431e35a2c716c4bae85b032486c8b29
+size 23653393

models/word_markov/cv_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "cv",
-  "unique_contexts": 506656,
-  "total_transitions": 9821555
+  "unique_contexts": 352008,
+  "total_transitions": 4030456
 }

models/word_markov/cv_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1af075ec6a60aac75f50d3f9adf5baa713f51bb79a14579b3d8e1fc200dd8e75
-size 84795515
+oid sha256:04538a1a074396c3e2dba751a45663e0683f8dd3242406fc2c55ff9ab7c488ea
+size 58306430

models/word_markov/cv_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "cv",
-  "unique_contexts": 2525380,
-  "total_transitions": 9763121
+  "unique_contexts": 1864001,
+  "total_transitions": 3975968
 }

models/word_markov/cv_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:016ff6763f4bfbbd7e9b1d29de9699da3b2867257421a370710a2467cb579369
-size 126945531
+oid sha256:5f9bf1561d5e4c2f59aea4e5d0e22f1c5ed5c0fc363e90752c90f774d484528d
+size 75887845

models/word_markov/cv_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "cv",
-  "unique_contexts": 4399352,
-  "total_transitions": 9704872
+  "unique_contexts": 2580848,
+  "total_transitions": 3921480
 }

models/word_markov/cv_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4e2a48783822b8da73e476d87647a68ed50d3a7a73b879157618ecd49025627d
-size 158955552
+oid sha256:46742be5a48378178ea3dee073a78c91db68e460ddec4e29dc408eeb40652bff
+size 89451560

models/word_markov/cv_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "cv",
-  "unique_contexts": 5463793,
-  "total_transitions": 9646658
+  "unique_contexts": 2780564,
+  "total_transitions": 3866992
 }

models/word_ngram/cv_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a5f0565170f38be4c508175a9c7c94fb19581765987513539dd7c984ade03c7f
-size 3445658
+oid sha256:a9f93e996019df665eadd40faa1aeb735cc69ff047ccd9636f7e6917eb11c508
+size 1597795

models/word_ngram/cv_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "cv",
-  "unique_ngrams": 161660,
-  "total_ngrams": 9821555
+  "unique_ngrams": 70721,
+  "total_ngrams": 4030456
 }

models/word_ngram/cv_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5c930d09fb4c0999648c800182537a0aadf9e2ae705e29357f84fa9fde02b840
-size 6867525
+oid sha256:02ab91ffcb5ae6d2e65d564774c0f1a12dff2b9fed368fa06033c350612cfc18
+size 2351032

models/word_ngram/cv_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "cv",
-  "unique_ngrams": 291182,
-  "total_ngrams": 9763121
+  "unique_ngrams": 88916,
+  "total_ngrams": 3975968
 }

models/word_ngram/cv_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:151bd68aee3fb844a9e45d64ef599e3a26e0c051633db4f89e6c1269c13e6a6c
-size 13785247
+oid sha256:21500810e9ccbb52ea6248a13552dd1b49cde067a353175d757ae451148bca40
+size 4854879

models/word_ngram/cv_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "cv",
-  "unique_ngrams": 552862,
-  "total_ngrams": 9704872
+  "unique_ngrams": 168587,
+  "total_ngrams": 3921480
 }