Upload all models and assets for cu (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.593
+    value: 4.945
   - name: best_isotropy
     type: isotropy
-    value: 0.3160
+    value: 0.2996
   - name: vocabulary_size
     type: vocab
-    value: 6898
-generated: 2025-12-29
+    value: 0
+generated: 2026-01-03
 ---
 
 # CU - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,54 +70,53 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.548x | 3.47 | 0.1382% | 136,059 |
-| **16k** | 3.988x | 3.90 | 0.1553% | 121,055 |
-| **32k** | 4.435x | 4.34 | 0.1727% | 108,853 |
-| **64k** | 4.593x 🏆 | 4.49 | 0.1789% | 105,095 |
+| **8k** | 3.879x | 3.88 | 0.1306% | 107,930 |
+| **16k** | 4.369x | 4.37 | 0.1472% | 95,813 |
+| **32k** | 4.945x 🏆 | 4.95 | 0.1666% | 84,659 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `thumb
-Ꙙ (имѧ ꙁатворѥнъ малъ юсъ или ѥнь) словѣньскаѥго ѩꙁꙑка боукꙑ ѥстъ
-
-Катигор...`
+**Sample 1:** `thumb Ꚛ (имѧ Ꙋкрестъ) словѣньскаѥго ѩꙁꙑка боукꙑ ѥстъ. ѩꙁꙑка боукъви`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁thumb ▁ ꙙ ▁( имѧ ▁ꙁа творѥ нъ ▁малъ ▁ю ... (+19 more)` | 29 |
-| 16k | `▁thumb ▁ ꙙ ▁( имѧ ▁ꙁа творѥ нъ ▁малъ ▁юсъ ... (+18 more)` | 28 |
-| 32k | `▁thumb ▁ꙙ ▁( имѧ ▁ꙁатворѥнъ ▁малъ ▁юсъ ▁или ▁ѥнь ) ... (+14 more)` | 24 |
-| 64k | `▁thumb ▁ꙙ ▁( имѧ ▁ꙁатворѥнъ ▁малъ ▁юсъ ▁или ▁ѥнь ) ... (+14 more)` | 24 |
+| 8k | `▁thumb ▁ ꚛ ▁( имѧ ▁ꙋ к ре стъ ) ... (+7 more)` | 17 |
+| 16k | `▁thumb ▁ ꚛ ▁( имѧ ▁ꙋ кре стъ ) ▁словѣньскаѥго ... (+6 more)` | 16 |
+| 32k | `▁thumb ▁ ꚛ ▁( имѧ ▁ꙋкрестъ ) ▁словѣньскаѥго ▁ѩꙁꙑка ▁боукꙑ ... (+4 more)` | 14 |
 
-**Sample 2:** `Илїѥ и  · ꙁнакъ He · аєрїо ѥстъ ⁙ Ѥгожє число въ пєрїодичьсцѣ сѷстимѣ 2 ѥстъ ⁙ А...`
+**Sample 2:** `Могилєвъ и · · градъ Бѣлꙑ Роуси ѥстъ ⁙ Людии обитаѥтъ 371 318 ⁙ Помѣновєнъ жє ꙁа...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁и лїѥ ▁и ▁· ▁ꙁнакъ ▁h e ▁· ▁ає рїо ... (+26 more)` | 36 |
-| 16k | `▁илїѥ ▁и ▁· ▁ꙁнакъ ▁he ▁· ▁ає рїо ▁ѥстъ ▁⁙ ... (+24 more)` | 34 |
-| 32k | `▁илїѥ ▁и ▁· ▁ꙁнакъ ▁he ▁· ▁аєрїо ▁ѥстъ ▁⁙ ▁ѥгожє ... (+23 more)` | 33 |
-| 64k | `▁илїѥ ▁и ▁· ▁ꙁнакъ ▁he ▁· ▁аєрїо ▁ѥстъ ▁⁙ ▁ѥгожє ... (+23 more)` | 33 |
+| 8k | `▁могилєвъ ▁и ▁· ▁· ▁градъ ▁бѣлꙑ ▁роуси ▁ѥстъ ▁⁙ ▁людии ... (+31 more)` | 41 |
+| 16k | `▁могилєвъ ▁и ▁· ▁· ▁градъ ▁бѣлꙑ ▁роуси ▁ѥстъ ▁⁙ ▁людии ... (+28 more)` | 38 |
+| 32k | `▁могилєвъ ▁и ▁· ▁· ▁градъ ▁бѣлꙑ ▁роуси ▁ѥстъ ▁⁙ ▁людии ... (+27 more)` | 37 |
 
-**Sample 3:** `Ха́сково () Блъгарі́ѩ Ха́сковьскꙑ области гла́вьнъ гра́дъ ѥ́стъ. Люди́и обита́ѥт...`
+**Sample 3:** `thumb Ѱ (имѧ ыпсьлон) словѣньскаѥго ѩꙁꙑка боукꙑ ѥстъ ѩꙁꙑка боукъви аꙁъбоукꙑ боук...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ха ́ ск ово ▁() ▁блъгарі́ѩ ▁ха ́ ск овьскꙑ ... (+14 more)` | 24 |
-| 16k | `▁ха́ск ово ▁() ▁блъгарі́ѩ ▁ха́ск овьскꙑ ▁области ▁гла́вьнъ ▁гра́дъ ▁ѥ́стъ ... (+10 more)` | 20 |
-| 32k | `▁ха́ск ово ▁() ▁блъгарі́ѩ ▁ха́ск овьскꙑ ▁области ▁гла́вьнъ ▁гра́дъ ▁ѥ́стъ ... (+10 more)` | 20 |
-| 64k | `▁ха́сково ▁() ▁блъгарі́ѩ ▁ха́сковьскꙑ ▁области ▁гла́вьнъ ▁гра́дъ ▁ѥ́стъ . ▁люди́и ... (+8 more)` | 18 |
+| 8k | `▁thumb ▁ѱ ▁( имѧ ▁ы п сь лон ) ▁словѣньскаѥго ... (+7 more)` | 17 |
+| 16k | `▁thumb ▁ѱ ▁( имѧ ▁ы п сь лон ) ▁словѣньскаѥго ... (+7 more)` | 17 |
+| 32k | `▁thumb ▁ѱ ▁( имѧ ▁ыпсьлон ) ▁словѣньскаѥго ▁ѩꙁꙑка ▁боукꙑ ▁ѥстъ ... (+4 more)` | 14 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.593x compression
-- **Lowest UNK Rate:** 8k with 0.1382% unknown tokens
+- **Best Compression:** 32k achieves 4.945x compression
+- **Lowest UNK Rate:** 8k with 0.1306% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -124,57 +125,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 1,147 🏆 | 10.16 | 2,704 | 35.7% | 76.6% |
-| **2-gram** | 544 🏆 | 9.09 | 3,037 | 52.6% | 93.9% |
-| **3-gram** | 1,920 | 10.91 | 3,713 | 28.5% | 66.1% |
-| **3-gram** | 3,142 | 11.62 | 14,844 | 24.1% | 63.9% |
-| **4-gram** | 3,217 | 11.65 | 6,299 | 23.2% | 54.0% |
-| **4-gram** | 9,292 | 13.18 | 39,612 | 16.1% | 43.7% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 803 | 9.65 | 1,418 | 38.7% | 88.8% |
+| **2-gram** | Subword | 451 🏆 | 8.82 | 2,626 | 56.3% | 95.5% |
+| **3-gram** | Word | 974 | 9.93 | 1,743 | 35.2% | 82.0% |
+| **3-gram** | Subword | 2,632 | 11.36 | 12,321 | 25.6% | 67.4% |
+| **4-gram** | Word | 1,602 | 10.65 | 2,970 | 29.2% | 66.7% |
+| **4-gram** | Subword | 8,242 | 13.01 | 33,307 | 16.1% | 45.2% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `катигорїꙗ :` | 1,702 |
-| 2 | `ѥстъ ⁙` | 1,213 |
-| 3 | `и ·` | 712 |
-| 4 | `ꙁьри такождє` | 432 |
-| 5 | `ѥстъ ·` | 367 |
+| 1 | `ꙁьри такождє` | 429 |
+| 2 | `людии обитаѥтъ` | 260 |
+| 3 | `ѥстъ людии` | 234 |
+| 4 | `градъ ѥстъ` | 230 |
+| 5 | `стольнъ градъ` | 186 |
 
-**3-grams:**
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `- 2 :` | 247 |
-| 2 | `3166 - 2` | 247 |
-| 3 | `⁙ людии обитаѥтъ` | 228 |
-| 4 | `ѥстъ ⁙ людии` | 226 |
-| 5 | `катигорїꙗ : повѣтъ` | 203 |
+| 1 | `ѥстъ людии обитаѥтъ` | 181 |
+| 2 | `въ дрьжавѣ бѣла` | 120 |
+| 3 | `дрьжавѣ бѣла роусь` | 120 |
+| 4 | `градъ ѥстъ людии` | 115 |
+| 5 | `роусь сѣи оудѣлъ` | 114 |
 
-**4-grams:**
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `3166 - 2 :` | 247 |
-| 2 | `ѥстъ ⁙ людии обитаѥтъ` | 173 |
-| 3 | `въ дрьжавѣ бѣла роусь` | 120 |
-| 4 | `ѥ ́ стъ ⁙` | 117 |
-| 5 | `оудѣлъ въ дрьжавѣ бѣла` | 114 |
+| 1 | `въ дрьжавѣ бѣла роусь` | 120 |
+| 2 | `ꙁємьскъ оудѣлъ въ дрьжавѣ` | 114 |
+| 3 | `оудѣлъ въ дрьжавѣ бѣла` | 114 |
+| 4 | `дрьжавѣ бѣла роусь сѣи` | 114 |
+| 5 | `ѥстъ ꙁємьскъ оудѣлъ въ` | 114 |
+
+**2-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ъ _` | 17,731 |
+| 2 | `и _` | 9,203 |
+| 3 | `а _` | 8,612 |
+| 4 | `с т` | 8,393 |
+| 5 | `_ с` | 6,604 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `т ъ _` | 5,941 |
+| 2 | `_ · _` | 4,423 |
+| 3 | `ь с к` | 3,904 |
+| 4 | `_ ⁙ _` | 3,096 |
+| 5 | `с т ъ` | 3,041 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ ѥ с т` | 2,898 |
+| 2 | `с т ъ _` | 2,880 |
+| 3 | `ѥ с т ъ` | 2,700 |
+| 4 | `ъ _ ⁙ _` | 1,900 |
+| 5 | `т ъ _ ⁙` | 1,811 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 544
+- **Best Perplexity:** 2-gram (subword) with 451
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~44% of corpus
+- **Coverage:** Top-1000 patterns cover ~45% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -182,55 +215,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.4597 | 1.375 | 2.91 | 20,872 | 54.0% |
-| **1** | 1.1094 | 2.158 | 8.69 | 991 | 0.0% |
-| **2** | 0.1521 | 1.111 | 1.33 | 60,436 | 84.8% |
-| **2** | 0.8922 | 1.856 | 4.49 | 8,609 | 10.8% |
-| **3** | 0.0670 | 1.048 | 1.12 | 80,298 | 93.3% |
-| **3** | 0.5843 | 1.499 | 2.46 | 38,630 | 41.6% |
-| **4** | 0.0320 🏆 | 1.022 | 1.05 | 89,220 | 96.8% |
-| **4** | 0.3376 🏆 | 1.264 | 1.67 | 95,077 | 66.2% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.4875 | 1.402 | 2.62 | 18,795 | 51.2% |
+| **1** | Subword | 0.9912 | 1.988 | 7.09 | 1,078 | 0.9% |
+| **2** | Word | 0.1229 | 1.089 | 1.22 | 48,721 | 87.7% |
+| **2** | Subword | 0.8197 | 1.765 | 4.19 | 7,639 | 18.0% |
+| **3** | Word | 0.0442 | 1.031 | 1.07 | 58,656 | 95.6% |
+| **3** | Subword | 0.5526 | 1.467 | 2.43 | 31,947 | 44.7% |
+| **4** | Word | 0.0206 🏆 | 1.014 | 1.03 | 61,545 | 97.9% |
+| **4** | Subword | 0.3393 | 1.265 | 1.70 | 77,657 | 66.1% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `· нарицаѥми словѣньскꙑ ѩꙁꙑкꙑ єѵрѡпѣ тъкъмо господь богъ ѥстъ ⁙ ꙁьданъ ѥстъ ⁙ ѥгда їѡаннъ сраѯимиръ`
-2. `⁙ бєꙁъ иꙁлѣвоу сѣмєнє ѥстъ a | | 33x22px тєѯасъ border 45x45px 39 эв − 1`
-3. `и ⁖ ( 466 , 342 оуранъ и 395 2 : бєрєстєйскаѧ · органїсма и ·`
+1. `и словѣньскъ ѩꙁꙑкъ ѥстъ людии обитаѥтъ стольнъ градъ ѥстъ ѥгожє потомъць тєодєнъ ꙗко идєжє kb постоꙗ...`
+2. `ѥстъ додєканъ ѫглѥводородьно вєщьство алканъ рѧдоу ѥстъ ꙗко нарочито поѩтиѥ паоуло коєлио пїитъ браꙁ...`
+3. `лѣта нарєчєнъ съ тꙑлоу жєнꙑ ѳєологїѩ вївлїи въ дрьжавѣ бѣла роусь сѣи оудѣлъ въ дрьжавѣ бѣла`
 
 **Context Size 2:**
 
-1. `катигорїꙗ : повѣтъ имаѥтъ оурѧдъ рѣкомъ сєльскъ съвѣтъ : катигорїꙗ : блъгарїꙗ катигорїꙗ : повѣтъ мѣн...`
-2. `ѥстъ ⁙ боукъвъ рѧдъ данъ съгласьно съ юникода осьмꙑимь ( 8 , 2 ꙁьри ́ та по`
-3. `и · ѯнѯѯ : ащє виждитє бингъ блѫдо · община єси блѫдописьплоцевы чын і яго апісаньне ·`
+1. `ꙁьри такождє владимѣръ мєждоусѣтии гради гради въ асии аꙁєрбаичаноѵ`
+2. `людии обитаѥтъ масачоусєтсѣ 7 лєѡдръ обитаѥтъ таджикистана дрьжавьнъ ѩꙁꙑкъ соуми ѥстъ симъ ѩꙁꙑкомъ 9...`
+3. `ѥстъ людии обитаѥтъ лѣта 788 лѣто 168 17 64 320 0 10 23 ꙁапражиѥиванофранковьска 13 9 13`
 
 **Context Size 3:**
 
-1. `3166 - 2 : mc 496mngmniso 3166 - 2 : qa 404kenkeiso 3166 - 2 : so 736sdnsdiso`
-2. `- 2 : fo 242fjifjiso 3166 - 2 : ml 581umiumiso 3166 - 2 : gg 300grcgriso 3166`
-3. `⁙ людии обитаѥтъ 1299 000 ⁙ єпїсимьнъ оукраиньскъ ѩꙁꙑкъ ѥстъ людиѥ лѣ ́ та ѥ ́ стъ ⁙`
+1. `ѥстъ людии обитаѥтъ 700 тꙑсѫщь основанъ ѥстъ лѣта нарєчєнъ градъ съ лѣта гєѡргїꙗ жє мьнитъ лꙑхнꙑ ꙗко...`
+2. `въ дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ ѡбласти рѣкома мѣньска ѡбласть повѣтъ имаѥтъ оурѧдъ рѣкомъ...`
+3. `дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ ѡбласти рѣкома бєрєстєиска ѡбласть повѣтъ имаѥтъ оурѧдъ рѣком...`
 
 **Context Size 4:**
 
-1. `3166 - 2 : tn 795tkmtmiso 3166 - 2 : lt 438lieliiso 3166 - 2 : nz 540nclnciso 3166`
-2. `ѥстъ ⁙ людии обитаѥтъ 55 997 ( 2011 ) " δείτε τη διοικητική διαίρεση " ꙁьри такождє катєрїни мєждоус...`
-3. `въ дрьжавѣ бѣла роусь : сѣи оудѣлъ бѣ члѣнъ ѡбласти · рѣкома витєбьска ѡбласть : повѣтъ имаѥтъ оурѧд...`
+1. `въ дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ ѡбласти рѣкома мѣньска ѡбласть конѣць иматъ оурѧдъ рѣкомъ ...`
+2. `ѥстъ ꙁємьскъ оудѣлъ въ дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ градоу витьбьскъ въ ѡбласти рѣкома вит...`
+3. `ꙁємьскъ оудѣлъ въ дрьжавѣ бѣла роусь сѣи оудѣлъ бѣ члѣнъ ѡбласти рѣкома бєрєстєиска ѡбласть повѣтъ и...`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_инъ_єньсєжапь_н`
+2. `а_вє_шємл҄итѣка_п`
+3. `о_стєрїтовоуспє_`
+
+**Context Size 2:**
+
+1. `ъ_ка_ѥстъ_бѣлꙗѥтъ`
+2. `и_·_тавѣ_коѩбр҄їꙗ:`
+3. `а_костомолїтарьно`
+
+**Context Size 3:**
+
+1. `тъ_словѣньскъ_ѥстъ`
+2. `_·_єпїсимь_40_грос`
+3. `ьскъвьсцѣ_на_оупи_`
+
+**Context Size 4:**
+
+1. `_ѥстъ_⁙_наи́бѫ́льша_г`
+2. `стъ_гоѵглъ_єси_и_8_`
+3. `ѥстъ_⁙_сѥго_ѩꙁꙑка_к`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 96.8% predictability
+- **Best Predictability:** Context-4 (word) with 97.9% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (95,077 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (77,657 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -246,64 +310,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 6,898 |
-| Total Tokens | 75,909 |
-| Mean Frequency | 11.00 |
+| Vocabulary Size | 6,213 |
+| Total Tokens | 63,034 |
+| Mean Frequency | 10.15 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 64.09 |
+| Frequency Std Dev | 60.04 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | и | 3,123 |
-| 2 | ѥстъ | 2,706 |
-| 3 | катигорїꙗ | 1,703 |
-| 4 | лѣта | 953 |
-| 5 | бѣ | 926 |
-| 6 | въ | 862 |
-| 7 | градъ | 792 |
-| 8 | ꙁьри | 569 |
-| 9 | такождє | 533 |
-| 10 | жє | 526 |
+| 1 | и | 2,825 |
+| 2 | ѥстъ | 2,697 |
+| 3 | лѣта | 958 |
+| 4 | бѣ | 912 |
+| 5 | въ | 843 |
+| 6 | градъ | 795 |
+| 7 | ꙁьри | 533 |
+| 8 | такождє | 529 |
+| 9 | жє | 512 |
+| 10 | людии | 470 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | катєгорїꙗ | 2 |
-| 2 | سخ | 2 |
-| 3 | هس | 2 |
-| 4 | ش | 2 |
-| 5 | ؤخخم | 2 |
-| 6 | خىث | 2 |
-| 7 | ىعةلاثق | 2 |
-| 8 | صشس | 2 |
-| 9 | пльсковьская | 2 |
-| 10 | маѭтъ | 2 |
+| 1 | статистичьского | 2 |
+| 2 | катєгорїꙗ | 2 |
+| 3 | سخ | 2 |
+| 4 | هس | 2 |
+| 5 | ش | 2 |
+| 6 | ؤخخم | 2 |
+| 7 | خىث | 2 |
+| 8 | ىعةلاثق | 2 |
+| 9 | صشس | 2 |
+| 10 | пльсковьская | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9646 |
-| R² (Goodness of Fit) | 0.987592 |
+| Zipf Coefficient | 0.9368 |
+| R² (Goodness of Fit) | 0.986351 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 39.8% |
-| Top 1,000 | 72.1% |
-| Top 5,000 | 95.0% |
+| Top 100 | 40.9% |
+| Top 1,000 | 72.7% |
+| Top 5,000 | 96.2% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9876 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 39.8% of corpus
-- **Long Tail:** -3,102 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9864 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 40.9% of corpus
+- **Long Tail:** -3,787 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -316,24 +380,128 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 2,259 | 32 | 3.128 | 0.714 | 0.3160 🏆 |
-| **mono_64d** | 2,259 | 64 | 3.031 | 0.702 | 0.0939 |
-| **mono_128d** | 2,259 | 128 | 3.024 | 0.720 | 0.0137 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.2996 🏆 | 0.4830 | N/A | N/A |
+| **mono_64d** | 64 | 0.0761 | 0.4499 | N/A | N/A |
+| **mono_128d** | 128 | 0.0111 | 0.4641 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.3160 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 2,259 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.2996 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.4657. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-пр` | правилъ, протєстантї́зма, прѣславъ |
+| `-по` | помꙑшлѥниѥ, послѣдни, польꙃєвати |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-ъ` | въсѣхъ, правилъ, кѷрїллъ |
+| `-къ` | арктїчьскъ, оучєникъ, оукъ |
+| `-ка` | владимѣрьска, банчьска, вльгоградьска |
+| `-нъ` | октадєканъ, ѥдьнѥнъ, дръжавьнъ |
+| `-ска` | владимѣрьска, банчьска, вльгоградьска |
+| `-скъ` | арктїчьскъ, лєниньскъ, въсточьнословѣньскъ |
+| `-кꙑ` | шавьльскꙑ, дрєвл҄ьнѥгрьчьскꙑ, аѵстрїискꙑ |
+| `-ьска` | владимѣрьска, банчьска, вльгоградьска |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `боук` | 1.84x | 14 contexts | боукꙑ, боуквꙑ, боукъвъ |
+| `ловѣ` | 1.56x | 18 contexts | словѣ, словѣнъ, словѣнє |
+| `слов` | 1.69x | 14 contexts | слово, слова, словѣ |
+| `ьжав` | 1.70x | 13 contexts | дрьжавѫ, дрьжавꙑ, дрьжавъ |
+| `ньск` | 1.60x | 15 contexts | мѣньска, жєньскъ, мѣньскъ |
+| `ласт` | 1.40x | 20 contexts | власти, властъ, власть |
+| `ьска` | 1.56x | 14 contexts | омьска, людьска, мѣньска |
+| `овѣн` | 1.77x | 9 contexts | словѣнъ, словѣнє, словѣнїꙗ |
+| `град` | 1.57x | 12 contexts | градъ, градѣ, гради |
+| `ьскъ` | 1.55x | 11 contexts | омьскъ, жєньскъ, томьскъ |
+| `блас` | 1.57x | 10 contexts | ѡбласти, область, ѡбласть |
+| `рьжа` | 1.62x | 9 contexts | дрьжавѫ, дрьжавꙑ, дрьжавъ |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-пр` | `-ъ` | 34 words | приморьскъ, природьнꙑхъ |
+| `-по` | `-ъ` | 34 words | польꙃоуѭтъ, полѫостровъ |
+| `-по` | `-нъ` | 11 words | подобьнъ, поушькинъ |
+| `-по` | `-тъ` | 7 words | польꙃоуѭтъ, польꙃоуѥтъ |
+| `-по` | `-къ` | 7 words | подъбрадъкъ, пол҄ьскъ |
+| `-по` | `-ка` | 7 words | политика, политическа |
+| `-пр` | `-къ` | 6 words | приморьскъ, прѣꙁъсибирьскъ |
+| `-по` | `-скъ` | 6 words | пол҄ьскъ, подольскъ |
+| `-пр` | `-нъ` | 6 words | природьнъ, прѡтонъ |
+| `-по` | `-ска` | 5 words | политическа, подъкарпатьска |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| самостоꙗтѣл҄ьнъ | **`самостоꙗтѣл҄ь-нъ`** | 4.5 | `самостоꙗтѣл҄ь` |
+| аѵстралїиска | **`аѵстралїи-ска`** | 4.5 | `аѵстралїи` |
+| аѵстрїиска | **`аѵстрїи-ска`** | 4.5 | `аѵстрїи` |
+| франкїиска | **`франкїи-ска`** | 4.5 | `франкїи` |
+| аѵстрїискъ | **`аѵстрїи-скъ`** | 4.5 | `аѵстрїи` |
+| сибирьскъ | **`сибирь-скъ`** | 4.5 | `сибирь` |
+| їталїискъ | **`їталїи-скъ`** | 4.5 | `їталїи` |
+| ꙗпѡнїискъ | **`ꙗпѡнїи-скъ`** | 4.5 | `ꙗпѡнїи` |
+| ꙗпѡнїиска | **`ꙗпѡнїи-ска`** | 4.5 | `ꙗпѡнїи` |
+| посєлєниѥ | **`по-сєлєниѥ`** | 4.5 | `сєлєниѥ` |
+| посєлєниꙗ | **`по-сєлєниꙗ`** | 4.5 | `сєлєниꙗ` |
+| поминаѭтъ | **`по-минаѭ-тъ`** | 3.0 | `минаѭ` |
+| подълѣсьскъ | **`по-дълѣсь-скъ`** | 3.0 | `дълѣсь` |
+| єѯадєканъ | **`єѯадє-ка-нъ`** | 3.0 | `єѯадє` |
+| политическꙑ | **`по-литиче-скꙑ`** | 3.0 | `литиче` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language CU appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -341,11 +509,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.59x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (544) |
-| Markov | **Context-4** | Highest predictability (96.8%) |
+| Tokenizer | **32k BPE** | Best compression (4.94x) |
+| N-gram | **2-gram** | Lowest perplexity (451) |
+| Markov | **Context-4** | Highest predictability (97.9%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -535,7 +704,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -551,7 +721,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-29 05:40:26*
+*Report Date: 2026-01-03 10:39:18*

models/embeddings/monolingual/cu_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9f8bf1a2cd1d39bc475e824c060c112d0eed6a4abce0f05f73e9b9ec5e60b68d
-size 1026363435
+oid sha256:8f180195153a0e4578f44c1f6b1c3eda4c3a7f0672314c1b3ce6aebed01755be
+size 1026177089

models/embeddings/monolingual/cu_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 2259
+  "vocab_size": 2082
 }

models/embeddings/monolingual/cu_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:161c4c5348c059b1ae0591e22cc433bc474df6bcce3838452936f16f3b5f7e60
-size 256628523
+oid sha256:f90429875eb75105fa8f12361ebaf97112dbdb9c2a3a585b6a0f5093c23fa4a9
+size 256578113

models/embeddings/monolingual/cu_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 2259
+  "vocab_size": 2082
 }

models/embeddings/monolingual/cu_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:15dc6b0abb2e7a3e27624c5b69ecba7dee2e3716811eadec27c745229453c1a3
-size 513206827
+oid sha256:13f1bd4d35a86f2cf0d70eb7c8c834bc4615d38220adae9405e7fddc748a894a
+size 513111105

models/embeddings/monolingual/cu_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 2259
+  "vocab_size": 2082
 }

models/subword_markov/cu_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:57ccc6b42dab1bf7c5a257dfb27195ea840f95be9970cc532f0210b5dd6387a8
-size 60348
+oid sha256:4e949a197cc604e7c761bd6bc85ce653be6b8e3a354f4cd10bcb179122c7c1bc
+size 61462

models/subword_markov/cu_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "cu",
-  "unique_contexts": 991,
-  "total_transitions": 603139
+  "unique_contexts": 1078,
+  "total_transitions": 515604
 }

models/subword_markov/cu_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ba074a4cc8fb5da21a6fcb219be60ee1499279188686c4396cd5598f80b568b1
-size 268803
+oid sha256:b941955b7b2c0d840400a334ec728d235f18de03f0b7d327b53a4c3fc2b0affd
+size 241073

models/subword_markov/cu_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "cu",
-  "unique_contexts": 8609,
-  "total_transitions": 601806
+  "unique_contexts": 7639,
+  "total_transitions": 514286
 }

models/subword_markov/cu_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6869f77e3f9c4db59fee37a52a6e162e414109135c62986a99c5ce684f227ed4
-size 793249
+oid sha256:f131478e09b8f63ad301796670764859e6dfa836f6b9d6ca0311e6ee4f91d06c
+size 678460

models/subword_markov/cu_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "cu",
-  "unique_contexts": 38630,
-  "total_transitions": 600473
+  "unique_contexts": 31947,
+  "total_transitions": 512968
 }

models/subword_markov/cu_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:19467222dd21df3f3e6e9fcdddc86c3b9372284a115fce02093145a8a8f5ffe9
-size 1565855
+oid sha256:56b15f422fd056498e6c88d7059108ebf1372e799ff8caeeefcadf86d5c18d69
+size 1344662

models/subword_markov/cu_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "cu",
-  "unique_contexts": 95077,
-  "total_transitions": 599140
+  "unique_contexts": 77657,
+  "total_transitions": 511650
 }

models/subword_ngram/cu_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:24e90f2a485f8f014b3627c50edfcd35f4212c928a8766692131bf13c7d96264
-size 37401
+oid sha256:2126e3213739aec87a6da0c10a5370c2aa9b1ac282cc16a13ff6bb7203679f6e
+size 33800

models/subword_ngram/cu_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "cu",
-  "unique_ngrams": 3037,
-  "total_ngrams": 603139
+  "unique_ngrams": 2626,
+  "total_ngrams": 515604
 }

models/subword_ngram/cu_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4046ccba23d8e946ef06d680c2b21d302abfd9d7e98db4ea628c00969f8eed2f
-size 188194
+oid sha256:b2cc8ff85bcc54e2024c90771fec3887648656c77e0464212ec94ebfeea2edc3
+size 160670

models/subword_ngram/cu_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "cu",
-  "unique_ngrams": 14844,
-  "total_ngrams": 601806
+  "unique_ngrams": 12321,
+  "total_ngrams": 514286
 }

models/subword_ngram/cu_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:74091d700f9ab56bc6c4bd2f3e7c9d6a775dcf465df00fb077e38b38cde258fd
-size 532101
+oid sha256:6e5f7ca57181d7dc4617a367f4ab643220e33820321439ffab3d419cf589a2a4
+size 464092

models/subword_ngram/cu_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "cu",
-  "unique_ngrams": 39612,
-  "total_ngrams": 600473
+  "unique_ngrams": 33307,
+  "total_ngrams": 512968
 }

models/tokenizer/cu_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cb448b26375611df94da44b847221dd5fdea5edce40f0ec9fe06e334c0174ff9
-size 572654
+oid sha256:97e6359a0f9b3b36740eb66281c9d74b4750ea93b6ae84d5b6666d1d0cb350c4
+size 574583

models/tokenizer/cu_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c96363f6098e71f315c123debcf72cd5c11d7ebf7ab46c5d09276e8d11266588
-size 936427
+oid sha256:64b3f6f09166812fbbb56c670818df981292963a9f019e1f0adfac21a18679da
+size 960296

models/tokenizer/cu_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bcb57ab736443d6385d346ab2f6748c3c0df07fc4448f236e0c042d182468009
-size 397450
+oid sha256:acc38a0fece8f1fa677a20fabe3d2d86630c9a3df52c07ec34dbad75382cb37a
+size 398366

models/vocabulary/cu_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e033bafb355a9f316216bd2db29d2167ca0da7874bc338ec478966432881af72
-size 120299
+oid sha256:430483667c6dfc4d8c34b266ab0e2f51a877989023eebb849030861107cfd909
+size 111841

models/vocabulary/cu_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "cu",
-  "vocabulary_size": 6898,
+  "vocabulary_size": 6213,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.23164024227309166,
+    "type_token_ratio": 0.24926341379629263,
     "coverage": {
-      "top_100": 0.33659926961788544,
-      "top_1000": 0.609757726908346,
-      "top_5000": 0.802897033936047,
-      "top_10000": 0.8796984947002762
+      "top_100": 0.34099647231360736,
+      "top_1000": 0.6052426440471944,
+      "top_5000": 0.8007715988214621,
+      "top_10000": 0.8828596720705009
     },
-    "hapax_count": 13907,
-    "hapax_ratio": 0.6684450853160298,
-    "total_documents": 1333
+    "hapax_count": 12653,
+    "hapax_ratio": 0.6706774090957278,
+    "total_documents": 1318
   }
 }

models/word_markov/cu_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:80450533f8c29713d84cca16915b30a007901f532ee608e9697cf178f9f62188
-size 747330
+oid sha256:aa5618d23ca23ed6f616602577c65e3e186ba372648b13a91d6915bcb3a1bca1
+size 679909

models/word_markov/cu_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "cu",
-  "unique_contexts": 20872,
-  "total_transitions": 115174
+  "unique_contexts": 18795,
+  "total_transitions": 74369
 }

models/word_markov/cu_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ae4559951b8aa0c3ca3c3345ca554ec48fe55b8532c66aaf8c99fe54219d6423
-size 1481222
+oid sha256:e53c547cb424d7129f42e0810a4886b3e115522fb5db795757ffaeff957e9df6
+size 1303006

models/word_markov/cu_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "cu",
-  "unique_contexts": 60436,
-  "total_transitions": 113841
+  "unique_contexts": 48721,
+  "total_transitions": 73051
 }

models/word_markov/cu_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e3c7c0e59a5e03eb30765d9ba9fbfb9c8fce86cb9d451e82c7ece47b86df8ad2
-size 1992856
+oid sha256:e1c4cf8e3dc9462e981f1ecec515a2d1249d9723320718db76503ee3b51ce571
+size 1658703

models/word_markov/cu_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "cu",
-  "unique_contexts": 80298,
-  "total_transitions": 112508
+  "unique_contexts": 58656,
+  "total_transitions": 71733
 }

models/word_markov/cu_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:41162062e37e77acb6090d0a4b86d87b321d4d3d19afb8620767688377b78741
-size 2393975
+oid sha256:1786989b68b43efad0cf9e8e8d222c1102102234836b14aabb943a1287d4f5bb
+size 1943027

models/word_markov/cu_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "cu",
-  "unique_contexts": 89220,
-  "total_transitions": 111176
+  "unique_contexts": 61545,
+  "total_transitions": 70415
 }

models/word_ngram/cu_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:189aa93a0dc65d6e6db6bae77afa9126cca9908f243661fe9cd8f4f99e639f4e
-size 52502
+oid sha256:019fe189687b07352dc7821236f9c5edb065885715366be8bad6fcd3fbc96f09
+size 33517

models/word_ngram/cu_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "cu",
-  "unique_ngrams": 2704,
-  "total_ngrams": 115174
+  "unique_ngrams": 1418,
+  "total_ngrams": 74369
 }

models/word_ngram/cu_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:54d7de5b539032d40f866f886c2dfa9c5f705b58962ecbb3c5552a7e12059e61
-size 86125
+oid sha256:30d2686e4c4da562a496eaa516690623f01aa672574a09a10e1dbc4c77f92f34
+size 49768

models/word_ngram/cu_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "cu",
-  "unique_ngrams": 3713,
-  "total_ngrams": 113841
+  "unique_ngrams": 1743,
+  "total_ngrams": 73051
 }

models/word_ngram/cu_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:75da47d27788f2dc037002883d57aec34e2b89f8c2929f4fcc5e33330133feca
-size 163706
+oid sha256:991aea88c68675eaf1bbe5a97bddb5818aa23dbfe762008c8d8ddfb58d7a40a9
+size 96377

models/word_ngram/cu_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "cu",
-  "unique_ngrams": 6299,
-  "total_ngrams": 112508
+  "unique_ngrams": 2970,
+  "total_ngrams": 71733
 }