Upload all models and assets for ab (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.203
+    value: 4.193
   - name: best_isotropy
     type: isotropy
-    value: 0.8443
+    value: 0.8185
   - name: vocabulary_size
     type: vocab
-    value: 34914
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # AB - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,59 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.211x | 3.15 | 0.1756% | 257,918 |
-| **16k** | 3.553x | 3.49 | 0.1943% | 233,133 |
-| **32k** | 3.880x | 3.81 | 0.2122% | 213,462 |
-| **64k** | 4.203x 🏆 | 4.13 | 0.2299% | 197,072 |
+| **8k** | 3.304x | 3.31 | 0.1494% | 223,505 |
+| **16k** | 3.652x | 3.66 | 0.1652% | 202,175 |
+| **32k** | 3.908x | 3.91 | 0.1768% | 188,952 |
+| **64k** | 4.193x 🏆 | 4.20 | 0.1897% | 176,103 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Ѫ, ѫ — кириллтәи аҩыратә архаикатә иажәхьоу нбан. 
-
-Азхьарԥшқәа
- Graphemica (Ѫ)
-...`
+**Sample 1:** `Ψ, ψ — бырзентәи аҩыратә нбан. Азхьарԥшқәа Graphemica (Ψ) Graphemica (ψ) аҩыратә...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ ѫ , ▁ ѫ ▁— ▁кириллтәи ▁аҩыратә ▁архаикатә ▁иажәхьоу ... (+11 more)` | 21 |
-| 16k | `▁ ѫ , ▁ ѫ ▁— ▁кириллтәи ▁аҩыратә ▁архаикатә ▁иажәхьоу ... (+11 more)` | 21 |
-| 32k | `▁ ѫ , ▁ ѫ ▁— ▁кириллтәи ▁аҩыратә ▁архаикатә ▁иажәхьоу ... (+11 more)` | 21 |
-| 64k | `▁ ѫ , ▁ ѫ ▁— ▁кириллтәи ▁аҩыратә ▁архаикатә ▁иажәхьоу ... (+11 more)` | 21 |
-
-**Sample 2:** `Аби́а () — ҵиаа. Ашәыр. Ашәырҵла.
+| 8k | `▁ ψ , ▁ ψ ▁— ▁бырзентәи ▁аҩыратә ▁нбан . ... (+11 more)` | 21 |
+| 16k | `▁ψ , ▁ψ ▁— ▁бырзентәи ▁аҩыратә ▁нбан . ▁азхьарԥшқәа ▁graphemica ... (+9 more)` | 19 |
+| 32k | `▁ψ , ▁ψ ▁— ▁бырзентәи ▁аҩыратә ▁нбан . ▁азхьарԥшқәа ▁graphemica ... (+9 more)` | 19 |
+| 64k | `▁ψ , ▁ψ ▁— ▁бырзентәи ▁аҩыратә ▁нбан . ▁азхьарԥшқәа ▁graphemica ... (+9 more)` | 19 |
 
-Ахьарԥшқәа
-
-б`
+**Sample 2:** `Амолдав бызшәа, амолдаван бызшәа (limba moldovenească, лимба молдовеняскэ) Азгәа...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁аби ́ а ▁() ▁— ▁ҵиаа . ▁ашәыр . ▁аш ... (+5 more)` | 15 |
-| 16k | `▁аби ́ а ▁() ▁— ▁ҵиаа . ▁ашәыр . ▁ашәырҵ ... (+4 more)` | 14 |
-| 32k | `▁аби ́а ▁() ▁— ▁ҵиаа . ▁ашәыр . ▁ашәырҵла . ... (+2 more)` | 12 |
-| 64k | `▁аби ́а ▁() ▁— ▁ҵиаа . ▁ашәыр . ▁ашәырҵла . ... (+2 more)` | 12 |
+| 8k | `▁ам ол да в ▁бызшәа , ▁ам ол да ван ... (+24 more)` | 34 |
+| 16k | `▁амол дав ▁бызшәа , ▁амол да ван ▁бызшәа ▁( l ... (+20 more)` | 30 |
+| 32k | `▁амолдав ▁бызшәа , ▁амол да ван ▁бызшәа ▁( l imba ... (+13 more)` | 23 |
+| 64k | `▁амолдав ▁бызшәа , ▁амолдаван ▁бызшәа ▁( limba ▁moldoveneasc ă , ... (+5 more)` | 15 |
 
-**Sample 3:** `Ҝ, ҝ — кириллтәи аҩыратә нбан.`
+**Sample 3:** `Ардешен () – Ҭырқтәыла ақалақь. Иаланхо . Ахьарԥшқәа ақалақьқәа`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ ҝ , ▁ ҝ ▁— ▁кириллтәи ▁аҩыратә ▁нбан .` | 10 |
-| 16k | `▁ ҝ , ▁ ҝ ▁— ▁кириллтәи ▁аҩыратә ▁нбан .` | 10 |
-| 32k | `▁ ҝ , ▁ ҝ ▁— ▁кириллтәи ▁аҩыратә ▁нбан .` | 10 |
-| 64k | `▁ ҝ , ▁ ҝ ▁— ▁кириллтәи ▁аҩыратә ▁нбан .` | 10 |
+| 8k | `▁ар деш ен ▁() ▁– ▁ҭырқтәыла ▁ақалақь . ▁иаланхо ▁. ... (+2 more)` | 12 |
+| 16k | `▁ар деш ен ▁() ▁– ▁ҭырқтәыла ▁ақалақь . ▁иаланхо ▁. ... (+2 more)` | 12 |
+| 32k | `▁ардешен ▁() ▁– ▁ҭырқтәыла ▁ақалақь . ▁иаланхо ▁. ▁ахьарԥшқәа ▁ақалақьқәа` | 10 |
+| 64k | `▁ардешен ▁() ▁– ▁ҭырқтәыла ▁ақалақь . ▁иаланхо ▁. ▁ахьарԥшқәа ▁ақалақьқәа` | 10 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.203x compression
-- **Lowest UNK Rate:** 8k with 0.1756% unknown tokens
+- **Best Compression:** 64k achieves 4.193x compression
+- **Lowest UNK Rate:** 8k with 0.1494% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -129,57 +129,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 2,750 🏆 | 11.43 | 13,494 | 35.3% | 57.9% |
-| **2-gram** | 464 🏆 | 8.86 | 5,850 | 56.1% | 94.4% |
-| **3-gram** | 2,460 | 11.26 | 16,782 | 38.6% | 56.9% |
-| **3-gram** | 3,385 | 11.72 | 40,776 | 25.5% | 64.3% |
-| **4-gram** | 3,267 | 11.67 | 27,732 | 37.4% | 51.5% |
-| **4-gram** | 13,192 | 13.69 | 145,474 | 16.1% | 43.3% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 724 | 9.50 | 5,814 | 51.4% | 72.0% |
+| **2-gram** | Subword | 363 | 8.50 | 4,104 | 60.3% | 96.8% |
+| **3-gram** | Word | 252 🏆 | 7.98 | 5,216 | 66.5% | 80.6% |
+| **3-gram** | Subword | 2,675 | 11.39 | 28,199 | 28.1% | 67.5% |
+| **4-gram** | Word | 342 | 8.42 | 9,800 | 64.0% | 74.0% |
+| **4-gram** | Subword | 11,090 | 13.44 | 112,541 | 16.8% | 44.7% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `акатегориа :` | 5,231 |
-| 2 | `рыԥсҭазаара иалҵит` | 3,971 |
-| 3 | `иит рыԥсҭазаара` | 3,938 |
-| 4 | `нанҳәамза цәыббрамза` | 3,601 |
-| 5 | `жәабранмза хәажәкырамза` | 3,601 |
+| 1 | `рыԥсҭазаара иалҵит` | 3,971 |
+| 2 | `иит рыԥсҭазаара` | 3,938 |
+| 3 | `рашәарамза ԥхынгәымза` | 3,603 |
+| 4 | `жәабранмза хәажәкырамза` | 3,603 |
+| 5 | `ажьырныҳәамза жәабранмза` | 3,602 |
 
-**3-grams:**
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
 | 1 | `иит рыԥсҭазаара иалҵит` | 3,938 |
-| 2 | `ажьырныҳәамза жәабранмза хәажәкырамза` | 3,601 |
-| 3 | `жьҭаарамза абҵарамза ԥхынҷкәынмза` | 3,601 |
-| 4 | `мшаԥымза лаҵарамза рашәарамза` | 3,601 |
-| 5 | `ԥхынгәымза нанҳәамза цәыббрамза` | 3,601 |
+| 2 | `цәыббрамза жьҭаарамза абҵарамза` | 3,602 |
+| 3 | `нанҳәамза цәыббрамза жьҭаарамза` | 3,601 |
+| 4 | `рашәарамза ԥхынгәымза нанҳәамза` | 3,601 |
+| 5 | `мшаԥымза лаҵарамза рашәарамза` | 3,601 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `мшаԥымза лаҵарамза рашәарамза ԥхынгәымза` | 3,601 |
+| 2 | `лаҵарамза рашәарамза ԥхынгәымза нанҳәамза` | 3,601 |
+| 3 | `рашәарамза ԥхынгәымза нанҳәамза цәыббрамза` | 3,601 |
+| 4 | `ԥхынгәымза нанҳәамза цәыббрамза жьҭаарамза` | 3,601 |
+| 5 | `нанҳәамза цәыббрамза жьҭаарамза абҵарамза` | 3,601 |
 
-**4-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `лаҵарамза рашәарамза ԥхынгәымза нанҳәамза` | 3,601 |
-| 2 | `рашәарамза ԥхынгәымза нанҳәамза цәыббрамза` | 3,601 |
-| 3 | `ахҭысқəа ажьырныҳәамза жәабранмза хәажәкырамза` | 3,601 |
-| 4 | `мшаԥымза лаҵарамза рашәарамза ԥхынгәымза` | 3,601 |
-| 5 | `ԥхынгәымза нанҳәамза цәыббрамза жьҭаарамза` | 3,601 |
+| 1 | `а _` | 154,707 |
+| 2 | `_ а` | 149,768 |
+| 3 | `р а` | 100,539 |
+| 4 | `а р` | 84,594 |
+| 5 | `ә а` | 75,990 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `а р а` | 50,267 |
+| 2 | `м з а` | 45,869 |
+| 3 | `з а _` | 44,868 |
+| 4 | `а _ а` | 35,436 |
+| 5 | `а м з` | 31,355 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `м з а _` | 44,436 |
+| 2 | `а м з а` | 30,784 |
+| 3 | `р а м з` | 22,744 |
+| 4 | `а р а _` | 19,480 |
+| 5 | `қ ә а _` | 17,506 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 464
+- **Best Perplexity:** 3-gram (word) with 252
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~43% of corpus
+- **Coverage:** Top-1000 patterns cover ~45% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -187,55 +219,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.5772 | 1.492 | 3.62 | 99,604 | 42.3% |
-| **1** | 1.5567 | 2.942 | 13.88 | 876 | 0.0% |
-| **2** | 0.1878 | 1.139 | 1.43 | 360,470 | 81.2% |
-| **2** | 1.2241 | 2.336 | 6.90 | 12,157 | 0.0% |
-| **3** | 0.0635 | 1.045 | 1.11 | 515,280 | 93.6% |
-| **3** | 0.7258 | 1.654 | 3.34 | 83,923 | 27.4% |
-| **4** | 0.0257 🏆 | 1.018 | 1.04 | 573,219 | 97.4% |
-| **4** | 0.4863 🏆 | 1.401 | 2.16 | 280,678 | 51.4% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.6658 | 1.586 | 3.61 | 90,583 | 33.4% |
+| **1** | Subword | 1.3373 | 2.527 | 10.82 | 872 | 0.0% |
+| **2** | Word | 0.1207 | 1.087 | 1.22 | 326,705 | 87.9% |
+| **2** | Subword | 1.0105 | 2.015 | 5.95 | 9,435 | 0.0% |
+| **3** | Word | 0.0295 | 1.021 | 1.04 | 396,742 | 97.1% |
+| **3** | Subword | 0.7768 | 1.713 | 3.69 | 56,063 | 22.3% |
+| **4** | Word | 0.0100 🏆 | 1.007 | 1.01 | 412,289 | 99.0% |
+| **4** | Subword | 0.5289 | 1.443 | 2.33 | 206,852 | 47.1% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `, аил - маклаи ихьӡ зху аҟәатәи аҳәынҭқарратә педагогтә институт . кёльн - рико ) ,`
-2. `. алитература ахырхарҭала . уи азҵаара азыҳәан қьалышь - 1528 ашықәсқәа рзы агазет « titus andronicu...`
-3. `- зшәышықәса агьама змоу акоуп азеипш гәабзиарахьчара аусхк аҿы ԥаҵаду ҳәа иашьҭан . акатегориа : в`
+1. `уи иалнаршеит абанкбжьаратә ҳасабкрақәа рҭыԥ инаркны шықәсанӡа наринџьованиархангельский а лазарев м...`
+2. `рыԥсҭазаара иалҵит шықәса нанҳәа 5 ԥхынҷкәы́н 22 27 азы иҭыҵит раԥхьа ақырҭуа ҭыԥҳа викторина ж ж`
+3. `иит рыԥсҭазаара иалҵит дидим халкентер абырзен бызшәала адраматургиа иақәшәаны ииашаҵәҟьаны еилкааӡа...`
 
 **Context Size 2:**
 
-1. `акатегориа : аԥсны аиҭагаҩцәа акатегориа : аҩада — атерриториа атерриториа – . ақалақьқәа ақалақь га...`
-2. `иит рыԥсҭазаара иалҵит : друз иулии цезарь – германики агриппинәи рԥа ( дыԥсит ? ? ) азхьарԥшқәа`
-3. `мшаԥымза лаҵарамза рашәарамза ԥхынгәымза нанҳә��мза цәыббрамза жьҭаарамза абҵарамза ԥхынҷкәынмза иит ...`
+1. `иит рыԥсҭазаара иалҵит клеопатра селена ii марк антонии клеопатреи antony and cleopatra кориолан cor...`
+2. `рашәарамза ԥхынгәымза нанҳәамза цәыббрамза жьҭаарамза абҵарамза ԥхынҷкәынмза иит рыԥсҭазаара иалҵит ...`
+3. `жәабранмза хәажәкырамза мшаԥымза лаҵарамза рашәарамза ԥхынгәымза нанҳәамза цәыббрамза жьҭаарамза абҵ...`
 
 **Context Size 3:**
 
-1. `мшаԥымза лаҵарамза рашәарамза ԥхынгәымза нанҳәамза цәыббрамза жьҭаарамза абҵарамза ԥхынҷкәынмза иит ...`
-2. `рашәарамза ԥхынгәымза нанҳәамза цәыббрамза жьҭаарамза абҵарамза ԥхынҷкәынмза иит рыԥсҭазаара иалҵит ...`
-3. `жьҭаарамза абҵарамза ԥхынҷкәынмза иит рыԥсҭазаара иалҵит арминии – германиатә херуски аимшьҭра рхада...`
+1. `цәыббрамза жьҭаарамза абҵарамза ԥхынҷкәынмза иит калигула римтәи аимператор дыԥсит 69 ш рыԥсҭазаара ...`
+2. `жьҭаарамза абҵарамза ԥхынҷкәынмза иит рыԥсҭазаара иалҵит аныҳәақәа араԥтә ар амш аҳәаахьчаҩцәа рамш ...`
+3. `ажьырныҳәамза жәабранмза хәажәкырамза мшаԥымза лаҵарамза рашәарамза ԥхынгәымза нанҳәамза цәыббрамза ...`
 
 **Context Size 4:**
 
-1. `мшаԥымза лаҵарамза рашәарамза ԥхынгәымза нанҳәамза цәыббрамза жьҭаарамза абҵарамза ԥхынҷкәынмза иит ...`
-2. `нанҳәамза цәыббрамза жьҭаарамза абҵарамза ԥхынҷкәынмза иит рыԥсҭазаара иалҵит арминии – германиатә х...`
-3. `цәыббрамза жьҭаарамза абҵарамза ԥхынҷкәынмза иит рыԥсҭазаара иалҵит арминии – германиатә херуски аим...`
+1. `нанҳәамза цәыббрамза жьҭаарамза абҵарамза ԥхынҷкәынмза иит рыԥсҭазаара иалҵит арминии германиатә хер...`
+2. `цәыббрамза жьҭаарамза абҵарамза ԥхынҷкәынмза иит рыԥсҭазаара иалҵит арминии германиатә херуски аимшь...`
+3. `рашәарамза ԥхынгәымза нанҳәамза цәыббрамза жьҭаарамза абҵарамза ԥхынҷкәынмза иит рыԥсҭазаара иалҵит ...`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `аил_даит.._аарԥа`
+2. `_дигьҭақәанбрббс`
+3. `иунмақә_ser_иала`
+
+**Context Size 2:**
+
+1. `а_иазара,_иа_мамҭ`
+2. `_ашықәаԥсҭақәа_из`
+3. `рамза_иционва,_30`
+
+**Context Size 3:**
+
+1. `ара_амшын._уи_аҩны`
+2. `мза_ԥхынгәымза_абе`
+3. `за_мшаԥысуа_кәу._а`
+
+**Context Size 4:**
+
+1. `мза_нанҳәамҭа._аҩаӡ`
+2. `амза_рашәара_мап_рц`
+3. `рамза_ԥхынгәы_9,_ш.`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 97.4% predictability
+- **Best Predictability:** Context-4 (word) with 99.0% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (280,678 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (206,852 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -251,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 34,914 |
-| Total Tokens | 483,415 |
-| Mean Frequency | 13.85 |
+| Vocabulary Size | 32,686 |
+| Total Tokens | 440,475 |
+| Mean Frequency | 13.48 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 106.12 |
+| Frequency Std Dev | 100.81 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | акатегориа | 5,263 |
-| 2 | уи | 4,164 |
-| 3 | рыԥсҭазаара | 4,025 |
-| 4 | иит | 3,987 |
-| 5 | иалҵит | 3,980 |
-| 6 | лаҵарамза | 3,888 |
-| 7 | жәабранмза | 3,837 |
-| 8 | хәажәкырамза | 3,833 |
-| 9 | ԥхынҷкәынмза | 3,805 |
-| 10 | абҵарамза | 3,804 |
+| 1 | уи | 4,159 |
+| 2 | рыԥсҭазаара | 4,025 |
+| 3 | иит | 3,987 |
+| 4 | иалҵит | 3,980 |
+| 5 | лаҵарамза | 3,752 |
+| 6 | жәабранмза | 3,722 |
+| 7 | хәажәкырамза | 3,701 |
+| 8 | абҵарамза | 3,701 |
+| 9 | нанҳәамза | 3,696 |
+| 10 | ԥхынҷкәынмза | 3,696 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | адрес | 2 |
-| 2 | extended | 2 |
-| 3 | stream | 2 |
-| 4 | block | 2 |
-| 5 | stru | 2 |
-| 6 | compressed | 2 |
-| 7 | draft | 2 |
-| 8 | preston | 2 |
-| 9 | видеохәмарроуп | 2 |
-| 10 | авидеохәмаррақәа | 2 |
+| 1 | slang | 2 |
+| 2 | пуэрто | 2 |
+| 3 | испантәи | 2 |
+| 4 | reggaetón | 2 |
+| 5 | маратон | 2 |
+| 6 | урымтәыла | 2 |
+| 7 | византии | 2 |
+| 8 | белобров | 2 |
+| 9 | длины | 2 |
+| 10 | акармара | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9724 |
-| R² (Goodness of Fit) | 0.994461 |
+| Zipf Coefficient | 0.9638 |
+| R² (Goodness of Fit) | 0.995375 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 30.1% |
-| Top 1,000 | 55.4% |
-| Top 5,000 | 76.6% |
-| Top 10,000 | 85.3% |
+| Top 100 | 30.3% |
+| Top 1,000 | 55.8% |
+| Top 5,000 | 76.9% |
+| Top 10,000 | 85.7% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9945 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 30.1% of corpus
-- **Long Tail:** 24,914 words needed for remaining 14.7% coverage
+- **Zipf Compliance:** R²=0.9954 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 30.3% of corpus
+- **Long Tail:** 22,686 words needed for remaining 14.3% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -321,24 +384,127 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 12,418 | 32 | 3.919 | 0.892 | 0.8443 🏆 |
-| **mono_64d** | 12,418 | 64 | 4.225 | 0.826 | 0.5913 |
-| **mono_128d** | 12,418 | 128 | 4.285 | 0.827 | 0.1726 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8185 🏆 | 0.3505 | N/A | N/A |
+| **mono_64d** | 64 | 0.6063 | 0.2895 | N/A | N/A |
+| **mono_128d** | 128 | 0.1746 | 0.2841 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8443 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 12,418 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8185 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3080. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-иа` | иаанхон, иаҳәшьа, иагьлыхьӡиҵоит |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-а` | азербайджана, рыҟамлара, аҭӡысахьақәа |
+| `-әа` | аҭ��ысахьақәа, льготақәа, ааимҭақәа |
+| `-қәа` | аҭӡысахьақәа, льготақәа, ааимҭақәа |
+| `-ит` | еиҭарҳәоит, иагьлыхьӡиҵоит, рыдиулоит |
+| `-ра` | рыҟамлара, архынҳәра, ахынҳәра |
+| `-еи` | аԥсреи, русқәеи, аглобализациеи |
+| `-тә` | антилопатә, аҳазылхратә, апрозатә |
+| `-ақәа` | аҭӡысахьақәа, льготақәа, ааимҭақәа |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `гыла` | 1.69x | 82 contexts | гылан, ргылан, игылаз |
+| `аҵар` | 1.72x | 38 contexts | аҵара, лаҵара, хаҵара |
+| `әыла` | 1.72x | 34 contexts | тәыла, атәыла, тәылан |
+| `ықәс` | 1.85x | 26 contexts | шықәс, шықәсы, шықәса |
+| `әара` | 1.47x | 58 contexts | шәара, акәара, аҟәара |
+| `арам` | 2.03x | 17 contexts | нарам, харам, арамка |
+| `қәса` | 2.02x | 16 contexts | шықәса, щықәса, жәшықәса |
+| `шәар` | 1.69x | 26 contexts | шәара, ршәарц, шәарах |
+| `ҭаза` | 2.47x | 8 contexts | иԥсҭазара, ԥсҭазаара, ипсҭазаара |
+| `азаа` | 1.69x | 23 contexts | лазаа, амазаап, иазааит |
+| `ыҳәа` | 1.62x | 22 contexts | ныҳәа, иныҳәа, ныҳәан |
+| `заар` | 2.08x | 10 contexts | акзаара, акзаарак, ракзаара |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-иа` | `-ит` | 97 words | иахьӡырҵеит, иаҭаауеит |
+| `-иа` | `-еит` | 54 words | иахьӡырҵеит, иаҭаауеит |
+| `-иа` | `-а` | 42 words | иалихуа, иадыруа |
+| `-иа` | `-әа` | 14 words | иаламҵакәа, иажәа |
+| `-иа` | `-ра` | 5 words | иабшьҭра, иааӡара |
+| `-иа` | `-тә` | 5 words | иаҳратә, иашахаҵаратә |
+| `-иа` | `-қәа` | 5 words | иашақәа, ианҵамҭақәа |
+| `-иа` | `-ақәа` | 4 words | иашақәа, ианҵамҭақәа |
+| `-иа` | `-еи` | 3 words | иашьцәеи, иамхреи |
+| `-иа` | `-әи` | 1 words | иапониатәи |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| аҳәынҭқарқәа | **`аҳәынҭқар-қәа`** | 4.5 | `аҳәынҭқар` |
+| мраҭашәаратә | **`мраҭаш-әа-ра-тә`** | 4.5 | `мраҭаш` |
+| ашьауӷатә | **`ашьауӷа-тә`** | 4.5 | `ашьауӷа` |
+| алахәылара | **`алахәыла-ра`** | 4.5 | `алахәыла` |
+| амитингқәа | **`амитинг-қәа`** | 4.5 | `амитинг` |
+| аметаллургиатә | **`аметаллургиа-тә`** | 4.5 | `аметаллургиа` |
+| рекологиатә | **`рекологиа-тә`** | 4.5 | `рекологиа` |
+| асиужетқәа | **`асиужет-қәа`** | 4.5 | `асиужет` |
+| аконцерттә | **`аконцерт-тә`** | 4.5 | `аконцерт` |
+| ипартиатә | **`ипартиа-тә`** | 4.5 | `ипартиа` |
+| афасадқәа | **`афасад-қәа`** | 4.5 | `афасад` |
+| астадионқәа | **`астадион-қәа`** | 4.5 | `астадион` |
+| аномерқәа | **`аномер-қәа`** | 4.5 | `аномер` |
+| аныҳәаратә | **`аныҳ-әа-ра-тә`** | 4.5 | `аныҳ` |
+| атерминқәа | **`атермин-қәа`** | 4.5 | `атермин` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language AB appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -346,11 +512,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.20x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (464) |
-| Markov | **Context-4** | Highest predictability (97.4%) |
+| Tokenizer | **64k BPE** | Best compression (4.19x) |
+| N-gram | **3-gram** | Lowest perplexity (252) |
+| Markov | **Context-4** | Highest predictability (99.0%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -540,7 +707,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -556,7 +724,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 04:31:24*
+*Report Date: 2026-01-03 05:05:55*

models/embeddings/monolingual/ab_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:70a5bb9264c0f53018d482c2616f76c0bf9699ff0b3215c919337610e18f88a9
-size 1037017106
+oid sha256:974accbf5e248437084213db34906d889d1387be4ba287f5bf3decc278094a76
+size 1036399852

models/embeddings/monolingual/ab_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 12418
+  "vocab_size": 11829
 }

models/embeddings/monolingual/ab_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f4040b5f28ef7e1b3cf96524fad4c6f4249e5107c0f78bdd6a3133c867ead2aa
-size 259480082
+oid sha256:20cc29bc8688764f66dd79168f5b38b9bc2c37d4cff74832ca2676787afff53e
+size 259315180

models/embeddings/monolingual/ab_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 12418
+  "vocab_size": 11829
 }

models/embeddings/monolingual/ab_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:59bba3121b8000d938989dc2a115057a7fd481c440ecd2c4af0c5f7711a53e1d
-size 518659090
+oid sha256:f6e336f851f9509b219587c555a83bc96b5e5f8af5ceca5f7c676a4c11e0eec7
+size 518343404

models/embeddings/monolingual/ab_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 12418
+  "vocab_size": 11829
 }

models/subword_markov/ab_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d5b0a600b840e3db1ee3806d9be588a8213f99ef3cce2c1cd544318a9585f986
-size 93410
+oid sha256:08d81996caede4f16305dff37bfaf00cc4aa080b329a05b6e813e43b32d4b5b1
+size 74300

models/subword_markov/ab_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "ab",
-  "unique_contexts": 876,
-  "total_transitions": 4575952
+  "unique_contexts": 872,
+  "total_transitions": 4086915
 }

models/subword_markov/ab_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3ea56c5cf197c6e7366129c2324fad6e8e710aca5379b47abfec22e6edd82fa6
-size 586863
+oid sha256:391adb8ed94471fe378fd39fa4bafc27bc22517c04a0688782555f658bfa5660
+size 430405

models/subword_markov/ab_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "ab",
-  "unique_contexts": 12157,
-  "total_transitions": 4569421
+  "unique_contexts": 9435,
+  "total_transitions": 4080892
 }

models/subword_markov/ab_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3d3626fdfb84a1b24e4b45c30124c54337aabdb6a8585a36031b40d4b2aea4c4
-size 2014070
+oid sha256:2554570f44dad84400a3a91e792ec596f90a065c3a673af002a820c70dca44e9
+size 1518240

models/subword_markov/ab_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "ab",
-  "unique_contexts": 83923,
-  "total_transitions": 4562890
+  "unique_contexts": 56063,
+  "total_transitions": 4074869
 }

models/subword_markov/ab_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:142c381e03e008f540daba013f253cbcc8bcf7cfe2a7bac360d368f678c4c000
-size 5510132
+oid sha256:a36f4483bb607e4182b1e8a367258cf1570a36ccf6debdfa0c5ed6d83eb6baa9
+size 4368269

models/subword_markov/ab_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "ab",
-  "unique_contexts": 280678,
-  "total_transitions": 4556359
+  "unique_contexts": 206852,
+  "total_transitions": 4068846
 }

models/subword_ngram/ab_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:25a3ad73144d6ab005e2dfb7deb33bbc6f3db63a36d07ff42172d7b2be73b608
-size 72236
+oid sha256:6546e93ab975088eb711841de0aac0eef95cedd64a0648eef8735695ed55ff88
+size 54917

models/subword_ngram/ab_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "ab",
-  "unique_ngrams": 5850,
-  "total_ngrams": 4575952
+  "unique_ngrams": 4104,
+  "total_ngrams": 4086915
 }

models/subword_ngram/ab_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e7da8578639b16b8512cca6705fe65f303df54fe9d9e019e89159e0e2b7f2ff5
-size 516357
+oid sha256:7b1b02523b1b148c8a70500352648cf3a5210acb9cff8bd3b8c613fbe55e72ef
+size 368926

models/subword_ngram/ab_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "ab",
-  "unique_ngrams": 40776,
-  "total_ngrams": 4569421
+  "unique_ngrams": 28199,
+  "total_ngrams": 4080892
 }

models/subword_ngram/ab_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3e898894c24a5ef426ee5c2cd23250aabe7e24d18ae03c6bbee902c09cf9ec33
-size 1774272
+oid sha256:d7eebb21889eb39845863d08c2bb3db59acb306cc610d9cff31719d1fccb8a9f
+size 1420721

models/subword_ngram/ab_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "ab",
-  "unique_ngrams": 145474,
-  "total_ngrams": 4562890
+  "unique_ngrams": 112541,
+  "total_ngrams": 4074869
 }

models/tokenizer/ab_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b92a09ffb9d9e16d3cfcd9d14625e37f4a5f64b96aa51c52d35d12a333ad633b
-size 582079
+oid sha256:5a2085566fc5290d88287b8216db2b990d022c905f7619c834e7a919406ca669
+size 569009

models/tokenizer/ab_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5030df6b09c2e503b39eb01d7fa0fccf197e83ae7fdb9fd7b16866de09bababe
-size 937553
+oid sha256:1e4a3fa167aa62fd56d1a517f1285f051a7587310895cdc82a80895e1bd8badc
+size 936940

models/tokenizer/ab_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:049440a566c70344456ff4c82202e7e102f4915ace284706ad7cc4c5fc73dba9
-size 1697167
+oid sha256:09581380da9729bf63f3885bf0e54ee06e9b0659cd517ad67eab83cf01150542
+size 1690301

models/tokenizer/ab_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:47045e15113d3f3907abf69824acfc10797a5b6e9942db1f6c5d4990df2dd043
-size 405395
+oid sha256:fb1da49fc5cbf61e37585d974f3bcb0262f7b8e403c201974781b65a1079fa28
+size 402830

models/vocabulary/ab_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1b85dbc4e6a7540ba1ff9dfa0ac21d279d0fce68b4180e8d6df6b29deb668b84
-size 683448
+oid sha256:1b9954423f516ffad7b6b395410e0a3ed33a13656770c69fcd1e37b9a2500496
+size 613460

models/vocabulary/ab_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "ab",
-  "vocabulary_size": 34914,
+  "vocabulary_size": 32686,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.18177207522936784,
+    "type_token_ratio": 0.18190225895758938,
     "coverage": {
-      "top_100": 0.2651855284354094,
-      "top_1000": 0.48867710079779325,
-      "top_5000": 0.6758529345765748,
-      "top_10000": 0.7518667778310897
+      "top_100": 0.26752598001845684,
+      "top_1000": 0.4927917987401196,
+      "top_5000": 0.6798940737471412,
+      "top_10000": 0.7575291899049071
     },
-    "hapax_count": 64722,
-    "hapax_ratio": 0.649584487534626,
-    "total_documents": 6531
+    "hapax_count": 57985,
+    "hapax_ratio": 0.639509876366203,
+    "total_documents": 6023
   }
 }

models/word_markov/ab_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b1bf752e561b655e36306643fcd5f9ee780835f3c8ab87825116a8028fcef88a
-size 4988023
+oid sha256:477c26b2510447d28ecaa37e0e7ab40a517773cfffb1210fbde670a44914bc00
+size 4536676

models/word_markov/ab_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "ab",
-  "unique_contexts": 99604,
-  "total_transitions": 715219
+  "unique_contexts": 90583,
+  "total_transitions": 492437
 }

models/word_markov/ab_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f00af198c4f86326bb4f959b22443089a350d189a2ac8d5415a26f3dcfb7f220
-size 10940272
+oid sha256:b4bc4862d6258993f1f724d0d5118dc5b0d53b173d1761b40efd9fab5f483159
+size 10110358

models/word_markov/ab_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "ab",
-  "unique_contexts": 360470,
-  "total_transitions": 708688
+  "unique_contexts": 326705,
+  "total_transitions": 486414
 }

models/word_markov/ab_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7bac1826a063cba6bbd8f61df21c4be2dba68dac0abfc311b80cd598e258a68e
-size 14657425
+oid sha256:fb37ec60704d2bebf6f001873a2a83cab37cde917c2788f8f44f143be73b3f5f
+size 12602543

models/word_markov/ab_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "ab",
-  "unique_contexts": 515280,
-  "total_transitions": 702160
+  "unique_contexts": 396742,
+  "total_transitions": 480391
 }

models/word_markov/ab_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:17958e76510075d2200f79a9911c3d1e9a3f4d0a87cd2a4cc0331afca431b300
-size 17354156
+oid sha256:811fb71775eaf00c7218d8a816f74bc26333e418e9d329068207739c393fd7bf
+size 14827550

models/word_markov/ab_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "ab",
-  "unique_contexts": 573219,
-  "total_transitions": 695636
+  "unique_contexts": 412289,
+  "total_transitions": 474369
 }

models/word_ngram/ab_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:acdfd16e121b263288a429d93a717f252969d4848648d80287275db5e16edc95
-size 282764
+oid sha256:d4a4d7d52cfdf902f9fe8cfe09802ecf37bacbd4cf5bf99b4545e9c58e1b6fe1
+size 148052

models/word_ngram/ab_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "ab",
-  "unique_ngrams": 13494,
-  "total_ngrams": 715219
+  "unique_ngrams": 5814,
+  "total_ngrams": 492437
 }

models/word_ngram/ab_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dab1f98b0cb7bd2d28f99f706169e76bd8a06a19b00b43438a7ba2c241b8d463
-size 389587
+oid sha256:3f22297b51acd3186bc5a9c808c852c94d515c15a5ae6556948ea6097484cb83
+size 162697

models/word_ngram/ab_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "ab",
-  "unique_ngrams": 16782,
-  "total_ngrams": 708688
+  "unique_ngrams": 5216,
+  "total_ngrams": 486414
 }

models/word_ngram/ab_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:592dd7484ba41a21c3e7a4f4fc8fc748b19346f94898e746c0776742ba664e4d
-size 699987
+oid sha256:ed909f3efc922de94c60f3fd686d519c1b57fb60acd95efc62e4dcd9179206f9
+size 334679

models/word_ngram/ab_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "ab",
-  "unique_ngrams": 27732,
-  "total_ngrams": 702160
+  "unique_ngrams": 9800,
+  "total_ngrams": 480391
 }