Upload all models and assets for ba (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.068
+    value: 4.673
   - name: best_isotropy
     type: isotropy
-    value: 0.7712
+    value: 0.7751
   - name: vocabulary_size
     type: vocab
-    value: 417410
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # BA - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,54 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.248x | 3.21 | 0.2917% | 1,877,404 |
-| **16k** | 3.576x | 3.53 | 0.3212% | 1,705,077 |
-| **32k** | 3.852x | 3.81 | 0.3460% | 1,582,867 |
-| **64k** | 4.068x 🏆 | 4.02 | 0.3653% | 1,499,077 |
+| **8k** | 3.556x | 3.56 | 0.3956% | 1,547,491 |
+| **16k** | 3.995x | 4.00 | 0.4444% | 1,377,561 |
+| **32k** | 4.373x | 4.37 | 0.4864% | 1,258,583 |
+| **64k** | 4.673x 🏆 | 4.68 | 0.5198% | 1,177,657 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Сыңрау торна:
- Сыңрау торна (йыр) — башҡорт халыҡ йыры.
- Сыңрау торна — өс актлы...`
+**Sample 1:** `йыл — шишәмбе көнөнән башланған йыл, кәбисә түгел. Ваҡиғалар Тыуғандар Вафат бул...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁с ың рау ▁тор на : ▁с ың рау ▁тор ... (+23 more)` | 33 |
-| 16k | `▁сың рау ▁торна : ▁сың рау ▁торна ▁( йыр ) ... (+17 more)` | 27 |
-| 32k | `▁сың рау ▁торна : ▁сың рау ▁торна ▁( йыр ) ... (+16 more)` | 26 |
-| 64k | `▁сың рау ▁торна : ▁сың рау ▁торна ▁( йыр ) ... (+16 more)` | 26 |
+| 8k | `▁йыл ▁— ▁шиш әм бе ▁көнөнән ▁башланған ▁йыл , ▁кәбисә ... (+10 more)` | 20 |
+| 16k | `▁йыл ▁— ▁шиш әм бе ▁көнөнән ▁башланған ▁йыл , ▁кәбисә ... (+10 more)` | 20 |
+| 32k | `▁йыл ▁— ▁шишәмбе ▁көнөнән ▁башланған ▁йыл , ▁кәбисә ▁түгел . ... (+8 more)` | 18 |
+| 64k | `▁йыл ▁— ▁шишәмбе ▁көнөнән ▁башланған ▁йыл , ▁кәбисә ▁түгел . ... (+8 more)` | 18 |
 
-**Sample 2:** `Бөйөк БританияБөйөк Британия`
+**Sample 2:** `Азимут: Азимут — геодезияла бирелгән йүнәлеш менән төньяҡҡа табан булған йүнәлеш...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁бөйөк ▁британия бөйөк ▁британия` | 4 |
-| 16k | `▁бөйөк ▁британия бөйөк ▁британия` | 4 |
-| 32k | `▁бөйөк ▁британия бөйөк ▁британия` | 4 |
-| 64k | `▁бөйөк ▁британия бөйөк ▁британия` | 4 |
+| 8k | `▁аз им ут : ▁аз им ут ▁— ▁ге од ... (+29 more)` | 39 |
+| 16k | `▁аз им ут : ▁аз им ут ▁— ▁геод ез ... (+27 more)` | 37 |
+| 32k | `▁аз им ут : ▁аз им ут ▁— ▁геодез ияла ... (+23 more)` | 33 |
+| 64k | `▁азим ут : ▁азим ут ▁— ▁геодез ияла ▁бирелгән ▁йүнәлеш ... (+19 more)` | 29 |
 
-**Sample 3:** `Австралия — Көньяҡ ярымшарҙарҙа урынлашҡан дәүләт.
- Австралия (ҡитға) — Көнсығыш...`
+**Sample 3:** `Апанай мәсете ( ) — Ҡазан мәсете , татар архитектура культы ҡомартҡыһы. Ҡаҙанда ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁австр алия ▁— ▁көньяҡ ▁ярым шар ҙарҙа ▁урынлашҡан ▁дәүләт . ... (+18 more)` | 28 |
-| 16k | `▁австралия ▁— ▁көньяҡ ▁ярым шар ҙарҙа ▁урынлашҡан ▁дәүләт . ▁австралия ... (+15 more)` | 25 |
-| 32k | `▁австралия ▁— ▁көньяҡ ▁ярымшар ҙарҙа ▁урынлашҡан ▁дәүләт . ▁австралия ▁( ... (+12 more)` | 22 |
-| 64k | `▁австралия ▁— ▁көньяҡ ▁ярымшар ҙарҙа ▁урынлашҡан ▁дәүләт . ▁австралия ▁( ... (+11 more)` | 21 |
+| 8k | `▁ап ан ай ▁мәсете ▁( ▁) ▁— ▁ҡазан ▁мәсете ▁, ... (+18 more)` | 28 |
+| 16k | `▁ап ан ай ▁мәсете ▁( ▁) ▁— ▁ҡазан ▁мәсете ▁, ... (+16 more)` | 26 |
+| 32k | `▁ап ан ай ▁мәсете ▁( ▁) ▁— ▁ҡазан ▁мәсете ▁, ... (+15 more)` | 25 |
+| 64k | `▁ап ан ай ▁мәсете ▁( ▁) ▁— ▁ҡазан ▁мәсете ▁, ... (+14 more)` | 24 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.068x compression
-- **Lowest UNK Rate:** 8k with 0.2917% unknown tokens
+- **Best Compression:** 64k achieves 4.673x compression
+- **Lowest UNK Rate:** 8k with 0.3956% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -124,57 +129,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 46,308 🏆 | 15.50 | 552,281 | 13.1% | 33.2% |
-| **2-gram** | 586 🏆 | 9.20 | 17,419 | 48.7% | 94.9% |
-| **3-gram** | 107,842 | 16.72 | 1,025,675 | 10.6% | 27.4% |
-| **3-gram** | 5,171 | 12.34 | 163,683 | 17.5% | 55.2% |
-| **4-gram** | 184,106 | 17.49 | 1,793,503 | 10.9% | 25.9% |
-| **4-gram** | 26,442 | 14.69 | 914,560 | 9.8% | 31.4% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 56,525 | 15.79 | 433,408 | 13.8% | 30.4% |
+| **2-gram** | Subword | 489 🏆 | 8.93 | 13,769 | 52.3% | 96.8% |
+| **3-gram** | Word | 53,989 | 15.72 | 563,973 | 18.1% | 34.8% |
+| **3-gram** | Subword | 4,226 | 12.04 | 117,773 | 18.9% | 58.5% |
+| **4-gram** | Word | 61,817 | 15.92 | 883,766 | 19.4% | 36.8% |
+| **4-gram** | Subword | 21,528 | 14.39 | 687,383 | 10.2% | 33.2% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `категория :` | 197,952 |
-| 2 | `. —` | 100,523 |
-| 3 | `) .` | 79,614 |
-| 4 | `) —` | 77,384 |
-| 5 | `) ,` | 71,872 |
+| 1 | `гө буйынса` | 60,195 |
+| 2 | `һыу реестры` | 40,405 |
+| 3 | `дәүләт һыу` | 40,403 |
+| 4 | `йылға бассейны` | 40,327 |
+| 5 | `рәсәй федерацияһы` | 37,241 |
 
-**3-grams:**
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `% d0 %` | 38,175 |
-| 2 | `йылға бассейны —` | 29,475 |
-| 3 | `. а .` | 21,364 |
-| 4 | `йылғалары категория :` | 20,772 |
-| 5 | `һыу реестры мәғлүмәттәре` | 20,323 |
+| 1 | `һыу реестры мәғлүмәттәре` | 20,323 |
+| 2 | `дәүләт һыу реестры` | 20,208 |
+| 3 | `рәсәй дәүләт һыу` | 20,202 |
+| 4 | `дәүләт һыу реестрында` | 20,168 |
+| 5 | `реестры мәғлүмәттәре рәсәй` | 20,167 |
 
-**4-grams:**
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
 | 1 | `рәсәй дәүләт һыу реестры` | 20,195 |
-| 2 | `мәғлүмәттәре рәсәй дәүләт һыу` | 20,169 |
-| 3 | `реестры мәғлүмәттәре рәсәй дәүләт` | 20,169 |
-| 4 | `һыу реестры мәғлүмәттәре рәсәй` | 20,166 |
+| 2 | `реестры мәғлүмәттәре рәсәй дәүләт` | 20,167 |
+| 3 | `мәғлүмәттәре рәсәй дәүләт һыу` | 20,167 |
+| 4 | `һыу реестры мәғлүмәттәре рәсәй` | 20,164 |
 | 5 | `дәүләт һыу реестрында һыу` | 20,160 |
 
+**2-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `а _` | 2,396,936 |
+| 2 | `а р` | 2,197,072 |
+| 3 | `ы _` | 2,104,654 |
+| 4 | `_ б` | 2,010,552 |
+| 5 | `а н` | 1,869,683 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ й ы` | 756,503 |
+| 2 | `й ы л` | 745,794 |
+| 3 | `н д а` | 679,041 |
+| 4 | `а н _` | 653,833 |
+| 5 | `ы ң _` | 648,174 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ й ы л` | 708,824 |
+| 2 | `ы н д а` | 469,174 |
+| 3 | `_ һ ә м` | 442,529 |
+| 4 | `һ ә м _` | 440,639 |
+| 5 | `н д а _` | 409,349 |
+
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 586
+- **Best Perplexity:** 2-gram (subword) with 489
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~31% of corpus
+- **Coverage:** Top-1000 patterns cover ~33% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -182,55 +219,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.7118 | 1.638 | 7.12 | 1,081,802 | 28.8% |
-| **1** | 1.2737 | 2.418 | 9.88 | 4,928 | 0.0% |
-| **2** | 0.3181 | 1.247 | 2.06 | 7,695,897 | 68.2% |
-| **2** | 0.9811 | 1.974 | 7.09 | 48,690 | 1.9% |
-| **3** | 0.1344 | 1.098 | 1.31 | 15,852,609 | 86.6% |
-| **3** | 0.8666 | 1.823 | 4.73 | 344,963 | 13.3% |
-| **4** | 0.0616 🏆 | 1.044 | 1.12 | 20,837,559 | 93.8% |
-| **4** | 0.6873 🏆 | 1.610 | 3.25 | 1,631,641 | 31.3% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.8998 | 1.866 | 8.99 | 915,102 | 10.0% |
+| **1** | Subword | 0.9916 | 1.988 | 7.48 | 5,664 | 0.8% |
+| **2** | Word | 0.2745 | 1.210 | 1.74 | 8,225,491 | 72.6% |
+| **2** | Subword | 0.8603 | 1.815 | 5.91 | 42,359 | 14.0% |
+| **3** | Word | 0.0884 | 1.063 | 1.17 | 14,302,544 | 91.2% |
+| **3** | Subword | 0.8235 | 1.770 | 4.71 | 250,210 | 17.6% |
+| **4** | Word | 0.0321 🏆 | 1.022 | 1.05 | 16,653,317 | 96.8% |
+| **4** | Subword | 0.7025 | 1.627 | 3.37 | 1,177,702 | 29.8% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `һәм инәйҙәре тәрбиәләп үҫтергәндәр улы сәғитов м стрельникова с григорьев а а преображенский верфенд...`
+2. `буйынса ла бүлә көньяҡ диалекты там где плещется форель фильм үҙенең ҡатнашыуын ылыҡтыра йылда саҡыр...`
+3. `һыу реестры мәғлүмәте буйынса асыш кубогын еңә йылдан гидромеханизация горных породах и любовь шевцо...`
+
+**Context Size 2:**
+
+1. `гө буйынса һаны номеры 15 гө буйынса коды бассейн коды гө буйынса һаны номеры 03 гө буйынса`
+2. `һыу реестры мәғлүмәте буйынса дәүләт һыу реестрында һыу объектының коды гидрологик өйрәнеү гө буйынс...`
+3. `дәүләт һыу реестры мәғлүмәттәре рәсәй дәүләт һыу реестрында һыу объектының коды гидрологик өйрәнеү г...`
+
+**Context Size 3:**
+
+1. `һыу реестры мәғлүмәттәре рәсәй дәүләт һыу реестры мәғлүмәте буйынса йылға түбәнге обь һыу бассейны о...`
+2. `дәүләт һыу реестры мәғлүмәте буйынса йылға кама һыу бассейны округында урынлашҡан һыу хужалығы участ...`
+3. `рәсәй дәүләт һыу реестры мәғлүмәте буйынса йылға кама һыу бассейны округында урынлашҡан һыу хужалығы...`
+
+**Context Size 4:**
+
+1. `рәсәй дәүләт һыу реестры мәғлүмәте буйынса йылға кубань һыу бассейны округында урынлашҡан һыу хужалы...`
+2. `реестры мәғлүмәттәре рәсәй дәүләт һыу реестры мәғлүмәте буйынса йылға көнбыйыш каспий һыу бассейны о...`
+3. `мәғлүмәттәре рәсәй дәүләт һыу реестры мәғлүмәте буйынса йылға иртыш һыу бассейны округында урынлашҡа...`
 
-### Generated Text Samples
 
-Below are text samples generated from each Markov chain model:
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. а . һыу бассейны ) . — шул таштар ҡыҙһа , 1998 ) — 0`
-2. `, суданды иҫәпләмәйенсә ) , 1978 йылда 768 ( номеры ) 9 октябрь 1918 йылдан административ`
-3. `— 28 тайфун ) , табак магнаты , 2006 ) — 13 ғинуарында бәләкәй йылға двина`
+1. `_бә_бемиәкәл)_өх`
+2. `ацине_аҡъя_тенан`
+3. `радъеле,_бесеср_`
 
 **Context Size 2:**
 
-1. `категория : ҡабарҙы - балҡар йылғалары категория : алфавит буйынса шәхестәр категория : рәсәй субъек...`
-2. `. — мәскәү 762 « сапсан » санкт - петербург собор майҙаны ансамбле , солист сифатында саҡыралар`
-3. `) . памятный знак на месте ќ , яңғырау диапозоны киң ( 30 сентябрь 1960 йыл ,`
+1. `а_тетыға_олкәр_ре`
+2. `ар,_двинфүргеҙмәт`
+3. `ы_былдағыный_мәһе`
 
 **Context Size 3:**
 
-1. `% d0 % b5 % d1 % 83 % d0 % b0 % d1 % 86 % d1`
-2. `йылға бассейны — печора һәм обь йылғалары араһындағы , баренц диңгеҙенә ҡойоусы , йылғалар бассейны ...`
-3. `. а . токарев тәҡдим итәләр . немецтарҙы аптыратып , сталин документтар өсөн түләргә ризалаша . әзер...`
+1. `_йылға_владионерҙә`
+2. `йылға_бүләт_ил_ажн`
+3. `нда_алек_тамблем,_`
 
 **Context Size 4:**
 
-1. `рәсәй дәүләт һыу реестры мәғлүмәте буйынса йылға үрге обь һыу бассейны округында урынлашҡан , һыу ху...`
-2. `реестры мәғлүмәттәре рәсәй дәүләт һыу реестры мәғлүмәте буйынса йылға үрге обь һыу бассейны округынд...`
-3. `мәғлүмәттәре рәсәй дәүләт һыу реестры мәғлүмәте буйынса йылға түбәнге волга һыу бассейны округында у...`
+1. `_йылған_күпкә_ҡушыл`
+2. `ында_ҡаршы_ҡустың_ү`
+3. `_һәм_төрлө_метрында`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 93.8% predictability
+- **Best Predictability:** Context-4 (word) with 96.8% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (1,631,641 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (1,177,702 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -246,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 417,410 |
-| Total Tokens | 23,479,822 |
-| Mean Frequency | 56.25 |
+| Vocabulary Size | 391,795 |
+| Total Tokens | 21,537,937 |
+| Mean Frequency | 54.97 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 1249.29 |
+| Frequency Std Dev | 1228.27 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | һәм | 442,975 |
-| 2 | буйынса | 199,955 |
-| 3 | категория | 198,342 |
-| 4 | һыу | 168,429 |
-| 5 | менән | 154,744 |
-| 6 | йылға | 141,138 |
-| 7 | йылда | 136,378 |
-| 8 | рәсәй | 111,896 |
-| 9 | йыл | 97,392 |
-| 10 | йылдың | 89,845 |
+| 1 | һәм | 442,727 |
+| 2 | буйынса | 199,652 |
+| 3 | һыу | 168,369 |
+| 4 | менән | 154,690 |
+| 5 | йылға | 141,126 |
+| 6 | йылда | 136,417 |
+| 7 | рәсәй | 107,366 |
+| 8 | йыл | 97,537 |
+| 9 | йылдың | 89,696 |
+| 10 | в | 87,704 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | совкомбанк | 2 |
-| 2 | маркетплейстың | 2 |
-| 3 | суларға | 2 |
-| 4 | кишлак | 2 |
-| 5 | пацанский | 2 |
-| 6 | мунден | 2 |
-| 7 | гертфордшир | 2 |
-| 8 | кроуға | 2 |
-| 9 | франклоу | 2 |
-| 10 | алтынкүлдән | 2 |
+| 1 | анкалаевҡа | 2 |
+| 2 | куцелаба | 2 |
+| 3 | хизарович | 2 |
+| 4 | чимаевтың | 2 |
+| 5 | уиттакерҙың | 2 |
+| 6 | дрикус | 2 |
+| 7 | шарабутдин | 2 |
+| 8 | rcc | 2 |
+| 9 | cosmetics | 2 |
+| 10 | kits | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.0644 |
-| R² (Goodness of Fit) | 0.989157 |
+| Zipf Coefficient | 1.0493 |
+| R² (Goodness of Fit) | 0.992213 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 23.2% |
-| Top 1,000 | 52.0% |
-| Top 5,000 | 71.9% |
-| Top 10,000 | 78.9% |
+| Top 100 | 23.9% |
+| Top 1,000 | 52.3% |
+| Top 5,000 | 71.5% |
+| Top 10,000 | 78.5% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9892 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 23.2% of corpus
-- **Long Tail:** 407,410 words needed for remaining 21.1% coverage
+- **Zipf Compliance:** R²=0.9922 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 23.9% of corpus
+- **Long Tail:** 381,795 words needed for remaining 21.5% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -316,24 +384,113 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 246,880 | 32 | 3.693 | 1.255 | 0.7645 |
-| **mono_64d** | 246,880 | 64 | 4.152 | 1.202 | 0.7712 🏆 |
-| **mono_128d** | 246,880 | 128 | 4.739 | 1.156 | 0.7517 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.7656 | 0.3637 | N/A | N/A |
+| **mono_64d** | 64 | 0.7751 🏆 | 0.2899 | N/A | N/A |
+| **mono_128d** | 128 | 0.7586 | 0.2211 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_64d with 0.7712 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 246,880 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_64d with 0.7751 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2916. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
+
+---
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-а` | симпозиумдарында, режица, оффенбаха |
+| `-ың` | тамаҡтың, ялкайндың, һуҙаһың |
+| `-ан` | ышанмаған, аҡсабан, гарнизондарынан |
+| `-ар` | стәрлетамаҡлылар, аныҡлаусылар, яндырылғандар |
+| `-ға` | ципрофлоксацинға, һауығырға, ҡыҫырыҡларға |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `ассе` | 2.59x | 57 contexts | сассе, массе, гассе |
+| `ссей` | 3.05x | 29 contexts | бассей, шоссей, иессей |
+| `олог` | 1.87x | 205 contexts | лолог, молог, полог |
+| `арҙа` | 1.74x | 267 contexts | дарҙа, арҙан, барҙа |
+| `арҙы` | 1.79x | 169 contexts | шарҙы, сарҙы, ҡарҙы |
+| `лған` | 1.60x | 230 contexts | алған, ялған, ҡлған |
+| `шҡор` | 3.05x | 15 contexts | башҡор, башҡорд, башҡорт |
+| `ылға` | 1.57x | 213 contexts | йылға, тылға, ҡылға |
+| `йылғ` | 1.88x | 73 contexts | йылға, йылғы, уйылға |
+| `әрен` | 1.63x | 140 contexts | йәрен, кәрен, дәрен |
+| `дәүл` | 2.80x | 16 contexts | дәүли, дәүлә, дәүләт |
+| `әүлә` | 1.99x | 39 contexts | хәүлә, дәүлә, мәүлә |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+*No significant affix co-occurrences detected.*
+
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| биониканың | **`бионик-ан-ың`** | 6.0 | `бионик` |
+| худякованың | **`худяков-ан-ың`** | 6.0 | `худяков` |
+| воронкованың | **`воронков-ан-ың`** | 6.0 | `воронков` |
+| давыдованың | **`давыдов-ан-ың`** | 6.0 | `давыдов` |
+| фонеманың | **`фонем-ан-ың`** | 6.0 | `фонем` |
+| балаһынан | **`балаһын-ан`** | 4.5 | `балаһын` |
+| фламенкоға | **`фламенко-ға`** | 4.5 | `фламенко` |
+| топонимияһынан | **`топонимияһын-ан`** | 4.5 | `топонимияһын` |
+| баштарының | **`баштарын-ың`** | 4.5 | `баштарын` |
+| людмилаға | **`людмила-ға`** | 4.5 | `людмила` |
+| мозаикаға | **`мозаика-ға`** | 4.5 | `мозаика` |
+| орлеанскийға | **`орлеанский-ға`** | 4.5 | `орлеанский` |
+| манараларының | **`манараларын-ың`** | 4.5 | `манараларын` |
+| начальнигынан | **`начальнигын-ан`** | 4.5 | `начальнигын` |
+| кинофильмының | **`кинофильмын-ың`** | 4.5 | `кинофильмын` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language BA appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
 
 ---
-## 6. Summary & Recommendations
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -341,11 +498,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.07x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (586) |
-| Markov | **Context-4** | Highest predictability (93.8%) |
+| Tokenizer | **64k BPE** | Best compression (4.67x) |
+| N-gram | **2-gram** | Lowest perplexity (489) |
+| Markov | **Context-4** | Highest predictability (96.8%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -535,7 +693,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -551,7 +710,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 23:45:09*
+*Report Date: 2026-01-03 07:03:34*

models/embeddings/monolingual/ba_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a5ab9a23748938ba7a7d67175df00ad9b8f68caab526a73b491efb79bbbbe158
-size 1283130940
+oid sha256:1d0506e69046f5b9eeed00571257333963c93985c08aaff04359a63f9e18966d
+size 1267773201

models/embeddings/monolingual/ba_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 246880
+  "vocab_size": 232257
 }

models/embeddings/monolingual/ba_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:04654a56597263edcecef3b373e2a46dc6600d993eb9bd089dd4f30b4f6271fc
-size 325527100
+oid sha256:857e0d1d2801efad1ea0ad2269a5064780058e0457d0dd04fc3e747f514bb814
+size 321399825

models/embeddings/monolingual/ba_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 246880
+  "vocab_size": 232257
 }

models/embeddings/monolingual/ba_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a93872afbd8f9317ccf9f48907b1875deb4c323daca798bc87cd8b1031f5280d
-size 644728380
+oid sha256:a68b56475da9d52e23c1e8987aa18959fc1800f3fc2373b5aa026f7005dc4bd0
+size 636857617

models/embeddings/monolingual/ba_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 246880
+  "vocab_size": 232257
 }

models/subword_markov/ba_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3d6868878ce8c3d2fd2c44240f2a970b42cb2586dbfde7025d1e72a3b6ac70b3
-size 342219
+oid sha256:72ab68634a6f78c810b9cb22d356ab586e0001dca90d89cea677632b627a61ff
+size 321684

models/subword_markov/ba_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "ba",
-  "unique_contexts": 4928,
-  "total_transitions": 182147511
+  "unique_contexts": 5664,
+  "total_transitions": 166306994
 }

models/subword_markov/ba_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a89552d8eccadab10a1ddfcdbc4ef36a2c61eb6c16185355b87ec80138c89456
-size 2589714
+oid sha256:64f356772a4c2f2fe17d0b3ced4a871083815f3306cc44a66e5ae43c45a11b68
+size 2038058

models/subword_markov/ba_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "ba",
-  "unique_contexts": 48690,
-  "total_transitions": 182083612
+  "unique_contexts": 42359,
+  "total_transitions": 166243136
 }

models/subword_markov/ba_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fe121d741c5aa034400c27fbbc59abf2e556ee7421504c98b9d515f7a84ad7bb
-size 12785656
+oid sha256:034412812ae35db516b6f169d1eaad99c3412d09b2b6b86dd34d6c091ce4cc35
+size 10033851

models/subword_markov/ba_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "ba",
-  "unique_contexts": 344963,
-  "total_transitions": 182019713
+  "unique_contexts": 250210,
+  "total_transitions": 166179278
 }

models/subword_markov/ba_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5ae5d341812bb14f14b59e593a901e72ed00651851f4e438b227da8ad19c4b3b
-size 45384065
+oid sha256:de9d856639518fda8614b7147be3a51f07dc47b782f251d16d66f550cd28ed7e
+size 34335118

models/subword_markov/ba_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "ba",
-  "unique_contexts": 1631641,
-  "total_transitions": 181955814
+  "unique_contexts": 1177702,
+  "total_transitions": 166115420
 }

models/subword_ngram/ba_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1fc455ec80702bbdfc394b20ad7e53af994541f5739559a039c1f14797a9da11
-size 239143
+oid sha256:d8bc5caa63739eb8bd546faf8fd400a3cfed2102cf28c5d178636fd18c4a8540
+size 193332

models/subword_ngram/ba_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "ba",
-  "unique_ngrams": 17419,
-  "total_ngrams": 182147511
+  "unique_ngrams": 13769,
+  "total_ngrams": 166306994
 }

models/subword_ngram/ba_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5c40130c1d50bed91350758c922cb88d6c733a82bc1c682c711ddb6b3a967810
-size 2055831
+oid sha256:92068c289061bb2e93b829c142f737236f826c376deac540afd5cf5264596713
+size 1542627

models/subword_ngram/ba_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "ba",
-  "unique_ngrams": 163683,
-  "total_ngrams": 182083612
+  "unique_ngrams": 117773,
+  "total_ngrams": 166243136
 }

models/subword_ngram/ba_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:69d73f681e1e459c2bb8a107c96c5c890e3a2e44d6ab89934d8b31a552557178
-size 11914292
+oid sha256:829baeb29467446ba03ac3c2e0fd8e4ce93634a4b5989afa2f4cbad92beba6d0
+size 9034722

models/subword_ngram/ba_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "ba",
-  "unique_ngrams": 914560,
-  "total_ngrams": 182019713
+  "unique_ngrams": 687383,
+  "total_ngrams": 166179278
 }

models/tokenizer/ba_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a9f7f8c6111aa25b0301c425ef2ca81bd1c492beb384a92794a1bb5cc8894883
-size 585441
+oid sha256:a303c0d026bb9c2d55bf88aea6c23a3e9ee5479dbe177e187ac24e298d72853f
+size 587702

models/tokenizer/ba_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6e332a25893c33db32e04b3f994922558d7dc8b80336f210ced52c2fd41f20f5
-size 959495
+oid sha256:c53247b474ba1fb2bc81a96123dbc8edfeb1a2aa07072d220e4422014f9aa0a7
+size 956108

models/tokenizer/ba_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cea1f24f48084bfadca804cc73807fcc76d7f6bb1dff92a5c5e66974005805d8
-size 1732060
+oid sha256:b92b6043e85b598f61bfc865f4cea415a1ba6735c830a01264fe91b74338e8a0
+size 1712379

models/tokenizer/ba_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1e002b5188a90a30bfff5759ab095f75651e042f7c82b1813539e5174c356ccc
-size 406416
+oid sha256:ebdbb6ac25e572b302f21c6cd6328ec1606feb430f390d602f0e1d7add179ee0
+size 408777

models/vocabulary/ba_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:649c9e7928c6d049c1807e44ee84ba39de0da8e0afe944c65f92c5e67dc152ea
-size 7166077
+oid sha256:65de254f1bc43c0356834a3df469079bf03def47f1cebad5fc6f9eda7540b462
+size 6781677

models/vocabulary/ba_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "ba",
-  "vocabulary_size": 417410,
+  "vocabulary_size": 391795,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.04478519689532156,
+    "type_token_ratio": 0.041486328965339804,
     "coverage": {
-      "top_100": 0.22514075312093942,
-      "top_1000": 0.505577357431498,
-      "top_5000": 0.6994683745965613,
-      "top_10000": 0.7677274461586074
+      "top_100": 0.23295039278580296,
+      "top_1000": 0.510335569094746,
+      "top_5000": 0.6976257341559833,
+      "top_10000": 0.7667897867532179
     },
-    "hapax_count": 663870,
-    "hapax_ratio": 0.6139667801124593,
-    "total_documents": 63899
+    "hapax_count": 523451,
+    "hapax_ratio": 0.5719238324996777,
+    "total_documents": 63858
   }
 }

models/word_markov/ba_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e4875129953afdc863c3b7840b357671100ecec9e86e7b3b1f5291e35e34721f
-size 96022509
+oid sha256:92fd33e481646024de79a8020c805420b7e263733616508172265ae2ae3ffd6d
+size 103200739

models/word_markov/ba_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "ba",
-  "unique_contexts": 1081802,
-  "total_transitions": 31976815
+  "unique_contexts": 915102,
+  "total_transitions": 21997530
 }

models/word_markov/ba_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:40cea10bfc8003e0da9502d75d5a3f40994d832dc0531213e610dc8d43e240b8
-size 318818703
+oid sha256:6e3439ca9b5e1619b07b22f4f1b78774fe37bef5ff88ba91a9f9bb6d367c1dba
+size 335420766

models/word_markov/ba_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "ba",
-  "unique_contexts": 7695897,
-  "total_transitions": 31912918
+  "unique_contexts": 8225491,
+  "total_transitions": 21933672
 }

models/word_markov/ba_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9f02c513c6045c41dd1b65bd8350a8ae70c94cc93ed3a0e138a41a538bcc05b9
-size 525618960
+oid sha256:1c2d14e6732bdebeb2da58ec10759660304522a4cfd88648d442df2b62e56654
+size 495650890

models/word_markov/ba_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "ba",
-  "unique_contexts": 15852609,
-  "total_transitions": 31849021
+  "unique_contexts": 14302544,
+  "total_transitions": 21869814
 }

models/word_markov/ba_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:285ccd82b56012a9d822c5a84b5bd795f9f01fdd006e41e3d182acacb79cd943
-size 670487523
+oid sha256:9f6f284d108dd9a0fa4ad4d988f64ed44de3e7ca0dcac08f5a86bf65953c220e
+size 603913588

models/word_markov/ba_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "ba",
-  "unique_contexts": 20837559,
-  "total_transitions": 31785127
+  "unique_contexts": 16653317,
+  "total_transitions": 21805956
 }

models/word_ngram/ba_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cfe67bcedd2ac32d53a8a2dd06d0192ad020853a8783d4bb0a6899c0ea5f9839
-size 13228712
+oid sha256:5d06c49aeeb05893184effe3f4626afe5bb8378179b82b0547da6bdf3858e1be
+size 11465370

models/word_ngram/ba_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "ba",
-  "unique_ngrams": 552281,
-  "total_ngrams": 31976815
+  "unique_ngrams": 433408,
+  "total_ngrams": 21997530
 }

models/word_ngram/ba_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8dfd471500e30bd3120ae58431c82aac2f9ea6b68195d44bb94fde6f64830a54
-size 27689470
+oid sha256:0ef620130d0e6e9252f714154c5b5c57b13961cab6f737793a94aa79445273f9
+size 18273381

models/word_ngram/ba_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "ba",
-  "unique_ngrams": 1025675,
-  "total_ngrams": 31912918
+  "unique_ngrams": 563973,
+  "total_ngrams": 21933672
 }

models/word_ngram/ba_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:577732cb5b7771e05860e02c4596a4b43feba90e0cbe682d7bca85459e12e11a
-size 51575545
+oid sha256:e495639d67edc292e0555a61932d40e85b6cca00135c5bb0dff00e942a310b68
+size 31535510

models/word_ngram/ba_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "ba",
-  "unique_ngrams": 1793503,
-  "total_ngrams": 31849021
+  "unique_ngrams": 883766,
+  "total_ngrams": 21869814
 }