Upload all models and assets for bxr (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.299
+    value: 4.390
   - name: best_isotropy
     type: isotropy
-    value: 0.8992
+    value: 0.8916
   - name: vocabulary_size
     type: vocab
-    value: 37973
-generated: 2025-12-28
+    value: 0
+generated: 2026-01-03
 ---
 
 # BXR - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,56 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.277x | 3.23 | 0.1090% | 644,799 |
-| **16k** | 3.643x | 3.59 | 0.1212% | 580,038 |
-| **32k** | 3.983x | 3.93 | 0.1325% | 530,456 |
-| **64k** | 4.299x 🏆 | 4.24 | 0.1430% | 491,550 |
+| **8k** | 3.450x | 3.45 | 0.1418% | 628,340 |
+| **16k** | 3.842x | 3.84 | 0.1579% | 564,308 |
+| **32k** | 4.148x | 4.15 | 0.1705% | 522,647 |
+| **64k** | 4.390x 🏆 | 4.39 | 0.1804% | 493,909 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Маршалл аралууд - Австралиин болон Океаниин улас.
-
-Категори:Австрали ба Океани`
+**Sample 1:** `Цзяинь - Ород Википеэдийн Үбэр Монголой долоо хоногой үгүүлэл. Мүн үзэхэ Үбэр Мо...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁мар шал л ▁арал ууд ▁- ▁австралиин ▁болон ▁оке аниин ... (+9 more)` | 19 |
-| 16k | `▁маршал л ▁аралууд ▁- ▁австралиин ▁болон ▁оке аниин ▁улас . ... (+5 more)` | 15 |
-| 32k | `▁маршал л ▁аралууд ▁- ▁австралиин ▁болон ▁океаниин ▁улас . ▁категори ... (+4 more)` | 14 |
-| 64k | `▁маршалл ▁аралууд ▁- ▁австралиин ▁болон ▁океаниин ▁улас . ▁категори : ... (+3 more)` | 13 |
-
-**Sample 2:** `Зимбабве - Африкийн улас.
+| 8k | `▁цз я инь ▁- ▁ород ▁википеэдийн ▁үбэр ▁монголой ▁долоо ▁хоногой ... (+8 more)` | 18 |
+| 16k | `▁цз я инь ▁- ▁ород ▁википеэдийн ▁үбэр ▁монголой ▁долоо ▁хоногой ... (+8 more)` | 18 |
+| 32k | `▁цзя инь ▁- ▁ород ▁википеэдийн ▁үбэр ▁монголой ▁долоо ▁хоногой ▁үгүүлэл ... (+7 more)` | 17 |
+| 64k | `▁цзяинь ▁- ▁ород ▁википеэдийн ▁үбэр ▁монголой ▁долоо ▁хоногой ▁үгүүлэл . ... (+6 more)` | 16 |
 
-President : Emmerson Mnangagwa
-Категори:Африка`
+**Sample 2:** `Мобилизаци гээшэ зэбсэгтэ хүсэниие энхэ тайбангай байдалһаань дайнай байдалда ор...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁з им ба б ве ▁- ▁африкийн ▁улас . ▁pr ... (+16 more)` | 26 |
-| 16k | `▁з имба б ве ▁- ▁африкийн ▁улас . ▁pres ident ... (+12 more)` | 22 |
-| 32k | `▁зимбабве ▁- ▁африкийн ▁улас . ▁president ▁: ▁em m erson ... (+8 more)` | 18 |
-| 64k | `▁зимбабве ▁- ▁африкийн ▁улас . ▁president ▁: ▁emm erson ▁mn ... (+6 more)` | 16 |
+| 8k | `▁м об ил изаци ���гээшэ ▁зэбсэгтэ ▁хүсэн иие ▁энхэ ▁тайбан ... (+11 more)` | 21 |
+| 16k | `▁м об ил изаци ▁гээшэ ▁зэбсэгтэ ▁хүсэниие ▁энхэ ▁тайбан гай ... (+9 more)` | 19 |
+| 32k | `▁м обилизаци ▁гээшэ ▁зэбсэгтэ ▁хүсэниие ▁энхэ ▁тайбангай ▁байдалһаань ▁дайнай ▁байдалда ... (+5 more)` | 15 |
+| 64k | `▁мобилизаци ▁гээшэ ▁зэбсэгтэ ▁хүсэниие ▁энхэ ▁тайбангай ▁байдалһаань ▁дайнай ▁байдалда ▁оруулха ... (+4 more)` | 14 |
 
-**Sample 3:** `Ваканси ( — «хооһон, сүлөө»)  эмхи зургаанда, һургуулиин газарта эзэлэгдээгүй ту...`
+**Sample 3:** `Гильзэ — буугай һомоной түмэр патрон. Зүүлтэ зэбсэг`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁вак ан си ▁( ▁— ▁« х ооһон , ▁сүлөө ... (+27 more)` | 37 |
-| 16k | `▁вак ан си ▁( ▁— ▁« х ооһон , ▁сүлөө ... (+26 more)` | 36 |
-| 32k | `▁вак ан си ▁( ▁— ▁« хооһон , ▁сүлөө ») ... (+22 more)` | 32 |
-| 64k | `▁вак ан си ▁( ▁— ▁« хооһон , ▁сүлөө ») ... (+19 more)` | 29 |
+| 8k | `▁г иль зэ ▁— ▁буу гай ▁һом оной ▁түмэр ▁патр ... (+4 more)` | 14 |
+| 16k | `▁г иль зэ ▁— ▁буу гай ▁һомоной ▁түмэр ▁патр он ... (+3 more)` | 13 |
+| 32k | `▁г иль зэ ▁— ▁буу гай ▁һомоной ▁түмэр ▁патр он ... (+3 more)` | 13 |
+| 64k | `▁г иль зэ ▁— ▁буугай ▁һомоной ▁түмэр ▁патрон . ▁зүүлтэ ... (+1 more)` | 11 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.299x compression
-- **Lowest UNK Rate:** 8k with 0.1090% unknown tokens
+- **Best Compression:** 64k achieves 4.390x compression
+- **Lowest UNK Rate:** 8k with 0.1418% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -126,57 +129,89 @@ President : Emmerson Mnangagwa
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 5,799 🏆 | 12.50 | 15,674 | 20.1% | 46.7% |
-| **2-gram** | 532 🏆 | 9.05 | 4,895 | 53.6% | 95.0% |
-| **3-gram** | 8,896 | 13.12 | 19,215 | 16.4% | 37.7% |
-| **3-gram** | 4,459 | 12.12 | 35,775 | 19.2% | 59.0% |
-| **4-gram** | 16,750 | 14.03 | 31,775 | 13.4% | 28.0% |
-| **4-gram** | 21,165 | 14.37 | 149,695 | 9.1% | 33.0% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 4,169 | 12.03 | 8,128 | 19.5% | 49.4% |
+| **2-gram** | Subword | 452 🏆 | 8.82 | 3,823 | 56.9% | 96.7% |
+| **3-gram** | Word | 3,724 | 11.86 | 7,805 | 24.5% | 47.7% |
+| **3-gram** | Subword | 3,736 | 11.87 | 29,340 | 20.6% | 62.1% |
+| **4-gram** | Word | 7,537 | 12.88 | 14,616 | 19.0% | 34.7% |
+| **4-gram** | Subword | 18,031 | 14.14 | 124,835 | 9.4% | 34.5% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `энэ үдэр` | 1,070 |
+| 2 | `гү али` | 1,030 |
+| 3 | `of the` | 465 |
+| 4 | `байна энэ` | 415 |
+| 5 | `бүгэдэ найрамдаха` | 396 |
+
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `категори :` | 4,149 |
-| 2 | `юм .` | 3,028 |
-| 3 | `) ,` | 2,485 |
-| 4 | `байна .` | 2,414 |
-| 5 | `) .` | 1,647 |
+| 1 | `үдэр наһа бараһаниинь` | 353 |
+| 2 | `үдэр тохёоһон үйлэ` | 353 |
+| 3 | `энэ үдэр түрэһэниинь` | 353 |
+| 4 | `үйлэ ябадалай жагсаалта` | 353 |
+| 5 | `энэ үдэр тохёоһон` | 353 |
 
-**3-grams:**
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `зүүлтэ категори :` | 1,025 |
-| 2 | `гү , али` | 1,017 |
-| 3 | `. зүүлтэ категори` | 687 |
-| 4 | `. категори :` | 682 |
-| 5 | `( ородоор :` | 647 |
+| 1 | `энэ үдэр наһа бараһаниинь` | 353 |
+| 2 | `тохёоһон үйлэ ябадалай жагсаалта` | 353 |
+| 3 | `үдэр тохёоһон үйлэ ябадалай` | 353 |
+| 4 | `энэ үдэр тохёоһон үйлэ` | 353 |
+| 5 | `энэ үдэрэй тэ��дэглэлтэ баяр` | 345 |
 
-**4-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `. зүүлтэ категори :` | 687 |
-| 2 | `тохёоһон үйлэ ябадалай жагсаалта` | 366 |
-| 3 | `үдэр тохёоһон үйлэ ябадалай` | 366 |
-| 4 | `энэ үдэр тохёоһон үйлэ` | 366 |
-| 5 | `энэ үдэр наһа бараһаниинь` | 366 |
+| 1 | `н _` | 82,295 |
+| 2 | `й _` | 56,691 |
+| 3 | `_ б` | 54,353 |
+| 4 | `_ х` | 50,092 |
+| 5 | `а й` | 48,574 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `а й _` | 24,558 |
+| 2 | `_ б а` | 24,246 |
+| 3 | `ы н _` | 18,435 |
+| 4 | `э й _` | 17,416 |
+| 5 | `а н _` | 16,805 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ б а й` | 12,907 |
+| 2 | `_ б о л` | 11,173 |
+| 3 | `б о л о` | 9,002 |
+| 4 | `и и н _` | 6,889 |
+| 5 | `_ у л а` | 6,870 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 532
+- **Best Perplexity:** 2-gram (subword) with 452
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~33% of corpus
+- **Coverage:** Top-1000 patterns cover ~35% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -184,55 +219,86 @@ President : Emmerson Mnangagwa
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.6555 | 1.575 | 4.12 | 100,619 | 34.4% |
-| **1** | 1.0618 | 2.088 | 7.29 | 2,105 | 0.0% |
-| **2** | 0.1980 | 1.147 | 1.45 | 414,597 | 80.2% |
-| **2** | 0.8900 | 1.853 | 5.27 | 15,331 | 11.0% |
-| **3** | 0.0669 | 1.047 | 1.12 | 601,940 | 93.3% |
-| **3** | 0.7767 | 1.713 | 3.63 | 80,818 | 22.3% |
-| **4** | 0.0247 🏆 | 1.017 | 1.04 | 671,341 | 97.5% |
-| **4** | 0.5432 🏆 | 1.457 | 2.28 | 293,211 | 45.7% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.7391 | 1.669 | 4.14 | 92,909 | 26.1% |
+| **1** | Subword | 0.8623 | 1.818 | 5.69 | 2,141 | 13.8% |
+| **2** | Word | 0.1430 | 1.104 | 1.26 | 383,260 | 85.7% |
+| **2** | Subword | 0.8174 | 1.762 | 5.04 | 12,176 | 18.3% |
+| **3** | Word | 0.0340 | 1.024 | 1.05 | 482,888 | 96.6% |
+| **3** | Subword | 0.7977 | 1.738 | 3.77 | 61,348 | 20.2% |
+| **4** | Word | 0.0111 🏆 | 1.008 | 1.02 | 504,904 | 98.9% |
+| **4** | Subword | 0.5768 | 1.491 | 2.40 | 230,966 | 42.3% |
+
+### Generated Text Samples (Word-based)
 
-### Generated Text Samples
+Below are text samples generated from each word-based Markov chain model:
 
-Below are text samples generated from each Markov chain model:
+**Context Size 1:**
+
+1. `ба үлэнтэй шэлэ нюруу шулуун харшанууд гэхэ мэтэ болобош 1 хушааһан 4 зуун орост нуран унажа`
+2. `юм зүүлтэ гадаада ба хүн гэжэ намые арадай хуралай депутатаар һунгагдаһан юрэнхылэгшэ байгаа тула тэ...`
+3. `энэ үедэ мадрид мадридынь шэнэ үгэнүүд бии гал носоохо гал задагай агаарта гү али алинда гарза`
+
+**Context Size 2:**
+
+1. `энэ үдэр тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь оной урда үе энэ үдэр тохёоһон үйлэ я...`
+2. `гү али бэеын дархалалай харюу урбалаар янза бүриин үнгэтэй улаан ногоон шара г м түлэб хиинүүдынь хи...`
+3. `of the american library association energy data statistics for russia from the principles of water i...`
+
+**Context Size 3:**
+
+1. `энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь николай островский зүблэлтэ зохёолшо как закалялась с...`
+2. `тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь энэ үдэрэй тэмдэглэл...`
+3. `энэ үдэр тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь энэ үдэрэй ...`
+
+**Context Size 4:**
+
+1. `үдэр тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь энэ үдэрэй тэмд...`
+2. `энэ үдэр тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь энэ үдэрэй ...`
+3. `тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь оной урда үе энэ үдэр наһа бараһаниинь энэ үдэ...`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `, 080 526 | | almeirim | 9 хуби . түмэр гарган абаха , тэршэлэн ,`
-2. `. металл болон туризм , ( 0 , амеэрикын нэгэдэһэн улас һуулшын 10 жилийн хугацаатай томилогджээ`
-3. `( 14 , ехэ , гурбан шажанууд болон xiii зуунай франциин уран үгэ гаралтай . аж`
+1. `_raps_s_бон_аали`
+2. `аандэрыноданаай_`
+3. `эругэнь_ой_оте_м`
 
 **Context Size 2:**
 
-1. `категори : зэмэ аймаг ( ) — оросой улас түрын , эдэй засагта шэлжэһэн . 1926 .`
-2. `юм . 48 c - һээ 10 ° c ( уһанай хүлдэхын сэг , шугам , сахилгаан`
-3. `) , пит бест хоёр бүрилдэхүүнһөө гарашье хожомынь ринго старр ( бүмбэршэн , дуушан ) дүрбэнһөө 1960`
+1. `н_хүн_юм.,_5_бари`
+2. `й_(ганые_/kazano!`
+3. `_бай_һар_180—512_`
 
 **Context Size 3:**
 
-1. `зүүлтэ категори : хүдэлмэри`
-2. `гү , али кесаревын зүһэлгын аргаар 39 долоо хоногой урда нарайлалга бусад элүүр мэндын шалтгаанай ул...`
-3. `. зүүлтэ категори : буддын шажан категори : хамба ламанар категори : 1935 ондо түрэгэһэд категори : ...`
+1. `ай_ботар_(үндэ_үед`
+2. `_баярын,_камын_ург`
+3. `ын_5-дуңма_хэрэ_өө`
 
 **Context Size 4:**
 
-1. `. зүүлтэ категори : ехэ британи категори : арадууд`
-2. `энэ үдэр тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь категори : ...`
-3. `энэ үдэр наһа бараһаниинь категори : үдэрнүүд`
+1. `_байлгаха_агналда_х`
+2. `_боложо_уласые_байр`
+3. `болон_тэнгисангуй_б`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 97.5% predictability
+- **Best Predictability:** Context-4 (word) with 98.9% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (293,211 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (230,966 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -248,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 37,973 |
-| Total Tokens | 528,039 |
-| Mean Frequency | 13.91 |
+| Vocabulary Size | 36,185 |
+| Total Tokens | 491,809 |
+| Mean Frequency | 13.59 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 76.38 |
+| Frequency Std Dev | 73.56 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | категори | 4,150 |
-| 2 | ба | 3,836 |
-| 3 | юм | 3,234 |
-| 4 | энэ | 3,067 |
-| 5 | ондо | 2,851 |
-| 6 | болон | 2,646 |
-| 7 | байна | 2,600 |
-| 8 | улас | 2,578 |
-| 9 | оной | 2,541 |
-| 10 | the | 2,196 |
+| 1 | ба | 3,838 |
+| 2 | юм | 3,200 |
+| 3 | энэ | 3,020 |
+| 4 | ондо | 2,873 |
+| 5 | болон | 2,652 |
+| 6 | оной | 2,566 |
+| 7 | байна | 2,566 |
+| 8 | улас | 2,455 |
+| 9 | the | 2,159 |
+| 10 | of | 2,042 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ᠲᠠᠢ | 2 |
-| 2 | ᠮᠣᠩᠭᠤᠯ | 2 |
-| 3 | ᠤᠷᠤᠨ | 2 |
-| 4 | ᠮᠢᠨᠢ | 2 |
-| 5 | ᠦᠷ | 2 |
-| 6 | ᠵᠢᠷᠭᠠᠯ | 2 |
-| 7 | дүхэригтэй | 2 |
-| 8 | e5 | 2 |
-| 9 | исибагай | 2 |
-| 10 | ылын | 2 |
+| 1 | үүсэбэринүүд | 2 |
+| 2 | ᠮᠠᠨᠠᠶ | 2 |
+| 3 | ᠲᠠᠢ | 2 |
+| 4 | ᠮᠣᠩᠭᠤᠯ | 2 |
+| 5 | ᠤᠷᠤᠨ | 2 |
+| 6 | ᠮᠢᠨᠢ | 2 |
+| 7 | ᠦᠷ | 2 |
+| 8 | ᠵᠢᠷᠭᠠᠯ | 2 |
+| 9 | дүхэригтэй | 2 |
+| 10 | исибагай | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9692 |
-| R² (Goodness of Fit) | 0.992557 |
+| Zipf Coefficient | 0.9662 |
+| R² (Goodness of Fit) | 0.993759 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 21.7% |
-| Top 1,000 | 51.3% |
-| Top 5,000 | 74.3% |
-| Top 10,000 | 83.7% |
+| Top 100 | 22.2% |
+| Top 1,000 | 52.2% |
+| Top 5,000 | 74.6% |
+| Top 10,000 | 84.1% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9926 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 21.7% of corpus
-- **Long Tail:** 27,973 words needed for remaining 16.3% coverage
+- **Zipf Compliance:** R²=0.9938 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 22.2% of corpus
+- **Long Tail:** 26,185 words needed for remaining 15.9% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -318,24 +384,124 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 14,816 | 32 | 3.693 | 0.816 | 0.8992 🏆 |
-| **mono_64d** | 14,816 | 64 | 4.178 | 0.727 | 0.8181 |
-| **mono_128d** | 14,816 | 128 | 4.428 | 0.707 | 0.4039 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8916 🏆 | 0.3371 | N/A | N/A |
+| **mono_64d** | 64 | 0.8046 | 0.2601 | N/A | N/A |
+| **mono_128d** | 128 | 0.3726 | 0.2357 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8992 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 14,816 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8916 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2776. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ба` | байхдаа, балнад, байһан |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-н` | хилын, биотехнологиин, зайдан |
+| `-й` | феодорой, намуудай, зангай |
+| `-ай` | намуудай, зангай, дарангылалай |
+| `-ан` | зайдан, хааншалһан, буруудхан |
+| `-эй` | сэнтэй, клэй, тэригүүдэй |
+| `-ын` | хилын, доржын, эмнэлгын |
+| `-ые` | хүүгэдые, различные, вермахтые |
+| `-эн` | үйһэн, нэмэгдэһэн, дэбжүүлэн |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `анай` | 1.88x | 75 contexts | ганай, манай, ханай |
+| `гуул` | 1.80x | 67 contexts | уугуул, агуулга, хайгуул |
+| `эгдэ` | 1.65x | 93 contexts | жэгдэ, нэгдэн, нэгдэл |
+| `азар` | 2.38x | 21 contexts | газар, базар, лазарь |
+| `дэһэ` | 1.85x | 44 contexts | үдэһэн, гэдэһэ, үндэһэ |
+| `энэй` | 1.75x | 53 contexts | сэнэй, үгэнэй, үһэнэй |
+| `эдэг` | 1.70x | 57 contexts | гэдэг, хэдэг, ерэдэг |
+| `алай` | 1.78x | 47 contexts | далай, малай, һалай |
+| `ниин` | 1.85x | 40 contexts | ниинь, даниин, линиин |
+| `нууд` | 1.62x | 56 contexts | онууд, орнууд, ионууд |
+| `үндэ` | 1.75x | 39 contexts | һүндэ, хүндэ, үндэр |
+| `айда` | 1.77x | 36 contexts | сайда, дайда, зайда |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-ба` | `-й` | 38 words | байнхэй, баримталалай |
+| `-ба` | `-ай` | 29 words | баримталалай, баталгаатай |
+| `-ба` | `-н` | 27 words | балжан, байн |
+| `-ба` | `-ан` | 17 words | балжан, барилгашан |
+| `-ба` | `-ые` | 13 words | байрлалые, баримтые |
+| `-ба` | `-ын` | 4 words | байгуулгын, багамын |
+| `-ба` | `-эй` | 1 words | байнхэй |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| түүхэшэдые | **`түүхэшэд-ые`** | 4.5 | `түүхэшэд` |
+| түшэмэлые | **`түшэмэл-ые`** | 4.5 | `түшэмэл` |
+| дамжуулгануудые | **`дамжуулганууд-ые`** | 4.5 | `дамжуулганууд` |
+| далайшадай | **`далайшад-ай`** | 4.5 | `далайшад` |
+| хубисалай | **`хубисал-ай`** | 4.5 | `хубисал` |
+| ниигэмүүдэй | **`ниигэмүүд-эй`** | 4.5 | `ниигэмүүд` |
+| хэлэгшэдэй | **`хэлэгшэд-эй`** | 4.5 | `хэлэгшэд` |
+| таряашадай | **`таряашад-ай`** | 4.5 | `таряашад` |
+| магадлалай | **`магадлал-ай`** | 4.5 | `магадлал` |
+| тогтоһоные | **`тогтоһон-ые`** | 4.5 | `тогтоһон` |
+| буурсагые | **`буурсаг-ые`** | 4.5 | `буурсаг` |
+| юумэнүүдые | **`юумэнүүд-ые`** | 4.5 | `юумэнүүд` |
+| дашинимаевай | **`дашинимаев-ай`** | 4.5 | `дашинимаев` |
+| найрамдалай | **`найрамдал-ай`** | 4.5 | `найрамдал` |
+| зохёолуудые | **`зохёолууд-ые`** | 4.5 | `зохёолууд` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language BXR appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -343,11 +509,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.30x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (532) |
-| Markov | **Context-4** | Highest predictability (97.5%) |
+| Tokenizer | **64k BPE** | Best compression (4.39x) |
+| N-gram | **2-gram** | Lowest perplexity (452) |
+| Markov | **Context-4** | Highest predictability (98.9%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -537,7 +704,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -553,7 +721,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-28 09:22:58*
+*Report Date: 2026-01-03 09:00:32*

models/embeddings/monolingual/bxr_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:74b8869363141d972d72be5bdf332a7e4fdd0330e861684e70a4b494e8291893
-size 1039505247
+oid sha256:9a68f121ed60ec837dcd0cd6dfe40ccae7bda27cefb1882915867e69840e1104
+size 1038925515

models/embeddings/monolingual/bxr_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 14816
+  "vocab_size": 14262
 }

models/embeddings/monolingual/bxr_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:003deff7c5ca0b619e47c9989e4f389629b2a0d871190e9d2902770d857ca46d
-size 260126559
+oid sha256:bf82d275c6ec9e3300c42d4bf6f1d83083710befa2f67e76dcf383dcfb7c187a
+size 259972299

models/embeddings/monolingual/bxr_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 14816
+  "vocab_size": 14262
 }

models/embeddings/monolingual/bxr_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ca0f7f1e62bb09a2478a59565e98e61f30486cbd4b309435d81d3530e1c2a6dc
-size 519919455
+oid sha256:846d84f5d2248a39c8ed392e72d06b7956f7871585cd833febab2c57b28b0799
+size 519623371

models/embeddings/monolingual/bxr_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 14816
+  "vocab_size": 14262
 }

models/subword_markov/bxr_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7f40c2222fbfccd237862781d0d0e955784a245a38a54dbe387b8cec2b9d2b82
-size 117796
+oid sha256:8ef28afcd8620169ed8fa2c34876e8625f7c7d635864fbd2796f37f30dba4cf1
+size 102324

models/subword_markov/bxr_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "bxr",
-  "unique_contexts": 2105,
-  "total_transitions": 4289081
+  "unique_contexts": 2141,
+  "total_transitions": 3947454
 }

models/subword_markov/bxr_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:608dcc0d1cb6248070161cbadb7420910339b0915882fb5dd132e4901a394daf
-size 623776
+oid sha256:e8eeb9396bbe777fa33dbcb0a026a02660f3421472954259ef5fb21e30676aba
+size 526873

models/subword_markov/bxr_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "bxr",
-  "unique_contexts": 15331,
-  "total_transitions": 4286148
+  "unique_contexts": 12176,
+  "total_transitions": 3944697
 }

models/subword_markov/bxr_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3ba216b7648c71d9678295b66f161d77cbdceb72f9be9fb280ff239f8df4097b
-size 2166085
+oid sha256:251c7577b1bff07e9bd8f78d3e3413f809f22133920660f8ff8fd5953d1df234
+size 1738691

models/subword_markov/bxr_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "bxr",
-  "unique_contexts": 80818,
-  "total_transitions": 4283215
+  "unique_contexts": 61348,
+  "total_transitions": 3941940
 }

models/subword_markov/bxr_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:720c028f0afa2beef418da44146e3aa7fdd4e8a84d87aaec098f46b0c4904607
-size 6019932
+oid sha256:e6704187c11e7bf62086d0438a41d230986b7075412d814372c74e9c42a43bb8
+size 4955182

models/subword_markov/bxr_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "bxr",
-  "unique_contexts": 293211,
-  "total_transitions": 4280282
+  "unique_contexts": 230966,
+  "total_transitions": 3939183
 }

models/subword_ngram/bxr_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b30174f435d373a7a684d0fdf82ab6f85dd7d97e258e9d2966c7f95ac04897bd
-size 63850
+oid sha256:4dd1815af3d79add27e46b93017d57f0b62838b802a873a7a07c137fb6540c3d
+size 51809

models/subword_ngram/bxr_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "bxr",
-  "unique_ngrams": 4895,
-  "total_ngrams": 4289081
+  "unique_ngrams": 3823,
+  "total_ngrams": 3947454
 }

models/subword_ngram/bxr_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:918475421fb0cf8063f92f0780e8b12e328c8328a0fc30ff110bb89e499d66ee
-size 467922
+oid sha256:63257791e85ed8eaf2b882c6823a3a702fe1a0ea07f2c59e64c50dd97653a352
+size 376817

models/subword_ngram/bxr_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "bxr",
-  "unique_ngrams": 35775,
-  "total_ngrams": 4286148
+  "unique_ngrams": 29340,
+  "total_ngrams": 3944697
 }

models/subword_ngram/bxr_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e1042ddad008a9599c0748487f78c73efa7d9b50d6e70a4a6fafe6c8db84ab71
-size 1826857
+oid sha256:7fc6a6ba4fa3d0662727116dddfc49d4ebb04b8fe918900b45bddaff9a5e475c
+size 1528848

models/subword_ngram/bxr_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "bxr",
-  "unique_ngrams": 149695,
-  "total_ngrams": 4283215
+  "unique_ngrams": 124835,
+  "total_ngrams": 3941940
 }

models/tokenizer/bxr_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c313a02b460324a3fd78aeeebbf024c367c66eec31762824e5d13559b06e96eb
-size 566509
+oid sha256:9ec3513a9e102b31e37ebbe9c74465e68178a75caa31640e2da4dd18964909d1
+size 572848

models/tokenizer/bxr_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:54bddc6897ff3219983c48eaffae095b8f605807ed27292c894a640c31089df6
-size 917207
+oid sha256:e83a11fb6d2c68714a56a8f79737f3bdf4020f6741555062a3277802b3b50563
+size 936687

models/tokenizer/bxr_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:29ab501244bdf544b03adcaaefea70ce2eccdb557572c1df01afcc5ba62a6ac1
-size 1665622
+oid sha256:1a9a5aa8bf4ae1cf3f260d408856d0de604cee508f58aa294462236f70101270
+size 1699640

models/tokenizer/bxr_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:421a45e29770748dce802c6ebcb76913d893b4e21831c4ceebf7fc41ffa3d537
-size 396771
+oid sha256:8bb41ea3e440a8f23950afc663ca4194ddb8fe1cbbfd3fc96c5e7a3187bdb9a0
+size 400889

models/vocabulary/bxr_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6f052baf5dc25aeabdf73c8b5285ec87abcc088d635e2973f70a8ffe31ed6b36
-size 717641
+oid sha256:71921497bda1a7f95ca89688025ce27f6680e9797e7176d62d5eeb5de52180fa
+size 687102

models/vocabulary/bxr_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "bxr",
-  "vocabulary_size": 37973,
+  "vocabulary_size": 36185,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.17001129648792698,
+    "type_token_ratio": 0.16949986693740954,
     "coverage": {
-      "top_100": 0.19418781300332458,
-      "top_1000": 0.4590506546712756,
-      "top_5000": 0.664120017139499,
-      "top_10000": 0.748515113074965
+      "top_100": 0.19864421979752614,
+      "top_1000": 0.46809049714371126,
+      "top_5000": 0.6691097930421025,
+      "top_10000": 0.7539690930235101
     },
-    "hapax_count": 62410,
-    "hapax_ratio": 0.6217188169311537,
-    "total_documents": 2933
+    "hapax_count": 56805,
+    "hapax_ratio": 0.6108721367889021,
+    "total_documents": 2757
   }
 }

models/word_markov/bxr_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:650b9705441f0a50ae4936a70a92a0cb793133d94dc1d0370ed9ed6d3c7dd84c
-size 5152081
+oid sha256:8d83d9d3e8e86f09c7979a3b6839e87cbf8a1c801cae88a26b8e2568e80c0af1
+size 4628667

models/word_markov/bxr_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "bxr",
-  "unique_contexts": 100619,
-  "total_transitions": 759550
+  "unique_contexts": 92909,
+  "total_transitions": 545857
 }

models/word_markov/bxr_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4ac1c70b5bf0d07084ffb47645e4f1fad509d187d04f4eeb81e55e1abc7af15e
-size 11903170
+oid sha256:08a52284e5b3cf3a5d895964ac029e73604bb505096a1643018fcbce3b7f0871
+size 11044627

models/word_markov/bxr_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "bxr",
-  "unique_contexts": 414597,
-  "total_transitions": 756617
+  "unique_contexts": 383260,
+  "total_transitions": 543100
 }

models/word_markov/bxr_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:902dc90ffc87fd7ada473260dd7c7a9bd442da2e1209173b26a8b5bb99539fbf
-size 16125345
+oid sha256:20b01b9dbd6b5fb671c3229725a1ed9ec3da87304bb0d69aa5d5c22d2d65e482
+size 14044491

models/word_markov/bxr_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "bxr",
-  "unique_contexts": 601940,
-  "total_transitions": 753684
+  "unique_contexts": 482888,
+  "total_transitions": 540343
 }

models/word_markov/bxr_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3f4c22cf906dbab93a3218dc56a6b362a41a9c0e192456e5f64beca972c76c3f
-size 19134570
+oid sha256:1672222d8edd090496caa2f1e1bd6c68fd909d2a58607dab3d33a10e8047ca99
+size 16507536

models/word_markov/bxr_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "bxr",
-  "unique_contexts": 671341,
-  "total_transitions": 750751
+  "unique_contexts": 504904,
+  "total_transitions": 537586
 }

models/word_ngram/bxr_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c2a3ebfb828cb5690e2603dc347ba8fa6c7472b2c4e334354e5ff525f64915e0
-size 320596
+oid sha256:5f22572c4d354fedb3053f8b2b9665062ea46ca2de919d373b3f625078118c3c
+size 192506

models/word_ngram/bxr_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "bxr",
-  "unique_ngrams": 15674,
-  "total_ngrams": 759550
+  "unique_ngrams": 8128,
+  "total_ngrams": 545857
 }

models/word_ngram/bxr_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0cda9295ac08e43661c1ef513f79da30c697a4ffd25d8b920bc6d5a8e91b67f2
-size 440192
+oid sha256:97ee45afd7fac64151a4862029dca6299fc0b2a8c65aa3d7868466142d152bd5
+size 220646

models/word_ngram/bxr_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "bxr",
-  "unique_ngrams": 19215,
-  "total_ngrams": 756617
+  "unique_ngrams": 7805,
+  "total_ngrams": 543100
 }

models/word_ngram/bxr_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c95d55178d9a68895f802a5b94c63c1a5584c899404e70ef1d1e0a17fa6afa20
-size 791045
+oid sha256:c0fe193bfe17ed9385235885ddb5fa79c77354a7a702f0a815c82ede303edbfc
+size 453116

models/word_ngram/bxr_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "bxr",
-  "unique_ngrams": 31775,
-  "total_ngrams": 753684
+  "unique_ngrams": 14616,
+  "total_ngrams": 540343
 }