Upload all models and assets for bxr (latest)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: bxr
-language_name: BXR
+language_name: Russia Buriat
 language_family: mongolic
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-mongolic
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.390
+    value: 4.402
   - name: best_isotropy
     type: isotropy
-    value: 0.8916
+    value: 0.9019
   - name: vocabulary_size
     type: vocab
     value: 0
 generated: 2026-01-03
 ---
 
-# BXR - Wikilangs Models
+# Russia Buriat - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **BXR** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Russia Buriat** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -60,7 +70,7 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
 - [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
@@ -80,47 +90,47 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.450x | 3.45 | 0.1418% | 628,340 |
-| **16k** | 3.842x | 3.84 | 0.1579% | 564,308 |
-| **32k** | 4.148x | 4.15 | 0.1705% | 522,647 |
-| **64k** | 4.390x 🏆 | 4.39 | 0.1804% | 493,909 |
+| **8k** | 3.459x | 3.46 | 0.1450% | 616,507 |
+| **16k** | 3.854x | 3.86 | 0.1615% | 553,408 |
+| **32k** | 4.159x | 4.16 | 0.1743% | 512,788 |
+| **64k** | 4.402x 🏆 | 4.40 | 0.1845% | 484,538 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Цзяинь - Ород Википеэдийн Үбэр Монголой долоо хоногой үгүүлэл. Мүн үзэхэ Үбэр Мо...`
+**Sample 1:** `Мэйси - Ород Википеэдийн Үбэр Монголой долоо хоногой үгүүлэл. Мүн үзэхэ Үбэр Мон...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁цз я инь ▁- ▁ород ▁википеэдийн ▁үбэр ▁монголой ▁долоо ▁хоногой ... (+8 more)` | 18 |
-| 16k | `▁цз я инь ▁- ▁ород ▁википеэдийн ▁үбэр ▁монголой ▁долоо ▁хоногой ... (+8 more)` | 18 |
-| 32k | `▁цзя инь ▁- ▁ород ▁википеэдийн ▁үбэр ▁монголой ▁долоо ▁хоногой ▁үгүүлэл ... (+7 more)` | 17 |
-| 64k | `▁цзяинь ▁- ▁ород ▁википеэдийн ▁үбэр ▁монголой ▁долоо ▁хоногой ▁үгүүлэл . ... (+6 more)` | 16 |
+| 8k | `▁мэй си ▁- ▁ород ▁википеэдийн ▁үбэр ▁монголой ▁долоо ▁хоногой ▁үгүүлэл ... (+7 more)` | 17 |
+| 16k | `▁мэй си ▁- ▁ород ▁википеэдийн ▁үбэр ▁монголой ▁долоо ▁хоногой ▁үгүүлэл ... (+7 more)` | 17 |
+| 32k | `▁мэй си ▁- ▁ород ▁википеэдийн ▁үбэр ▁монголой ▁долоо ▁хоногой ▁үгүүлэл ... (+7 more)` | 17 |
+| 64k | `▁мэйси ▁- ▁ород ▁википеэдийн ▁үбэр ▁монголой ▁долоо ▁хоногой ▁үгүүлэл . ... (+6 more)` | 16 |
 
-**Sample 2:** `Мобилизаци гээшэ зэбсэгтэ хүсэниие энхэ тайбангай байдалһаань дайнай байдалда ор...`
+**Sample 2:** `Уһан далайн сэрэгэй авиаци — уһан соо бууха ба уһан дээрэһээ ниидэжэ гараха онго...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁м об ил изаци ▁гээшэ ▁зэбсэгтэ ▁хүсэн иие ▁энхэ ▁тайбан ... (+11 more)` | 21 |
-| 16k | `▁м об ил изаци ▁гээшэ ▁зэбсэгтэ ▁хүсэниие ▁энх�� ▁тайбан гай ... (+9 more)` | 19 |
-| 32k | `▁м обилизаци ▁гээшэ ▁зэбсэгтэ ▁хүсэниие ▁энхэ ▁тайбангай ▁байдалһаань ▁дайнай ▁байдалда ... (+5 more)` | 15 |
-| 64k | `▁мобилизаци ▁гээшэ ▁зэбсэгтэ ▁хүсэниие ▁энхэ ▁тайбангай ▁байдалһаань ▁дайнай ▁байдалда ▁оруулха ... (+4 more)` | 14 |
+| 8k | `▁уһан ▁далайн ▁сэрэгэй ▁ав иа ци ▁— ▁уһан ▁соо ▁буу ... (+16 more)` | 26 |
+| 16k | `▁уһан ▁далайн ▁сэрэгэй ▁авиа ци ▁— ▁уһан ▁соо ▁бууха ▁ба ... (+13 more)` | 23 |
+| 32k | `▁уһан ▁далайн ▁сэрэгэй ▁авиаци ▁— ▁уһан ▁соо ▁бууха ▁ба ▁уһан ... (+12 more)` | 22 |
+| 64k | `▁уһан ▁далайн ▁сэрэгэй ▁авиаци ▁— ▁уһан ▁соо ▁бууха ▁ба ▁уһан ... (+12 more)` | 22 |
 
-**Sample 3:** `Гильзэ — буугай һомоной түмэр патрон. Зүүлтэ зэбсэг`
+**Sample 3:** `Денонсаци — нэгэ гүрэнэй нүгөө гүрэндэ өөр—хоорондохи ябажа байгаа хэрээ, хэлсээ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁г иль зэ ▁— ▁буу гай ▁һом оной ▁түмэр ▁патр ... (+4 more)` | 14 |
-| 16k | `▁г иль зэ ▁— ▁буу гай ▁һомоной ▁түмэр ▁патр он ... (+3 more)` | 13 |
-| 32k | `▁г иль зэ ▁— ▁буу гай ▁һомоной ▁түмэр ▁патр он ... (+3 more)` | 13 |
-| 64k | `▁г иль зэ ▁— ▁буугай ▁һомоной ▁түмэр ▁патрон . ▁зүүлтэ ... (+1 more)` | 11 |
+| 8k | `▁д ен он са ци ▁— ▁нэгэ ▁гүрэнэй ▁нүгөө ▁гүрэндэ ... (+16 more)` | 26 |
+| 16k | `▁ден он са ци ▁— ▁нэгэ ▁гүрэнэй ▁нүгөө ▁гүрэндэ ▁өөр ... (+14 more)` | 24 |
+| 32k | `▁ден он са ци ▁— ▁нэгэ ▁гүрэнэй ▁нүгөө ▁гүрэндэ ▁өөр ... (+14 more)` | 24 |
+| 64k | `▁денонсаци ▁— ▁нэгэ ▁гүрэнэй ▁нүгөө ▁гүрэндэ ▁өөр — хоорондохи ▁ябажа ... (+9 more)` | 19 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.390x compression
-- **Lowest UNK Rate:** 8k with 0.1418% unknown tokens
+- **Best Compression:** 64k achieves 4.402x compression
+- **Lowest UNK Rate:** 8k with 0.1450% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -137,12 +147,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
 |--------|---------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | Word | 4,169 | 12.03 | 8,128 | 19.5% | 49.4% |
-| **2-gram** | Subword | 452 🏆 | 8.82 | 3,823 | 56.9% | 96.7% |
-| **3-gram** | Word | 3,724 | 11.86 | 7,805 | 24.5% | 47.7% |
-| **3-gram** | Subword | 3,736 | 11.87 | 29,340 | 20.6% | 62.1% |
-| **4-gram** | Word | 7,537 | 12.88 | 14,616 | 19.0% | 34.7% |
-| **4-gram** | Subword | 18,031 | 14.14 | 124,835 | 9.4% | 34.5% |
+| **2-gram** | Word | 4,087 | 12.00 | 8,036 | 19.8% | 49.7% |
+| **2-gram** | Subword | 452 🏆 | 8.82 | 3,815 | 56.9% | 96.7% |
+| **3-gram** | Word | 3,571 | 11.80 | 7,655 | 25.2% | 48.6% |
+| **3-gram** | Subword | 3,726 | 11.86 | 29,176 | 20.6% | 62.2% |
+| **4-gram** | Word | 7,283 | 12.83 | 14,462 | 19.6% | 35.4% |
+| **4-gram** | Subword | 17,919 | 14.13 | 123,764 | 9.4% | 34.6% |
+| **5-gram** | Word | 5,323 | 12.38 | 10,833 | 22.1% | 38.6% |
+| **5-gram** | Subword | 48,261 | 15.56 | 234,708 | 6.1% | 22.3% |
 
 ### Top 5 N-grams by Size
 
@@ -150,68 +162,88 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `энэ үдэр` | 1,070 |
-| 2 | `гү али` | 1,030 |
-| 3 | `of the` | 465 |
-| 4 | `байна энэ` | 415 |
+| 1 | `энэ үдэр` | 1,109 |
+| 2 | `гү али` | 1,021 |
+| 3 | `of the` | 462 |
+| 4 | `байна энэ` | 425 |
 | 5 | `бүгэдэ найрамдаха` | 396 |
 
 **3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `үдэр наһа бараһаниинь` | 353 |
-| 2 | `үдэр тохёоһон үйлэ` | 353 |
-| 3 | `энэ үдэр түрэһэниинь` | 353 |
-| 4 | `үйлэ ябадалай жагсаалта` | 353 |
-| 5 | `энэ үдэр тохёоһон` | 353 |
+| 1 | `үйлэ ябадалай жагсаалта` | 366 |
+| 2 | `энэ үдэр тохёоһон` | 366 |
+| 3 | `тохёоһон үйлэ ябадалай` | 366 |
+| 4 | `үдэр наһа бараһаниинь` | 366 |
+| 5 | `энэ үдэр наһа` | 366 |
 
 **4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `энэ үдэр наһа бараһаниинь` | 353 |
-| 2 | `тохёоһон үйлэ ябадалай жагсаалта` | 353 |
-| 3 | `үдэр тохёоһон үйлэ ябадалай` | 353 |
-| 4 | `энэ үдэр тохёоһон үйлэ` | 353 |
-| 5 | `энэ үдэрэй тэмдэглэлтэ баяр` | 345 |
+| 1 | `үдэр тохёоһон үйлэ ябадалай` | 366 |
+| 2 | `энэ үдэр наһа бараһаниинь` | 366 |
+| 3 | `энэ үдэр тохёоһон үйлэ` | 366 |
+| 4 | `тохёоһон үйлэ ябадалай жагсаалта` | 366 |
+| 5 | `энэ үдэрэй тэмдэглэлтэ баяр` | 358 |
+
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `энэ үдэр тохёоһон үйлэ ябадалай` | 366 |
+| 2 | `үдэр тохёоһон үйлэ ябадалай жагсаалта` | 366 |
+| 3 | `тохёоһон үйлэ ябадалай жагсаалта энэ` | 340 |
+| 4 | `ябадалай жагсаалта энэ үдэр түрэһэниинь` | 340 |
+| 5 | `үйлэ ябадалай жагсаалта энэ үдэр` | 340 |
 
 **2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `н _` | 82,295 |
-| 2 | `й _` | 56,691 |
-| 3 | `_ б` | 54,353 |
-| 4 | `_ х` | 50,092 |
-| 5 | `а й` | 48,574 |
+| 1 | `н _` | 81,065 |
+| 2 | `й _` | 55,911 |
+| 3 | `_ б` | 53,676 |
+| 4 | `_ х` | 49,355 |
+| 5 | `а й` | 47,888 |
 
 **3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `а й _` | 24,558 |
-| 2 | `_ б а` | 24,246 |
-| 3 | `ы н _` | 18,435 |
-| 4 | `э й _` | 17,416 |
-| 5 | `а н _` | 16,805 |
+| 1 | `а й _` | 24,178 |
+| 2 | `_ б а` | 23,944 |
+| 3 | `ы н _` | 18,168 |
+| 4 | `э й _` | 17,283 |
+| 5 | `а н _` | 16,564 |
 
 **4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `_ б а й` | 12,907 |
-| 2 | `_ б о л` | 11,173 |
-| 3 | `б о л о` | 9,002 |
-| 4 | `и и н _` | 6,889 |
-| 5 | `_ у л а` | 6,870 |
+| 1 | `_ б а й` | 12,726 |
+| 2 | `_ б о л` | 11,040 |
+| 3 | `б о л о` | 8,901 |
+| 4 | `и и н _` | 6,846 |
+| 5 | `_ у л а` | 6,751 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ б о л о` | 8,849 |
+| 2 | `_ у л а с` | 5,743 |
+| 3 | `о н о й _` | 4,950 |
+| 4 | `а н а й _` | 4,619 |
+| 5 | `э һ э н _` | 4,162 |
 
 
 ### Key Findings
 
 - **Best Perplexity:** 2-gram (subword) with 452
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~35% of corpus
+- **Coverage:** Top-1000 patterns cover ~22% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -227,14 +259,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
 |---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.7391 | 1.669 | 4.14 | 92,909 | 26.1% |
-| **1** | Subword | 0.8623 | 1.818 | 5.69 | 2,141 | 13.8% |
-| **2** | Word | 0.1430 | 1.104 | 1.26 | 383,260 | 85.7% |
-| **2** | Subword | 0.8174 | 1.762 | 5.04 | 12,176 | 18.3% |
-| **3** | Word | 0.0340 | 1.024 | 1.05 | 482,888 | 96.6% |
-| **3** | Subword | 0.7977 | 1.738 | 3.77 | 61,348 | 20.2% |
-| **4** | Word | 0.0111 🏆 | 1.008 | 1.02 | 504,904 | 98.9% |
-| **4** | Subword | 0.5768 | 1.491 | 2.40 | 230,966 | 42.3% |
+| **1** | Word | 0.7365 | 1.666 | 4.12 | 92,015 | 26.3% |
+| **1** | Subword | 0.8645 | 1.821 | 5.69 | 2,131 | 13.5% |
+| **2** | Word | 0.1428 | 1.104 | 1.26 | 378,037 | 85.7% |
+| **2** | Subword | 0.8166 | 1.761 | 5.04 | 12,123 | 18.3% |
+| **3** | Word | 0.0341 | 1.024 | 1.05 | 476,205 | 96.6% |
+| **3** | Subword | 0.7973 | 1.738 | 3.76 | 61,012 | 20.3% |
+| **4** | Word | 0.0112 🏆 | 1.008 | 1.02 | 497,992 | 98.9% |
+| **4** | Subword | 0.5747 | 1.489 | 2.39 | 229,261 | 42.5% |
 
 ### Generated Text Samples (Word-based)
 
@@ -242,27 +274,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `ба үлэнтэй шэлэ нюруу шулуун харшанууд гэхэ мэтэ болобош 1 хушааһан 4 зуун орост нуран унажа`
-2. `юм зүүлтэ гадаада ба хүн гэжэ намые арадай хуралай депутатаар һунгагдаһан юрэнхылэгшэ байгаа тула тэ...`
-3. `энэ үедэ мадрид мадридынь шэнэ үгэнүүд бии гал носоохо гал задагай агаарта гү али алинда гарза`
+1. `ба дайшадай толгойнууд олдоо һэн мүн магрибай ар��б уласай 5 сая ажаһуугшад боложо үгэһэн бэлэй ниисл...`
+2. `юм исаак ньютон джон нэрэтэй байгаад наһа бараа үйлэшэлгын хэлтэстэ хубаагдана эдэ олон жэлэй 189 дэ...`
+3. `энэ үдэр түрэһэниинь парацельс алхимик эмшэ эсперантогой байгуулагша гээд хэдэн нөлөө дэндүү их гүрн...`
 
 **Context Size 2:**
 
-1. `энэ үдэр тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь оной урда үе энэ үдэр тохёоһон үйлэ я...`
-2. `гү али бэеын дархалалай харюу урбалаар янза бүриин үнгэтэй улаан ногоон шара г м түлэб хиинүүдынь хи...`
-3. `of the american library association energy data statistics for russia from the principles of water i...`
+1. `энэ үдэр тохёоһон үйлэ ябадалай жагсаалта 324 римэй эзэнтэ гүрэнэй үндэһэлэгшэд отто фон бисмарк фри...`
+2. `гү али зүрхэнэй өөрынхинь мэдэрэлэй тогтолсоогоор ябагдана агшалтын үеэр шуһанай һудаһуудта шуһан ша...`
+3. `of the iaea itu upu and wipo and a permanently functioning legislative administrative and supervisor...`
 
 **Context Size 3:**
 
-1. `энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь николай островский зүблэлтэ зохёолшо как закалялась с...`
-2. `тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь энэ үдэрэй тэмдэглэл...`
-3. `энэ үдэр тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь энэ үдэрэй ...`
+1. `тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь энэ үдэрэй тэмдэглэл...`
+2. `үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь энэ үдэрэй тэмдэглэлтэ баяр э...`
+3. `энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь энэ үдэрэй тэмдэглэлтэ баяр энэ үдэр тохёоһон үйлэ яб...`
 
 **Context Size 4:**
 
-1. `үдэр тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь энэ үдэрэй тэмд...`
-2. `энэ үдэр тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь энэ үдэрэй ...`
-3. `тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь оной урда үе энэ үдэр наһа бараһаниинь энэ үдэ...`
+1. `үдэр тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь оной урда үе энэ үдэр наһа бараһаниинь эн...`
+2. `тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь энэ үдэр наһа бараһаниинь энэ үдэрэй тэмдэглэл...`
+3. `энэ үдэр тохёоһон үйлэ ябадалай жагсаалта энэ үдэр түрэһэниинь оной урда үе энэ үдэр наһа бараһаниин...`
 
 
 ### Generated Text Samples (Subword-based)
@@ -271,34 +303,34 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_raps_s_бон_аали`
-2. `аандэрыноданаай_`
-3. `эругэнь_ой_оте_м`
+1. `_6,_сэн»_г,_үүга`
+2. `а_тэршэгай_гаһэд`
+3. `эраре_бан_каасэй`
 
 **Context Size 2:**
 
-1. `н_хүн_юм.,_5_бари`
-2. `й_(ганые_/kazano!`
-3. `_бай_һар_180—512_`
+1. `н_зари,_хажа._бан`
+2. `й_лэгэ,_plearunt_`
+3. `_баран._захмерита`
 
 **Context Size 3:**
 
-1. `ай_ботар_(үндэ_үед`
-2. `_баярын,_камын_ург`
-3. `ын_5-дуңма_хэрэ_өө`
+1. `ай_гэшүүн_хубиин_1`
+2. `_бан_холбоон_ба_ту`
+3. `ын_аралай_марилсуу`
 
 **Context Size 4:**
 
-1. `_байлгаха_агналда_х`
-2. `_боложо_уласые_байр`
-3. `болон_тэнгисангуй_б`
+1. `_байна._антика._мож`
+2. `_болоһоншье_үлүү_эр`
+3. `болобошье,_каирай_н`
 
 
 ### Key Findings
 
 - **Best Predictability:** Context-4 (word) with 98.9% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (230,966 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (229,261 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -314,48 +346,48 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 36,185 |
-| Total Tokens | 491,809 |
-| Mean Frequency | 13.59 |
+| Vocabulary Size | 35,751 |
+| Total Tokens | 485,385 |
+| Mean Frequency | 13.58 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 73.56 |
+| Frequency Std Dev | 73.26 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ба | 3,838 |
-| 2 | юм | 3,200 |
-| 3 | энэ | 3,020 |
-| 4 | ондо | 2,873 |
-| 5 | болон | 2,652 |
-| 6 | оной | 2,566 |
-| 7 | байна | 2,566 |
-| 8 | улас | 2,455 |
-| 9 | the | 2,159 |
-| 10 | of | 2,042 |
+| 1 | ба | 3,777 |
+| 2 | юм | 3,165 |
+| 3 | энэ | 3,056 |
+| 4 | ондо | 2,831 |
+| 5 | болон | 2,629 |
+| 6 | байна | 2,533 |
+| 7 | оной | 2,521 |
+| 8 | улас | 2,428 |
+| 9 | the | 2,147 |
+| 10 | үдэр | 2,079 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | үүсэбэринүүд | 2 |
-| 2 | ᠮᠠᠨᠠᠶ | 2 |
-| 3 | ᠲᠠᠢ | 2 |
-| 4 | ᠮᠣᠩᠭᠤᠯ | 2 |
-| 5 | ᠤᠷᠤᠨ | 2 |
-| 6 | ᠮᠢᠨᠢ | 2 |
-| 7 | ᠦᠷ | 2 |
-| 8 | ᠵᠢᠷᠭᠠᠯ | 2 |
-| 9 | дүхэригтэй | 2 |
-| 10 | исибагай | 2 |
+| 1 | ᠮᠠᠨᠠᠶ | 2 |
+| 2 | ᠲᠠᠢ | 2 |
+| 3 | ᠮᠣᠩᠭᠤᠯ | 2 |
+| 4 | ᠤᠷᠤᠨ | 2 |
+| 5 | ᠮᠢᠨᠢ | 2 |
+| 6 | ᠦᠷ | 2 |
+| 7 | ᠵᠢᠷᠭᠠᠯ | 2 |
+| 8 | дүхэригтэй | 2 |
+| 9 | исибагай | 2 |
+| 10 | ылын | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9662 |
-| R² (Goodness of Fit) | 0.993759 |
+| Zipf Coefficient | 0.9688 |
+| R² (Goodness of Fit) | 0.993514 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
@@ -363,15 +395,15 @@ Below are text samples generated from each subword-based Markov chain model:
 | Top N Words | Coverage |
 |-------------|----------|
 | Top 100 | 22.2% |
-| Top 1,000 | 52.2% |
-| Top 5,000 | 74.6% |
-| Top 10,000 | 84.1% |
+| Top 1,000 | 52.4% |
+| Top 5,000 | 74.8% |
+| Top 10,000 | 84.3% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9938 indicates excellent adherence to Zipf's law
+- **Zipf Compliance:** R²=0.9935 indicates excellent adherence to Zipf's law
 - **High Frequency Dominance:** Top 100 words cover 22.2% of corpus
-- **Long Tail:** 26,185 words needed for remaining 15.9% coverage
+- **Long Tail:** 25,751 words needed for remaining 15.7% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -387,37 +419,40 @@ Below are text samples generated from each subword-based Markov chain model:
 
 ### 5.1 Cross-Lingual Alignment
 
-> *Note: Multilingual alignment visualization not available for this language.*
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
 
 
 ### 5.2 Model Comparison
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.8916 🏆 | 0.3371 | N/A | N/A |
-| **mono_64d** | 64 | 0.8046 | 0.2601 | N/A | N/A |
-| **mono_128d** | 128 | 0.3726 | 0.2357 | N/A | N/A |
+| **mono_32d** | 32 | 0.9019 🏆 | 0.3176 | N/A | N/A |
+| **mono_64d** | 64 | 0.7924 | 0.2625 | N/A | N/A |
+| **mono_128d** | 128 | 0.3620 | 0.2359 | N/A | N/A |
+| **aligned_32d** | 32 | 0.9019 | 0.3203 | 0.0100 | 0.1160 |
+| **aligned_64d** | 64 | 0.7924 | 0.2588 | 0.0220 | 0.1580 |
+| **aligned_128d** | 128 | 0.3620 | 0.2402 | 0.0480 | 0.2140 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8916 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.2776. Lower values indicate better semantic separation.
-- **Alignment Quality:** No aligned models evaluated in this run.
+- **Best Isotropy:** mono_32d with 0.9019 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2725. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 4.8% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
 ## 6.  Morphological Analysis (Experimental)
 
-> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
-
 This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
 
 ### 6.1 Productivity & Complexity
 
 | Metric | Value | Interpretation | Recommendation |
 |--------|-------|----------------|----------------|
-| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
-| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **0.728** | High formulaic/idiomatic content | - |
 
 ### 6.2 Affix Inventory (Productive Units)
 
@@ -426,19 +461,20 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Prefixes
 | Prefix | Examples |
 |--------|----------|
-| `-ба` | байхдаа, балнад, байһан |
+| `-ба` | байгаар, байр, баряуд |
+| `-ха` | харагдана, халимагууд, хангахын |
 
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-н` | хилын, биотехнологиин, зайдан |
-| `-й` | феодорой, намуудай, зангай |
-| `-ай` | намуудай, зангай, дарангылалай |
-| `-ан` | зайдан, хааншалһан, буруудхан |
-| `-эй` | сэнтэй, клэй, тэригүүдэй |
-| `-ын` | хилын, доржын, эмнэлгын |
-| `-ые` | хүүгэдые, различные, вермахтые |
-| `-эн` | үйһэн, нэмэгдэһэн, дэбжүүлэн |
+| `-н` | шатааһан, португалиин, догшин |
+| `-й` | монголой, шэрхэгтэй, санхүүгай |
+| `-ай` | санхүүгай, билзуухай, байгуулгануудтай |
+| `-ан` | шатааһан, урлаһан, абатан |
+| `-эй` | шэрхэгтэй, ерэнхэй, нүхэтэй |
+| `-ые` | диграфые, конгрессые, логикые |
+| `-ын` | хилын, нэмэгдэхын, хангахын |
+| `-нь` | уклонь, утаашань, вангиинь |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -446,18 +482,18 @@ Bound stems are high-frequency subword units that are semantically cohesive but
 
 | Stem | Cohesion | Substitutability | Examples |
 |------|----------|------------------|----------|
-| `анай` | 1.88x | 75 contexts | ганай, манай, ханай |
-| `гуул` | 1.80x | 67 contexts | уугуул, агуулга, хайгуул |
-| `эгдэ` | 1.65x | 93 contexts | жэгдэ, нэгдэн, нэгдэл |
-| `азар` | 2.38x | 21 contexts | газар, базар, лазарь |
-| `дэһэ` | 1.85x | 44 contexts | үдэһэн, гэдэһэ, үндэһэ |
-| `энэй` | 1.75x | 53 contexts | сэнэй, үгэнэй, үһэнэй |
-| `эдэг` | 1.70x | 57 contexts | гэдэг, хэдэг, ерэдэг |
-| `алай` | 1.78x | 47 contexts | далай, малай, һалай |
-| `ниин` | 1.85x | 40 contexts | ниинь, даниин, линиин |
-| `нууд` | 1.62x | 56 contexts | онууд, орнууд, ионууд |
-| `үндэ` | 1.75x | 39 contexts | һүндэ, хүндэ, үндэр |
-| `айда` | 1.77x | 36 contexts | сайда, дайда, зайда |
+| `гуул` | 1.87x | 66 contexts | уугуул, хайгуул, агуулжа |
+| `энэй` | 1.92x | 53 contexts | сэнэй, эзэнэй, энэнэй |
+| `анай` | 1.74x | 74 contexts | манай, танай, ванай |
+| `ниин` | 1.99x | 40 contexts | ниинь, даниин, кениин |
+| `азар` | 2.36x | 21 contexts | газар, базар, лазарь |
+| `нүүд` | 1.92x | 41 contexts | үенүүд, гүнүүд, эснүүд |
+| `алай` | 1.85x | 47 contexts | һалай, малай, алайр |
+| `дэһэ` | 1.87x | 44 contexts | гэдэһэ, үндэһэ, үдэһэн |
+| `эдэг` | 1.76x | 56 contexts | хэдэг, гэдэг, үзэдэг |
+| `эгдэ` | 1.57x | 91 contexts | жэгдэ, дэгдэн, нэгдэн |
+| `оһон` | 1.91x | 40 contexts | тоһон, хооһон, ороһон |
+| `ууда` | 1.72x | 57 contexts | уудам, уудаг, буудал |
 
 ### 6.4 Affix Compatibility (Co-occurrence)
 
@@ -465,13 +501,16 @@ This table shows which prefixes and suffixes most frequently co-occur on the sam
 
 | Prefix | Suffix | Frequency | Examples |
 |--------|--------|-----------|----------|
-| `-ба` | `-й` | 38 words | байнхэй, баримталалай |
-| `-ба` | `-ай` | 29 words | баримталалай, баталгаатай |
-| `-ба` | `-н` | 27 words | балжан, байн |
-| `-ба` | `-ан` | 17 words | балжан, барилгашан |
-| `-ба` | `-ые` | 13 words | байрлалые, баримтые |
-| `-ба` | `-ын` | 4 words | байгуулгын, багамын |
-| `-ба` | `-эй` | 1 words | байнхэй |
+| `-ба` | `-н` | 36 words | багамын, байгуулсан |
+| `-ха` | `-н` | 29 words | хамаарһан, харбаан |
+| `-ба` | `-й` | 28 words | байгууламжануудай, баттерфляй |
+| `-ха` | `-й` | 26 words | харбинай, хатарай |
+| `-ха` | `-ай` | 23 words | харбинай, хатарай |
+| `-ха` | `-ан` | 21 words | хамаарһан, харбаан |
+| `-ба` | `-ан` | 21 words | байгуулсан, барилдаан |
+| `-ба` | `-ай` | 18 words | байгууламжануудай, баатарай |
+| `-ха` | `-аа` | 13 words | хаанһаа, харууллаа |
+| `-ба` | `-аа` | 11 words | байдалаараа, бараа |
 
 ### 6.5 Recursive Morpheme Segmentation
 
@@ -479,26 +518,28 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 
 | Word | Suggested Split | Confidence | Stem |
 |------|-----------------|------------|------|
-| түүхэшэдые | **`түүхэшэд-ые`** | 4.5 | `түүхэшэд` |
-| түшэмэлые | **`түшэмэл-ые`** | 4.5 | `түшэмэл` |
-| дамжуулгануудые | **`дамжуулганууд-ые`** | 4.5 | `дамжуулганууд` |
-| далайшадай | **`далайшад-ай`** | 4.5 | `далайшад` |
-| хубисалай | **`хубисал-ай`** | 4.5 | `хубисал` |
-| ниигэмүүдэй | **`ниигэмүүд-эй`** | 4.5 | `ниигэмүүд` |
-| хэлэгшэдэй | **`хэлэгшэд-эй`** | 4.5 | `хэлэгшэд` |
-| таряашадай | **`таряашад-ай`** | 4.5 | `таряашад` |
-| магадлалай | **`магадлал-ай`** | 4.5 | `магадлал` |
-| тогтоһоные | **`тогтоһон-ые`** | 4.5 | `тогтоһон` |
-| буурсагые | **`буурсаг-ые`** | 4.5 | `буурсаг` |
-| юумэнүүдые | **`юумэнүүд-ые`** | 4.5 | `юумэнүүд` |
-| дашинимаевай | **`дашинимаев-ай`** | 4.5 | `дашинимаев` |
-| найрамдалай | **`найрамдал-ай`** | 4.5 | `найрамдал` |
-| зохёолуудые | **`зохёолууд-ые`** | 4.5 | `зохёолууд` |
+| басаганай | **`ба-саган-ай`** | 6.0 | `саган` |
+| онсолигые | **`онсолиг-ые`** | 4.5 | `онсолиг` |
+| гибралтарай | **`гибралтар-ай`** | 4.5 | `гибралтар` |
+| оронуудаа | **`оронууд-аа`** | 4.5 | `оронууд` |
+| туристуудай | **`туристууд-ай`** | 4.5 | `туристууд` |
+| эблэрэлэй | **`эблэрэл-эй`** | 4.5 | `эблэрэл` |
+| шалгалтые | **`шалгалт-ые`** | 4.5 | `шалгалт` |
+| шулуунуудые | **`шулуунууд-ые`** | 4.5 | `шулуунууд` |
+| хүсэнүүдые | **`хүсэнүүд-ые`** | 4.5 | `хүсэнүүд` |
+| бэшэхэдэнь | **`бэшэхэдэ-нь`** | 4.5 | `бэшэхэдэ` |
+| хубилбаринь | **`хубилбари-нь`** | 4.5 | `хубилбари` |
+| үзүүрнүүдые | **`үзүүрнүүд-ые`** | 4.5 | `үзүүрнүүд` |
+| моринойнь | **`мориной-нь`** | 4.5 | `мориной` |
+| реализмын | **`реализм-ын`** | 4.5 | `реализм` |
+| сэрэгүүдые | **`сэрэгүүд-ые`** | 4.5 | `сэрэгүүд` |
 
 ### 6.6 Linguistic Interpretation
 
 > **Automated Insight:**
-The language BXR appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+The language Russia Buriat shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
 
 ---
 ## 7. Summary & Recommendations
@@ -509,7 +550,7 @@ The language BXR appears to be more isolating or has a highly fixed vocabulary.
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **64k BPE** | Best compression (4.39x) |
+| Tokenizer | **64k BPE** | Best compression (4.40x) |
 | N-gram | **2-gram** | Lowest perplexity (452) |
 | Markov | **Context-4** | Highest predictability (98.9%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
@@ -725,4 +766,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 09:00:32*
+*Report Date: 2026-01-03 19:55:46*

models/embeddings/aligned/bxr_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:172f899284639436ddce6499851ed3eed6da42fc9a384c3b5308803d0e390be5
+size 1038708787

models/embeddings/aligned/bxr_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "bxr", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/bxr_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c785d969d3099dc4bd076c150506f7c27c81cfb77cc44d85ffc4161801ca957f
+size 65664

models/embeddings/aligned/bxr_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "bxr",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 2377,
+  "vocab_size": 14055
+}

models/embeddings/aligned/bxr_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a57d92bbd9734f329c9a94b4a167c386486deabe8011adb84b0dc6b41665fdc
+size 259914547

models/embeddings/aligned/bxr_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "bxr", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/bxr_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d84d55644d798c788da0b7a35dc71d4d28e0af9ddac8942ed99182153b9022e9
+size 4224

models/embeddings/aligned/bxr_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "bxr",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 2377,
+  "vocab_size": 14055
+}

models/embeddings/aligned/bxr_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31797d27ef08af5cfc33cd37eb6403b8830b43a2384cee115e2187e33f6f3567
+size 519512627

models/embeddings/aligned/bxr_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "bxr", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/bxr_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:296853c47c6dca141665279cf92cd6239125203b235581ebbb8b4e0df0d4f179
+size 16512

models/embeddings/aligned/bxr_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "bxr",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 2377,
+  "vocab_size": 14055
+}

models/embeddings/monolingual/bxr_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9a68f121ed60ec837dcd0cd6dfe40ccae7bda27cefb1882915867e69840e1104
-size 1038925515
+oid sha256:172f899284639436ddce6499851ed3eed6da42fc9a384c3b5308803d0e390be5
+size 1038708787

models/embeddings/monolingual/bxr_128d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 128
   },
-  "vocab_size": 14262
+  "vocab_size": 14055
 }

models/embeddings/monolingual/bxr_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bf82d275c6ec9e3300c42d4bf6f1d83083710befa2f67e76dcf383dcfb7c187a
-size 259972299
+oid sha256:8a57d92bbd9734f329c9a94b4a167c386486deabe8011adb84b0dc6b41665fdc
+size 259914547

models/embeddings/monolingual/bxr_32d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 32
   },
-  "vocab_size": 14262
+  "vocab_size": 14055
 }

models/embeddings/monolingual/bxr_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:846d84f5d2248a39c8ed392e72d06b7956f7871585cd833febab2c57b28b0799
-size 519623371
+oid sha256:31797d27ef08af5cfc33cd37eb6403b8830b43a2384cee115e2187e33f6f3567
+size 519512627

models/embeddings/monolingual/bxr_64d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 64
   },
-  "vocab_size": 14262
+  "vocab_size": 14055
 }

models/subword_markov/bxr_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8ef28afcd8620169ed8fa2c34876e8625f7c7d635864fbd2796f37f30dba4cf1
-size 102324
+oid sha256:293e9e02dce724d7d299cbe107a10523d43b37eb72da8462882e0eac348ac329
+size 101727

models/subword_markov/bxr_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "bxr",
-  "unique_contexts": 2141,
-  "total_transitions": 3947454
+  "unique_contexts": 2131,
+  "total_transitions": 3900304
 }

models/subword_markov/bxr_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e8eeb9396bbe777fa33dbcb0a026a02660f3421472954259ef5fb21e30676aba
-size 526873
+oid sha256:6a615345c8b10c613c94aab3e7c6fb5f2734e848c8b331ca146732c286324642
+size 513516

models/subword_markov/bxr_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "bxr",
-  "unique_contexts": 12176,
-  "total_transitions": 3944697
+  "unique_contexts": 12123,
+  "total_transitions": 3897537
 }

models/subword_markov/bxr_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:251c7577b1bff07e9bd8f78d3e3413f809f22133920660f8ff8fd5953d1df234
-size 1738691
+oid sha256:a9ee2902f7aaf5d3edd1ea0dc81f7cfb62b6abca7e206fc0397c007629224ba4
+size 1722195

models/subword_markov/bxr_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "bxr",
-  "unique_contexts": 61348,
-  "total_transitions": 3941940
+  "unique_contexts": 61012,
+  "total_transitions": 3894770
 }

models/subword_markov/bxr_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e6704187c11e7bf62086d0438a41d230986b7075412d814372c74e9c42a43bb8
-size 4955182
+oid sha256:5e580fc7987ee03846340336c7e0ab11562bc1fc258080cffed351b5d8f6d789
+size 4915742

models/subword_markov/bxr_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "bxr",
-  "unique_contexts": 230966,
-  "total_transitions": 3939183
+  "unique_contexts": 229261,
+  "total_transitions": 3892003
 }

models/subword_ngram/bxr_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4dd1815af3d79add27e46b93017d57f0b62838b802a873a7a07c137fb6540c3d
-size 51809
+oid sha256:8c970573bc00321c29090c3e39278271d9babe2c7ebe7651224fe27e2403e724
+size 51663

models/subword_ngram/bxr_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "bxr",
-  "unique_ngrams": 3823,
-  "total_ngrams": 3947454
+  "unique_ngrams": 3815,
+  "total_ngrams": 3900304
 }

models/subword_ngram/bxr_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:63257791e85ed8eaf2b882c6823a3a702fe1a0ea07f2c59e64c50dd97653a352
-size 376817
+oid sha256:53f60887f7680b105d429d09e3e4b7131fca53225c28baa8a7770f72f960f665
+size 376607

models/subword_ngram/bxr_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "bxr",
-  "unique_ngrams": 29340,
-  "total_ngrams": 3944697
+  "unique_ngrams": 29176,
+  "total_ngrams": 3897537
 }

models/subword_ngram/bxr_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7fc6a6ba4fa3d0662727116dddfc49d4ebb04b8fe918900b45bddaff9a5e475c
-size 1528848
+oid sha256:d8fa2736d561ab20de5bd87197ef8df71272258581c81664b282841815e95d5d
+size 1520439

models/subword_ngram/bxr_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "bxr",
-  "unique_ngrams": 124835,
-  "total_ngrams": 3941940
+  "unique_ngrams": 123764,
+  "total_ngrams": 3894770
 }

models/subword_ngram/bxr_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:04b1c6cdffcae19db3787c20a35580f9838d314ef2c046bce0608922d9539fdb
+size 3080903

models/subword_ngram/bxr_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "bxr",
+  "unique_ngrams": 234708,
+  "total_ngrams": 3892003
+}

models/tokenizer/bxr_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9ec3513a9e102b31e37ebbe9c74465e68178a75caa31640e2da4dd18964909d1
-size 572848
+oid sha256:a6466fb53efc61a06539c545999886f09575ed8148f8ba5aa60ba12484ecf9b7
+size 573527

models/tokenizer/bxr_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e83a11fb6d2c68714a56a8f79737f3bdf4020f6741555062a3277802b3b50563
-size 936687
+oid sha256:ba98ca87d23060587cc040f5d457fe1e109a3f6624d46e586d436f50d6b2e882
+size 936516

models/tokenizer/bxr_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1a9a5aa8bf4ae1cf3f260d408856d0de604cee508f58aa294462236f70101270
-size 1699640
+oid sha256:c684b9c660cac22c56d1016985f83e4dd024a78bce0217470464e28a93c9b151
+size 1700593

models/tokenizer/bxr_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8bb41ea3e440a8f23950afc663ca4194ddb8fe1cbbfd3fc96c5e7a3187bdb9a0
-size 400889
+oid sha256:e73d24aa19dde68bc2e0c4c40fc5f45d0509dd052bb2223834139dce115379b6
+size 401028

models/vocabulary/bxr_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:71921497bda1a7f95ca89688025ce27f6680e9797e7176d62d5eeb5de52180fa
-size 687102
+oid sha256:80f255597637065e8418da1b760b1e22f97981a2723c96db42725d30ff084b8b
+size 675289

models/vocabulary/bxr_vocabulary_metadata.json

@@ -1,17 +1,17 @@
 {
   "language": "bxr",
-  "vocabulary_size": 36185,
+  "vocabulary_size": 35751,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.16949986693740954,
+    "type_token_ratio": 0.17000503940147416,
     "coverage": {
-      "top_100": 0.19864421979752614,
-      "top_1000": 0.46809049714371126,
-      "top_5000": 0.6691097930421025,
-      "top_10000": 0.7539690930235101
+      "top_100": 0.1989179131340093,
+      "top_1000": 0.469338103228355,
+      "top_5000": 0.6704471407395921,
+      "top_10000": 0.7549872538215461
     },
-    "hapax_count": 56805,
-    "hapax_ratio": 0.6108721367889021,
-    "total_documents": 2757
+    "hapax_count": 56346,
+    "hapax_ratio": 0.6118114596566664,
+    "total_documents": 2767
   }
 }

models/word_markov/bxr_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8d83d9d3e8e86f09c7979a3b6839e87cbf8a1c801cae88a26b8e2568e80c0af1
-size 4628667
+oid sha256:ba2bcae0c44eed43d2c7e7a02dab530b098ba46720f12a947d40c50c626b6744
+size 4569472

models/word_markov/bxr_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "bxr",
-  "unique_contexts": 92909,
-  "total_transitions": 545857
+  "unique_contexts": 92015,
+  "total_transitions": 538964
 }

models/word_markov/bxr_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:08a52284e5b3cf3a5d895964ac029e73604bb505096a1643018fcbce3b7f0871
-size 11044627
+oid sha256:aa645f6c7ca05c76ed36c372451efd8f9c8eaf4ef88fdbe687946397959ae364
+size 10866637

models/word_markov/bxr_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "bxr",
-  "unique_contexts": 383260,
-  "total_transitions": 543100
+  "unique_contexts": 378037,
+  "total_transitions": 536197
 }