Upload all models and assets for be (latest)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: be
-language_name: BE
+language_name: Belarusian
 language_family: slavic_east
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-slavic_east
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.769
+    value: 4.771
   - name: best_isotropy
     type: isotropy
-    value: 0.6512
+    value: 0.6444
   - name: vocabulary_size
     type: vocab
     value: 0
-generated: 2026-01-03
+generated: 2026-01-06
 ---
 
-# BE - Wikilangs Models
+# Belarusian - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **BE** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Belarusian** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -60,7 +70,7 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
 - [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
@@ -80,47 +90,47 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.593x | 3.60 | 0.0487% | 287,700 |
-| **16k** | 4.036x | 4.04 | 0.0547% | 256,163 |
-| **32k** | 4.451x | 4.46 | 0.0603% | 232,280 |
-| **64k** | 4.769x 🏆 | 4.77 | 0.0646% | 216,795 |
+| **8k** | 3.599x | 3.60 | 0.0489% | 286,335 |
+| **16k** | 4.042x | 4.05 | 0.0549% | 254,965 |
+| **32k** | 4.455x | 4.46 | 0.0605% | 231,292 |
+| **64k** | 4.771x 🏆 | 4.78 | 0.0648% | 215,975 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Грынчэнкавэ () — вёска ў Ахтырскім раёне Сумскай вобласці Украіны. Уваходзіць у ...`
+**Sample 1:** `Ланавычы () — вёска ў Самбірскім раёне Львоўскай вобласці Украіны. Крыніцы пункт...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁гры н чэн ка вэ ▁() ▁— ▁вёска ▁ў ▁ах ... (+23 more)` | 33 |
-| 16k | `▁грын чэнка вэ ▁() ▁— ▁вёска ▁ў ▁ах ты рскім ... (+21 more)` | 31 |
-| 32k | `▁грын чэнка вэ ▁() ▁— ▁вёска ▁ў ▁ахты рскім ▁раёне ... (+19 more)` | 29 |
-| 64k | `▁грын чэнка вэ ▁() ▁— ▁вёска ▁ў ▁ахтырскім ▁раёне ▁сумскай ... (+17 more)` | 27 |
+| 8k | `▁ла на вы чы ▁() ▁— ▁вёска ▁ў ▁сам бі ... (+12 more)` | 22 |
+| 16k | `▁ла на вы чы ▁() ▁— ▁вёска ▁ў ▁сам бі ... (+12 more)` | 22 |
+| 32k | `▁ла на вычы ▁() ▁— ▁вёска ▁ў ▁самбі рскім ▁раёне ... (+9 more)` | 19 |
+| 64k | `▁лана вычы ▁() ▁— ▁вёска ▁ў ▁самбірскім ▁раёне ▁львоўскай ▁вобласці ... (+6 more)` | 16 |
 
-**Sample 2:** `Лугавэ () — вёска ў Бродыўскім раёне Львоўскай вобласці Украіны. Крыніцы пункты ...`
+**Sample 2:** `Марсо () — французскае прозвішча. Вядомыя носьбіты Марсель Марсо, французскі арт...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁луга вэ ▁() ▁— ▁вёска ▁ў ▁б роды ўскім ▁раёне ... (+15 more)` | 25 |
-| 16k | `▁луга вэ ▁() ▁— ▁вёска ▁ў ▁б роды ўскім ▁раёне ... (+15 more)` | 25 |
-| 32k | `▁луга вэ ▁() ▁— ▁вёска ▁ў ▁броды ўскім ▁раёне ▁львоўскай ... (+13 more)` | 23 |
-| 64k | `▁луга вэ ▁() ▁— ▁вёска ▁ў ▁бродыўскім ▁раёне ▁львоўскай ▁вобласці ... (+11 more)` | 21 |
+| 8k | `▁мар со ▁() ���— ▁француз скае ▁прозвішча . ▁вядомыя ▁носьбіты ... (+17 more)` | 27 |
+| 16k | `▁мар со ▁() ▁— ▁француз скае ▁прозвішча . ▁вядомыя ▁носьбіты ... (+16 more)` | 26 |
+| 32k | `▁мар со ▁() ▁— ▁француз скае ▁прозвішча . ▁вядомыя ▁носьбіты ... (+15 more)` | 25 |
+| 64k | `▁мар со ▁() ▁— ▁французскае ▁прозвішча . ▁вядомыя ▁носьбіты ▁марсель ... (+14 more)` | 24 |
 
-**Sample 3:** `Косарэвэ () — вёска ў Млыніўскім раёне Ровенскай вобласці Украіны. Уваходзіць у ...`
+**Sample 3:** `Вораніў () — вёска ў Гарадэнкіўскім раёне Івана-Франкоўскай вобласці Украіны. Кр...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ко са рэ вэ ▁() ▁— ▁вёска ▁ў ▁млы ніў ... (+21 more)` | 31 |
-| 16k | `▁ко са рэ вэ ▁() ▁— ▁вёска ▁ў ▁млы ніўскім ... (+19 more)` | 29 |
-| 32k | `▁коса рэ вэ ▁() ▁— ▁вёска ▁ў ▁млы ніўскім ▁раёне ... (+17 more)` | 27 |
-| 64k | `▁коса рэ вэ ▁() ▁— ▁вёска ▁ў ▁млыніўскім ▁раёне ▁ровенскай ... (+15 more)` | 25 |
+| 8k | `▁вора ніў ▁() ▁— ▁вёска ▁ў ▁гарад эн кі ўскім ... (+21 more)` | 31 |
+| 16k | `▁вора ніў ▁() ▁— ▁вёска ▁ў ▁гарад эн кіўскім ▁раёне ... (+18 more)` | 28 |
+| 32k | `▁вора ніў ▁() ▁— ▁вёска ▁ў ▁гарад эн кіўскім ▁раёне ... (+17 more)` | 27 |
+| 64k | `▁вора ніў ▁() ▁— ▁вёска ▁ў ▁гарад эн кіўскім ▁раёне ... (+17 more)` | 27 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.769x compression
-- **Lowest UNK Rate:** 8k with 0.0487% unknown tokens
+- **Best Compression:** 64k achieves 4.771x compression
+- **Lowest UNK Rate:** 8k with 0.0489% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -137,12 +147,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
 |--------|---------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | Word | 114,899 | 16.81 | 1,095,876 | 11.4% | 25.2% |
-| **2-gram** | Subword | 453 🏆 | 8.82 | 15,607 | 55.9% | 96.8% |
-| **3-gram** | Word | 176,550 | 17.43 | 1,682,544 | 11.7% | 25.2% |
-| **3-gram** | Subword | 4,192 | 12.03 | 145,836 | 18.7% | 59.5% |
-| **4-gram** | Word | 286,677 | 18.13 | 2,809,290 | 9.5% | 25.0% |
-| **4-gram** | Subword | 25,337 | 14.63 | 930,596 | 8.0% | 29.4% |
+| **2-gram** | Word | 115,602 | 16.82 | 1,101,685 | 11.4% | 25.2% |
+| **2-gram** | Subword | 453 🏆 | 8.82 | 15,623 | 55.9% | 96.8% |
+| **3-gram** | Word | 178,210 | 17.44 | 1,692,602 | 11.7% | 25.1% |
+| **3-gram** | Subword | 4,191 | 12.03 | 146,010 | 18.7% | 59.5% |
+| **4-gram** | Word | 289,150 | 18.14 | 2,823,610 | 9.4% | 24.9% |
+| **4-gram** | Subword | 25,327 | 14.63 | 932,448 | 8.0% | 29.4% |
+| **5-gram** | Word | 212,986 | 17.70 | 2,118,708 | 8.7% | 25.2% |
+| **5-gram** | Subword | 104,621 | 16.67 | 3,234,164 | 4.5% | 17.2% |
 
 ### Top 5 N-grams by Size
 
@@ -151,19 +163,19 @@ Below are sample sentences tokenized with each vocabulary size:
 | Rank | N-gram | Count |
 |------|--------|-------|
 | 1 | `0 10` | 188,589 |
-| 2 | `10 0` | 184,433 |
-| 3 | `0 09` | 178,218 |
-| 4 | `09 0` | 172,686 |
-| 5 | `у годзе` | 140,117 |
+| 2 | `10 0` | 184,434 |
+| 3 | `0 09` | 178,217 |
+| 4 | `09 0` | 172,685 |
+| 5 | `у годзе` | 141,829 |
 
 **3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `0 10 0` | 183,056 |
-| 2 | `0 09 0` | 171,686 |
-| 3 | `0 11 0` | 133,046 |
-| 4 | `0 08 0` | 125,664 |
+| 1 | `0 10 0` | 183,055 |
+| 2 | `0 09 0` | 171,685 |
+| 3 | `0 11 0` | 133,047 |
+| 4 | `0 08 0` | 125,665 |
 | 5 | `0 07 0` | 84,761 |
 
 **4-grams (Word):**
@@ -176,42 +188,62 @@ Below are sample sentences tokenized with each vocabulary size:
 | 4 | `47 0 10 0` | 26,709 |
 | 5 | `0 50 0 10` | 26,628 |
 
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `0 44 0 10 0` | 27,892 |
+| 2 | `0 47 0 10 0` | 26,707 |
+| 3 | `0 50 0 10 0` | 26,249 |
+| 4 | `0 45 0 10 0` | 25,524 |
+| 5 | `0 49 0 10 0` | 24,716 |
+
 **2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `а _` | 7,375,676 |
-| 2 | `н а` | 5,829,339 |
-| 3 | `р а` | 5,735,773 |
-| 4 | `к а` | 4,959,811 |
-| 5 | `_ п` | 4,750,427 |
+| 1 | `а _` | 7,411,164 |
+| 2 | `н а` | 5,858,867 |
+| 3 | `р а` | 5,764,007 |
+| 4 | `к а` | 4,983,576 |
+| 5 | `_ п` | 4,779,657 |
 
 **3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `_ п а` | 2,102,007 |
-| 2 | `_ 0 ,` | 1,872,298 |
-| 3 | `_ н а` | 1,670,363 |
-| 4 | `н а _` | 1,424,587 |
-| 5 | `_ п р` | 1,341,590 |
+| 1 | `_ п а` | 2,113,963 |
+| 2 | `_ 0 ,` | 1,872,411 |
+| 3 | `_ н а` | 1,678,358 |
+| 4 | `н а _` | 1,430,853 |
+| 5 | `_ п р` | 1,351,115 |
 
 **4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `а г а _` | 980,628 |
-| 2 | `_ п р а` | 746,402 |
-| 3 | `_ г о д` | 708,921 |
-| 4 | `_ н а _` | 692,237 |
-| 5 | `к а й _` | 545,902 |
+| 1 | `а г а _` | 985,197 |
+| 2 | `_ п р а` | 752,091 |
+| 3 | `_ г о д` | 714,067 |
+| 4 | `_ н а _` | 694,537 |
+| 5 | `к а й _` | 548,513 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `к а г а _` | 467,479 |
+| 2 | `с к а й _` | 409,977 |
+| 3 | `с к а г а` | 393,058 |
+| 4 | `б е л а р` | 392,561 |
+| 5 | `е л а р у` | 392,043 |
 
 
 ### Key Findings
 
 - **Best Perplexity:** 2-gram (subword) with 453
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~29% of corpus
+- **Coverage:** Top-1000 patterns cover ~17% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -227,14 +259,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
 |---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.9806 | 1.973 | 10.65 | 1,594,726 | 1.9% |
-| **1** | Subword | 0.4731 | 1.388 | 3.96 | 16,459 | 52.7% |
-| **2** | Word | 0.3129 | 1.242 | 1.94 | 16,955,773 | 68.7% |
-| **2** | Subword | 0.6387 | 1.557 | 4.81 | 65,143 | 36.1% |
-| **3** | Word | 0.1126 | 1.081 | 1.23 | 32,878,014 | 88.7% |
-| **3** | Subword | 0.8192 | 1.764 | 4.91 | 313,186 | 18.1% |
-| **4** | Word | 0.0455 🏆 | 1.032 | 1.08 | 40,250,681 | 95.5% |
-| **4** | Subword | 0.7603 | 1.694 | 3.75 | 1,537,647 | 24.0% |
+| **1** | Word | 0.9802 | 1.973 | 10.66 | 1,600,794 | 2.0% |
+| **1** | Subword | 0.4743 | 1.389 | 3.96 | 16,475 | 52.6% |
+| **2** | Word | 0.3132 | 1.242 | 1.95 | 17,028,048 | 68.7% |
+| **2** | Subword | 0.6391 | 1.557 | 4.81 | 65,298 | 36.1% |
+| **3** | Word | 0.1128 | 1.081 | 1.23 | 33,045,925 | 88.7% |
+| **3** | Subword | 0.8191 | 1.764 | 4.91 | 313,830 | 18.1% |
+| **4** | Word | 0.0455 🏆 | 1.032 | 1.08 | 40,473,004 | 95.4% |
+| **4** | Subword | 0.7606 | 1.694 | 3.75 | 1,541,159 | 23.9% |
 
 ### Generated Text Samples (Word-based)
 
@@ -242,27 +274,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `0 57 0 09 0 67 0 07 0 58 км на 1 20 лютага жэнева`
-2. `і стаўшы першым урадзе і гітарыст разам з поўдня сутыкненні прыпыніліся на кіргізскай сср 10 0`
-3. `у годзе гэтыя эксперыменты па год 11 0 56 0 75 0 50 0 08 0`
+1. `0 06 0 1 мінскай вобласці беларусі ў раёне віцебскай губерні земскага самакіравання якая выказалася ...`
+2. `і дзіцячы сад каралевы якія выменьвалі ў эджбастане бірмінгем сіці манчэстэр юнайтэд дзе адносна нев...`
+3. `у годзе стала ўскосным выглядзе шоу consecința istorică sibiu mitropolitul andrei yahorau alena маё ...`
 
 **Context Size 2:**
 
-1. `0 10 0 50 0 10 0 39 0 11 0 36 0 12 0 54 0`
-2. `10 0 68 0 25 0 6 1 52 1 25 джэсіка пегула эна сібахара 7 6`
-3. `0 09 0 46 0 10 0 35 0 12 0 37 0 12 0 д2 прамень`
+1. `0 10 0 34 0 12 0 38 0 11 0 53 0 09 0 41 0`
+2. `10 0 55 0 09 0 46 0 10 0 63 0 08 0 75 0 07`
+3. `0 09 0 54 0 09 0 47 0 10 0 48 0 10 0 45 0`
 
 **Context Size 3:**
 
-1. `0 10 0 37 0 12 0 35 0 48 0 10 0 56 0 09 0 51`
-2. `0 09 0 37 0 12 0 57 0 09 0 41 0 11 0 45 0 10`
-3. `0 11 0 42 0 11 0 61 0 08 0 51 0 09 0 37 0 12`
+1. `0 10 0 37 0 12 0 45 0 10 0 60 0 08 0 58 0 09`
+2. `0 09 0 54 0 09 0 50 0 09 so a 0 67 0 08 0 79`
+3. `0 11 0 47 0 10 0 54 0 09 0 48 0 10 0 43 0 11`
 
 **Context Size 4:**
 
-1. `0 44 0 10 0 52 0 09 0 43 0 11 0 76 0 07 0 37 0`
-2. `44 0 10 0 51 0 09 0 51 0 09 0 42 0 11 0 60 0 08`
-3. `0 47 0 10 0 54 0 09 0 65 0 08 0 38 0 11 0 46 0`
+1. `0 44 0 10 0 40 0 11 0 54 0 32 0 45 0 32 0 56 0`
+2. `44 0 10 0 47 0 10 0 48 0 10 0 48 0 10 0 57 0 06`
+3. `0 47 0 10 0 54 0 09 0 87 0 06 sbbc 0 78 0 07 0 47`
 
 
 ### Generated Text Samples (Subword-based)
@@ -271,34 +303,34 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_irone_саджырода`
-2. `аса._бетвекаенсы`
-3. `ных_г._тэні_09_��`
+1. `_бек»_мано_szk._`
+2. `аёрларныкльбеніц`
+3. `нагркаў_вай_stol`
 
 **Context Size 2:**
 
-1. `а_абто_чальны,_пр`
-2. `наяны_нькімпіныма`
-3. `раён_з_10),_якаге`
+1. `а_вылкі_ў_парышша`
+2. `на_апілік_вы,_які`
+3. `раў_звагарскаў_вы`
 
 **Context Size 3:**
 
-1. `_паднакадэміі_пало`
-2. `_0,40_0,56_0,50_0,`
-3. `_на_паданні._перац`
+1. `_памка:_ю._тайскаг`
+2. `_0,53_0,42_0,43_0,`
+3. `_насцю_і_тавіч_см.`
 
 **Context Size 4:**
 
-1. `ага_адсек_нацыя_4_т`
-2. `_пра_ў_сваюць_62-я_`
-3. `_годзе._жывяць_дызе`
+1. `ага_заняў_і_паведа,`
+2. `_прасійскаў_супольс`
+3. `_годзе_прыезда_філь`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 (word) with 95.5% predictability
+- **Best Predictability:** Context-4 (word) with 95.4% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (1,537,647 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (1,541,159 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -314,48 +346,48 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 739,605 |
-| Total Tokens | 54,963,738 |
-| Mean Frequency | 74.31 |
+| Vocabulary Size | 741,819 |
+| Total Tokens | 55,243,342 |
+| Mean Frequency | 74.47 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 3865.57 |
+| Frequency Std Dev | 3873.91 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | 0 | 1,944,698 |
-| 2 | і | 1,322,186 |
-| 3 | у | 1,231,156 |
-| 4 | ў | 1,155,870 |
-| 5 | з | 858,124 |
-| 6 | на | 705,989 |
-| 7 | года | 365,156 |
-| 8 | да | 288,350 |
-| 9 | годзе | 255,744 |
-| 10 | 10 | 239,762 |
+| 1 | 0 | 1,944,910 |
+| 2 | і | 1,331,350 |
+| 3 | у | 1,238,468 |
+| 4 | ў | 1,161,043 |
+| 5 | з | 862,221 |
+| 6 | на | 708,262 |
+| 7 | года | 367,568 |
+| 8 | да | 290,434 |
+| 9 | годзе | 258,378 |
+| 10 | 10 | 239,964 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | іцуно | 2 |
-| 2 | міурай | 2 |
-| 3 | kodanshas | 2 |
-| 4 | llb | 2 |
-| 5 | давы́даўскае | 2 |
-| 6 | эльханон | 2 |
-| 7 | vilner | 2 |
-| 8 | emes | 2 |
-| 9 | folkstsaytung | 2 |
-| 10 | dertseyln | 2 |
+| 1 | девятке | 2 |
+| 2 | дэкунаў | 2 |
+| 3 | iovine | 2 |
+| 4 | іавін | 2 |
+| 5 | аёвіну | 2 |
+| 6 | джэніка | 2 |
+| 7 | мэрылінам | 2 |
+| 8 | сардэшная | 2 |
+| 9 | івасю | 2 |
+| 10 | стеценко | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
 | Zipf Coefficient | 0.9714 |
-| R² (Goodness of Fit) | 0.997385 |
+| R² (Goodness of Fit) | 0.997383 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
@@ -371,7 +403,7 @@ Below are text samples generated from each subword-based Markov chain model:
 
 - **Zipf Compliance:** R²=0.9974 indicates excellent adherence to Zipf's law
 - **High Frequency Dominance:** Top 100 words cover 29.3% of corpus
-- **Long Tail:** 729,605 words needed for remaining 25.5% coverage
+- **Long Tail:** 731,819 words needed for remaining 25.5% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -387,37 +419,40 @@ Below are text samples generated from each subword-based Markov chain model:
 
 ### 5.1 Cross-Lingual Alignment
 
-> *Note: Multilingual alignment visualization not available for this language.*
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
 
 
 ### 5.2 Model Comparison
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.6148 | 0.3550 | N/A | N/A |
-| **mono_64d** | 64 | 0.6479 | 0.2915 | N/A | N/A |
-| **mono_128d** | 128 | 0.6512 🏆 | 0.2220 | N/A | N/A |
+| **mono_32d** | 32 | 0.6096 | 0.3533 | N/A | N/A |
+| **mono_64d** | 64 | 0.6408 | 0.2859 | N/A | N/A |
+| **mono_128d** | 128 | 0.6444 | 0.2271 | N/A | N/A |
+| **aligned_32d** | 32 | 0.6096 | 0.3568 | 0.0440 | 0.3040 |
+| **aligned_64d** | 64 | 0.6408 | 0.2908 | 0.1380 | 0.5080 |
+| **aligned_128d** | 128 | 0.6444 🏆 | 0.2362 | 0.2300 | 0.6220 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_128d with 0.6512 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.2895. Lower values indicate better semantic separation.
-- **Alignment Quality:** No aligned models evaluated in this run.
+- **Best Isotropy:** aligned_128d with 0.6444 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2917. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 23.0% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
 ## 6.  Morphological Analysis (Experimental)
 
-> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
-
 This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
 
 ### 6.1 Productivity & Complexity
 
 | Metric | Value | Interpretation | Recommendation |
 |--------|-------|----------------|----------------|
-| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
-| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **0.467** | High formulaic/idiomatic content | - |
 
 ### 6.2 Affix Inventory (Productive Units)
 
@@ -426,21 +461,21 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Prefixes
 | Prefix | Examples |
 |--------|----------|
-| `-ка` | каганаў, кайлі, карэлятыўных |
-| `-па` | пасуэлу, падую, паліцыянтаў |
-| `-пр` | протестантами, провозглашении, принципу |
+| `-па` | параллельной, падаплёка, падкіданні |
+| `-ка` | канавалава, кафедрамі, калеснікава |
+| `-пр` | прышчэпаўшчына, прыпяцкі, прапіткі |
 
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-а` | кішскага, краснасельскага, апельсіна |
-| `-кі` | ліпнякі, чарашкі, вярцінскі |
-| `-га` | кішскага, краснасельскага, луэнга |
-| `-ай` | абнаўленчай, пустэльніцай, факталагічнай |
-| `-ага` | кішскага, краснасельскага, найбагацейшага |
-| `-мі` | неадмоўнымі, контурамі, абрамі |
-| `-ая` | наватухінская, загорская, чакаўская |
-| `-ыя` | шматбаковыя, перанятыя, узбагачаныя |
+| `-а` | гароха, прышчэпаўшчына, падаплёка |
+| `-га` | паўднёвага, іпацеўскага, міжазёрнага |
+| `-кі` | леанінскі, прыпяцкі, прапіткі |
+| `-ай` | кіянкай, ольстэрскай, найноўшай |
+| `-ага` | паўднёвага, іпацеўскага, міжазёрнага |
+| `-ая` | рудэральная, прымененая, свальбардская |
+| `-аў` | шакіраваў, вігаў, шукальнікаў |
+| `-на` | прышчэпаўшчына, непэсрэдна, скампанавана |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -448,18 +483,18 @@ Bound stems are high-frequency subword units that are semantically cohesive but
 
 | Stem | Cohesion | Substitutability | Examples |
 |------|----------|------------------|----------|
-| `насц` | 1.82x | 190 contexts | насцю, насць, насці |
-| `елар` | 2.47x | 46 contexts | белар, гелар, келар |
-| `анск` | 1.35x | 1021 contexts | ганск, данск, канск |
-| `асел` | 2.07x | 87 contexts | расел, насел, асель |
-| `нскі` | 1.43x | 414 contexts | янскі, енскі, інскі |
-| `ання` | 1.67x | 173 contexts | рання, вання, ранняе |
-| `аецц` | 2.21x | 48 contexts | ваецца, каецца, лаецца |
-| `нска` | 1.35x | 500 contexts | унска, янска, минска |
-| `ўска` | 1.52x | 236 contexts | еўска, іўска, еўская |
-| `ленн` | 1.48x | 234 contexts | гленн, ленны, ленная |
-| `йска` | 1.59x | 149 contexts | йская, ейска, войска |
-| `уска` | 1.36x | 263 contexts | буска, гуска, ускат |
+| `анск` | 1.51x | 1027 contexts | ганск, данск, канск |
+| `нска` | 1.55x | 503 contexts | унска, янска, інская |
+| `насц` | 1.79x | 190 contexts | насце, насця, насцю |
+| `асел` | 2.08x | 87 contexts | асель, аселі, расел |
+| `елар` | 2.39x | 47 contexts | белар, селар, гелар |
+| `ўска` | 1.58x | 236 contexts | еўска, іўска, ёўскае |
+| `аецц` | 2.20x | 48 contexts | маецца, каецца, лаецца |
+| `тычн` | 1.49x | 233 contexts | этычны, стычня, этычна |
+| `нскі` | 1.34x | 416 contexts | енскі, янс��і, інскі |
+| `ельн` | 1.32x | 342 contexts | ельню, ельна, ельні |
+| `ходз` | 1.47x | 182 contexts | ходзі, ходза, ходзь |
+| `ання` | 1.47x | 174 contexts | рання, вання, арання |
 
 ### 6.4 Affix Compatibility (Co-occurrence)
 
@@ -467,16 +502,16 @@ This table shows which prefixes and suffixes most frequently co-occur on the sam
 
 | Prefix | Suffix | Frequency | Examples |
 |--------|--------|-----------|----------|
-| `-ка` | `-а` | 66 words | каміна, камунізма |
-| `-па` | `-а` | 55 words | паступаленка, панінскага |
-| `-пр` | `-а` | 28 words | прыкладвацца, прынада |
-| `-па` | `-ай` | 21 words | паплаўковай, пастаяннай |
-| `-па` | `-мі` | 17 words | пасіўнымі, паказнікамі |
-| `-па` | `-кі` | 16 words | палінскі, падзьячаскі |
-| `-ка` | `-га` | 16 words | какамега, калобжагскага |
-| `-ка` | `-ага` | 15 words | калобжагскага, каламойскага |
-| `-ка` | `-кі` | 14 words | кадомскі, каўхаёкі |
-| `-ка` | `-аў` | 12 words | карыбаў, катэрынычаў |
+| `-па` | `-а` | 57 words | падлічваюцца, павета |
+| `-ка` | `-а` | 51 words | карахана, каралькова |
+| `-пр` | `-а` | 33 words | прынцэса, працягваюцца |
+| `-па` | `-ыя` | 14 words | падпружныя, пасярэбраныя |
+| `-па` | `-ай` | 14 words | паўлавіцкай, пагібельнай |
+| `-ка` | `-ая` | 14 words | карнуая, карэспандэнцкая |
+| `-ка` | `-на` | 13 words | карахана, кадрына |
+| `-ка` | `-га` | 13 words | калевальскага, каларадскага |
+| `-па` | `-кі` | 13 words | пакупкі, палачанкі |
+| `-па` | `-га` | 13 words | папаленага, палаткавага |
 
 ### 6.5 Recursive Morpheme Segmentation
 
@@ -484,26 +519,28 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 
 | Word | Suggested Split | Confidence | Stem |
 |------|-----------------|------------|------|
-| барыёнамі | **`барыё-на-мі`** | 6.0 | `барыё` |
-| курапаткіна | **`курапат-кі-на`** | 6.0 | `курапат` |
-| хакеістаў | **`хакеіст-аў`** | 4.5 | `хакеіст` |
-| навасібірская | **`навасібірск-ая`** | 4.5 | `навасібірск` |
-| пірамідаў | **`пірамід-аў`** | 4.5 | `пірамід` |
-| трансфарматараў | **`трансфарматар-аў`** | 4.5 | `трансфарматар` |
-| участковыя | **`участков-ыя`** | 4.5 | `участков` |
-| вузельчыкамі | **`вузельчыка-мі`** | 4.5 | `вузельчыка` |
-| мікрараёнаў | **`мікрараён-аў`** | 4.5 | `мікрараён` |
-| патраціць | **`па-траціць`** | 4.5 | `траціць` |
-| папоўніцца | **`па-поўніцца`** | 4.5 | `поўніцца` |
-| капашчэўскі | **`ка-па-шчэўс-кі`** | 4.5 | `шчэўс` |
-| накрыўкамі | **`накрыўка-мі`** | 4.5 | `накрыўка` |
-| наведвальніцкі | **`наведвальніц-кі`** | 4.5 | `наведвальніц` |
-| беспартыйнымі | **`беспартыйны-мі`** | 4.5 | `беспартыйны` |
+| галіцынаўка | **`галіцын-аў-ка`** | 6.0 | `галіцын` |
+| перакладчыкаў | **`перакладчык-аў`** | 4.5 | `перакладчык` |
+| зікуратаў | **`зікурат-аў`** | 4.5 | `зікурат` |
+| астраблемай | **`астраблем-ай`** | 4.5 | `астраблем` |
+| авіяатрадаў | **`авіяатрад-аў`** | 4.5 | `авіяатрад` |
+| гукарадаў | **`гукарад-аў`** | 4.5 | `гукарад` |
+| цырульнікаў | **`цырульнік-аў`** | 4.5 | `цырульнік` |
+| адпраўшчыкаў | **`адпраўшчык-аў`** | 4.5 | `адпраўшчык` |
+| рэдэмптарыстаў | **`рэдэмптарыст-аў`** | 4.5 | `рэдэмптарыст` |
+| кулінараў | **`кулінар-аў`** | 4.5 | `кулінар` |
+| іньігесаў | **`іньігес-аў`** | 4.5 | `іньігес` |
+| гэлтахтаў | **`гэлтахт-аў`** | 4.5 | `гэлтахт` |
+| рэгістрацыйна | **`рэгістрацый-на`** | 4.5 | `рэгістрацый` |
+| чапаеўскага | **`чапаеўск-ага`** | 4.5 | `чапаеўск` |
+| грунтоўка | **`грунтоў-ка`** | 4.5 | `грунтоў` |
 
 ### 6.6 Linguistic Interpretation
 
 > **Automated Insight:**
-The language BE appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+The language Belarusian shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
 
 ---
 ## 7. Summary & Recommendations
@@ -516,7 +553,7 @@ The language BE appears to be more isolating or has a highly fixed vocabulary. W
 |-----------|-------------|-----------|
 | Tokenizer | **64k BPE** | Best compression (4.77x) |
 | N-gram | **2-gram** | Lowest perplexity (453) |
-| Markov | **Context-4** | Highest predictability (95.5%) |
+| Markov | **Context-4** | Highest predictability (95.4%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
 
@@ -730,4 +767,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 11:32:17*
+*Report Date: 2026-01-06 15:57:39*

models/embeddings/aligned/be_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2398fdf9124672a6f7ed2d7a9dba453f1c6fc1da62df8fd1b78517f026d18c39
+size 1569698641

models/embeddings/aligned/be_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "be", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/be_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:96fcf5038c28eaead63c420c94055bca962719cdbb23ca63661cdc0becf7130d
+size 65664

models/embeddings/aligned/be_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "be",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 42773,
+  "vocab_size": 519797
+}

models/embeddings/aligned/be_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4755e61bf97fcc4ab93a50344a113542111b8613ffa53c5507175f2b8dae6dd
+size 402494545

models/embeddings/aligned/be_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "be", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/be_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e7612eb2b5ad863c05f6f99e62aa672d4fc162776296f2a9d41a0d77a46c676
+size 4224

models/embeddings/aligned/be_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "be",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 42773,
+  "vocab_size": 519797
+}

models/embeddings/aligned/be_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ded6f659a5f25bd1e4c2cf7f0634e7fc41467c0b8fffc1cfabec6463e11ecc8d
+size 791562577

models/embeddings/aligned/be_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "be", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/be_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3958212057a2f8723f332186599eb6d9450132cf5b6366dc313f0fd9c84760c2
+size 16512

models/embeddings/aligned/be_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "be",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 42773,
+  "vocab_size": 519797
+}

models/embeddings/monolingual/be_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e80dd83fd9b000c473bacdfc520317bc08c8e6232f6acc8ddf47a4dc636212b7
-size 1567868138
+oid sha256:2398fdf9124672a6f7ed2d7a9dba453f1c6fc1da62df8fd1b78517f026d18c39
+size 1569698641

models/embeddings/monolingual/be_128d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 128
   },
-  "vocab_size": 518052
+  "vocab_size": 519797
 }

models/embeddings/monolingual/be_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e15ec6617f84546d2951de84ffe80fbfa2280da80a7135e996e30747c163a575
-size 402004202
+oid sha256:a4755e61bf97fcc4ab93a50344a113542111b8613ffa53c5507175f2b8dae6dd
+size 402494545

models/embeddings/monolingual/be_32d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 32
   },
-  "vocab_size": 518052
+  "vocab_size": 519797
 }

models/embeddings/monolingual/be_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0dca4824861fd94e6b9de472d555ae08662bf04a8795cab1ac77097e32c191f3
-size 790625514
+oid sha256:ded6f659a5f25bd1e4c2cf7f0634e7fc41467c0b8fffc1cfabec6463e11ecc8d
+size 791562577

models/embeddings/monolingual/be_64d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 64
   },
-  "vocab_size": 518052
+  "vocab_size": 519797
 }

models/subword_markov/be_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:16ec89ccbf7b33b419dff7091cc3396dd6b9c2f2d9e7b4aaa101c1f6dc261e98
-size 528755
+oid sha256:3942a714b5a38d5d96ee158d63c1aa919d9ab86f0e43931415d4af822ef1069e
+size 534939

models/subword_markov/be_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "be",
-  "unique_contexts": 16459,
-  "total_transitions": 384276543
+  "unique_contexts": 16475,
+  "total_transitions": 386334702
 }

models/subword_markov/be_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:75a72bc43ff9fcb1e07421d9900ef838856a31dd2e997a85da1ec51c2da7313f
-size 2698586
+oid sha256:dac602100171ff68ef9af0bed9e4c36f887a6ef52a25684636d67ff1ca55a61e
+size 2719683

models/subword_markov/be_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "be",
-  "unique_contexts": 65143,
-  "total_transitions": 384021043
+  "unique_contexts": 65298,
+  "total_transitions": 386077966
 }

models/subword_markov/be_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3cc5bbbf80158973cace1739a5b2da93ae4aba1805dddfad45e13be87b4dd5b4
-size 12779069
+oid sha256:3b861f2a419cdf3ef03894e8873567b72ea96075efe4764bb551f15037bff314
+size 12801191

models/subword_markov/be_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "be",
-  "unique_contexts": 313186,
-  "total_transitions": 383765543
+  "unique_contexts": 313830,
+  "total_transitions": 385821230
 }

models/subword_markov/be_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:78d8dd2621f613c8c1d06109067bcf9cbac4f41f2929f949639de78672590907
-size 48720729
+oid sha256:53d211d50f2ccdea6ee36e7c38dca0f43324033cc38b6518b3a16d49c5e9c97a
+size 48789277

models/subword_markov/be_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "be",
-  "unique_contexts": 1537647,
-  "total_transitions": 383510043
+  "unique_contexts": 1541159,
+  "total_transitions": 385564494
 }

models/subword_ngram/be_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aafc4ee9f69f303f6f198618f5bee0cac66a99dacae147499dc0cae12854a772
-size 221209
+oid sha256:7a20ed88f731298f393820cc8e6a49e7e9ad366b29d99b05efe77c0d29a8897b
+size 221285

models/subword_ngram/be_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "be",
-  "unique_ngrams": 15607,
-  "total_ngrams": 384276543
+  "unique_ngrams": 15623,
+  "total_ngrams": 386334702
 }

models/subword_ngram/be_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:87fb73101845b2c8cdea801fcdcd4465df82baa9bba94ed1aefa8c506c088840
-size 1907996
+oid sha256:b3f393ed68e52b9087a46bac8620c06fcc6c26109ce49b55725df279ef40727b
+size 1898998

models/subword_ngram/be_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "be",
-  "unique_ngrams": 145836,
-  "total_ngrams": 384021043
+  "unique_ngrams": 146010,
+  "total_ngrams": 386077966
 }

models/subword_ngram/be_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0a1d30f2ce0acd57d7abf371e1916b606dd8f958d5fc479f3dc5736b5bb18b10
-size 12274905
+oid sha256:4830ea433f90560d2d7544d8e762d5cfc269a9c2d43405ae2bfc9b424f8022e0
+size 12284822

models/subword_ngram/be_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "be",
-  "unique_ngrams": 930596,
-  "total_ngrams": 383765543
+  "unique_ngrams": 932448,
+  "total_ngrams": 385821230
 }

models/subword_ngram/be_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62972cf284ad315d93677c8b7f92a406ee75b38d2a82ff40e134a09c75f18e04
+size 45152293

models/subword_ngram/be_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "be",
+  "unique_ngrams": 3234164,
+  "total_ngrams": 385564494
+}

models/tokenizer/be_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:008fe4df9c07918b817613d49143c9d406e08cd7c95f2c94d7e35e4d7af0322f
-size 592885
+oid sha256:4f3de32b0a7b3b3a3b69d24f6b697d9794ee59c08d0ffc70f7d561ebad1d439f
+size 592882

models/tokenizer/be_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f2db34459f167d40ce24759a3730279bf398faad2bcfe0de422d5a1ec7a70ffc
-size 969782
+oid sha256:3ecb23f7b5f2f82a8fd7aea00b3054d7ecd036fe45503b17c3ad800cc12d9bb2
+size 969548

models/tokenizer/be_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:df2ee1b2850c4e4bd93d09aa2f1f4c06b4fd62dd623170b145a36f61154961b9
-size 1751650
+oid sha256:4dfed6e10895ec9f14ab639f9cbc12b5c079d85b4ab209949758712be587f3fe
+size 1749733

models/tokenizer/be_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e07b5ee32211d68f303eb0ca2473ef5a3e47cf3d435dbe20a3f50b5e40747119
-size 410385
+oid sha256:2d29ff98b89bf26ec846ef03f8b5039985fda77c537f34a01eb209ee2abcb87d
+size 410417

models/vocabulary/be_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:eaef8e90391cc62be2430106a5b0b4c67cc2dfacdb35f517f62c46107295d042
-size 12490294
+oid sha256:60f54584eb998e8e4cec858bc3bc846ac14f0cc3cb7cb2b11da43a339eb5bfda
+size 12528391

models/vocabulary/be_vocabulary_metadata.json

@@ -1,17 +1,17 @@
 {
   "language": "be",
-  "vocabulary_size": 739605,
+  "vocabulary_size": 741819,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.028584751562556937,
+    "type_token_ratio": 0.028548400082640594,
     "coverage": {
-      "top_100": 0.28834518105660356,
-      "top_1000": 0.49802155961854777,
-      "top_5000": 0.6639092244917146,
-      "top_10000": 0.7333083111156797
+      "top_100": 0.2882919700503959,
+      "top_1000": 0.4979523531099726,
+      "top_5000": 0.663883930712732,
+      "top_10000": 0.7333078754770741
     },
-    "hapax_count": 855988,
-    "hapax_ratio": 0.5364701399417019,
-    "total_documents": 255500
+    "hapax_count": 859837,
+    "hapax_ratio": 0.5368424930197245,
+    "total_documents": 256736
   }
 }

models/word_markov/be_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e8935add6e30b042b611c05c62b5e95de82abb4595dfd2e015226bf394cfb1f0
-size 207227789
+oid sha256:e032b1d50d56ef25717d00fa1d93c643346f946c30281462d04acb7143fee7bf
+size 207708356

models/word_markov/be_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "be",
-  "unique_contexts": 1594726,
-  "total_transitions": 55564226
+  "unique_contexts": 1600794,
+  "total_transitions": 55846443
 }

models/word_markov/be_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ecade528e4ed410d8e502020cb4476eb3034bb468a559acee35d2d25b0b413e1
-size 740234356
+oid sha256:c1ae989a56b7f61469acbe2340dda964b8474df26de7089f3a1ff8a2ff624b38
+size 743701483

models/word_markov/be_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "be",
-  "unique_contexts": 16955773,
-  "total_transitions": 55308726
+  "unique_contexts": 17028048,
+  "total_transitions": 55589707
 }