Upload all models and assets for be (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 3.609
+    value: 4.769
   - name: best_isotropy
     type: isotropy
-    value: 0.6652
+    value: 0.6512
   - name: vocabulary_size
     type: vocab
-    value: 834514
-generated: 2025-12-28
+    value: 0
+generated: 2026-01-03
 ---
 
 # BE - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,57 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 2.945x | 2.88 | 0.0099% | 395,619 |
-| **16k** | 3.198x | 3.13 | 0.0107% | 364,322 |
-| **32k** | 3.434x | 3.36 | 0.0115% | 339,332 |
-| **64k** | 3.609x 🏆 | 3.53 | 0.0121% | 322,873 |
+| **8k** | 3.593x | 3.60 | 0.0487% | 287,700 |
+| **16k** | 4.036x | 4.04 | 0.0547% | 256,163 |
+| **32k** | 4.451x | 4.46 | 0.0603% | 232,280 |
+| **64k** | 4.769x 🏆 | 4.77 | 0.0646% | 216,795 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Адаі () — вёска ў Акнянскім раёне Адэскай вобласці Украіны.
-
- Крыніцы 
-
-Катэгоры...`
+**Sample 1:** `Грынчэнкавэ () — вёска ў Ахтырскім раёне Сумскай вобласці Украіны. Уваходзіць у ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ада і ▁() ▁— ▁вёска ▁ў ▁ак ня нскім ▁раёне ... (+14 more)` | 24 |
-| 16k | `▁ада і ▁() ▁— ▁вёска ▁ў ▁ак ня нскім ▁раёне ... (+13 more)` | 23 |
-| 32k | `▁ада і ▁() ▁— ▁вёска ▁ў ▁ак ня нскім ▁раёне ... (+12 more)` | 22 |
-| 64k | `▁ада і ▁() ▁— ▁вёска ▁ў ▁ак нянскім ▁раёне ▁адэскай ... (+11 more)` | 21 |
+| 8k | `▁гры н чэн ка вэ ▁() ▁— ▁вёска ▁ў ▁ах ... (+23 more)` | 33 |
+| 16k | `▁грын чэнка вэ ▁() ▁— ▁вёска ▁ў ▁ах ты рскім ... (+21 more)` | 31 |
+| 32k | `▁грын чэнка вэ ▁() ▁— ▁вёска ▁ў ▁ахты рскім ▁раёне ... (+19 more)` | 29 |
+| 64k | `▁грын чэнка вэ ▁() ▁— ▁вёска ▁ў ▁ахтырскім ▁раёне ▁сумскай ... (+17 more)` | 27 |
 
-**Sample 2:** `Назву Асманіе маюць: горад і правінцыя ў Турцыі.`
+**Sample 2:** `Лугавэ () — вёска ў Бродыўскім раёне Львоўскай вобласці Украіны. Крыніцы пункты ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁назву ▁ас мані е ▁маюць : ▁горад ▁і ▁правінцыя ▁ў ... (+2 more)` | 12 |
-| 16k | `▁назву ▁ас мані е ▁маюць : ▁горад ▁і ▁правінцыя ▁ў ... (+2 more)` | 12 |
-| 32k | `▁назву ▁ас мані е ▁маюць : ▁горад ▁і ▁правінцыя ▁ў ... (+2 more)` | 12 |
-| 64k | `▁назву ▁ас мані е ▁маюць : ▁горад ▁і ▁правінцыя ▁ў ... (+2 more)` | 12 |
+| 8k | `▁луга вэ ▁() ▁— ▁вёска ▁ў ▁б роды ўскім ▁раёне ... (+15 more)` | 25 |
+| 16k | `▁луга вэ ▁() ▁— ▁вёска ▁ў ▁б роды ўскім ▁раёне ... (+15 more)` | 25 |
+| 32k | `▁луга вэ ▁() ▁— ▁вёска ▁ў ▁броды ўскім ▁раёне ▁львоўскай ... (+13 more)` | 23 |
+| 64k | `▁луга вэ ▁() ▁— ▁вёска ▁ў ▁бродыўскім ▁раёне ▁львоўскай ▁вобласці ... (+11 more)` | 21 |
 
-**Sample 3:** `M21 (каталог Месье) — рассеянае скопішча ў сузор'і Стральца.
-
-Катэгорыя:Астранам...`
+**Sample 3:** `Косарэвэ () — вёска ў Млыніўскім раёне Ровенскай вобласці Украіны. Уваходзіць у ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁m 2 1 ▁( ката лог ▁месье ) ▁— ▁рас ... (+39 more)` | 49 |
-| 16k | `▁m 2 1 ▁( ката лог ▁месье ) ▁— ▁рассея ... (+36 more)` | 46 |
-| 32k | `▁m 2 1 ▁( ката лог ▁месье ) ▁— ▁рассеянае ... (+35 more)` | 45 |
-| 64k | `▁m 2 1 ▁( каталог ▁месье ) ▁— ▁рассеянае ▁скопішча ... (+34 more)` | 44 |
+| 8k | `▁ко са рэ вэ ▁() ▁— ▁вёска ▁ў ▁млы ніў ... (+21 more)` | 31 |
+| 16k | `▁ко са рэ вэ ▁() ▁— ▁вёска ▁ў ▁млы ніўскім ... (+19 more)` | 29 |
+| 32k | `▁коса рэ вэ ▁() ▁— ▁вёска ▁ў ▁млы ніўскім ▁раёне ... (+17 more)` | 27 |
+| 64k | `▁коса рэ вэ ▁() ▁— ▁вёска ▁ў ▁млыніўскім ▁раёне ▁ровенскай ... (+15 more)` | 25 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 3.609x compression
-- **Lowest UNK Rate:** 8k with 0.0099% unknown tokens
+- **Best Compression:** 64k achieves 4.769x compression
+- **Lowest UNK Rate:** 8k with 0.0487% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -127,57 +129,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 79,086 🏆 | 16.27 | 1,374,832 | 15.1% | 29.5% |
-| **2-gram** | 534 🏆 | 9.06 | 19,108 | 52.5% | 95.6% |
-| **3-gram** | 294,087 | 18.17 | 2,802,568 | 8.0% | 20.4% |
-| **3-gram** | 5,046 | 12.30 | 199,361 | 17.9% | 56.0% |
-| **4-gram** | 713,543 | 19.44 | 5,150,781 | 6.7% | 16.8% |
-| **4-gram** | 30,738 | 14.91 | 1,228,443 | 8.4% | 28.2% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 114,899 | 16.81 | 1,095,876 | 11.4% | 25.2% |
+| **2-gram** | Subword | 453 🏆 | 8.82 | 15,607 | 55.9% | 96.8% |
+| **3-gram** | Word | 176,550 | 17.43 | 1,682,544 | 11.7% | 25.2% |
+| **3-gram** | Subword | 4,192 | 12.03 | 145,836 | 18.7% | 59.5% |
+| **4-gram** | Word | 286,677 | 18.13 | 2,809,290 | 9.5% | 25.0% |
+| **4-gram** | Subword | 25,337 | 14.63 | 930,596 | 8.0% | 29.4% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `0 10` | 188,589 |
+| 2 | `10 0` | 184,433 |
+| 3 | `0 09` | 178,218 |
+| 4 | `09 0` | 172,686 |
+| 5 | `у годзе` | 140,117 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `0 10 0` | 183,056 |
+| 2 | `0 09 0` | 171,686 |
+| 3 | `0 11 0` | 133,046 |
+| 4 | `0 08 0` | 125,664 |
+| 5 | `0 07 0` | 84,761 |
+
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `0 ,` | 1,884,848 |
-| 2 | `катэгорыя :` | 589,578 |
-| 3 | `. у` | 390,424 |
-| 4 | `) .` | 239,852 |
-| 5 | `) —` | 224,324 |
+| 1 | `0 44 0 10` | 28,229 |
+| 2 | `44 0 10 0` | 27,892 |
+| 3 | `0 47 0 10` | 27,125 |
+| 4 | `47 0 10 0` | 26,709 |
+| 5 | `0 50 0 10` | 26,628 |
 
-**3-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `0 , 10` | 188,226 |
-| 2 | `0 , 09` | 178,159 |
-| 3 | `0 , 11` | 136,658 |
-| 4 | `0 , 08` | 133,746 |
-| 5 | `0 , 07` | 96,108 |
+| 1 | `а _` | 7,375,676 |
+| 2 | `н а` | 5,829,339 |
+| 3 | `р а` | 5,735,773 |
+| 4 | `к а` | 4,959,811 |
+| 5 | `_ п` | 4,750,427 |
 
-**4-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `катэгорыя : населеныя пункты` | 53,581 |
-| 2 | `) — вёска ў` | 38,538 |
-| 3 | `. уваходзіць у склад` | 36,002 |
-| 4 | `. — с .` | 29,359 |
-| 5 | `, 44 0 ,` | 29,322 |
+| 1 | `_ п а` | 2,102,007 |
+| 2 | `_ 0 ,` | 1,872,298 |
+| 3 | `_ н а` | 1,670,363 |
+| 4 | `н а _` | 1,424,587 |
+| 5 | `_ п ��` | 1,341,590 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `а г а _` | 980,628 |
+| 2 | `_ п р а` | 746,402 |
+| 3 | `_ г о д` | 708,921 |
+| 4 | `_ н а _` | 692,237 |
+| 5 | `к а й _` | 545,902 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 534
+- **Best Perplexity:** 2-gram (subword) with 453
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~28% of corpus
+- **Coverage:** Top-1000 patterns cover ~29% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -185,55 +219,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.5334 | 1.447 | 6.21 | 2,760,646 | 46.7% |
-| **1** | 0.5275 | 1.441 | 4.42 | 17,006 | 47.2% |
-| **2** | 0.3318 | 1.259 | 2.22 | 17,120,073 | 66.8% |
-| **2** | 0.7155 | 1.642 | 5.63 | 75,115 | 28.5% |
-| **3** | 0.1509 | 1.110 | 1.38 | 37,905,234 | 84.9% |
-| **3** | 0.8770 | 1.837 | 5.02 | 422,960 | 12.3% |
-| **4** | 0.0724 🏆 | 1.051 | 1.16 | 52,452,226 | 92.8% |
-| **4** | 0.7380 🏆 | 1.668 | 3.59 | 2,124,122 | 26.2% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.9806 | 1.973 | 10.65 | 1,594,726 | 1.9% |
+| **1** | Subword | 0.4731 | 1.388 | 3.96 | 16,459 | 52.7% |
+| **2** | Word | 0.3129 | 1.242 | 1.94 | 16,955,773 | 68.7% |
+| **2** | Subword | 0.6387 | 1.557 | 4.81 | 65,143 | 36.1% |
+| **3** | Word | 0.1126 | 1.081 | 1.23 | 32,878,014 | 88.7% |
+| **3** | Subword | 0.8192 | 1.764 | 4.91 | 313,186 | 18.1% |
+| **4** | Word | 0.0455 🏆 | 1.032 | 1.08 | 40,250,681 | 95.5% |
+| **4** | Subword | 0.7603 | 1.694 | 3.75 | 1,537,647 | 24.0% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `, аэрапорт знаходзіцца ў свеце вядома з сакрысціямі і іншыя матэрыялы , створаных самім фактам ,`
-2. `. 3 адкрыты чэмпіянат па 21 кастрычніка 1957 — супраціўнае звычайна была прысвоена званне « за`
-3. `0 , лячэбны факультэт кіравання версіямі , евангелле паводле ніпурскага , а ў сялянскай сям ’`
+1. `0 57 0 09 0 67 0 07 0 58 км на 1 20 лютага жэнева`
+2. `і стаўшы першым урадзе і гітарыст разам з поўдня сутыкненні прыпыніліся на кіргізскай сср 10 0`
+3. `у годзе гэтыя эксперыменты па год 11 0 56 0 75 0 50 0 08 0`
 
 **Context Size 2:**
 
-1. `0 , 41 0 , 11 1912897 0 , 47 0 , 44 0 , 81 0`
-2. `катэгорыя : лігатуры кірыліцы катэгорыя : паняцці індуізму катэгорыя : будынкі і збудаванні бабруйск...`
-3. `. у 1932 — 1982 ) , святы рпцз . 12 . 3 . игнатенко , в`
+1. `0 10 0 50 0 10 0 39 0 11 0 36 0 12 0 54 0`
+2. `10 0 68 0 25 0 6 1 52 1 25 джэсіка пегула эна сібахара 7 6`
+3. `0 09 0 46 0 10 0 35 0 12 0 37 0 12 0 д2 прамень`
 
 **Context Size 3:**
 
-1. `0 , 10 47053 sd 1 , 20 0 , 09 21985 e - so 0 , 93`
-2. `0 , 09 637479 0 , 39 0 , 11 88688 0 , 60 0 , 08 310690`
-3. `0 , 11 348939 0 , 60 0 , 08 2587753 0 , 73 0 , 07 838578`
+1. `0 10 0 37 0 12 0 35 0 48 0 10 0 56 0 09 0 51`
+2. `0 09 0 37 0 12 0 57 0 09 0 41 0 11 0 45 0 10`
+3. `0 11 0 42 0 11 0 61 0 08 0 51 0 09 0 37 0 12`
 
 **Context Size 4:**
 
-1. `катэгорыя : населеныя пункты гарадэнкіўскага раёна катэгорыя : населеныя пункты на дняпры катэгорыя ...`
-2. `) — вёска ў сосныцкім раёне чарнігаўскай вобласці украіны . крыніцы катэгорыя : населеныя пункты аст...`
-3. `. уваходзіць у склад сямлёўскага сельскага паселішча . гісторыя 25 сакавіка 1918 года згодна з трэця...`
+1. `0 44 0 10 0 52 0 09 0 43 0 11 0 76 0 07 0 37 0`
+2. `44 0 10 0 51 0 09 0 51 0 09 0 42 0 11 0 60 0 08`
+3. `0 47 0 10 0 54 0 09 0 65 0 08 0 38 0 11 0 46 0`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_irone_саджырода`
+2. `аса._бетвекаенсы`
+3. `ных_г._тэні_09_—`
+
+**Context Size 2:**
+
+1. `а_абто_чальны,_пр`
+2. `наяны_нькімпіныма`
+3. `раён_з_10),_якаге`
+
+**Context Size 3:**
+
+1. `_паднакадэміі_пало`
+2. `_0,40_0,56_0,50_0,`
+3. `_на_паданні._перац`
+
+**Context Size 4:**
+
+1. `ага_адсек_нацыя_4_т`
+2. `_пра_ў_сваюць_62-я_`
+3. `_годзе._жывяць_дызе`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 92.8% predictability
+- **Best Predictability:** Context-4 (word) with 95.5% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (2,124,122 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (1,537,647 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -249,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 834,514 |
-| Total Tokens | 59,867,674 |
-| Mean Frequency | 71.74 |
+| Vocabulary Size | 739,605 |
+| Total Tokens | 54,963,738 |
+| Mean Frequency | 74.31 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 3746.92 |
+| Frequency Std Dev | 3865.57 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | 0 | 1,972,029 |
-| 2 | і | 1,334,509 |
-| 3 | у | 1,241,263 |
-| 4 | ў | 1,163,394 |
-| 5 | з | 869,929 |
-| 6 | на | 710,982 |
-| 7 | катэгорыя | 590,266 |
-| 8 | года | 368,272 |
-| 9 | да | 291,578 |
-| 10 | годзе | 258,701 |
+| 1 | 0 | 1,944,698 |
+| 2 | і | 1,322,186 |
+| 3 | у | 1,231,156 |
+| 4 | ў | 1,155,870 |
+| 5 | з | 858,124 |
+| 6 | на | 705,989 |
+| 7 | года | 365,156 |
+| 8 | да | 288,350 |
+| 9 | годзе | 255,744 |
+| 10 | 10 | 239,762 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | мурашава | 2 |
-| 2 | девятке | 2 |
-| 3 | дэкунаў | 2 |
-| 4 | iovine | 2 |
-| 5 | аёвіну | 2 |
-| 6 | джэніка | 2 |
-| 7 | мэрылінам | 2 |
-| 8 | сардэшная | 2 |
-| 9 | івасю | 2 |
-| 10 | стеценко | 2 |
+| 1 | іцуно | 2 |
+| 2 | міурай | 2 |
+| 3 | kodanshas | 2 |
+| 4 | llb | 2 |
+| 5 | давы́даўскае | 2 |
+| 6 | эльханон | 2 |
+| 7 | vilner | 2 |
+| 8 | emes | 2 |
+| 9 | folkstsaytung | 2 |
+| 10 | dertseyln | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9824 |
-| R² (Goodness of Fit) | 0.995868 |
+| Zipf Coefficient | 0.9714 |
+| R² (Goodness of Fit) | 0.997385 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 28.3% |
-| Top 1,000 | 50.1% |
+| Top 100 | 29.3% |
+| Top 1,000 | 50.6% |
 | Top 5,000 | 67.4% |
-| Top 10,000 | 74.4% |
+| Top 10,000 | 74.5% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9959 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 28.3% of corpus
-- **Long Tail:** 824,514 words needed for remaining 25.6% coverage
+- **Zipf Compliance:** R²=0.9974 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 29.3% of corpus
+- **Long Tail:** 729,605 words needed for remaining 25.5% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -319,24 +384,129 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 563,209 | 32 | 4.064 | 1.812 | 0.6194 |
-| **mono_64d** | 563,209 | 64 | 4.475 | 1.623 | 0.6528 |
-| **mono_128d** | 563,209 | 128 | 4.940 | 1.374 | 0.6652 🏆 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.6148 | 0.3550 | N/A | N/A |
+| **mono_64d** | 64 | 0.6479 | 0.2915 | N/A | N/A |
+| **mono_128d** | 128 | 0.6512 🏆 | 0.2220 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_128d with 0.6652 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 563,209 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_128d with 0.6512 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2895. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ка` | каганаў, кайлі, карэлятыўных |
+| `-па` | пасуэлу, падую, паліцыянтаў |
+| `-пр` | протестантами, провозглашении, принципу |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-а` | кішскага, краснасельскага, апельсіна |
+| `-кі` | ліпнякі, чарашкі, вярцінскі |
+| `-га` | кішскага, краснасельскага, луэнга |
+| `-ай` | абнаўленчай, пустэльніцай, факталагічнай |
+| `-ага` | кішскага, краснасельскага, найбагацейшага |
+| `-мі` | неадмоўнымі, контурамі, абрамі |
+| `-ая` | наватухінская, загорская, чакаўская |
+| `-ыя` | шматбаковыя, перанятыя, узбагачаныя |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `насц` | 1.82x | 190 contexts | насцю, насць, насці |
+| `елар` | 2.47x | 46 contexts | белар, гелар, келар |
+| `анск` | 1.35x | 1021 contexts | ганск, данск, канск |
+| `асел` | 2.07x | 87 contexts | расел, насел, асель |
+| `нскі` | 1.43x | 414 contexts | янскі, енскі, інскі |
+| `ання` | 1.67x | 173 contexts | рання, вання, ранняе |
+| `аецц` | 2.21x | 48 contexts | ваецца, каецца, лаецца |
+| `нска` | 1.35x | 500 contexts | унска, янска, минска |
+| `ўска` | 1.52x | 236 contexts | еўска, іўска, еўская |
+| `ленн` | 1.48x | 234 contexts | гленн, ленны, ленная |
+| `йска` | 1.59x | 149 contexts | йская, ейска, войска |
+| `уска` | 1.36x | 263 contexts | буска, гуска, ускат |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-ка` | `-а` | 66 words | каміна, камунізма |
+| `-па` | `-а` | 55 words | паступаленка, панінскага |
+| `-пр` | `-а` | 28 words | прыкладвацца, прынада |
+| `-па` | `-ай` | 21 words | паплаўковай, пастаяннай |
+| `-па` | `-мі` | 17 words | пасіўнымі, паказнікамі |
+| `-па` | `-кі` | 16 words | палінскі, падзьячаскі |
+| `-ка` | `-га` | 16 words | какамега, калобжагскага |
+| `-ка` | `-ага` | 15 words | калобжагскага, каламойскага |
+| `-ка` | `-кі` | 14 words | кадомскі, каўхаёкі |
+| `-ка` | `-аў` | 12 words | карыбаў, катэрынычаў |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| барыёнамі | **`барыё-на-мі`** | 6.0 | `барыё` |
+| курапаткіна | **`курапат-кі-на`** | 6.0 | `курапат` |
+| хакеістаў | **`хакеіст-аў`** | 4.5 | `хакеіст` |
+| навасібірская | **`навасібірск-ая`** | 4.5 | `навасібірск` |
+| пірамідаў | **`пірамід-аў`** | 4.5 | `пірамід` |
+| трансфарматараў | **`трансфарматар-аў`** | 4.5 | `трансфарматар` |
+| участковыя | **`участков-ыя`** | 4.5 | `участков` |
+| вузельчыкамі | **`вузельчыка-мі`** | 4.5 | `вузельчыка` |
+| мікрараёнаў | **`мікрараён-аў`** | 4.5 | `мікрараён` |
+| патраціць | **`па-траціць`** | 4.5 | `траціць` |
+| папоўніцца | **`па-поўніцца`** | 4.5 | `поўніцца` |
+| капашчэўскі | **`ка-па-шчэўс-кі`** | 4.5 | `шчэўс` |
+| накрыўкамі | **`накрыўка-мі`** | 4.5 | `накрыўка` |
+| наведвальніцкі | **`наведвальніц-кі`** | 4.5 | `наведвальніц` |
+| беспартыйнымі | **`беспартыйны-мі`** | 4.5 | `беспартыйны` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language BE appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -344,11 +514,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (3.61x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (534) |
-| Markov | **Context-4** | Highest predictability (92.8%) |
+| Tokenizer | **64k BPE** | Best compression (4.77x) |
+| N-gram | **2-gram** | Lowest perplexity (453) |
+| Markov | **Context-4** | Highest predictability (95.5%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -538,7 +709,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -554,7 +726,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-28 02:15:50*
+*Report Date: 2026-01-03 11:32:17*

models/embeddings/monolingual/be_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:348b8d47c3adfcab8cf7ed5c5c126bac20a90632d48e0a96d1ac58cf34bdd206
-size 1615310423
+oid sha256:e80dd83fd9b000c473bacdfc520317bc08c8e6232f6acc8ddf47a4dc636212b7
+size 1567868138

models/embeddings/monolingual/be_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 563209
+  "vocab_size": 518052
 }

models/embeddings/monolingual/be_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:95ba41adfe852e9124f1f4e2a999e53493f8ac5a72620a903103616d33e1d3f8
-size 414765911
+oid sha256:e15ec6617f84546d2951de84ffe80fbfa2280da80a7135e996e30747c163a575
+size 402004202

models/embeddings/monolingual/be_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 563209
+  "vocab_size": 518052
 }

models/embeddings/monolingual/be_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0aae5fa847ee5bb9a765d41f00f80662622108ba7c2da83c491b6ee1aaff3cbc
-size 814947415
+oid sha256:0dca4824861fd94e6b9de472d555ae08662bf04a8795cab1ac77097e32c191f3
+size 790625514

models/embeddings/monolingual/be_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 563209
+  "vocab_size": 518052
 }

models/subword_markov/be_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:613af873a5c8ee531d4fd55c96b5d2024b23df806ba9c46820017e322c5c5213
-size 581483
+oid sha256:16ec89ccbf7b33b419dff7091cc3396dd6b9c2f2d9e7b4aaa101c1f6dc261e98
+size 528755

models/subword_markov/be_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "be",
-  "unique_contexts": 17006,
-  "total_transitions": 433862151
+  "unique_contexts": 16459,
+  "total_transitions": 384276543
 }

models/subword_markov/be_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e0c1db4d2e461008750e32dfa8de1dc9c3409c516afaa54674aef8e0c2ab8830
-size 3387673
+oid sha256:75a72bc43ff9fcb1e07421d9900ef838856a31dd2e997a85da1ec51c2da7313f
+size 2698586

models/subword_markov/be_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "be",
-  "unique_contexts": 75115,
-  "total_transitions": 433604630
+  "unique_contexts": 65143,
+  "total_transitions": 384021043
 }

models/subword_markov/be_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ab73b13dfa84faefbb17dd279eb97e6291463168a9832e9f2d357b36483bf97b
-size 16387590
+oid sha256:3cc5bbbf80158973cace1739a5b2da93ae4aba1805dddfad45e13be87b4dd5b4
+size 12779069

models/subword_markov/be_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "be",
-  "unique_contexts": 422960,
-  "total_transitions": 433347109
+  "unique_contexts": 313186,
+  "total_transitions": 383765543
 }

models/subword_markov/be_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:85fb9c5f2e85e087f9fd863b630a9776994de89274e1d9581cb75947969ee1f6
-size 64453185
+oid sha256:78d8dd2621f613c8c1d06109067bcf9cbac4f41f2929f949639de78672590907
+size 48720729

models/subword_markov/be_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "be",
-  "unique_contexts": 2124122,
-  "total_transitions": 433089588
+  "unique_contexts": 1537647,
+  "total_transitions": 383510043
 }

models/subword_ngram/be_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c0e30a48df55f96d4bf2b8dc208c62d144ebf34b1a70cfcf18baab44a7ed51fd
-size 267023
+oid sha256:aafc4ee9f69f303f6f198618f5bee0cac66a99dacae147499dc0cae12854a772
+size 221209

models/subword_ngram/be_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "be",
-  "unique_ngrams": 19108,
-  "total_ngrams": 433862151
+  "unique_ngrams": 15607,
+  "total_ngrams": 384276543
 }

models/subword_ngram/be_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1f8fca4862a13251eca0d62c746599da6a1c83bee3b4fd07ef8d26fd3782087b
-size 2504735
+oid sha256:87fb73101845b2c8cdea801fcdcd4465df82baa9bba94ed1aefa8c506c088840
+size 1907996

models/subword_ngram/be_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "be",
-  "unique_ngrams": 199361,
-  "total_ngrams": 433604630
+  "unique_ngrams": 145836,
+  "total_ngrams": 384021043
 }

models/subword_ngram/be_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:258fc772791e206747c2feee7088b7f28d6c39225074e25abce4217fc6aa0cba
-size 15989176
+oid sha256:0a1d30f2ce0acd57d7abf371e1916b606dd8f958d5fc479f3dc5736b5bb18b10
+size 12274905

models/subword_ngram/be_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "be",
-  "unique_ngrams": 1228443,
-  "total_ngrams": 433347109
+  "unique_ngrams": 930596,
+  "total_ngrams": 383765543
 }

models/tokenizer/be_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1402065668cb58d0465980ced26b6d3d48c3faaebd6dda93b82b5cbe1aabbe99
-size 589454
+oid sha256:008fe4df9c07918b817613d49143c9d406e08cd7c95f2c94d7e35e4d7af0322f
+size 592885

models/tokenizer/be_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1e0a60bc4e1c3914f6172171b41c29fd097713bfa2666961c50cd45a0a57c495
-size 968380
+oid sha256:f2db34459f167d40ce24759a3730279bf398faad2bcfe0de422d5a1ec7a70ffc
+size 969782

models/tokenizer/be_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7dd70bc8b3492061d8b4f5bfbcfdc3a51c6d45fd46e0c5ca413a9b0977d1bbb1
-size 1750083
+oid sha256:df2ee1b2850c4e4bd93d09aa2f1f4c06b4fd62dd623170b145a36f61154961b9
+size 1751650

models/tokenizer/be_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:335477bcdfc1cb61bc835c34d31b30091119b1b7473fa74c622aca874b26dab0
-size 408070
+oid sha256:e07b5ee32211d68f303eb0ca2473ef5a3e47cf3d435dbe20a3f50b5e40747119
+size 410385

models/vocabulary/be_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:67ae10394b92721eb663679625db202a64a880e70109d0b21040a910e8804efb
-size 13628498
+oid sha256:eaef8e90391cc62be2430106a5b0b4c67cc2dfacdb35f517f62c46107295d042
+size 12490294

models/vocabulary/be_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "be",
-  "vocabulary_size": 834514,
+  "vocabulary_size": 739605,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.044671476336453776,
+    "type_token_ratio": 0.028584751562556937,
     "coverage": {
-      "top_100": 0.2744355925705539,
-      "top_1000": 0.48551658368338324,
-      "top_5000": 0.6526621297329156,
-      "top_10000": 0.7207621763882552
+      "top_100": 0.28834518105660356,
+      "top_1000": 0.49802155961854777,
+      "top_5000": 0.6639092244917146,
+      "top_10000": 0.7333083111156797
     },
-    "hapax_count": 1925896,
-    "hapax_ratio": 0.697684764219808,
-    "total_documents": 257521
+    "hapax_count": 855988,
+    "hapax_ratio": 0.5364701399417019,
+    "total_documents": 255500
   }
 }

models/word_markov/be_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f573302999a72f7aa3e9d142ae089b7b678ffd15700999d05203009d925512c0
-size 196984346
+oid sha256:e8935add6e30b042b611c05c62b5e95de82abb4595dfd2e015226bf394cfb1f0
+size 207227789

models/word_markov/be_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "be",
-  "unique_contexts": 2760646,
-  "total_transitions": 81548782
+  "unique_contexts": 1594726,
+  "total_transitions": 55564226
 }

models/word_markov/be_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:63dffe71cfd86f4ff61c37367e2dbee74fa2f7c50e05a4825b7739595e1b2155
-size 696994958
+oid sha256:ecade528e4ed410d8e502020cb4476eb3034bb468a559acee35d2d25b0b413e1
+size 740234356

models/word_markov/be_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "be",
-  "unique_contexts": 17120073,
-  "total_transitions": 81291262
+  "unique_contexts": 16955773,
+  "total_transitions": 55308726
 }

models/word_markov/be_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fd8f08d3be91c4c054ae6f0aaa136416885527a2d29113415ab337c834789d0e
-size 1226716369
+oid sha256:79ab049dc05e060d35810bba686870b9c9a206eddbb744584e16841aca50e4f2
+size 1158519507

models/word_markov/be_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "be",
-  "unique_contexts": 37905234,
-  "total_transitions": 81033747
+  "unique_contexts": 32878014,
+  "total_transitions": 55053226
 }

models/word_markov/be_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:33587a71fb18a5c4cc838d154494581511cee2776f0934c16659de4c95e12078
-size 1617896422
+oid sha256:5efe48d536fb8243e5c2508fc3a751114aaf85a61fbf563ecc161ba97c65c3dc
+size 1439522702

models/word_markov/be_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "be",
-  "unique_contexts": 52452226,
-  "total_transitions": 80776256
+  "unique_contexts": 40250681,
+  "total_transitions": 54797726
 }

models/word_ngram/be_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2e4fc961aa5fc25059f776fbf54bffac241e518ff469fdaf22b2abdea5a86e4d
-size 33153735
+oid sha256:3a25d00ebef666273e7160d97bdef6a637b74a4a36ef712be7aa842c92bbff7d
+size 28486547

models/word_ngram/be_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "be",
-  "unique_ngrams": 1374832,
-  "total_ngrams": 81548782
+  "unique_ngrams": 1095876,
+  "total_ngrams": 55564226
 }

models/word_ngram/be_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:58da42f46883e63cb423ab5d1f9a73b92e98b59bc8da341ae4aaed966d43beef
-size 75522602
+oid sha256:18acde570dcbb5bbf4141bebdadf37c9ea7cd46ab121e7d42466e7b12d35d67a
+size 51844728

models/word_ngram/be_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "be",
-  "unique_ngrams": 2802568,
-  "total_ngrams": 81291262
+  "unique_ngrams": 1682544,
+  "total_ngrams": 55308726
 }

models/word_ngram/be_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c5e3e3cc455ebf7ad153584a4edb334f24017f1636a5e52b898c56b9a3a24755
-size 147804215
+oid sha256:556869d5b61ce881002e2f0826d29a633f55212072deae87c3eea091a943b1ad
+size 95068402

models/word_ngram/be_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "be",
-  "unique_ngrams": 5150781,
-  "total_ngrams": 81033747
+  "unique_ngrams": 2809290,
+  "total_ngrams": 55053226
 }