Upload all models and assets for blk (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 6.247
+    value: 4.845
   - name: best_isotropy
     type: isotropy
-    value: 0.8571
+    value: 0.8617
   - name: vocabulary_size
     type: vocab
-    value: 8617
-generated: 2025-12-28
+    value: 0
+generated: 2026-01-03
 ---
 
 # BLK - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,53 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 4.512x | 4.49 | 0.0852% | 975,649 |
-| **16k** | 5.081x | 5.06 | 0.0959% | 866,368 |
-| **32k** | 5.658x | 5.63 | 0.1068% | 777,918 |
-| **64k** | 6.247x 🏆 | 6.22 | 0.1179% | 704,624 |
+| **8k** | 4.016x | 4.02 | 0.0510% | 1,061,065 |
+| **16k** | 4.425x | 4.43 | 0.0562% | 962,856 |
+| **32k** | 4.609x | 4.61 | 0.0585% | 924,505 |
+| **64k** | 4.845x 🏆 | 4.85 | 0.0615% | 879,406 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `မျန်မာခမ်းထီကိုယို တွိုင်ꩻဒေႏသတန် အဝ်ႏ ( ၇ )တွိုင်ꩻ နဝ်ꩻသွူ ။`
+**Sample 1:** `အမုဲင် ခမ်းထီ ကသှိုပ်စဒါႏ ငဝ်းလဝ်းနီꩻ ၃၅လာအို ၉၄ ထူႏတောမ်`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁မျန်မာခမ်းထီ ကိုယို ▁တွိုင်ꩻ ဒေႏသတန် ▁အဝ်ႏ ▁( ▁၇ ▁) တွိုင်ꩻ ▁နဝ်ꩻ ... (+2 more)` | 12 |
-| 16k | `▁မျန်မာခမ်းထီ ကိုယို ▁တွိုင်ꩻဒေႏသတန် ▁အဝ်ႏ ▁( ▁၇ ▁) တွိုင်ꩻ ▁နဝ်ꩻသွူ ▁။` | 10 |
-| 32k | `▁မျန်မာခမ်းထီ ကိုယို ▁တွိုင်ꩻဒေႏသတန် ▁အဝ်ႏ ▁( ▁၇ ▁) တွိုင်ꩻ ▁နဝ်ꩻသွူ ▁။` | 10 |
-| 64k | `▁မျန်မာခမ်းထီ ကိုယို ▁တွိုင်ꩻဒေႏသတန် ▁အဝ်ႏ ▁( ▁၇ ▁) တွိုင်ꩻ ▁နဝ်ꩻသွူ ▁။` | 10 |
+| 8k | `▁အမုဲင် ▁ခမ်းထီ ▁က သ ှို ပ် စဒါႏ ▁ငဝ်း လ ဝ်း ... (+7 more)` | 17 |
+| 16k | `▁အမုဲင် ▁ခမ်းထီ ▁ကသ ှိုပ် စဒါႏ ▁ငဝ်း လဝ်း နီꩻ ▁၃၅ လာအို ... (+3 more)` | 13 |
+| 32k | `▁အမုဲင် ▁ခမ်းထီ ▁ကသှိုပ်စဒါႏ ▁ငဝ်း လဝ်း နီꩻ ▁၃၅လာအို ▁၉ ၄ ▁ထူႏတောမ်` | 10 |
+| 64k | `▁အမုဲင် ▁ခမ်းထီ ▁ကသှိုပ်စဒါႏ ▁ငဝ်းလဝ်းနီꩻ ▁၃၅လာအို ▁၉၄ ▁ထူႏတောမ်` | 7 |
 
-**Sample 2:** `မျန်မာခမ်းထီကိုယို ခမ်းနယ်ႏ အဝ်ႏ ( ၇ )ခမ်းနယ်ႏ နဝ်ꩻသွူ ။`
+**Sample 2:** `the war is very bad!a website to summarise the war`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁မျန်မာခမ်းထီ ကိုယို ▁ခမ်းနယ်ႏ ▁အဝ်ႏ ▁( ▁၇ ▁) ခမ်းနယ်ႏ ▁နဝ်ꩻ သွူ ... (+1 more)` | 11 |
-| 16k | `▁မျန်မာခမ်���ထီ ကိုယို ▁ခမ်းနယ်ႏ ▁အဝ်ႏ ▁( ▁၇ ▁) ခမ်းနယ်ႏ ▁နဝ်ꩻသွူ ▁။` | 10 |
-| 32k | `▁မျန်မာခမ်းထီ ကိုယို ▁ခမ်းနယ်ႏ ▁အဝ်ႏ ▁( ▁၇ ▁) ခမ်းနယ်ႏ ▁နဝ်ꩻသွူ ▁။` | 10 |
-| 64k | `▁မျန်မာခမ်းထီ ကိုယို ▁ခမ်းနယ်ႏ ▁အဝ်ႏ ▁( ▁၇ ▁) ခမ်းနယ်ႏ ▁နဝ်ꩻသွူ ▁။` | 10 |
-
-**Sample 3:** `လွူးဖွာꩻဇာႏတိပအိုဝ်ႏခမ်း ထွာဒေါ့ꩻဖြဝ်ႏပအိုဝ်ႏငဝ်းငွါနဝ်ꩻသွူ။
+| 8k | `▁the ▁w ar ▁is ▁ver y ▁b ad ! a ... (+11 more)` | 21 |
+| 16k | `▁the ▁war ▁is ▁ver y ▁b ad ! a ▁website ... (+6 more)` | 16 |
+| 32k | `▁the ▁war ▁is ▁very ▁b ad ! a ▁website ▁to ... (+3 more)` | 13 |
+| 64k | `▁the ▁war ▁is ▁very ▁bad ! a ▁website ▁to ▁summarise ... (+2 more)` | 12 |
 
-ကဏ္ဍ:ဘာႏသာႏငဝ်းငွါ...`
+**Sample 3:** `မျန်မာခမ်းထီကိုယို ခမ်းနယ်ႏ အဝ်ႏ ( ၇ )ခမ်းနယ်ႏ နဝ်ꩻသွူ ။`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁လွူးဖွာꩻ ဇာႏတိ ပအိုဝ်ႏ ခမ်း ▁ထွာ ဒေါ့ꩻ ဖြဝ်ႏ ပအိုဝ်ႏ ငဝ်းငွါ နဝ်ꩻသွူ။ ... (+10 more)` | 20 |
-| 16k | `▁လွူးဖွာꩻ ဇာႏတိ ပအိုဝ်ႏ ခမ်း ▁ထွာ ဒေါ့ꩻဖြဝ်ႏ ပအိုဝ်ႏ ငဝ်းငွါ နဝ်ꩻသွူ။ ▁ကဏ္ဍ ... (+9 more)` | 19 |
-| 32k | `▁လွူးဖွာꩻ ဇာႏတိ ပအိုဝ်ႏခမ်း ▁ထွာ ဒေါ့ꩻဖြဝ်ႏ ပအိုဝ်ႏ ငဝ်းငွါ နဝ်ꩻသွူ။ ▁ကဏ္ဍ : ... (+7 more)` | 17 |
-| 64k | `▁လွူးဖွာꩻဇာႏတိ ပအိုဝ်ႏခမ်း ▁ထွာ ဒေါ့ꩻဖြဝ်ႏ ပအိုဝ်ႏ ငဝ်းငွါ နဝ်ꩻသွူ။ ▁ကဏ္ဍ : ဘာႏသာႏငဝ်းငွါ ... (+6 more)` | 16 |
+| 8k | `▁မျန်မာခမ်းထီ ကိုယို ▁ခမ်းနယ်ႏ ▁အဝ်ႏ ▁( ▁၇ ▁) ခမ်းနယ်ႏ ▁နဝ်ꩻ သွူ ... (+1 more)` | 11 |
+| 16k | `▁မျန်မာခမ်းထီ ကိုယို ▁ခမ်းနယ်ႏ ▁အဝ်ႏ ▁( ▁၇ ▁) ခမ်းနယ်ႏ ▁နဝ်ꩻသွူ ▁။` | 10 |
+| 32k | `▁မျန်မာခမ်းထီ ကိုယို ▁ခမ်းနယ်ႏ ▁အဝ်ႏ ▁( ▁၇ ▁) ခမ်းနယ်ႏ ▁နဝ်ꩻသွူ ▁။` | 10 |
+| 64k | `▁မျန်မာခမ်းထီ ကိုယို ▁ခမ်းနယ်ႏ ▁အဝ်ႏ ▁( ▁၇ ▁) ခမ်းနယ်ႏ ▁နဝ်ꩻသွူ ▁။` | 10 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 6.247x compression
-- **Lowest UNK Rate:** 8k with 0.0852% unknown tokens
+- **Best Compression:** 64k achieves 4.845x compression
+- **Lowest UNK Rate:** 8k with 0.0510% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -123,57 +129,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 605 🏆 | 9.24 | 17,140 | 53.4% | 90.3% |
-| **2-gram** | 457 🏆 | 8.84 | 5,233 | 54.9% | 95.5% |
-| **3-gram** | 4,230 | 12.05 | 80,070 | 26.8% | 61.4% |
-| **3-gram** | 3,254 | 11.67 | 49,454 | 27.4% | 64.5% |
-| **4-gram** | 20,609 | 14.33 | 276,816 | 14.4% | 37.5% |
-| **4-gram** | 15,807 | 13.95 | 213,495 | 15.2% | 39.6% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 2,554 | 11.32 | 4,328 | 21.2% | 57.8% |
+| **2-gram** | Subword | 1,405 🏆 | 10.46 | 24,351 | 42.7% | 76.9% |
+| **3-gram** | Word | 3,876 | 11.92 | 6,558 | 18.8% | 47.3% |
+| **3-gram** | Subword | 11,360 | 13.47 | 129,980 | 18.9% | 45.0% |
+| **4-gram** | Word | 16,945 | 14.05 | 23,380 | 8.9% | 21.9% |
+| **4-gram** | Subword | 54,384 | 15.73 | 407,218 | 10.1% | 25.7% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `နဝ်ꩻ အဝ��ႏဒျာႏ` | 718 |
+| 2 | `အဝ်ႏဒျာႏ မျန်မာခမ်းထီ` | 691 |
+| 3 | `ခရိစ်နေင်ႏ ဗာႏ` | 404 |
+| 4 | `ဗာႏ စာႏရင်ꩻအလꩻ` | 320 |
+| 5 | `မျန်မာခမ်းထီ အခဝ်ထာႏဝ` | 295 |
+
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `် ꩻ` | 273,897 |
-| 2 | `ိ ု` | 259,205 |
-| 3 | `် ႏ` | 224,944 |
-| 4 | `င ်` | 218,997 |
-| 5 | `ေ ာ` | 183,337 |
+| 1 | `နဝ်ꩻ အဝ်ႏဒျာႏ မျန်မာခမ်းထီ` | 624 |
+| 2 | `အဝ်ႏဒျာႏ မျန်မာခမ်းထီ အခဝ်ထာႏဝ` | 295 |
+| 3 | `ခရိစ်နေင်ႏ ဗာႏ စာႏရင်ꩻအလꩻ` | 261 |
+| 4 | `ဗာႏ စာႏရင်ꩻအလꩻ ဝေင်ꩻကိုနဝ်ꩻ` | 161 |
+| 5 | `ထာꩻထွာဖုံႏ လွူးဖွာꩻသားဖုံႏ သီမားသားဖုံႏ` | 153 |
 
-**3-grams:**
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ဲ င ်` | 92,543 |
-| 2 | `နဝ ် ꩻ` | 76,996 |
-| 3 | `ိ ု ꩻ` | 76,094 |
-| 4 | `င ် ꩻ` | 70,792 |
-| 5 | `ွ ိ ု` | 70,250 |
+| 1 | `နဝ်ꩻ အဝ်ႏဒျာႏ မျန်မာခမ်းထီ အခဝ်ထာႏဝ` | 282 |
+| 2 | `ခရိစ်နေင်ႏ ဗာႏ စာႏရင်ꩻအလꩻ ဝေင်ꩻကိုနဝ်ꩻ` | 161 |
+| 3 | `သီမားသားဖုံႏ မွူးနီꩻအုံပဆားနီꩻဖုံႏတောမ်ႏ အထွတ်အမျတ်မွူးနီꩻဖုံႏ အာႏကွိုꩻ` | 153 |
+| 4 | `လွူးဖွာꩻသားဖုံႏ သီမားသားဖုံႏ မွူးနီꩻအုံပဆားနီꩻဖုံႏတောမ်ႏ အထွတ်အမျတ်မွူးနီꩻဖုံႏ` | 153 |
+| 5 | `ထာꩻထွာဖုံႏ လွူးဖွာꩻသားဖုံႏ သီမားသားဖုံႏ မွူးနီꩻအုံပဆားနီꩻဖုံႏတောမ်ႏ` | 153 |
 
-**4-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ေ ာ င ်` | 53,600 |
-| 2 | `ု ဲ င ်` | 46,319 |
-| 3 | `ေ ာ ဝ ်` | 45,569 |
-| 4 | `သ ွ ူ ။` | 34,315 |
-| 5 | `ဖ ု ံ ႏ` | 27,123 |
+| 1 | `ာ ႏ` | 142,556 |
+| 2 | `၊ _` | 135,431 |
+| 3 | `ꩻ _` | 126,463 |
+| 4 | `ဝ် ꩻ` | 102,686 |
+| 5 | `င် ꩻ` | 97,010 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `န ဝ် ꩻ` | 77,017 |
+| 2 | `ဝ် ꩻ _` | 57,560 |
+| 3 | `ꩻ ၊ _` | 31,807 |
+| 4 | `သွူ ။ _` | 31,585 |
+| 5 | `ႏ ၊ _` | 30,939 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `န ဝ် ꩻ _` | 45,458 |
+| 2 | `နေ ာ ဝ် ꩻ` | 23,540 |
+| 3 | `ꩻ သွူ ။ _` | 18,995 |
+| 4 | `ꩻ န ဝ် ꩻ` | 18,028 |
+| 5 | `ႏ န ဝ် ꩻ` | 17,062 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 457
+- **Best Perplexity:** 2-gram (subword) with 1,405
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~40% of corpus
+- **Coverage:** Top-1000 patterns cover ~26% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -181,55 +219,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.4548 | 1.371 | 4.45 | 18,934 | 54.5% |
-| **1** | 0.9647 | 1.952 | 10.06 | 1,025 | 3.5% |
-| **2** | 0.3715 | 1.294 | 3.04 | 84,154 | 62.9% |
-| **2** | 1.2736 | 2.418 | 8.47 | 10,314 | 0.0% |
-| **3** | 0.3756 | 1.297 | 2.48 | 255,732 | 62.4% |
-| **3** | 0.9122 | 1.882 | 4.21 | 87,402 | 8.8% |
-| **4** | 0.3254 🏆 | 1.253 | 1.98 | 633,828 | 67.5% |
-| **4** | 0.5603 🏆 | 1.475 | 2.51 | 367,932 | 44.0% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.2313 | 1.174 | 1.60 | 382,155 | 76.9% |
+| **1** | Subword | 1.2242 | 2.336 | 21.07 | 2,910 | 0.0% |
+| **2** | Word | 0.0413 | 1.029 | 1.06 | 611,948 | 95.9% |
+| **2** | Subword | 0.7533 | 1.686 | 5.49 | 61,297 | 24.7% |
+| **3** | Word | 0.0155 | 1.011 | 1.02 | 648,373 | 98.5% |
+| **3** | Subword | 0.4736 | 1.389 | 2.77 | 336,631 | 52.6% |
+| **4** | Word | 0.0088 🏆 | 1.006 | 1.01 | 660,080 | 99.1% |
+| **4** | Subword | 0.3161 | 1.245 | 1.90 | 934,101 | 68.4% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `် ꩻ ပ ြ ွ ဉ ာ ႏ ၊ သ ူ ႏ န ် ꩻ နဝ`
-2. `ꩻ မ ီ း ၊ ဒလ ိ ု ꩻ လက ် ႏ သ ာ ဖ ေ`
-3. `ႏ မ ် ႏ ထ ဲ ( မ ွ ိ ု င ် ꩻ ပအ ိ`
+1. `၂ ဖြားတန်မောင်ငွေ ဥက္ကဋ္ဌ နာယက ဗွေႏဗွန်ကျောင်ꩻဝေင်ꩻသီႏသဲင်ႏ အကျိုꩻဆောင်ႏနဝ်ꩻ ဗွေႏဗွန်လုံးဖိုးကျောင်ꩻ...`
+2. `၃ နီꩻကို နုဲင်ႏငံႏတောႏအစိုႏရ ကတဲမ်းထွို့ꩻဒါႏ ဗုဲင်းရတ်သ်ပုဂ္ဂိုလ်ႏယင်ဟန်ႏသားနဝ်ꩻ ဝွေꩻသီး အွဉ်မာꩻချွမ...`
+3. `၁ အဟွိုန်အထီႏ မဉ်ႏအာနောဝ်ꩻ ဗွေႏမျတ်ဘုရာꩻ ကဟော်ꩻသေꩻခါꩻအတွိုင်ꩻသွူ ပိဋကတ်ပြန်ႏပအိုဝ်ႏစွိုꩻကထေသေꩻခါꩻ ပြ...`
 
 **Context Size 2:**
 
-1. `် ꩻ ထ ွ ူ ။ မန ် း ဆ ာ သ ူ ဘ ာ ႏ သ`
-2. `ိ ု င ် ꩻ တ ဲ ့ ၊ မ ီ ႏ ဟ ံ ႏ မဉ ်`
-3. `် ႏ စ ွ ု ံ ႏ န ေ ာ င ် ႏ ယ ူ ကရ ေ`
+1. `နဝ်ꩻ အဝ်ႏဒျာႏ မျန်မာခမ်းထီ အခဝ်ထာႏဝ မကွေးတွိုင်ꩻဒေႏသတန် ချောက်ခရဲင်ႏ ဝေင်ꩻနယ်ႏချောက်ကို ကအဝ်ႏဒါႏ ဧရာ...`
+2. `အဝ်ႏဒျာႏ မျန်မာခမ်းထီ ဖြဝ်ꩻခမ်းနယ်ႏအခဝ်နဝ် ကလော်ꩻခရဲင်ႏ ဝေင်ꩻနယ်ႏညောင်ႏရွီႏကို ကအဝ်ႏဒါႏ ဝေင်ꩻတဝေင်ꩻဒ...`
+3. `ခရိစ်နေင်ႏ ဗာႏ မွူးကွို့ꩻနာꩻစာႏရင်ꩻအလꩻ အဝ်ႏ ဖြာꩻသွူ အွိုင်ႏလွယ်အွန်ႏကို လိုꩻဖြာꩻခြွဉ်းနဝ်ꩻ နေင်ႏ ဗာႏ...`
 
 **Context Size 3:**
 
-1. `ဲ င ် း - " ခ ေ တ ် ယ ာ နဝ ် ꩻ နမ ် း`
-2. `နဝ ် ꩻ အ ွ ဉ ် ႏ ဒ ါ ႏ ခမ ် း က ြ ီ း`
-3. `ိ ု ꩻ မ ု ဲ င ် း နဝ ် ꩻ သ ွ ူ ။ အ ာ`
+1. `နဝ်ꩻ အဝ်ႏဒျာႏ မျန်မာခမ်းထီ အခဝ်ခိုႏ သကုဲင်းတွိုင်ꩻဒေႏသတန် အခဝ်ကွဉ်ႏထင်ꩻ ကတာခရဲင်ႏ ဝေင်ꩻနယ်ႏအိဉ်းတော်...`
+2. `အဝ်ႏဒျာႏ မျန်မာခမ်းထီ အခဝ်ထာႏဝ မန္တလေꩻတွိုင်ꩻဒေႏသတန် ဝေင်ꩻနယ်ႏနာထိုးစီးကို ကအဝ်ႏဒါႏ ဝေင်ꩻနယ်ႏရွုမ်ꩻ ...`
+3. `ခရိစ်နေင်ႏ ဗာႏ စာႏရင်ꩻအလꩻ ဝေင်ꩻကိုနဝ်ꩻ လိုꩻဖြာꩻခြွဉ်းအဝ်ႏ ၈ ၂၅၂ ဖြာꩻသွူ အာႏကွိုꩻ`
 
 **Context Size 4:**
 
-1. `ေ ာ င ် း ခ ါ ႏ ဓလ ေ ့ ꩻ ထ ိ ု ꩻ ဝင ် ꩻ`
-2. `ု ဲ င ် း လ ဲ ့ ခ ွ ေ အ ီ ႏ အ ာ ႏ အရသ ာ`
-3. `ေ ာ ဝ ် ꩻ လ ိ ု ႏ သ ံ ႏ ဃ ာ ႏ လ ေ ာ င`
+1. `နဝ်ꩻ အဝ်ႏဒျာႏ မျန်မာခမ်းထီ အခဝ်ထာႏဝ မန်ႏတလေꩻတွိုင်ꩻဒေႏသတန် မိဉ်းဆျန်ခရဲင်ႏကို ကအဝ်ႏပါသော့ꩻဒါႏ ဝေင်ꩻန...`
+2. `ခရိစ်နေင်ႏ ဗာႏ စာႏရင်ꩻအလꩻ ဝေင်ꩻကိုနဝ်ꩻ လိုꩻဖြာꩻခြွဉ်းအဝ်ႏ ၇ ၅၈၇ ရပ်သွူ အာႏကွိုꩻ`
+3. `လွူးဖွာꩻသားဖုံႏ သီမားသားဖုံႏ မွူးနီꩻအုံပဆားနီꩻဖုံႏတောမ်ႏ အထွတ်အမျတ်မွူးနီꩻဖုံႏ အာႏကွိုꩻ`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_လွေꩻလꩻဗာꩻ_၈-_နာတ်ꩻ`
+2. `ꩻငဝ်း၊_ထွာယ်ဆည်းရွယ်ႏန`
+3. `ႏ_သမ်းပလောက်ထာႏတသီးထ`
+
+**Context Size 2:**
+
+1. `ာႏရေꩻတွယ်ႏတယယ်ꩻထင်ႏနောဝ်`
+2. `၊_နာႏလွယ်၊_ထွာဆေ့ꩻရွစ်ဟော`
+3. `ꩻ_။_ကတ္တရာꩻနုဲင်းယိုသွူ-_သ`
+
+**Context Size 3:**
+
+1. `နဝ်ꩻတဲ့_စူꩻအကိုသူꩻ_ထွာ_ပသေဒဲ`
+2. `ဝ်ꩻ_ကဖေႏအံႏ_ခမ်းနေႏရာႏသွု`
+3. `ꩻ၊_၃။_ဖြော်ချာ_လူကြိုက်များရ`
+
+**Context Size 4:**
+
+1. `နဝ်ꩻ_လိုꩻဖြာꩻ၊_နောဝ်ထင်ꩻ_ဖုံ`
+2. `နောဝ်ꩻ_အွဉ်အကိုရေꩻ_(၂၂၂)။_`
+3. `ꩻသွူ။_အားထွုတ်"ဗုဒ္ဓါနုဿတိကမ္မဋ္ဌ`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 67.5% predictability
+- **Best Predictability:** Context-4 (word) with 99.1% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (367,932 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (934,101 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -245,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 8,617 |
-| Total Tokens | 3,060,323 |
-| Mean Frequency | 355.15 |
-| Median Frequency | 4 |
-| Frequency Std Dev | 5961.41 |
+| Vocabulary Size | 68,078 |
+| Total Tokens | 398,630 |
+| Mean Frequency | 5.86 |
+| Median Frequency | 2 |
+| Frequency Std Dev | 39.89 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | င | 240,072 |
-| 2 | န | 174,719 |
-| 3 | မ | 171,648 |
-| 4 | သ | 171,366 |
-| 5 | တ | 167,103 |
-| 6 | လ | 163,416 |
-| 7 | က | 130,859 |
-| 8 | ထ | 101,875 |
-| 9 | ဝ | 96,145 |
-| 10 | ခ | 89,693 |
+| 1 | ၂ | 3,802 |
+| 2 | ၃ | 3,377 |
+| 3 | ၁ | 3,336 |
+| 4 | အာႏကွိုꩻ | 3,141 |
+| 5 | နဝ်ꩻ | 2,713 |
+| 6 | ၄ | 2,610 |
+| 7 | ၅ | 2,059 |
+| 8 | ထွာဒျာႏ | 1,628 |
+| 9 | ၆ | 1,583 |
+| 10 | အဝ်ႏဒျာႏ | 1,493 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ဗဃသ | 2 |
-| 2 | ဆဿ | 2 |
-| 3 | ခဏဇ | 2 |
-| 4 | subatomic | 2 |
-| 5 | nashi | 2 |
-| 6 | bridges | 2 |
-| 7 | antihistamine | 2 |
-| 8 | histamine | 2 |
-| 9 | ၁၃၈၇ | 2 |
-| 10 | ဗခရ | 2 |
+| 1 | ယံဇိတံ | 2 |
+| 2 | ထာꩻအောင်ႏယမျိုꩻနွ်ꩻ | 2 |
+| 3 | လိုꩻစင်ꩻတဖဲ့ꩻသား | 2 |
+| 4 | ပစ္စာဟရိဿာမိ | 2 |
+| 5 | အဖွန်ႏအခွမ်ꩻတွယ်ꩻမုꩻ | 2 |
+| 6 | တပုဲင်ႏဗွာ | 2 |
+| 7 | nashi | 2 |
+| 8 | ညီႏလာႏခံႏအကို | 2 |
+| 9 | ပြဲႏထောင်ႏစု | 2 |
+| 10 | ဆောင်ႏရွတ်ဖေႏ | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.5483 |
-| R² (Goodness of Fit) | 0.988794 |
+| Zipf Coefficient | 0.7925 |
+| R² (Goodness of Fit) | 0.997962 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 90.5% |
-| Top 1,000 | 98.5% |
-| Top 5,000 | 99.7% |
-| Top 10,000 | 0.0% |
+| Top 100 | 18.0% |
+| Top 1,000 | 34.4% |
+| Top 5,000 | 51.9% |
+| Top 10,000 | 61.5% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9888 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 90.5% of corpus
-- **Long Tail:** -1,383 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9980 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 18.0% of corpus
+- **Long Tail:** 58,078 words needed for remaining 38.5% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -315,24 +384,117 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 11,378 | 32 | 4.944 | 1.091 | 0.8571 🏆 |
-| **mono_64d** | 11,378 | 64 | 5.763 | 0.821 | 0.8516 |
-| **mono_128d** | 11,378 | 128 | 6.405 | 0.791 | 0.6804 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8617 🏆 | 0.3357 | N/A | N/A |
+| **mono_64d** | 64 | 0.8600 | 0.2769 | N/A | N/A |
+| **mono_128d** | 128 | 0.6775 | 0.2411 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8571 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 11,378 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8617 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2846. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
+
+---
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-လိ` | လိတ်ပအိုဝ်ႏဗဟိုႏသွဉ်တန်ꩻ, လိုꩻမျတ်ဖုံႏ, လိတ်ပအိုဝ်ႏစောင်ႏကို |
+| `-လို` | လိုꩻမျတ်ဖုံႏ, လိုꩻသီးဖုံႏယို, လိုꩻသီးယိုနဝ်ꩻ |
+| `-လိုꩻ` | လိုꩻမျတ်ဖုံႏ, လိုꩻသီးဖုံႏယို, လိုꩻသီးယိုနဝ်ꩻ |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-ꩻ` | လိတ်ပအိုဝ်ႏဗဟိုႏသွဉ်တန်ꩻ, ခိုမူႏခွန်နင်ꩻ, အလင်နဝ်ꩻ |
+| `-ႏ` | ပအိုဝ်ႏတာႏ, ကျင်ꩻလွဉ်ꩻမဲဉ်ႏမဲဉ်ႏဒျာႏ, ခမ်းထီနဲင်ႏငန်ႏ |
+| `-်ꩻ` | လိတ်ပအိုဝ်ႏဗဟိုႏသွဉ်တန်ꩻ, ခိုမူႏခွန်နင်ꩻ, အလင်နဝ်ꩻ |
+| `-ဝ်ꩻ` | အလင်နဝ်ꩻ, အမတ်ဖုံႏနောဝ်ꩻ, ပါꩻမုဲင်ꩻနဝ်ꩻ |
+| `-နဝ်ꩻ` | အလင်နဝ်ꩻ, ပါꩻမုဲင်ꩻနဝ်ꩻ, ခွန်ထွန်းအောင်နဝ်ꩻ |
+| `-်း` | လုမ်းလုမ်း, ဘဝလိုꩻခမ်း, အုံတပန်း |
+| `-ာႏ` | ပအိုဝ်ႏတာႏ, ကျင်ꩻလွဉ်ꩻမဲဉ်ႏမဲဉ်ႏဒျာႏ, ကိုꩻကွယ်ႏဆရာႏမာႏ |
+| `-်ႏ` | ခမ်းထီနဲင်ႏငန်ႏ, ကမ္ဘာႏဟမ်ႏ, သကဒါဂါမိဖိုလ်ႏ |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+*No significant bound stems detected.*
+
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-လိ` | `-ꩻ` | 82 words | လိုꩻရွိုင်ꩻ, လိုꩻဘဝခြွေနယ်ꩻ |
+| `-လိ` | `-ႏ` | 54 words | လိုႏဖေႏအာႏငါႏ, လိုꩻနွို့လိုꩻထန်ႏ |
+| `-လိ` | `-်ꩻ` | 50 words | လိုꩻရွိုင်ꩻ, လိုꩻဘဝခြွေနယ်ꩻ |
+| `-လိ` | `-ဝ်ꩻ` | 38 words | လိတ်လုဲင်ꩻပညာႏသျင်ႏသီးနဝ်ꩻ, လိတ်မွူးတွယ်ꩻနဝ်ꩻ |
+| `-လိ` | `-နဝ်ꩻ` | 30 words | လိတ်လုဲင်ꩻပညာႏသျင်ႏသီးနဝ်ꩻ, လိတ်မွူးတွယ်ꩻနဝ်ꩻ |
+| `-လိ` | `-ို` | 24 words | လိုႏသော့ꩻလိတ်မွူးကို, လိုꩻတဟဝ်တဝ်းယို |
+| `-လိ` | `-်ႏ` | 18 words | လိုꩻနွို့လိုꩻထန်ႏ, လိုꩻဖြာꩻခြွဉ်းအောဝ်ႏ |
+| `-လိ` | `-်း` | 17 words | လိုꩻတဲ့ယဝ်း, လိုꩻတသေတဝ်း |
+| `-လိ` | `-ာႏ` | 16 words | လိုꩻသꩻရာႏ, လိုꩻခြွေလိုꩻခြာႏ |
+| `-လိ` | `-ွူ` | 7 words | လိုꩻမျိုꩻဖုံႏဒျာႏသွူ, လိုꩻအွူးဟွူ |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| ဥပဇ္ဈာယ်ႏ | **`ဥပဇ္ဈာယ-်ႏ`** | 4.5 | `ဥပဇ္ဈာယ` |
+| ပငါပရာꩻဖုံႏနဝ်ꩻ | **`ပငါပရာꩻဖုံႏ-နဝ်ꩻ`** | 4.5 | `ပငါပရာꩻဖုံႏ` |
+| အနမ်းနဝ်ꩻ | **`အနမ်း-နဝ်ꩻ`** | 4.5 | `အနမ်း` |
+| မွူးရဝ်ꩻနီꩻနဝ်ꩻ | **`မွူးရဝ်ꩻနီꩻ-နဝ်ꩻ`** | 4.5 | `မွူးရဝ်ꩻနီꩻ` |
+| ရဟန္တာႏသီးနဝ်ꩻ | **`ရဟန္တာႏသီး-နဝ်ꩻ`** | 4.5 | `ရဟန္တာႏသီး` |
+| ယိုခါနဝ်ꩻ | **`ယိုခါ-နဝ်ꩻ`** | 4.5 | `ယိုခါ` |
+| စဲ့ꩻရေꩻနဝ်ꩻ | **`စဲ့ꩻရေꩻ-နဝ်ꩻ`** | 4.5 | `စဲ့ꩻရေꩻ` |
+| လိတ်ကရိုꩻယိုနဝ်ꩻ | **`လိ-တ်ကရိုꩻယ-ို-နဝ်ꩻ`** | 4.5 | `တ်ကရိုꩻယ` |
+| ကုဲင်ထိုꩻနဝ်ꩻ | **`ကုဲင်ထိုꩻ-နဝ်ꩻ`** | 4.5 | `ကုဲင်ထိုꩻ` |
+| လိုꩻသꩻရာႏ | **`လိုꩻ-သꩻရာႏ`** | 4.5 | `သꩻရာႏ` |
+| လိက်ဖြိုင်ႏ | **`လိ-က်ဖြိုင-်ႏ`** | 3.0 | `က်ဖြိုင` |
+| လိုꩻအဆင်ႏအရန်း | **`လိုꩻ-အဆင်ႏအရန-်း`** | 3.0 | `အဆင်ႏအရန` |
+| လိတ်ကျမ်ꩻ | **`လိ-တ်ကျမ-်ꩻ`** | 3.0 | `တ်ကျမ` |
+| လိုꩻသဒ္ဓါႏအဝ်ႏ | **`လိုꩻ-သဒ္ဓါႏအဝ-်ႏ`** | 3.0 | `သဒ္ဓါႏအဝ` |
+| ဘာႏဝနာႏနဝ်ꩻ | **`ဘာႏဝန-ာႏ-နဝ်ꩻ`** | 3.0 | `ဘာႏဝန` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language BLK appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
 
 ---
-## 6. Summary & Recommendations
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -340,11 +502,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (6.25x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (457) |
-| Markov | **Context-4** | Highest predictability (67.5%) |
+| Tokenizer | **64k BPE** | Best compression (4.85x) |
+| N-gram | **2-gram** | Lowest perplexity (1,405) |
+| Markov | **Context-4** | Highest predictability (99.1%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -534,7 +697,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -550,7 +714,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-28 05:26:48*
+*Report Date: 2026-01-03 07:25:42*

models/embeddings/monolingual/blk_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9dcbc456b0ca70e1cd011b575bd9a37b0edd1ea13d8af6d238e3048786a2b808
-size 1036136519
+oid sha256:849c0bf818b84aff6ad56616057a43cf1087071bb06730ea49f4f4fd35fc3106
+size 1035828932

models/embeddings/monolingual/blk_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 11378
+  "vocab_size": 11091
 }

models/embeddings/monolingual/blk_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d9d2ac1a810d1ca485988e821ea93f7cead81bbd01e1f702df5cdbefc9a3c329
-size 259398215
+oid sha256:de9eeb9d196eece4ed3965d511a1314e0f2cf59efc97054b08865446875f3bbd
+size 259311044

models/embeddings/monolingual/blk_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 11378
+  "vocab_size": 11091
 }

models/embeddings/monolingual/blk_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:679c1174fff4a0b60564e2d7d8fdcbe9d0dde2081a31b4cf3a0bc36e40b11054
-size 518310983
+oid sha256:27b17305d78322b33cc4baaad0b9e0e0fdb42f39380214b5f5965697b4cad3ae
+size 518150340

models/embeddings/monolingual/blk_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 11378
+  "vocab_size": 11091
 }

models/subword_markov/blk_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:28dce6541a3e6f62522a48c3ef3514e8cd30f7dac2a543c36e0e68e4b21df124
-size 88966
+oid sha256:f3b2f1848d241b73c04054b4240c8432860a0e2355d8a3823c68d65c2387e9f3
+size 370202

models/subword_markov/blk_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "blk",
-  "unique_contexts": 1025,
-  "total_transitions": 10706412
+  "unique_contexts": 2910,
+  "total_transitions": 6938576
 }

models/subword_markov/blk_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4c20da6a3e5ff426ad75e363a2cd4f2ef7188f31ae936d6c0bba7c9c73c0754c
-size 658773
+oid sha256:8d89d3ad1a647826ea8221d87d4f188b1de1a7d698456940763c0aed02388fdf
+size 2224824

models/subword_markov/blk_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "blk",
-  "unique_contexts": 10314,
-  "total_transitions": 10703146
+  "unique_contexts": 61297,
+  "total_transitions": 6935314
 }

models/subword_markov/blk_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4cf767c94ae3978ae158e0d1f8a9d830971b5e375df8e585577a6a1c89dc94b9
-size 2591408
+oid sha256:e7a1aadc96aaeb58a6546d15275e93c39e8ee61c836927fd4300c346d76a391b
+size 8382547

models/subword_markov/blk_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "blk",
-  "unique_contexts": 87402,
-  "total_transitions": 10699880
+  "unique_contexts": 336631,
+  "total_transitions": 6932052
 }

models/subword_markov/blk_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aac807dc615e13f9ccf7ecfa9e5c98c71ac84a741721686673e5d7acc098d699
-size 8209651
+oid sha256:ea3c67743f31a2c851e8f3f3a6592c350caec10a77d9fea4ac375a862f6be6ae
+size 20618538

models/subword_markov/blk_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "blk",
-  "unique_contexts": 367932,
-  "total_transitions": 10696614
+  "unique_contexts": 934101,
+  "total_transitions": 6928790
 }

models/subword_ngram/blk_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:785d36bf903a97e7d0b67facea78f39c29c8951b37b51046dc0e9e6a3121fff2
-size 71268
+oid sha256:aa3fd33203c185a7173b0a19a097c225f98847fe4b5e8d2470493098adeffc4f
+size 361985

models/subword_ngram/blk_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "blk",
-  "unique_ngrams": 5233,
-  "total_ngrams": 10706412
+  "unique_ngrams": 24351,
+  "total_ngrams": 6938576
 }

models/subword_ngram/blk_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:817fe896b2adcfbbdea063189814cfc80a5a45f48614b113767ff47aba9be90d
-size 640449
+oid sha256:34a673bb8ff0f8d0ce8a38c20764c99b533e5d3b05e2b0409cec35a4dc63f108
+size 1910510

models/subword_ngram/blk_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "blk",
-  "unique_ngrams": 49454,
-  "total_ngrams": 10703146
+  "unique_ngrams": 129980,
+  "total_ngrams": 6935314
 }

models/subword_ngram/blk_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:46efd55e134decbcba53878ef9b0b64a85a6ea82da4a5d9ceabee64b941b8499
-size 2718677
+oid sha256:ad489f9217fc89e1f791b12a3222263cddcb0e0633e8c10a4c0c47aedfdc9570
+size 6363654

models/subword_ngram/blk_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "blk",
-  "unique_ngrams": 213495,
-  "total_ngrams": 10699880
+  "unique_ngrams": 407218,
+  "total_ngrams": 6932052
 }

models/tokenizer/blk_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a882bbc85e0088d322795810ccf7677cd1e0687db984eb3f6e052da093a8b253
-size 699244
+oid sha256:cc0e9e63fac0ed90decd12841053888549e74d29340b045b453d1d22af1d7ddc
+size 681362

models/tokenizer/blk_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:97396401f1b9e515cb1a804b070d86c9be9cc3b2c6cb65121a39bd2cfef99a6f
-size 1224157
+oid sha256:f126ac43be0e6ce337bb595e3f2c105d53ace6fe352dcb131fd8f674b76159cf
+size 1258797

models/tokenizer/blk_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8553bc14dffbd9d31982ec3838cdb6eb14c44dad2593cd5e1d426fa5140e747f
-size 2317849
+oid sha256:167575152f3e5342b9a090a90d5e343cfd3cdd0597adfa44421d24c67af92969
+size 2501030

models/tokenizer/blk_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1d5181efa4d066f873bbbb32c1e783f73d03a3598a685ed7552a267adc845add
-size 455001
+oid sha256:7b0d3dcf17c7a21318f97a63127193ca8986b12cc4621950f17ca81225afe7fa
+size 458144

models/vocabulary/blk_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:966f586e9619499d26b0a6deabf681ad871d25f84758c959d4491c35d7eb6275
-size 138576
+oid sha256:55be2c429edb280fe0b2070ded305af1d3f07559a14b5a9bac8cf0766d0ff793
+size 1555201

models/vocabulary/blk_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "blk",
-  "vocabulary_size": 8617,
+  "vocabulary_size": 68078,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.006118561416684552,
+    "type_token_ratio": 0.5362379026940269,
     "coverage": {
-      "top_100": 0.9023449980182335,
-      "top_1000": 0.9818019451599467,
-      "top_5000": 0.9940286462141421,
-      "top_10000": 0.9971382445750568
+      "top_100": 0.10057494647582223,
+      "top_1000": 0.19264495020778324,
+      "top_5000": 0.2903072834971561,
+      "top_10000": 0.34396334260243955
     },
-    "hapax_count": 10170,
-    "hapax_ratio": 0.5413317719699793,
-    "total_documents": 3266
+    "hapax_count": 314132,
+    "hapax_ratio": 0.8218832578948746,
+    "total_documents": 3262
   }
 }

models/word_markov/blk_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:77b59373213e6ab9640ab9c00a6741256931eb6e6080b6c1228b4cfd9e47e9d5
-size 714869
+oid sha256:cdc7fcee57f6aee4ec698d86445b90105b2a0598b31b2c1cfe8e2a3d113bedf3
+size 21335834

models/word_markov/blk_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "blk",
-  "unique_contexts": 18934,
-  "total_transitions": 9027298
+  "unique_contexts": 382155,
+  "total_transitions": 709500
 }

models/word_markov/blk_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:db890b6582dae73672210c7a36187420bb86323cc97b865f35e25a7b8dbad0b5
-size 2226465
+oid sha256:0a68a819a624f91e8cbe47de3432324bf5c1e928d4c5aa0ccbbbf3f6b3680065
+size 29817330

models/word_markov/blk_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "blk",
-  "unique_contexts": 84154,
-  "total_transitions": 9024032
+  "unique_contexts": 611948,
+  "total_transitions": 706238
 }

models/word_markov/blk_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6b18a7502e58725000c65d47a227365c8462f9ab1f11229730233cedebecd447
-size 5898376
+oid sha256:525e381b3da1d52c16764a3f122122a78c7c8df2ab6c6393c010fd01ad2fae92
+size 35006655

models/word_markov/blk_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "blk",
-  "unique_contexts": 255732,
-  "total_transitions": 9020766
+  "unique_contexts": 648373,
+  "total_transitions": 702976
 }

models/word_markov/blk_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:805b9b9bed5af1500385ee02e4a7f75f1c419bc845698805c7f754f34a7f31ca
-size 13367482
+oid sha256:f077563299e6d8f13a44373afbc79c9fcf727ab7a5173e5157b9c53ad2f465bf
+size 40499621

models/word_markov/blk_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "blk",
-  "unique_contexts": 633828,
-  "total_transitions": 9017500
+  "unique_contexts": 660080,
+  "total_transitions": 699715
 }

models/word_ngram/blk_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6ec3b3a7e228f1305279a8afcc9b63a55bc7523bf3bcae6d23243ab7fc15ae11
-size 239461
+oid sha256:43d3a00ca49ee7f9fbdcc30a26d114dbd65120b3ece9b1ec2aa600399cc94300
+size 140248

models/word_ngram/blk_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "blk",
-  "unique_ngrams": 17140,
-  "total_ngrams": 9027298
+  "unique_ngrams": 4328,
+  "total_ngrams": 709500
 }

models/word_ngram/blk_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:73f83847e6fc345533e4891492d21cd2f6194794c0f5524c05bbcfcb810a0edc
-size 1085208
+oid sha256:c0f576f51b6aeb30d288d1a4ffb7348a425b04b91bee2b5ae5825d5b311bafce
+size 274266

models/word_ngram/blk_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "blk",
-  "unique_ngrams": 80070,
-  "total_ngrams": 9024032
+  "unique_ngrams": 6558,
+  "total_ngrams": 706238
 }

models/word_ngram/blk_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1c605e0f07ec921c6027b5788696cd11844dc252eae56d5ab8c19884e04e41e5
-size 3849193
+oid sha256:794371edf3eae195c85e115616f5018de7f8d6ab381948c5e08a18938a32ca1a
+size 1087943

models/word_ngram/blk_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "blk",
-  "unique_ngrams": 276816,
-  "total_ngrams": 9020766
+  "unique_ngrams": 23380,
+  "total_ngrams": 702976
 }