Upload all models and assets for bug (latest)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: bug
-language_name: BUG
+language_name: Buginese
 language_family: austronesian_sulawesi
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-austronesian_sulawesi
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.924
+    value: 4.927
   - name: best_isotropy
     type: isotropy
-    value: 0.0631
+    value: 0.0849
   - name: vocabulary_size
     type: vocab
     value: 0
 generated: 2026-01-03
 ---
 
-# BUG - Wikilangs Models
+# Buginese - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **BUG** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Buginese** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -60,7 +70,7 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
 - [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
@@ -80,43 +90,43 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 4.284x | 4.31 | 0.4916% | 36,818 |
-| **16k** | 4.514x | 4.54 | 0.5180% | 34,940 |
-| **32k** | 4.924x 🏆 | 4.96 | 0.5650% | 32,035 |
+| **8k** | 4.286x | 4.31 | 0.4928% | 36,732 |
+| **16k** | 4.517x | 4.55 | 0.5194% | 34,850 |
+| **32k** | 4.927x 🏆 | 4.96 | 0.5665% | 31,952 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Ita to Komun ri déparetema Allier Kategori:Komun ri Allier`
+**Sample 1:** `Dammartin-sur-Meuse iyanaritu séuwa komun ri déparetema Haute-Marne ri Perancis....`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ita ▁to ▁komun ▁ri ▁déparetema ▁allier ▁kategori : komun ▁ri ... (+1 more)` | 11 |
-| 16k | `▁ita ▁to ▁komun ▁ri ▁déparetema ▁allier ▁kategori : komun ▁ri ... (+1 more)` | 11 |
-| 32k | `▁ita ▁to ▁komun ▁ri ▁déparetema ▁allier ▁kategori : komun ▁ri ... (+1 more)` | 11 |
+| 8k | `▁dam martin - sur - meuse ▁iyanaritu ▁séuwa ▁komun ▁ri ... (+22 more)` | 32 |
+| 16k | `▁dammartin - sur - meuse ▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ... (+21 more)` | 31 |
+| 32k | `▁dammartin - sur - meuse ▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ... (+21 more)` | 31 |
 
-**Sample 2:** `iyanaritu séuwa komun ri déparetema Manche ri Perancis. Ita to Komun ri déparete...`
+**Sample 2:** `Bussières iyanaritu séuwa komun ri déparetema Yonne ri Perancis. Ita to Komun ri...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ▁manche ▁ri ▁perancis . ▁ita ... (+10 more)` | 20 |
-| 16k | `▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ▁manche ▁ri ▁perancis . ▁ita ... (+10 more)` | 20 |
-| 32k | `▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ▁manche ▁ri ▁perancis . ▁ita ... (+10 more)` | 20 |
+| 8k | `▁bussières ▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ▁yonne ▁ri ▁perancis . ... (+11 more)` | 21 |
+| 16k | `▁bussières ▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ▁yonne ▁ri ▁perancis . ... (+11 more)` | 21 |
+| 32k | `▁bussières ▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ▁yonne ▁ri ▁perancis . ... (+11 more)` | 21 |
 
-**Sample 3:** `iyanaritu séuwa komun ri déparetema Gard ri Perancis. Ita to Komun ri déparetema...`
+**Sample 3:** `Pujols iyanaritu séuwa komun ri déparetema Gironde ri Perancis. Ita to Komun ri ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ▁gard ▁ri ▁perancis . ▁ita ... (+10 more)` | 20 |
-| 16k | `▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ▁gard ▁ri ▁perancis . ▁ita ... (+10 more)` | 20 |
-| 32k | `▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ▁gard ▁ri ▁perancis . ▁ita ... (+10 more)` | 20 |
+| 8k | `▁pujols ▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ▁gironde ▁ri ▁perancis . ... (+11 more)` | 21 |
+| 16k | `▁pujols ▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ▁gironde ▁ri ▁perancis . ... (+11 more)` | 21 |
+| 32k | `▁pujols ▁iyanaritu ▁séuwa ▁komun ▁ri ▁déparetema ▁gironde ▁ri ▁perancis . ... (+11 more)` | 21 |
 
 
 ### Key Findings
 
-- **Best Compression:** 32k achieves 4.924x compression
-- **Lowest UNK Rate:** 8k with 0.4916% unknown tokens
+- **Best Compression:** 32k achieves 4.927x compression
+- **Lowest UNK Rate:** 8k with 0.4928% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -134,11 +144,13 @@ Below are sample sentences tokenized with each vocabulary size:
 | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
 |--------|---------|------------|---------|----------------|------------------|-------------------|
 | **2-gram** | Word | 75 🏆 | 6.23 | 1,721 | 84.8% | 98.5% |
-| **2-gram** | Subword | 168 | 7.39 | 2,166 | 81.3% | 99.5% |
+| **2-gram** | Subword | 167 | 7.39 | 2,161 | 81.3% | 99.5% |
 | **3-gram** | Word | 118 | 6.89 | 2,060 | 74.9% | 98.6% |
-| **3-gram** | Subword | 512 | 9.00 | 10,883 | 62.7% | 89.5% |
-| **4-gram** | Word | 228 | 7.84 | 4,992 | 61.5% | 96.5% |
-| **4-gram** | Subword | 938 | 9.87 | 41,978 | 58.6% | 80.3% |
+| **3-gram** | Subword | 511 | 9.00 | 10,879 | 62.7% | 89.5% |
+| **4-gram** | Word | 229 | 7.84 | 4,999 | 61.5% | 96.5% |
+| **4-gram** | Subword | 938 | 9.87 | 41,989 | 58.6% | 80.3% |
+| **5-gram** | Word | 304 | 8.25 | 4,200 | 51.5% | 97.0% |
+| **5-gram** | Subword | 1,221 | 10.25 | 76,709 | 57.6% | 78.3% |
 
 ### Top 5 N-grams by Size
 
@@ -146,9 +158,9 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `komun ri` | 40,954 |
+| 1 | `komun ri` | 40,953 |
 | 2 | `ri déparetema` | 25,713 |
-| 3 | `kategori komun` | 15,119 |
+| 3 | `kategori komun` | 15,118 |
 | 4 | `ita to` | 13,903 |
 | 5 | `to komun` | 13,889 |
 
@@ -157,7 +169,7 @@ Below are sample sentences tokenized with each vocabulary size:
 | Rank | N-gram | Count |
 |------|--------|-------|
 | 1 | `komun ri déparetema` | 25,709 |
-| 2 | `kategori komun ri` | 15,118 |
+| 2 | `kategori komun ri` | 15,117 |
 | 3 | `to komun ri` | 13,889 |
 | 4 | `ita to komun` | 13,889 |
 | 5 | `iyanaritu séuwa komun` | 13,324 |
@@ -166,48 +178,68 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ita to komun ri` | 13,889 |
-| 2 | `to komun ri déparetema` | 13,889 |
-| 3 | `perancis ita to komun` | 12,095 |
+| 1 | `to komun ri déparetema` | 13,889 |
+| 2 | `ita to komun ri` | 13,889 |
+| 3 | `perancis ita to komun` | 12,104 |
 | 4 | `iyanaritu séuwa komun ri` | 11,780 |
 | 5 | `séuwa komun ri déparetema` | 11,779 |
 
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ita to komun ri déparetema` | 13,889 |
+| 2 | `perancis ita to komun ri` | 12,104 |
+| 3 | `iyanaritu séuwa komun ri déparetema` | 11,779 |
+| 4 | `ri perancis ita to komun` | 10,125 |
+| 5 | `to komun ri déparetema haute` | 1,825 |
+
 **2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `r i` | 90,073 |
-| 2 | `a _` | 63,521 |
-| 3 | `i _` | 58,103 |
+| 1 | `r i` | 90,059 |
+| 2 | `a _` | 63,515 |
+| 3 | `i _` | 58,114 |
 | 4 | `_ r` | 57,562 |
-| 5 | `t e` | 57,384 |
+| 5 | `t e` | 57,375 |
 
 **3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
 | 1 | `_ r i` | 56,241 |
-| 2 | `r i _` | 55,682 |
-| 3 | `m u n` | 43,032 |
-| 4 | `u n _` | 42,982 |
-| 5 | `k o m` | 42,818 |
+| 2 | `r i _` | 55,684 |
+| 3 | `m u n` | 43,031 |
+| 4 | `u n _` | 42,981 |
+| 5 | `k o m` | 42,817 |
 
 **4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `_ r i _` | 55,380 |
-| 2 | `o m u n` | 42,739 |
-| 3 | `k o m u` | 42,738 |
-| 4 | `m u n _` | 42,683 |
-| 5 | `n _ r i` | 41,407 |
+| 1 | `_ r i _` | 55,382 |
+| 2 | `o m u n` | 42,738 |
+| 3 | `k o m u` | 42,737 |
+| 4 | `m u n _` | 42,682 |
+| 5 | `n _ r i` | 41,406 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `k o m u n` | 42,737 |
+| 2 | `o m u n _` | 42,672 |
+| 3 | `n _ r i _` | 41,389 |
+| 4 | `u n _ r i` | 40,955 |
+| 5 | `m u n _ r` | 40,953 |
 
 
 ### Key Findings
 
 - **Best Perplexity:** 2-gram (word) with 75
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~80% of corpus
+- **Coverage:** Top-1000 patterns cover ~78% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -223,14 +255,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
 |---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.5094 | 1.423 | 2.20 | 33,148 | 49.1% |
-| **1** | Subword | 0.6420 | 1.560 | 6.04 | 1,115 | 35.8% |
-| **2** | Word | 0.1229 | 1.089 | 1.21 | 72,816 | 87.7% |
-| **2** | Subword | 0.6776 | 1.600 | 3.79 | 6,727 | 32.2% |
-| **3** | Word | 0.0488 | 1.034 | 1.07 | 87,927 | 95.1% |
-| **3** | Subword | 0.6911 | 1.614 | 3.05 | 25,458 | 30.9% |
-| **4** | Word | 0.0143 🏆 | 1.010 | 1.02 | 93,631 | 98.6% |
-| **4** | Subword | 0.5492 | 1.463 | 2.16 | 77,506 | 45.1% |
+| **1** | Word | 0.5091 | 1.423 | 2.20 | 33,150 | 49.1% |
+| **1** | Subword | 0.6409 | 1.559 | 6.02 | 1,114 | 35.9% |
+| **2** | Word | 0.1228 | 1.089 | 1.21 | 72,762 | 87.7% |
+| **2** | Subword | 0.6769 | 1.599 | 3.79 | 6,702 | 32.3% |
+| **3** | Word | 0.0488 | 1.034 | 1.07 | 87,846 | 95.1% |
+| **3** | Subword | 0.6926 | 1.616 | 3.05 | 25,381 | 30.7% |
+| **4** | Word | 0.0142 🏆 | 1.010 | 1.02 | 93,544 | 98.6% |
+| **4** | Subword | 0.5499 | 1.464 | 2.16 | 77,409 | 45.0% |
 
 ### Generated Text Samples (Word-based)
 
@@ -238,27 +270,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `ri déparetema gironde ri déparetema eure et loir kategori komun ri déparetema manche ri aisne katego...`
-2. `komun ri perancis ita to komun ri dordogne ri déparetema somme kategori kota ri déparetema haute`
-3. `déparetema gironde ri haute saône ri déparetema haute provence ri perancis ita to komun ri déparetem...`
+1. `ri haute loire rocé roches avrillé caa guillaucourt guillemont guizancourt guyencourt saulcourt iyan...`
+2. `komun ri déparetema dordogne ri déparetema somme ri lino kaminang maégai napunnai peddang malampe si...`
+3. `déparetema aube ri déparetema vosges kategori komun ri manoraŋna perancis ita to komun ri perancis i...`
 
 **Context Size 2:**
 
-1. `komun ri provinsi messina komun ri déparetema haute loire ri perancis ita to komun ri déparetema dor...`
-2. `ri déparetema manche ri perancis ita to komun ri déparetema dordogne ri perancis ita to komun ri`
-3. `kategori komun ri déparetema manche kategori komun ri déparetema ain kategori komun ri alpes de haut...`
+1. `komun ri ardennes`
+2. `ri déparetema somme ri perancis ita to komun ri finistère`
+3. `kategori komun ri déparetema somme kategori komun ri déparetema haute saône kategori komun ri gard`
 
 **Context Size 3:**
 
-1. `komun ri déparetema somme kategori komun ri eure et loir ri perancis ita to komun ri déparetema haut...`
-2. `kategori komun ri aisne`
-3. `to komun ri déparetema dordogne ri perancis ita to komun ri déparetema vosges ri perancis ita to kom...`
+1. `komun ri déparetema somme ri perancis ita to komun ri déparetema somme ri perancis ita to komun ri`
+2. `kategori komun ri guadeloupe`
+3. `ita to komun ri déparetema eure et loir kategori komun ri hautes pyrénées`
 
 **Context Size 4:**
 
-1. `ita to komun ri déparetema gironde ri perancis ita to komun ri déparetema haute saône ri perancis it...`
-2. `to komun ri déparetema vosges kategori komun ri vosges`
-3. `perancis ita to komun ri déparetema côtes d armor kategori komun ri côtes d armor`
+1. `to komun ri déparetema ain kategori komun ri ain`
+2. `ita to komun ri déparetema vosges ri perancis ita to komun ri déparetema gard ri perancis ita to kom...`
+3. `perancis ita to komun ri déparetema haute saône ri perancis ita to komun ri déparetema yvelines kate...`
 
 
 ### Generated Text Samples (Subword-based)
@@ -267,26 +299,26 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_séunas._koiya_r`
-2. `ares._retoépesat`
-3. `riséunanetaŋn_pa`
+1. `_te_raweri:korom`
+2. `apajesaniritori_`
+3. `resèséun_i:ko_ay`
 
 **Context Size 2:**
 
-1. `ri_ube_katespia_f`
-2. `a_to_komun_ri:kom`
-3. `i_aretemay_(caven`
+1. `ritu_séuwa_katema`
+2. `a_agny-saônes_bin`
+3. `i_dépari_lancis_s`
 
 **Context Size 3:**
 
-1. `_ri_déparetema_vos`
-2. `ri_déparetema_eure`
-3. `mun_ri_perancis._i`
+1. `_ri_aisnes_kategor`
+2. `ri_déparetema_eurc`
+3. `mun_ri_allers_kate`
 
 **Context Size 4:**
 
-1. `_ri_perancis._ita_t`
-2. `omun_ri_aisne_ri_pe`
+1. `_ri_déparetema_côte`
+2. `omun_ri_ain_vignoll`
 3. `komun_ri_déparetema`
 
 
@@ -294,7 +326,7 @@ Below are text samples generated from each subword-based Markov chain model:
 
 - **Best Predictability:** Context-4 (word) with 98.6% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (77,506 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (77,409 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -310,64 +342,64 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 13,441 |
-| Total Tokens | 358,235 |
-| Mean Frequency | 26.65 |
+| Vocabulary Size | 13,449 |
+| Total Tokens | 358,170 |
+| Mean Frequency | 26.63 |
 | Median Frequency | 2 |
-| Frequency Std Dev | 719.11 |
+| Frequency Std Dev | 718.89 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ri | 55,390 |
-| 2 | komun | 42,680 |
+| 1 | ri | 55,392 |
+| 2 | komun | 42,679 |
 | 3 | déparetema | 27,244 |
-| 4 | kategori | 15,401 |
-| 5 | to | 14,028 |
+| 4 | kategori | 15,395 |
+| 5 | to | 14,029 |
 | 6 | ita | 13,904 |
-| 7 | iyanaritu | 13,503 |
+| 7 | iyanaritu | 13,505 |
 | 8 | séuwa | 13,393 |
 | 9 | perancis | 12,636 |
-| 10 | haute | 6,207 |
+| 10 | haute | 6,206 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ᨆᨘᨄᨗ | 2 |
-| 2 | ᨕᨗᨊᨘᨂᨛᨊ | 2 |
-| 3 | ᨒᨘ | 2 |
-| 4 | ᨅᨀᨗ | 2 |
-| 5 | ᨀᨀᨀᨀ | 2 |
-| 6 | ᨉᨗᨛᨄᨗᨛ | 2 |
-| 7 | days | 2 |
-| 8 | after | 2 |
-| 9 | federal | 2 |
-| 10 | ᨔᨛᨀᨗᨈ | 2 |
+| 1 | museum | 2 |
+| 2 | tychy | 2 |
+| 3 | tangnga | 2 |
+| 4 | miniaturowej | 2 |
+| 5 | sztuki | 2 |
+| 6 | profesjonalnej | 2 |
+| 7 | wideo | 2 |
+| 8 | nietypowe | 2 |
+| 9 | sztalugi | 2 |
+| 10 | zapałek | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9107 |
-| R² (Goodness of Fit) | 0.956604 |
+| Zipf Coefficient | 0.9102 |
+| R² (Goodness of Fit) | 0.956494 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 83.0% |
+| Top 100 | 83.1% |
 | Top 1,000 | 89.7% |
 | Top 5,000 | 95.1% |
 | Top 10,000 | 98.1% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9566 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 83.0% of corpus
-- **Long Tail:** 3,441 words needed for remaining 1.9% coverage
+- **Zipf Compliance:** R²=0.9565 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 83.1% of corpus
+- **Long Tail:** 3,449 words needed for remaining 1.9% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -383,37 +415,40 @@ Below are text samples generated from each subword-based Markov chain model:
 
 ### 5.1 Cross-Lingual Alignment
 
-> *Note: Multilingual alignment visualization not available for this language.*
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
 
 
 ### 5.2 Model Comparison
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.0631 🏆 | 0.6817 | N/A | N/A |
-| **mono_64d** | 64 | 0.0264 | 0.6525 | N/A | N/A |
-| **mono_128d** | 128 | 0.0035 | 0.7173 | N/A | N/A |
+| **mono_32d** | 32 | 0.0849 🏆 | 0.7683 | N/A | N/A |
+| **mono_64d** | 64 | 0.0269 | 0.6385 | N/A | N/A |
+| **mono_128d** | 128 | 0.0039 | 0.6251 | N/A | N/A |
+| **aligned_32d** | 32 | 0.0849 | 0.7636 | 0.0000 | 0.0300 |
+| **aligned_64d** | 64 | 0.0269 | 0.6542 | 0.0120 | 0.1200 |
+| **aligned_128d** | 128 | 0.0039 | 0.6125 | 0.0300 | 0.1620 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.0631 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.6838. Lower values indicate better semantic separation.
-- **Alignment Quality:** No aligned models evaluated in this run.
+- **Best Isotropy:** mono_32d with 0.0849 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.6770. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 3.0% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
 ## 6.  Morphological Analysis (Experimental)
 
-> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
-
 This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
 
 ### 6.1 Productivity & Complexity
 
 | Metric | Value | Interpretation | Recommendation |
 |--------|-------|----------------|----------------|
-| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
-| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **0.239** | High formulaic/idiomatic content | - |
 
 ### 6.2 Affix Inventory (Productive Units)
 
@@ -422,23 +457,21 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Prefixes
 | Prefix | Examples |
 |--------|----------|
-| `-ma` | marainville, mailhac, mauges |
-| `-mo` | molliens, montmotier, morin |
-| `-ch` | châtel, chèze, chauffourt |
-| `-co` | confort, coulombiers, coux |
-| `-la` | lacalm, lasse, lacour |
+| `-ma` | marson, massoins, maël |
+| `-mo` | montégut, moncale, morton |
+| `-ch` | chépy, cheylard, chatel |
 
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-s` | pozières, serbonnes, molliens |
-| `-e` | givonne, marainville, roville |
-| `-es` | pozières, serbonnes, mauges |
-| `-rt` | confort, chauffourt, saucourt |
-| `-le` | marainville, roville, touille |
-| `-urt` | chauffourt, saucourt, pignicourt |
-| `-ourt` | chauffourt, saucourt, pignicourt |
-| `-lle` | marainville, roville, touille |
+| `-s` | siprus, massoins, hiis |
+| `-e` | épagne, aizanville, vesle |
+| `-es` | barges, vellèches, laspènes |
+| `-le` | aizanville, vesle, gameville |
+| `-lle` | aizanville, gameville, girondelle |
+| `-rt` | begnécourt, hinacourt, bouzincourt |
+| `-urt` | begnécourt, hinacourt, bouzincourt |
+| `-ourt` | begnécourt, hinacourt, bouzincourt |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -446,14 +479,15 @@ Bound stems are high-frequency subword units that are semantically cohesive but
 
 | Stem | Cohesion | Substitutability | Examples |
 |------|----------|------------------|----------|
-| `ngka` | 1.37x | 20 contexts | éngka, angka, engka |
-| `appa` | 1.39x | 15 contexts | lappa, nappa, tappa |
-| `engk` | 1.41x | 9 contexts | engka, engkai, engkaé |
-| `seng` | 1.34x | 10 contexts | aseng, naseng, siseng |
-| `asen` | 1.35x | 8 contexts | aseng, naseng, asenna |
-| `unna` | 1.32x | 6 contexts | punna, umunna, punnai |
-| `enna` | 1.37x | 5 contexts | asenna, lalenna, sisenna |
-| `yana` | 1.30x | 5 contexts | iyana, iyanae, iyanaé |
+| `ngka` | 1.51x | 20 contexts | angka, engka, éngka |
+| `appa` | 1.55x | 15 contexts | cappa, nappa, lappa |
+| `engk` | 1.57x | 9 contexts | engka, engkaé, engkai |
+| `seng` | 1.50x | 10 contexts | aseng, siseng, naseng |
+| `asen` | 1.46x | 8 contexts | aseng, asenna, naseng |
+| `unna` | 1.46x | 6 contexts | punna, punnai, umunna |
+| `enna` | 1.46x | 5 contexts | asenna, sisenna, lalenna |
+| `yana` | 1.38x | 5 contexts | iyana, iyanaé, iyanae |
+| `iyan` | 1.37x | 5 contexts | iyana, iyanaé, iyanae |
 
 ### 6.4 Affix Compatibility (Co-occurrence)
 
@@ -461,16 +495,16 @@ This table shows which prefixes and suffixes most frequently co-occur on the sam
 
 | Prefix | Suffix | Frequency | Examples |
 |--------|--------|-----------|----------|
-| `-la` | `-e` | 76 words | lagardelle, lange |
-| `-ch` | `-s` | 63 words | charnois, chassagnes |
-| `-ma` | `-e` | 54 words | marville, maddare |
-| `-co` | `-s` | 53 words | collonges, corps |
-| `-mo` | `-s` | 46 words | moffans, moulédous |
-| `-ch` | `-es` | 44 words | chassagnes, chaumes |
-| `-ma` | `-s` | 43 words | mazères, martigues |
-| `-ch` | `-e` | 41 words | challerange, champclause |
-| `-co` | `-e` | 35 words | corbière, conie |
-| `-co` | `-es` | 28 words | collonges, coyolles |
+| `-ch` | `-s` | 56 words | chaulnes, champdeniers |
+| `-ch` | `-e` | 46 words | châtaigneraie, chabre |
+| `-ma` | `-e` | 44 words | maritime, maire |
+| `-ma` | `-s` | 43 words | mainvilliers, mandres |
+| `-mo` | `-s` | 41 words | molins, moulines |
+| `-ch` | `-es` | 40 words | chaulnes, chamvres |
+| `-mo` | `-e` | 19 words | motteville, moulière |
+| `-ma` | `-es` | 18 words | mandres, maulichères |
+| `-mo` | `-on` | 18 words | monthodon, montfaucon |
+| `-mo` | `-rt` | 13 words | montlibert, montescourt |
 
 ### 6.5 Recursive Morpheme Segmentation
 
@@ -478,26 +512,26 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 
 | Word | Suggested Split | Confidence | Stem |
 |------|-----------------|------------|------|
-| lagardelle | **`la-garde-lle`** | 6.0 | `garde` |
-| lavilledieu | **`la-villedieu`** | 4.5 | `villedieu` |
-| laboissière | **`la-boissière`** | 4.5 | `boissière` |
-| malaincourt | **`ma-la-inco-urt`** | 4.5 | `inco` |
-| colleville | **`co-llev-ille`** | 3.0 | `llev` |
-| maizières | **`ma-izièr-es`** | 3.0 | `izièr` |
-| champenoises | **`ch-ampenois-es`** | 3.0 | `ampenois` |
-| chavignon | **`ch-avign-on`** | 3.0 | `avign` |
-| montescourt | **`mo-ntesc-ourt`** | 3.0 | `ntesc` |
-| châteauredon | **`ch-âteaured-on`** | 3.0 | `âteaured` |
-| chevannes | **`ch-evann-es`** | 3.0 | `evann` |
-| lamasquère | **`la-ma-squère`** | 3.0 | `squère` |
-| landaville | **`la-ndav-ille`** | 3.0 | `ndav` |
-| mourvilles | **`mo-urvill-es`** | 3.0 | `urvill` |
-| malvières | **`ma-lvièr-es`** | 3.0 | `lvièr` |
+| lagardelle | **`lagarde-lle`** | 4.5 | `lagarde` |
+| motteville | **`mo-ttev-ille`** | 3.0 | `ttev` |
+| chalencon | **`ch-alenc-on`** | 3.0 | `alenc` |
+| champignelles | **`ch-ampignell-es`** | 3.0 | `ampignell` |
+| chamarandes | **`ch-amarand-es`** | 3.0 | `amarand` |
+| martinsart | **`ma-rtinsa-rt`** | 3.0 | `rtinsa` |
+| manancourt | **`ma-nanc-ourt`** | 3.0 | `nanc` |
+| charleville | **`ch-arlev-ille`** | 3.0 | `arlev` |
+| montheries | **`mo-ntheri-es`** | 3.0 | `ntheri` |
+| marseille | **`ma-rsei-lle`** | 3.0 | `rsei` |
+| champvallon | **`ch-ampvall-on`** | 3.0 | `ampvall` |
+| monthodon | **`mo-nthod-on`** | 3.0 | `nthod` |
+| mazerolles | **`ma-zeroll-es`** | 3.0 | `zeroll` |
+| chevrières | **`ch-evrièr-es`** | 3.0 | `evrièr` |
+| montagnes | **`mo-ntagn-es`** | 3.0 | `ntagn` |
 
 ### 6.6 Linguistic Interpretation
 
 > **Automated Insight:**
-The language BUG appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+The language Buginese shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
 
 ---
 ## 7. Summary & Recommendations
@@ -508,7 +542,7 @@ The language BUG appears to be more isolating or has a highly fixed vocabulary.
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.92x) |
+| Tokenizer | **32k BPE** | Best compression (4.93x) |
 | N-gram | **2-gram** | Lowest perplexity (75) |
 | Markov | **Context-4** | Highest predictability (98.6%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
@@ -724,4 +758,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 08:55:12*
+*Report Date: 2026-01-03 19:48:58*

models/embeddings/aligned/bug_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6284ca3d9a3a42e01b0cf4545e24bb0c1d475694590698c54d9c4a80e7d75c2
+size 1025501057

models/embeddings/aligned/bug_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "bug", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/bug_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd3eb6e77750f40e5d5f91e6f3ab65e312b47e691439b0f2060b297f687449ed
+size 65664

models/embeddings/aligned/bug_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "bug",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 613,
+  "vocab_size": 1441
+}

models/embeddings/aligned/bug_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40e2a1a18de65d5de1a8fc806805d83593b39dc96137b5abea4a5e1da6a60cb1
+size 256394369

models/embeddings/aligned/bug_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "bug", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/bug_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:945f384ccc372cc36717260f6373c3e1f47782edbffbea2a47dd061ff46e24fe
+size 4224

models/embeddings/aligned/bug_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "bug",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 613,
+  "vocab_size": 1441
+}

models/embeddings/aligned/bug_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53e95dd4467c2ace8757700001cab0f690cefdd92e9b6f10383f6a6bcd5d1273
+size 512763265

models/embeddings/aligned/bug_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "bug", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/bug_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e681dd6f006c857e0909eac3925604dc1646a2a2b381678f32f7a1692d6d69a4
+size 16512

models/embeddings/aligned/bug_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "bug",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 613,
+  "vocab_size": 1441
+}

models/embeddings/monolingual/bug_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:60e643ec1ece75f273fcafba6e20d15dd273d9a060acc636c62b079cc3827632
-size 1025503150
+oid sha256:e6284ca3d9a3a42e01b0cf4545e24bb0c1d475694590698c54d9c4a80e7d75c2
+size 1025501057

models/embeddings/monolingual/bug_128d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 128
   },
-  "vocab_size": 1443
+  "vocab_size": 1441
 }

models/embeddings/monolingual/bug_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:becf192777316400ac67a33873637aadaeabcd3ab768294b2064c3c0615f144b
-size 256394926
+oid sha256:40e2a1a18de65d5de1a8fc806805d83593b39dc96137b5abea4a5e1da6a60cb1
+size 256394369

models/embeddings/monolingual/bug_32d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 32
   },
-  "vocab_size": 1443
+  "vocab_size": 1441
 }

models/embeddings/monolingual/bug_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d9823040ce7cd355f045c8caad10719684ec4e18bf1bef4ea891663c355cb360
-size 512764334
+oid sha256:53e95dd4467c2ace8757700001cab0f690cefdd92e9b6f10383f6a6bcd5d1273
+size 512763265

models/embeddings/monolingual/bug_64d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 64
   },
-  "vocab_size": 1443
+  "vocab_size": 1441
 }

models/subword_markov/bug_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ea6b54847a784f51e2f16c837e7951d96c474085d40f16a957be9eecbdceb2d7
-size 56451
+oid sha256:b389c35c312b58052a2dc019fbad8167b79bf68a5405f5dff1de2d71a71a8d36
+size 56660

models/subword_markov/bug_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "bug",
-  "unique_contexts": 1115,
-  "total_transitions": 2377233
+  "unique_contexts": 1114,
+  "total_transitions": 2376929
 }

models/subword_markov/bug_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fc9770904232ce07f4aa4fa00d4f31741086ce42dde9337c557a94fcd91601d3
-size 229577
+oid sha256:182b9dbf6626ea9e4f0e4b177f5c95143dca05ad5add967e2b27000e96233bcd
+size 227384

models/subword_markov/bug_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "bug",
-  "unique_contexts": 6727,
-  "total_transitions": 2361714
+  "unique_contexts": 6702,
+  "total_transitions": 2361412
 }

models/subword_markov/bug_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1851961810e4cd0121317f3bf6e74a5b2ad2d767fbf3ea615e494b1b33a0d9ff
-size 651956
+oid sha256:b5e40536d41489c4041689a30506095ba3fb77da5c8868e6886f33d7c7450a01
+size 645144

models/subword_markov/bug_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "bug",
-  "unique_contexts": 25458,
-  "total_transitions": 2346195
+  "unique_contexts": 25381,
+  "total_transitions": 2345895
 }

models/subword_markov/bug_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1991e713220325ecadbfc81c2871898ab38be9aff994ef7d45517d9081e2d843
-size 1459206
+oid sha256:aa35faf25a2d4148ed795ce56be93ae55d1cc642ad22468a17e480de6aef5662
+size 1439503

models/subword_markov/bug_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "bug",
-  "unique_contexts": 77506,
-  "total_transitions": 2330676
+  "unique_contexts": 77409,
+  "total_transitions": 2330378
 }

models/subword_ngram/bug_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bc0ffd67460f75034bcef5b208e6dcebf5026c731829639ee1c9b37cd94f0255
-size 29845
+oid sha256:a8f6874249088a09af3813be47318476386d90591fb377899a82d434cfd66855
+size 29748

models/subword_ngram/bug_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "bug",
-  "unique_ngrams": 2166,
-  "total_ngrams": 2377233
+  "unique_ngrams": 2161,
+  "total_ngrams": 2376929
 }

models/subword_ngram/bug_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:27dae62a90b04631ca45a23226c4fb6e3e5676dd3b56fda2064206ccd05cc271
-size 133308
+oid sha256:0ff980159f39d0d0606f2c18b8ee48a9d727109530ac5cab0342b528b119f52d
+size 132924

models/subword_ngram/bug_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "bug",
-  "unique_ngrams": 10883,
-  "total_ngrams": 2361714
+  "unique_ngrams": 10879,
+  "total_ngrams": 2361412
 }

models/subword_ngram/bug_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4d7a75327568e461ca60da5b19c31846f906735013d0944cb9687eb7e175681a
-size 522479
+oid sha256:fc1c9e4f98153ce090d2e73131732fab8fe46b10d66108a97bf20d45dac52d58
+size 517391

models/subword_ngram/bug_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "bug",
-  "unique_ngrams": 41978,
-  "total_ngrams": 2346195
+  "unique_ngrams": 41989,
+  "total_ngrams": 2345895
 }

models/subword_ngram/bug_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc40434fb726a016b8e0b7bbbe1ebcde234880b7b1b285183236316a5506f456
+size 895843

models/subword_ngram/bug_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "bug",
+  "unique_ngrams": 76709,
+  "total_ngrams": 2330378
+}

models/tokenizer/bug_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f695d915517db069983efcf78f1827a707c9d7e5df1fe96fe7b390346018196e
-size 518421
+oid sha256:26fa62a9b406e1e9bba2baf575cfc8f9a17114a97539834419efa0e46a2d504c
+size 518399

models/tokenizer/bug_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:41e741a9c77783f82b6eef2bd92933e422989f91b37dafb131fcbf96a6c44a90
-size 821175
+oid sha256:0655348011befa58bdb22261e527b681b994f038c4ad3fe7c9c06fe3ae0cdce6
+size 821102

models/tokenizer/bug_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ae5e6410beed7f97e2ba0a2f82bd06f81b54b111021264a322f628b6ebd7930f
+oid sha256:af26935d851b7fa79d11da73a096e34eea80260b1ae4cccc94755ff6a9a21e29
 size 372086

models/vocabulary/bug_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:eaf37b2a8977cb6828c5e00937c105d6dd1b5753eb506d9a9c3d3715c43116ba
-size 212963
+oid sha256:d9040b7aa9e107c5948c3f57a85fb919bf9a3a99bc1156138a662f6b971b4187
+size 212837

models/vocabulary/bug_vocabulary_metadata.json

@@ -1,17 +1,17 @@
 {
   "language": "bug",
-  "vocabulary_size": 13441,
+  "vocabulary_size": 13449,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.08785621316176548,
+    "type_token_ratio": 0.08787801012348749,
     "coverage": {
-      "top_100": 0.7870075448937048,
-      "top_1000": 0.8499830689622332,
-      "top_5000": 0.9016359615242167,
-      "top_10000": 0.9294954550745494
+      "top_100": 0.7871183516655068,
+      "top_1000": 0.8499601781268108,
+      "top_5000": 0.9015830848324967,
+      "top_10000": 0.9294557500932705
     },
-    "hapax_count": 19769,
-    "hapax_ratio": 0.5952725082806384,
-    "total_documents": 15519
+    "hapax_count": 19763,
+    "hapax_ratio": 0.5950560038540287,
+    "total_documents": 15517
   }
 }

models/word_markov/bug_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:11fe5e192c8147c9659e92fea9d4a1deda14c8f7dd8f48d353ccca9695ddc8e5
-size 879712
+oid sha256:922e8750e9a5e1b6e55c94fd16434c4d350a0ac640cc39525d559081508ff8c4
+size 879339

models/word_markov/bug_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "bug",
-  "unique_contexts": 33148,
-  "total_transitions": 362485
+  "unique_contexts": 33150,
+  "total_transitions": 362416
 }

models/word_markov/bug_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6e286442d9daf9a0a40fbc601c40874bbd94485367efec9e0aa5fa8ada829a45
-size 1390138
+oid sha256:2e10ff8d20655316f2b1ccb414fc37e3821b2969a9d32563d07bf03438321740
+size 1390068

models/word_markov/bug_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "bug",
-  "unique_contexts": 72816,
-  "total_transitions": 346966
+  "unique_contexts": 72762,
+  "total_transitions": 346899
 }

models/word_markov/bug_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c638220df23f4ff0bdbe9ae2d3083f2ce355e1c83d1a468b2bea91d89552f40e
-size 1674825
+oid sha256:79bd483a98b3279fa6076f7b52fc380c847e0ab70fe130326276f19e268140cb
+size 1675726

models/word_markov/bug_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "bug",
-  "unique_contexts": 87927,
-  "total_transitions": 331447
+  "unique_contexts": 87846,
+  "total_transitions": 331382
 }