Upload all models and assets for bdr (latest)

.gitattributes

@@ -38,3 +38,4 @@ visualizations/performance_dashboard.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: bdr
-language_name: BDR
+language_name: West Coast Bajau
 language_family: austronesian_other
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-austronesian_other
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.792
+    value: 4.803
   - name: best_isotropy
     type: isotropy
-    value: 0.0482
+    value: 0.0390
   - name: vocabulary_size
     type: vocab
     value: 0
 generated: 2026-01-03
 ---
 
-# BDR - Wikilangs Models
+# West Coast Bajau - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **BDR** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **West Coast Bajau** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -60,7 +70,7 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
 - [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
@@ -80,35 +90,35 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 4.792x 🏆 | 4.81 | 0.1661% | 33,107 |
+| **8k** | 4.803x 🏆 | 4.82 | 0.1461% | 32,844 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Nimbug iyono indu' manuk nuut ngentelo ta' keteraan manuk lain.`
+**Sample 1:** `Dugal tu io akan bungkar pedih ni amun niak mangam buas dembangi , Dugal tu baya...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁nimbug ▁iyono ▁indu ' ▁manuk ▁nuut ▁ngentelo ▁ta ' ▁keteraan ... (+3 more)` | 13 |
+| 8k | `▁dugal ▁tu ▁io ▁akan ▁bungkar ▁pedih ▁ni ▁amun ▁niak ▁mangam ... (+15 more)` | 25 |
 
-**Sample 2:** `Raja iyo no' dangan jomo kuleh kuasa diom pemerintah dikau kerajaan.Endo rojo pi...`
+**Sample 2:** `Bul (Ling Melayu: Bola) iyo dembua barang pinakai untuk besukan`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁raja ▁iyo ▁no ' ▁dangan ▁jomo ▁kuleh ▁kuasa ▁diom ▁pemerintah ... (+20 more)` | 30 |
+| 8k | `▁bul ▁( ling ▁melayu : ▁bola ) ▁iyo ▁dembua ▁barang ... (+3 more)` | 13 |
 
-**Sample 3:** `Para-para iyo no tempat ngena segala barang enjata rak`
+**Sample 3:** `Tupi sungku tu sejenis tupi tradisional jomo sama. Tupi sungku pinakai untuk nge...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁para - para ▁iyo ▁no ▁tempat ▁ngena ▁segala ▁barang ▁enjata ... (+1 more)` | 11 |
+| 8k | `▁tupi ▁sungku ▁tu ▁sejenis ▁tupi ▁tradisional ▁jomo ▁sama . ▁tupi ... (+10 more)` | 20 |
 
 
 ### Key Findings
 
-- **Best Compression:** 8k achieves 4.792x compression
-- **Lowest UNK Rate:** 8k with 0.1661% unknown tokens
+- **Best Compression:** 8k achieves 4.803x compression
+- **Lowest UNK Rate:** 8k with 0.1461% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -126,11 +136,13 @@ Below are sample sentences tokenized with each vocabulary size:
 | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
 |--------|---------|------------|---------|----------------|------------------|-------------------|
 | **2-gram** | Word | 287 | 8.16 | 401 | 53.3% | 100.0% |
-| **2-gram** | Subword | 181 🏆 | 7.50 | 597 | 77.1% | 100.0% |
-| **3-gram** | Word | 221 | 7.79 | 271 | 59.6% | 100.0% |
-| **3-gram** | Subword | 1,140 | 10.15 | 3,421 | 32.8% | 85.1% |
-| **4-gram** | Word | 273 | 8.09 | 346 | 51.1% | 100.0% |
-| **4-gram** | Subword | 4,426 | 12.11 | 11,413 | 17.0% | 52.6% |
+| **2-gram** | Subword | 180 | 7.49 | 593 | 77.1% | 100.0% |
+| **3-gram** | Word | 219 | 7.78 | 269 | 59.9% | 100.0% |
+| **3-gram** | Subword | 1,136 | 10.15 | 3,407 | 32.8% | 85.1% |
+| **4-gram** | Word | 272 | 8.09 | 345 | 51.2% | 100.0% |
+| **4-gram** | Subword | 4,404 | 12.10 | 11,348 | 17.0% | 52.7% |
+| **5-gram** | Word | 110 🏆 | 6.78 | 144 | 81.2% | 100.0% |
+| **5-gram** | Subword | 8,815 | 13.11 | 18,466 | 12.6% | 37.5% |
 
 ### Top 5 N-grams by Size
 
@@ -139,7 +151,7 @@ Below are sample sentences tokenized with each vocabulary size:
 | Rank | N-gram | Count |
 |------|--------|-------|
 | 1 | `tungan metelak` | 162 |
-| 2 | `iyo no` | 138 |
+| 2 | `iyo no` | 137 |
 | 3 | `iyo noh` | 69 |
 | 4 | `iyo tu` | 68 |
 | 5 | `bioso ni` | 45 |
@@ -164,42 +176,62 @@ Below are sample sentences tokenized with each vocabulary size:
 | 4 | `iyo no endangan jomo` | 12 |
 | 5 | `no endangan jomo politik` | 12 |
 
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `no endangan jomo politik ta` | 12 |
+| 2 | `iyo no endangan jomo politik` | 12 |
+| 3 | `beliau tu ahli dewan undangan` | 11 |
+| 4 | `malaysia beliau tu ahli dewan` | 11 |
+| 5 | `ahli dewan undangan negeri sabah` | 11 |
+
 **2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `a n` | 5,437 |
-| 2 | `n _` | 3,734 |
-| 3 | `n g` | 3,473 |
-| 4 | `i _` | 3,019 |
-| 5 | `_ t` | 2,998 |
+| 1 | `a n` | 5,403 |
+| 2 | `n _` | 3,715 |
+| 3 | `n g` | 3,458 |
+| 4 | `i _` | 3,000 |
+| 5 | `_ t` | 2,981 |
 
 **3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `a n _` | 2,443 |
-| 2 | `a n g` | 1,577 |
-| 3 | `n g _` | 1,357 |
-| 4 | `_ t a` | 1,076 |
-| 5 | `_ n i` | 987 |
+| 1 | `a n _` | 2,433 |
+| 2 | `a n g` | 1,567 |
+| 3 | `n g _` | 1,349 |
+| 4 | `_ t a` | 1,073 |
+| 5 | `_ n i` | 983 |
 
 **4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `a n g _` | 910 |
-| 2 | `_ n i _` | 650 |
-| 3 | `_ i y o` | 643 |
-| 4 | `n g a n` | 619 |
-| 5 | `g a n _` | 579 |
+| 1 | `a n g _` | 903 |
+| 2 | `_ n i _` | 648 |
+| 3 | `_ i y o` | 641 |
+| 4 | `n g a n` | 618 |
+| 5 | `g a n _` | 578 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `n g a n _` | 565 |
+| 2 | `_ i y o _` | 504 |
+| 3 | `y a n g _` | 410 |
+| 4 | `_ y a n g` | 370 |
+| 5 | `_ t a ' _` | 355 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram (subword) with 181
+- **Best Perplexity:** 5-gram (word) with 110
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~53% of corpus
+- **Coverage:** Top-1000 patterns cover ~37% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -215,14 +247,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
 |---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.8053 | 1.747 | 3.60 | 5,241 | 19.5% |
-| **1** | Subword | 1.4652 | 2.761 | 11.14 | 104 | 0.0% |
-| **2** | Word | 0.1666 | 1.122 | 1.26 | 18,592 | 83.3% |
-| **2** | Subword | 1.1951 | 2.290 | 5.74 | 1,154 | 0.0% |
-| **3** | Word | 0.0377 | 1.026 | 1.05 | 22,996 | 96.2% |
-| **3** | Subword | 0.7985 | 1.739 | 3.15 | 6,603 | 20.2% |
-| **4** | Word | 0.0104 🏆 | 1.007 | 1.01 | 23,590 | 99.0% |
-| **4** | Subword | 0.5443 | 1.458 | 2.09 | 20,699 | 45.6% |
+| **1** | Word | 0.8062 | 1.749 | 3.61 | 5,206 | 19.4% |
+| **1** | Subword | 1.4517 | 2.735 | 11.24 | 101 | 0.0% |
+| **2** | Word | 0.1664 | 1.122 | 1.26 | 18,482 | 83.4% |
+| **2** | Subword | 1.2091 | 2.312 | 5.80 | 1,130 | 0.0% |
+| **3** | Word | 0.0377 | 1.026 | 1.05 | 22,853 | 96.2% |
+| **3** | Subword | 0.8020 | 1.744 | 3.16 | 6,542 | 19.8% |
+| **4** | Word | 0.0104 🏆 | 1.007 | 1.01 | 23,441 | 99.0% |
+| **4** | Subword | 0.5453 | 1.459 | 2.09 | 20,556 | 45.5% |
 
 ### Generated Text Samples (Word-based)
 
@@ -230,27 +262,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `ni ta lok kuah engko tangsi selegubdi tu terhasil moko dangan pelego dendo malaysia beliau tu`
-2. `tu tungan ni un duo ni mediam tepung buas tak sekul tena tana amun pinapi enggo`
-3. `iyo boi nilego oleg ni jomo yang bok ni pan akan buan raya kota belud tu`
+1. `ni tak sekolah kebangsanaan puteri ngerujuk ta dikau bumbung laat atau pan buli kinurban iyo no`
+2. `tu pan kuleh status teralap malaysia diom arena seni soro tungan metelak politik malaysia iko pinaka...`
+3. `iyo menjadi budaya bajau sama ngeruo elau ule bedagang tradisi boi penenakan tak taun iyo pan`
 
 **Context Size 2:**
 
-1. `iyo no nyaun preskripsi toos bineli ta farmasi atau mediam kadai yang nyaun sebarang halangan engko ...`
-2. `iyo noh kui tradisional jomo mitu sabah kui tu bentuk ni dokon indung jari engko binuat lua`
-3. `iyo tu boi ni urus le ni gua a masi un sampai betiru terutama ni sembiang pardu`
+1. `iyo no dangan jomo mediom menjogo keselamatan ko kestabilan masyarakat nuut ta diom undang undang lu...`
+2. `iyo noh tun dr hasmah binti haji mohamad ali nganak 12 julai hasmah iyono doktor dendo yang`
+3. `iyo tu pelego pemuzik kok pelakun dendo malaysia iyo tekilo kok watak ni lua kawasan asahan sumatera`
 
 **Context Size 3:**
 
-1. `ma na ni teko ta tampat tungan setemu tapi jomo tenemuan ai no lumaan`
-2. `dewan undangan negeri sabah ta kewasan tempasuk lua tungan metelak politik malaysia di pertua laat a...`
-3. `undangan negeri sabah betiru`
+1. `ma na ni dediki bana`
+2. `dewan undangan negeri sabah dewan undangan negeri sabah betiru`
+3. `undangan negeri sabah boi nilantik lua 8 oktober beliau betiru ngentan jewatan ketua parti islam se ...`
 
 **Context Size 4:**
 
-1. `dewan undangan negeri sabah dun lua september tu anggota pertubuhan kebangsaan melayu bersatu malays...`
-2. `sama ma na ni ai ngemban matai`
-3. `ahli dewan undangan negeri sabah dewan undangan negeri sabah dewan undangan negeri sabah ta kewasan ...`
+1. `dewan undangan negeri sabah betiru`
+2. `sama ma na ni telampau oyo antawa oyo bana`
+3. `ahli dewan undangan negeri sabah dewan undangan negeri sabah ta kewasan matunggong lua tungan metela...`
 
 
 ### Generated Text Samples (Subword-based)
@@ -259,34 +291,34 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_njano-bo_cseria`
-2. `anegal_8_t_bu"_b`
-3. `ngim_nd_bo_isaup`
+1. `_amagri_dintutha`
+2. `a'_tandartino_il`
+3. `ntan_bik_di_a_bi`
 
 **Context Size 2:**
 
-1. `an_kain_tamuḥamal`
-2. `n_jom_no_turi_mud`
-3. `ngko_ta_tang_boi_`
+1. `angerangerebini_p`
+2. `n_tau_us_amungine`
+3. `ng._tan_fa_langha`
 
 **Context Size 3:**
 
-1. `an_ni_ana'_nakasal`
-2. `ang_jomo_untuan_ta`
-3. `ng_teali_pulo_ko'_`
+1. `an_ole_ta'_mapas_d`
+2. `ang_ma'na_tang_di_`
+3. `ng_un_pan_atley_ma`
 
 **Context Size 4:**
 
-1. `ang_sefalopod_lua'_`
-2. `_ni_denga_septembag`
-3. `_iyo_no_telia_punya`
+1. `ang_semek_regisin,_`
+2. `_ni_untuk_pelbagas_`
+3. `_iyo_no_un_pakai_bi`
 
 
 ### Key Findings
 
 - **Best Predictability:** Context-4 (word) with 99.0% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (20,699 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (20,556 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -302,25 +334,25 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 2,342 |
-| Total Tokens | 23,366 |
-| Mean Frequency | 9.98 |
+| Vocabulary Size | 2,333 |
+| Total Tokens | 23,243 |
+| Mean Frequency | 9.96 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 33.27 |
+| Frequency Std Dev | 33.24 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ni | 760 |
-| 2 | tu | 584 |
-| 3 | iyo | 549 |
-| 4 | ta | 455 |
-| 5 | yang | 382 |
-| 6 | boi | 354 |
+| 1 | ni | 758 |
+| 2 | tu | 583 |
+| 3 | iyo | 548 |
+| 4 | ta | 452 |
+| 5 | yang | 381 |
+| 6 | boi | 353 |
 | 7 | pan | 303 |
 | 8 | kok | 280 |
-| 9 | jomo | 275 |
+| 9 | jomo | 273 |
 | 10 | tungan | 250 |
 
 ### Least Common Words (from vocabulary)
@@ -342,24 +374,24 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9532 |
-| R² (Goodness of Fit) | 0.984280 |
+| Zipf Coefficient | 0.9534 |
+| R² (Goodness of Fit) | 0.984288 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 45.5% |
-| Top 1,000 | 85.6% |
+| Top 100 | 45.6% |
+| Top 1,000 | 85.7% |
 | Top 5,000 | 0.0% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
 - **Zipf Compliance:** R²=0.9843 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 45.5% of corpus
-- **Long Tail:** -7,658 words needed for remaining 100.0% coverage
+- **High Frequency Dominance:** Top 100 words cover 45.6% of corpus
+- **Long Tail:** -7,667 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -375,37 +407,40 @@ Below are text samples generated from each subword-based Markov chain model:
 
 ### 5.1 Cross-Lingual Alignment
 
-> *Note: Multilingual alignment visualization not available for this language.*
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
 
 
 ### 5.2 Model Comparison
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.0482 🏆 | 0.8825 | N/A | N/A |
-| **mono_64d** | 64 | 0.0132 | 0.9050 | N/A | N/A |
-| **mono_128d** | 128 | 0.0053 | 0.9273 | N/A | N/A |
+| **mono_32d** | 32 | 0.0390 🏆 | 0.8823 | N/A | N/A |
+| **mono_64d** | 64 | 0.0242 | 0.9395 | N/A | N/A |
+| **mono_128d** | 128 | 0.0071 | 0.9393 | N/A | N/A |
+| **aligned_32d** | 32 | 0.0390 | 0.8940 | 0.0078 | 0.0667 |
+| **aligned_64d** | 64 | 0.0242 | 0.9410 | 0.0039 | 0.0627 |
+| **aligned_128d** | 128 | 0.0071 | 0.9401 | 0.0039 | 0.0627 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.0482 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.9049. Lower values indicate better semantic separation.
-- **Alignment Quality:** No aligned models evaluated in this run.
+- **Best Isotropy:** mono_32d with 0.0390 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.9227. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 0.8% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
 ## 6.  Morphological Analysis (Experimental)
 
-> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
-
 This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
 
 ### 6.1 Productivity & Complexity
 
 | Metric | Value | Interpretation | Recommendation |
 |--------|-------|----------------|----------------|
-| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
-| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+| Productivity Index | **2.767** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **1.669** | High formulaic/idiomatic content | - |
 
 ### 6.2 Affix Inventory (Productive Units)
 
@@ -414,21 +449,21 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Prefixes
 | Prefix | Examples |
 |--------|----------|
-| `-pe` | petaling, pekakas, peketa |
-| `-se` | sejak, seniram, sejati |
-| `-ke` | kenangan, kerita, keratas |
-| `-te` | tehe, tempoh, tetiak |
-| `-me` | meruma, menurut, melioro |
-| `-be` | berukuran, berfikir, benua |
+| `-pe` | petaling, pekakasan, pesat |
+| `-se` | sepanjang, serupo, seri |
+| `-ke` | keempat, kempen, kemudahan |
+| `-te` | tena, teposok, terbaik |
+| `-me` | melodi, mencakup, menduo |
+| `-be` | bege, begiang, been |
 
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-n` | kumpulan, regisin, haiwan |
-| `-an` | kumpulan, haiwan, berukuran |
-| `-ng` | petaling, kantung, ngulang |
-| `-ang` | ngulang, manang, sayang |
-| `-ah` | tah, fatimah, umrah |
+| `-n` | sangkan, intan, kempen |
+| `-an` | sangkan, intan, pekakasan |
+| `-ng` | suang, sepanjang, petaling |
+| `-ang` | suang, sepanjang, begiang |
+| `-ah` | tanah, buah, majalah |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -443,16 +478,16 @@ This table shows which prefixes and suffixes most frequently co-occur on the sam
 
 | Prefix | Suffix | Frequency | Examples |
 |--------|--------|-----------|----------|
-| `-pe` | `-n` | 55 words | pelan, pentaran |
-| `-pe` | `-an` | 49 words | pelan, pentaran |
-| `-ke` | `-n` | 42 words | kenangan, keteraan |
-| `-ke` | `-an` | 36 words | kenangan, keteraan |
-| `-se` | `-n` | 11 words | sebahagian, selain |
-| `-te` | `-n` | 9 words | temban, tenomon |
-| `-se` | `-ng` | 9 words | sedong, sepanjang |
-| `-me` | `-n` | 9 words | mesimpon, meluman |
-| `-pe` | `-ng` | 7 words | petaling, pelancong |
-| `-be` | `-n` | 7 words | berukuran, been |
+| `-pe` | `-n` | 55 words | pekakasan, pernikahan |
+| `-pe` | `-an` | 49 words | pekakasan, pernikahan |
+| `-ke` | `-n` | 42 words | kempen, kemudahan |
+| `-ke` | `-an` | 36 words | kemudahan, keadilan |
+| `-se` | `-n` | 11 words | semimon, sembilan |
+| `-se` | `-ng` | 9 words | sepanjang, sesambung |
+| `-te` | `-n` | 9 words | temban, teniman |
+| `-me` | `-n` | 9 words | mekitoon, mesakan |
+| `-pe` | `-ng` | 7 words | petaling, perang |
+| `-se` | `-an` | 7 words | sembilan, sebahagian |
 
 ### 6.5 Recursive Morpheme Segmentation
 
@@ -461,25 +496,27 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 | Word | Suggested Split | Confidence | Stem |
 |------|-----------------|------------|------|
 | kebenyakan | **`ke-be-nyak-an`** | 7.5 | `nyak` |
-| kebangsaan | **`ke-bangsa-an`** | 6.0 | `bangsa` |
-| kelebihan | **`ke-lebih-an`** | 6.0 | `lebih` |
-| kelahiran | **`ke-lahir-an`** | 6.0 | `lahir` |
+| kebudayaan | **`ke-budaya-an`** | 6.0 | `budaya` |
 | keramaian | **`ke-ramai-an`** | 6.0 | `ramai` |
 | kepulauan | **`ke-pulau-an`** | 6.0 | `pulau` |
-| kebudayaan | **`ke-budaya-an`** | 6.0 | `budaya` |
 | keputeraan | **`ke-putera-an`** | 6.0 | `putera` |
-| sedembila | **`se-dembila`** | 4.5 | `dembila` |
-| perpisahan | **`pe-rpis-ah-an`** | 4.5 | `rpis` |
-| keselamatan | **`ke-se-lamat-an`** | 4.5 | `lamat` |
+| kelahiran | **`ke-lahir-an`** | 6.0 | `lahir` |
+| kelebihan | **`ke-lebih-an`** | 6.0 | `lebih` |
+| kebangsaan | **`ke-bangsa-an`** | 6.0 | `bangsa` |
 | pernikahan | **`pe-rnik-ah-an`** | 4.5 | `rnik` |
-| perjuangan | **`pe-rjua-ng-an`** | 4.5 | `rjua` |
+| pertandingan | **`pe-rtandi-ng-an`** | 4.5 | `rtandi` |
+| pelancongan | **`pe-lanco-ng-an`** | 4.5 | `lanco` |
+| persembahan | **`pe-rsemb-ah-an`** | 4.5 | `rsemb` |
 | kemerdekaan | **`ke-me-rdeka-an`** | 4.5 | `rdeka` |
-| kepelbagaian | **`ke-pe-lbagai-an`** | 4.5 | `lbagai` |
+| keselamatan | **`ke-se-lamat-an`** | 4.5 | `lamat` |
+| sedembila | **`se-dembila`** | 4.5 | `dembila` |
 
 ### 6.6 Linguistic Interpretation
 
 > **Automated Insight:**
-The language BDR appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+The language West Coast Bajau shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
 
 ---
 ## 7. Summary & Recommendations
@@ -490,8 +527,8 @@ The language BDR appears to be more isolating or has a highly fixed vocabulary.
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **8k BPE** | Best compression (4.79x) |
-| N-gram | **2-gram** | Lowest perplexity (181) |
+| Tokenizer | **8k BPE** | Best compression (4.80x) |
+| N-gram | **5-gram** | Lowest perplexity (110) |
 | Markov | **Context-4** | Highest predictability (99.0%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
@@ -706,4 +743,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 06:44:23*
+*Report Date: 2026-01-03 18:34:47*

models/embeddings/aligned/bdr_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e41667680a8bc0cadec67edf4743451f35cd7995f62085ad35d96463f5772de
+size 1024832531

models/embeddings/aligned/bdr_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "bdr", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/bdr_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7a12fadca82ce6dfea45d130790fcac9cd31a5b472f332630040e1cfe4a2787f
+size 65664

models/embeddings/aligned/bdr_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "bdr",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 255,
+  "vocab_size": 801
+}

models/embeddings/aligned/bdr_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ae1bb7c338b11bc2a28dfc0e13abbbd2a39098370cf28ad78b0514a6f5863aa
+size 256217363

models/embeddings/aligned/bdr_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "bdr", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/bdr_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aad3c75607eb6c5b251c400c7ff7627a3c005d45d6cb24c0abcd80758e3be665
+size 4224

models/embeddings/aligned/bdr_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "bdr",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 255,
+  "vocab_size": 801
+}

models/embeddings/aligned/bdr_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37213f3cd2a0a803f1b3c410bc6e46b64095e2d36a2dc927c279189c10fdde99
+size 512422419

models/embeddings/aligned/bdr_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "bdr", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/bdr_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ff9f1c468e4c40cecffd704552c1ed2039f271a861bc14cd8ba44252ed0c8dd
+size 16512

models/embeddings/aligned/bdr_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "bdr",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 255,
+  "vocab_size": 801
+}

models/embeddings/monolingual/bdr_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:14901cc49f6d41fceff30e160e6f74a2d3e4c31128383c0eb0c581df8f55667d
-size 1024837728
+oid sha256:0e41667680a8bc0cadec67edf4743451f35cd7995f62085ad35d96463f5772de
+size 1024832531

models/embeddings/monolingual/bdr_128d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 128
   },
-  "vocab_size": 806
+  "vocab_size": 801
 }

models/embeddings/monolingual/bdr_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fa045fafcb5ac078ce6dc442fa1420ea920b282300bd6a2013469a40763dfb5e
-size 256218720
+oid sha256:9ae1bb7c338b11bc2a28dfc0e13abbbd2a39098370cf28ad78b0514a6f5863aa
+size 256217363

models/embeddings/monolingual/bdr_32d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 32
   },
-  "vocab_size": 806
+  "vocab_size": 801
 }

models/embeddings/monolingual/bdr_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0f57494e6d97c810c174b928ec3c054ba096b896edd93f5bb2951c51ea1469fb
-size 512425056
+oid sha256:37213f3cd2a0a803f1b3c410bc6e46b64095e2d36a2dc927c279189c10fdde99
+size 512422419

models/embeddings/monolingual/bdr_64d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 64
   },
-  "vocab_size": 806
+  "vocab_size": 801
 }

models/subword_markov/bdr_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:90dd99baf68b20403d569b8e5f8d37366e4ddf2960b74590e45c40e1931d3cf1
-size 12465
+oid sha256:9c01103c97f395e9b1a6df414c90c77b0441b957516d54a843bb55d5ae0454da
+size 12256

models/subword_markov/bdr_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "bdr",
-  "unique_contexts": 104,
-  "total_transitions": 158102
+  "unique_contexts": 101,
+  "total_transitions": 157193
 }

models/subword_markov/bdr_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:de0458067f06b19cd92069895f072cb47f6cdac7b1aec47edc3f54439b9dc8fb
-size 51058
+oid sha256:4a1eaf53753bc5cd77e26c762c70b525953aea8bbbdaba88d81f8b89f3cad7d8
+size 50308

models/subword_markov/bdr_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "bdr",
-  "unique_contexts": 1154,
-  "total_transitions": 157564
+  "unique_contexts": 1130,
+  "total_transitions": 156655
 }

models/subword_markov/bdr_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8a603938c76585d0058fba20159b63f48cc8de1a081d60c9e812d5ef4cc856c2
-size 147977
+oid sha256:125c046ee06d3fd97f3d071738c1eee6377f5c7c4013bd62eef4fb484d2c9ccc
+size 146590

models/subword_markov/bdr_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "bdr",
-  "unique_contexts": 6603,
-  "total_transitions": 157026
+  "unique_contexts": 6542,
+  "total_transitions": 156117
 }

models/subword_markov/bdr_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:40b65aa668a1cff884dd7d1981f09619017109f7d40edc49b7916cb9d523100b
-size 355053
+oid sha256:b1cfe96d25d60cd6b2f2a9c4d6224d5dc2d7e34f5199adc174e0e948de40a415
+size 353233

models/subword_markov/bdr_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "bdr",
-  "unique_contexts": 20699,
-  "total_transitions": 156488
+  "unique_contexts": 20556,
+  "total_transitions": 155579
 }

models/subword_ngram/bdr_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f89d4cd1bab1bda41e8fb6e1bcbdcfb402307503bc759166623f23f6684cb7d8
-size 9761
+oid sha256:81ad86f878dbd31142b31c0bceefa29fec8ddff13564e7a1537d9e13f941884e
+size 9737

models/subword_ngram/bdr_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "bdr",
-  "unique_ngrams": 597,
-  "total_ngrams": 158102
+  "unique_ngrams": 593,
+  "total_ngrams": 157193
 }

models/subword_ngram/bdr_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:82c353ab6740086eface298a08842c1494816358218a9e1924c16d5cc9298c3d
-size 37547
+oid sha256:5536c1c7357dd2f396f0e7af3a16bbf82d7e0abefe4045ab9afe2432147f21ae
+size 37388

models/subword_ngram/bdr_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "bdr",
-  "unique_ngrams": 3421,
-  "total_ngrams": 157564
+  "unique_ngrams": 3407,
+  "total_ngrams": 156655
 }

models/subword_ngram/bdr_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:00c9461926041d02b2401848dda26b82b5b87974371b657a4eeb9a268d495f8c
-size 128743
+oid sha256:805cf63298b535adbcf60a5db766aade000901bf269c45ce290338156be8f197
+size 129044

models/subword_ngram/bdr_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "bdr",
-  "unique_ngrams": 11413,
-  "total_ngrams": 157026
+  "unique_ngrams": 11348,
+  "total_ngrams": 156117
 }

models/subword_ngram/bdr_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f4e8e2a8a2fd55877f905703b6d37187254f576327870646a2a80da1780f039
+size 224851

models/subword_ngram/bdr_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "bdr",
+  "unique_ngrams": 18466,
+  "total_ngrams": 155579
+}

models/tokenizer/bdr_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:60470ba72369b5d6feb2a6ddd8122b4eb757065597901f8d33f1f47db9d61152
-size 373260
+oid sha256:d3257481d465f4f6cee30948577989924f9bf988a0865d4b1d66ac5beb8c0a6c
+size 373305

models/vocabulary/bdr_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a58092d4262f6aa43f6194c3af31f93b2c60e2e1f1457610992d6848f4156246
-size 38048
+oid sha256:4b427705d359f9ebf3fa546a3c56857b1b319cd6cc0cfdb8bec8e9e95e7f20b9
+size 37799

models/vocabulary/bdr_vocabulary_metadata.json

@@ -1,16 +1,16 @@
 {
   "language": "bdr",
-  "vocabulary_size": 2342,
+  "vocabulary_size": 2333,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.20221606648199447,
+    "type_token_ratio": 0.2020302243932224,
     "coverage": {
-      "top_100": 0.4034075816795052,
-      "top_1000": 0.7593442871779305,
-      "top_5000": 0.9875156528668463
+      "top_100": 0.4047855289268814,
+      "top_1000": 0.760418256754694,
+      "top_5000": 0.9887803388795604
     },
-    "hapax_count": 2987,
-    "hapax_ratio": 0.5605179208106587,
+    "hapax_count": 2961,
+    "hapax_ratio": 0.5593124291650926,
     "total_documents": 538
   }
 }

models/word_markov/bdr_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:50da15bb2d596e55b1af74da418be234ae20d7994f359665a73457c179ace2c4
-size 149999
+oid sha256:93344d32ca7f52ac3cf06047cffe71756e8f7a067baa2b2d0e581bff40c4ad9b
+size 149528

models/word_markov/bdr_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "bdr",
-  "unique_contexts": 5241,
-  "total_transitions": 25815
+  "unique_contexts": 5206,
+  "total_transitions": 25666
 }

models/word_markov/bdr_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a70c0745d64f93f951581ccf1e770bc0198416279a90b1c76af931bc397bb65f
-size 308032
+oid sha256:b2ff731f7444897501c9da118612008f21992d71fa9e8d4c5c82bd2db7ebb4c6
+size 306258

models/word_markov/bdr_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "bdr",
-  "unique_contexts": 18592,
-  "total_transitions": 25277
+  "unique_contexts": 18482,
+  "total_transitions": 25128
 }

models/word_markov/bdr_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:223447078bb309ba5d758145f41c4571806689ffaf1efe8aef015c68c49c7526
-size 388681
+oid sha256:b86db7e1621c9ba0690146e1820f26862e9beaaa79b8cfbf63842988dbb2e4e7
+size 386401

models/word_markov/bdr_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "bdr",
-  "unique_contexts": 22996,
-  "total_transitions": 24739
+  "unique_contexts": 22853,
+  "total_transitions": 24590
 }

models/word_markov/bdr_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:66c6e307074a65e79883d656ec156325dab1be18c7c2082eab0c9665ad6e3133
-size 429090
+oid sha256:dff2b6fb681e66f38ede9f1c2614a2c163ec702f2199b843e0e84e335a3600ce
+size 426318

models/word_markov/bdr_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "bdr",
-  "unique_contexts": 23590,
-  "total_transitions": 24201
+  "unique_contexts": 23441,
+  "total_transitions": 24052
 }

models/word_ngram/bdr_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3f7dbbf4075f6494f4e948bc0417ec1b327b33cb3676c74c44c5cf7b15c76e93
-size 7711
+oid sha256:dc048e89e93b12320521aa74196bfb21637e0a035925b52b7ef01bae1000a530
+size 7714

models/word_ngram/bdr_2gram_word_metadata.json

@@ -3,5 +3,5 @@
   "variant": "word",
   "language": "bdr",
   "unique_ngrams": 401,
-  "total_ngrams": 25815
+  "total_ngrams": 25666
 }