Upload all models and assets for br (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 3.617
+    value: 3.786
   - name: best_isotropy
     type: isotropy
-    value: 0.8322
+    value: 0.8171
   - name: vocabulary_size
     type: vocab
-    value: 263085
-generated: 2025-12-28
+    value: 0
+generated: 2026-01-03
 ---
 
 # BR - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,55 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.122x | 3.07 | 0.4020% | 881,322 |
-| **16k** | 3.329x | 3.28 | 0.4286% | 826,606 |
-| **32k** | 3.492x | 3.44 | 0.4496% | 788,002 |
-| **64k** | 3.617x 🏆 | 3.56 | 0.4657% | 760,713 |
+| **8k** | 3.235x | 3.24 | 0.4490% | 793,680 |
+| **16k** | 3.460x | 3.46 | 0.4803% | 742,059 |
+| **32k** | 3.645x | 3.65 | 0.5060% | 704,331 |
+| **64k** | 3.786x 🏆 | 3.79 | 0.5255% | 678,179 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Fuentes de Jiloca zo ur gumun eus Spagn e Proviñs Zaragoza, en Aragon.`
+**Sample 1:** `Monsano zo ur gumun italian e proviñs Ancona, er Marche. Marche Proviñs Ancona`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁fu ent es ▁de ▁j il oc a ▁zo ▁ur ... (+12 more)` | 22 |
-| 16k | `▁fu entes ▁de ▁j il oca ▁zo ▁ur ▁gumun ▁eus ... (+8 more)` | 18 |
-| 32k | `▁fuentes ▁de ▁jil oca ▁zo ▁ur ▁gumun ▁eus ▁spagn ▁e ... (+6 more)` | 16 |
-| 64k | `▁fuentes ▁de ▁jil oca ▁zo ▁ur ▁gumun ▁eus ▁spagn ▁e ... (+6 more)` | 16 |
+| 8k | `▁mon s ano ▁zo ▁ur ▁gumun ▁italian ▁e ▁proviñs ▁anc ... (+9 more)` | 19 |
+| 16k | `▁mons ano ▁zo ▁ur ▁gumun ▁italian ▁e ▁proviñs ▁ancona , ... (+6 more)` | 16 |
+| 32k | `▁mons ano ▁zo ▁ur ▁gumun ▁italian ▁e ▁proviñs ▁ancona , ... (+6 more)` | 16 |
+| 64k | `▁mons ano ▁zo ▁ur ▁gumun ▁italian ▁e ▁proviñs ▁ancona , ... (+6 more)` | 16 |
 
-**Sample 2:** `Barromán zo ur gumun eus Spagn, e proviñs Ávila, en Kastilha ha León. 
-
-Rummad:K...`
+**Sample 2:** `San Asensio zo ur gumun e proviñs La Rioja en Spagn. Rioja`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁bar rom án ▁zo ▁ur ▁gumun ▁eus ▁spagn , ▁e ... (+14 more)` | 24 |
-| 16k | `▁bar rom án ▁zo ▁ur ▁gumun ▁eus ▁spagn , ▁e ... (+14 more)` | 24 |
-| 32k | `▁bar rom án ▁zo ▁ur ▁gumun ▁eus ▁spagn , ▁e ... (+14 more)` | 24 |
-| 64k | `▁bar rom án ▁zo ▁ur ▁gumun ▁eus ▁spagn , ▁e ... (+14 more)` | 24 |
-
-**Sample 3:** `Tolbaños zo ur gumun eus Spagn, e proviñs Ávila, en Kastilha ha León. 
+| 8k | `▁san ▁as ens io ▁zo ▁ur ▁gumun ▁e ▁proviñs ▁la ... (+5 more)` | 15 |
+| 16k | `▁san ▁as ens io ▁zo ▁ur ▁gumun ▁e ▁proviñs ▁la ... (+5 more)` | 15 |
+| 32k | `▁san ▁as ens io ▁zo ▁ur ▁gumun ▁e ▁proviñs ▁la ... (+5 more)` | 15 |
+| 64k | `▁san ▁as ens io ▁zo ▁ur ▁gumun ▁e ▁proviñs ▁la ... (+5 more)` | 15 |
 
-Rummad:K...`
+**Sample 3:** `Segusino zo ur gumun e proviñs Treviso e Veneto, en Italia.`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁tol b añ os ▁zo ▁ur ▁gumun ▁eus ▁spagn , ... (+15 more)` | 25 |
-| 16k | `▁tol b añ os ▁zo ▁ur ▁gumun ▁eus ▁spagn , ... (+15 more)` | 25 |
-| 32k | `▁tol b años ▁zo ▁ur ▁gumun ▁eus ▁spagn , ▁e ... (+14 more)` | 24 |
-| 64k | `▁tol b años ▁zo ▁ur ▁gumun ▁eus ▁spagn , ▁e ... (+14 more)` | 24 |
+| 8k | `▁seg us ino ▁zo ▁ur ▁gumun ▁e ▁proviñs ▁trev iso ... (+6 more)` | 16 |
+| 16k | `▁seg us ino ▁zo ▁ur ▁gumun ▁e ▁proviñs ▁treviso ▁e ... (+5 more)` | 15 |
+| 32k | `▁seg us ino ▁zo ▁ur ▁gumun ▁e ▁proviñs ▁treviso ▁e ... (+5 more)` | 15 |
+| 64k | `▁seg us ino ▁zo ▁ur ▁gumun ▁e ▁proviñs ▁treviso ▁e ... (+5 more)` | 15 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 3.617x compression
-- **Lowest UNK Rate:** 8k with 0.4020% unknown tokens
+- **Best Compression:** 64k achieves 3.786x compression
+- **Lowest UNK Rate:** 8k with 0.4490% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -125,57 +129,89 @@ Rummad:K...`
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 37,795 🏆 | 15.21 | 396,301 | 15.5% | 32.7% |
-| **2-gram** | 361 🏆 | 8.49 | 13,774 | 60.5% | 98.2% |
-| **3-gram** | 147,885 | 17.17 | 858,386 | 7.1% | 19.7% |
-| **3-gram** | 3,461 | 11.76 | 106,981 | 21.9% | 63.7% |
-| **4-gram** | 382,971 | 18.55 | 1,581,104 | 3.6% | 13.2% |
-| **4-gram** | 22,182 | 14.44 | 580,225 | 10.3% | 33.1% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 37,349 | 15.19 | 296,192 | 13.7% | 32.0% |
+| **2-gram** | Subword | 294 🏆 | 8.20 | 11,776 | 65.3% | 98.9% |
+| **3-gram** | Word | 128,487 | 16.97 | 570,380 | 5.9% | 19.5% |
+| **3-gram** | Subword | 2,726 | 11.41 | 81,142 | 23.8% | 68.1% |
+| **4-gram** | Word | 279,047 | 18.09 | 973,376 | 4.1% | 14.8% |
+| **4-gram** | Subword | 17,313 | 14.08 | 421,873 | 10.8% | 35.5% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `e voe` | 59,782 |
+| 2 | `ar c` | 55,139 |
+| 3 | `a viz` | 53,711 |
+| 4 | `e oa` | 52,022 |
+| 5 | `d ar` | 47,935 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `zo ur gumun` | 17,678 |
+| 2 | `bro c hall` | 15,638 |
+| 3 | `a zo ur` | 15,315 |
+| 4 | `e oa bet` | 12,845 |
+| 5 | `ur gumun eus` | 8,897 |
+
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `c '` | 168,356 |
-| 2 | `rummad :` | 166,928 |
-| 3 | `d '` | 113,851 |
-| 4 | `' h` | 99,333 |
-| 5 | `, e` | 74,128 |
+| 1 | `zo ur gumun eus` | 8,261 |
+| 2 | `monumantoù ha traoù heverk` | 5,435 |
+| 3 | `a zo ur gumun` | 5,065 |
+| 4 | `zo ur gumun e` | 4,314 |
+| 5 | `monumant ar re varv` | 3,991 |
 
-**3-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ar c '` | 51,117 |
-| 2 | `d ' ar` | 44,541 |
-| 3 | `d ' an` | 33,579 |
-| 4 | `. rummad :` | 24,562 |
-| 5 | `c ' hall` | 23,162 |
+| 1 | `_ a` | 1,901,092 |
+| 2 | `_ e` | 1,675,345 |
+| 3 | `a n` | 1,608,231 |
+| 4 | `e _` | 1,592,896 |
+| 5 | `r _` | 1,428,493 |
 
-**4-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `bro - c '` | 16,536 |
-| 2 | `- c ' hall` | 15,614 |
-| 3 | `zo ur gumun eus` | 8,365 |
-| 4 | `ha daveennoù rummad :` | 7,182 |
-| 5 | `notennoù ha daveennoù rummad` | 7,173 |
+| 1 | `a r _` | 640,562 |
+| 2 | `_ e _` | 639,818 |
+| 3 | `e t _` | 623,800 |
+| 4 | `_ a r` | 555,503 |
+| 5 | `e n n` | 467,995 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ a r _` | 456,425 |
+| 2 | `_ a n _` | 280,111 |
+| 3 | `a n t _` | 269,237 |
+| 4 | `_ g a n` | 228,714 |
+| 5 | `_ h a _` | 221,977 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 361
+- **Best Perplexity:** 2-gram (subword) with 294
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~33% of corpus
+- **Coverage:** Top-1000 patterns cover ~35% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -183,55 +219,86 @@ Rummad:K...`
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.7133 | 1.640 | 6.37 | 642,948 | 28.7% |
-| **1** | 1.1559 | 2.228 | 7.70 | 6,849 | 0.0% |
-| **2** | 0.3934 | 1.313 | 2.35 | 4,087,653 | 60.7% |
-| **2** | 0.7490 | 1.681 | 4.86 | 52,722 | 25.1% |
-| **3** | 0.1872 | 1.139 | 1.44 | 9,600,283 | 81.3% |
-| **3** | 0.7699 | 1.705 | 4.16 | 256,241 | 23.0% |
-| **4** | 0.0950 🏆 | 1.068 | 1.19 | 13,824,129 | 90.5% |
-| **4** | 0.6874 🏆 | 1.610 | 3.24 | 1,065,026 | 31.3% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.8900 | 1.853 | 7.60 | 546,189 | 11.0% |
+| **1** | Subword | 0.8953 | 1.860 | 5.85 | 8,402 | 10.5% |
+| **2** | Word | 0.3300 | 1.257 | 2.04 | 4,129,549 | 67.0% |
+| **2** | Subword | 0.6669 | 1.588 | 4.21 | 49,100 | 33.3% |
+| **3** | Word | 0.1561 | 1.114 | 1.34 | 8,377,190 | 84.4% |
+| **3** | Subword | 0.6656 | 1.586 | 3.74 | 206,555 | 33.4% |
+| **4** | Word | 0.0728 🏆 | 1.052 | 1.13 | 11,216,136 | 92.7% |
+| **4** | Subword | 0.6497 | 1.569 | 3.22 | 772,246 | 35.0% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `e breizh e vennozhioù diskouez an 18 muhelder bihanañ 34 evel caravan palace da vezañ ivez`
+2. `ar zastava e amzer tredeoged betek da vare a eskemm ha pterosaurus petra a ra an`
+3. `a oa un heñvelster gant dean bounce prison gang crime deep grand prix de france bleu`
+
+**Context Size 2:**
+
+1. `e voe embannet an testennoù klasel e vez da 800 000 den pe gant kraf ar revelezh`
+2. `ar c hembraeg mawr bras tolkien avat en doa bet ur gwadliñvadur e barzh ar c haramel`
+3. `a viz eost a oa ul livour hag un impalaer e voe kumun kernitron al lann e`
+
+**Context Size 3:**
+
+1. `zo ur gumun en italia e proviñs cuneo etre kumunioù entracque ha valdieri anezhañ unan eus ugent lev...`
+2. `a zo ur bronneg geotdebrer a vev er meurvor atlantel belle isle distaget bɛl ˈaɪl e saozneg zo`
+3. `bro c hall zo un tiern ag ar morioù a zo e kenver ar sonerezh met evel ul`
 
-### Generated Text Samples
+**Context Size 4:**
+
+1. `zo ur gumun eus meurgêr palermo e sikilia un 3 400 a dud zo enni o chom anezhi an`
+2. `monumantoù ha traoù heverk iliz katolik saint eustacheclochers de france douaroniezh emdroadur ar bo...`
+3. `a zo ur gumun eus italia e proviñs piacenza e rannvro emilia romagna ha 525 940 a dud o`
+
+
+### Generated Text Samples (Subword-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `, job de reims . emañ o embann levrioù gant ur yezh romanek . ramsay ha`
-2. `- kreiz kalotenn skorn . sonet gantañ cheñch anv ar - se ne em gavas youenn`
-3. `. notennoù rummad : nobiliaire et tableaux d ' orgeraie ; barnet rust e - labour`
+1. `_t_onur_b_el_gan`
+2. `er_pakoumiat_g_v`
+3. `adoù-stiz_aleuri`
 
 **Context Size 2:**
 
-1. `c ' hommonwealth e beredoù ar brezel en ur bolz - enor e 1785 . darn all`
-2. `rummad : geomorfologiezh rummad : pladennoù brezhonek rummad : ganedigezhioù 1958 rummad : ganedigez...`
-3. `d ' ar gonfusianegezh lakaet e voe seitek nijour , ne c ' hoar vras paula (`
+1. `_a_un_erl_ezenner`
+2. `_e_he_c'hhn_/_niz`
+3. `an_ero_liged_;_ev`
 
 **Context Size 3:**
 
-1. `ar c ' hentañ derez ) ofis publik ar brezhoneg . in : studia celtica18 / 19 :`
-2. `d ' ar 26 a viz eost . an anv implij al lerc ' h pa ' z`
-3. `d ' an arabegerion eta , evit skrivañ ar sañskriteg e vez implijet ar sistem - mañ gant`
+1. `ar_bretek_ilioù,_o`
+2. `_e_pyrrarkva,_leon`
+3. `et_gant_franne,_ga`
 
 **Context Size 4:**
 
-1. `bro - c ' hall ) , e - lec ' h zo anvet pentre e kembre . hen`
-2. `- c ' hall ) d ' an 3 a viz genver 1871 . krouet e oa bet gant`
-3. `zo ur gumun eus italia , e proviñs cremona , e rannvro lombardia . rummad : kumunioù lombardia rumma...`
+1. `_ar_spesadoù_war_ar`
+2. `_an_emsavid_fy_nhad`
+3. `ant_an_alamanentiad`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 90.5% predictability
+- **Best Predictability:** Context-4 (word) with 92.7% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (1,065,026 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (772,246 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -247,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 263,085 |
-| Total Tokens | 16,823,868 |
-| Mean Frequency | 63.95 |
+| Vocabulary Size | 242,115 |
+| Total Tokens | 15,327,088 |
+| Mean Frequency | 63.30 |
 | Median Frequency | 4 |
-| Frequency Std Dev | 2476.07 |
+| Frequency Std Dev | 2500.68 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | e | 717,291 |
-| 2 | ar | 526,767 |
-| 3 | a | 475,234 |
-| 4 | an | 331,609 |
-| 5 | ha | 233,395 |
-| 6 | c | 194,241 |
-| 7 | gant | 192,785 |
-| 8 | en | 189,181 |
-| 9 | da | 173,430 |
-| 10 | rummad | 169,617 |
+| 1 | e | 701,948 |
+| 2 | ar | 517,584 |
+| 3 | a | 464,667 |
+| 4 | an | 326,300 |
+| 5 | ha | 228,454 |
+| 6 | gant | 189,759 |
+| 7 | c | 186,830 |
+| 8 | en | 181,309 |
+| 9 | da | 170,732 |
+| 10 | ur | 158,708 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | maghrebonkoud | 2 |
-| 2 | fidefide | 2 |
-| 3 | 2024he | 2 |
-| 4 | ougandachess | 2 |
-| 5 | ouganda365 | 2 |
-| 6 | inmediares | 2 |
-| 7 | cytonn | 2 |
-| 8 | malinga | 2 |
-| 9 | ablainville | 2 |
-| 10 | remonter | 2 |
+| 1 | nfpb | 2 |
+| 2 | konjic | 2 |
+| 3 | formoraich | 2 |
+| 4 | vsn | 2 |
+| 5 | moldavie | 2 |
+| 6 | yankovich | 2 |
+| 7 | gueydon | 2 |
+| 8 | tréhouart | 2 |
+| 9 | bouguen | 2 |
+| 10 | shimosa | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.1027 |
-| R² (Goodness of Fit) | 0.995216 |
+| Zipf Coefficient | 1.1106 |
+| R² (Goodness of Fit) | 0.996763 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 40.0% |
-| Top 1,000 | 64.0% |
-| Top 5,000 | 79.7% |
-| Top 10,000 | 85.1% |
+| Top 100 | 41.7% |
+| Top 1,000 | 65.8% |
+| Top 5,000 | 80.4% |
+| Top 10,000 | 85.6% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9952 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 40.0% of corpus
-- **Long Tail:** 253,085 words needed for remaining 14.9% coverage
+- **Zipf Compliance:** R²=0.9968 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 41.7% of corpus
+- **Long Tail:** 232,115 words needed for remaining 14.4% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -317,24 +384,114 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 165,065 | 32 | 3.529 | 1.159 | 0.8322 🏆 |
-| **mono_64d** | 165,065 | 64 | 4.045 | 1.139 | 0.8224 |
-| **mono_128d** | 165,065 | 128 | 4.668 | 1.127 | 0.8005 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8120 | 0.3605 | N/A | N/A |
+| **mono_64d** | 64 | 0.8171 🏆 | 0.2761 | N/A | N/A |
+| **mono_128d** | 128 | 0.7922 | 0.2119 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8322 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 165,065 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_64d with 0.8171 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2828. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-s` | mariånas, gilgamès, battalions |
+| `-er` | tufer, hutier, beaver |
+| `-où` | damkanadoù, barrennoù, heitioù |
+| `-es` | tarbes, marcondes, cordes |
+| `-us` | luchinus, menenius, fulmarus |
+| `-en` | weyden, minchen, wageningen |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `tion` | 2.51x | 77 contexts | tione, metion, aktion |
+| `emba` | 2.14x | 41 contexts | pemba, emban, bemba |
+| `adoù` | 1.80x | 74 contexts | dadoù, kadoù, zadoù |
+| `nnet` | 1.78x | 70 contexts | annet, bonnet, linnet |
+| `iamm` | 2.35x | 24 contexts | liamm, fiamma, fiamme |
+| `ouar` | 1.48x | 126 contexts | douar, zouar, mouar |
+| `nnad` | 1.52x | 97 contexts | bennad, rannad, hannad |
+| `zhio` | 1.87x | 40 contexts | uzhioù, lezhioù, bezhioù |
+| `zhañ` | 1.91x | 35 contexts | ezhañ, kozhañ, dizhañ |
+| `nnoù` | 1.84x | 39 contexts | tennoù, vannoù, bennoù |
+| `hone` | 1.81x | 40 contexts | honeg, khone, dhone |
+| `reze` | 1.46x | 94 contexts | breze, dreze, rezet |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+*No significant affix co-occurrences detected.*
+
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| eildelwennoù | **`eildelwenn-où`** | 4.5 | `eildelwenn` |
+| hejadennoù | **`hejadenn-où`** | 4.5 | `hejadenn` |
+| wissenschaften | **`wissenschaft-en`** | 4.5 | `wissenschaft` |
+| beauvaisen | **`beauvais-en`** | 4.5 | `beauvais` |
+| wellaennoù | **`wellaenn-où`** | 4.5 | `wellaenn` |
+| antoninus | **`antonin-us`** | 4.5 | `antonin` |
+| pluñvennoù | **`pluñvenn-où`** | 4.5 | `pluñvenn` |
+| kementadoù | **`kementad-où`** | 4.5 | `kementad` |
+| compositores | **`compositor-es`** | 4.5 | `compositor` |
+| garidelloù | **`garidell-où`** | 4.5 | `garidell` |
+| hromozomoù | **`hromozom-où`** | 4.5 | `hromozom` |
+| reolennoù | **`reolenn-où`** | 4.5 | `reolenn` |
+| barringer | **`barring-er`** | 4.5 | `barring` |
+| diamantes | **`diamant-es`** | 4.5 | `diamant` |
+| stradivarius | **`stradivari-us`** | 4.5 | `stradivari` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language BR appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -342,11 +499,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (3.62x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (361) |
-| Markov | **Context-4** | Highest predictability (90.5%) |
+| Tokenizer | **64k BPE** | Best compression (3.79x) |
+| N-gram | **2-gram** | Lowest perplexity (294) |
+| Markov | **Context-4** | Highest predictability (92.7%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -536,7 +694,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -552,7 +711,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-28 08:17:06*
+*Report Date: 2026-01-03 08:48:16*

models/embeddings/monolingual/br_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d3454154976810c137904a12ad207abedf1ec5887e8aca2f5a7122ad0f8d5930
-size 1195958360
+oid sha256:2cc2dcf0e5dbf7b0a47f5b7b84203934a4ef4a94c793c7cf148e47bcf6831e94
+size 1181713085

models/embeddings/monolingual/br_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 165065
+  "vocab_size": 151402
 }

models/embeddings/monolingual/br_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:64e33fc4be67c9fbddd42b666550e7082d827aabcc323efa5fa8f0d43bbce29b
-size 301188440
+oid sha256:563c972d1b01c4444517fc5d7b7951b8d9f5cf9c450bf8bfa161f4a7504bc953
+size 297436349

models/embeddings/monolingual/br_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 165065
+  "vocab_size": 151402
 }

models/embeddings/monolingual/br_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d9f750fa84e5a98167cab2dbbbb436fa61f2fe773cee5d410fe13aeafa89b674
-size 599445080
+oid sha256:64947ec6dfc340b15fe1245dfbf27eb885378a9c32be40f616fce71e20a67416
+size 592195261

models/embeddings/monolingual/br_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 165065
+  "vocab_size": 151402
 }

models/subword_markov/br_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7589832008594b2044eb3a660f9c0ee1c38b0a71dc2cf76a5e11e054aa0709c2
-size 359681
+oid sha256:364c87eeed84e49a322b272f9060cbaf46e7cf91d5fc9cf833ebcd76d814171e
+size 371383

models/subword_markov/br_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "br",
-  "unique_contexts": 6849,
-  "total_transitions": 100507380
+  "unique_contexts": 8402,
+  "total_transitions": 88562222
 }

models/subword_markov/br_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b80fd5521997f26184b281b5d6b8bbdead1d92da1152a2bc63ff0293af7aaeaa
-size 1941406
+oid sha256:3b2ffa4ddd6ad6edd05335dca480707c3b64c0a14c509081d1d74dfe72edc981
+size 1665166

models/subword_markov/br_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "br",
-  "unique_contexts": 52722,
-  "total_transitions": 100417214
+  "unique_contexts": 49100,
+  "total_transitions": 88473399
 }

models/subword_markov/br_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:909df4c05b66b69687d1d8f920db593ab05009de50d4a603df49a4228042cb35
-size 8952059
+oid sha256:692d3af21734dce1c8cf7fcea827b781d47ef5c8b25654782ff6145d753309c2
+size 6642409

models/subword_markov/br_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "br",
-  "unique_contexts": 256241,
-  "total_transitions": 100327048
+  "unique_contexts": 206555,
+  "total_transitions": 88384576
 }

models/subword_markov/br_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f2615bccc86f4696fdc966eac1f934f561672e8ed319db55e0b79b9044123108
-size 27438080
+oid sha256:02fe0d42cac9d5c8869d073c8570acc4ae0f693f23327c55de4b987d7ddb8e52
+size 20533909

models/subword_markov/br_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "br",
-  "unique_contexts": 1065026,
-  "total_transitions": 100236882
+  "unique_contexts": 772246,
+  "total_transitions": 88295753
 }

models/subword_ngram/br_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:97148ef1780fce9d7c84e24a99d7d18050738bc639866e47145c2650f4d24549
-size 180245
+oid sha256:7aa9a0309201a5bff85bb732039ec09cee34921527090479840454db58c6414f
+size 154793

models/subword_ngram/br_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "br",
-  "unique_ngrams": 13774,
-  "total_ngrams": 100507380
+  "unique_ngrams": 11776,
+  "total_ngrams": 88562222
 }

models/subword_ngram/br_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5896217e8144736c8569249112eb653f0c7159070b9cdcc4fab60365ffc6f9ea
-size 1306949
+oid sha256:3f11838b2e1dd3db7e5060c435f0ed9b2d633720992bd49d8414dc3ebb1947f1
+size 1025779

models/subword_ngram/br_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "br",
-  "unique_ngrams": 106981,
-  "total_ngrams": 100417214
+  "unique_ngrams": 81142,
+  "total_ngrams": 88473399
 }

models/subword_ngram/br_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:018393f222505ae1b2a7dc78a37fb2b916100fe4fd5ae7b93b4852d77884fc8d
-size 6695773
+oid sha256:c0662a30ed0f866ba0ef46fba70681b5e219c20b4d5f818869aeaf346ad95cea
+size 4921369

models/subword_ngram/br_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "br",
-  "unique_ngrams": 580225,
-  "total_ngrams": 100327048
+  "unique_ngrams": 421873,
+  "total_ngrams": 88384576
 }

models/tokenizer/br_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a752f92ace2ce968d9f9080803babfaec23f0f941b90aeaef31fda3dfce785ad
-size 501248
+oid sha256:3a34b2820290797923a2b18aa49d5154f1b7c7d12b7564cace04d896d97c6e53
+size 502646

models/tokenizer/br_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c1de8e69da6c92acfc056827b6f959e3a09c7f5e3833dbf06d735ef89d1a4415
-size 772253
+oid sha256:26ea0db3a85a9e9055b43984cf4ac687e795f5825553ccc302a5abc29d9b8273
+size 774769

models/tokenizer/br_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:952d3588c7c165d850f099355da1f2fb472e90d978c3fca8d4064f335f551dce
-size 1327800
+oid sha256:d3d3d03f79fa86777af32c2e613a308e2b7c627fbf49f5d7a3c97678ebafad13
+size 1335604

models/tokenizer/br_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ae0f58e053910435e1f0c3469037b7e403142c364ac16080087a71a6a1757cb9
-size 369734
+oid sha256:b6a1ce4f9267e8a3f8951e0d2dfd0751006c25871dfdcd890cceebf4f01248eb
+size 370491

models/vocabulary/br_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3e743e0213a6e24e64e5597ee9208bbcc913f68ffe7dfd9358fa210925a8f67c
-size 4050849
+oid sha256:5ace7ffc403a7ed53e5646074aa9d3b0acceee4c23eb067355b372c233b0488f
+size 3765893

models/vocabulary/br_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "br",
-  "vocabulary_size": 263085,
+  "vocabulary_size": 242115,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.03735837819296646,
+    "type_token_ratio": 0.03497837029849385,
     "coverage": {
-      "top_100": 0.3910641436030635,
-      "top_1000": 0.6260291451547847,
-      "top_5000": 0.779592055722224,
-      "top_10000": 0.8327078880623957
+      "top_100": 0.40905545676087623,
+      "top_1000": 0.6447280425360034,
+      "top_5000": 0.7885968080597011,
+      "top_10000": 0.8392496909814169
     },
-    "hapax_count": 379609,
-    "hapax_ratio": 0.5906527834397085,
-    "total_documents": 90166
+    "hapax_count": 304658,
+    "hapax_ratio": 0.5571928387100314,
+    "total_documents": 88823
   }
 }

models/word_markov/br_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:29d6439c7c56bab6e2e37e1f57fd518d0ba61a6913f7d0eb2a4fea4b55dbef56
-size 35882245
+oid sha256:be693644d5a0ebbb01f9e30236eea7c454a9d88268b56513e574925b7ff00051
+size 35952141

models/word_markov/br_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "br",
-  "unique_contexts": 642948,
-  "total_transitions": 22080722
+  "unique_contexts": 546189,
+  "total_transitions": 15542923
 }

models/word_markov/br_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ceff4f844e237ead4ea08a340d17ef9f3e7fe62d5bc6d7e8c3d3b31b59d2cf3d
-size 102417911
+oid sha256:a8fda1c756cb7fc2b4ab6e70c2cd1d67819d416d7e280f2e0548839fb1da8d2f
+size 97354526

models/word_markov/br_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "br",
-  "unique_contexts": 4087653,
-  "total_transitions": 21990556
+  "unique_contexts": 4129549,
+  "total_transitions": 15454100
 }

models/word_markov/br_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6a8a1170156b1dcf6b52e7667e73eb5c87efc4eed7afa9a113410a6fc6268928
-size 178712506
+oid sha256:e416824c2150276925507a7611e286a76e96671f648bc40d13846ae080b28802
+size 157121741

models/word_markov/br_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "br",
-  "unique_contexts": 9600283,
-  "total_transitions": 21900429
+  "unique_contexts": 8377190,
+  "total_transitions": 15365277
 }

models/word_markov/br_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4dc51b5cdaf1e6d6f00bfc342106c2a491d891dca552edee6c7dc09f3e43da76
-size 237588645
+oid sha256:2dd980b2747cf4b46ceef14ea14dd7faf42da8d72d82141bf5e94c7943d016ce
+size 198463924

models/word_markov/br_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "br",
-  "unique_contexts": 13824129,
-  "total_transitions": 21810331
+  "unique_contexts": 11216136,
+  "total_transitions": 15276454
 }

models/word_ngram/br_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0378b53defe0eedb7989c2010d751a14d99928386aba3ced76bad82ec402ec36
-size 5232515
+oid sha256:8cf0a3e72d51913a31496c69f8b0c3a5557067df085981f0f30eb30c446795de
+size 4100389

models/word_ngram/br_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "br",
-  "unique_ngrams": 396301,
-  "total_ngrams": 22080722
+  "unique_ngrams": 296192,
+  "total_ngrams": 15542923
 }

models/word_ngram/br_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:513c163c0b3b0ac784029384146f1f680f6310d92cb835eb7aad27b794f28240
-size 12329602
+oid sha256:14593e06d89cdb20d9c666016043454320f17e45c7383903f2d8fe59206d47be
+size 8580184

models/word_ngram/br_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "br",
-  "unique_ngrams": 858386,
-  "total_ngrams": 21990556
+  "unique_ngrams": 570380,
+  "total_ngrams": 15454100
 }

models/word_ngram/br_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6e17780562a29c8720acf9f11550b417c291cd4606856491e09f594c95e9fc88
-size 23878363
+oid sha256:53200ebaddcf544fba280fb0b628cb226d70d854902efc6cb0a1b28fe2934c37
+size 15534803

models/word_ngram/br_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "br",
-  "unique_ngrams": 1581104,
-  "total_ngrams": 21900429
+  "unique_ngrams": 973376,
+  "total_ngrams": 15365277
 }