Upload all models and assets for bar (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 3.790
+    value: 4.002
   - name: best_isotropy
     type: isotropy
-    value: 0.8361
+    value: 0.8442
   - name: vocabulary_size
     type: vocab
-    value: 225914
-generated: 2025-12-28
+    value: 0
+generated: 2026-01-03
 ---
 
 # BAR - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,59 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.051x | 3.01 | 0.0348% | 1,184,756 |
-| **16k** | 3.320x | 3.27 | 0.0378% | 1,088,767 |
-| **32k** | 3.568x | 3.52 | 0.0407% | 1,013,044 |
-| **64k** | 3.790x 🏆 | 3.74 | 0.0432% | 953,541 |
+| **8k** | 3.167x | 3.17 | 0.0429% | 1,049,729 |
+| **16k** | 3.475x | 3.48 | 0.0470% | 956,699 |
+| **32k** | 3.752x | 3.75 | 0.0508% | 885,998 |
+| **64k** | 4.002x 🏆 | 4.00 | 0.0542% | 830,614 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Platte County is a County in Nebraska in da USA.
-
- Beleg 
-
- Im Netz 
-
-Kategorie:...`
+**Sample 1:** `Buffalo County Obgruafa am 22. Feba is a County in Wisconsin in da USA. Beleg Im...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁plat te ▁county ▁is ▁a ▁county ▁in ▁nebraska ▁in ▁da ... (+10 more)` | 20 |
-| 16k | `▁plat te ▁county ▁is ▁a ▁county ▁in ▁nebraska ▁in ▁da ... (+10 more)` | 20 |
-| 32k | `▁platte ▁county ▁is ▁a ▁county ▁in ▁nebraska ▁in ▁da ▁usa ... (+9 more)` | 19 |
-| 64k | `▁platte ▁county ▁is ▁a ▁county ▁in ▁nebraska ▁in ▁da ▁usa ... (+9 more)` | 19 |
+| 8k | `▁buffalo ▁county ▁obgruafa ▁am ▁ 2 2 . ▁feba ▁is ... (+13 more)` | 23 |
+| 16k | `▁buffalo ▁county ▁obgruafa ▁am ▁ 2 2 . ▁feba ▁is ... (+13 more)` | 23 |
+| 32k | `▁buffalo ▁county ▁obgruafa ▁am ▁ 2 2 . ▁feba ▁is ... (+13 more)` | 23 |
+| 64k | `▁buffalo ▁county ▁obgruafa ▁am ▁ 2 2 . ▁feba ▁is ... (+13 more)` | 23 |
 
-**Sample 2:** `Union County. Obgruafa am 22. Feba 2011 is a County in South Carolina in da USA....`
+**Sample 2:** `Fauquier County. Obgruafa am 22. Feba is a County in Virginia in da USA. Beleg I...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁union ▁county . ▁obgruafa ▁am ▁ 2 2 . ▁feba ... (+24 more)` | 34 |
-| 16k | `▁union ▁county . ▁obgruafa ▁am ▁ 2 2 . ▁feba ... (+24 more)` | 34 |
-| 32k | `▁union ▁county . ▁obgruafa ▁am ▁ 2 2 . ▁feba ... (+24 more)` | 34 |
-| 64k | `▁union ▁county . ▁obgruafa ▁am ▁ 2 2 . ▁feba ... (+24 more)` | 34 |
-
-**Sample 3:** `Des is a Iwablick iwas Joar 1561.
+| 8k | `▁f au qui er ▁county . ▁obgruafa ▁am ▁ 2 ... (+17 more)` | 27 |
+| 16k | `▁f au qui er ▁county . ▁obgruafa ▁am ▁ 2 ... (+17 more)` | 27 |
+| 32k | `▁fau qui er ▁county . ▁obgruafa ▁am ▁ 2 2 ... (+16 more)` | 26 |
+| 64k | `▁fau qui er ▁county . ▁obgruafa ▁am ▁ 2 2 ... (+16 more)` | 26 |
 
- Im Netz`
+**Sample 3:** `Carlow stähd fia: Carlow, Stod in Irland County Carlow, irische Grofschoft Carlo...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁des ▁is ▁a ▁iwablick ▁iwas ▁joar ▁ 1 5 6 ... (+4 more)` | 14 |
-| 16k | `▁des ▁is ▁a ▁iwablick ▁iwas ▁joar ▁ 1 5 6 ... (+4 more)` | 14 |
-| 32k | `▁des ▁is ▁a ▁iwablick ▁iwas ▁joar ▁ 1 5 6 ... (+4 more)` | 14 |
-| 64k | `▁des ▁is ▁a ▁iwablick ▁iwas ▁joar ▁ 1 5 6 ... (+4 more)` | 14 |
+| 8k | `▁carl ow ▁stähd ▁fia : ▁carl ow , ▁stod ▁in ... (+18 more)` | 28 |
+| 16k | `▁carl ow ▁stähd ▁fia : ▁carl ow , ▁stod ▁in ... (+17 more)` | 27 |
+| 32k | `▁carl ow ▁stähd ▁fia : ▁carl ow , ▁stod ▁in ... (+16 more)` | 26 |
+| 64k | `▁carlow ▁stähd ▁fia : ▁carlow , ▁stod ▁in ▁irland ▁county ... (+10 more)` | 20 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 3.790x compression
-- **Lowest UNK Rate:** 8k with 0.0348% unknown tokens
+- **Best Compression:** 64k achieves 4.002x compression
+- **Lowest UNK Rate:** 8k with 0.0429% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -129,57 +129,89 @@ Kategorie:...`
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 26,677 🏆 | 14.70 | 159,756 | 14.4% | 35.4% |
-| **2-gram** | 438 🏆 | 8.77 | 9,188 | 56.6% | 97.2% |
-| **3-gram** | 61,390 | 15.91 | 246,468 | 9.0% | 25.7% |
-| **3-gram** | 4,733 | 12.21 | 83,186 | 18.8% | 56.7% |
-| **4-gram** | 99,269 | 16.60 | 386,315 | 9.1% | 23.6% |
-| **4-gram** | 33,953 | 15.05 | 473,217 | 8.7% | 26.5% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 27,417 | 14.74 | 110,635 | 12.9% | 31.4% |
+| **2-gram** | Subword | 362 🏆 | 8.50 | 7,805 | 60.7% | 98.3% |
+| **3-gram** | Word | 41,058 | 15.33 | 129,534 | 12.6% | 26.5% |
+| **3-gram** | Subword | 3,802 | 11.89 | 63,080 | 20.6% | 60.8% |
+| **4-gram** | Word | 57,367 | 15.81 | 187,348 | 13.7% | 25.1% |
+| **4-gram** | Subword | 27,463 | 14.75 | 363,810 | 9.1% | 28.4% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `vo da` | 26,665 |
+| 2 | `is a` | 22,998 |
+| 3 | `in da` | 22,567 |
+| 4 | `im netz` | 14,649 |
+| 5 | `vo de` | 13,503 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `beleg im netz` | 3,527 |
+| 2 | `in da usa` | 3,478 |
+| 3 | `da beziak hod` | 2,393 |
+| 4 | `des is a` | 2,037 |
+| 5 | `im netz in` | 2,001 |
+
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `kategorie :` | 38,959 |
-| 2 | `vo da` | 26,615 |
-| 3 | `is a` | 23,040 |
-| 4 | `in da` | 22,485 |
-| 5 | `. de` | 21,082 |
+| 1 | `beleg im netz in` | 1,573 |
+| 2 | `da sitz vo da` | 1,483 |
+| 3 | `is a county in` | 1,429 |
+| 4 | `in da usa da` | 1,407 |
+| 5 | `a katastralgmoa in da` | 1,387 |
 
-**3-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `beleg kategorie :` | 6,999 |
-| 2 | `isbn 3 -` | 5,924 |
-| 3 | `. im netz` | 5,460 |
-| 4 | `kategorie : ort` | 5,121 |
-| 5 | `| | |` | 4,937 |
+| 1 | `n _` | 706,670 |
+| 2 | `a _` | 671,532 |
+| 3 | `c h` | 640,658 |
+| 4 | `_ d` | 560,830 |
+| 5 | `e _` | 482,452 |
 
-**4-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `, isbn 3 -` | 4,499 |
-| 2 | `| align = "` | 3,879 |
-| 3 | `align = " center` | 3,590 |
-| 4 | `= " center "` | 3,590 |
-| 5 | `kategorie : ort im` | 3,505 |
+| 1 | `s c h` | 305,657 |
+| 2 | `_ d e` | 255,292 |
+| 3 | `_ d a` | 174,094 |
+| 4 | `n d _` | 170,331 |
+| 5 | `d a _` | 169,070 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ d a _` | 133,118 |
+| 2 | `_ d e _` | 131,138 |
+| 3 | `u n d _` | 128,509 |
+| 4 | `_ u n d` | 120,455 |
+| 5 | `i s c h` | 100,072 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 438
+- **Best Perplexity:** 2-gram (subword) with 362
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~27% of corpus
+- **Coverage:** Top-1000 patterns cover ~28% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -187,55 +219,86 @@ Kategorie:...`
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.6074 | 1.523 | 4.85 | 638,594 | 39.3% |
-| **1** | 1.0706 | 2.100 | 7.75 | 3,379 | 0.0% |
-| **2** | 0.2487 | 1.188 | 1.74 | 3,093,978 | 75.1% |
-| **2** | 0.9487 | 1.930 | 6.44 | 26,199 | 5.1% |
-| **3** | 0.0996 | 1.071 | 1.21 | 5,375,695 | 90.0% |
-| **3** | 0.9366 | 1.914 | 4.86 | 168,731 | 6.3% |
-| **4** | 0.0401 🏆 | 1.028 | 1.07 | 6,470,090 | 96.0% |
-| **4** | 0.7672 🏆 | 1.702 | 3.38 | 820,482 | 23.3% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.7091 | 1.635 | 5.19 | 569,846 | 29.1% |
+| **1** | Subword | 0.9426 | 1.922 | 6.61 | 3,388 | 5.7% |
+| **2** | Word | 0.2116 | 1.158 | 1.52 | 2,948,968 | 78.8% |
+| **2** | Subword | 0.9158 | 1.887 | 5.85 | 22,382 | 8.4% |
+| **3** | Word | 0.0664 | 1.047 | 1.12 | 4,475,523 | 93.4% |
+| **3** | Subword | 0.8683 | 1.826 | 4.67 | 130,801 | 13.2% |
+| **4** | Word | 0.0224 🏆 | 1.016 | 1.04 | 4,973,907 | 97.8% |
+| **4** | Subword | 0.7777 | 1.714 | 3.53 | 610,360 | 22.2% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. 524 angewachsen und die fläche vo dera zoagt , par les ' n ) u`
-2. `, isbn 978 - mal 1920 bis auf des is kemnath ( " franz kafka .`
-3. `- deitscha schauspuia und a zaumgroida schdrudl is ois broad bekaunnt hans ( 2016 saha air`
+1. `de flugsauria buidns de knaj oda negakuss domois ghairat hod direkt in den jüngling oder goar`
+2. `da insl blaagad hom niks gwisst hod a öatschoft im netz hoamseitn vo mercia zrugg mei`
+3. `und is er im neich augleande mocha oda z himinbjörg und ů und san letztle zua`
 
 **Context Size 2:**
 
-1. `kategorie : artikel auf niederösterreichisch kategorie : ortsteil von wieseth kategorie : geboren 18...`
-2. `vo da mathematik , schau gorkhaland`
-3. `is a bruck ' nschlåg ; da linné aun an rio xingu . as gericht håtn zu`
+1. `vo da vawoitung is in n gensatz za altn welt is a bleamalkiag ausgoat und in bemen`
+2. `is a urtschoft und a jeda miassat eintritt brandln dann war des eagebnis vo de großn industriezentre...`
+3. `in da usa on da anderson mesa in da langobardischn ehefrow vom kini ludwig i vo habsbuag`
 
 **Context Size 3:**
 
-1. `beleg kategorie : johann nestroy ; stücke 21 . s . highway 84 mindt . uma 32 kilometa`
-2. `isbn 3 - 417 - 20675 - 8 ( teubner - studienbücher der geographie ) . im joa`
-3. `kategorie : ort auf den färöern kategorie : streymoy`
+1. `in da usa beleg im netz in south carolina in da usa da beziak hod a fläche vo`
+2. `beleg im netz eana hoamseitn eana myspace seitn eana facebook seitn volksmusik`
+3. `da beziak hod a fläch vo 802 km af dena 49 970 eihwohna lem stond gmoana da powiat`
 
 **Context Size 4:**
 
-1. `, isbn 3 - 406 - 46224 - 3 birgit zotz : destination tibet . touristisches image zwischen politik`
-2. `| align = " center " | | - | berleu | | | | | | | |`
-3. `= " center " | | | align = " center " | | | fatututa | | align`
+1. `beleg im netz in der normandie im département seine maritime in da region normandie ea liegd im arro...`
+2. `da sitz vo da vawoitung is lake city da beziak hod a flächn vo quadratkilometa dovo san 1 quadratkil...`
+3. `is a county in virginia in da usa beleg im netz in missouri`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_heerinznachrnea`
+2. `an_knenant_getun`
+3. `eichaumo_bh_ll,_`
+
+**Context Size 2:**
+
+1. `n_de_autz)_val_(z`
+2. `a_berseeka)_trejn`
+3. `chulretiveicittbo`
+
+**Context Size 3:**
+
+1. `sch-wei_in_de_im_o`
+2. `_der_schaubind_so_`
+3. `_da_hamation_phana`
+
+**Context Size 4:**
+
+1. `_da_sicht_große_sog`
+2. `_de_kompillatinen_t`
+3. `und_europäischen_(a`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 96.0% predictability
+- **Best Predictability:** Context-4 (word) with 97.8% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (820,482 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (610,360 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -251,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 225,914 |
-| Total Tokens | 5,874,699 |
-| Mean Frequency | 26.00 |
+| Vocabulary Size | 213,465 |
+| Total Tokens | 5,378,004 |
+| Mean Frequency | 25.19 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 709.74 |
+| Frequency Std Dev | 715.69 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | de | 139,734 |
-| 2 | da | 136,994 |
-| 3 | und | 120,375 |
-| 4 | in | 102,834 |
-| 5 | a | 93,585 |
-| 6 | vo | 92,275 |
-| 7 | is | 88,045 |
-| 8 | im | 71,546 |
-| 9 | kategorie | 39,103 |
-| 10 | des | 34,614 |
+| 1 | de | 137,737 |
+| 2 | da | 137,316 |
+| 3 | und | 119,692 |
+| 4 | in | 102,651 |
+| 5 | a | 92,739 |
+| 6 | vo | 92,570 |
+| 7 | is | 86,950 |
+| 8 | im | 71,173 |
+| 9 | des | 34,457 |
+| 10 | hod | 30,772 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | mechanisches | 2 |
-| 2 | stabilisierungssystem | 2 |
-| 3 | voeffentlecht | 2 |
-| 4 | innpuls | 2 |
-| 5 | buagstej | 2 |
-| 6 | nuwenburg | 2 |
-| 7 | kulturweges | 2 |
-| 8 | spessartprojektes | 2 |
-| 9 | terrassnfermig | 2 |
-| 10 | tuamhigi | 2 |
+| 1 | vorarlberga | 2 |
+| 2 | opfenbach | 2 |
+| 3 | raubibafäi | 2 |
+| 4 | marcianopel | 2 |
+| 5 | sachtler | 2 |
+| 6 | vitec | 2 |
+| 7 | videocom | 2 |
+| 8 | promovierten | 2 |
+| 9 | mechanisches | 2 |
+| 10 | stabilisierungssystem | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9896 |
-| R² (Goodness of Fit) | 0.999155 |
+| Zipf Coefficient | 0.9728 |
+| R² (Goodness of Fit) | 0.999432 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 32.7% |
-| Top 1,000 | 54.6% |
-| Top 5,000 | 70.2% |
-| Top 10,000 | 76.9% |
+| Top 100 | 34.1% |
+| Top 1,000 | 55.0% |
+| Top 5,000 | 70.0% |
+| Top 10,000 | 76.7% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9992 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 32.7% of corpus
-- **Long Tail:** 215,914 words needed for remaining 23.1% coverage
+- **Zipf Compliance:** R²=0.9994 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 34.1% of corpus
+- **Long Tail:** 203,465 words needed for remaining 23.3% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -321,24 +384,125 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 92,573 | 32 | 3.954 | 1.316 | 0.8131 |
-| **mono_64d** | 92,573 | 64 | 4.642 | 1.256 | 0.8361 🏆 |
-| **mono_128d** | 92,573 | 128 | 5.543 | 1.143 | 0.8310 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8230 | 0.3300 | N/A | N/A |
+| **mono_64d** | 64 | 0.8442 🏆 | 0.2564 | N/A | N/A |
+| **mono_128d** | 128 | 0.8427 | 0.1773 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_64d with 0.8361 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 92,573 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_64d with 0.8442 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2546. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-be` | begleitendn, bewohnde, bewiabt |
+| `-sc` | schwingt, schienengebundenen, schwefelhölzern |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-n` | begleitendn, lesegerätn, clipperton |
+| `-en` | warmgemäßigten, alanen, aussen |
+| `-er` | puppentheater, kirchenmusiker, rothmüller |
+| `-ng` | hamhŭng, polung, urauffüahrung |
+| `-ch` | woifschbouch, mittlboarisch, meafoch |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `icht` | 1.84x | 346 contexts | richt, eicht, dicht |
+| `schr` | 2.11x | 137 contexts | schrei, schräg, schrag |
+| `gsch` | 1.93x | 181 contexts | gschdö, gscher, gschod |
+| `schl` | 1.64x | 288 contexts | eschl, ischl, göschl |
+| `chte` | 1.70x | 217 contexts | åchte, echte, ochte |
+| `itsc` | 2.10x | 64 contexts | gitsch, kitsch, nitsch |
+| `chof` | 2.22x | 50 contexts | schof, schofn, schoft |
+| `tlic` | 1.76x | 137 contexts | etlich, etlichs, rötlich |
+| `atio` | 2.18x | 45 contexts | natio, ratio, nation |
+| `nisc` | 1.72x | 127 contexts | nisch, nischt, nischn |
+| `ichn` | 1.97x | 68 contexts | eichn, suichn, zoichn |
+| `uach` | 1.78x | 99 contexts | duach, buach, suach |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-sc` | `-n` | 53 words | schleierbaracken, schüidln |
+| `-be` | `-n` | 43 words | betroffanan, berichtigungen |
+| `-sc` | `-en` | 13 words | schleierbaracken, schnupfen |
+| `-be` | `-ng` | 13 words | bevejkarungsentwigglung, bereicherung |
+| `-sc` | `-er` | 13 words | schimpfkalender, schweller |
+| `-sc` | `-ch` | 10 words | schrambach, schpruch |
+| `-be` | `-en` | 9 words | berichtigungen, beten |
+| `-be` | `-ch` | 4 words | besuch, bessenbach |
+| `-be` | `-er` | 3 words | bettinger, berghammer |
+| `-sc` | `-ng` | 3 words | schoidruckpeglmindarung, schiefling |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| betreiber | **`be-treib-er`** | 6.0 | `treib` |
+| vorarlberger | **`vorarlberg-er`** | 4.5 | `vorarlberg` |
+| verkaufen | **`verkauf-en`** | 4.5 | `verkauf` |
+| grotesken | **`grotesk-en`** | 4.5 | `grotesk` |
+| schwabinger | **`sc-hwabi-ng-er`** | 4.5 | `hwabi` |
+| gsprochenen | **`gspro-ch-en-en`** | 4.5 | `gspro` |
+| waxenberger | **`waxenberg-er`** | 4.5 | `waxenberg` |
+| scheazhoft | **`sc-heazhoft`** | 4.5 | `heazhoft` |
+| gebildeten | **`gebildet-en`** | 4.5 | `gebildet` |
+| carstensen | **`carstens-en`** | 4.5 | `carstens` |
+| bewundern | **`be-wundern`** | 4.5 | `wundern` |
+| dornröschen | **`dornrös-ch-en`** | 3.0 | `dornrös` |
+| überetscher | **`überets-ch-er`** | 3.0 | `überets` |
+| betrieblich | **`be-triebli-ch`** | 3.0 | `triebli` |
+| umgebungen | **`umgebu-ng-en`** | 3.0 | `umgebu` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language BAR appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -346,11 +510,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (3.79x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (438) |
-| Markov | **Context-4** | Highest predictability (96.0%) |
+| Tokenizer | **64k BPE** | Best compression (4.00x) |
+| N-gram | **2-gram** | Lowest perplexity (362) |
+| Markov | **Context-4** | Highest predictability (97.8%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -540,7 +705,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -556,7 +722,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-28 00:09:41*
+*Report Date: 2026-01-03 06:42:51*

models/embeddings/monolingual/bar_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b67e002458c6ffe3851456e1d214d989436508fafb3940aa0a7aeac7f9501600
-size 1120431691
+oid sha256:6256ddf046563a10ad0e96ef58deffac4839b1eb48bc65facd0dd74ba05e830b
+size 1115788582

models/embeddings/monolingual/bar_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 92573
+  "vocab_size": 88124
 }

models/embeddings/monolingual/bar_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d60a4fc93329fd2886ca8100a062412d4d07afea28e09e264dc800ac247d496b
-size 281335627
+oid sha256:cc7a6c48e154f63a623a8376127a01b222af8bf2a04aba41f602a929eb6c27af
+size 280109350

models/embeddings/monolingual/bar_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 92573
+  "vocab_size": 88124
 }

models/embeddings/monolingual/bar_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8669bc90711c772638673e6ddeca2a5ceb915402131cac9b7c3628c941baa702
-size 561034315
+oid sha256:9998545a04e63264d36cca0eb51c1e069804e9cdba44f02039cd26cd05531df8
+size 558669094

models/embeddings/monolingual/bar_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 92573
+  "vocab_size": 88124
 }

models/subword_markov/bar_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:60e1fcddc267b2ddb4763a4f04458f5fc0a3864d4d3b65b068a66791c6d11ca9
-size 199512
+oid sha256:60e6c3a72c3c4d893f6ddf06c56e1234d3475b1b1443e819143470ab5f1b4e23
+size 182402

models/subword_markov/bar_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "bar",
-  "unique_contexts": 3379,
-  "total_transitions": 41526016
+  "unique_contexts": 3388,
+  "total_transitions": 36898936
 }

models/subword_markov/bar_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:44ac0dfee6ebc508b0694bd88f8515a3ecd00e7b531cab5055a6e18d776b7900
-size 1274970
+oid sha256:dc48393a4abf97a7708eb8626bb77b238a51c44b6bba67b7e1d4fd0bd11ec189
+size 1029228

models/subword_markov/bar_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "bar",
-  "unique_contexts": 26199,
-  "total_transitions": 41498726
+  "unique_contexts": 22382,
+  "total_transitions": 36872004
 }

models/subword_markov/bar_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dab3f88d8a597024d6c2d45205f49335376cff21c6acc22705b1aeb5a3d6d9b7
-size 5898144
+oid sha256:2791996798e11cff4740288bb4e88ef150e9193843760570285c334a6ad21342
+size 4585569

models/subword_markov/bar_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "bar",
-  "unique_contexts": 168731,
-  "total_transitions": 41471436
+  "unique_contexts": 130801,
+  "total_transitions": 36845072
 }

models/subword_markov/bar_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e7d35e1ed6aa219268b066af5ab218034a029fe871fbf27a8621d574e9ea9bf2
-size 20162408
+oid sha256:19edd5d2767e201a87efc206fec70368c6dc23b8ed658bbf39f26635b1f373d3
+size 15789022

models/subword_markov/bar_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "bar",
-  "unique_contexts": 820482,
-  "total_transitions": 41444146
+  "unique_contexts": 610360,
+  "total_transitions": 36818140
 }

models/subword_ngram/bar_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:98fc1ea2db56ddb0c1e4d55f5df47bce76f4c454fad62a3186181d6e59bf7819
-size 120210
+oid sha256:9e38e7d14a00c23438ba07cef06a0be700a6eaea54c9aa917c3319ce7eba165f
+size 102154

models/subword_ngram/bar_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "bar",
-  "unique_ngrams": 9188,
-  "total_ngrams": 41526016
+  "unique_ngrams": 7805,
+  "total_ngrams": 36898936
 }

models/subword_ngram/bar_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:eb5ad1ec42d5688187fa2244bcea28e012a96c3954cbff00e1919cc637e39cfa
-size 980179
+oid sha256:ff09d3ef6e1cfa552449a8dedd31d9754c60b8a91db7126c3cc9c4c71957830a
+size 780954

models/subword_ngram/bar_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "bar",
-  "unique_ngrams": 83186,
-  "total_ngrams": 41498726
+  "unique_ngrams": 63080,
+  "total_ngrams": 36872004
 }

models/subword_ngram/bar_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:74cb0c9a29de3bf166be1817974bf03a79e844a9172dca886d1e6c11444fe105
-size 5250784
+oid sha256:72f067181a08a87d97a0065d3a1bdaeb78404363053efae74d287fdc5e58187b
+size 4090739

models/subword_ngram/bar_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "bar",
-  "unique_ngrams": 473217,
-  "total_ngrams": 41471436
+  "unique_ngrams": 363810,
+  "total_ngrams": 36845072
 }

models/tokenizer/bar_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:86990145267f2f23bece3cf170063df25b610b0172bb06154c137114776c8277
-size 497891
+oid sha256:70ddb2da3b8a1d35b41491bf8a0ea06f6e0357f42e81f619258dafcbc3567d87
+size 500516

models/tokenizer/bar_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:75f629bdc6954475d208780fe096b578aa8ab68a13993884c783d727d30b36c9
-size 769242
+oid sha256:f7c6a1964c1d0a5e0f3f657f5fe39d0fc4524fef607b13be5a9d6be1c48e71b2
+size 773157

models/tokenizer/bar_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3817f1c29c6409fc0d09a5641df6d546b4d73373e90aff314bd676d06f34afe9
-size 1329778
+oid sha256:0a6c179e6feecfed4b2ee3d6bb1547e591a0fdece5e1f9b05bc8dc198ae6ac1b
+size 1335748

models/tokenizer/bar_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cc3653e3a2ae044e86fc99a6e9f56a602c7345188f2a36c7e517e330131174fa
-size 367058
+oid sha256:77089651ccfa512d13316ef10f75bd2ec3ead99b2fce243a47c46f50b404db47
+size 368818

models/vocabulary/bar_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3c50b70ce0ccf85aa73d5c98c50c1637e81c02c9079bae2cfca6036b4a7fd444
-size 3610908
+oid sha256:d7edcd6cf37858bad1ad7f0f51a2247a6bb728fc9c8b48ffb32bfd46cfe4ee3f
+size 3431123

models/vocabulary/bar_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "bar",
-  "vocabulary_size": 225914,
+  "vocabulary_size": 213465,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.1016273267091797,
+    "type_token_ratio": 0.0995109201675251,
     "coverage": {
-      "top_100": 0.3056598360284214,
-      "top_1000": 0.5100686297601529,
-      "top_5000": 0.6556657283942213,
-      "top_10000": 0.7183725874101852
+      "top_100": 0.3198175845558876,
+      "top_1000": 0.5158986689356716,
+      "top_5000": 0.6559721791165527,
+      "top_10000": 0.7190383696641477
     },
-    "hapax_count": 413098,
-    "hapax_ratio": 0.6464636031874206,
-    "total_documents": 27290
+    "hapax_count": 357256,
+    "hapax_ratio": 0.6259731112049496,
+    "total_documents": 26932
   }
 }

models/word_markov/bar_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b3dd31c7a802cfbef574568e2ba6b615e18144bf29db409393fb10850929bd7d
-size 30129994
+oid sha256:51608c66d7996b04b88bec1cf78dd596b044f1df0c12652fcb092ce187c00456
+size 29383606

models/word_markov/bar_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "bar",
-  "unique_contexts": 638594,
-  "total_transitions": 8127466
+  "unique_contexts": 569846,
+  "total_transitions": 5708328
 }

models/word_markov/bar_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3db5d683eb8fa5679d0baab234482104310aed5348832ff37632b96f81bac464
-size 67578257
+oid sha256:7fa0c4b83dfa569be9152f9aa90cd11f1e0d964b9f72a6e43fd8f8c02a1c4229
+size 62105262

models/word_markov/bar_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "bar",
-  "unique_contexts": 3093978,
-  "total_transitions": 8100177
+  "unique_contexts": 2948968,
+  "total_transitions": 5681396
 }

models/word_markov/bar_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9253b3e64a7a3aa9bd84e29a7bf91e310187acd7b81345ac3362887b3c6a9f7b
-size 96618918
+oid sha256:563c3760b53559738c956fa71ade32f9ab51d036d6cbaf2f0eb9276d18a64547
+size 82156156

models/word_markov/bar_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "bar",
-  "unique_contexts": 5375695,
-  "total_transitions": 8072889
+  "unique_contexts": 4475523,
+  "total_transitions": 5654464
 }

models/word_markov/bar_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ab88f182fc703c53bba76e38edc370a4c8ed1ad8a0ec15d58155beab90c2d334
-size 113834013
+oid sha256:c30b22ec78b8b8678b0b13acf2ffae3de2029fda73f5076c124066a8dc0af5c9
+size 93499333

models/word_markov/bar_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "bar",
-  "unique_contexts": 6470090,
-  "total_transitions": 8045602
+  "unique_contexts": 4973907,
+  "total_transitions": 5627532
 }

models/word_ngram/bar_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:305be2b4a731fc13d78d8a31b51d838be655ee3899d4a3479c5d81c5bc114a10
-size 2084252
+oid sha256:d9d1fd00d794d3cbcff9b6abaf96b0600f5711bb2e3767de57949c3b54808ba8
+size 1517204

models/word_ngram/bar_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "bar",
-  "unique_ngrams": 159756,
-  "total_ngrams": 8127466
+  "unique_ngrams": 110635,
+  "total_ngrams": 5708328
 }

models/word_ngram/bar_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:817288d320b289c5f49b63cb4084a8d5c1cfaf5535e133621f6117c6ac86bce6
-size 3391005
+oid sha256:ddf18025f58d5d74e756ca438583de919245a4a89dafdf1e1698d3c0ad41d5da
+size 1935462

models/word_ngram/bar_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "bar",
-  "unique_ngrams": 246468,
-  "total_ngrams": 8100177
+  "unique_ngrams": 129534,
+  "total_ngrams": 5681396
 }

models/word_ngram/bar_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:70d5bc187fee7c4a76f7ca6f6f46bdc686870760f287cface360e35199b8986c
-size 5697675
+oid sha256:4c7b231b1b6bf00b737f2acedc276944c25abba63ec6d096669529bb9951ec97
+size 3042682

models/word_ngram/bar_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "bar",
-  "unique_ngrams": 386315,
-  "total_ngrams": 8072889
+  "unique_ngrams": 187348,
+  "total_ngrams": 5654464
 }