Upload all models and assets for ang (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.001
+    value: 4.021
   - name: best_isotropy
     type: isotropy
-    value: 0.7980
+    value: 0.7825
   - name: vocabulary_size
     type: vocab
-    value: 32745
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # ANG - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,53 +70,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.091x | 3.04 | 0.0788% | 272,719 |
-| **16k** | 3.414x | 3.36 | 0.0871% | 246,960 |
-| **32k** | 3.716x | 3.66 | 0.0948% | 226,874 |
-| **64k** | 4.001x 🏆 | 3.94 | 0.1020% | 210,690 |
+| **8k** | 3.112x | 3.12 | 0.0790% | 253,185 |
+| **16k** | 3.447x | 3.45 | 0.0875% | 228,585 |
+| **32k** | 3.771x | 3.78 | 0.0957% | 208,909 |
+| **64k** | 4.021x 🏆 | 4.03 | 0.1021% | 195,954 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Yun Chi-ho (Corēanisc: 윤치호, 26 Gēolmōnaþ, 1864 – 9 Gēolmōnaþ, 1945) ƿæs Corēanis...`
+**Sample 1:** `Ƿada (tacn: 16px|♆) is þæt eahtoþa planēta þǣre sunnlican endebyrdnesse. tungol`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁y un ▁ch i - ho ▁( cor ēan isc ... (+37 more)` | 47 |
-| 16k | `▁y un ▁chi - ho ▁( cor ēan isc : ... (+35 more)` | 45 |
-| 32k | `▁y un ▁chi - ho ▁( corēan isc : ▁ ... (+33 more)` | 43 |
-| 64k | `▁yun ▁chi - ho ▁( corēanisc : ▁ 윤치호 , ... (+31 more)` | 41 |
+| 8k | `▁ƿa da ▁( tac n : ▁ 1 6 px ... (+13 more)` | 23 |
+| 16k | `▁ƿa da ▁( tacn : ▁ 1 6 px | ... (+12 more)` | 22 |
+| 32k | `▁ƿada ▁( tacn : ▁ 1 6 px | ♆ ... (+10 more)` | 20 |
+| 64k | `▁ƿada ▁( tacn : ▁ 1 6 px | ♆ ... (+10 more)` | 20 |
 
-**Sample 2:** `Þega  ƿæs se cyning þāra Ēastgotena fram þǣm 552. gēare oþ þæt læt 552. gēare oþ...`
+**Sample 2:** `Caþerine, Wēala Þēodienen, (ġeboren Caþerine Elisabeþ Middeltūn; 9 Æfterra Gēola...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁þe ga ▁ƿæs ▁se ▁cyning ▁þāra ▁ēast gotena ▁fram ▁þǣm ... (+32 more)` | 42 |
-| 16k | `▁þe ga ▁ƿæs ▁se ▁cyning ▁þāra ▁ēastgotena ▁fram ▁þǣm ▁ ... (+30 more)` | 40 |
-| 32k | `▁þe ga ▁ƿæs ▁se ▁cyning ▁þāra ▁ēastgotena ▁fram ▁þǣm ▁ ... (+30 more)` | 40 |
-| 64k | `▁þe ga ▁ƿæs ▁se ▁cyning ▁þāra ▁ēastgotena ▁fram ▁þǣm ▁ ... (+30 more)` | 40 |
-
-**Sample 3:** `Ælbūrcerrce () oþþe Ælbūrccerrcke is sēo mǣsteburg on Nīƿemexico.
+| 8k | `▁caþ er ine , ▁wēala ▁þēod ien en , ▁( ... (+28 more)` | 38 |
+| 16k | `▁caþerine , ▁wēala ▁þēod ien en , ▁( ġe boren ... (+24 more)` | 34 |
+| 32k | `▁caþerine , ▁wēala ▁þēodienen , ▁( ġeboren ▁caþerine ▁elisabeþ ▁middeltūn ... (+18 more)` | 28 |
+| 64k | `▁caþerine , ▁wēala ▁þēodienen , ▁( ġeboren ▁caþerine ▁elisabeþ ▁middeltūn ... (+18 more)` | 28 |
 
-Flocc:Byrig o...`
+**Sample 3:** `Seo burg Hƿītburg ( oþþe Belgrade) oþþe Singidceaster is sēo hēafodburg and sēo ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁æl bū r cer r ce ▁() ▁oþþe ▁æl bū ... (+27 more)` | 37 |
-| 16k | `▁æl būr cer r ce ▁() ▁oþþe ▁æl būr ccer ... (+24 more)` | 34 |
-| 32k | `▁æl būr cer r ce ▁() ▁oþþe ▁æl būr ccer ... (+23 more)` | 33 |
-| 64k | `▁ælbūr cer r ce ▁() ▁oþþe ▁ælbūr ccer rc ke ... (+19 more)` | 29 |
+| 8k | `▁seo ▁burg ▁hƿīt burg ▁( ▁oþþe ▁belg ra de ) ... (+20 more)` | 30 |
+| 16k | `▁seo ▁burg ▁hƿīt burg ▁( ▁oþþe ▁belg rade ) ▁oþþe ... (+19 more)` | 29 |
+| 32k | `▁seo ▁burg ▁hƿīt burg ▁( ▁oþþe ▁belgrade ) ▁oþþe ▁sing ... (+17 more)` | 27 |
+| 64k | `▁seo ▁burg ▁hƿītburg ▁( ▁oþþe ▁belgrade ) ▁oþþe ▁sing id ... (+16 more)` | 26 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.001x compression
-- **Lowest UNK Rate:** 8k with 0.0788% unknown tokens
+- **Best Compression:** 64k achieves 4.021x compression
+- **Lowest UNK Rate:** 8k with 0.0790% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -123,57 +129,89 @@ Flocc:Byrig o...`
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 4,036 🏆 | 11.98 | 12,127 | 24.9% | 51.6% |
-| **2-gram** | 432 🏆 | 8.76 | 3,589 | 56.9% | 97.1% |
-| **3-gram** | 5,292 | 12.37 | 13,432 | 21.9% | 44.6% |
-| **3-gram** | 3,950 | 11.95 | 28,365 | 20.9% | 59.7% |
-| **4-gram** | 10,291 | 13.33 | 22,900 | 17.1% | 35.0% |
-| **4-gram** | 21,387 | 14.38 | 125,189 | 10.5% | 31.5% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 3,511 | 11.78 | 7,045 | 21.4% | 53.3% |
+| **2-gram** | Subword | 365 🏆 | 8.51 | 3,016 | 61.0% | 98.1% |
+| **3-gram** | Word | 3,285 | 11.68 | 6,002 | 21.8% | 50.6% |
+| **3-gram** | Subword | 3,330 | 11.70 | 23,727 | 22.3% | 62.8% |
+| **4-gram** | Word | 6,683 | 12.71 | 11,447 | 16.8% | 36.4% |
+| **4-gram** | Subword | 18,648 | 14.19 | 105,485 | 10.6% | 32.7% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `flocc :` | 7,126 |
-| 2 | `, and` | 3,048 |
-| 3 | `. flocc` | 2,814 |
-| 4 | `) is` | 1,674 |
-| 5 | `: byrig` | 1,552 |
+| 1 | `in þǣm` | 768 |
+| 2 | `on þǣm` | 759 |
+| 3 | `in þæm` | 693 |
+| 4 | `of the` | 648 |
+| 5 | `se is` | 547 |
 
-**3-grams:**
+**3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `. flocc :` | 2,813 |
-| 2 | `flocc : byrig` | 1,551 |
-| 3 | `: byrig on` | 1,319 |
-| 4 | `( ) is` | 943 |
-| 5 | `< td valign` | 629 |
+| 1 | `td valign top` | 529 |
+| 2 | `is þorp in` | 313 |
+| 3 | `þæs geānedan cynerīces` | 312 |
+| 4 | `eoferwicscīre þæs geānedan` | 248 |
+| 5 | `on eoferwicscīre þæs` | 248 |
 
-**4-grams:**
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `flocc : byrig on` | 1,319 |
-| 2 | `. flocc : byrig` | 999 |
-| 3 | `< td valign =` | 629 |
-| 4 | `td valign = top` | 627 |
-| 5 | `valign = top >` | 615 |
+| 1 | `on eoferwicscīre þæs geānedan` | 248 |
+| 2 | `eoferwicscīre þæs geānedan cynerīces` | 248 |
+| 3 | `is eoferƿicscire dǣl on` | 244 |
+| 4 | `eoferƿicscire dǣl on englum` | 244 |
+| 5 | `se is eoferƿicscire dǣl` | 242 |
+
+**2-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `e _` | 68,661 |
+| 2 | `a n` | 60,782 |
+| 3 | `n _` | 55,172 |
+| 4 | `s _` | 47,775 |
+| 5 | `n d` | 40,577 |
+
+**3-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `a n d` | 24,204 |
+| 2 | `n d _` | 20,527 |
+| 3 | `a n _` | 17,020 |
+| 4 | `_ a n` | 16,519 |
+| 5 | `o n _` | 15,999 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `a n d _` | 16,546 |
+| 2 | `_ a n d` | 14,727 |
+| 3 | `_ o n _` | 10,205 |
+| 4 | `_ i s _` | 10,081 |
+| 5 | `_ i n _` | 9,962 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 432
+- **Best Perplexity:** 2-gram (subword) with 365
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~32% of corpus
+- **Coverage:** Top-1000 patterns cover ~33% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -181,55 +219,86 @@ Flocc:Byrig o...`
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.5832 | 1.498 | 3.71 | 92,131 | 41.7% |
-| **1** | 1.0106 | 2.015 | 7.57 | 1,214 | 0.0% |
-| **2** | 0.1973 | 1.147 | 1.44 | 340,526 | 80.3% |
-| **2** | 1.0352 | 2.049 | 6.24 | 9,184 | 0.0% |
-| **3** | 0.0600 | 1.042 | 1.10 | 488,936 | 94.0% |
-| **3** | 0.8839 | 1.845 | 4.02 | 57,270 | 11.6% |
-| **4** | 0.0221 🏆 | 1.015 | 1.03 | 535,590 | 97.8% |
-| **4** | 0.6030 🏆 | 1.519 | 2.46 | 230,176 | 39.7% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.6222 | 1.539 | 3.58 | 86,720 | 37.8% |
+| **1** | Subword | 0.8536 | 1.807 | 6.47 | 1,240 | 14.6% |
+| **2** | Word | 0.1549 | 1.113 | 1.30 | 307,843 | 84.5% |
+| **2** | Subword | 0.9630 | 1.949 | 5.87 | 8,021 | 3.7% |
+| **3** | Word | 0.0382 | 1.027 | 1.05 | 397,541 | 96.2% |
+| **3** | Subword | 0.8613 | 1.817 | 4.01 | 47,051 | 13.9% |
+| **4** | Word | 0.0126 🏆 | 1.009 | 1.02 | 415,179 | 98.7% |
+| **4** | Subword | 0.6212 | 1.538 | 2.55 | 188,289 | 37.9% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. magnus hēold brytenƿealdan sƿā þæt land æt 77 . 00 solidarity tax rates vary in`
-2. `, 568 , canadian golfer 1960 / haː / 1743 flocc : erica durance sƿā sumum`
-3. `and scotlandes heallum , se mǣsta luh onmiddan þām þēgnum þæs australisca ƿerungþrēates , cent and`
+1. `and ūtƿeardra ēarena dat gearƿum gearƿum acc hāligne scole basic properties geometry type of snāwdūn...`
+2. `on læt westseaxisc norþanhymbrisc miercisc arun cisseceaster craƿley hæstingas ge forlēosaþ hiera ag...`
+3. `is lēoþ þe roðberht roðberhting beweddod æþelhæþ of indian islamic scholar kenichi fukui geapanisc s...`
 
 **Context Size 2:**
 
-1. `flocc : sċīrbyrig on colorado flocc : crabbas`
-2. `, and is hēofodburg paranā þæs rīces and valetta þæs cynelican hired þæs spræce gesprōcen on west`
-3. `. flocc : byrig on þeodsclande . flocc : byrig on eoferwicscīre flocc : ceastra þæs geānedan`
+1. `in þǣm æt paris and roðem liciaþ on hiere rīce ƿæron corsica sardinia and sicilia īege cartaine`
+2. `on þǣm trēoƿenan hrōfe þǣre byrgenne þæt mægdnes ƿelgeāspared līc nēodlīce geāspared mid mēose and b...`
+3. `in þæm geāre bī marianland and þam sæfaroþum þeodsclandes niðerlandes belgican and franclandes in þæ...`
 
 **Context Size 3:**
 
-1. `. flocc : geboren in 1989 flocc : ƿīf flocc : angelseaxisc englaland flocc : stǣr flocc :`
-2. `flocc : byrig on eoferwicscīre flocc : ceastra þæs geānedan cynerīces ‎ flocc : sūþcorēa`
-3. `: byrig on ġeolurēadsċīr ( californie ) flocc : sċīrbyrig on cǣnsasum flocc : byrig and þorpas on`
+1. `td valign top td valign top imperator caesar lvcivs septimvs severvs pertinax avgvstvs small procons...`
+2. `is þorp in þæm east þriding se is eoferƿicscire dǣl on englum hit hæfþ 3 178 būendas on`
+3. `on eoferwicscīre þæs geānedan cynerīces fram þǣm gēare oþ þæt gēar belamp þæt hūs and his foregenga ...`
 
 **Context Size 4:**
 
-1. `flocc : byrig on orlēanascīre flocc : sċīrbyrig on nīwe eoforwīc flocc : sċīrbyrig on miscegan`
-2. `. flocc : byrig and þorpas on sorie`
-3. `< td valign = top > < small > 24 sēremōnaþ 79 oþ 13 hærfestmōnaþ 81 < td valign`
+1. `on eoferwicscīre þæs geānedan cynerīces`
+2. `is eoferƿicscire dǣl on englum hit hæfþ 105 būend on eoferwicscīre þæs geānedan cynerīces`
+3. `eoferƿicscire dǣl on englum heo hæfþ 975 buend on eoferwicscīre þæs geānedan cynerīces`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_t_ate_inn,_mis_`
+2. `egmbrōðon,_s_on_`
+3. `n_brls_þ_k_aliea`
+
+**Context Size 2:**
+
+1. `e_mand_heaxum_sæ_`
+2. `and_ploƿealin_dæg`
+3. `n_enganvicipez)_v`
+
+**Context Size 3:**
+
+1. `andūnsta_hild_on_p`
+2. `nd_(ælesta_æcgrung`
+3. `an_þissibbe._æfn_r`
+
+**Context Size 4:**
+
+1. `and_ƿæs_ƿrīteresfel`
+2. `_and_hīe_(se_ƿord_e`
+3. `_on_villelme._7_heo`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 97.8% predictability
+- **Best Predictability:** Context-4 (word) with 98.7% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (230,176 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (188,289 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -245,64 +314,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 32,745 |
-| Total Tokens | 440,987 |
-| Mean Frequency | 13.47 |
+| Vocabulary Size | 31,177 |
+| Total Tokens | 402,508 |
+| Mean Frequency | 12.91 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 159.89 |
+| Frequency Std Dev | 155.94 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | and | 14,426 |
-| 2 | on | 10,769 |
-| 3 | is | 10,434 |
-| 4 | in | 10,245 |
-| 5 | flocc | 7,161 |
-| 6 | of | 6,202 |
-| 7 | se | 4,329 |
-| 8 | the | 4,026 |
-| 9 | þǣm | 3,673 |
-| 10 | þæs | 3,617 |
+| 1 | and | 14,190 |
+| 2 | on | 10,528 |
+| 3 | in | 10,215 |
+| 4 | is | 10,204 |
+| 5 | of | 6,064 |
+| 6 | se | 4,321 |
+| 7 | the | 3,988 |
+| 8 | þǣm | 3,644 |
+| 9 | þæs | 3,627 |
+| 10 | his | 3,498 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ƿīleacstede | 2 |
-| 2 | cōcsċīre | 2 |
-| 3 | winnebagsċīre | 2 |
-| 4 | ælfrēdingtūn | 2 |
-| 5 | irfung | 2 |
-| 6 | dællassċīr | 2 |
-| 7 | lubbecsċīr | 2 |
-| 8 | larēodo | 2 |
-| 9 | grœndā | 2 |
-| 10 | dǣlungs | 2 |
+| 1 | orcaneġe | 2 |
+| 2 | laguna | 2 |
+| 3 | stātwīca | 2 |
+| 4 | seolhstrand | 2 |
+| 5 | crosern | 2 |
+| 6 | crosernes | 2 |
+| 7 | sīdesċipes | 2 |
+| 8 | heardran | 2 |
+| 9 | caysċīre | 2 |
+| 10 | gjirokastër | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9423 |
-| R² (Goodness of Fit) | 0.997151 |
+| Zipf Coefficient | 0.9331 |
+| R² (Goodness of Fit) | 0.998051 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 37.1% |
-| Top 1,000 | 58.8% |
-| Top 5,000 | 77.7% |
-| Top 10,000 | 86.0% |
+| Top 100 | 37.9% |
+| Top 1,000 | 59.5% |
+| Top 5,000 | 77.9% |
+| Top 10,000 | 86.2% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9972 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 37.1% of corpus
-- **Long Tail:** 22,745 words needed for remaining 14.0% coverage
+- **Zipf Compliance:** R²=0.9981 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 37.9% of corpus
+- **Long Tail:** 21,177 words needed for remaining 13.8% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -315,24 +384,121 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 10,885 | 32 | 3.878 | 0.872 | 0.7980 🏆 |
-| **mono_64d** | 10,885 | 64 | 4.075 | 0.834 | 0.4885 |
-| **mono_128d** | 10,885 | 128 | 4.131 | 0.840 | 0.1418 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.7825 🏆 | 0.3427 | N/A | N/A |
+| **mono_64d** | 64 | 0.4658 | 0.3135 | N/A | N/A |
+| **mono_128d** | 128 | 0.1306 | 0.3083 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.7980 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 10,885 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.7825 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3215. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ge` | gesetedum, germanisca, getimbrod |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-e` | participle, ċeampscipe, smǣte |
+| `-es` | pirates, cromwelles, stranges |
+| `-an` | onginnan, lǣdnan, praetorian |
+| `-um` | gesetedum, mǣnum, strengum |
+| `-de` | onƿendode, landede, aspreade |
+| `-ng` | manufacturing, bringing, georging |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `mani` | 2.06x | 43 contexts | amani, maniȝ, manig |
+| `enne` | 1.98x | 49 contexts | fenne, vienne, etenne |
+| `unge` | 1.86x | 46 contexts | tunge, jungen, ēðunge |
+| `ster` | 1.69x | 59 contexts | buster, easter, ēaster |
+| `tion` | 2.27x | 19 contexts | nation, motion, action |
+| `inga` | 1.74x | 34 contexts | ðinga, minga, þinga |
+| `ning` | 1.67x | 36 contexts | mining, ininga, cyning |
+| `aste` | 1.77x | 27 contexts | easte, ēaste, taste |
+| `ynin` | 2.27x | 11 contexts | cynin, cyninᵹ, cyning |
+| `nden` | 1.74x | 24 contexts | finden, bunden, funden |
+| `afod` | 1.89x | 18 contexts | hēafod, heafod, ƿafode |
+| `nisc` | 1.56x | 30 contexts | denisc, rūnisc, dēnisc |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-ge` | `-e` | 89 words | gesealde, geƿorhte |
+| `-ge` | `-de` | 28 words | gesealde, gebede |
+| `-ge` | `-an` | 21 words | georgian, geƿunelican |
+| `-ge` | `-es` | 19 words | gereces, gewitnes |
+| `-ge` | `-um` | 14 words | gerādum, gelicum |
+| `-ge` | `-ng` | 8 words | gegaderung, gewrixlung |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| gereordes | **`ge-reord-es`** | 6.0 | `reord` |
+| cræftigum | **`cræftig-um`** | 4.5 | `cræftig` |
+| dƿeligendes | **`dƿeligend-es`** | 4.5 | `dƿeligend` |
+| bisceopes | **`bisceop-es`** | 4.5 | `bisceop` |
+| fylgendan | **`fylgend-an`** | 4.5 | `fylgend` |
+| swisslandes | **`swissland-es`** | 4.5 | `swissland` |
+| norðiscan | **`norðisc-an`** | 4.5 | `norðisc` |
+| þēodlican | **`þēodlic-an`** | 4.5 | `þēodlic` |
+| gregoriscan | **`gregorisc-an`** | 4.5 | `gregorisc` |
+| blōtmōnaðes | **`blōtmōnað-es`** | 4.5 | `blōtmōnað` |
+| dufenales | **`dufenal-es`** | 4.5 | `dufenal` |
+| healdende | **`healden-de`** | 4.5 | `healden` |
+| þrōndhāmes | **`þrōndhām-es`** | 4.5 | `þrōndhām` |
+| niðerlendiscan | **`niðerlendisc-an`** | 4.5 | `niðerlendisc` |
+| antarctiscum | **`antarctisc-um`** | 4.5 | `antarctisc` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language ANG appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -340,11 +506,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.00x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (432) |
-| Markov | **Context-4** | Highest predictability (97.8%) |
+| Tokenizer | **64k BPE** | Best compression (4.02x) |
+| N-gram | **2-gram** | Lowest perplexity (365) |
+| Markov | **Context-4** | Highest predictability (98.7%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -534,7 +701,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -550,7 +718,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 06:04:51*
+*Report Date: 2026-01-03 05:11:41*

models/embeddings/monolingual/ang_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fc194d4a31942e2e32c3a8dae780c49b26b5b2dcf2d1a0743c00d42c18909362
-size 1035329826
+oid sha256:dc7bae19e3f22929a7ae4549e0307cea4fff7f0a27889fdd52644f4d57718ccd
+size 1034428777

models/embeddings/monolingual/ang_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 10885
+  "vocab_size": 10020
 }

models/embeddings/monolingual/ang_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6df9faaec5697096804c52251169b69509f4b58ecec42b9c97f06f9b99d3a2d6
-size 258970146
+oid sha256:76152ef959e26af0b83086f074dcef1d392da8b664962055a3b2fc4489805dfb
+size 258733417

models/embeddings/monolingual/ang_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 10885
+  "vocab_size": 10020
 }

models/embeddings/monolingual/ang_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:07df11aa5c3e43134f3a3f21f7f1109a62605b73ffd0840c3a062fefcaac8477
-size 517756706
+oid sha256:46f6519234c95a833d0492a96515afc2ecb16722d834b8a1bb54212baff798b7
+size 517298537

models/embeddings/monolingual/ang_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 10885
+  "vocab_size": 10020
 }

models/subword_markov/ang_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:96efa4fd32f8723ecce729acbd1c362fe3222b904974ce6779fa687e76c388d0
-size 73394
+oid sha256:90a9a8841ba5050d61a53c8c4b882418b11758d12d4e4bfb0e1c35962a5ff66e
+size 67847

models/subword_markov/ang_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "ang",
-  "unique_contexts": 1214,
-  "total_transitions": 3248126
+  "unique_contexts": 1240,
+  "total_transitions": 2931555
 }

models/subword_markov/ang_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:baf874239e5f656387a4b53b454dad05437a34b8038d1d1b3d564c5cba24be6a
-size 453198
+oid sha256:9862fd96d60b2ddcba7f9f843ca6ff64c8e2609911fdd8fe6814efae3de58e89
+size 359618

models/subword_markov/ang_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "ang",
-  "unique_contexts": 9184,
-  "total_transitions": 3242991
+  "unique_contexts": 8021,
+  "total_transitions": 2926656
 }

models/subword_markov/ang_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:43326969d8e8254044d5345d200b3dc66db5c4bc7c296648dd4075f37a900d29
-size 1646395
+oid sha256:bdba84f6d74bc0545de74369839c3baecf845b91493ccd850300aa17dc493368
+size 1382361

models/subword_markov/ang_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "ang",
-  "unique_contexts": 57270,
-  "total_transitions": 3237856
+  "unique_contexts": 47051,
+  "total_transitions": 2921757
 }

models/subword_markov/ang_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2300760c01bd695ee195c7c4bbe9527831968d70b9e8a499bbdf3dbb2acc8094
-size 4538468
+oid sha256:2975bbac2b2a98d8f25d17a40f1919b0cb2c89af3cb5e283029e0814841eb485
+size 3773700

models/subword_markov/ang_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "ang",
-  "unique_contexts": 230176,
-  "total_transitions": 3232721
+  "unique_contexts": 188289,
+  "total_transitions": 2916858
 }

models/subword_ngram/ang_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9d0aef14dece44db6119149b33b24ba911374e76abaa8d1c9930045cc8cd84a9
-size 47002
+oid sha256:ae5772afcd4edeeb505f3f15218338f4ab947f1eb61616ecf75e9477428c3bdf
+size 40718

models/subword_ngram/ang_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "ang",
-  "unique_ngrams": 3589,
-  "total_ngrams": 3248126
+  "unique_ngrams": 3016,
+  "total_ngrams": 2931555
 }

models/subword_ngram/ang_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b4356d4e74991c0b2f9daec42ea9d0f193e6e25fc974deb779a45134d6258b09
-size 345647
+oid sha256:d2c6ba1309e9103dc84eb58507d098ed662ea9cfea0a9c9f9b74ba147f439950
+size 290728

models/subword_ngram/ang_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "ang",
-  "unique_ngrams": 28365,
-  "total_ngrams": 3242991
+  "unique_ngrams": 23727,
+  "total_ngrams": 2926656
 }

models/subword_ngram/ang_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e4867afacfc08d62f62d898a90eed09e96af5d562b3a54b91b0acde18a8c8c96
-size 1444787
+oid sha256:214ae5f8a7aa366afa79d7b55156420e4cde925f114aacb43a0e8b1677ac34eb
+size 1206179

models/subword_ngram/ang_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "ang",
-  "unique_ngrams": 125189,
-  "total_ngrams": 3237856
+  "unique_ngrams": 105485,
+  "total_ngrams": 2921757
 }

models/tokenizer/ang_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:10e4d0807a40b872f29528b5c23f9841503ebf5aab866002da9ea38b52f50d27
-size 506547
+oid sha256:10b809027a944dfea82e7565213a5a0dd09ee883ea5c6abc3dc9dae21515155d
+size 507706

models/tokenizer/ang_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:334a8b8b3a0ebc409c568a256e9614276ad9185b0da9661339bec148fc9baa34
-size 779127
+oid sha256:882f479e6372ebf5cf806ece607984b48ce4e93dc139971c738b81acd86b41e5
+size 785063

models/tokenizer/ang_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b03ece4ccfeeb1e095675628468e8394e4b1b8f9b3dc73660c628eabfb345361
-size 1372424
+oid sha256:1e5786df616a5b116d734bb3ad420c192cfac85b05de0422ef4883ffab1a5188
+size 1394695

models/tokenizer/ang_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:951f11d514320c9af6f06489cd33c223767ce8110d61ae10acfd10c2f4c1d819
-size 370408
+oid sha256:9350642caa9ab4fcb09383368a4dd339cdebaa4100ee5aa4e4b6128518a978ae
+size 372384

models/vocabulary/ang_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8293a2194f8ee9bccc9ba1e789d27affd0a5326a35adb253b8aa12dd20a95bee
-size 558369
+oid sha256:ee7206585fc58294269ce4be08117acf83241efa4ebab2fe8ed326955b226129
+size 531959

models/vocabulary/ang_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "ang",
-  "vocabulary_size": 32745,
+  "vocabulary_size": 31177,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.18429092362255856,
+    "type_token_ratio": 0.19034733881342258,
     "coverage": {
-      "top_100": 0.3267196163644538,
-      "top_1000": 0.5182676457365503,
-      "top_5000": 0.6850711823767421,
-      "top_10000": 0.7579399570408112
+      "top_100": 0.33234415541940127,
+      "top_1000": 0.5225039792425267,
+      "top_5000": 0.6832392124370408,
+      "top_10000": 0.7561302138979134
     },
-    "hapax_count": 59488,
-    "hapax_ratio": 0.6449752257868658,
-    "total_documents": 5135
+    "hapax_count": 56122,
+    "hapax_ratio": 0.6428710523602791,
+    "total_documents": 4899
   }
 }

models/word_markov/ang_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2a00910d0170554920791ac4446cf494c25bbb7bd8699047f94b43700084dc3c
-size 3148814
+oid sha256:94e97bb71a4cb316f1df8db8559995a89599d765c1ea3b77e8d9298dde91cee5
+size 2838106

models/word_markov/ang_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "ang",
-  "unique_contexts": 92131,
-  "total_transitions": 614610
+  "unique_contexts": 86720,
+  "total_transitions": 453731
 }

models/word_markov/ang_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9d2a28e8e4c8f3183b5a563700bef62bed372f66fd54af666b1adc514bcaee1a
-size 6460657
+oid sha256:6ed47074ae9df17987a96dedf82859c432c3c11ac930279d48acda20815ee000
+size 5756944

models/word_markov/ang_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "ang",
-  "unique_contexts": 340526,
-  "total_transitions": 609476
+  "unique_contexts": 307843,
+  "total_transitions": 448832
 }

models/word_markov/ang_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4013cfaf6a7e5fb5b05fd16c6fb82a6b07a3986ca2ba084cb4249f829ba48fe6
-size 8659399
+oid sha256:cfebbca94358174706ac1a2151652a5d78ba509f5a59365ca6564d84afbf29d6
+size 7363185

models/word_markov/ang_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "ang",
-  "unique_contexts": 488936,
-  "total_transitions": 604342
+  "unique_contexts": 397541,
+  "total_transitions": 443933
 }

models/word_markov/ang_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:248fdd8a9842bf29f9fb3c6c2994608e1182bf86996d9b007f01871b5c8c6fb8
-size 9896185
+oid sha256:dda31535b2a9881cfcdb06c59a64b5c7407b33461372890fa430f9decfd3a283
+size 8215964

models/word_markov/ang_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "ang",
-  "unique_contexts": 535590,
-  "total_transitions": 599209
+  "unique_contexts": 415179,
+  "total_transitions": 439034
 }

models/word_ngram/ang_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:77969c802023316308f47b559b4654b98f131e91a3e596b4349461e965a4d7a0
-size 171492
+oid sha256:f17dec169cfd4d3cf850b8e92ac17563fdf4214bacccbc07cec77dfe89d84e59
+size 107591

models/word_ngram/ang_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "ang",
-  "unique_ngrams": 12127,
-  "total_ngrams": 614610
+  "unique_ngrams": 7045,
+  "total_ngrams": 453731
 }

models/word_ngram/ang_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ddb4a50ee6552832977f0d2b46d56ea53cfa83bf4d67e4c1cc23feaf8a715561
-size 216617
+oid sha256:7be6fd1d36090e7ad2c61a0009ff0329136a70c8f8e21c354e8ab0d0313ec4a7
+size 109462

models/word_ngram/ang_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "ang",
-  "unique_ngrams": 13432,
-  "total_ngrams": 609476
+  "unique_ngrams": 6002,
+  "total_ngrams": 448832
 }

models/word_ngram/ang_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4c0cefee4edc601a3bdad8fbf3b8f6e175478c16df0bfcb72ac4db6fe7981f8e
-size 402550
+oid sha256:26d6e9ad44f0065bd7d8547dae3860db92ca45a6a819fab269604436a369897a
+size 221303

models/word_ngram/ang_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "ang",
-  "unique_ngrams": 22900,
-  "total_ngrams": 604342
+  "unique_ngrams": 11447,
+  "total_ngrams": 443933
 }