Upload all models and assets for alt (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.265
+    value: 3.681
   - name: best_isotropy
     type: isotropy
-    value: 0.8322
+    value: 0.8352
   - name: vocabulary_size
     type: vocab
-    value: 27823
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # ALT - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,59 +70,49 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.662x | 3.60 | 0.1372% | 996,183 |
-| **16k** | 3.919x | 3.85 | 0.1469% | 930,651 |
-| **32k** | 4.115x | 4.05 | 0.1542% | 886,408 |
-| **64k** | 4.265x 🏆 | 4.19 | 0.1598% | 855,191 |
+| **8k** | 3.483x | 3.48 | 0.3997% | 976,020 |
+| **16k** | 3.681x 🏆 | 3.68 | 0.4223% | 923,645 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Шӱлӱк () — эмдеер тынду, чойлошкон.
-
- Тайантылар 
-
-Категория:Азыранты`
+**Sample 1:** `Тижимеева Галина Ивановна — Кан-Оозы аймактыҥ аймак депутатды. Ӱстӱги Јалаҥый Ба...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁шӱл ӱк ▁() ▁— ▁эмде ер ▁тынду , ▁ч ой ... (+9 more)` | 19 |
-| 16k | `▁шӱл ӱк ▁() ▁— ▁эмдеер ▁тынду , ▁чой ло шкон ... (+6 more)` | 16 |
-| 32k | `▁шӱл ӱк ▁() ▁— ▁эмдеер ▁тынду , ▁чой ло шкон ... (+5 more)` | 15 |
-| 64k | `▁шӱл ӱк ▁() ▁— ▁эмдеер ▁тынду , ▁чойлошкон . ▁тайантылар ... (+3 more)` | 13 |
+| 8k | `▁ти жи ме ева ▁галина ▁ивановна ▁— ▁кан - оозы ... (+12 more)` | 22 |
+| 16k | `▁тижимеева ▁галина ▁ивановна ▁— ▁кан - оозы ▁аймактыҥ ▁аймак ▁депутатды ... (+8 more)` | 18 |
 
-**Sample 2:** `Казахтар кош-агаштыҥ — Алтайыста јадып турган казахтар. Тӧрӧли Казахстан.
-
-Катег...`
+**Sample 2:** `«Кызалаҥду јылдар» (орус. «Трудные годы») — баштапкы алтай тӱӱкилик роман. Автор...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁казах тар ▁кош - агаштыҥ ▁— ▁алтай ы ста ▁јадып ... (+15 more)` | 25 |
-| 16k | `▁казахтар ▁кош - агаштыҥ ▁— ▁алтай ы ста ▁јадып ▁турган ... (+12 more)` | 22 |
-| 32k | `▁казахтар ▁кош - агаштыҥ ▁— ▁алтай ыста ▁јадып ▁турган ▁казахтар ... (+11 more)` | 21 |
-| 64k | `▁казахтар ▁кош - агаштыҥ ▁— ▁алтайыста ▁јадып ▁турган ▁казахтар . ... (+10 more)` | 20 |
+| 8k | `▁« кы за ла ҥ ду ▁јылдар » ▁( орус ... (+19 more)` | 29 |
+| 16k | `▁« кызалаҥду ▁јылдар » ▁( орус . ▁« трудные ▁годы ... (+14 more)` | 24 |
 
-**Sample 3:** `Тура:
- кижи јадар айыл.
- кала (темдектезе: Ойрот-Тура, Јаш-Тура, Том-Тура).`
+**Sample 3:** `Эски Чечкаб (, ) — јурт Россияда Татарстан Республиканыҥ Кайбыч аймагында кирет....`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁тура : ▁кижи ▁јадар ▁айыл . ▁кала ▁( темде кт ... (+14 more)` | 24 |
-| 16k | `▁тура : ▁кижи ▁јадар ▁айыл . ▁кала ▁( темдектезе : ... (+12 more)` | 22 |
-| 32k | `▁тура : ▁кижи ▁јадар ▁айыл . ▁кала ▁( темдектезе : ... (+12 more)` | 22 |
-| 64k | `▁тура : ▁кижи ▁јадар ▁айыл . ▁кала ▁( темдектезе : ... (+12 more)` | 22 |
+| 8k | `▁эски ▁че ч ка б ▁(, ▁) ▁— ▁јурт ▁россияда ... (+12 more)` | 22 |
+| 16k | `▁эски ▁чечкаб ▁(, ▁) ▁— ▁јурт ▁россияда ▁татарстан ▁республиканыҥ ▁кайбыч ... (+7 more)` | 17 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.265x compression
-- **Lowest UNK Rate:** 8k with 0.1372% unknown tokens
+- **Best Compression:** 16k achieves 3.681x compression
+- **Lowest UNK Rate:** 8k with 0.3997% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -129,57 +121,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 5,603 🏆 | 12.45 | 19,162 | 17.9% | 51.7% |
-| **2-gram** | 479 🏆 | 8.91 | 3,376 | 52.1% | 97.1% |
-| **3-gram** | 9,322 | 13.19 | 31,313 | 12.0% | 45.1% |
-| **3-gram** | 3,850 | 11.91 | 26,889 | 18.2% | 59.9% |
-| **4-gram** | 14,040 | 13.78 | 53,203 | 10.9% | 40.5% |
-| **4-gram** | 16,189 | 13.98 | 114,000 | 9.9% | 34.0% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 4,436 | 12.12 | 12,008 | 16.5% | 55.5% |
+| **2-gram** | Subword | 413 🏆 | 8.69 | 2,712 | 55.2% | 98.2% |
+| **3-gram** | Word | 5,478 | 12.42 | 16,272 | 15.6% | 52.1% |
+| **3-gram** | Subword | 3,295 | 11.69 | 22,501 | 19.5% | 62.8% |
+| **4-gram** | Word | 8,026 | 12.97 | 27,756 | 15.3% | 46.2% |
+| **4-gram** | Subword | 14,033 | 13.78 | 96,739 | 10.5% | 35.6% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `республики алтай` | 1,480 |
+| 2 | `ј чык` | 1,391 |
+| 3 | `горно алтайск` | 1,246 |
+| 4 | `алтай республиканыҥ` | 1,222 |
+| 5 | `ј бож` | 1,072 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `јылдыҥ ӱлӱрген айыныҥ` | 755 |
+| 2 | `ӱлӱрген айыныҥ 15` | 730 |
+| 3 | `алтайск ау ра` | 511 |
+| 4 | `горно алтайск ау` | 511 |
+| 5 | `јон јаткан јерлери` | 504 |
+
+**4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `) ,` | 6,036 |
-| 2 | `. —` | 5,108 |
-| 3 | `) .` | 3,265 |
-| 4 | `. )` | 3,242 |
-| 5 | `( )` | 2,922 |
+| 1 | `јылдыҥ ӱлӱрген айыныҥ 15` | 730 |
+| 2 | `горно алтайск ау ра` | 511 |
+| 3 | `болгон јылдыҥ ӱлӱрген айыныҥ` | 367 |
+| 4 | `тоолоорго окылу конвертер датла` | 365 |
+| 5 | `окылу конвертер датла тузаланарга` | 365 |
 
-**3-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `. ) ,` | 2,899 |
-| 2 | `чык . )` | 1,583 |
-| 3 | `. чык .` | 1,572 |
-| 4 | `ј . чык` | 1,391 |
-| 5 | `. бож .` | 1,334 |
+| 1 | `_ к` | 74,491 |
+| 2 | `, _` | 64,716 |
+| 3 | `_ ј` | 55,670 |
+| 4 | `а _` | 55,340 |
+| 5 | `ҥ _` | 54,127 |
 
-**4-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `. чык . )` | 1,570 |
-| 2 | `чык . ) ,` | 1,561 |
-| 3 | `. бож . )` | 1,331 |
-| 4 | `ј . чык .` | 1,308 |
-| 5 | `бож . ) ,` | 1,276 |
+| 1 | `ы ҥ _` | 34,280 |
+| 2 | `д а _` | 17,047 |
+| 3 | `_ — _` | 16,876 |
+| 4 | `н ы ҥ` | 15,865 |
+| 5 | `_ к а` | 15,102 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `н ы ҥ _` | 15,267 |
+| 2 | `д ы ҥ _` | 13,210 |
+| 3 | `_ к ӱ н` | 11,149 |
+| 4 | `а л т а` | 9,638 |
+| 5 | `_ ј ы л` | 9,359 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 479
+- **Best Perplexity:** 2-gram (subword) with 413
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~34% of corpus
+- **Coverage:** Top-1000 patterns cover ~36% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -187,55 +211,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.6271 | 1.544 | 4.13 | 68,485 | 37.3% |
-| **1** | 1.7812 | 3.437 | 20.47 | 296 | 0.0% |
-| **2** | 0.2402 | 1.181 | 1.61 | 283,129 | 76.0% |
-| **2** | 1.3538 | 2.556 | 8.10 | 6,058 | 0.0% |
-| **3** | 0.0999 | 1.072 | 1.20 | 455,274 | 90.0% |
-| **3** | 0.8810 | 1.842 | 4.06 | 49,063 | 11.9% |
-| **4** | 0.0482 🏆 | 1.034 | 1.09 | 544,523 | 95.2% |
-| **4** | 0.5803 🏆 | 1.495 | 2.45 | 199,058 | 42.0% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.7272 | 1.655 | 4.24 | 64,506 | 27.3% |
+| **1** | Subword | 1.6383 | 3.113 | 16.08 | 301 | 0.0% |
+| **2** | Word | 0.1675 | 1.123 | 1.34 | 273,261 | 83.2% |
+| **2** | Subword | 1.3152 | 2.488 | 8.05 | 4,839 | 0.0% |
+| **3** | Word | 0.0551 | 1.039 | 1.10 | 366,294 | 94.5% |
+| **3** | Subword | 0.8839 | 1.845 | 4.16 | 38,940 | 11.6% |
+| **4** | Word | 0.0265 🏆 | 1.019 | 1.05 | 402,354 | 97.4% |
+| **4** | Subword | 0.6047 | 1.521 | 2.55 | 162,075 | 39.5% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `, карасал . труд » — 40 салковой безналичныйла тӧлӧзӧ ( 1354 ) , 2017 ,`
-2. `. 2011 ) — день памяти святой софии ( ортозында јадат . скобканыҥ ичинде кереестиҥ аҥылу`
-3. `- оозы 19 категория : электронный . чаган айдыҥ 21 паспаул16 22 кӱнинде пермский государственный уни...`
+1. `ла эмчиликте фундаментал шиҥжӱлер эдип чотолот чике тоозын айдып салган аш курсактыҥ томский пивоныҥ...`
+2. `ле бийик эмес ортолой кеми 27 ноября года n 107 об образовании муниципальных образований наделении с...`
+3. `алтай республиканыҥ јурт јеезезине статус ла лесопильный ла иш аайынча министр сорокин почвоведение ...`
 
 **Context Size 2:**
 
-1. `) , совет гимнаст , кöп сабазында бу профильный федерал министерстволор кандидатуралар аайынча jöптö...`
-2. `. — 267 с . . мигранттардыҥ тоозы москваныҥ јоныныҥ тоозы астаганыныҥ шылтагы — миграционный отток н...`
-3. `) . јылдыҥ учына јетире 37 кӱн арткан . куран айдыҥ 20 кӱни григориан кӱнтизӱ юлиан кӱнтизӱни`
+1. `республики алтай и верхний иртыш под ред и м краевед ада тӧрӧл учун улу јууныҥ туружаачызы канча`
+2. `ј чык британ черӱниҥ баштапкы јаан чууганга туштаган театрдыҥ сценазында јылда ачылган зимняя вишня ...`
+3. `горно алтайск гагу 267 с ил библиогр с 233 256 isbn текст электронный сууларда азый балыктыҥ кандыйы`
 
 **Context Size 3:**
 
-1. `. ) , орус литературалык критик , кеендикте романтизм деп ууламјыга кирген . 1826 — алхазов , яков`
-2. `чык . ) , совет архитектор ( днепрогэс , театр - студия , студенческий театр « ринхбург »`
-3. `. чык . ) , актёр театр ла киноныҥ . 2016 — алиева фазу гамзатовна ( 1899 ј`
+1. `јылдыҥ ӱлӱрген айыныҥ 15 кӱнинеҥ ала кочкор айдыҥ 18 кӱнинде восход 2 корабльда космонавт а а леонов...`
+2. `ӱлӱрген айыныҥ 15 кӱнинеҥ ала кочкор айдыҥ 3 кӱни григориан кӱнтизӱде јылдыҥ 208 кӱни високосный јыл...`
+3. `алтайск ау ра литературно издательский дом алтын туу јери ле јолдоры јуртта 3 ором казаковтыҥ кыдраш...`
 
 **Context Size 4:**
 
-1. `. чык . ) , армян билимчи - монах , просветитель , армян алфавитти эткен . 1673 — мольер`
-2. `чык . ) , немец тÿÿкичи , литературовед , политик . — отниел чарльз марш ( j . чык`
-3. `. бож . ) , совет тӱӱкичи , археолог , топонимик ле этнограф . 1862 — джон тайлер (`
+1. `јылдыҥ ӱлӱрген айыныҥ 15 кӱнинеҥ ала чаган айдыҥ 17 кӱни юлиан кӱнтизӱ аайынча јылдыҥ ӱлӱрген айыныҥ...`
+2. `горно алтайск ау ра литературно издательский дом алтын туу јери ле јолдоры јурттыҥ текши јери 124 4 ...`
+3. `болгон јылдыҥ ӱлӱрген айыныҥ 15 кӱнине јетире болгон јылдыҥ ӱлӱрген айыныҥ 15 кӱнинеҥ ала кандык айд...`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_эдыҥ_оваралетик`
+2. `акен._ј._бачӱ_10`
+3. `рн_орнфилтӧрораа`
+
+**Context Size 2:**
+
+1. `_ка_мештай,_эдищн`
+2. `,_ӱйматкальдынде_`
+3. `_јылдыҥ_мет_башен`
+
+**Context Size 3:**
+
+1. `ыҥ_бичинентизӱлери`
+2. `да_эмчилевич_ј.бож`
+3. `_—_грицаныҥ_јаҥыс_`
+
+**Context Size 4:**
+
+1. `ныҥ_15_кӱнде_фоновы`
+2. `дыҥ_эдеги_келтейинд`
+3. `_кӱн_айдыҥ_15_айдыҥ`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 95.2% predictability
+- **Best Predictability:** Context-4 (word) with 97.4% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (199,058 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (162,075 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -251,64 +306,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 27,823 |
-| Total Tokens | 605,437 |
-| Mean Frequency | 21.76 |
+| Vocabulary Size | 26,456 |
+| Total Tokens | 567,020 |
+| Mean Frequency | 21.43 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 124.54 |
+| Frequency Std Dev | 124.45 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ла | 6,610 |
-| 2 | алтай | 5,102 |
-| 3 | ле | 4,975 |
-| 4 | с | 3,949 |
-| 5 | деп | 3,904 |
-| 6 | јылда | 3,748 |
-| 7 | айдыҥ | 3,442 |
+| 1 | ла | 6,612 |
+| 2 | ле | 4,973 |
+| 3 | алтай | 4,656 |
+| 4 | деп | 3,921 |
+| 5 | с | 3,896 |
+| 6 | јылда | 3,763 |
+| 7 | айдыҥ | 3,450 |
 | 8 | болгон | 3,231 |
-| 9 | јурт | 3,141 |
-| 10 | и | 3,049 |
+| 9 | км | 3,151 |
+| 10 | јурт | 3,140 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | туузаланат | 2 |
-| 2 | узаныш | 2 |
-| 3 | эрессейде | 2 |
-| 4 | метеметике | 2 |
-| 5 | јеткилдери | 2 |
-| 6 | кӧмпӱтерлик | 2 |
-| 7 | чоотош | 2 |
-| 8 | кошлык | 2 |
-| 9 | програмалары | 2 |
-| 10 | türkiye | 2 |
+| 1 | таскадуларды | 2 |
+| 2 | туузаланат | 2 |
+| 3 | узаныш | 2 |
+| 4 | эрессейде | 2 |
+| 5 | метеметике | 2 |
+| 6 | јеткилдери | 2 |
+| 7 | кӧмпӱтерлик | 2 |
+| 8 | чоотош | 2 |
+| 9 | кошлык | 2 |
+| 10 | програмалары | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.1608 |
-| R² (Goodness of Fit) | 0.984170 |
+| Zipf Coefficient | 1.1623 |
+| R² (Goodness of Fit) | 0.985922 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 26.1% |
-| Top 1,000 | 64.2% |
-| Top 5,000 | 85.3% |
-| Top 10,000 | 91.9% |
+| Top 100 | 27.1% |
+| Top 1,000 | 65.6% |
+| Top 5,000 | 85.8% |
+| Top 10,000 | 92.3% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9842 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 26.1% of corpus
-- **Long Tail:** 17,823 words needed for remaining 8.1% coverage
+- **Zipf Compliance:** R²=0.9859 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 27.1% of corpus
+- **Long Tail:** 16,456 words needed for remaining 7.7% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -321,24 +376,128 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 12,740 | 32 | 4.584 | 1.210 | 0.8322 🏆 |
-| **mono_64d** | 12,740 | 64 | 4.972 | 1.017 | 0.7431 |
-| **mono_128d** | 12,740 | 128 | 5.170 | 0.932 | 0.3673 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8352 🏆 | 0.3587 | N/A | N/A |
+| **mono_64d** | 64 | 0.7406 | 0.3005 | N/A | N/A |
+| **mono_128d** | 128 | 0.3709 | 0.2867 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8322 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 12,740 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8352 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3153. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ко` | корнелия, концертные, коруланар |
+| `-ка` | каа, каталанской, казанды |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-ыҥ` | пятницаныҥ, јазатырдыҥ, экспедициязыныҥ |
+| `-ий` | автобиографический, университетский, кентерберийский |
+| `-кий` | автобиографический, университетский, кентерберийский |
+| `-ский` | автобиографический, университетский, кентерберийский |
+| `-ныҥ` | пятницаныҥ, экспедициязыныҥ, тартканыныҥ |
+| `-иҥ` | унсеттиҥ, билимдериниҥ, эштектиҥ |
+| `-да` | фонында, лида, украинада |
+| `-ый` | сосновый, туберкулезный, маршрутный |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `ский` | 2.13x | 43 contexts | южский, айский, омский |
+| `ында` | 1.56x | 51 contexts | мында, адында, ойында |
+| `ыныҥ` | 1.77x | 30 contexts | зыныҥ, мыныҥ, ажыныҥ |
+| `лтай` | 1.93x | 21 contexts | алтай, шылтай, алтайды |
+| `лгон` | 2.28x | 12 contexts | болгон, толгон, болгоны |
+| `аныҥ` | 1.77x | 23 contexts | кааныҥ, уфаныҥ, оканыҥ |
+| `олго` | 1.78x | 22 contexts | јолго, колго, иолго |
+| `осси` | 2.07x | 13 contexts | россии, россий, россия |
+| `алта` | 1.64x | 26 contexts | алтам, алтан, алтая |
+| `лган` | 1.67x | 24 contexts | алган, салган, алганы |
+| `рген` | 1.53x | 27 contexts | юрген, мерген, тӱрген |
+| `ылда` | 1.69x | 19 contexts | јылда, дылда, тылда |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-ко` | `-ыҥ` | 26 words | комедияныҥ, командазыныҥ |
+| `-ка` | `-ыҥ` | 23 words | каспаныҥ, кардыҥ |
+| `-ко` | `-ныҥ` | 16 words | комедияныҥ, командазыныҥ |
+| `-ка` | `-ий` | 15 words | калий, кавказский |
+| `-ка` | `-ныҥ` | 13 words | каспаныҥ, калаларыныҥ |
+| `-ка` | `-да` | 13 words | картазында, кампанияда |
+| `-ка` | `-кий` | 12 words | кавказский, каледонский |
+| `-ка` | `-ский` | 12 words | кавказский, каледонский |
+| `-ка` | `-ар` | 11 words | кайыҥдар, каналдар |
+| `-ко` | `-ар` | 11 words | космонавттар, коллекциялар |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| молотовский | **`молот-ов-ский`** | 6.0 | `молот` |
+| логиканыҥ | **`логика-ныҥ`** | 4.5 | `логика` |
+| кереестериниҥ | **`кереестерин-иҥ`** | 4.5 | `кереестерин` |
+| тӱӱкизиниҥ | **`тӱӱкизин-иҥ`** | 4.5 | `тӱӱкизин` |
+| швейцарияда | **`швейцария-да`** | 4.5 | `швейцария` |
+| съездиниҥ | **`съездин-иҥ`** | 4.5 | `съездин` |
+| јӱрӱминиҥ | **`јӱрӱмин-иҥ`** | 4.5 | `јӱрӱмин` |
+| политиканыҥ | **`политика-ныҥ`** | 4.5 | `политика` |
+| алексеевский | **`алексеев-ский`** | 4.5 | `алексеев` |
+| субъектов | **`субъект-ов`** | 4.5 | `субъект` |
+| фабриканыҥ | **`фабрика-ныҥ`** | 4.5 | `фабрика` |
+| улаганский | **`улаган-ский`** | 4.5 | `улаган` |
+| бийигиниҥ | **`бийигин-иҥ`** | 4.5 | `бийигин` |
+| черӱлериниҥ | **`черӱлерин-иҥ`** | 4.5 | `черӱлерин` |
+| мьянманыҥ | **`мьянма-ныҥ`** | 4.5 | `мьянма` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language ALT appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -346,11 +505,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.27x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (479) |
-| Markov | **Context-4** | Highest predictability (95.2%) |
+| Tokenizer | **16k BPE** | Best compression (3.68x) |
+| N-gram | **2-gram** | Lowest perplexity (413) |
+| Markov | **Context-4** | Highest predictability (97.4%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -540,7 +700,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -556,7 +717,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 05:34:43*
+*Report Date: 2026-01-03 05:04:55*

models/embeddings/monolingual/alt_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:25f39c60e203d2438da48c866dc0122d86895494620c2edb87b2287604fd5a3c
-size 1037346864
+oid sha256:e396190247b1c989d377e3a31a5ca94405fd3ee9794d9a1f7bafcef3e5cf2c32
+size 1036365432

models/embeddings/monolingual/alt_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 12740
+  "vocab_size": 11800
 }

models/embeddings/monolingual/alt_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3bd06d4b82d8979ba635eb09cfed9e13630bc7fee3c9520ad4c1989e80b32a1f
-size 259562544
+oid sha256:61f263d2302c0b79944fb6dea7a5410f34344972105b8624236c585557cd9b72
+size 259303032

models/embeddings/monolingual/alt_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 12740
+  "vocab_size": 11800
 }

models/embeddings/monolingual/alt_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:db52613d7b4587ee9e0a772f70c33f1c2803a3b6ded1304f0d08823bb6254261
-size 518823984
+oid sha256:0af70072fe6e458bf918c9d11f1a56126a09d6b5cade10dc1cf79494ec3cad2b
+size 518323832

models/embeddings/monolingual/alt_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 12740
+  "vocab_size": 11800
 }

models/subword_markov/alt_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:588f76957b8af3c47a7147669c4d774dc2bfa12d36bb00f3f9b9c83098608e1e
-size 50442
+oid sha256:9835d764d81a373a43d4c69af7b885b80b3e7e6708cce0d6899e9b5ea4672187
+size 43649

models/subword_markov/alt_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "alt",
-  "unique_contexts": 296,
-  "total_transitions": 4709685
+  "unique_contexts": 301,
+  "total_transitions": 4392884
 }

models/subword_markov/alt_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:202f7a9500c7c4a3d4e0c876818f2e61129809a6bcce24d5a91465259febeeab
-size 374335
+oid sha256:8d66028d1955552f26a81718406af2dce47d0f3dedd00f8bb0b84c80b869c131
+size 310925

models/subword_markov/alt_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "alt",
-  "unique_contexts": 6058,
-  "total_transitions": 4708580
+  "unique_contexts": 4839,
+  "total_transitions": 4391785
 }

models/subword_markov/alt_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4dace38255fd3c7ec3e49d088e674c5b23bf4c3c7755cbf939a65bfbd1c8b904
-size 1452315
+oid sha256:93204d2e51859eacc761cb0757c0e6456d7c7bc8f68ee9401b438c2b0f12f236
+size 1232693

models/subword_markov/alt_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "alt",
-  "unique_contexts": 49063,
-  "total_transitions": 4707475
+  "unique_contexts": 38940,
+  "total_transitions": 4390686
 }

models/subword_markov/alt_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9e8cd16eed7b77e9d51bc740beb2ec21f62d42dbe8b7f1b31b5235ad41e9fb70
-size 4344236
+oid sha256:24dc5e341215c2a39c5ae7484dd8bd985f42205d269ff7f3a91b8cc25d862939
+size 3689341

models/subword_markov/alt_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "alt",
-  "unique_contexts": 199058,
-  "total_transitions": 4706370
+  "unique_contexts": 162075,
+  "total_transitions": 4389587
 }

models/subword_ngram/alt_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1aa2c2c1c4c6fe8449d0737457837d21c94791c0ece82c6602065fb5b6c4e42f
-size 46118
+oid sha256:a6cafd90f885b32dd8861ed71808430f5107a59536dc5f4e342a7bdc0fbbba4c
+size 38120

models/subword_ngram/alt_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "alt",
-  "unique_ngrams": 3376,
-  "total_ngrams": 4709685
+  "unique_ngrams": 2712,
+  "total_ngrams": 4392884
 }

models/subword_ngram/alt_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b7f013dc15eeae2ca077839f1e1f4a82ef2dcf39c0b39576ba5a79d0572f7302
-size 354929
+oid sha256:34a836528a3396f9306f4ecc25205690e3dd0d56877599f1efb2e2f194507c84
+size 295825

models/subword_ngram/alt_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "alt",
-  "unique_ngrams": 26889,
-  "total_ngrams": 4708580
+  "unique_ngrams": 22501,
+  "total_ngrams": 4391785
 }

models/subword_ngram/alt_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ab13f71def8ec0b710fcbdef73871e6cd1a0b69db0016a454c131495f4d82bf9
-size 1469638
+oid sha256:884f3c7557c6823455b4677c54106f38a9691634c5e0cfe29fac18815f11c7a2
+size 1241123

models/subword_ngram/alt_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "alt",
-  "unique_ngrams": 114000,
-  "total_ngrams": 4707475
+  "unique_ngrams": 96739,
+  "total_ngrams": 4390686
 }

models/tokenizer/alt_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:980b4133ea1daddc81173676db443164466831f3ca0e9db86af16461414f5a6e
-size 592388
+oid sha256:6ed995fcbde5668b2f32931d416ecfd444547f4fccf04118ff4bf11e3c248ef4
+size 600334

models/tokenizer/alt_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dbc2117cbe37352fbe0fb0a78326dd7a71e661fda8e7fb425ca1f800ac235000
-size 411942
+oid sha256:452a0aec3e7e4b4e17384e2ff0d3b52a51f9cb273b8e8bbc7addbb7f2e51363f
+size 410662

models/vocabulary/alt_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a885262275b1763315e2f0b46972b066080bc341581c9b3933083e4efc882732
-size 527033
+oid sha256:2c316fcfa2120415d93073b97d54878932bd6da42c81696b2da0093488988631
+size 512673

models/vocabulary/alt_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "alt",
-  "vocabulary_size": 27823,
+  "vocabulary_size": 26456,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.1059063546289431,
+    "type_token_ratio": 0.10662391749851259,
     "coverage": {
-      "top_100": 0.2449394999945823,
-      "top_1000": 0.6020404530418725,
-      "top_5000": 0.7997656466465335,
-      "top_10000": 0.8615411287039516
+      "top_100": 0.2540457460170556,
+      "top_1000": 0.6146790507040392,
+      "top_5000": 0.8042837310768824,
+      "top_10000": 0.8649781847028493
     },
-    "hapax_count": 40596,
-    "hapax_ratio": 0.5933439541647788,
-    "total_documents": 1105
+    "hapax_count": 38060,
+    "hapax_ratio": 0.5899311798623598,
+    "total_documents": 1099
   }
 }

models/word_markov/alt_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:072d8e4929be96c8f98e65837e9bb62d7c0ad7b9399dd020e1db736fa445cf7d
-size 3441947
+oid sha256:6ed78756b17891f1d853be9b080f0dbe62a2e12d8ac311c5761369520b78a512
+size 3264406

models/word_markov/alt_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "alt",
-  "unique_contexts": 68485,
-  "total_transitions": 917583
+  "unique_contexts": 64506,
+  "total_transitions": 603981
 }

models/word_markov/alt_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:33891ed476ac087a46973edb85932340c5aa8a095bf6d74e8c7f5aecb2036ac2
-size 8484870
+oid sha256:99cc5861c45675e04db4b52347a14f4dad4cced5b1fde724f96eb05b82e2b557
+size 8258854

models/word_markov/alt_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "alt",
-  "unique_contexts": 283129,
-  "total_transitions": 916478
+  "unique_contexts": 273261,
+  "total_transitions": 602882
 }

models/word_markov/alt_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c8c2326ff49e21f16d6438f45aa87c46782d543383b0a3ca4bb3c2a033c97859
-size 12475746
+oid sha256:761a9c53530fa3d9cba8ad3b211af79457306c88812173b71d35bdd3d1faedac
+size 11105253

models/word_markov/alt_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "alt",
-  "unique_contexts": 455274,
-  "total_transitions": 915373
+  "unique_contexts": 366294,
+  "total_transitions": 601783
 }

models/word_markov/alt_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6bfb1a38815ea112cbe8d8484df23d09293f2f182435f4caf9bfb8cc389d527f
-size 15452158
+oid sha256:4483468f8c0c180566bb8295e44c88b85cd214ea65a3fca1dfdc4c1fd87d8d95
+size 13560943

models/word_markov/alt_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "alt",
-  "unique_contexts": 544523,
-  "total_transitions": 914268
+  "unique_contexts": 402354,
+  "total_transitions": 600684
 }

models/word_ngram/alt_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:06433ca90a29b7fc4f3dd8b44d5a816235fdd8957c630b28fcb469db2389c420
-size 412706
+oid sha256:5493baaabf8c3309446cf79df0d0518dd14efc63de47d09ffd902abb6b59cf0d
+size 301065

models/word_ngram/alt_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "alt",
-  "unique_ngrams": 19162,
-  "total_ngrams": 917583
+  "unique_ngrams": 12008,
+  "total_ngrams": 603981
 }

models/word_ngram/alt_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9396925135c898d843fdd5b52c715879fd9baa7387ffb5e60715bd57deafc479
-size 733761
+oid sha256:8a74da8a3a5caaddba92e392597b92357d19a6fc6d4c6d2f0b5c8824ced3fa20
+size 470151

models/word_ngram/alt_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "alt",
-  "unique_ngrams": 31313,
-  "total_ngrams": 916478
+  "unique_ngrams": 16272,
+  "total_ngrams": 602882
 }

models/word_ngram/alt_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:31e26d5fad3df318aea57c7bf608a001e4a1f4d6fea056b2a8dd62fd899d62e1
-size 1354732
+oid sha256:e79b9594793378b010dc641e26308ed766fabaa853ad5069f4bcdf76467d0a3d
+size 898038

models/word_ngram/alt_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "alt",
-  "unique_ngrams": 53203,
-  "total_ngrams": 915373
+  "unique_ngrams": 27756,
+  "total_ngrams": 601783
 }