Upload all models and assets for ady (latest)

.gitattributes

@@ -40,3 +40,4 @@ visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
 visualizations/ngram_coverage.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: ady
-language_name: ADY
+language_name: Adyghe
 language_family: caucasian_northwest
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-caucasian_northwest
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.231
+    value: 4.197
   - name: best_isotropy
     type: isotropy
-    value: 0.4850
+    value: 0.4929
   - name: vocabulary_size
     type: vocab
     value: 0
 generated: 2026-01-03
 ---
 
-# ADY - Wikilangs Models
+# Adyghe - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **ADY** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Adyghe** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -60,7 +70,7 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
 - [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
@@ -80,43 +90,43 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.442x | 3.45 | 0.1638% | 134,283 |
-| **16k** | 3.798x | 3.80 | 0.1808% | 121,676 |
-| **32k** | 4.231x 🏆 | 4.24 | 0.2014% | 109,215 |
+| **8k** | 3.406x | 3.41 | 0.1685% | 137,125 |
+| **16k** | 3.759x | 3.76 | 0.1859% | 124,248 |
+| **32k** | 4.197x 🏆 | 4.20 | 0.2076% | 111,273 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Киев — Украинэ и Нэбгырэ млн 2.9 фэдиз дэс. Къалэм и – Кличко Виталий Владимир ы...`
+**Sample 1:** `(Пынарбашы), Къайсэр къалэм и район. Адыгэхэ нахь бэрэу мы лъэныком щыӏпсэу.`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁киев ▁— ▁украинэ ▁и ▁нэбгырэ ▁млн ▁ 2 . 9 ... (+34 more)` | 44 |
-| 16k | `▁киев ▁— ▁украинэ ▁и ▁нэбгырэ ▁млн ▁ 2 . 9 ... (+25 more)` | 35 |
-| 32k | `▁киев ▁— ▁украинэ ▁и ▁нэбгырэ ▁млн ▁ 2 . 9 ... (+17 more)` | 27 |
+| 8k | `▁( пы н арбашы ), ▁къайсэр ▁къалэм ▁и ▁район . ... (+10 more)` | 20 |
+| 16k | `▁( пы н арбашы ), ▁къайсэр ▁къалэм ▁и ▁район . ... (+10 more)` | 20 |
+| 32k | `▁( пынарбашы ), ▁къайсэр ▁къалэм ▁и ▁район . ▁адыгэхэ ▁нахь ... (+5 more)` | 15 |
 
-**Sample 2:** `Пётровице – Полшэм и Нэбгырэ 352 фэдиз Bank Danych Lokalnych. ТехьэпӀэхэр Пётров...`
+**Sample 2:** `Орэдус — орэдхэр зыусырэр. пае классикэ орэд е мэкъамэ ягугъу – композитор нахьы...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁пёт ров ице ▁– ▁пол шэм ▁и ▁нэбгырэ ▁ 3 ... (+27 more)` | 37 |
-| 16k | `▁пётровице ▁– ▁полшэм ▁и ▁нэбгырэ ▁ 3 5 2 ▁фэдиз ... (+16 more)` | 26 |
-| 32k | `▁пётровице ▁– ▁полшэм ▁и ▁нэбгырэ ▁ 3 5 2 ▁фэдиз ... (+13 more)` | 23 |
+| 8k | `▁орэдус ▁— ▁орэдхэр ▁зы ус ырэр . ▁пае ▁класс икэ ... (+18 more)` | 28 |
+| 16k | `▁орэдус ▁— ▁орэдхэр ▁зы ус ырэр . ▁пае ▁класс икэ ... (+15 more)` | 25 |
+| 32k | `▁орэдус ▁— ▁орэдхэр ▁зыусырэр . ▁пае ▁классикэ ▁орэд ▁е ▁мэкъамэ ... (+10 more)` | 20 |
 
-**Sample 3:** `пшъэшъэ – пшъашъэхэм алъыплъэу, якъэшъон, языгъэпсэфын гъунэ алъызыфэу джэгур пш...`
+**Sample 3:** `Эбрар Каракурт 17 Щылэмаз Балыкесирым къэхъугъ, Тыркуе Волэйболым и джэгуакӀу,Ты...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁пшъэшъэ ▁– ▁пшъашъэ хэм ▁алъ ып лъэу , ▁я къэ ... (+17 more)` | 27 |
-| 16k | `▁пшъэшъэ ▁– ▁пшъашъэ хэм ▁алъыплъэу , ▁я къэ шъо н ... (+10 more)` | 20 |
-| 32k | `▁пшъэшъэ ▁– ▁пшъашъэхэм ▁алъыплъэу , ▁якъэшъон , ▁языгъэпсэфын ▁гъунэ ▁алъызыфэу ... (+3 more)` | 13 |
+| 8k | `▁э б рар ▁кара к урт ▁ 1 7 ▁щы ... (+21 more)` | 31 |
+| 16k | `▁эбрар ▁карак урт ▁ 1 7 ▁щы л эм аз ... (+15 more)` | 25 |
+| 32k | `▁эбрар ▁каракурт ▁ 1 7 ▁щылэмаз ▁балыкесирым ▁къэхъугъ , ▁тыркуе ... (+10 more)` | 20 |
 
 
 ### Key Findings
 
-- **Best Compression:** 32k achieves 4.231x compression
-- **Lowest UNK Rate:** 8k with 0.1638% unknown tokens
+- **Best Compression:** 32k achieves 4.197x compression
+- **Lowest UNK Rate:** 8k with 0.1685% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -133,12 +143,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
 |--------|---------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | Word | 418 | 8.71 | 593 | 45.3% | 100.0% |
-| **2-gram** | Subword | 399 🏆 | 8.64 | 2,072 | 57.0% | 97.4% |
-| **3-gram** | Word | 706 | 9.46 | 922 | 33.9% | 100.0% |
-| **3-gram** | Subword | 2,788 | 11.44 | 11,614 | 24.5% | 65.1% |
-| **4-gram** | Word | 2,848 | 11.48 | 3,264 | 13.1% | 44.3% |
-| **4-gram** | Subword | 10,651 | 13.38 | 35,316 | 12.4% | 39.6% |
+| **2-gram** | Word | 453 | 8.82 | 625 | 42.6% | 100.0% |
+| **2-gram** | Subword | 407 🏆 | 8.67 | 2,126 | 56.6% | 97.3% |
+| **3-gram** | Word | 759 | 9.57 | 977 | 31.6% | 100.0% |
+| **3-gram** | Subword | 2,854 | 11.48 | 11,856 | 24.3% | 64.6% |
+| **4-gram** | Word | 2,909 | 11.51 | 3,378 | 13.2% | 45.0% |
+| **4-gram** | Subword | 10,911 | 13.41 | 36,062 | 12.4% | 39.2% |
+| **5-gram** | Word | 2,658 | 11.38 | 2,950 | 12.2% | 45.6% |
+| **5-gram** | Subword | 21,199 | 14.37 | 52,393 | 8.2% | 28.3% |
 
 ### Top 5 N-grams by Size
 
@@ -146,11 +158,11 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `нэбгырэ млн` | 169 |
+| 1 | `нэбгырэ млн` | 168 |
 | 2 | `къехъу щэпсэу` | 104 |
-| 3 | `картым тетэу` | 100 |
-| 4 | `м къехъу` | 89 |
-| 5 | `дло м` | 87 |
+| 3 | `м къехъу` | 89 |
+| 4 | `дло м` | 87 |
+| 5 | `адыгэ республикэм` | 80 |
 
 **3-grams (Word):**
 
@@ -158,7 +170,7 @@ Below are sample sentences tokenized with each vocabulary size:
 |------|--------|-------|
 | 1 | `м къехъу щэпсэу` | 76 |
 | 2 | `къехъу щэпсэу хэгэгум` | 70 |
-| 3 | `адыгэ республикэм и` | 48 |
+| 3 | `адыгэ республикэм и` | 46 |
 | 4 | `дло м хахьэ` | 44 |
 | 5 | `м хахьэ хэгъэгу` | 39 |
 
@@ -172,42 +184,62 @@ Below are sample sentences tokenized with each vocabulary size:
 | 4 | `америкэм ит къэралыгъу къэлэ` | 19 |
 | 5 | `азием ит къэралыгъу къэлэ` | 18 |
 
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `км гъогу щыӏ къуаджэм ис` | 17 |
+| 2 | `гъогу щыӏ къуаджэм ис цӏыфхэр` | 17 |
+| 3 | `щыӏ къуаджэм ис цӏыфхэр илъэсхэм` | 17 |
+| 4 | `къуаджэм ис цӏыфхэр илъэсхэм тетэу` | 17 |
+| 5 | `ис цӏыфхэр илъэсхэм тетэу къуаджэм` | 17 |
+
 **2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `г ъ` | 9,349 |
-| 2 | `ъ э` | 9,255 |
-| 3 | `э _` | 8,719 |
-| 4 | `м _` | 7,823 |
-| 5 | `э р` | 6,778 |
+| 1 | `г ъ` | 9,326 |
+| 2 | `ъ э` | 9,249 |
+| 3 | `э _` | 8,792 |
+| 4 | `м _` | 7,740 |
+| 5 | `э р` | 6,822 |
 
 **3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `г ъ э` | 4,967 |
-| 2 | `_ к ъ` | 4,149 |
-| 3 | `э м _` | 3,582 |
-| 4 | `ы г ъ` | 3,357 |
-| 5 | `э р _` | 3,016 |
+| 1 | `г ъ э` | 4,961 |
+| 2 | `_ к ъ` | 4,140 |
+| 3 | `э м _` | 3,581 |
+| 4 | `ы г ъ` | 3,362 |
+| 5 | `э р _` | 3,020 |
 
 **4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ы г ъ э` | 1,903 |
-| 2 | `х э р _` | 1,450 |
-| 3 | `а г ъ э` | 1,351 |
-| 4 | `х э м _` | 1,305 |
+| 1 | `ы г ъ э` | 1,902 |
+| 2 | `х э р _` | 1,448 |
+| 3 | `а г �� э` | 1,342 |
+| 4 | `х э м _` | 1,303 |
 | 5 | `_ к ъ э` | 1,289 |
 
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ а д ы г` | 1,062 |
+| 2 | `а д ы г э` | 978 |
+| 3 | `_ и л ъ э` | 670 |
+| 4 | `д ы г э _` | 651 |
+| 5 | `и л ъ э с` | 627 |
+
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram (subword) with 399
+- **Best Perplexity:** 2-gram (subword) with 407
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~40% of corpus
+- **Coverage:** Top-1000 patterns cover ~28% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -223,14 +255,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
 |---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.4365 | 1.353 | 2.10 | 22,306 | 56.3% |
-| **1** | Subword | 1.4909 | 2.811 | 10.56 | 410 | 0.0% |
-| **2** | Word | 0.0764 | 1.054 | 1.12 | 46,305 | 92.4% |
-| **2** | Subword | 1.1481 | 2.216 | 5.61 | 4,325 | 0.0% |
-| **3** | Word | 0.0240 | 1.017 | 1.03 | 51,243 | 97.6% |
-| **3** | Subword | 0.7541 | 1.687 | 2.97 | 24,260 | 24.6% |
-| **4** | Word | 0.0128 🏆 | 1.009 | 1.02 | 52,387 | 98.7% |
-| **4** | Subword | 0.4304 | 1.348 | 1.86 | 72,077 | 57.0% |
+| **1** | Word | 0.4341 | 1.351 | 2.09 | 22,655 | 56.6% |
+| **1** | Subword | 1.4193 | 2.674 | 10.02 | 450 | 0.0% |
+| **2** | Word | 0.0766 | 1.055 | 1.12 | 46,851 | 92.3% |
+| **2** | Subword | 1.1376 | 2.200 | 5.57 | 4,503 | 0.0% |
+| **3** | Word | 0.0248 | 1.017 | 1.04 | 51,794 | 97.5% |
+| **3** | Subword | 0.7466 | 1.678 | 2.95 | 25,044 | 25.3% |
+| **4** | Word | 0.0130 🏆 | 1.009 | 1.02 | 53,002 | 98.7% |
+| **4** | Subword | 0.4264 | 1.344 | 1.85 | 73,859 | 57.4% |
 
 ### Generated Text Samples (Word-based)
 
@@ -238,27 +270,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `и 99 86 561 3 щагъэпсыгъ ахэр къуаджэ адыгэ алфавитэу зэхигъэуцуа гъэм ди лъэхъэнэм хухуабжэ зэкӏэпс...`
-2. `адыгэ автоном хэку британиешхо пачъыхьыгъо темыр скотландием уэлсым пэгъунэгъу къэлэ нейпьидо мьянмэ...`
-3. `м къехъу щэпсэу къэралыгъом икъэлэ хъугъэ шӏагъэхэм ащ ипшъэкӏэрэ лъэныкъо щыӏ сауд арабие бахрейн к...`
+1. `и нэхъышъхьэ лэжьыгъэм статистикэм теухуауэ интервью къэрал хассан аль джадид зэхащагъ илъэсым тэуфи...`
+2. `адыгэ литературэм ихьаси лъэшэу фэӏэзагъэх синдикэр къэралыгъоу тунисым и 20 м нэс тхыдэр нэхь мэхъу...`
+3. `м хахьэ ыужрэр алтай бзэунагъом хахьэ хэгъэгу тхьаматэр инь юн`
 
 **Context Size 2:**
 
-1. `нэбгырэ млн 13 м паплъэхэзэ пчэдыжьым сыхьатыр 11 м машинэм дахэхэр къежьэхи исполкомым а ябыракъэу ...`
-2. `къехъу щэпсэу хэгэгум 718 км китаибзэ англыбзэ малаибзэ тамилыбзэ дло м ез м и кандидат хэгъэгу фили...`
-3. `картым тетэу экуадор къыблэ америкэм ит къэралыгъу къэлэ братиславэ нэбгырэ млн 8 8 фэдиз щэпсэу хэг...`
+1. `нэбгырэ млн 7 къехъу щэпсэу хэгэгум 51 100 км арапыбзэ дло м еуро зэкъотыныгъэм ахахьэ хэгъэгу колин...`
+2. `къехъу щэпсэу хэгэгум 718 км китаибзэ англыбзэ малаибзэ тамилыбзэ дло м хахьэ хэгъэгу пачъыхьэу абду...`
+3. `м къехъу щэпсэу хэгэгум 267 667 км францыбзэ къэрал фор эссозимна гнассингбе хэгъэгу тхьаматэр даниэ...`
 
 **Context Size 3:**
 
-1. `м къехъу щэпсэу хэгэгум 23 200 км арапыбзэ францыбзэ къэрал мохаммед ульд абдель азиз гуадзэр яхья у...`
-2. `къехъу щэпсэу хэгэгум чӏырэу иӏэр 17 820 км бзэшъхьаӏэр арапыбз дло м хахьэ хэгъэгу гувернатор петер...`
-3. `адыгэ республикэм и теуцуожь къедзыгъор къыгот краснодар краим и къэлэ нэбгырэ млн 1 3 фэдиз дэс къэ...`
+1. `м къехъу щэпсэу хэгэгум 765 км арапыбз арап къэралмэ анахь баймэ а��ыщ нефтыр лъэшдэдэу дло м хахьэ х...`
+2. `къехъу щэпсэу хэгэгум 147 570 км бенгалыбзэ дло м хахьэ хэгъэгу алмазбек атамбаев къэрал тхьэматэр т...`
+3. `адыгэ республикэм и шэуджэн къедзыгъом и къоджэ км 42 мыекъуапэ пэчыжь хэкум къинэжьыгъэ абдзэхэ къо...`
 
 **Context Size 4:**
 
-1. `м къехъу щэпсэу хэгэгум 82 880 км арапыбз дло м хахьэ хэгъэгу лӏышъхьэр сабах я 6 аль ахьмэд аль`
-2. `дло м хахьэ хэгъэгу хамед бен исса аль халифа хэгъэгу тхьаматэр халифа бен салман аль халифа географ...`
-3. `еуропэм хэт къэралыгъу къэлэ софие нэбгырэ млн 7 м къехъу щэпсэу хэгэгум 198 500 км кыргызыбзэрэ уры...`
+1. `м къехъу щэпсэу хэгэгум чӏырэу иӏэр 17 820 км бзэшъхьаӏэр арапыбз дло м хахьэ хэгъэгу хассанал болки...`
+2. `дло м хахьэ хэгъэгу тейн сейн географие азием и гъунэгъухэр урысые казахстан кыргызстан монголие ишъ...`
+3. `еуропэм хэт къэралыгъу къэлэ загреб нэбгырэ млн 4 м къехъу щэпсэу я 116 хэгэгум 49 035 км я 129`
 
 
 ### Generated Text Samples (Subword-based)
@@ -267,34 +299,34 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_хъ._зэтм_ьэлыбе`
-2. `эмыны_ар,_е,_щӏэ`
-3. `ыхьэ_хъайстоджау`
+1. `_пчӏэр_гокъэме_д`
+2. `эр_цӏыӏэзэзынэ_я`
+3. `ыем_щщэра,_фадж.`
 
 **Context Size 2:**
 
-1. `гъ._грайономафедо`
-2. `ъэгугъэзетэм_зэра`
-3. `э_сырикэмрэ_хылъэ`
+1. `гъэмьяхэр_арт_пре`
+2. `ъэу_дэхъ_зышӏэным`
+3. `э_гъэ_ратымэ_лъхь`
 
 **Context Size 3:**
 
-1. `гъэлэ_зинскэ_хы_фэ`
-2. `_къикӏым_научнэр_б`
-3. `эм_гу_рэ_исэугъэзе`
+1. `гъэзекӏожьыдзэнэжы`
+2. `_къагъэхьыбэмэ,_гу`
+3. `эм_къурэтхъум_↔_ищ`
 
 **Context Size 4:**
 
-1. `ыгъэ_поясхэм_сурэтх`
-2. `хэр_зыдэщыӏу_ыкӏи_и`
-3. `агъэцэкӏэ_иӏэм_итіо`
+1. `ыгъэ_хасэмрэ_млн_89`
+2. `хэр_къолэжъхэр_тхыг`
+3. `агъэкӏотэщтыр_ары._`
 
 
 ### Key Findings
 
 - **Best Predictability:** Context-4 (word) with 98.7% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (72,077 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (73,859 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -310,64 +342,64 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 7,032 |
-| Total Tokens | 44,503 |
-| Mean Frequency | 6.33 |
+| Vocabulary Size | 7,120 |
+| Total Tokens | 45,308 |
+| Mean Frequency | 6.36 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 22.13 |
+| Frequency Std Dev | 22.08 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | и | 1,013 |
-| 2 | адыгэ | 666 |
-| 3 | м | 489 |
-| 4 | илъэсым | 398 |
+| 1 | и | 999 |
+| 2 | адыгэ | 660 |
+| 3 | м | 508 |
+| 4 | илъэсым | 406 |
 | 5 | ащ | 391 |
-| 6 | я | 309 |
-| 7 | ары | 271 |
-| 8 | нэбгырэ | 247 |
-| 9 | а | 243 |
-| 10 | ыкӏи | 211 |
+| 6 | я | 320 |
+| 7 | ары | 274 |
+| 8 | а | 257 |
+| 9 | нэбгырэ | 250 |
+| 10 | е | 223 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | рсфср | 2 |
-| 2 | серийнэ | 2 |
-| 3 | ныбжьыкӏэхэри | 2 |
-| 4 | зэратебэнагъэр | 2 |
-| 5 | хираганэ | 2 |
-| 6 | катаканэ | 2 |
-| 7 | сербыбзэм | 2 |
-| 8 | къыздикӏыгъэр | 2 |
-| 9 | тыванбзэ | 2 |
-| 10 | къызыл | 2 |
+| 1 | muzea | 2 |
+| 2 | britishpedia | 2 |
+| 3 | encyklopedia | 2 |
+| 4 | osobistości | 2 |
+| 5 | rzeczypospolitej | 2 |
+| 6 | polskiej | 2 |
+| 7 | bph | 2 |
+| 8 | british | 2 |
+| 9 | publishing | 2 |
+| 10 | ltd | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.7821 |
-| R² (Goodness of Fit) | 0.977951 |
+| Zipf Coefficient | 0.7863 |
+| R² (Goodness of Fit) | 0.977814 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 29.3% |
-| Top 1,000 | 60.6% |
-| Top 5,000 | 90.9% |
+| Top 100 | 28.8% |
+| Top 1,000 | 60.5% |
+| Top 5,000 | 90.6% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9780 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 29.3% of corpus
-- **Long Tail:** -2,968 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9778 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 28.8% of corpus
+- **Long Tail:** -2,880 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -383,37 +415,40 @@ Below are text samples generated from each subword-based Markov chain model:
 
 ### 5.1 Cross-Lingual Alignment
 
-> *Note: Multilingual alignment visualization not available for this language.*
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
 
 
 ### 5.2 Model Comparison
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.4850 🏆 | 0.4355 | N/A | N/A |
-| **mono_64d** | 64 | 0.2076 | 0.3984 | N/A | N/A |
-| **mono_128d** | 128 | 0.0353 | 0.4111 | N/A | N/A |
+| **mono_32d** | 32 | 0.4929 🏆 | 0.4238 | N/A | N/A |
+| **mono_64d** | 64 | 0.2008 | 0.4008 | N/A | N/A |
+| **mono_128d** | 128 | 0.0373 | 0.3931 | N/A | N/A |
+| **aligned_32d** | 32 | 0.4929 | 0.4303 | 0.0632 | 0.4080 |
+| **aligned_64d** | 64 | 0.2008 | 0.3933 | 0.2011 | 0.7586 |
+| **aligned_128d** | 128 | 0.0373 | 0.3923 | 0.2701 | 0.8046 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.4850 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.4150. Lower values indicate better semantic separation.
-- **Alignment Quality:** No aligned models evaluated in this run.
+- **Best Isotropy:** mono_32d with 0.4929 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.4056. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 27.0% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
 ## 6.  Morphological Analysis (Experimental)
 
-> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
-
 This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
 
 ### 6.1 Productivity & Complexity
 
 | Metric | Value | Interpretation | Recommendation |
 |--------|-------|----------------|----------------|
-| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
-| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **0.610** | High formulaic/idiomatic content | - |
 
 ### 6.2 Affix Inventory (Productive Units)
 
@@ -422,21 +457,20 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Prefixes
 | Prefix | Examples |
 |--------|----------|
-| `-къ` | къатыгъ, къуаншэмрэ, къунетрэр |
-| `-зэ` | зэрэзэхагъэуцорэр, зэреджэхэрэр, зэкӏэм |
-| `-къы` | къыхэуутыным, къычӏэкӏы, къырафыгъэ |
+| `-къ` | къахэщых, къэштэжь, къыхагъэщэу |
+| `-зэ` | зэмыпэсырэм, зэрагъэзэкӏуагъэу, зэфэшъхьафыбэмэ |
 
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-э` | зыхъожьыгъэ, инароднэ, шапсыгъэбзэкӏэ |
-| `-р` | зэрэзэхагъэуцорэр, зэреджэхэрэр, къунетрэр |
-| `-м` | къутамэхэм, къушъхьэм, ӏушъом |
-| `-эр` | зэрэзэхагъэуцорэр, зэреджэхэрэр, къунетрэр |
-| `-эм` | къутамэхэм, къушъхьэм, зэкӏэм |
-| `-эу` | псынкіэу, тамыгъэу, доцентэу |
-| `-хэр` | урымхэр, кӏэмгуехэр, зэлъаштагъэхэр |
-| `-рэ` | къуаншэмрэ, швециемрэ, тыркурэ |
+| `-э` | инджылыбзэ, шъхьэгуащэ, ыкурэ |
+| `-р` | хъулъфыгъэхэр, егъэблэгъэныр, усэхэр |
+| `-м` | зэмыпэсырэм, бысымым, м |
+| `-эр` | хъулъфыгъэхэр, усэхэр, благъэр |
+| `-эм` | зэмыпэсырэм, къутамэм, пхъэм |
+| `-эу` | бэрэу, зэрагъэзэкӏуагъэу, къыхагъэщэу |
+| `-хэр` | хъулъфыгъэхэр, усэхэр, ӏутыхэр |
+| `-рэ` | ыкурэ, цӏэмрэ, чэщрэ |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -444,18 +478,18 @@ Bound stems are high-frequency subword units that are semantically cohesive but
 
 | Stem | Cohesion | Substitutability | Examples |
 |------|----------|------------------|----------|
-| `тыгъ` | 1.78x | 28 contexts | тыгъу, итыгъ, тыгъэ |
-| `эпкъ` | 1.81x | 25 contexts | нэпкъ, нэпкъы, инэпкъ |
-| `ъагъ` | 2.15x | 14 contexts | пчъагъ, лъагъо, пчъагъэ |
-| `агъэ` | 1.54x | 41 contexts | тхагъэ, благъэ, багъэх |
-| `къуа` | 2.14x | 10 contexts | къуае, къуадж, къуажэ |
-| `дыгэ` | 1.88x | 14 contexts | адыгэ, адыгэш, адыгэм |
-| `ъхьэ` | 1.77x | 16 contexts | шъхьэ, пшъхьэ, шъхьэм |
-| `эхэр` | 1.59x | 21 contexts | бэхэр, дзэхэр, бзэхэр |
-| `шъхь` | 1.49x | 24 contexts | шъхьэ, пшъхьэ, ышъхьа |
-| `ыгъо` | 1.59x | 19 contexts | мыгъо, цыгъо, пщыгъо |
-| `псэу` | 1.57x | 19 contexts | нэпсэу, сыпсэу, щэпсэу |
-| `эхэм` | 1.55x | 17 contexts | бзэхэм, блэхэм, цӏэхэм |
+| `тыгъ` | 1.78x | 28 contexts | тыгъу, тыгъэ, итыгъ |
+| `ъагъ` | 2.17x | 14 contexts | пчъагъ, лъагъо, тхъагъо |
+| `эпкъ` | 1.76x | 25 contexts | нэпкъ, нэпкъы, инэпкъ |
+| `агъэ` | 1.55x | 39 contexts | тхагъэ, багъэх, благъэ |
+| `къуа` | 2.17x | 10 contexts | къуае, къуадж, къуажэ |
+| `дыгэ` | 1.90x | 14 contexts | адыгэ, адыгэу, адыгэм |
+| `псэу` | 1.64x | 20 contexts | упсэу, нэпсэу, щэпсэу |
+| `эхэр` | 1.61x | 20 contexts | бэхэр, усэхэр, унэхэр |
+| `ъхьэ` | 1.72x | 16 contexts | шъхьэ, ишъхьэ, шъхьэм |
+| `ыгъо` | 1.62x | 19 contexts | цыгъо, мыгъо, мыгъом |
+| `шъхь` | 1.51x | 23 contexts | шъхьэ, шъхьаф, ишъхьэ |
+| `гъэх` | 1.67x | 14 contexts | багъэх, тхыгъэх, ежагъэх |
 
 ### 6.4 Affix Compatibility (Co-occurrence)
 
@@ -463,16 +497,16 @@ This table shows which prefixes and suffixes most frequently co-occur on the sam
 
 | Prefix | Suffix | Frequency | Examples |
 |--------|--------|-----------|----------|
-| `-къ` | `-э` | 96 words | къэттыгъэ, къэлэшхомэ |
-| `-къ` | `-р` | 64 words | къэплъаныр, къэплъанхэр |
-| `-къ` | `-м` | 56 words | къызэхуэсахэм, къытыралъхьагъэм |
-| `-къ` | `-эр` | 52 words | къэплъанхэр, къэралыгъохэр |
-| `-зэ` | `-р` | 42 words | зэраукӏырэр, зэпэуцужьыныгъэхэр |
-| `-зэ` | `-м` | 41 words | зэрэратырэм, зэратебанэщтыгъэхэм |
-| `-къ` | `-эм` | 36 words | къызэхуэсахэм, къытыралъхьагъэм |
-| `-къ` | `-эу` | 34 words | къехыжьэу, къыдамылъытагъэу |
-| `-зэ` | `-эр` | 34 words | зэраукӏырэр, зэпэуцужьыныгъэхэр |
-| `-зэ` | `-э` | 31 words | зэхэзгъэуцуагъэ, зэхищэгъэгъэ |
+| `-къ` | `-э` | 94 words | къызэриӏорэмкӏэ, къыгъэпсыщтыгъэ |
+| `-къ` | `-р` | 64 words | къалъхуахэр, къызэдыхэфэныр |
+| `-къ` | `-м` | 56 words | къэралхэм, къожъхэм |
+| `-къ` | `-эр` | 52 words | къалъхуахэр, къалэр |
+| `-зэ` | `-р` | 43 words | зэрэзэтекӏыхэрэр, зэрыхъур |
+| `-зэ` | `-м` | 41 words | зэрэхъурэм, зэкъотыныгъэм |
+| `-къ` | `-эм` | 36 words | къэралхэм, къожъхэм |
+| `-зэ` | `-эр` | 34 words | зэрэзэтекӏыхэрэр, зэриукъорэр |
+| `-къ` | `-эу` | 33 words | къыхахыгъэу, къыдыхэлъытагъэу |
+| `-зэ` | `-э` | 31 words | зэралэжьырэ, зэгъусэмэ |
 
 ### 6.5 Recursive Morpheme Segmentation
 
@@ -480,26 +514,28 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 
 | Word | Suggested Split | Confidence | Stem |
 |------|-----------------|------------|------|
-| бзылъфыгъэмрэ | **`бзылъфыгъ-эм-рэ`** | 6.0 | `бзылъфыгъ` |
-| сарматхэмрэ | **`сармат-хэм-рэ`** | 6.0 | `сармат` |
-| меотхэмрэ | **`меот-хэм-рэ`** | 6.0 | `меот` |
-| адыгабзэмрэ | **`адыгабз-эм-рэ`** | 6.0 | `адыгабз` |
-| макъэхэмрэ | **`макъэ-хэм-рэ`** | 6.0 | `макъэ` |
-| республикэмрэ | **`республик-эм-рэ`** | 6.0 | `республик` |
-| зэхэлъхэм | **`зэ-хэлъ-хэм`** | 6.0 | `хэлъ` |
-| арапыбзэрэ | **`арапыбз��-рэ`** | 4.5 | `арапыбзэ` |
-| ягъунэгъухэр | **`ягъунэгъу-хэр`** | 4.5 | `ягъунэгъу` |
-| унагъохэр | **`унагъо-хэр`** | 4.5 | `унагъо` |
-| жьыбгъэхэр | **`жьыбгъэ-хэр`** | 4.5 | `жьыбгъэ` |
-| елъытыгъэу | **`елъытыгъ-эу`** | 4.5 | `елъытыгъ` |
-| чӏыпӏэхэм | **`чӏыпӏэ-хэм`** | 4.5 | `чӏыпӏэ` |
-| зыщыпсэухэрэр | **`зыщыпс-эу-хэр-эр`** | 4.5 | `зыщыпс` |
-| журналхэм | **`журнал-хэм`** | 4.5 | `журнал` |
+| щыпсэухэрэр | **`щыпс-эу-хэр-эр`** | 7.5 | `щыпс` |
+| литературэмрэ | **`литератур-эм-рэ`** | 6.0 | `литератур` |
+| мыхъунхэр | **`мыхъун-хэр`** | 4.5 | `мыхъун` |
+| джуртыбзэрэ | **`джуртыбзэ-рэ`** | 4.5 | `джуртыбзэ` |
+| тхьаматэр | **`тхьамат-эр`** | 4.5 | `тхьамат` |
+| фэхъугъэм | **`фэхъугъ-эм`** | 4.5 | `фэхъугъ` |
+| игъунэгъухэр | **`игъунэгъу-хэр`** | 4.5 | `игъунэгъу` |
+| нэмыкӏхэр | **`нэмыкӏ-хэр`** | 4.5 | `нэмыкӏ` |
+| зэкъотыныгъэм | **`зэ-къ-отыныгъ-эм`** | 4.5 | `отыныгъ` |
+| ипрезидентэу | **`ипрезидент-эу`** | 4.5 | `ипрезидент` |
+| литературэр | **`литератур-эр`** | 4.5 | `литератур` |
+| хъыбархэм | **`хъыбар-хэм`** | 4.5 | `хъыбар` |
+| культурэм | **`культур-эм`** | 4.5 | `культур` |
+| къыхафыгъэхэр | **`къ-ыхафыг-ъэ-хэр`** | 4.5 | `ыхафыг` |
+| зэрэгущаӏэхэрэр | **`зэ-рэгущаӏэ-хэр-эр`** | 4.5 | `рэгущаӏэ` |
 
 ### 6.6 Linguistic Interpretation
 
 > **Automated Insight:**
-The language ADY appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+The language Adyghe shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
 
 ---
 ## 7. Summary & Recommendations
@@ -510,8 +546,8 @@ The language ADY appears to be more isolating or has a highly fixed vocabulary.
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.23x) |
-| N-gram | **2-gram** | Lowest perplexity (399) |
+| Tokenizer | **32k BPE** | Best compression (4.20x) |
+| N-gram | **2-gram** | Lowest perplexity (407) |
 | Markov | **Context-4** | Highest predictability (98.7%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
@@ -726,4 +762,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 12:36:55*
+*Report Date: 2026-01-03 14:02:39*

models/embeddings/aligned/ady_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f176dc3785048428a3b30c841bd84ed79b92ff591378272a599482b167641cd
+size 1025644289

models/embeddings/aligned/ady_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "ady", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/ady_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d83549b0e20236b2c0988f4f13b7266f6f78751ae5dde671daea405bba933aae
+size 65664

models/embeddings/aligned/ady_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "ady",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 174,
+  "vocab_size": 1573
+}

models/embeddings/aligned/ady_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:663d1bbe256cb5339ab746ba37d0f5e5ab034bafd747c390ebbdd1eabbe1b5fc
+size 256436225

models/embeddings/aligned/ady_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "ady", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/ady_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7057b6fbc0c824ffad21041b164ea17f161de3d678d0a454ec12b77d682c863
+size 4224

models/embeddings/aligned/ady_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "ady",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 174,
+  "vocab_size": 1573
+}

models/embeddings/aligned/ady_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a10090792b8196b70efae444f73787c1ec5103fd55dc37ceb14d647443ad1a4
+size 512838913

models/embeddings/aligned/ady_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "ady", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/ady_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84ba46ef4bf4f82df308790cb1ba9f63722408b474837aeab9f8fc47f81c7e11
+size 16512

models/embeddings/aligned/ady_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "ady",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 174,
+  "vocab_size": 1573
+}

models/embeddings/monolingual/ady_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:15314cc76788ddb26c68cad7ccc4d1a2fa1c42bf0d8a23163020f6c1be5e011c
-size 1025621365
+oid sha256:6f176dc3785048428a3b30c841bd84ed79b92ff591378272a599482b167641cd
+size 1025644289

models/embeddings/monolingual/ady_128d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 128
   },
-  "vocab_size": 1551
+  "vocab_size": 1573
 }

models/embeddings/monolingual/ady_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0a7d65d778045d5838991706962f1f2fda476d83f0eaf0d25e006a2c54b0addb
-size 256430197
+oid sha256:663d1bbe256cb5339ab746ba37d0f5e5ab034bafd747c390ebbdd1eabbe1b5fc
+size 256436225

models/embeddings/monolingual/ady_32d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 32
   },
-  "vocab_size": 1551
+  "vocab_size": 1573
 }

models/embeddings/monolingual/ady_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:462cda6a4c5bb17912b29e9f210bd916044788b457f8b2cd61463a18150385c6
-size 512827253
+oid sha256:5a10090792b8196b70efae444f73787c1ec5103fd55dc37ceb14d647443ad1a4
+size 512838913

models/embeddings/monolingual/ady_64d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 64
   },
-  "vocab_size": 1551
+  "vocab_size": 1573
 }

models/subword_markov/ady_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:326d1899bc21c88f5d2ecbb60df90091a3f81877b05f9f3880179e64fc82bc39
-size 35559
+oid sha256:ea1f21df4b685dfea6b5db6e44e57a8bb0ac92311c91c308143cf35fcbb61d68
+size 38398

models/subword_markov/ady_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "ady",
-  "unique_contexts": 410,
-  "total_transitions": 461186
+  "unique_contexts": 450,
+  "total_transitions": 466024
 }

models/subword_markov/ady_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:439580e1d85b149b7a4e12434dcacbd8c63ce4f422949f7787d12f51a3b06768
-size 169546
+oid sha256:cc4a78482132260c4af1de9c10f9cc91758d344526e771dd9ccb4fc1f194464c
+size 177308

models/subword_markov/ady_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "ady",
-  "unique_contexts": 4325,
-  "total_transitions": 460515
+  "unique_contexts": 4503,
+  "total_transitions": 465338
 }

models/subword_markov/ady_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ba32cd485f139de6037badf290db15170159de5ebb63242fabe6473e50963984
-size 554556
+oid sha256:5b821b2be71b4b958f03177625985421729fee5ea5f35f0bcbbba7f891cf8e38
+size 567095

models/subword_markov/ady_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "ady",
-  "unique_contexts": 24260,
-  "total_transitions": 459844
+  "unique_contexts": 25044,
+  "total_transitions": 464652
 }

models/subword_markov/ady_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a23a4022efd4fb6c5360771eb3945cadabce7d4e08d9a04895e4535a30c12dad
-size 1240010
+oid sha256:a5acbad74a99249e96dd951ee18917f894f3669704783fd95161beee040f3fce
+size 1265262

models/subword_markov/ady_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "ady",
-  "unique_contexts": 72077,
-  "total_transitions": 459173
+  "unique_contexts": 73859,
+  "total_transitions": 463966
 }

models/subword_ngram/ady_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f2744392ac6f30902dc24ef71be21ed3d0027816ff58c51cc117dbe79b60b6eb
-size 26615
+oid sha256:d9de0521f3604c270d9d5334436fe81fb540e8ea5906b0be25a092aa6179cf45
+size 27108

models/subword_ngram/ady_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "ady",
-  "unique_ngrams": 2072,
-  "total_ngrams": 461186
+  "unique_ngrams": 2126,
+  "total_ngrams": 466024
 }

models/subword_ngram/ady_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3e27920b4a896f7b33778a4da60561fc4bf2c40fe17414d4721a023a815cc6e6
-size 144946
+oid sha256:ed0ae00b308ead9c908cdaabf26a84bec9172251d443c0d3d3aa02081e24292d
+size 147714

models/subword_ngram/ady_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "ady",
-  "unique_ngrams": 11614,
-  "total_ngrams": 460515
+  "unique_ngrams": 11856,
+  "total_ngrams": 465338
 }

models/subword_ngram/ady_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a1189db5bc7c400c2d162eabf55e0ca7ce6b8403c020f3330fe200b437eab55c
-size 467994
+oid sha256:d8d87c1c9407f9d5599e4e07229e5b7ccf78b280fc06922a9134d9e6f48368fa
+size 470826

models/subword_ngram/ady_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "ady",
-  "unique_ngrams": 35316,
-  "total_ngrams": 459844
+  "unique_ngrams": 36062,
+  "total_ngrams": 464652
 }

models/subword_ngram/ady_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5b99c1f7304e466988fd44a8987f2393709285de0fa76cc71513adbc5c16ebb
+size 704587

models/subword_ngram/ady_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "ady",
+  "unique_ngrams": 52393,
+  "total_ngrams": 463966
+}

models/tokenizer/ady_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:39a0cc5483f8b60e51f162e1451531fbd79f4a7ef64ed4d9770c72634ad2f63f
-size 582264
+oid sha256:907c1ae616234d526f5961f0c4f163fb497ac9c3805d3e14a7fe3fabf96c944f
+size 579551

models/tokenizer/ady_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:60fe921e9bf5abac9a26d2e450368741ec923239ef1bfe3380b5fecc9f1ebc5e
-size 924924
+oid sha256:62873f8357831fb35e5a2f441ad474a6c2eb8cc0ae9734eb5c04190135969486
+size 926359

models/tokenizer/ady_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:eed6e8aa7a197f3cf93f47869aff6cc5bec5173974995af92b59a00c2be84d9a
-size 396510
+oid sha256:0c0595fa5cbdefef5ffe8230419254db68de7f2fb652181e980c62b94f6eea3a
+size 395183

models/vocabulary/ady_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0394f3c6c2f9871c8756de87f7c0ab00cba32facf111d0c2027f0844c932241b
-size 124070
+oid sha256:e4972182eb8bd7e39e0e37b85fab21a678c05c8dec6fd4c262eca1c1d0e7439b
+size 125482

models/vocabulary/ady_vocabulary_metadata.json

@@ -1,17 +1,17 @@
 {
   "language": "ady",
-  "vocabulary_size": 7032,
+  "vocabulary_size": 7120,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.37416908841901325,
+    "type_token_ratio": 0.3733302701106042,
     "coverage": {
-      "top_100": 0.21755686942579416,
-      "top_1000": 0.4501954103617597,
-      "top_5000": 0.6754016768547283,
-      "top_10000": 0.7928483147944015
+      "top_100": 0.21408644852144804,
+      "top_1000": 0.45016246020545475,
+      "top_5000": 0.6739308805671338,
+      "top_10000": 0.7907709475204306
     },
-    "hapax_count": 15371,
-    "hapax_ratio": 0.6861134669463911,
-    "total_documents": 671
+    "hapax_count": 15630,
+    "hapax_ratio": 0.687032967032967,
+    "total_documents": 686
   }
 }

models/word_markov/ady_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e529d92df652217192b1a53b43d340b1b2d134ff78b718c2331e8296f5e8a183
-size 772451
+oid sha256:c10b7085544a5b8115070723022c0bdfcead811182d57a2c877f7604d1ba7a53
+size 782266

models/word_markov/ady_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "ady",
-  "unique_contexts": 22306,
-  "total_transitions": 59203
+  "unique_contexts": 22655,
+  "total_transitions": 60252
 }

models/word_markov/ady_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:949fea0fcff222936353244ef6edc60951c50ac50290dc3b857a31ce73658051
-size 1310186
+oid sha256:e0bd26c61a14f1e19b592dca73900dc19aa0d3e8839ea60b95c3643dba42ae82
+size 1323569

models/word_markov/ady_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "ady",
-  "unique_contexts": 46305,
-  "total_transitions": 58532
+  "unique_contexts": 46851,
+  "total_transitions": 59566
 }

models/word_markov/ady_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0edd29aaf52ac72ad56819e317223cc422f1de2fd93feb950a6fd0c9d7772f20
-size 1585798
+oid sha256:6ede7cceaaeebf470cbbc85a329ad43223fcfd6126b983a75c435ce8f14238ff
+size 1598892

models/word_markov/ady_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "ady",
-  "unique_contexts": 51243,
-  "total_transitions": 57861
+  "unique_contexts": 51794,
+  "total_transitions": 58880
 }