Upload all models and assets for chr (latest)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: chr
-language_name: CHR
+language_name: Cherokee
 language_family: american_iroquoian
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-american_iroquoian
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 3.573
+    value: 3.552
   - name: best_isotropy
     type: isotropy
-    value: 0.2005
+    value: 0.2412
   - name: vocabulary_size
     type: vocab
     value: 0
 generated: 2026-01-03
 ---
 
-# CHR - Wikilangs Models
+# Cherokee - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **CHR** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Cherokee** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -60,7 +70,7 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
 - [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
@@ -80,42 +90,42 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 2.938x | 2.95 | 0.1472% | 83,560 |
-| **16k** | 3.361x | 3.37 | 0.1684% | 73,046 |
-| **32k** | 3.573x 🏆 | 3.59 | 0.1790% | 68,709 |
+| **8k** | 2.919x | 2.93 | 0.1472% | 82,177 |
+| **16k** | 3.358x | 3.37 | 0.1694% | 71,429 |
+| **32k** | 3.552x 🏆 | 3.57 | 0.1792% | 67,524 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `ᎠᏥᏂᏘᏂᎠ ᏙᏱᏗᏢᎦᏚᎲ ᏧᎦᎾᏮ ᎠᎹᏰᏟ-Ꮒ. Buenos Aires ᎠᏰᎵᏗᎦᎳᏫᎢᏍᏗ ᎦᏚᎲᎢ. ᏓᏓᏚᎬ ᎪᏪᎵ ᏙᏯᏗᏢ ᏗᏕᎬᏔᏛ ᎠᎹ...`
+**Sample 1:** `ᏅᏓᎩ"Consortium Word List." (nvdagi) () ᎦᏚᎲᎢ ᎡᏉ ᏄᏲᎪᎢ, ᏄᏲᎩ, ᎠᎹᏰᏟ. ᏙᏯᏗᏢ ᏗᏕᎬᏔᏛ be ch...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ᎠᏥᏂᏘᏂᎠ ▁ᏙᏱᏗᏢᎦᏚᎲ ▁ᏧᎦᎾᏮ ▁ᎠᎹᏰᏟ - Ꮒ . ▁buenos ▁aires ▁ᎠᏰᎵᏗᎦᎳᏫᎢᏍᏗ ... (+10 more)` | 20 |
-| 16k | `▁ᎠᏥᏂᏘᏂᎠ ▁ᏙᏱᏗᏢᎦᏚᎲ ▁ᏧᎦᎾᏮ ▁ᎠᎹᏰᏟ - Ꮒ . ▁buenos ▁aires ▁ᎠᏰᎵᏗᎦᎳᏫᎢᏍᏗ ... (+10 more)` | 20 |
-| 32k | `▁ᎠᏥᏂᏘᏂᎠ ▁ᏙᏱᏗᏢᎦᏚᎲ ▁ᏧᎦᎾᏮ ▁ᎠᎹᏰᏟ - Ꮒ . ▁buenos ▁aires ▁ᎠᏰᎵᏗᎦᎳᏫᎢᏍᏗ ... (+10 more)` | 20 |
+| 8k | `▁ᏅᏓᎩ " consortium ▁word ▁list ." ▁( nvda gi ) ... (+13 more)` | 23 |
+| 16k | `▁ᏅᏓᎩ " consortium ▁word ▁list ." ▁( nvdagi ) ▁() ... (+12 more)` | 22 |
+| 32k | `▁ᏅᏓᎩ " consortium ▁word ▁list ." ▁( nvdagi ) ▁() ... (+12 more)` | 22 |
 
-**Sample 2:** `ᎠᎷᏆ"Tsalagi Anagalisgi." ᏙᏱᏗᏢᎦᏚᎲ ᎾᏍᎩ ᏁᏛᎳᏂ-Ꮒ. Oranjestad ᎠᏰᎵᏗᎦᎳᏫᎢᏍᏗ ᎦᏚᎲᎢ. ᏓᏓᏚᎬ ᎪᏪ...`
+**Sample 2:** `ᏳᏈᎳ"Consortium Word List." (yuquila) (). ᏓᏓᏚᎬ ᎪᏪᎵ ᏙᏯᏗᏢ ᏗᏕᎬᏔᏛ be checked`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ᎠᎷ Ꮖ " tsalagi ▁anagalisgi ." ▁ᏙᏱᏗᏢᎦᏚᎲ ▁ᎾᏍᎩ ▁ᏁᏛᎳᏂ - ... (+18 more)` | 28 |
-| 16k | `▁ᎠᎷᏆ " tsalagi ▁anagalisgi ." ▁ᏙᏱᏗᏢᎦᏚᎲ ▁ᎾᏍᎩ ▁ᏁᏛᎳᏂ - Ꮒ ... (+14 more)` | 24 |
-| 32k | `▁ᎠᎷᏆ " tsalagi ▁anagalisgi ." ▁ᏙᏱᏗᏢᎦᏚᎲ ▁ᎾᏍᎩ ▁ᏁᏛᎳᏂ - Ꮒ ... (+13 more)` | 23 |
+| 8k | `▁Ᏻ Ꮘ Ꮃ " consortium ▁word ▁list ." ▁( yu ... (+9 more)` | 19 |
+| 16k | `▁ᏳᏈᎳ " consortium ▁word ▁list ." ▁( yuquila ) ▁(). ... (+6 more)` | 16 |
+| 32k | `▁ᏳᏈᎳ " consortium ▁word ▁list ." ▁( yuquila ) ▁(). ... (+6 more)` | 16 |
 
-**Sample 3:** `ᎴᎹᏂ (Lemani) ᎠᏓᎶᏂᎨᎢ ᎤᏓᏔᏅᎯ ᎠᎩᏍᏗ ᎨᏐᎢ. ᎠᎩᏍᏗ be checked`
+**Sample 3:** `ᎦᏢᏍᏙᏗ"Consortium Word List." (gatlvsdodi). ᏓᏓᏚᎬ ᎪᏪ��� ᏙᏯᏗᏢ ᏗᏕᎬᏔᏛ ᎠᎦᏎᏍᏔᏅ be checked`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁Ꮄ ᎹᏂ ▁( le mani ) ▁ᎠᏓᎶᏂᎨᎢ ▁ᎤᏓᏔᏅᎯ ▁ᎠᎩᏍᏗ ▁ᎨᏐᎢ ... (+4 more)` | 14 |
-| 16k | `▁ᎴᎹᏂ ▁( lemani ) ▁ᎠᏓᎶᏂᎨᎢ ▁ᎤᏓᏔᏅᎯ ▁ᎠᎩᏍᏗ ▁ᎨᏐᎢ . ▁ᎠᎩᏍᏗ ... (+2 more)` | 12 |
-| 32k | `▁ᎴᎹᏂ ▁( lemani ) ▁ᎠᏓᎶᏂᎨᎢ ▁ᎤᏓᏔᏅᎯ ▁ᎠᎩᏍᏗ ▁ᎨᏐᎢ . ▁ᎠᎩᏍᏗ ... (+2 more)` | 12 |
+| 8k | `▁Ꭶ Ꮲ ᏍᏙᏗ " consortium ▁word ▁list ." ▁( gat ... (+10 more)` | 20 |
+| 16k | `▁ᎦᏢᏍᏙᏗ " consortium ▁word ▁list ." ▁( gatlvs dodi ). ... (+7 more)` | 17 |
+| 32k | `▁ᎦᏢᏍᏙᏗ " consortium ▁word ▁list ." ▁( gatlvsdodi ). ▁ᏓᏓᏚᎬ ... (+6 more)` | 16 |
 
 
 ### Key Findings
 
-- **Best Compression:** 32k achieves 3.573x compression
+- **Best Compression:** 32k achieves 3.552x compression
 - **Lowest UNK Rate:** 8k with 0.1472% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
@@ -133,12 +143,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
 |--------|---------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | Word | 160 🏆 | 7.32 | 494 | 67.3% | 100.0% |
-| **2-gram** | Subword | 938 | 9.87 | 3,277 | 40.0% | 86.1% |
-| **3-gram** | Word | 222 | 7.80 | 667 | 63.1% | 100.0% |
-| **3-gram** | Subword | 4,484 | 12.13 | 12,903 | 21.8% | 52.1% |
-| **4-gram** | Word | 508 | 8.99 | 1,308 | 48.2% | 89.2% |
-| **4-gram** | Subword | 9,908 | 13.27 | 28,877 | 18.5% | 39.2% |
+| **2-gram** | Word | 151 🏆 | 7.24 | 471 | 68.7% | 100.0% |
+| **2-gram** | Subword | 931 | 9.86 | 3,244 | 40.2% | 86.2% |
+| **3-gram** | Word | 218 | 7.76 | 655 | 63.6% | 100.0% |
+| **3-gram** | Subword | 4,428 | 12.11 | 12,716 | 22.1% | 52.3% |
+| **4-gram** | Word | 483 | 8.91 | 1,256 | 49.2% | 90.8% |
+| **4-gram** | Subword | 9,728 | 13.25 | 28,356 | 18.7% | 39.4% |
+| **5-gram** | Word | 414 | 8.69 | 901 | 51.7% | 100.0% |
+| **5-gram** | Subword | 9,506 | 13.21 | 27,480 | 20.2% | 39.5% |
 
 ### Top 5 N-grams by Size
 
@@ -146,68 +158,88 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `be checked` | 850 |
-| 2 | `ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ` | 578 |
-| 3 | `ꮣꮣꮪꭼ ꭺꮺꮅ` | 472 |
-| 4 | `ꭺꮺꮅ ꮩꮿꮧꮲ` | 431 |
-| 5 | `word list` | 345 |
+| 1 | `be checked` | 841 |
+| 2 | `ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ` | 577 |
+| 3 | `ꮣꮣꮪꭼ ꭺꮺꮅ` | 470 |
+| 4 | `ꭺꮺꮅ ꮩꮿꮧꮲ` | 430 |
+| 5 | `word list` | 344 |
 
 **3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ` | 431 |
-| 2 | `ꮣꮣꮪꭼ ꭺꮺꮅ ꮩꮿꮧꮲ` | 431 |
-| 3 | `consortium word list` | 343 |
-| 4 | `ꮧꮥꭼꮤꮫ be checked` | 227 |
-| 5 | `ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be` | 216 |
+| 1 | `ꮣꮣꮪꭼ ꭺꮺꮅ ꮩꮿꮧꮲ` | 430 |
+| 2 | `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ` | 430 |
+| 3 | `consortium word list` | 342 |
+| 4 | `ꮧꮥꭼꮤꮫ be checked` | 226 |
+| 5 | `ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be` | 215 |
 
 **4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ꮣꮣꮪꭼ ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ` | 431 |
-| 2 | `ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be checked` | 216 |
-| 3 | `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be` | 163 |
+| 1 | `ꮣꮣꮪꭼ ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ` | 430 |
+| 2 | `ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be checked` | 215 |
+| 3 | `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be` | 162 |
 | 4 | `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ ꭰꭶꮞꮝꮤꮕ` | 96 |
 | 5 | `ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ ꭰꭶꮞꮝꮤꮕ be` | 96 |
 
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be checked` | 162 |
+| 2 | `ꮣꮣꮪꭼ ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be` | 162 |
+| 3 | `ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ ꭰꭶꮞꮝꮤꮕ be checked` | 96 |
+| 4 | `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ ꭰꭶꮞꮝꮤꮕ be` | 96 |
+| 5 | `ꮣꮣꮪꭼ ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ ꭰꭶꮞꮝꮤꮕ` | 96 |
+
 **2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `_ ꭰ` | 5,396 |
-| 2 | `_ ꭴ` | 3,446 |
-| 3 | `ꮧ _` | 2,866 |
-| 4 | `. _` | 2,628 |
-| 5 | `, _` | 2,120 |
+| 1 | `_ ꭰ` | 5,288 |
+| 2 | `_ ꭴ` | 3,380 |
+| 3 | `ꮧ _` | 2,778 |
+| 4 | `. _` | 2,562 |
+| 5 | `, _` | 2,084 |
 
 **3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ꮝ ꮧ _` | 1,406 |
-| 2 | `_ c h` | 987 |
-| 3 | `_ ꭰ ꮄ` | 975 |
-| 4 | `c h e` | 965 |
-| 5 | `ꭰ ꮄ _` | 899 |
+| 1 | `ꮝ ꮧ _` | 1,355 |
+| 2 | `_ c h` | 978 |
+| 3 | `c h e` | 956 |
+| 4 | `_ ꭰ ꮄ` | 955 |
+| 5 | `ꮧ ꮲ _` | 882 |
 
 **4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `_ c h e` | 918 |
-| 2 | `_ ꭰ ꮄ _` | 892 |
-| 3 | `e _ c h` | 857 |
-| 4 | `_ b e _` | 851 |
-| 5 | `c k e d` | 850 |
+| 1 | `_ c h e` | 909 |
+| 2 | `_ ꭰ ꮄ _` | 874 |
+| 3 | `e _ c h` | 848 |
+| 4 | `_ b e _` | 842 |
+| 5 | `c h e c` | 841 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `e _ c h e` | 846 |
+| 2 | `_ c h e c` | 841 |
+| 3 | `e c k e d` | 841 |
+| 4 | `_ b e _ c` | 841 |
+| 5 | `c h e c k` | 841 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram (word) with 160
+- **Best Perplexity:** 2-gram (word) with 151
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~39% of corpus
+- **Coverage:** Top-1000 patterns cover ~40% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -223,14 +255,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
 |---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.4942 | 1.409 | 2.32 | 13,243 | 50.6% |
-| **1** | Subword | 1.6130 | 3.059 | 16.20 | 448 | 0.0% |
-| **2** | Word | 0.0935 | 1.067 | 1.16 | 30,660 | 90.6% |
-| **2** | Subword | 1.0024 | 2.003 | 4.68 | 7,258 | 0.0% |
-| **3** | Word | 0.0291 | 1.020 | 1.05 | 35,274 | 97.1% |
-| **3** | Subword | 0.5824 | 1.497 | 2.33 | 33,930 | 41.8% |
-| **4** | Word | 0.0141 🏆 | 1.010 | 1.02 | 36,782 | 98.6% |
-| **4** | Subword | 0.2770 | 1.212 | 1.46 | 79,071 | 72.3% |
+| **1** | Word | 0.4882 | 1.403 | 2.29 | 13,116 | 51.2% |
+| **1** | Subword | 1.6098 | 3.052 | 16.02 | 447 | 0.0% |
+| **2** | Word | 0.0920 | 1.066 | 1.15 | 29,975 | 90.8% |
+| **2** | Subword | 1.0061 | 2.008 | 4.67 | 7,162 | 0.0% |
+| **3** | Word | 0.0290 | 1.020 | 1.05 | 34,378 | 97.1% |
+| **3** | Subword | 0.5823 | 1.497 | 2.32 | 33,475 | 41.8% |
+| **4** | Word | 0.0141 🏆 | 1.010 | 1.02 | 35,846 | 98.6% |
+| **4** | Subword | 0.2760 | 1.211 | 1.46 | 77,796 | 72.4% |
 
 ### Generated Text Samples (Word-based)
 
@@ -238,27 +270,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `ꭰꮄ ꭿꭰ ꮳꮃꭹ ꮧꭶꮳꭶꮈᏼ ꮧꮜꮓ ꮪꮎꮜꮓꭾꭲ ꭰꮒꮝꭶꮿꮓ ꭰꮗꮱꮝꮧ ꭰꮏꮼꭲ ꭵꮝꮚ ꭰꭺꮩꮝꭶ ꭴꮒꭺꮫ ᏼꮻ ꭴꮕꮤꮒꮣꮝꮩꮧ v ꮝ`
-2. `be checked ꭿꭺꮹꮤ ꮎꮝꭶ ꮎꮝꮗ ꭰꮝꮪꭲꮫ ꭿꭰ ꭰꮣꮄꮒꮝꭼ ꭰꮫꮝꭼ ꭰᏸꮈ ꭴꮣꮄꮴꮂ consortium word list sdagoi ꮣꮣꮪꭼ`
-3. `ꭿꭰ ꮳꮃꭹ ꭴꭼꮻᏻꭿ ꭸꮢ ꮎꮏ ꮌꮞꮋꮗꭹ ꮎ ꮎꮝꭹ ꮑꮫꮃꮒ ꮪꮎꮩꮲꮹꮧꮢ be checked ꭱꮃꮧꮬ thumb none thumb`
+1. `ꭰꮄ ꭳꮒꮿꭸꮝꮩꮧ ꭳꭶꮃꮀꮋ contributed ꮎꭵ ꮝꮖꮄꮝꮧ ᏹꮹꭹ ꮮꭶ ꮵꮿꮢꮒꮅꮩꮈꭲ ꮓꮚꮕ ꭳꭹꮎꮅꮝꮣᏼꮕꭲ ꭰꮳꮧ ꮄꭼꮎꮋ animalia ꭰꮭꭵꭲ phylum`
+2. `be checked ꮪꮎꮩꮲꮹꮧꮢ be checked ꭱꮃꮧꮬ ꭰꮥꮫꮝꭺꭲ ꭶꮳꮔꮃ ꭴꮭꮕꮣꮥꮂ ꭰꮣꮕꮝꮧ ordo artiodactyla ꮟꮣꮑꮈꭿ ꭴꮝꮧ subspecies c`
+3. `ꭿꭰ 50 41 fꭶꮈꮃꮧ 65 f ꭶꮈꭰꮥꭴ 49 fꭶꮈꮃꮧ 50 ꭷꮓꭾꭽ ꭰᏸꮅ ꭴꮢꭷꮅ ꭿꭰ ꭲᏼ ꭰꮒꮩꮎꭵ`
 
 **Context Size 2:**
 
-1. `ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be checked ꮪꮎꮩꮲꮹꮧꮢ`
-2. `ꮣꮣꮪꭼ ꭺꮺꮅ ꭴꮣꮞꭶꮴꮧ ꮧꮥꭼꮤꮫ ꮣꮆꮒꭶꮝꮫ ᏻꮃꮫ ꭼꮏꭸꮝꮫ ꮷᏼꮲ ꭰꮉᏸꮯ be checked ꭱꮃꮧꮬ ꮖꮗꭲꮴꭹꭲꮒ`
-3. `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be checked ꮤꮝꮊꮒꮿ ꮷꮑꭿ ꭰꮄ ꭿꮈꮝꮧ ꭰᏺꮅꭸ ꭰꮉᏸꮅ ꭷꮓꭾꭽ ꭰꮄ ꮶꭲ ꮏꮎ ꭸꮢ yolngu`
+1. `ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be checked ꮷᏼꮲ ꭰꮉᏸꮯ ꭰꮒꭲꮴꭲᏻꮝꮧ ꭲꮴꭲᏻꮝꮧ ꭰꮒꮼꮒꭽ ꮧꮣꮯꮆꮝꮤꮕ be checked ꭱꮃꮧꮬ ꮖꮗꭲꮴꭹꭲꮒ`
+2. `ꮣꮣꮪꭼ ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be checked ꮷᏼꮲ ꭰꮉᏸꮯ ᏻꮃꮫ ꮣꮆꮒꭶꮝꮫ ꭼꮏꭸꮝꮫ`
+3. `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ ꮳꮃꭹ ꮷꮒᏼꮻ ꭰꮒꮳꭻꭲ ꭸꭺꮞꮈꭲ ꭰꮄ ꮜꮚ ꭴꮝꮧ ꮴꮆꭿ ꮠꮑꮅꮑ safire william the way we`
 
 **Context Size 3:**
 
-1. `ꮣꮣꮪꭼ ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ ꮪꮎꮩꮲꮹꮧꮢ be checked`
+1. `ꮣꮣꮪꭼ ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ ꭰꮉᏸꮅ ꮪꮎꮩꮲꮹꮧꮢ be checked`
 2. `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be checked ꭱꮃꮧꮬ ꮖꮗꭲꮴꭹꭲꮒ`
-3. `consortium word list dikaneisd digoluwatisgi ꮣꮣꮪꭼ ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ ꮔꮫꮏꮥꭼ be checked`
+3. `consortium word list amayutlidi saluyi ꮣꮣꮪꭼ ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be checked ꭱꮃꮧꮬ ꮖꮗꭲꮴꭹꭲꮒ`
 
 **Context Size 4:**
 
-1. `ꮣꮣꮪꭼ ꭺꮺꮅ ꮩ��ꮧꮲ ꮧꮥꭼꮤꮫ ꭰꭶꮞꮝꮩꮧ be checked`
-2. `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be checked ꮷᏼꮲ ꭰꮉᏸꮯ ᏻꮃꮫ ꮣꮆꮒꭶꮝꮫ ꭼꮏꭸꮝꮫ`
-3. `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ ꭰꭶꮞꮝꮤꮕ be checked`
+1. `ꮣꮣꮪꭼ ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be checked ꮷᏼꮲ ꭰꮉᏸꮯ ᏻꮃꮫ ꮣꮆꮒꭶꮝꮫ ꭼꮏꭸꮝꮫ`
+2. `ꭺꮺꮅ ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ be checked ꭱꮃꮧꮬ`
+3. `ꮩꮿꮧꮲ ꮧꮥꭼꮤꮫ ꭰꭶꮞꮝꮤꮕ be checked`
 
 
 ### Generated Text Samples (Subword-based)
@@ -267,34 +299,34 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_ꭻꮹꮧꮳꮕᏹꭹꮸꮟꮅꭳꮜꭲꮧ_`
-2. `ꮧꮲ_ꮣꮞꭲᏻꭿꮵꮅ_blipe`
-3. `ꮝꮩꮲꮹꮢ_ꭰ_ꭴꮓꭿᏻꭿꭰ_ꮣ`
+1. `_ꭿ_-_ꮻꮫ_ꭻꭰꮑꭶꮜꮕꭲ_`
+2. `ꮧꮭꭵ_manotatrdo,_`
+3. `ꮝꮦꮡꮣꮕꮫꮃꮜꮒꮿꮧ._ꮣꮯ.`
 
 **Context Size 2:**
 
-1. `_ꭰꮄ_ꭴꮎꮴꮅ_ꮧꭶꮎꮕꮟꮣꮑꮈ`
-2. `_ꭴꮓꮈꮤꮕꭲ,_.._tr>_<`
-3. `ꮧ_list."_(iaꭰꮒꭺꮫ_`
+1. `_ꭰꮑ,_be_ꮳꮃꭹ,_ꭳꮻ_ꮎ`
+2. `_ꭴꭼꮻᏻꭿ_ꭶꮑꭶ_ꮒꮧꮝ_ꭲꮿ`
+3. `ꮧ_ꮣꮒꭺꮫꮢ_be_ꮣꮣꮄꭹ_ꮧ`
 
 **Context Size 3:**
 
-1. `ꮝꮧ_ble._galv)._ꮣꮣꮪ`
-2. `_checked_(gila)_ꮗꮅ`
-3. `_ꭰꮄ_ꭴꮣꮔꮦᏺꮂ_ꮞꮅꮎ_seq`
+1. `ꮝꮧ_ꭸꮢꭹ_ꭽꮻꮎꮧꮲ_tassi`
+2. `_chemispherokee_na`
+3. `checked_(ꭱꮅꮯꮿ_ꭰꮅꮠ_`
 
 **Context Size 4:**
 
-1. `_checked_(ꭱꮃꮧꮭꭲ_ꭰꮣꭿ`
-2. `_ꭰꮄ_ꭰꮣꭿꭿ_polydeuces`
-3. `e_checked_(ꭱꮃꮧꮬ._re`
+1. `_checked_(ꭱꮃꮧꮬ)_(ꮖꮗ`
+2. `_ꭰꮄ_80,000_ꮎꮝꭶꮕꮎ_ꭶꮆ`
+3. `e_checked_ꮪᏻꭺꮫ_ꮹꮞꮝꮧ`
 
 
 ### Key Findings
 
 - **Best Predictability:** Context-4 (word) with 98.6% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (79,071 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (77,796 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -310,26 +342,26 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 4,236 |
-| Total Tokens | 35,186 |
-| Mean Frequency | 8.31 |
+| Vocabulary Size | 4,160 |
+| Total Tokens | 34,218 |
+| Mean Frequency | 8.23 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 35.59 |
+| Frequency Std Dev | 35.30 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | ꭰꮄ | 903 |
-| 2 | be | 852 |
-| 3 | checked | 850 |
-| 4 | ꭿꭰ | 797 |
-| 5 | ꮧꮥꭼꮤꮫ | 611 |
-| 6 | ꮩꮿꮧꮲ | 580 |
-| 7 | ꭺꮺꮅ | 524 |
-| 8 | ꮣꮣꮪꭼ | 482 |
-| 9 | ꮳꮃꭹ | 475 |
-| 10 | word | 346 |
+| 1 | ꭰꮄ | 885 |
+| 2 | be | 843 |
+| 3 | checked | 841 |
+| 4 | ꭿꭰ | 767 |
+| 5 | ꮧꮥꭼꮤꮫ | 610 |
+| 6 | ꮩꮿꮧꮲ | 579 |
+| 7 | ꭺꮺꮅ | 521 |
+| 8 | ꮣꮣꮪꭼ | 480 |
+| 9 | ꮳꮃꭹ | 468 |
+| 10 | word | 345 |
 
 ### Least Common Words (from vocabulary)
 
@@ -350,24 +382,24 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.8714 |
-| R² (Goodness of Fit) | 0.984546 |
+| Zipf Coefficient | 0.8676 |
+| R² (Goodness of Fit) | 0.984121 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 39.8% |
-| Top 1,000 | 74.2% |
+| Top 100 | 40.0% |
+| Top 1,000 | 74.3% |
 | Top 5,000 | 0.0% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9845 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 39.8% of corpus
-- **Long Tail:** -5,764 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9841 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 40.0% of corpus
+- **Long Tail:** -5,840 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -383,37 +415,40 @@ Below are text samples generated from each subword-based Markov chain model:
 
 ### 5.1 Cross-Lingual Alignment
 
-> *Note: Multilingual alignment visualization not available for this language.*
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
 
 
 ### 5.2 Model Comparison
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.2005 🏆 | 0.5045 | N/A | N/A |
-| **mono_64d** | 64 | 0.0662 | 0.4736 | N/A | N/A |
-| **mono_128d** | 128 | 0.0103 | 0.4890 | N/A | N/A |
+| **mono_32d** | 32 | 0.2412 🏆 | 0.5036 | N/A | N/A |
+| **mono_64d** | 64 | 0.0627 | 0.4822 | N/A | N/A |
+| **mono_128d** | 128 | 0.0098 | 0.4702 | N/A | N/A |
+| **aligned_32d** | 32 | 0.2412 | 0.4975 | 0.0596 | 0.3311 |
+| **aligned_64d** | 64 | 0.0627 | 0.4601 | 0.0861 | 0.4702 |
+| **aligned_128d** | 128 | 0.0098 | 0.4781 | 0.1325 | 0.5033 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.2005 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.4890. Lower values indicate better semantic separation.
-- **Alignment Quality:** No aligned models evaluated in this run.
+- **Best Isotropy:** mono_32d with 0.2412 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.4820. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 13.2% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
 ## 6.  Morphological Analysis (Experimental)
 
-> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
-
 This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
 
 ### 6.1 Productivity & Complexity
 
 | Metric | Value | Interpretation | Recommendation |
 |--------|-------|----------------|----------------|
-| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
-| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **1.531** | High formulaic/idiomatic content | - |
 
 ### 6.2 Affix Inventory (Productive Units)
 
@@ -426,7 +461,7 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-ꮝꮧ` | ꮵꮣꮯꮆꮝꮧ, ꭴꮺꮕꮝꮧ, ꭰꭶꮤꮂꮝꮧ |
+| `-ꮝꮧ` | ꭴꮒꭹꮝꮧ, ꮞꮧᏻꮝꮧ, ꭰꮣꮿꮝꮧ |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -448,14 +483,15 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 
 | Word | Suggested Split | Confidence | Stem |
 |------|-----------------|------------|------|
-| ꮒꭶꮅꮝꮧꮝꭸꮝꮧ | **`ꮒꭶꮅꮝꮧꮝꭸ-ꮝꮧ`** | 1.5 | `ꮒꭶꮅꮝꮧꮝꭸ` |
 | ꭰᏸꮅꮧꭶꮃꮻꭲꮝꮧ | **`ꭰᏸꮅꮧꭶꮃꮻꭲ-ꮝꮧ`** | 1.5 | `ꭰᏸꮅꮧꭶꮃꮻꭲ` |
-| ꭰꮜꮓꮧꮥꮆꮖꮝꮧ | **`ꭰꮜꮓꮧꮥꮆꮖ-ꮝꮧ`** | 1.5 | `ꭰꮜꮓꮧꮥꮆꮖ` |
+| ꮒꭶꮅꮝꮧꮝꭸꮝꮧ | **`ꮒꭶꮅꮝꮧꮝꭸ-ꮝꮧ`** | 1.5 | `ꮒꭶꮅꮝꮧꮝꭸ` |
 
 ### 6.6 Linguistic Interpretation
 
 > **Automated Insight:**
-The language CHR appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+The language Cherokee shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
 
 ---
 ## 7. Summary & Recommendations
@@ -466,8 +502,8 @@ The language CHR appears to be more isolating or has a highly fixed vocabulary.
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (3.57x) |
-| N-gram | **2-gram** | Lowest perplexity (160) |
+| Tokenizer | **32k BPE** | Best compression (3.55x) |
+| N-gram | **2-gram** | Lowest perplexity (151) |
 | Markov | **Context-4** | Highest predictability (98.6%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
@@ -682,4 +718,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 10:09:15*
+*Report Date: 2026-01-03 20:28:09*

models/embeddings/aligned/chr_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9608e76f7cb96be932c1395d1bcfefe1c081dd492d18cab0b28d835a637f44fd
+size 1025224158

models/embeddings/aligned/chr_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "chr", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/chr_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46bc20b462846457385b6068fd6f90238c7d468e33af3f998fb331ad235f7d6d
+size 65664

models/embeddings/aligned/chr_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "chr",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 151,
+  "vocab_size": 1172
+}

models/embeddings/aligned/chr_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f760c33d716161b029922d0b7c3599884ca393f9cd64c7db1c0a7e2f1ec41ad5
+size 256324062

models/embeddings/aligned/chr_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "chr", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/chr_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31f01b51e05d62314cbc89d3fe685678d852c389ad02104624c8c3e80bb8b6af
+size 4224

models/embeddings/aligned/chr_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "chr",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 151,
+  "vocab_size": 1172
+}

models/embeddings/aligned/chr_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e9a38a8bd94958ef1ccef12f12d92223fefb6a7bed727f0bb340377e1fd05640
+size 512624094

models/embeddings/aligned/chr_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "chr", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/chr_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4df8debcbf314f96d35f7d6dc2782e956cd957c7de8bd6fe988ee635cf73d7e7
+size 16512

models/embeddings/aligned/chr_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "chr",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 151,
+  "vocab_size": 1172
+}

models/embeddings/monolingual/chr_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b65a441c9c4908f0ffa5c095ba64382b4b323cde80a4f6d979c87e4b93102427
-size 1025246107
+oid sha256:9608e76f7cb96be932c1395d1bcfefe1c081dd492d18cab0b28d835a637f44fd
+size 1025224158

models/embeddings/monolingual/chr_128d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 128
   },
-  "vocab_size": 1193
+  "vocab_size": 1172
 }

models/embeddings/monolingual/chr_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d9b5ecb349ca3c9d1d8f1f38b34647edbd029de966e73fe6f8043e5b64357e71
-size 256329883
+oid sha256:f760c33d716161b029922d0b7c3599884ca393f9cd64c7db1c0a7e2f1ec41ad5
+size 256324062

models/embeddings/monolingual/chr_32d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 32
   },
-  "vocab_size": 1193
+  "vocab_size": 1172
 }

models/embeddings/monolingual/chr_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2ff3baa5798aac84304b756ac363fd816043d3a9775848343c3a39af84cf09bf
-size 512635291
+oid sha256:e9a38a8bd94958ef1ccef12f12d92223fefb6a7bed727f0bb340377e1fd05640
+size 512624094

models/embeddings/monolingual/chr_64d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 64
   },
-  "vocab_size": 1193
+  "vocab_size": 1172
 }

models/subword_markov/chr_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:47a719104c2e085b3f0210500f2e0d35dd3fad4c62155a28ff1885ad952f603e
-size 48128
+oid sha256:3b4f4c43e8d3846167bdc43f2a3dc4c86699e912a96a34eebf31879dfeda3133
+size 50406

models/subword_markov/chr_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "chr",
-  "unique_contexts": 448,
-  "total_transitions": 244557
+  "unique_contexts": 447,
+  "total_transitions": 238935
 }

models/subword_markov/chr_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:90017c14d3413b7443aa8cfd4e8186bbbfa36eb5c061301510a67e33d59ced11
-size 213598
+oid sha256:167401283766adb0d352a1e7805c6c52332a9bc3f6da39a8ebf72250311d9c2e
+size 213571

models/subword_markov/chr_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "chr",
-  "unique_contexts": 7258,
-  "total_transitions": 243650
+  "unique_contexts": 7162,
+  "total_transitions": 238035
 }

models/subword_markov/chr_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3aa3158ad438f31c7b0fe3d234fab09e4e04d062e867a5f9276cd4c9c48b1d0c
-size 611513
+oid sha256:bd7ba6e70ee8f0c93fe6eaf000495cbe01f56f2d91d10c8ecda1ae5671786b71
+size 601400

models/subword_markov/chr_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "chr",
-  "unique_contexts": 33930,
-  "total_transitions": 242743
+  "unique_contexts": 33475,
+  "total_transitions": 237135
 }

models/subword_markov/chr_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:041f8fb81d9f01bcb72c1f16d3266d79dd5c44b4454b909b4095a6dc7c4e41c9
-size 1161581
+oid sha256:283f2f059945bfb99068c55820b70093ad31dce7d3d763e2c6051a868b6d6b33
+size 1141008

models/subword_markov/chr_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "chr",
-  "unique_contexts": 79071,
-  "total_transitions": 241836
+  "unique_contexts": 77796,
+  "total_transitions": 236235
 }

models/subword_ngram/chr_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8d97d4c240d2e544cee601293886cf200a3ca9744dc6f05309d9f3f6dedb5701
-size 37349
+oid sha256:9b4791f7439af8a9b3785e2ec53c371f4e22342cdd1182d9ee51b3680f1eb235
+size 37002

models/subword_ngram/chr_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "chr",
-  "unique_ngrams": 3277,
-  "total_ngrams": 244557
+  "unique_ngrams": 3244,
+  "total_ngrams": 238935
 }

models/subword_ngram/chr_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4909748af2830cbec527b301ed9c11dc5fba2c6a9e7f48e1b13a633d5cfdcd87
-size 157760
+oid sha256:b56340ab97d7f0fb81ed597d12abf6ab8e79e4b92a8df5871dc964f8393a888e
+size 156597

models/subword_ngram/chr_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "chr",
-  "unique_ngrams": 12903,
-  "total_ngrams": 243650
+  "unique_ngrams": 12716,
+  "total_ngrams": 238035
 }

models/subword_ngram/chr_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a19dcd2eca820072c87f80cca01672f0fafd765e2ca81e6dbfd95249c77828dc
-size 374199
+oid sha256:57dc768d1347044e414d260830723b3977b4d8ae1fb1a9fd2e9c8d3254477faf
+size 367072

models/subword_ngram/chr_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "chr",
-  "unique_ngrams": 28877,
-  "total_ngrams": 242743
+  "unique_ngrams": 28356,
+  "total_ngrams": 237135
 }

models/subword_ngram/chr_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b62758c1284d9a0997d838e016f14c1198d6ed9997e4161cb502bd8a54ce09d9
+size 383421

models/subword_ngram/chr_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "chr",
+  "unique_ngrams": 27480,
+  "total_ngrams": 236235
+}

models/tokenizer/chr_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:359423c436a228311e5a63ed11e5f00c42f4e1f43e6dab0726841d8a8cea7899
-size 558319
+oid sha256:8302ab6dd99220f27f6a4b918a475cead1f491ee5ac099d34ad1c49a2c58d3ff
+size 558565

models/tokenizer/chr_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bd42bf932a4c3f148e083933ebcc3b363a57f53b5b4db22c1e6a60d5017edabe
-size 835851
+oid sha256:f4a07a8d5403d079ea7eaf7dc3d296dc7f93ca7ebb01caf3019f14dbd83fb421
+size 836496

models/tokenizer/chr_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cb436a7afc74fa9ba90ff21df46b3064327fe8aa4d69c5a5859a405d4b67a6c2
-size 392805
+oid sha256:428c97610731c1a1360888f8d67cbdb313e6a2cef8b6bd0816578205bbb32231
+size 392349

models/vocabulary/chr_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8068c404442dc10b84690c7100fc813e925cbe51c1e787ef250cbc0cea84d3d3
-size 73331
+oid sha256:6a51ad98285655f3a93b82e37b2328d6103a522ea9879c30c242647089e8a666
+size 71930

models/vocabulary/chr_vocabulary_metadata.json

@@ -1,17 +1,17 @@
 {
   "language": "chr",
-  "vocabulary_size": 4236,
+  "vocabulary_size": 4160,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.2998054386679336,
+    "type_token_ratio": 0.3039551685809559,
     "coverage": {
-      "top_100": 0.3170671010361522,
-      "top_1000": 0.590561513053708,
-      "top_5000": 0.8133116148590561,
-      "top_10000": 0.9264286683860459
+      "top_100": 0.3169692478695813,
+      "top_1000": 0.5890839199703594,
+      "top_5000": 0.8118284549833271,
+      "top_10000": 0.9276120785476102
     },
-    "hapax_count": 9016,
-    "hapax_ratio": 0.6803501358285542,
-    "total_documents": 907
+    "hapax_count": 8966,
+    "hapax_ratio": 0.6830717659606887,
+    "total_documents": 900
   }
 }

models/word_markov/chr_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2e06367eae370c3314f083ee9fc62374591dc05fc32a00400e11de6f6a357de8
-size 388759
+oid sha256:bbc9639767558234a47208531de6544ec229a2fca763ae0d3fc97ea5b2b25584
+size 383564

models/word_markov/chr_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "chr",
-  "unique_contexts": 13243,
-  "total_transitions": 43295
+  "unique_contexts": 13116,
+  "total_transitions": 42284
 }

models/word_markov/chr_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:efac8874c4bac9908d723a0cd4c5b8e19f9b13e1c696f3e7076b4376fd315124
-size 694832
+oid sha256:ff38caa6b856e38a004a07ba1e2a158a2a6e4c726f68e712654eb3fba3960208
+size 679573

models/word_markov/chr_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "chr",
-  "unique_contexts": 30660,
-  "total_transitions": 42388
+  "unique_contexts": 29975,
+  "total_transitions": 41384
 }

models/word_markov/chr_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:61472a61178571af03b7dbace2a9fde2feec3747df77700588b54d959fe32645
-size 845527
+oid sha256:0b351dfd6a07f58fcb32eb97bbc0e03270e06acb0d0985c95f1e74f0cb41c05c
+size 826860

models/word_markov/chr_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "chr",
-  "unique_contexts": 35274,
-  "total_transitions": 41481
+  "unique_contexts": 34378,
+  "total_transitions": 40484
 }