Upload all models and assets for chy (latest)

.gitattributes

@@ -38,3 +38,4 @@ visualizations/performance_dashboard.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: chy
-language_name: CHY
+language_name: Cheyenne
 language_family: american_algonquian
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-american_algonquian
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,7 +33,7 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 3.497
+    value: 3.494
   - name: best_isotropy
     type: isotropy
     value: 0.0023
@@ -33,10 +43,10 @@ metrics:
 generated: 2026-01-03
 ---
 
-# CHY - Wikilangs Models
+# Cheyenne - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **CHY** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Cheyenne** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -60,7 +70,7 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
 - [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
@@ -80,35 +90,35 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.497x 🏆 | 3.52 | 0.0960% | 19,785 |
+| **8k** | 3.494x 🏆 | 3.52 | 0.1022% | 18,598 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Vášêtaëno, Amâho'hestôtse (Pl: amâho'héstotôtse) Ama'éno'hamémôxe'êstoo'o Ama'én...`
+**Sample 1:** `Vóo'kooma, vóo'ooma (Melanerpes erythrocephalus) ve'kêseho-éve. Tôhohko`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁vášêtaëno , ▁amâho ' hestôtse ▁( pl : ▁amâho ' ... (+16 more)` | 26 |
+| 8k | `▁vóo ' kooma , ▁vóo ' ooma ▁( melanerpes ▁erythrocephalus ... (+8 more)` | 18 |
 
-**Sample 2:** `Ma'xeamóvôhtó'hestôtse, Éam-óvôhtó'heo'o. thumb|right thumb|right thumb|Daimler-...`
+**Sample 2:** `Hestaahtsémeno (Ribes floridum), heso'xêhestaahtsémeno, na'éstse máhtáme.`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ma ' xeamóvôhtó ' hestôtse , ▁éam - óvôhtó ' ... (+16 more)` | 26 |
+| 8k | `▁hestaahtsémeno ▁( ribes ▁floridum ), ▁heso ' xêhestaahtsémeno , ▁na ... (+4 more)` | 14 |
 
-**Sample 3:** `Mâhpémo'éhe (Alces alces) máto héva popóhpoévêsémo'éhe váótséva-éve.`
+**Sample 3:** `Vó'aehesanestôtse (vé'ho'énêstsestôtse: buckskin suit; "antelope-dress") Pl: vó'...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁mâhpémo ' éhe ▁( alces ▁alces ) ▁máto ▁héva ▁popóhpoévêsémo ... (+6 more)` | 16 |
+| 8k | `▁vó ' aehesanestôtse ▁( vé ' ho ' énêstsestôtse : ... (+20 more)` | 30 |
 
 
 ### Key Findings
 
-- **Best Compression:** 8k achieves 3.497x compression
-- **Lowest UNK Rate:** 8k with 0.0960% unknown tokens
+- **Best Compression:** 8k achieves 3.494x compression
+- **Lowest UNK Rate:** 8k with 0.1022% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -125,12 +135,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
 |--------|---------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | Word | 102 🏆 | 6.68 | 159 | 86.3% | 100.0% |
-| **2-gram** | Subword | 330 | 8.37 | 871 | 59.3% | 100.0% |
-| **3-gram** | Word | 156 | 7.28 | 245 | 72.6% | 100.0% |
-| **3-gram** | Subword | 1,700 | 10.73 | 3,811 | 27.1% | 73.0% |
-| **4-gram** | Word | 310 | 8.27 | 449 | 52.1% | 100.0% |
-| **4-gram** | Subword | 4,072 | 11.99 | 8,559 | 18.3% | 52.6% |
+| **2-gram** | Word | 98 🏆 | 6.62 | 148 | 88.0% | 100.0% |
+| **2-gram** | Subword | 325 | 8.34 | 853 | 59.8% | 100.0% |
+| **3-gram** | Word | 150 | 7.23 | 229 | 74.0% | 100.0% |
+| **3-gram** | Subword | 1,635 | 10.67 | 3,634 | 27.6% | 73.9% |
+| **4-gram** | Word | 301 | 8.23 | 420 | 52.7% | 100.0% |
+| **4-gram** | Subword | 3,873 | 11.92 | 8,064 | 18.7% | 53.5% |
+| **5-gram** | Word | 213 | 7.74 | 290 | 59.9% | 100.0% |
+| **5-gram** | Subword | 4,512 | 12.14 | 8,516 | 17.1% | 49.3% |
 
 ### Top 5 N-grams by Size
 
@@ -138,68 +150,88 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `na éstse` | 161 |
+| 1 | `na éstse` | 140 |
 | 2 | `vé ho` | 119 |
 | 3 | `ho énêstsestôtse` | 72 |
 | 4 | `republic of` | 67 |
-| 5 | `ho e` | 57 |
+| 5 | `éstse manâhéno` | 55 |
 
 **3-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
 | 1 | `vé ho énêstsestôtse` | 72 |
-| 2 | `na éstse manâhéno` | 56 |
-| 3 | `ho e éve` | 49 |
-| 4 | `na éstse ho` | 48 |
-| 5 | `éstse ho e` | 48 |
+| 2 | `na éstse manâhéno` | 55 |
+| 3 | `ho honáéšé e` | 44 |
+| 4 | `ho e éve` | 33 |
+| 5 | `éstse ho e` | 32 |
 
 **4-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `éstse ho e éve` | 48 |
-| 2 | `na éstse ho e` | 48 |
+| 1 | `na éstse ho e` | 32 |
+| 2 | `éstse ho e éve` | 32 |
 | 3 | `ma kaetaévôxe êstoo o` | 25 |
 | 4 | `toháano éve ho etse` | 23 |
-| 5 | `na éstse manâhéno ho` | 22 |
+| 5 | `manâhéno ho honáéšé e` | 22 |
+
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `na éstse ho e éve` | 32 |
+| 2 | `ho honáéšé e united states` | 22 |
+| 3 | `éstse manâhéno ho honáéšé e` | 22 |
+| 4 | `na éstse manâhéno ho honáéšé` | 22 |
+| 5 | `manâhéno ho honáéšé e united` | 21 |
 
 **2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `e _` | 1,534 |
-| 2 | `s e` | 1,395 |
-| 3 | `s t` | 1,310 |
-| 4 | `t s` | 1,310 |
-| 5 | `h e` | 1,012 |
+| 1 | `e _` | 1,450 |
+| 2 | `s e` | 1,334 |
+| 3 | `s t` | 1,269 |
+| 4 | `t s` | 1,249 |
+| 5 | `h e` | 974 |
 
 **3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `t s e` | 1,002 |
-| 2 | `s e _` | 580 |
-| 3 | `e s t` | 468 |
-| 4 | `s t s` | 459 |
-| 5 | `h o '` | 443 |
+| 1 | `t s e` | 956 |
+| 2 | `s e _` | 548 |
+| 3 | `e s t` | 461 |
+| 4 | `s t s` | 436 |
+| 5 | `h o '` | 420 |
 
 **4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `t s e _` | 456 |
-| 2 | `s t s e` | 436 |
-| 3 | `ô t s e` | 287 |
-| 4 | `t ô t s` | 208 |
-| 5 | `e s t ô` | 198 |
+| 1 | `t s e _` | 427 |
+| 2 | `s t s e` | 413 |
+| 3 | `ô t s e` | 276 |
+| 4 | `t ô t s` | 204 |
+| 5 | `e s t ô` | 194 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `s t s e _` | 216 |
+| 2 | `t ô t s e` | 203 |
+| 3 | `s t ô t s` | 190 |
+| 4 | `e s t ô t` | 190 |
+| 5 | `ê s t s e` | 170 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram (word) with 102
+- **Best Perplexity:** 2-gram (word) with 98
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~53% of corpus
+- **Coverage:** Top-1000 patterns cover ~49% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -215,14 +247,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
 |---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.4118 | 1.330 | 2.00 | 3,383 | 58.8% |
-| **1** | Subword | 1.3726 | 2.589 | 9.72 | 175 | 0.0% |
-| **2** | Word | 0.1118 | 1.081 | 1.20 | 6,516 | 88.8% |
-| **2** | Subword | 1.2032 | 2.303 | 5.05 | 1,699 | 0.0% |
-| **3** | Word | 0.0474 | 1.033 | 1.08 | 7,515 | 95.3% |
-| **3** | Subword | 0.6524 | 1.572 | 2.34 | 8,541 | 34.8% |
-| **4** | Word | 0.0269 🏆 | 1.019 | 1.04 | 7,792 | 97.3% |
-| **4** | Subword | 0.2844 | 1.218 | 1.44 | 19,944 | 71.6% |
+| **1** | Word | 0.4049 | 1.324 | 1.97 | 3,214 | 59.5% |
+| **1** | Subword | 1.3402 | 2.532 | 9.42 | 172 | 0.0% |
+| **2** | Word | 0.1099 | 1.079 | 1.20 | 6,126 | 89.0% |
+| **2** | Subword | 1.2169 | 2.324 | 5.05 | 1,620 | 0.0% |
+| **3** | Word | 0.0453 | 1.032 | 1.08 | 7,065 | 95.5% |
+| **3** | Subword | 0.6471 | 1.566 | 2.32 | 8,158 | 35.3% |
+| **4** | Word | 0.0256 🏆 | 1.018 | 1.04 | 7,317 | 97.4% |
+| **4** | Subword | 0.2799 | 1.214 | 1.44 | 18,852 | 72.0% |
 
 ### Generated Text Samples (Word-based)
 
@@ -230,27 +262,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `e he óonéma enóne éohkê héška ó he hohamháa há continentan naa nêhéóhe násáahéne enomóvóhe tsé`
-2. `ho honáemanėstóoseo o united states manâhestôtse 1 188 lwanda 100px mogadishu somali shilling swahil...`
-3. `o vé keehoohtsêstse vó kaehevôtse vé ho énêstsestôtse billingscheyenne english dictionary chief dull...`
+1. `e éve ho honáéšé e cfa ma kaetaévôxe êstoo o toháano éve hóxovê hooma naa kánome`
+2. `ho éstova éhe nėstaane néstse vóonotse 30 hestáotse naa unie van zuid afrika hotómá e great`
+3. `o gdp ppp 72 7 afrikaans vé ho etse 56 785 6 coloured 9 indian tséh`
 
 **Context Size 2:**
 
-1. `na éstse ho e éve asia center thumb handelskade in willemstad curaçao`
-2. `vé ho énêstsestôtse black hills ho honáéšé e missouri ó he e pónoeo hé e na éstse`
-3. `ho énêstsestôtse cimarron river bull river forgan heévȧhetanéno`
+1. `na éstse manâhéno ho honáéšé e vehicle license kȧhkoetohko prefix 29 hotómá e mo hetaneho e hánêsóvó...`
+2. `vé ho énestse 71 740 6 144 562 903 somali federal republic of the congo congo kinshasa`
+3. `ho énêstsestôtse wyolacheyenne english dictionarychief dull knife college hoig stan the peace chiefs...`
 
 **Context Size 3:**
 
-1. `vé ho énêstsestôtse bay horse variant tsé vó névóvâtse`
-2. `na éstse manâhéno ho honáéšé e united states óoetaneo o óoetanéno tsé amo eétâhéstove vé ho énêstses...`
-3. `ho e éve hóxovê hooma center frameless upright 1 5`
+1. `vé ho énêstsestôtse airplane this is`
+2. `na éstse manâhéno china republic of china republic of china republic of china republic of china repu...`
+3. `ho honáéšé e native news project`
 
 **Context Size 4:**
 
-1. `éstse ho e éve meško`
-2. `na éstse ho e éve vietnam dong hoi airport`
-3. `ma kaetaévôxe êstoo o sango toháano éve ho etse 622 984 4 216 666 1 198 chad republic of`
+1. `éstse ho e éve vietnam dong hoi airport`
+2. `na éstse ho e éve united states states of america`
+3. `ma kaetaévôxe êstoo o toháano éve ho etse 322 460 1 600 democratic republic of the congo of the`
 
 
 ### Generated Text Samples (Subword-based)
@@ -259,34 +291,34 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `e_a_29150002)_te`
-2. `_rkul_mâxpoeése.`
-3. `a_xema'ėh-évese:`
+1. `etokfive_piente'`
+2. `_t:_manésé'e'e,_`
+3. `aliotse'étinoo's`
 
 **Context Size 2:**
 
-1. `e_(vé'še'tó'neadd`
-2. `seotó'o_poestôtse`
-3. `ts_wymnetugual_ju`
+1. `e_100px_minestȯts`
+2. `se_cre_manéó'ho'ô`
+3. `stanjunt.thumb_la`
 
 **Context Size 3:**
 
-1. `tse._vé'ho'hé'e_bo`
-2. `se_rokese_mâhestȯt`
-3. `estôtse_manâhá'e_(`
+1. `tse_(lephonáéšé'e,`
+2. `se_odom_capid_city`
+3. `estôtsestôtsestôts`
 
 **Context Size 4:**
 
-1. `tse_hotómá'e_12_évȯ`
-2. `stseévenomo_hovanan`
-3. `ôtse:_ten_sage";_ar`
+1. `tse_évȯhkėha'etaneh`
+2. `stsestȯtse_kóhkonôh`
+3. `ôtsenáesëö'o_môxeov`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 (word) with 97.3% predictability
+- **Best Predictability:** Context-4 (word) with 97.4% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (19,944 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (18,852 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -302,64 +334,64 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 1,237 |
-| Total Tokens | 8,401 |
-| Mean Frequency | 6.79 |
+| Vocabulary Size | 1,174 |
+| Total Tokens | 7,828 |
+| Mean Frequency | 6.67 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 21.86 |
+| Frequency Std Dev | 21.01 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | e | 431 |
-| 2 | ho | 377 |
-| 3 | o | 237 |
-| 4 | na | 165 |
-| 5 | vé | 161 |
-| 6 | éstse | 161 |
-| 7 | éve | 154 |
-| 8 | of | 118 |
-| 9 | he | 108 |
-| 10 | naa | 108 |
+| 1 | e | 407 |
+| 2 | ho | 351 |
+| 3 | o | 229 |
+| 4 | vé | 159 |
+| 5 | na | 144 |
+| 6 | éstse | 140 |
+| 7 | éve | 133 |
+| 8 | of | 117 |
+| 9 | naa | 104 |
+| 10 | he | 103 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | mustangs | 2 |
-| 2 | sevonévo | 2 |
-| 3 | ėstovátamevéotse | 2 |
-| 4 | ėstova | 2 |
-| 5 | nėstse | 2 |
-| 6 | kūnas | 2 |
-| 7 | epsteins | 2 |
-| 8 | ir | 2 |
-| 9 | felon | 2 |
-| 10 | immigrants | 2 |
+| 1 | pack | 2 |
+| 2 | evenóse | 2 |
+| 3 | mountain | 2 |
+| 4 | cal | 2 |
+| 5 | poly | 2 |
+| 6 | mustangs | 2 |
+| 7 | sevonévo | 2 |
+| 8 | ėstovátamevéotse | 2 |
+| 9 | ėstova | 2 |
+| 10 | nėstse | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.8151 |
-| R² (Goodness of Fit) | 0.976018 |
+| Zipf Coefficient | 0.8142 |
+| R² (Goodness of Fit) | 0.973597 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 54.8% |
-| Top 1,000 | 94.4% |
+| Top 100 | 55.3% |
+| Top 1,000 | 95.6% |
 | Top 5,000 | 0.0% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9760 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 54.8% of corpus
-- **Long Tail:** -8,763 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9736 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 55.3% of corpus
+- **Long Tail:** -8,826 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -375,37 +407,40 @@ Below are text samples generated from each subword-based Markov chain model:
 
 ### 5.1 Cross-Lingual Alignment
 
-> *Note: Multilingual alignment visualization not available for this language.*
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
 
 
 ### 5.2 Model Comparison
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.0023 🏆 | 0.8533 | N/A | N/A |
-| **mono_64d** | 64 | 0.0008 | 0.9264 | N/A | N/A |
-| **mono_128d** | 128 | 0.0002 | 0.9821 | N/A | N/A |
+| **mono_32d** | 32 | 0.0023 🏆 | 0.8896 | N/A | N/A |
+| **mono_64d** | 64 | 0.0007 | 0.9590 | N/A | N/A |
+| **mono_128d** | 128 | 0.0002 | 0.9907 | N/A | N/A |
+| **aligned_32d** | 32 | 0.0023 | 0.8896 | 0.0513 | 0.2179 |
+| **aligned_64d** | 64 | 0.0007 | 0.9590 | 0.0385 | 0.1795 |
+| **aligned_128d** | 128 | 0.0002 | 0.9907 | 0.0128 | 0.1667 |
 
 ### Key Findings
 
 - **Best Isotropy:** mono_32d with 0.0023 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.9206. Lower values indicate better semantic separation.
-- **Alignment Quality:** No aligned models evaluated in this run.
+- **Semantic Density:** Average pairwise similarity of 0.9464. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 5.1% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
 ## 6.  Morphological Analysis (Experimental)
 
-> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
-
 This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
 
 ### 6.1 Productivity & Complexity
 
 | Metric | Value | Interpretation | Recommendation |
 |--------|-------|----------------|----------------|
-| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
-| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **1.027** | High formulaic/idiomatic content | - |
 
 ### 6.2 Affix Inventory (Productive Units)
 
@@ -414,18 +449,18 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Prefixes
 | Prefix | Examples |
 |--------|----------|
-| `-ho` | horse, hotoa, hoohëö |
+| `-ho` | hotóao, hohtóvá, hoéstónéó |
 
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-e` | néstovátamevéotse, kôhtse, where |
-| `-se` | néstovátamevéotse, kôhtse, kaehevotôtse |
-| `-tse` | néstovátamevéotse, kôhtse, kaehevotôtse |
-| `-ne` | mâhoestôtsene, kane, mâheóne |
-| `-ôtse` | kaehevotôtse, oestôtse, xemenôtse |
-| `-ia` | anastacia, abkhazia, shepherdia |
-| `-ve` | êstonêstove, native, hestoháatamaahéstove |
+| `-e` | ôhkêhenove, háahpe, manâhestôtse |
+| `-se` | manâhestôtse, tsétsêhéstâhese, xaénéhetse |
+| `-tse` | manâhestôtse, xaénéhetse, ôhnéménêstse |
+| `-ôtse` | manâhestôtse, mâhoestôtse, ôtse |
+| `-ne` | lione, mâhoestôtsene, nemâhmoteone |
+| `-ve` | ôhkêhenove, ôhkemôxeonêstove, kêsaéve |
+| `-ia` | alnifolia, austria, nitsvia |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -440,9 +475,9 @@ This table shows which prefixes and suffixes most frequently co-occur on the sam
 
 | Prefix | Suffix | Frequency | Examples |
 |--------|--------|-----------|----------|
-| `-ho` | `-e` | 5 words | horse, hováhne |
-| `-ho` | `-ne` | 2 words | hováhne, hovahne |
-| `-ho` | `-se` | 1 words | horse, hotse |
+| `-ho` | `-e` | 5 words | house, hovahne |
+| `-ho` | `-ne` | 2 words | hovahne, hovane |
+| `-ho` | `-se` | 1 words | house, hotse |
 | `-ho` | `-tse` | 1 words | hotse, hohpâhtsenámenôtse |
 | `-ho` | `-ôtse` | 1 words | hohpâhtsenámenôtse |
 
@@ -454,24 +489,26 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 |------|-----------------|------------|------|
 | mâhoestôtsene | **`mâhoest-ôtse-ne`** | 3.0 | `mâhoest` |
 | sevoneóneve | **`sevoneó-ne-ve`** | 3.0 | `sevoneó` |
-| náhkȯhehetanetse | **`náhkȯheheta-ne-tse`** | 3.0 | `náhkȯheheta` |
-| enóseoneve | **`enóseo-ne-ve`** | 3.0 | `enóseo` |
 | éestsėstóseoneve | **`éestsėst��seo-ne-ve`** | 3.0 | `éestsėstóseo` |
-| kaehevotôtse | **`kaehevot-ôtse`** | 1.5 | `kaehevot` |
-| anastacia | **`anastac-ia`** | 1.5 | `anastac` |
-| névóvâtse | **`névóvâ-tse`** | 1.5 | `névóvâ` |
+| enóseoneve | **`enóseo-ne-ve`** | 3.0 | `enóseo` |
+| náhkȯhehetanetse | **`náhkȯheheta-ne-tse`** | 3.0 | `náhkȯheheta` |
+| ôhkêhenove | **`ôhkêheno-ve`** | 1.5 | `ôhkêheno` |
+| manâhestôtse | **`manâhest-ôtse`** | 1.5 | `manâhest` |
+| alnifolia | **`alnifol-ia`** | 1.5 | `alnifol` |
+| ôhkemôxeonêstove | **`ôhkemôxeonêsto-ve`** | 1.5 | `ôhkemôxeonêsto` |
+| hoéstónéó | **`ho-éstónéó`** | 1.5 | `éstónéó` |
+| nemâhmoteone | **`nemâhmoteo-ne`** | 1.5 | `nemâhmoteo` |
+| tsétsêhéstâhese | **`tsétsêhéstâhe-se`** | 1.5 | `tsétsêhéstâhe` |
+| australia | **`austral-ia`** | 1.5 | `austral` |
 | shepherdia | **`shepherd-ia`** | 1.5 | `shepherd` |
-| êstonêstove | **`êstonêsto-ve`** | 1.5 | `êstonêsto` |
-| yellowstone | **`yellowsto-ne`** | 1.5 | `yellowsto` |
-| hoohtseto | **`ho-ohtseto`** | 1.5 | `ohtseto` |
-| xemenôtse | **`xemen-ôtse`** | 1.5 | `xemen` |
-| véhonevoemėstse | **`véhonevoemės-tse`** | 1.5 | `véhonevoemės` |
-| manestôtse | **`manest-ôtse`** | 1.5 | `manest` |
+| xaénéhetse | **`xaénéhe-tse`** | 1.5 | `xaénéhe` |
 
 ### 6.6 Linguistic Interpretation
 
 > **Automated Insight:**
-The language CHY appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+The language Cheyenne shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
 
 ---
 ## 7. Summary & Recommendations
@@ -482,9 +519,9 @@ The language CHY appears to be more isolating or has a highly fixed vocabulary.
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **8k BPE** | Best compression (3.50x) |
-| N-gram | **2-gram** | Lowest perplexity (102) |
-| Markov | **Context-4** | Highest predictability (97.3%) |
+| Tokenizer | **8k BPE** | Best compression (3.49x) |
+| N-gram | **2-gram** | Lowest perplexity (98) |
+| Markov | **Context-4** | Highest predictability (97.4%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
 
@@ -698,4 +735,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 10:13:21*
+*Report Date: 2026-01-03 20:28:03*

models/embeddings/aligned/chy_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4fcc832a966fb87260314fdde26c7aaca3eb6f87727d6172f71003e9cbd2c5eb
+size 1024154505

models/embeddings/aligned/chy_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "chy", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/chy_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c45ba64aef67dfa385dc9f8ca17d2d96ad776d8ccbe75505f9e1f6875aad67b
+size 65664

models/embeddings/aligned/chy_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "chy",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 78,
+  "vocab_size": 148
+}

models/embeddings/aligned/chy_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:207f94e4c6d8a4f9a8bc11e98f994580ddfbed45c5a57f6cc17e016f113b2fa3
+size 256040841

models/embeddings/aligned/chy_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "chy", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/chy_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf25a3e94561de4f68e7c3c373a0986dd3586b6921811406570a582aa2c047ad
+size 4224

models/embeddings/aligned/chy_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "chy",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 78,
+  "vocab_size": 148
+}

models/embeddings/aligned/chy_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f311db029de4600904491344614a6a0e06edf2dfd38492760620dcec861d4e73
+size 512078729

models/embeddings/aligned/chy_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "chy", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/chy_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5bade9f56bce2587b66a3e08b84b9549bcdb85c4f121a359ad5166f3f0e2fcf0
+size 16512

models/embeddings/aligned/chy_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "chy",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 78,
+  "vocab_size": 148
+}

models/embeddings/monolingual/chy_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fe82c87b0cea7042a6a16f85cf1875011ca32f13b8c941e25d5c1d57849b2eed
-size 1024170112
+oid sha256:4fcc832a966fb87260314fdde26c7aaca3eb6f87727d6172f71003e9cbd2c5eb
+size 1024154505

models/embeddings/monolingual/chy_128d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 128
   },
-  "vocab_size": 163
+  "vocab_size": 148
 }

models/embeddings/monolingual/chy_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:70c8741bb1d28e9983cdf32850bd345db541b195a40f92c1ebc8ba49374864be
-size 256044928
+oid sha256:207f94e4c6d8a4f9a8bc11e98f994580ddfbed45c5a57f6cc17e016f113b2fa3
+size 256040841

models/embeddings/monolingual/chy_32d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 32
   },
-  "vocab_size": 163
+  "vocab_size": 148
 }

models/embeddings/monolingual/chy_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:235dc5785bb5f5c2123f5d894c3c7266a0f239882928a7be48b4ddbb112ff63d
-size 512086656
+oid sha256:f311db029de4600904491344614a6a0e06edf2dfd38492760620dcec861d4e73
+size 512078729

models/embeddings/monolingual/chy_64d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 64
   },
-  "vocab_size": 163
+  "vocab_size": 148
 }

models/subword_markov/chy_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d18af851c234d84f1c8c493282bad10c2ce3cf38e5f884b0def3b69a8cc46a72
-size 16739
+oid sha256:1c35468a046d4cdb8600723c2cc9f33f643f687ec3db096b191c13c05400bf4a
+size 15097

models/subword_markov/chy_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "chy",
-  "unique_contexts": 175,
-  "total_transitions": 68722
+  "unique_contexts": 172,
+  "total_transitions": 64543
 }

models/subword_markov/chy_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8ecd88390cc78af12a57e86676cd4ac3d721a2a3b6105dedf56498e4f9198ba6
-size 55410
+oid sha256:68f0dc3358f63ea1ede0f6586e8b16d1fd372465d1bc2955b2c27e2422d47ff4
+size 53541

models/subword_markov/chy_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "chy",
-  "unique_contexts": 1699,
-  "total_transitions": 68263
+  "unique_contexts": 1620,
+  "total_transitions": 64114
 }

models/subword_markov/chy_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fd7301d331a8f6e5cf3904b5bd4befc8f3edf1b29659b35c4c4df3b64b73ed4d
-size 150466
+oid sha256:8d479318e787a4b6b2b6582e01bf5581981ed29a293db4c95bfe4803a02ceac4
+size 141411

models/subword_markov/chy_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "chy",
-  "unique_contexts": 8541,
-  "total_transitions": 67804
+  "unique_contexts": 8158,
+  "total_transitions": 63685
 }

models/subword_markov/chy_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3e44005f2ed1b58ece31011bfe69d86d11229eda4839c36faadb0b8ac3af5479
-size 281701
+oid sha256:2a00c38db3647e5284daec765ca6502387e939c467196ad926faa95198224d6c
+size 266437

models/subword_markov/chy_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "chy",
-  "unique_contexts": 19944,
-  "total_transitions": 67345
+  "unique_contexts": 18852,
+  "total_transitions": 63256
 }

models/subword_ngram/chy_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0228df27f63ed413c576b4cbe4ab6f76c9785023f717763c1927ee318cba7061
-size 11557
+oid sha256:34dc6910378237012a37e238d7df974f017c91b1c3ebd46bf8d76be772f32ac2
+size 11321

models/subword_ngram/chy_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "chy",
-  "unique_ngrams": 871,
-  "total_ngrams": 68722
+  "unique_ngrams": 853,
+  "total_ngrams": 64543
 }

models/subword_ngram/chy_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c6c8121df7c4fff9c78c7deca418c3eaecb7394cf750b4681239ca320d9977e8
-size 41181
+oid sha256:c62fad42c8e95d0d3266d32d66f8bd47a309860a70bb08f72065f1851f496ca7
+size 39559

models/subword_ngram/chy_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "chy",
-  "unique_ngrams": 3811,
-  "total_ngrams": 68263
+  "unique_ngrams": 3634,
+  "total_ngrams": 64114
 }

models/subword_ngram/chy_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4bfb26d6de423f9bad18bfb5f3a57a90fada9cbd23138dc9ac7e2fcdbe0c7831
-size 100088
+oid sha256:7f24a9511d5150fb308647b27c266b5a6f588d314eec67eede769dc1df1d15a0
+size 93862

models/subword_ngram/chy_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "chy",
-  "unique_ngrams": 8559,
-  "total_ngrams": 67804
+  "unique_ngrams": 8064,
+  "total_ngrams": 63685
 }

models/subword_ngram/chy_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a5a05456a14449cd7ecfb447e8bd0b51ddacc83add84722e977dd5f7c8b8b64
+size 107070

models/subword_ngram/chy_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "chy",
+  "unique_ngrams": 8516,
+  "total_ngrams": 63256
+}

models/tokenizer/chy_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7224c406ce7f0ceae9ae28bfb815c3ebc81aafabd231728ea1823e5477c23e0d
-size 374664
+oid sha256:f1c8a724c8c94c9c917aae80be9128fd26d48b89f29362db6780a3920aa83b5f
+size 373477

models/vocabulary/chy_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:557659dcccab98922c4fc45c22d401a01f92a33330e2d06cbdbafd9e3d8f37d2
-size 22392
+oid sha256:b9b062a41dc39802b42fd2a6c1857efd43a84af7da75bb98cfda744f6ad48c62
+size 21524

models/vocabulary/chy_vocabulary_metadata.json

@@ -1,15 +1,15 @@
 {
   "language": "chy",
-  "vocabulary_size": 1237,
+  "vocabulary_size": 1174,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.32707120045087357,
+    "type_token_ratio": 0.33165930092406587,
     "coverage": {
-      "top_100": 0.4324628968626714,
-      "top_1000": 0.7445989103888785
+      "top_100": 0.4347127360385697,
+      "top_1000": 0.7513057452792286
     },
-    "hapax_count": 2245,
-    "hapax_ratio": 0.644744399770247,
-    "total_documents": 459
+    "hapax_count": 2128,
+    "hapax_ratio": 0.6444579043004239,
+    "total_documents": 429
   }
 }

models/word_markov/chy_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:386ae58f4271bb09f2a7300c1ffb43cb66268f9329c798213791212b7375c544
-size 86902
+oid sha256:d274a31557ebe9914adbbba1a99399f013526f094054439fc9c1a835e8d5e6a4
+size 82511

models/word_markov/chy_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "chy",
-  "unique_contexts": 3383,
-  "total_transitions": 10187
+  "unique_contexts": 3214,
+  "total_transitions": 9527
 }

models/word_markov/chy_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:25b29a5fd2a035a5761b624f9763e3b2f1bd7218a31608a0a6f94197490478db
-size 131833
+oid sha256:72ec884b474155c4fb42b336ac8db08240a62ef9d9b96702036915eb8ece49a9
+size 124517

models/word_markov/chy_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "chy",
-  "unique_contexts": 6516,
-  "total_transitions": 9728
+  "unique_contexts": 6126,
+  "total_transitions": 9098
 }

models/word_markov/chy_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b4b7d0cf8b279e25f0c1dd5d12c4331f6647670846d67a4e626aa888e8687564
-size 158524
+oid sha256:17f7aa921fc6af5108a2d17c0dfb401db36aa8745424ebfb48f77007070d9508
+size 149507

models/word_markov/chy_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "chy",
-  "unique_contexts": 7515,
-  "total_transitions": 9269
+  "unique_contexts": 7065,
+  "total_transitions": 8669
 }

models/word_markov/chy_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:46f60b22a8fcac07a7de898ad66d40c570966ec70737f00b5cf2d7b22bf2c6a2
-size 174153
+oid sha256:459da114df8b2b97b442e07a5ee700b713cda57712639a7ac65a2bb20621e0d4
+size 163772

models/word_markov/chy_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "chy",
-  "unique_contexts": 7792,
-  "total_transitions": 8810
+  "unique_contexts": 7317,
+  "total_transitions": 8240
 }

models/word_ngram/chy_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f3cf90b3734f097b7881f2bd6766a12f1c1a78a49ef2c9af43b8bb1c04f684de
-size 5216
+oid sha256:d3c85c0ba694c7e59c5083db686db2a289b676f8dc72587178d4e8b7e405f9d5
+size 5096

models/word_ngram/chy_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "chy",
-  "unique_ngrams": 159,
-  "total_ngrams": 10187
+  "unique_ngrams": 148,
+  "total_ngrams": 9527
 }