Upload all models and assets for chy (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 3.456
+    value: 3.497
   - name: best_isotropy
     type: isotropy
-    value: 0.0028
+    value: 0.0023
   - name: vocabulary_size
     type: vocab
-    value: 1659
-generated: 2025-12-28
+    value: 0
+generated: 2026-01-03
 ---
 
 # CHY - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,50 +70,45 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.426x | 3.37 | 0.0811% | 33,276 |
-| **16k** | 3.456x 🏆 | 3.40 | 0.0819% | 32,987 |
+| **8k** | 3.497x 🏆 | 3.52 | 0.0960% | 19,785 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Môxéhéó'o (vé'ho'énêstsestôtse: broom, "sweeping [thing]") Pl: môxéheonôtse.
-
-
-C...`
+**Sample 1:** `Vášêtaëno, Amâho'hestôtse (Pl: amâho'héstotôtse) Ama'éno'hamémôxe'êstoo'o Ama'én...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁môxéhéó ' o ▁( vé ' ho ' énêstsestôtse : ... (+18 more)` | 28 |
-| 16k | `▁môxéhéó ' o ▁( vé ' ho ' énêstsestôtse : ... (+17 more)` | 27 |
+| 8k | `▁vášêtaëno , ▁amâho ' hestôtse ▁( pl : ▁amâho ' ... (+16 more)` | 26 |
 
-**Sample 2:** `Brazil, na'éstse ho'e-éve, Amérika.
-
-Category:Brazil`
+**Sample 2:** `Ma'xeamóvôhtó'hestôtse, Éam-óvôhtó'heo'o. thumb|right thumb|right thumb|Daimler-...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁brazil , ▁na ' éstse ▁ho ' e - éve ... (+6 more)` | 16 |
-| 16k | `▁brazil , ▁na ' éstse ▁ho ' e - éve ... (+6 more)` | 16 |
-
-**Sample 3:** `Boise, na'éstse manâhéno, Idaho.
+| 8k | `▁ma ' xeamóvôhtó ' hestôtse , ▁éam - óvôhtó ' ... (+16 more)` | 26 |
 
-Category:Mâhoestôtse`
+**Sample 3:** `Mâhpémo'éhe (Alces alces) máto héva popóhpoévêsémo'éhe váótséva-éve.`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁boise , ▁na ' éstse ▁manâhéno , ▁idaho . ▁category ... (+2 more)` | 12 |
-| 16k | `▁boise , ▁na ' éstse ▁manâhéno , ▁idaho . ▁category ... (+2 more)` | 12 |
+| 8k | `▁mâhpémo ' éhe ▁( alces ▁alces ) ▁máto ▁héva ▁popóhpoévêsémo ... (+6 more)` | 16 |
 
 
 ### Key Findings
 
-- **Best Compression:** 16k achieves 3.456x compression
-- **Lowest UNK Rate:** 8k with 0.0811% unknown tokens
+- **Best Compression:** 8k achieves 3.497x compression
+- **Lowest UNK Rate:** 8k with 0.0960% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -120,57 +117,89 @@ Category:Mâhoestôtse`
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 237 🏆 | 7.89 | 654 | 65.5% | 100.0% |
-| **2-gram** | 360 🏆 | 8.49 | 1,127 | 58.7% | 99.4% |
-| **3-gram** | 533 | 9.06 | 1,211 | 49.6% | 95.0% |
-| **3-gram** | 1,561 | 10.61 | 4,876 | 32.9% | 74.3% |
-| **4-gram** | 1,077 | 10.07 | 2,302 | 37.1% | 77.9% |
-| **4-gram** | 3,419 | 11.74 | 11,151 | 25.8% | 57.5% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 102 🏆 | 6.68 | 159 | 86.3% | 100.0% |
+| **2-gram** | Subword | 330 | 8.37 | 871 | 59.3% | 100.0% |
+| **3-gram** | Word | 156 | 7.28 | 245 | 72.6% | 100.0% |
+| **3-gram** | Subword | 1,700 | 10.73 | 3,811 | 27.1% | 73.0% |
+| **4-gram** | Word | 310 | 8.27 | 449 | 52.1% | 100.0% |
+| **4-gram** | Subword | 4,072 | 11.99 | 8,559 | 18.3% | 52.6% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `na éstse` | 161 |
+| 2 | `vé ho` | 119 |
+| 3 | `ho énêstsestôtse` | 72 |
+| 4 | `republic of` | 67 |
+| 5 | `ho e` | 57 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `vé ho énêstsestôtse` | 72 |
+| 2 | `na éstse manâhéno` | 56 |
+| 3 | `ho e éve` | 49 |
+| 4 | `na éstse ho` | 48 |
+| 5 | `éstse ho e` | 48 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `éstse ho e éve` | 48 |
+| 2 | `na éstse ho e` | 48 |
+| 3 | `ma kaetaévôxe êstoo o` | 25 |
+| 4 | `toháano éve ho etse` | 23 |
+| 5 | `na éstse manâhéno ho` | 22 |
+
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `category :` | 973 |
-| 2 | `' e` | 663 |
-| 3 | `ho '` | 511 |
-| 4 | `' o` | 391 |
-| 5 | `. category` | 332 |
+| 1 | `e _` | 1,534 |
+| 2 | `s e` | 1,395 |
+| 3 | `s t` | 1,310 |
+| 4 | `t s` | 1,310 |
+| 5 | `h e` | 1,012 |
 
-**3-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `. category :` | 331 |
-| 2 | `na ' éstse` | 288 |
-| 3 | `' ho '` | 225 |
-| 4 | `| thumb |` | 204 |
-| 5 | `| right |` | 201 |
+| 1 | `t s e` | 1,002 |
+| 2 | `s e _` | 580 |
+| 3 | `e s t` | 468 |
+| 4 | `s t s` | 459 |
+| 5 | `h o '` | 443 |
 
-**4-grams:**
+**4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `, na ' éstse` | 199 |
-| 2 | `| thumb | right` | 167 |
-| 3 | `thumb | right |` | 155 |
-| 4 | `vé ' ho '` | 131 |
-| 5 | `300px | thumb |` | 128 |
+| 1 | `t s e _` | 456 |
+| 2 | `s t s e` | 436 |
+| 3 | `ô t s e` | 287 |
+| 4 | `t ô t s` | 208 |
+| 5 | `e s t ô` | 198 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 237
+- **Best Perplexity:** 2-gram (word) with 102
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~58% of corpus
+- **Coverage:** Top-1000 patterns cover ~53% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -178,55 +207,86 @@ Category:Mâhoestôtse`
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.3489 | 1.274 | 2.42 | 4,255 | 65.1% |
-| **1** | 1.3997 | 2.638 | 10.97 | 189 | 0.0% |
-| **2** | 0.1560 | 1.114 | 1.36 | 10,197 | 84.4% |
-| **2** | 1.2345 | 2.353 | 5.27 | 2,073 | 0.0% |
-| **3** | 0.0936 | 1.067 | 1.18 | 13,745 | 90.6% |
-| **3** | 0.6401 | 1.558 | 2.32 | 10,919 | 36.0% |
-| **4** | 0.0555 🏆 | 1.039 | 1.10 | 16,004 | 94.5% |
-| **4** | 0.2796 🏆 | 1.214 | 1.44 | 25,260 | 72.0% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.4118 | 1.330 | 2.00 | 3,383 | 58.8% |
+| **1** | Subword | 1.3726 | 2.589 | 9.72 | 175 | 0.0% |
+| **2** | Word | 0.1118 | 1.081 | 1.20 | 6,516 | 88.8% |
+| **2** | Subword | 1.2032 | 2.303 | 5.05 | 1,699 | 0.0% |
+| **3** | Word | 0.0474 | 1.033 | 1.08 | 7,515 | 95.3% |
+| **3** | Subword | 0.6524 | 1.572 | 2.34 | 8,541 | 34.8% |
+| **4** | Word | 0.0269 🏆 | 1.019 | 1.04 | 7,792 | 97.3% |
+| **4** | Subword | 0.2844 | 1.218 | 1.44 | 19,944 | 71.6% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `e he óonéma enóne éohkê héška ó he hohamháa há continentan naa nêhéóhe násáahéne enomóvóhe tsé`
+2. `ho honáemanėstóoseo o united states manâhestôtse 1 188 lwanda 100px mogadishu somali shilling swahil...`
+3. `o vé keehoohtsêstse vó kaehevôtse vé ho énêstsestôtse billingscheyenne english dictionary chief dull...`
+
+**Context Size 2:**
+
+1. `na éstse ho e éve asia center thumb handelskade in willemstad curaçao`
+2. `vé ho énêstsestôtse black hills ho honáéšé e missouri ó he e pónoeo hé e na éstse`
+3. `ho énêstsestôtse cimarron river bull river forgan heévȧhetanéno`
+
+**Context Size 3:**
+
+1. `vé ho énêstsestôtse bay horse variant tsé vó névóvâtse`
+2. `na éstse manâhéno ho honáéšé e united states óoetaneo o óoetanéno tsé amo eétâhéstove vé ho énêstses...`
+3. `ho e éve hóxovê hooma center frameless upright 1 5`
+
+**Context Size 4:**
+
+1. `éstse ho e éve meško`
+2. `na éstse ho e éve vietnam dong hoi airport`
+3. `ma kaetaévôxe êstoo o sango toháano éve ho etse 622 984 4 216 666 1 198 chad republic of`
 
-### Generated Text Samples
 
-Below are text samples generated from each Markov chain model:
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `' konénėhesó - éve . manâhestôtse 7 heše ' tavö ' éhoo ' he tsénėxhésemé '`
-2. `, na ' he tsénėxhésemé ' evo ' éno ' e 1904 , na ' otsenáhkohe`
-3. `: turkey , na ' o tsétsêhéstâhese - éve . gus . curitiba - éve .`
+1. `e_a_29150002)_te`
+2. `_rkul_mâxpoeése.`
+3. `a_xema'ėh-évese:`
 
 **Context Size 2:**
 
-1. `category : ó ' he ( vé ' ho ' énestse 71 , 740 6 , 418`
-2. `' e 300px | thumb | amâho ' hestôtse amêške tsémo ' ôhtávoome amâho ' hestôtse category`
-3. `ho ' xó ' mâhoéve ' ho ' énêstsestôtse : purgatoire river , picketwire river ) -`
+1. `e_(vé'še'tó'neadd`
+2. `seotó'o_poestôtse`
+3. `ts_wymnetugual_ju`
 
 **Context Size 3:**
 
-1. `. category : mâhoestôtse category : california`
-2. `na ' éstse ho ' e - éve hóxovê - hooma , asia ) . *`
-3. `' ho ' e - éve , meško . category : mâhoestôtse category : ho ' honáeo '`
+1. `tse._vé'ho'hé'e_bo`
+2. `se_rokese_mâhestȯt`
+3. `estôtse_manâhá'e_(`
 
 **Context Size 4:**
 
-1. `, na ' éstse ho ' e - éve , amérika . *`
-2. `| thumb | right | hóxeeséeto ' hamestôtse 300px | thumb | right | méstaa ' êhéhe category :`
-3. `thumb | right | hotóhkeo ' o tsénésôhtôxese 300px | thumb | right | mámaa ' e mámaa '`
+1. `tse_hotómá'e_12_évȯ`
+2. `stseévenomo_hovanan`
+3. `ôtse:_ten_sage";_ar`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 94.5% predictability
+- **Best Predictability:** Context-4 (word) with 97.3% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (25,260 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (19,944 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -242,64 +302,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 1,659 |
-| Total Tokens | 14,360 |
-| Mean Frequency | 8.66 |
+| Vocabulary Size | 1,237 |
+| Total Tokens | 8,401 |
+| Mean Frequency | 6.79 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 38.60 |
+| Frequency Std Dev | 21.86 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | category | 974 |
-| 2 | e | 690 |
-| 3 | ho | 531 |
-| 4 | o | 414 |
-| 5 | na | 293 |
-| 6 | éstse | 288 |
-| 7 | right | 260 |
-| 8 | éve | 259 |
-| 9 | thumb | 226 |
-| 10 | vé | 180 |
+| 1 | e | 431 |
+| 2 | ho | 377 |
+| 3 | o | 237 |
+| 4 | na | 165 |
+| 5 | vé | 161 |
+| 6 | éstse | 161 |
+| 7 | éve | 154 |
+| 8 | of | 118 |
+| 9 | he | 108 |
+| 10 | naa | 108 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | evenóse | 2 |
-| 2 | mountain | 2 |
-| 3 | cal | 2 |
-| 4 | poly | 2 |
-| 5 | mustangs | 2 |
-| 6 | sevonévo | 2 |
-| 7 | ėstovátamevéotse | 2 |
-| 8 | ėstova | 2 |
-| 9 | nėstse | 2 |
-| 10 | 2025 | 2 |
+| 1 | mustangs | 2 |
+| 2 | sevonévo | 2 |
+| 3 | ėstovátamevéotse | 2 |
+| 4 | ėstova | 2 |
+| 5 | nėstse | 2 |
+| 6 | kūnas | 2 |
+| 7 | epsteins | 2 |
+| 8 | ir | 2 |
+| 9 | felon | 2 |
+| 10 | immigrants | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.8829 |
-| R² (Goodness of Fit) | 0.980523 |
+| Zipf Coefficient | 0.8151 |
+| R² (Goodness of Fit) | 0.976018 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 57.7% |
-| Top 1,000 | 90.8% |
+| Top 100 | 54.8% |
+| Top 1,000 | 94.4% |
 | Top 5,000 | 0.0% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9805 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 57.7% of corpus
-- **Long Tail:** -8,341 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9760 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 54.8% of corpus
+- **Long Tail:** -8,763 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -312,24 +372,109 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 223 | 32 | 1.577 | 0.877 | 0.0028 🏆 |
-| **mono_64d** | 223 | 64 | 1.556 | 0.897 | 0.0009 |
-| **mono_128d** | 223 | 128 | 1.593 | 0.888 | 0.0002 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.0023 🏆 | 0.8533 | N/A | N/A |
+| **mono_64d** | 64 | 0.0008 | 0.9264 | N/A | N/A |
+| **mono_128d** | 128 | 0.0002 | 0.9821 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.0028 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 223 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.0023 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.9206. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
+
+---
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ho` | horse, hotoa, hoohëö |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-e` | néstovátamevéotse, kôhtse, where |
+| `-se` | néstovátamevéotse, kôhtse, kaehevotôtse |
+| `-tse` | néstovátamevéotse, kôhtse, kaehevotôtse |
+| `-ne` | mâhoestôtsene, kane, mâheóne |
+| `-ôtse` | kaehevotôtse, oestôtse, xemenôtse |
+| `-ia` | anastacia, abkhazia, shepherdia |
+| `-ve` | êstonêstove, native, hestoháatamaahéstove |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+*No significant bound stems detected.*
+
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-ho` | `-e` | 5 words | horse, hováhne |
+| `-ho` | `-ne` | 2 words | hováhne, hovahne |
+| `-ho` | `-se` | 1 words | horse, hotse |
+| `-ho` | `-tse` | 1 words | hotse, hohpâhtsenámenôtse |
+| `-ho` | `-ôtse` | 1 words | hohpâhtsenámenôtse |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| mâhoestôtsene | **`mâhoest-ôtse-ne`** | 3.0 | `mâhoest` |
+| sevoneóneve | **`sevoneó-ne-ve`** | 3.0 | `sevoneó` |
+| náhkȯhehetanetse | **`náhkȯheheta-ne-tse`** | 3.0 | `náhkȯheheta` |
+| enóseoneve | **`enóseo-ne-ve`** | 3.0 | `enóseo` |
+| éestsėstóseoneve | **`éestsėstóseo-ne-ve`** | 3.0 | `éestsėstóseo` |
+| kaehevotôtse | **`kaehevot-ôtse`** | 1.5 | `kaehevot` |
+| anastacia | **`anastac-ia`** | 1.5 | `anastac` |
+| névóvâtse | **`névóvâ-tse`** | 1.5 | `névóvâ` |
+| shepherdia | **`shepherd-ia`** | 1.5 | `shepherd` |
+| êstonêstove | **`êstonêsto-ve`** | 1.5 | `êstonêsto` |
+| yellowstone | **`yellowsto-ne`** | 1.5 | `yellowsto` |
+| hoohtseto | **`ho-ohtseto`** | 1.5 | `ohtseto` |
+| xemenôtse | **`xemen-ôtse`** | 1.5 | `xemen` |
+| véhonevoemėstse | **`véhonevoemės-tse`** | 1.5 | `véhonevoemės` |
+| manestôtse | **`manest-ôtse`** | 1.5 | `manest` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language CHY appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
 
 ---
-## 6. Summary & Recommendations
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -337,11 +482,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (3.46x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (237) |
-| Markov | **Context-4** | Highest predictability (94.5%) |
+| Tokenizer | **8k BPE** | Best compression (3.50x) |
+| N-gram | **2-gram** | Lowest perplexity (102) |
+| Markov | **Context-4** | Highest predictability (97.3%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -531,7 +677,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -547,7 +694,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-28 22:42:59*
+*Report Date: 2026-01-03 10:13:21*

models/embeddings/monolingual/chy_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9abb00efdaee95eb235694ba4738708972eef813303beafecc2c4852c0a11360
-size 1024233321
+oid sha256:fe82c87b0cea7042a6a16f85cf1875011ca32f13b8c941e25d5c1d57849b2eed
+size 1024170112

models/embeddings/monolingual/chy_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 223
+  "vocab_size": 163
 }

models/embeddings/monolingual/chy_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8c8e4f25fd101e1917ce06bf1768de9245362d868e23a5ebbbc943b0e4650a60
-size 256062057
+oid sha256:70c8741bb1d28e9983cdf32850bd345db541b195a40f92c1ebc8ba49374864be
+size 256044928

models/embeddings/monolingual/chy_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 223
+  "vocab_size": 163
 }

models/embeddings/monolingual/chy_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4c920935ac87a1441f862361919b9313ce17a00634fb4b6dc1a1bb483d79a2f5
-size 512119145
+oid sha256:235dc5785bb5f5c2123f5d894c3c7266a0f239882928a7be48b4ddbb112ff63d
+size 512086656

models/embeddings/monolingual/chy_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 223
+  "vocab_size": 163
 }

models/subword_markov/chy_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6ea83b117f3a486981315ed6c06f66c8b179b52afa64753b10adcb037d01a299
-size 19111
+oid sha256:d18af851c234d84f1c8c493282bad10c2ce3cf38e5f884b0def3b69a8cc46a72
+size 16739

models/subword_markov/chy_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "chy",
-  "unique_contexts": 189,
-  "total_transitions": 113177
+  "unique_contexts": 175,
+  "total_transitions": 68722
 }

models/subword_markov/chy_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cc4389e4a77f6c71ab2c352c280e4fc1281ab64c25a40abbcf7af73411115bda
-size 71565
+oid sha256:8ecd88390cc78af12a57e86676cd4ac3d721a2a3b6105dedf56498e4f9198ba6
+size 55410

models/subword_markov/chy_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "chy",
-  "unique_contexts": 2073,
-  "total_transitions": 112352
+  "unique_contexts": 1699,
+  "total_transitions": 68263
 }

models/subword_markov/chy_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e7f1d3cabcf310e4f6579f8d79db5b6ca9e6cb8c218d840ca5d728e17f8a9d72
-size 189294
+oid sha256:fd7301d331a8f6e5cf3904b5bd4befc8f3edf1b29659b35c4c4df3b64b73ed4d
+size 150466

models/subword_markov/chy_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "chy",
-  "unique_contexts": 10919,
-  "total_transitions": 111527
+  "unique_contexts": 8541,
+  "total_transitions": 67804
 }

models/subword_markov/chy_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6f537764e3ba64873ce0be3e727efce7efa3b3ee9a512520fdff8fe7b9f23660
-size 357786
+oid sha256:3e44005f2ed1b58ece31011bfe69d86d11229eda4839c36faadb0b8ac3af5479
+size 281701

models/subword_markov/chy_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "chy",
-  "unique_contexts": 25260,
-  "total_transitions": 110702
+  "unique_contexts": 19944,
+  "total_transitions": 67345
 }

models/subword_ngram/chy_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2ed544f13314cfc2324a3b1462dfcc47e71f1236aeb6319b89b7d6e18ac3920a
-size 14546
+oid sha256:0228df27f63ed413c576b4cbe4ab6f76c9785023f717763c1927ee318cba7061
+size 11557

models/subword_ngram/chy_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "chy",
-  "unique_ngrams": 1127,
-  "total_ngrams": 113177
+  "unique_ngrams": 871,
+  "total_ngrams": 68722
 }

models/subword_ngram/chy_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4b077e5de241ab6ccaa7e1f45221e1202536459be7a01c5b4b63e9fd20ba730d
-size 54182
+oid sha256:c6c8121df7c4fff9c78c7deca418c3eaecb7394cf750b4681239ca320d9977e8
+size 41181

models/subword_ngram/chy_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "chy",
-  "unique_ngrams": 4876,
-  "total_ngrams": 112352
+  "unique_ngrams": 3811,
+  "total_ngrams": 68263
 }

models/subword_ngram/chy_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9b1c577153cd04ac40d0b50f9a07ec2270c218aff4d156a1ece01fd0edfec031
-size 130151
+oid sha256:4bfb26d6de423f9bad18bfb5f3a57a90fada9cbd23138dc9ac7e2fcdbe0c7831
+size 100088

models/subword_ngram/chy_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "chy",
-  "unique_ngrams": 11151,
-  "total_ngrams": 111527
+  "unique_ngrams": 8559,
+  "total_ngrams": 67804
 }

models/tokenizer/chy_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:17ad6a6d3558277d6925eb68f3855cb005e2fe3557d8485bf68a6f7274296dab
-size 375317
+oid sha256:7224c406ce7f0ceae9ae28bfb815c3ebc81aafabd231728ea1823e5477c23e0d
+size 374664

models/vocabulary/chy_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:470592a0b112b9a3a65f69af1557219cccb407f27f3a4236e60e71c7c2f40313
-size 29068
+oid sha256:557659dcccab98922c4fc45c22d401a01f92a33330e2d06cbdbafd9e3d8f37d2
+size 22392

models/vocabulary/chy_vocabulary_metadata.json

@@ -1,14 +1,15 @@
 {
   "language": "chy",
-  "vocabulary_size": 1659,
+  "vocabulary_size": 1237,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.24974895150333748,
+    "type_token_ratio": 0.32707120045087357,
     "coverage": {
-      "top_100": 0.4891015417331207,
-      "top_1000": 0.7703939984641739
+      "top_100": 0.4324628968626714,
+      "top_1000": 0.7445989103888785
     },
-    "hapax_count": 2569,
-    "hapax_ratio": 0.6076158940397351,
-    "total_documents": 825
+    "hapax_count": 2245,
+    "hapax_ratio": 0.644744399770247,
+    "total_documents": 459
   }
 }

models/word_markov/chy_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6fb77d185c917a1f8ece3debd54279262bbc2e1a01799fcb691731a7561bee31
-size 113562
+oid sha256:386ae58f4271bb09f2a7300c1ffb43cb66268f9329c798213791212b7375c544
+size 86902

models/word_markov/chy_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "chy",
-  "unique_contexts": 4255,
-  "total_transitions": 27559
+  "unique_contexts": 3383,
+  "total_transitions": 10187
 }

models/word_markov/chy_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c1602a760151707012bb6aef66d181795727fa9d8cb5780233a0b6c1c90a2312
-size 189750
+oid sha256:25b29a5fd2a035a5761b624f9763e3b2f1bd7218a31608a0a6f94197490478db
+size 131833

models/word_markov/chy_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "chy",
-  "unique_contexts": 10197,
-  "total_transitions": 26734
+  "unique_contexts": 6516,
+  "total_transitions": 9728
 }

models/word_markov/chy_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7a91e6dbb80dac8f5da14abcdfc3ba5c7122ed0a171bc88e3d506ea5a62e4790
-size 251639
+oid sha256:b4b7d0cf8b279e25f0c1dd5d12c4331f6647670846d67a4e626aa888e8687564
+size 158524

models/word_markov/chy_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "chy",
-  "unique_contexts": 13745,
-  "total_transitions": 25909
+  "unique_contexts": 7515,
+  "total_transitions": 9269
 }

models/word_markov/chy_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b827d8d8deb6ee8ee5a1c2f7e9164c6009220d08c889406bed208c143c67ef6f
-size 299200
+oid sha256:46f60b22a8fcac07a7de898ad66d40c570966ec70737f00b5cf2d7b22bf2c6a2
+size 174153

models/word_markov/chy_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "chy",
-  "unique_contexts": 16004,
-  "total_transitions": 25085
+  "unique_contexts": 7792,
+  "total_transitions": 8810
 }

models/word_ngram/chy_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0dc44fada7846034391ea2690c56b864a908054940a91ca581f91cf63ba43b79
-size 11396
+oid sha256:f3cf90b3734f097b7881f2bd6766a12f1c1a78a49ef2c9af43b8bb1c04f684de
+size 5216

models/word_ngram/chy_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "chy",
-  "unique_ngrams": 654,
-  "total_ngrams": 27559
+  "unique_ngrams": 159,
+  "total_ngrams": 10187
 }

models/word_ngram/chy_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ac58b70ff5ea478cd859a06e2722ba50d36f1d0a7d83c8e35282a144a67b1369
-size 20739
+oid sha256:18e0cd0236f9672b7cdc3748740203bb789cab4d98a9bba4f58941ef3779da01
+size 7252

models/word_ngram/chy_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "chy",
-  "unique_ngrams": 1211,
-  "total_ngrams": 26734
+  "unique_ngrams": 245,
+  "total_ngrams": 9728
 }

models/word_ngram/chy_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8fb9ac65e984d6fb53483e3e4ddaf089e90baeef0b877dc6f9e5067d9f4d0a9d
-size 39062
+oid sha256:be5660989ec57f68e126189b60da16af9da01d4a13b7e53d4bc22f9849e36215
+size 11340

models/word_ngram/chy_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "chy",
-  "unique_ngrams": 2302,
-  "total_ngrams": 25909
+  "unique_ngrams": 449,
+  "total_ngrams": 9269
 }