Upload all models and assets for atj (20251001)

README.md

@@ -23,14 +23,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 5.808
+    value: 5.949
   - name: best_isotropy
     type: isotropy
-    value: 0.1478
+    value: 0.1619
   - name: vocabulary_size
     type: vocab
-    value: 6720
-generated: 2025-12-27
+    value: 0
+generated: 2026-01-03
 ---
 
 # ATJ - Wikilangs Models
@@ -44,12 +44,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +60,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,47 +70,53 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 4.986x | 4.94 | 0.2011% | 100,930 |
-| **16k** | 5.358x | 5.31 | 0.2161% | 93,917 |
-| **32k** | 5.808x 🏆 | 5.76 | 0.2343% | 86,639 |
+| **8k** | 5.115x | 5.13 | 0.1890% | 92,078 |
+| **16k** | 5.507x | 5.52 | 0.2035% | 85,522 |
+| **32k** | 5.949x 🏆 | 5.96 | 0.2198% | 79,160 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Nipicapo ka pictewakamik e wapakaminikaniwok.  Ciwakamin kaie micta wikicin mito...`
+**Sample 1:** `Thetford Mines oteno Kepek askik ici actew, Kanata. Irikik e tacinaniwok 25 649 ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁nipicapo ▁ka ▁picte wakamik ▁e ▁wapa kami ni kaniwok . ... (+19 more)` | 29 |
-| 16k | `▁nipicapo ▁ka ▁pictewakamik ▁e ▁wapa kami nikaniwok . ▁ciwakamin ▁kaie ... (+14 more)` | 24 |
-| 32k | `▁nipicapo ▁ka ▁pictewakamik ▁e ▁wapakami nikaniwok . ▁ciwakamin ▁kaie ▁micta ... (+12 more)` | 22 |
+| 8k | `▁the t ford ▁mi ne s ▁oteno ▁kepek ▁askik ▁ici ... (+15 more)` | 25 |
+| 16k | `▁thetford ▁mi nes ▁oteno ▁kepek ▁askik ▁ici ▁actew , ▁kanata ... (+12 more)` | 22 |
+| 32k | `▁thetford ▁mines ▁oteno ▁kepek ▁askik ▁ici ▁actew , ▁kanata . ... (+11 more)` | 21 |
 
-**Sample 2:** `Fossambault-sur-le-Lac oteno Kepek askik ici actew, Kanata. Irikik e tacinaniwok...`
+**Sample 2:** `Ka Oskiskakamaksource CNA - Atikamekw Kinokewin, sakihikan Kepek askik ici actew...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁fo ssa mb ault - sur - le - lac ... (+30 more)` | 40 |
-| 16k | `▁fo ssa mb ault - sur - le - lac ... (+30 more)` | 40 |
-| 32k | `▁fossambault - sur - le - lac ▁oteno ▁kepek ▁askik ... (+27 more)` | 37 |
+| 8k | `▁ka ▁oski skakamak source ▁cna ▁- ▁atikamekw ▁kino kewin , ... (+9 more)` | 19 |
+| 16k | `▁ka ▁oski skakamak source ▁cna ▁- ▁atikamekw ▁kinokewin , ▁sakihikan ... (+8 more)` | 18 |
+| 32k | `▁ka ▁oskiskakamak source ▁cna ▁- ▁atikamekw ▁kinokewin , ▁sakihikan ▁kepek ... (+7 more)` | 17 |
 
-**Sample 3:** `Saint-Basile-le-Grand oteno Kepek askik ici actew, Kanata. Irikik e tacinaniwok ...`
+**Sample 3:** `Stellarton oteno Nouvelle-Écosse aski ici actew, Kanata. Irikik e tacinaniwok 4 ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁saint - ba sile - le - grand ▁oteno ▁kepek ... (+30 more)` | 40 |
-| 16k | `▁saint - basile - le - grand ▁oteno ▁kepek ▁askik ... (+29 more)` | 39 |
-| 32k | `▁saint - basile - le - grand ▁oteno ▁kepek ▁askik ... (+29 more)` | 39 |
+| 8k | `▁ste lla r ton ▁oteno ▁nouvelle - écosse ▁aski ▁ici ... (+14 more)` | 24 |
+| 16k | `▁ste lla r ton ▁oteno ▁nouvelle - écosse ▁aski ▁ici ... (+14 more)` | 24 |
+| 32k | `▁stellarton ▁oteno ▁nouvelle - écosse ▁aski ▁ici ▁actew , ▁kanata ... (+11 more)` | 21 |
 
 
 ### Key Findings
 
-- **Best Compression:** 32k achieves 5.808x compression
-- **Lowest UNK Rate:** 8k with 0.2011% unknown tokens
+- **Best Compression:** 32k achieves 5.949x compression
+- **Lowest UNK Rate:** 8k with 0.1890% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -117,57 +125,89 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 862 🏆 | 9.75 | 3,083 | 46.3% | 79.6% |
-| **2-gram** | 152 🏆 | 7.25 | 1,194 | 85.3% | 99.9% |
-| **3-gram** | 572 | 9.16 | 3,170 | 56.1% | 80.5% |
-| **3-gram** | 883 | 9.79 | 7,630 | 40.9% | 90.0% |
-| **4-gram** | 615 | 9.26 | 4,529 | 57.5% | 75.7% |
-| **4-gram** | 3,239 | 11.66 | 24,127 | 22.8% | 65.1% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 756 | 9.56 | 2,026 | 44.6% | 84.1% |
+| **2-gram** | Subword | 129 🏆 | 7.01 | 992 | 88.9% | 100.0% |
+| **3-gram** | Word | 540 | 9.08 | 1,856 | 50.0% | 84.6% |
+| **3-gram** | Subword | 761 | 9.57 | 5,493 | 41.8% | 92.5% |
+| **4-gram** | Word | 578 | 9.18 | 2,537 | 50.5% | 75.6% |
+| **4-gram** | Subword | 3,042 | 11.57 | 19,249 | 21.7% | 65.9% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ici actew` | 889 |
+| 2 | `actew kanata` | 771 |
+| 3 | `manawan wemotaci` | 722 |
+| 4 | `e ici` | 686 |
+| 5 | `irikik e` | 672 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ici actew kanata` | 770 |
+| 2 | `irikik e tacinaniwok` | 633 |
+| 3 | `kanata irikik e` | 620 |
+| 4 | `actew kanata irikik` | 620 |
+| 5 | `askik ici actew` | 500 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `kanata irikik e tacinaniwok` | 620 |
+| 2 | `actew kanata irikik e` | 620 |
+| 3 | `ici actew kanata irikik` | 620 |
+| 4 | `askik ici actew kanata` | 490 |
+| 5 | `kepek askik ici actew` | 457 |
+
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `tipanictawin :` | 2,442 |
-| 2 | `: e` | 1,606 |
-| 3 | `e matcectakaniwok` | 1,602 |
-| 4 | `ici actew` | 889 |
-| 5 | `matcectakaniwok tipanictawin` | 812 |
+| 1 | `c i` | 23,693 |
+| 2 | `k a` | 23,558 |
+| 3 | `_ k` | 23,282 |
+| 4 | `t c` | 23,205 |
+| 5 | `i k` | 21,042 |
 
-**3-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `tipanictawin : e` | 1,604 |
-| 2 | `: e matcectakaniwok` | 1,602 |
-| 3 | `matcectakaniwok tipanictawin :` | 812 |
-| 4 | `e matcectakaniwok tipanictawin` | 811 |
-| 5 | `ici actew ,` | 774 |
+| 1 | `t c i` | 11,312 |
+| 2 | `_ k i` | 10,112 |
+| 3 | `i t c` | 10,006 |
+| 4 | `_ k a` | 9,178 |
+| 5 | `c i _` | 8,654 |
 
-**4-grams:**
+**4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `tipanictawin : e matcectakaniwok` | 1,602 |
-| 2 | `: e matcectakaniwok tipanictawin` | 811 |
-| 3 | `e matcectakaniwok tipanictawin :` | 811 |
-| 4 | `ici actew , kanata` | 759 |
-| 5 | `actew , kanata .` | 756 |
+| 1 | `i t c i` | 5,889 |
+| 2 | `a n i w` | 5,154 |
+| 3 | `_ k a _` | 4,777 |
+| 4 | `n i w o` | 4,370 |
+| 5 | `k a n i` | 4,232 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 152
+- **Best Perplexity:** 2-gram (subword) with 129
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~65% of corpus
+- **Coverage:** Top-1000 patterns cover ~66% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -175,55 +215,86 @@ Below are sample sentences tokenized with each vocabulary size:
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.5314 | 1.445 | 3.58 | 20,568 | 46.9% |
-| **1** | 1.7637 | 3.396 | 16.31 | 126 | 0.0% |
-| **2** | 0.2296 | 1.172 | 1.52 | 73,484 | 77.0% |
-| **2** | 1.2892 | 2.444 | 6.48 | 2,049 | 0.0% |
-| **3** | 0.0715 | 1.051 | 1.12 | 111,484 | 92.9% |
-| **3** | 0.9367 | 1.914 | 3.52 | 13,270 | 6.3% |
-| **4** | 0.0212 🏆 | 1.015 | 1.04 | 124,488 | 97.9% |
-| **4** | 0.5020 🏆 | 1.416 | 2.09 | 46,658 | 49.8% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.5831 | 1.498 | 3.56 | 19,290 | 41.7% |
+| **1** | Subword | 1.5451 | 2.918 | 13.90 | 118 | 0.0% |
+| **2** | Word | 0.1879 | 1.139 | 1.41 | 67,726 | 81.2% |
+| **2** | Subword | 1.2667 | 2.406 | 6.32 | 1,639 | 0.0% |
+| **3** | Word | 0.0530 | 1.037 | 1.09 | 93,891 | 94.7% |
+| **3** | Subword | 0.7981 | 1.739 | 3.30 | 10,345 | 20.2% |
+| **4** | Word | 0.0145 🏆 | 1.010 | 1.02 | 100,093 | 98.6% |
+| **4** | Subword | 0.5495 | 1.464 | 2.26 | 34,073 | 45.1% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
 
-### Generated Text Samples
+**Context Size 1:**
+
+1. `e takonikatek anihe wikasi aka nipoane aka ewi tipapasotc tawatcikaniw apitcitaw iskwew kata nespito...`
+2. `ka icinikasotc ki nti matce kisinarik micta kackitatc e ici sikinikatek rasop e wamowsotc kaskina wi...`
+3. `ki micta kackitatc nictam ka ickwa aiketcik mitcetowaw nikamohinik tekera weckatc nehirowisikw ni ap...`
+
+**Context Size 2:**
+
+1. `ici actew kanata irikik e tacinaniwok matcectakaniwok`
+2. `actew kanata matcectakaniwok opitciwan matcectakaniwok matcectakaniwok`
+3. `manawan wemotaci nabesipi sipi ekote e otci katcitcipitak niheriw kitci aititosketc mitowi ka takoki...`
+
+**Context Size 3:**
+
+1. `ici actew kanata irikik e tacinaniwok 5 037 matcectakaniwok`
+2. `kanata irikik e tacinaniwok 7 200 oteno ote itekera ka icitiperitakok comté portneuf rareak micta si...`
+3. `actew kanata irikik e tacinaniwok 71 419 matcectakaniwok`
+
+**Context Size 4:**
 
-Below are text samples generated from each Markov chain model:
+1. `ici actew kanata irikik e tacinaniwok 403 390 matcectakaniwok`
+2. `actew kanata irikik e tacinaniwok 3 930 matcectakaniwok`
+3. `kanata irikik e tacinaniwok 552`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. kaskina kekocic ocitakaniwon ickwatem aka kitci wectcikaniwok mitwi mitcet wapamew e matcectakaniw...`
-2. `e totwakiniwitc , ekota awik e kicekawok " type " : e ici matisitcik tca mia`
-3. `, e matcectakaniwok tipanictawin : sakihikana . irikik e ki masinahotiso kitci ki pekociw ni apitc`
+1. `i_w_acikaraska_p`
+2. `_thitcicisan_ta_`
+3. `ak_kitciwik_ours`
 
 **Context Size 2:**
 
-1. `tipanictawin : e matcectakaniwok ‎ tipanictawin : e matcectakaniwok tipanictawin : nehirowisi otenow...`
-2. `: e matcectakaniwok tipanictawin : e matcectakaniwok tipanictawin : e matcectakaniwok tipanictawin :...`
-3. `e matcectakaniwok tipanictawin : e matcectakaniwok ‎ taci we otciparik anihe itewin . nictam awacak ...`
+1. `cira._ok_takanetc`
+2. `katcik._ka_ictert`
+3. `_koki_sinikaniwan`
 
 **Context Size 3:**
 
-1. `tipanictawin : e matcectakaniwok tipanictawin : otenowa tipanictawin : e matcectakaniwok ‎ tipanicta...`
-2. `: e matcectakaniwok ‎ tipanictawin : pirecicak grand héron`
-3. `matcectakaniwok tipanictawin : otenowa tipanictawin : e matcectakaniwok tipanictawin : otenowa tipan...`
+1. `tcikamooseph_du_qu`
+2. `_kirikanikaniwok._`
+3. `itc_iskakwasotc._e`
 
 **Context Size 4:**
 
-1. `tipanictawin : e matcectakaniwok tipanictawin : otenowa tipanictawin : e matcectakaniwok ‎ tipanicta...`
-2. `: e matcectakaniwok tipanictawin : otenowa tipanictawin : e matcectakaniwok tipanictawin : otenowa t...`
-3. `e matcectakaniwok tipanictawin : otenowa tipanictawin : e matcectakaniwok ‎ tipanictawin : ka kisker...`
+1. `itciwan_nehiriwa_on`
+2. `aniwiw_ka_taci_matc`
+3. `_ka_iti_ici_nictahi`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 97.9% predictability
+- **Best Predictability:** Context-4 (word) with 98.6% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (46,658 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (34,073 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -239,35 +310,35 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 6,720 |
-| Total Tokens | 113,194 |
-| Mean Frequency | 16.84 |
+| Vocabulary Size | 6,479 |
+| Total Tokens | 105,209 |
+| Mean Frequency | 16.24 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 144.86 |
+| Frequency Std Dev | 131.12 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | e | 7,966 |
-| 2 | ka | 4,835 |
-| 3 | ki | 3,659 |
-| 4 | ici | 2,657 |
-| 5 | tipanictawin | 2,442 |
-| 6 | kitci | 1,874 |
-| 7 | kaie | 1,655 |
-| 8 | matcectakaniwok | 1,604 |
-| 9 | micta | 1,222 |
-| 10 | kirika | 1,112 |
+| 1 | e | 6,370 |
+| 2 | ka | 4,817 |
+| 3 | ki | 3,656 |
+| 4 | ici | 2,654 |
+| 5 | kitci | 1,874 |
+| 6 | kaie | 1,655 |
+| 7 | matcectakaniwok | 1,604 |
+| 8 | micta | 1,222 |
+| 9 | kirika | 1,111 |
+| 10 | manawan | 973 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | nehirosi | 2 |
-| 2 | cikomewokw | 2 |
-| 3 | miitaw | 2 |
-| 4 | droits | 2 |
+| 1 | cikomewokw | 2 |
+| 2 | miitaw | 2 |
+| 3 | droits | 2 |
+| 4 | ntokiw | 2 |
 | 5 | kiskinohamato | 2 |
 | 6 | banque | 2 |
 | 7 | mawotcicorianionik | 2 |
@@ -279,24 +350,24 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.0564 |
-| R² (Goodness of Fit) | 0.988465 |
+| Zipf Coefficient | 1.0501 |
+| R² (Goodness of Fit) | 0.987715 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 54.9% |
-| Top 1,000 | 81.7% |
-| Top 5,000 | 97.0% |
+| Top 100 | 54.5% |
+| Top 1,000 | 81.8% |
+| Top 5,000 | 97.2% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9885 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 54.9% of corpus
-- **Long Tail:** -3,280 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9877 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 54.5% of corpus
+- **Long Tail:** -3,521 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -309,24 +380,133 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 2,371 | 32 | 2.682 | 0.752 | 0.1478 🏆 |
-| **mono_64d** | 2,371 | 64 | 2.686 | 0.746 | 0.0305 |
-| **mono_128d** | 2,371 | 128 | 2.694 | 0.728 | 0.0055 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+> *Note: Multilingual alignment visualization not available for this language.*
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.1619 🏆 | 0.4893 | N/A | N/A |
+| **mono_64d** | 64 | 0.0330 | 0.5001 | N/A | N/A |
+| **mono_128d** | 128 | 0.0058 | 0.5012 | N/A | N/A |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.1478 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 2,371 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.1619 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.4969. Lower values indicate better semantic separation.
+- **Alignment Quality:** No aligned models evaluated in this run.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
+| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-ki` | kictapatisiw, kiwew, kiskinohomakewiniw |
+| `-mi` | mikomin, mitataw, mirokotek |
+| `-ma` | mamowinikatek, masinatew, manikaniwon |
+| `-ni` | ninan, nikikw, niheritamokw |
+| `-ot` | otcepikiwitc, otactimikok, otcotcoma |
+| `-ta` | tanetci, tatiniw, tatow |
+| `-ic` | icakopan, icinikatakiniw, icinikatakaniwitcik |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-k` | nosinetakaniwonik, rarewak, mamowinikatek |
+| `-w` | kictapatisiw, masinatew, kiwew |
+| `-c` | awotatokwetc, otcepikiwitc, taciketc |
+| `-n` | potatcikan, mikomin, icakopan |
+| `-ik` | nosinetakaniwonik, pakacik, icinikatakaniwitcik |
+| `-tc` | awotatokwetc, otcepikiwitc, taciketc |
+| `-iw` | kictapatisiw, kiskinohomakewiniw, icinikatakiniw |
+| `-ok` | petakok, otactimikok, wapitamok |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `taka` | 1.44x | 22 contexts | otakai, pataka, matakaw |
+| `tako` | 1.32x | 29 contexts | takon, takok, takoki |
+| `apit` | 1.50x | 17 contexts | apitc, apita, tapit |
+| `atis` | 1.49x | 17 contexts | matisiw, matisin, batiste |
+| `mitc` | 1.33x | 22 contexts | mitca, mitci, mitcin |
+| `aniw` | 1.38x | 19 contexts | aniwe, nikaniw, oskaniw |
+| `iwok` | 1.43x | 16 contexts | apiwok, aipiwok, irniwok |
+| `erit` | 1.48x | 14 contexts | iteritam, iteritak, oreritam |
+| `niwo` | 1.51x | 13 contexts | irniwok, koniwok, kaniwok |
+| `tcik` | 1.30x | 19 contexts | tatcik, mitcik, motcik |
+| `irow` | 1.56x | 11 contexts | kirowe, wirowow, kewirow |
+| `kate` | 1.33x | 16 contexts | katek, makate, kateri |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-ki` | `-k` | 127 words | kiskeritakik, kiskeritakositcik |
+| `-ma` | `-k` | 89 words | masinapiskikatek, mackikirinikwecik |
+| `-mi` | `-k` | 88 words | mitcenik, mickaniwok |
+| `-ki` | `-w` | 68 words | kiskinohamakewiniw, kictaw |
+| `-mi` | `-w` | 65 words | miromakosiw, mitcetwaw |
+| `-ni` | `-k` | 61 words | nitwakik, nisitowinikatek |
+| `-ot` | `-k` | 57 words | otek, otcirowek |
+| `-ki` | `-ik` | 56 words | kiskeritakik, kiskeritakositcik |
+| `-ki` | `-c` | 51 words | kicterimitisotc, kiciwahikotc |
+| `-ta` | `-k` | 49 words | tacikewok, takociparitcik |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| kitotakaniw | **`ki-totak-an-iw`** | 7.5 | `totak` |
+| pimatisinaniwok | **`pimatisin-an-iw-ok`** | 7.5 | `pimatisin` |
+| icipekahikaniwok | **`ic-ipekah-ik-an-iw-ok`** | 7.5 | `ipekah` |
+| masinahikaniwok | **`ma-sinah-ik-an-iw-ok`** | 7.5 | `sinah` |
+| icitatcik | **`ic-itat-cik`** | 6.0 | `itat` |
+| masinahikanik | **`ma-sinah-ik-an-ik`** | 6.0 | `sinah` |
+| mipariwakaniwok | **`mi-pariwak-an-iw-ok`** | 6.0 | `pariwak` |
+| osawapisikaniwok | **`osawapis-ik-an-iw-ok`** | 6.0 | `osawapis` |
+| matcehonaniwok | **`ma-tcehon-an-iw-ok`** | 6.0 | `tcehon` |
+| nipimatisiwinik | **`ni-pimatisiwin-ik`** | 6.0 | `pimatisiwin` |
+| misinhikaniw | **`mi-sinh-ik-an-iw`** | 6.0 | `sinh` |
+| nikamonaniwok | **`ni-kamon-an-iw-ok`** | 6.0 | `kamon` |
+| metowaniwok | **`metow-an-iw-ok`** | 4.5 | `metow` |
+| miremakanik | **`mi-remak-an-ik`** | 4.5 | `remak` |
+| acamakaniwok | **`acamak-an-iw-ok`** | 4.5 | `acamak` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language ATJ appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -334,11 +514,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (5.81x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (152) |
-| Markov | **Context-4** | Highest predictability (97.9%) |
+| Tokenizer | **32k BPE** | Best compression (5.95x) |
+| N-gram | **2-gram** | Lowest perplexity (129) |
+| Markov | **Context-4** | Highest predictability (98.6%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -528,7 +709,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -544,7 +726,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-27 20:37:02*
+*Report Date: 2026-01-03 05:18:59*

models/embeddings/monolingual/atj_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5e4f652757f81386724dd7cf5c8d22549280e2e24aabab3b92abac08935d3c4a
-size 1026469326
+oid sha256:7f3d6139699a8a77dd41505f198f1088f6ea783407b1398a6369141ef13d2adc
+size 1026354933

models/embeddings/monolingual/atj_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 2371
+  "vocab_size": 2261
 }

models/embeddings/monolingual/atj_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1aac0c6be5de24ce9a05654a95b0945cccec0cac9fee51d6b0dfbb3df8672f74
-size 256648398
+oid sha256:733947e40bfc2a29297e78ab7590c70db193bca524959bb47af3e2ea45087684
+size 256618485

models/embeddings/monolingual/atj_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 2371
+  "vocab_size": 2261
 }

models/embeddings/monolingual/atj_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2e026ba4da09c003162df384f6aa6973c82d536f19dc68f82368126723c690cb
-size 513255374
+oid sha256:cf1bb93977201f5841f00da8ac2e6f2915a651bf8892a51f91d3c84233f3fae3
+size 513197301

models/embeddings/monolingual/atj_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 2371
+  "vocab_size": 2261
 }

models/subword_markov/atj_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d39bea41083fabbd4be030111bb4376a8345ee2c24cec3e7b066979621d62313
-size 19976
+oid sha256:01ab1d155ad7b2173943a42ef233b27a6456b68458231c9472df823263be9ec6
+size 16979

models/subword_markov/atj_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "atj",
-  "unique_contexts": 126,
-  "total_transitions": 971476
+  "unique_contexts": 118,
+  "total_transitions": 875958
 }

models/subword_markov/atj_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f0fe5cfd0b592f5cd30abbcc82868c6ee2c08671541e1e9c0d7aae11b4877a0a
-size 101456
+oid sha256:01c646d4de1b50ab2af2e42b22f63c9283f46a99722629ba2187c5b17039d034
+size 80338

models/subword_markov/atj_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "atj",
-  "unique_contexts": 2049,
-  "total_transitions": 969388
+  "unique_contexts": 1639,
+  "total_transitions": 873879
 }

models/subword_markov/atj_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:283a408f8f1bcb986dfc6a2779a3fb12448ab6bbe5183f07e442fbfe885e28a5
-size 337589
+oid sha256:d29a79e07eb2572a881d989c62b2b464a2e86ce95f9e1ca1cfc17ebb6f7fdda4
+size 260964

models/subword_markov/atj_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "atj",
-  "unique_contexts": 13270,
-  "total_transitions": 967300
+  "unique_contexts": 10345,
+  "total_transitions": 871800
 }

models/subword_markov/atj_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7d18d91899c8a29a661cfc050414a089b1a63c9c87ca71a786329c19df6b48ad
-size 812220
+oid sha256:c92961d642ccb93882ab7a777fc55a418e457e8f9843b90ea4d3869eb4115b68
+size 635430

models/subword_markov/atj_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "atj",
-  "unique_contexts": 46658,
-  "total_transitions": 965212
+  "unique_contexts": 34073,
+  "total_transitions": 869721
 }

models/subword_ngram/atj_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:511b2678c1e7285952fdcf58c2a8d8a86667a817f0871985f9327da706d62c1f
-size 16760
+oid sha256:83d8aa6cff41c2e4e7567422c922f50fad121f6d85606b56af394b2714de92de
+size 14344

models/subword_ngram/atj_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "atj",
-  "unique_ngrams": 1194,
-  "total_ngrams": 971476
+  "unique_ngrams": 992,
+  "total_ngrams": 875958
 }

models/subword_ngram/atj_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b0ec7189eab62ae8af249edf91953dbc77bf9ffb24f1864289cbbd10d584e18f
-size 88729
+oid sha256:8cde20675b31ba779a55241e29b9edb6286c6f51bb3fa511c63b49f3d14f3e10
+size 65963

models/subword_ngram/atj_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "atj",
-  "unique_ngrams": 7630,
-  "total_ngrams": 969388
+  "unique_ngrams": 5493,
+  "total_ngrams": 873879
 }

models/subword_ngram/atj_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6e75807d9f320a6f542d1776448ca86b0607bc7ad4c3c997ae27386964cafe9f
-size 288361
+oid sha256:c7765e9eede8cde44f7885c15dc01158f7b11d31566539ee029e4555794cf9cf
+size 231317

models/subword_ngram/atj_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "atj",
-  "unique_ngrams": 24127,
-  "total_ngrams": 967300
+  "unique_ngrams": 19249,
+  "total_ngrams": 871800
 }

models/tokenizer/atj_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a063ccee59eb9db3942d7290d6ff30567b41f568b01820e485d279d8849d9849
-size 533850
+oid sha256:087da7688c1626c425cee991b2afa0071074d6ba52750f654d69b2ab05b54160
+size 536207

models/tokenizer/atj_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e5f7bdd1eb535e2ec8d48cea6bf379c07f11dfd7923a25bdeb412bf5e8de7a49
-size 855496
+oid sha256:e8fa5e1ab7cf748ada69603f315ae6c44dbb95d4e07cabda9709bfbe2f875a70
+size 846451

models/tokenizer/atj_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a9511d57125eca644a7e92c031fb618e671127112c3f8ae4fb8d86bba50f9f88
-size 381099
+oid sha256:678b99dd207da5596a12f7a9f3f36ad762fad42d71893cebac1f4c67048caa88
+size 384081

models/vocabulary/atj_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bd2aba50829aac40db6f44873c3c728b3e7032be8f025cb9a1f670041e355de9
-size 110501
+oid sha256:00f58d6e6175ddabe7c28e84d3c39c5e5ba1832a75f401ec087f627ab36c7ca5
+size 106314

models/vocabulary/atj_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "atj",
-  "vocabulary_size": 6720,
+  "vocabulary_size": 6479,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.16166825450565717,
+    "type_token_ratio": 0.16444512148678497,
     "coverage": {
-      "top_100": 0.48908327887439274,
-      "top_1000": 0.7285031533694993,
-      "top_5000": 0.8641570937034967,
-      "top_10000": 0.9170675632051777
+      "top_100": 0.485278560607984,
+      "top_1000": 0.7279643875728878,
+      "top_5000": 0.865353204526028,
+      "top_10000": 0.9201851710801364
     },
-    "hapax_count": 13813,
-    "hapax_ratio": 0.6727219597720743,
-    "total_documents": 2088
+    "hapax_count": 12952,
+    "hapax_ratio": 0.6665637383562348,
+    "total_documents": 2079
   }
 }

models/word_markov/atj_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5cc0671b8feee03a545436e64c2433b0fae5929afd839c95895a1dd7a786b31b
-size 647870
+oid sha256:bdf082d2a0a4824649c5b861ab9afabf286d0dbb86ad23a70e172a5b2eb5897a
+size 602424

models/word_markov/atj_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "atj",
-  "unique_contexts": 20568,
-  "total_transitions": 155273
+  "unique_contexts": 19290,
+  "total_transitions": 116082
 }

models/word_markov/atj_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1bad041177a4fab91ca325954d98e00ec826c5f9a895eabdb741027fcd2741c5
-size 1343290
+oid sha256:e76203e3dfef52c4916a31791ddc3e203330c5ef4fa528a0653be84bb8bcec4e
+size 1248971

models/word_markov/atj_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "atj",
-  "unique_contexts": 73484,
-  "total_transitions": 153185
+  "unique_contexts": 67726,
+  "total_transitions": 114003
 }

models/word_markov/atj_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ebc592e1cd2fb89bf6091b62444d792f176ba63a922be974dd636b06b6f98d1c
-size 1854587
+oid sha256:8e80b9624c1459ecf42b9b53bfb0b754b70c4448cfbdda5f761c274441f7abec
+size 1620476

models/word_markov/atj_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "atj",
-  "unique_contexts": 111484,
-  "total_transitions": 151097
+  "unique_contexts": 93891,
+  "total_transitions": 111924
 }

models/word_markov/atj_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0985586ccd19bf6a0dc2637f1333d7183f5038b6938b444a5176ef7d509f514e
-size 2130768
+oid sha256:ef0092fba285ff8cbcd27b44e31bb50e02ea170185e6d35262b62e980fc6bdc2
+size 1821627

models/word_markov/atj_markov_ctx4_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "word",
   "language": "atj",
-  "unique_contexts": 124488,
-  "total_transitions": 149010
+  "unique_contexts": 100093,
+  "total_transitions": 109845
 }

models/word_ngram/atj_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:62105b25627ffc7e1d5c69b732181827784ed359ea429a44beec674d369493e8
-size 41316
+oid sha256:ea561dca24844eb9229ee5eab7bfb36cf51aa25a215c4e73586a94bb426e4237
+size 28681

models/word_ngram/atj_2gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "word",
   "language": "atj",
-  "unique_ngrams": 3083,
-  "total_ngrams": 155273
+  "unique_ngrams": 2026,
+  "total_ngrams": 116082
 }

models/word_ngram/atj_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:386ec64bf8b3c1e5e4d542440e374d9e21afc5bea7cec276940db9523d92dd10
-size 48660
+oid sha256:9d2ef5961a8f1df2b0cfe0ae1086e241f6274d85e63a7d821468ecde1848f6bc
+size 30298

models/word_ngram/atj_3gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "word",
   "language": "atj",
-  "unique_ngrams": 3170,
-  "total_ngrams": 153185
+  "unique_ngrams": 1856,
+  "total_ngrams": 114003
 }

models/word_ngram/atj_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:47ed6531e2d3bc1d45b63a5cd4c8ef732bb0478a628f2ad02029971a6484ccfb
-size 76192
+oid sha256:14c469ed9b37804dc6410be9e384c9f2fbfe4f07a4d0a79de09028386726d42e
+size 46381

models/word_ngram/atj_4gram_word_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "word",
   "language": "atj",
-  "unique_ngrams": 4529,
-  "total_ngrams": 151097
+  "unique_ngrams": 2537,
+  "total_ngrams": 111924
 }