Upload all models and assets for atj (latest)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: atj
-language_name: ATJ
+language_name: Atikamekw
 language_family: american_algonquian
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-american_algonquian
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 5.949
+    value: 5.953
   - name: best_isotropy
     type: isotropy
-    value: 0.1619
+    value: 0.1437
   - name: vocabulary_size
     type: vocab
     value: 0
 generated: 2026-01-03
 ---
 
-# ATJ - Wikilangs Models
+# Atikamekw - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **ATJ** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Atikamekw** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -60,7 +70,7 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Morphological Analysis (Experimental)](#6-morphological-analysis)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
 - [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
@@ -80,43 +90,43 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 5.115x | 5.13 | 0.1890% | 92,078 |
-| **16k** | 5.507x | 5.52 | 0.2035% | 85,522 |
-| **32k** | 5.949x 🏆 | 5.96 | 0.2198% | 79,160 |
+| **8k** | 5.122x | 5.13 | 0.1886% | 91,751 |
+| **16k** | 5.512x | 5.52 | 0.2029% | 85,261 |
+| **32k** | 5.953x 🏆 | 5.97 | 0.2191% | 78,943 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Thetford Mines oteno Kepek askik ici actew, Kanata. Irikik e tacinaniwok 25 649 ...`
+**Sample 1:** `Sainte-Anne-des-Monts oteno Kepek askik ici actew, Kanata. Irikik e tacinaniwok ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁the t ford ▁mi ne s ▁oteno ▁kepek ▁askik ▁ici ... (+15 more)` | 25 |
-| 16k | `▁thetford ▁mi nes ▁oteno ▁kepek ▁askik ▁ici ▁actew , ▁kanata ... (+12 more)` | 22 |
-| 32k | `▁thetford ▁mines ▁oteno ▁kepek ▁askik ▁ici ▁actew , ▁kanata . ... (+11 more)` | 21 |
+| 8k | `▁sainte - anne - des - mont s ▁oteno ▁kepek ... (+16 more)` | 26 |
+| 16k | `▁sainte - anne - des - monts ▁oteno ▁kepek ▁askik ... (+15 more)` | 25 |
+| 32k | `▁sainte - anne - des - monts ▁oteno ▁kepek ▁askik ... (+15 more)` | 25 |
 
-**Sample 2:** `Ka Oskiskakamaksource CNA - Atikamekw Kinokewin, sakihikan Kepek askik ici actew...`
+**Sample 2:** `Mulgrave oteno Nouvelle-Écosse aski ici actew, Kanata. Irikik e tacinaniwok 879 ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ka ▁oski skakamak source ▁cna ▁- ▁atikamekw ▁kino kewin , ... (+9 more)` | 19 |
-| 16k | `▁ka ▁oski skakamak source ▁cna ▁- ▁atikamekw ▁kinokewin , ▁sakihikan ... (+8 more)` | 18 |
-| 32k | `▁ka ▁oskiskakamak source ▁cna ▁- ▁atikamekw ▁kinokewin , ▁sakihikan ▁kepek ... (+7 more)` | 17 |
+| 8k | `▁m ul gra ve ▁oteno ▁nouvelle - écosse ▁aski ▁ici ... (+12 more)` | 22 |
+| 16k | `▁mulgrave ▁oteno ▁nouvelle - écosse ▁aski ▁ici ▁actew , ▁kanata ... (+9 more)` | 19 |
+| 32k | `▁mulgrave ▁oteno ▁nouvelle - écosse ▁aski ▁ici ▁actew , ▁kanata ... (+9 more)` | 19 |
 
-**Sample 3:** `Stellarton oteno Nouvelle-Écosse aski ici actew, Kanata. Irikik e tacinaniwok 4 ...`
+**Sample 3:** `Gracefield oteno Kepek askik ici actew, Kanata. Irikik e tacinaniwok 2 462 matce...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ste lla r ton ▁oteno ▁nouvelle - écosse ▁aski ▁ici ... (+14 more)` | 24 |
-| 16k | `▁ste lla r ton ▁oteno ▁nouvelle - écosse ▁aski ▁ici ... (+14 more)` | 24 |
-| 32k | `▁stellarton ▁oteno ▁nouvelle - écosse ▁aski ▁ici ▁actew , ▁kanata ... (+11 more)` | 21 |
+| 8k | `▁gra ce field ▁oteno ▁kepek ▁askik ▁ici ▁actew , ▁kanata ... (+11 more)` | 21 |
+| 16k | `▁gra ce field ▁oteno ▁kepek ▁askik ��ici ▁actew , ▁kanata ... (+11 more)` | 21 |
+| 32k | `▁gracefield ▁oteno ▁kepek ▁askik ▁ici ▁actew , ▁kanata . ▁irikik ... (+9 more)` | 19 |
 
 
 ### Key Findings
 
-- **Best Compression:** 32k achieves 5.949x compression
-- **Lowest UNK Rate:** 8k with 0.1890% unknown tokens
+- **Best Compression:** 32k achieves 5.953x compression
+- **Lowest UNK Rate:** 8k with 0.1886% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -133,12 +143,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
 |--------|---------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | Word | 756 | 9.56 | 2,026 | 44.6% | 84.1% |
-| **2-gram** | Subword | 129 🏆 | 7.01 | 992 | 88.9% | 100.0% |
-| **3-gram** | Word | 540 | 9.08 | 1,856 | 50.0% | 84.6% |
-| **3-gram** | Subword | 761 | 9.57 | 5,493 | 41.8% | 92.5% |
-| **4-gram** | Word | 578 | 9.18 | 2,537 | 50.5% | 75.6% |
-| **4-gram** | Subword | 3,042 | 11.57 | 19,249 | 21.7% | 65.9% |
+| **2-gram** | Word | 755 | 9.56 | 2,021 | 44.7% | 84.2% |
+| **2-gram** | Subword | 129 🏆 | 7.01 | 987 | 89.0% | 100.0% |
+| **3-gram** | Word | 540 | 9.08 | 1,854 | 50.0% | 84.6% |
+| **3-gram** | Subword | 759 | 9.57 | 5,467 | 41.9% | 92.6% |
+| **4-gram** | Word | 584 | 9.19 | 2,555 | 50.3% | 75.4% |
+| **4-gram** | Subword | 3,031 | 11.57 | 19,166 | 21.7% | 66.0% |
+| **5-gram** | Word | 345 | 8.43 | 1,658 | 58.1% | 85.5% |
+| **5-gram** | Subword | 7,892 | 12.95 | 37,893 | 14.8% | 46.5% |
 
 ### Top 5 N-grams by Size
 
@@ -146,10 +158,10 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `ici actew` | 889 |
+| 1 | `ici actew` | 888 |
 | 2 | `actew kanata` | 771 |
-| 3 | `manawan wemotaci` | 722 |
-| 4 | `e ici` | 686 |
+| 3 | `manawan wemotaci` | 721 |
+| 4 | `e ici` | 685 |
 | 5 | `irikik e` | 672 |
 
 **3-grams (Word):**
@@ -172,42 +184,62 @@ Below are sample sentences tokenized with each vocabulary size:
 | 4 | `askik ici actew kanata` | 490 |
 | 5 | `kepek askik ici actew` | 457 |
 
+**5-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `ici actew kanata irikik e` | 620 |
+| 2 | `actew kanata irikik e tacinaniwok` | 620 |
+| 3 | `kepek askik ici actew kanata` | 455 |
+| 4 | `askik ici actew kanata irikik` | 358 |
+| 5 | `oteno kepek askik ici actew` | 326 |
+
 **2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `c i` | 23,693 |
-| 2 | `k a` | 23,558 |
-| 3 | `_ k` | 23,282 |
-| 4 | `t c` | 23,205 |
-| 5 | `i k` | 21,042 |
+| 1 | `c i` | 23,681 |
+| 2 | `k a` | 23,540 |
+| 3 | `_ k` | 23,289 |
+| 4 | `t c` | 23,201 |
+| 5 | `i k` | 21,032 |
 
 **3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
 | 1 | `t c i` | 11,312 |
-| 2 | `_ k i` | 10,112 |
-| 3 | `i t c` | 10,006 |
-| 4 | `_ k a` | 9,178 |
-| 5 | `c i _` | 8,654 |
+| 2 | `_ k i` | 10,113 |
+| 3 | `i t c` | 10,005 |
+| 4 | `_ k a` | 9,180 |
+| 5 | `c i _` | 8,655 |
 
 **4-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `i t c i` | 5,889 |
+| 1 | `i t c i` | 5,891 |
 | 2 | `a n i w` | 5,154 |
 | 3 | `_ k a _` | 4,777 |
-| 4 | `n i w o` | 4,370 |
-| 5 | `k a n i` | 4,232 |
+| 4 | `n i w o` | 4,372 |
+| 5 | `k a n i` | 4,233 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `a n i w o` | 3,980 |
+| 2 | `n i w o k` | 3,620 |
+| 3 | `k a n i w` | 3,557 |
+| 4 | `a k a n i` | 3,262 |
+| 5 | `_ m a t c` | 2,919 |
 
 
 ### Key Findings
 
 - **Best Perplexity:** 2-gram (subword) with 129
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~66% of corpus
+- **Coverage:** Top-1000 patterns cover ~47% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -223,14 +255,14 @@ Below are sample sentences tokenized with each vocabulary size:
 
 | Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
 |---------|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | Word | 0.5831 | 1.498 | 3.56 | 19,290 | 41.7% |
-| **1** | Subword | 1.5451 | 2.918 | 13.90 | 118 | 0.0% |
-| **2** | Word | 0.1879 | 1.139 | 1.41 | 67,726 | 81.2% |
-| **2** | Subword | 1.2667 | 2.406 | 6.32 | 1,639 | 0.0% |
-| **3** | Word | 0.0530 | 1.037 | 1.09 | 93,891 | 94.7% |
-| **3** | Subword | 0.7981 | 1.739 | 3.30 | 10,345 | 20.2% |
-| **4** | Word | 0.0145 🏆 | 1.010 | 1.02 | 100,093 | 98.6% |
-| **4** | Subword | 0.5495 | 1.464 | 2.26 | 34,073 | 45.1% |
+| **1** | Word | 0.5828 | 1.498 | 3.55 | 19,248 | 41.7% |
+| **1** | Subword | 1.5433 | 2.915 | 13.86 | 118 | 0.0% |
+| **2** | Word | 0.1881 | 1.139 | 1.41 | 67,567 | 81.2% |
+| **2** | Subword | 1.2598 | 2.395 | 6.30 | 1,635 | 0.0% |
+| **3** | Word | 0.0530 | 1.037 | 1.09 | 93,703 | 94.7% |
+| **3** | Subword | 0.7971 | 1.738 | 3.30 | 10,279 | 20.3% |
+| **4** | Word | 0.0146 🏆 | 1.010 | 1.02 | 99,898 | 98.5% |
+| **4** | Subword | 0.5503 | 1.464 | 2.26 | 33,860 | 45.0% |
 
 ### Generated Text Samples (Word-based)
 
@@ -238,27 +270,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `e takonikatek anihe wikasi aka nipoane aka ewi tipapasotc tawatcikaniw apitcitaw iskwew kata nespito...`
-2. `ka icinikasotc ki nti matce kisinarik micta kackitatc e ici sikinikatek rasop e wamowsotc kaskina wi...`
-3. `ki micta kackitatc nictam ka ickwa aiketcik mitcetowaw nikamohinik tekera weckatc nehirowisikw ni ap...`
+1. `e totcikatek arimatc aric kirowe warowik e iti matce tipaskonikik ka tato piponikarik awik e kitotc`
+2. `ka takocinokopanen 22 otatakon pisimw nac mocak ki tesinikew kaie e tacinaniwok 352 395 matcectakani...`
+3. `ki pe ocitakaniwoki mikiwama ki ponimatisirikopon marianne ki kicikateriw kitci matcihitisotc nehiro...`
 
 **Context Size 2:**
 
-1. `ici actew kanata irikik e tacinaniwok matcectakaniwok`
-2. `actew kanata matcectakaniwok opitciwan matcectakaniwok matcectakaniwok`
-3. `manawan wemotaci nabesipi sipi ekote e otci katcitcipitak niheriw kitci aititosketc mitowi ka takoki...`
+1. `ici actew kanata irikik e tacinaniwok 53 939 matcectakaniwok`
+2. `actew kanata irikik e tacinaniwok 10 051 matcectakaniwok`
+3. `manawan wemotaci patak apitisiw anihe kirowe ka atiparik kecpin e orowinaniwok pitakamik e tacikaniw...`
 
 **Context Size 3:**
 
-1. `ici actew kanata irikik e tacinaniwok 5 037 matcectakaniwok`
-2. `kanata irikik e tacinaniwok 7 200 oteno ote itekera ka icitiperitakok comté portneuf rareak micta si...`
-3. `actew kanata irikik e tacinaniwok 71 419 matcectakaniwok`
+1. `ici actew kanata irikik e tacinaniwok 20 161 e ici tipatcimomakak nicw takon anohwe nehiro oteno ket...`
+2. `kanata irikik e tacinaniwok 10 051 matcectakaniwok`
+3. `actew kanata irikik e tacinaniwok 2 216 matcectakaniwok`
 
 **Context Size 4:**
 
-1. `ici actew kanata irikik e tacinaniwok 403 390 matcectakaniwok`
-2. `actew kanata irikik e tacinaniwok 3 930 matcectakaniwok`
-3. `kanata irikik e tacinaniwok 552`
+1. `actew kanata irikik e tacinaniwok 7 347 matcectakaniwok`
+2. `ici actew kanata irikik e tacinaniwok 7 282 matcectakaniwok`
+3. `kanata irikik e tacinaniwok 973 matcectakaniwok`
 
 
 ### Generated Text Samples (Subword-based)
@@ -267,34 +299,34 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `i_w_acikaraska_p`
-2. `_thitcicisan_ta_`
-3. `ak_kitciwik_ours`
+1. `iwoka_di_naw_k_m`
+2. `_m._ki_nanew._ka`
+3. `atcotakie_ak,_ac`
 
 **Context Size 2:**
 
-1. `cira._ok_takanetc`
-2. `katcik._ka_ictert`
-3. `_koki_sinikaniwan`
+1. `cina._tacimoodre_`
+2. `kaniniwee_icitci_`
+3. `_ki_ek_itcik._mot`
 
 **Context Size 3:**
 
-1. `tcikamooseph_du_qu`
-2. `_kirikanikaniwok._`
-3. `itc_iskakwasotc._e`
+1. `tcik._matcectapwat`
+2. `_ki_icitc_kitc_aga`
+3. `itciwok._kaie_nta_`
 
 **Context Size 4:**
 
-1. `itciwan_nehiriwa_on`
-2. `aniwiw_ka_taci_matc`
-3. `_ka_iti_ici_nictahi`
+1. `itcisowapinaniwiw_k`
+2. `aniwonik_meka_ki_oc`
+3. `_ka_tatopiponen_nip`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 (word) with 98.6% predictability
+- **Best Predictability:** Context-4 (word) with 98.5% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (34,073 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (33,860 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -310,35 +342,35 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 6,479 |
-| Total Tokens | 105,209 |
-| Mean Frequency | 16.24 |
+| Vocabulary Size | 6,458 |
+| Total Tokens | 105,050 |
+| Mean Frequency | 16.27 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 131.12 |
+| Frequency Std Dev | 131.25 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | e | 6,370 |
+| 1 | e | 6,358 |
 | 2 | ka | 4,817 |
-| 3 | ki | 3,656 |
-| 4 | ici | 2,654 |
+| 3 | ki | 3,659 |
+| 4 | ici | 2,655 |
 | 5 | kitci | 1,874 |
 | 6 | kaie | 1,655 |
 | 7 | matcectakaniwok | 1,604 |
 | 8 | micta | 1,222 |
 | 9 | kirika | 1,111 |
-| 10 | manawan | 973 |
+| 10 | manawan | 972 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | cikomewokw | 2 |
-| 2 | miitaw | 2 |
-| 3 | droits | 2 |
-| 4 | ntokiw | 2 |
+| 1 | nehirosi | 2 |
+| 2 | cikomewokw | 2 |
+| 3 | miitaw | 2 |
+| 4 | droits | 2 |
 | 5 | kiskinohamato | 2 |
 | 6 | banque | 2 |
 | 7 | mawotcicorianionik | 2 |
@@ -350,24 +382,24 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.0501 |
-| R² (Goodness of Fit) | 0.987715 |
+| Zipf Coefficient | 1.0505 |
+| R² (Goodness of Fit) | 0.987789 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 54.5% |
+| Top 100 | 54.6% |
 | Top 1,000 | 81.8% |
 | Top 5,000 | 97.2% |
 | Top 10,000 | 0.0% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9877 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 54.5% of corpus
-- **Long Tail:** -3,521 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9878 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 54.6% of corpus
+- **Long Tail:** -3,542 words needed for remaining 100.0% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -383,37 +415,40 @@ Below are text samples generated from each subword-based Markov chain model:
 
 ### 5.1 Cross-Lingual Alignment
 
-> *Note: Multilingual alignment visualization not available for this language.*
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
 
 
 ### 5.2 Model Comparison
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.1619 🏆 | 0.4893 | N/A | N/A |
-| **mono_64d** | 64 | 0.0330 | 0.5001 | N/A | N/A |
-| **mono_128d** | 128 | 0.0058 | 0.5012 | N/A | N/A |
+| **mono_32d** | 32 | 0.1437 🏆 | 0.4915 | N/A | N/A |
+| **mono_64d** | 64 | 0.0311 | 0.5012 | N/A | N/A |
+| **mono_128d** | 128 | 0.0055 | 0.4973 | N/A | N/A |
+| **aligned_32d** | 32 | 0.1437 | 0.4825 | 0.0091 | 0.1088 |
+| **aligned_64d** | 64 | 0.0311 | 0.5079 | 0.0136 | 0.1066 |
+| **aligned_128d** | 128 | 0.0055 | 0.4960 | 0.0317 | 0.1565 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.1619 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.4969. Lower values indicate better semantic separation.
-- **Alignment Quality:** No aligned models evaluated in this run.
+- **Best Isotropy:** mono_32d with 0.1437 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.4961. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 3.2% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
 ## 6.  Morphological Analysis (Experimental)
 
-> ⚠️ **Warning:** This language shows low morphological productivity. The statistical signals used for this analysis may be noisy or less reliable than for morphologically rich languages.
-
 This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
 
 ### 6.1 Productivity & Complexity
 
 | Metric | Value | Interpretation | Recommendation |
 |--------|-------|----------------|----------------|
-| Productivity Index | **0.000** | Low morphological productivity | ⚠️ Likely unreliable |
-| Idiomaticity Gap | **-1.000** | Low formulaic content | - |
+| Productivity Index | **4.183** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **0.838** | High formulaic/idiomatic content | - |
 
 ### 6.2 Affix Inventory (Productive Units)
 
@@ -422,25 +457,26 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Prefixes
 | Prefix | Examples |
 |--------|----------|
-| `-ki` | kictapatisiw, kiwew, kiskinohomakewiniw |
-| `-mi` | mikomin, mitataw, mirokotek |
-| `-ma` | mamowinikatek, masinatew, manikaniwon |
-| `-ni` | ninan, nikikw, niheritamokw |
-| `-ot` | otcepikiwitc, otactimikok, otcotcoma |
-| `-ta` | tanetci, tatiniw, tatow |
-| `-ic` | icakopan, icinikatakiniw, icinikatakaniwitcik |
+| `-ki` | kitciki, kimosapitc, kinowapitamokw |
+| `-mi` | mireritamiriwa, mitciso, mirokiw |
+| `-ma` | maninikatew, matcectakaniwok, mars |
+| `-ot` | ototokon, otenocic, otenawa |
+| `-ni` | nitowakik, nikomesak, nitawikiritci |
+| `-ic` | icikapowiw, icinikatikik, icinkatew |
+| `-wi` | wirino, witamotcik, wirtip |
+| `-ta` | takociretc, tacikeriwa, taritci |
 
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-k` | nosinetakaniwonik, rarewak, mamowinikatek |
-| `-w` | kictapatisiw, masinatew, kiwew |
-| `-c` | awotatokwetc, otcepikiwitc, taciketc |
-| `-n` | potatcikan, mikomin, icakopan |
-| `-ik` | nosinetakaniwonik, pakacik, icinikatakaniwitcik |
-| `-tc` | awotatokwetc, otcepikiwitc, taciketc |
-| `-iw` | kictapatisiw, kiskinohomakewiniw, icinikatakiniw |
-| `-ok` | petakok, otactimikok, wapitamok |
+| `-k` | titopiponikak, kanawapitcikatek, nitowakik |
+| `-w` | pakonehohakiniwiw, kinowapitamokw, nipiriw |
+| `-c` | kimosapitc, ponihatc, pamatisitc |
+| `-n` | ototokon, owen, foundation |
+| `-ik` | nitowakik, witamotcik, totowakaniwitcik |
+| `-tc` | kimosapitc, ponihatc, pamatisitc |
+| `-ok` | itakiniwok, ntokihitisohok, nakapewonok |
+| `-iw` | pakonehohakiniwiw, nipiriw, mowakiniwiw |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -448,18 +484,18 @@ Bound stems are high-frequency subword units that are semantically cohesive but
 
 | Stem | Cohesion | Substitutability | Examples |
 |------|----------|------------------|----------|
-| `taka` | 1.44x | 22 contexts | otakai, pataka, matakaw |
-| `tako` | 1.32x | 29 contexts | takon, takok, takoki |
-| `apit` | 1.50x | 17 contexts | apitc, apita, tapit |
-| `atis` | 1.49x | 17 contexts | matisiw, matisin, batiste |
-| `mitc` | 1.33x | 22 contexts | mitca, mitci, mitcin |
-| `aniw` | 1.38x | 19 contexts | aniwe, nikaniw, oskaniw |
-| `iwok` | 1.43x | 16 contexts | apiwok, aipiwok, irniwok |
-| `erit` | 1.48x | 14 contexts | iteritam, iteritak, oreritam |
-| `niwo` | 1.51x | 13 contexts | irniwok, koniwok, kaniwok |
-| `tcik` | 1.30x | 19 contexts | tatcik, mitcik, motcik |
-| `irow` | 1.56x | 11 contexts | kirowe, wirowow, kewirow |
-| `kate` | 1.33x | 16 contexts | katek, makate, kateri |
+| `tako` | 1.33x | 29 contexts | takok, takon, takoke |
+| `taka` | 1.42x | 22 contexts | pataka, otakai, otakaci |
+| `mitc` | 1.35x | 22 contexts | mitci, mitca, mitcim |
+| `erit` | 1.54x | 14 contexts | wewerita, oreritam, iteritci |
+| `apit` | 1.44x | 17 contexts | apita, tapit, apitc |
+| `aniw` | 1.36x | 19 contexts | aniwe, kaniwok, nikaniw |
+| `iwok` | 1.42x | 16 contexts | apiwok, irniwok, askiwok |
+| `niwo` | 1.50x | 13 contexts | irniwok, koniwok, kaniwok |
+| `kana` | 1.36x | 15 contexts | kanapé, kanada, oskana |
+| `irow` | 1.51x | 11 contexts | kirowe, kewirow, wirowaw |
+| `itak` | 1.35x | 15 contexts | witak, titak, kitaki |
+| `kate` | 1.32x | 16 contexts | katek, makate, kateri |
 
 ### 6.4 Affix Compatibility (Co-occurrence)
 
@@ -467,16 +503,16 @@ This table shows which prefixes and suffixes most frequently co-occur on the sam
 
 | Prefix | Suffix | Frequency | Examples |
 |--------|--------|-----------|----------|
-| `-ki` | `-k` | 127 words | kiskeritakik, kiskeritakositcik |
-| `-ma` | `-k` | 89 words | masinapiskikatek, mackikirinikwecik |
-| `-mi` | `-k` | 88 words | mitcenik, mickaniwok |
-| `-ki` | `-w` | 68 words | kiskinohamakewiniw, kictaw |
-| `-mi` | `-w` | 65 words | miromakosiw, mitcetwaw |
-| `-ni` | `-k` | 61 words | nitwakik, nisitowinikatek |
-| `-ot` | `-k` | 57 words | otek, otcirowek |
-| `-ki` | `-ik` | 56 words | kiskeritakik, kiskeritakositcik |
-| `-ki` | `-c` | 51 words | kicterimitisotc, kiciwahikotc |
-| `-ta` | `-k` | 49 words | tacikewok, takociparitcik |
+| `-ki` | `-k` | 127 words | kiceriniwok, kinokepitcikanik |
+| `-mi` | `-k` | 89 words | mirwacinik, mictikok |
+| `-ma` | `-k` | 89 words | matakanik, matcikonak |
+| `-ki` | `-w` | 68 words | kicteritakoniw, kiskinohamakew |
+| `-mi` | `-w` | 65 words | mitcetaw, micaw |
+| `-ni` | `-k` | 60 words | nikickowatcik, nikapewnok |
+| `-ot` | `-k` | 57 words | ototewok, otcikowik |
+| `-ki` | `-ik` | 56 words | kinokepitcikanik, kickapiskarik |
+| `-ki` | `-c` | 51 words | kinikositc, kictapeitc |
+| `-ta` | `-k` | 49 words | tarasak, tacikaniwonik |
 
 ### 6.5 Recursive Morpheme Segmentation
 
@@ -484,26 +520,28 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 
 | Word | Suggested Split | Confidence | Stem |
 |------|-----------------|------------|------|
-| kitotakaniw | **`ki-totak-an-iw`** | 7.5 | `totak` |
-| pimatisinaniwok | **`pimatisin-an-iw-ok`** | 7.5 | `pimatisin` |
-| icipekahikaniwok | **`ic-ipekah-ik-an-iw-ok`** | 7.5 | `ipekah` |
-| masinahikaniwok | **`ma-sinah-ik-an-iw-ok`** | 7.5 | `sinah` |
-| icitatcik | **`ic-itat-cik`** | 6.0 | `itat` |
-| masinahikanik | **`ma-sinah-ik-an-ik`** | 6.0 | `sinah` |
-| mipariwakaniwok | **`mi-pariwak-an-iw-ok`** | 6.0 | `pariwak` |
-| osawapisikaniwok | **`osawapis-ik-an-iw-ok`** | 6.0 | `osawapis` |
-| matcehonaniwok | **`ma-tcehon-an-iw-ok`** | 6.0 | `tcehon` |
-| nipimatisiwinik | **`ni-pimatisiwin-ik`** | 6.0 | `pimatisiwin` |
-| misinhikaniw | **`mi-sinh-ik-an-iw`** | 6.0 | `sinh` |
-| nikamonaniwok | **`ni-kamon-an-iw-ok`** | 6.0 | `kamon` |
-| metowaniwok | **`metow-an-iw-ok`** | 4.5 | `metow` |
-| miremakanik | **`mi-remak-an-ik`** | 4.5 | `remak` |
-| acamakaniwok | **`acamak-an-iw-ok`** | 4.5 | `acamak` |
+| otaskitcik | **`ot-aski-tc-ik`** | 7.5 | `aski` |
+| wikiconvention | **`wi-ki-convention`** | 6.0 | `convention` |
+| nehirowisitcik | **`nehirowisi-tc-ik`** | 6.0 | `nehirowisi` |
+| kiskerimakaniwiw | **`ki-skerimak-an-iw-iw`** | 6.0 | `skerimak` |
+| takapikenikaniw | **`ta-kapiken-ik-an-iw`** | 6.0 | `kapiken` |
+| wicamakaniwiw | **`wi-camak-an-iw-iw`** | 6.0 | `camak` |
+| nikickotatotcik | **`ni-ki-ckotato-tc-ik`** | 6.0 | `ckotato` |
+| kackihotcik | **`kackiho-tc-ik`** | 6.0 | `kackiho` |
+| tipatcimotcik | **`tipatcimo-tc-ik`** | 6.0 | `tipatcimo` |
+| takociretcik | **`ta-kocire-tc-ik`** | 4.5 | `kocire` |
+| apatcihakaniwiw | **`apatcihak-an-iw-iw`** | 4.5 | `apatcihak` |
+| takocinitcik | **`ta-kocini-tc-ik`** | 4.5 | `kocini` |
+| kicowekaniw | **`ki-cowek-an-iw`** | 4.5 | `cowek` |
+| emitcikocimotc | **`emitcikocimo-tc`** | 4.5 | `emitcikocimo` |
+| apitcihakaniwiw | **`apitcihak-an-iw-iw`** | 4.5 | `apitcihak` |
 
 ### 6.6 Linguistic Interpretation
 
 > **Automated Insight:**
-The language ATJ appears to be more isolating or has a highly fixed vocabulary. Word-level models perform nearly as well as subword models, indicating fewer productive morphological processes.
+The language Atikamekw shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
 
 ---
 ## 7. Summary & Recommendations
@@ -516,7 +554,7 @@ The language ATJ appears to be more isolating or has a highly fixed vocabulary.
 |-----------|-------------|-----------|
 | Tokenizer | **32k BPE** | Best compression (5.95x) |
 | N-gram | **2-gram** | Lowest perplexity (129) |
-| Markov | **Context-4** | Highest predictability (98.6%) |
+| Markov | **Context-4** | Highest predictability (98.5%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
 
@@ -730,4 +768,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 05:18:59*
+*Report Date: 2026-01-03 17:35:34*

models/embeddings/aligned/atj_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6130a611520c215b9ac6ebea267d66817769f589f58ebe5c05e99f7420b064e
+size 1026353893

models/embeddings/aligned/atj_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "atj", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/atj_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33150d433b40d1b10ce7bdc8c2e8697e00c05631b53be4898bc117fbd0489dd8
+size 65664

models/embeddings/aligned/atj_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "atj",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 441,
+  "vocab_size": 2260
+}

models/embeddings/aligned/atj_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8529399a9f32d7bd0b38d045844e20575dee1c6493e28cbc23a9d582416ff9c5
+size 256618213

models/embeddings/aligned/atj_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "atj", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/atj_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28d73a4e9c3fd5f6ed08fffa3e6147ec085965b077279a46e22510d6d221f699
+size 4224

models/embeddings/aligned/atj_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "atj",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 441,
+  "vocab_size": 2260
+}

models/embeddings/aligned/atj_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ca57067a2f64cc6ad0147cef85910d8f9df025d6fd14d22c782a9f234e6937d
+size 513196773

models/embeddings/aligned/atj_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "atj", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/atj_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c923d5bd3352a3faa7a5182ef39511cec09a5087ed55b61bed41274332033b63
+size 16512

models/embeddings/aligned/atj_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "atj",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 441,
+  "vocab_size": 2260
+}

models/embeddings/monolingual/atj_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7f3d6139699a8a77dd41505f198f1088f6ea783407b1398a6369141ef13d2adc
-size 1026354933
+oid sha256:e6130a611520c215b9ac6ebea267d66817769f589f58ebe5c05e99f7420b064e
+size 1026353893

models/embeddings/monolingual/atj_128d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 128
   },
-  "vocab_size": 2261
+  "vocab_size": 2260
 }

models/embeddings/monolingual/atj_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:733947e40bfc2a29297e78ab7590c70db193bca524959bb47af3e2ea45087684
-size 256618485
+oid sha256:8529399a9f32d7bd0b38d045844e20575dee1c6493e28cbc23a9d582416ff9c5
+size 256618213

models/embeddings/monolingual/atj_32d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 32
   },
-  "vocab_size": 2261
+  "vocab_size": 2260
 }

models/embeddings/monolingual/atj_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cf1bb93977201f5841f00da8ac2e6f2915a651bf8892a51f91d3c84233f3fae3
-size 513197301
+oid sha256:5ca57067a2f64cc6ad0147cef85910d8f9df025d6fd14d22c782a9f234e6937d
+size 513196773

models/embeddings/monolingual/atj_64d_metadata.json

@@ -11,5 +11,5 @@
     "encoding_method": "rope",
     "dim": 64
   },
-  "vocab_size": 2261
+  "vocab_size": 2260
 }

models/subword_markov/atj_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:01ab1d155ad7b2173943a42ef233b27a6456b68458231c9472df823263be9ec6
-size 16979
+oid sha256:dd708dfe5d10ec2476635da14a3b43caba934d16773347ae9cf596429c6e3373
+size 16288

models/subword_markov/atj_markov_ctx1_subword_metadata.json

@@ -3,5 +3,5 @@
   "variant": "subword",
   "language": "atj",
   "unique_contexts": 118,
-  "total_transitions": 875958
+  "total_transitions": 874657
 }

models/subword_markov/atj_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:01c646d4de1b50ab2af2e42b22f63c9283f46a99722629ba2187c5b17039d034
-size 80338
+oid sha256:996774f3d175aa2890351830b406e5a9b2a393be5fb1eecbcc8d1f589a1f77b5
+size 80406

models/subword_markov/atj_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "atj",
-  "unique_contexts": 1639,
-  "total_transitions": 873879
+  "unique_contexts": 1635,
+  "total_transitions": 872580
 }

models/subword_markov/atj_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d29a79e07eb2572a881d989c62b2b464a2e86ce95f9e1ca1cfc17ebb6f7fdda4
-size 260964
+oid sha256:e24405c9b09e7d8ff8662b4985a40e956d74330435ebf8106460b67335d7c44e
+size 263602

models/subword_markov/atj_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "atj",
-  "unique_contexts": 10345,
-  "total_transitions": 871800
+  "unique_contexts": 10279,
+  "total_transitions": 870503
 }

models/subword_markov/atj_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c92961d642ccb93882ab7a777fc55a418e457e8f9843b90ea4d3869eb4115b68
-size 635430
+oid sha256:f48aeb954a19ee1301b10e0fef337eae02e6f994caaede81c71444e2f1af68aa
+size 641517

models/subword_markov/atj_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "atj",
-  "unique_contexts": 34073,
-  "total_transitions": 869721
+  "unique_contexts": 33860,
+  "total_transitions": 868426
 }

models/subword_ngram/atj_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:83d8aa6cff41c2e4e7567422c922f50fad121f6d85606b56af394b2714de92de
-size 14344
+oid sha256:d3799428a10cba603c7b2f4814c59e9cd5872370f5467695c9426a648c99c3cb
+size 14196

models/subword_ngram/atj_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "atj",
-  "unique_ngrams": 992,
-  "total_ngrams": 875958
+  "unique_ngrams": 987,
+  "total_ngrams": 874657
 }

models/subword_ngram/atj_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8cde20675b31ba779a55241e29b9edb6286c6f51bb3fa511c63b49f3d14f3e10
-size 65963
+oid sha256:966bf4e8908c9d17020611e82eb1be771c0c5df3735ddbeba78e86de2471958b
+size 65636

models/subword_ngram/atj_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "atj",
-  "unique_ngrams": 5493,
-  "total_ngrams": 873879
+  "unique_ngrams": 5467,
+  "total_ngrams": 872580
 }

models/subword_ngram/atj_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c7765e9eede8cde44f7885c15dc01158f7b11d31566539ee029e4555794cf9cf
-size 231317
+oid sha256:edbe2a0815dec3054db2fb4e69fbe3ed1532712e8d4c0a64278e3aee39e10e11
+size 237255

models/subword_ngram/atj_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "atj",
-  "unique_ngrams": 19249,
-  "total_ngrams": 871800
+  "unique_ngrams": 19166,
+  "total_ngrams": 870503
 }

models/subword_ngram/atj_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8e1f9377f75fa4bb01638ff895452454f2676e14228adedd2210fd36c351005
+size 485082

models/subword_ngram/atj_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "atj",
+  "unique_ngrams": 37893,
+  "total_ngrams": 868426
+}

models/tokenizer/atj_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:087da7688c1626c425cee991b2afa0071074d6ba52750f654d69b2ab05b54160
-size 536207
+oid sha256:4ee5f96570e4aacf7848e744c8bb433d80038e85b5c7ac50737866784abc7f9b
+size 536410

models/tokenizer/atj_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e8fa5e1ab7cf748ada69603f315ae6c44dbb95d4e07cabda9709bfbe2f875a70
-size 846451
+oid sha256:6e78beef37ef47c219fa4eb9a5a1a4190325e01537fa6a5973e51f50cdf02ed8
+size 846260

models/tokenizer/atj_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:678b99dd207da5596a12f7a9f3f36ad762fad42d71893cebac1f4c67048caa88
-size 384081
+oid sha256:6d2bb59092f250508dd8e288f06ba2f245aff67ed2a53185891a086c6a4c306f
+size 384085

models/vocabulary/atj_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:00f58d6e6175ddabe7c28e84d3c39c5e5ba1832a75f401ec087f627ab36c7ca5
-size 106314
+oid sha256:4e18b6af6dfe426e9680782b6878f7d0ec03ea6f8f2275dadbb7a2196f27448c
+size 106614

models/vocabulary/atj_vocabulary_metadata.json

@@ -1,17 +1,17 @@
 {
   "language": "atj",
-  "vocabulary_size": 6479,
+  "vocabulary_size": 6458,
   "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.16444512148678497,
+    "type_token_ratio": 0.16433293778606545,
     "coverage": {
-      "top_100": 0.485278560607984,
-      "top_1000": 0.7279643875728878,
-      "top_5000": 0.865353204526028,
-      "top_10000": 0.9201851710801364
+      "top_100": 0.48592134260044073,
+      "top_1000": 0.7283183590439057,
+      "top_5000": 0.8656890998474318,
+      "top_10000": 0.9204271910493304
     },
-    "hapax_count": 12952,
-    "hapax_ratio": 0.6665637383562348,
-    "total_documents": 2079
+    "hapax_count": 12930,
+    "hapax_ratio": 0.6669073653806479,
+    "total_documents": 2077
   }
 }

models/word_markov/atj_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bdf082d2a0a4824649c5b861ab9afabf286d0dbb86ad23a70e172a5b2eb5897a
-size 602424
+oid sha256:90fb65d31c01aca6262a2e2ae9422225a3a0388f1facf05e1c137bf7f3cfc717
+size 601383

models/word_markov/atj_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "atj",
-  "unique_contexts": 19290,
-  "total_transitions": 116082
+  "unique_contexts": 19248,
+  "total_transitions": 115903
 }

models/word_markov/atj_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e76203e3dfef52c4916a31791ddc3e203330c5ef4fa528a0653be84bb8bcec4e
-size 1248971
+oid sha256:f7a60672dda7073fbd5ca7cc7da1407e9c2152c056a974cbb72967caf48c4574
+size 1246010

models/word_markov/atj_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "atj",
-  "unique_contexts": 67726,
-  "total_transitions": 114003
+  "unique_contexts": 67567,
+  "total_transitions": 113826
 }

models/word_markov/atj_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8e80b9624c1459ecf42b9b53bfb0b754b70c4448cfbdda5f761c274441f7abec
-size 1620476
+oid sha256:eab51866fc2f9aa26d9a443eca0453d51dc36083a80b4f481d7a231a6365a955
+size 1617567

models/word_markov/atj_markov_ctx3_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "word",
   "language": "atj",
-  "unique_contexts": 93891,
-  "total_transitions": 111924
+  "unique_contexts": 93703,
+  "total_transitions": 111749
 }