Upload all models and assets for dty (latest)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: dty
-language_name: DTY
+language_name: Dotyali
 language_family: indoaryan_central
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-indoaryan_central
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.898
+    value: 4.539
   - name: best_isotropy
     type: isotropy
-    value: 0.8984
+    value: 0.9032
   - name: vocabulary_size
     type: vocab
-    value: 9201
-generated: 2025-12-30
+    value: 0
+generated: 2026-01-04
 ---
 
-# DTY - Wikilangs Models
+# Dotyali - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **DTY** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Dotyali** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -44,12 +54,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +70,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,55 +80,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.624x | 3.58 | 0.0317% | 170,528 |
-| **16k** | 4.090x | 4.04 | 0.0357% | 151,110 |
-| **32k** | 4.513x | 4.46 | 0.0394% | 136,934 |
-| **64k** | 4.898x 🏆 | 4.84 | 0.0428% | 126,165 |
+| **8k** | 3.506x | 3.51 | 0.1249% | 181,747 |
+| **16k** | 3.906x | 3.91 | 0.1391% | 163,156 |
+| **32k** | 4.207x | 4.21 | 0.1499% | 151,469 |
+| **64k** | 4.539x 🏆 | 4.55 | 0.1617% | 140,390 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `पूर्णकला बिसी   नेपाली चर्चित लोक गायिका हुन।उनको जन्म बर्दिया जिल्ला  मी भया हो...`
+**Sample 1:** `सुखविंदर सिंह भारतीय सांगीतिक क्षेत्रका पाश्व गायक हुन। सन्दर्भ गिदाराअन`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁पूर्ण कला ▁बि सी ▁नेपाली ▁चर्चित ▁लोक ▁गायिका ▁हुन । ... (+13 more)` | 23 |
-| 16k | `▁पूर्ण कला ▁बि सी ▁नेपाली ▁चर्चित ▁लोक ▁गायिका ▁हुन । ... (+13 more)` | 23 |
-| 32k | `▁पूर्ण कला ▁बिसी ▁नेपाली ▁चर्चित ▁लोक ▁गायिका ▁हुन । उनको ... (+12 more)` | 22 |
-| 64k | `▁पूर्ण कला ▁बिसी ▁नेपाली ▁चर्चित ▁लोक ▁गायिका ▁हुन । उनको ... (+12 more)` | 22 |
-
-**Sample 2:** `प्रकाश केसी नेपाली क्रिकेट खेलाडी हुन।
+| 8k | `▁सुख वि ंदर ▁सिंह ▁भारतीय ▁सांगीतिक ▁क्षे��्रका ▁पाश्व ▁गायक ▁हुन ... (+3 more)` | 13 |
+| 16k | `▁सुख वि ंदर ▁सिंह ▁भारतीय ▁सांगीतिक ▁क्षेत्रका ▁पाश्व ▁गायक ▁हुन ... (+3 more)` | 13 |
+| 32k | `▁सुख विंदर ▁सिंह ▁भारतीय ▁सांगीतिक ▁क्षेत्रका ▁पाश्व ▁गायक ▁हुन । ... (+2 more)` | 12 |
+| 64k | `▁सुखविंदर ▁सिंह ▁भारतीय ▁सांगीतिक ▁क्षेत्रका ▁पाश्व ▁गायक ▁हुन । ▁सन्दर्भ ... (+1 more)` | 11 |
 
-सन्दर्भ`
+**Sample 2:** `सिंगौडी दैलेख जिल्लामी पडडे एक गाऊ विकास समिति हो । यी पनि हेर जिल्ला विकास समित...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁प्रकाश ▁केसी ▁नेपाली ▁क्रिकेट ▁खेलाडी ▁हुन । ▁सन्दर्भ` | 8 |
-| 16k | `▁प्रकाश ▁केसी ▁नेपाली ▁क्रिकेट ▁खेलाडी ▁हुन । ▁सन्दर्भ` | 8 |
-| 32k | `▁प्रकाश ▁केसी ▁नेपाली ▁क्रिकेट ▁खेलाडी ▁हुन । ▁सन्दर्भ` | 8 |
-| 64k | `▁प्रकाश ▁केसी ▁नेपाली ▁क्रिकेट ▁खेलाडी ▁हुन । ▁सन्दर्भ` | 8 |
+| 8k | `▁सि ंग ौ डी ▁दैलेख ▁जिल्लामी ▁पडडे ▁एक ▁गाऊ ▁विकास ... (+9 more)` | 19 |
+| 16k | `▁सिंग ौ डी ▁दैलेख ▁जिल्लामी ▁पडडे ▁एक ▁गाऊ ▁विकास ▁समिति ... (+8 more)` | 18 |
+| 32k | `▁सिंग ौडी ▁दैलेख ▁जिल्लामी ▁पडडे ▁एक ▁गाऊ ▁विकास ▁समिति ▁हो ... (+7 more)` | 17 |
+| 64k | `▁सिंगौडी ▁दैलेख ▁जिल्लामी ▁पडडे ▁एक ▁गाऊ ▁विकास ▁समिति ▁हो ▁। ... (+6 more)` | 16 |
 
-**Sample 3:** `मार्तडी बाजुरा जिल्लामि रया एक गाविस हो । 
-Category:बाजुरा जिल्ला
-श्रेणी:गाउँ वि...`
+**Sample 3:** `बेनिन अफ्रिका महाद्वीपमाई रयाको एक देश हो। सन्दर्भ देशअन`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁मा र्त डी ▁बाजुरा ▁जिल्ला मि ▁रया ▁एक ▁गाविस ▁हो ... (+11 more)` | 21 |
-| 16k | `▁मा र्त डी ▁बाजुरा ▁जिल्लामि ▁रया ▁एक ▁गाविस ▁हो ▁। ... (+10 more)` | 20 |
-| 32k | `▁मार्त डी ▁बाजुरा ▁जिल्लामि ▁रया ▁एक ▁गाविस ▁हो ▁। ▁category ... (+8 more)` | 18 |
-| 64k | `▁मार्तडी ▁बाजुरा ▁जिल्लामि ▁रया ▁एक ▁गाविस ▁हो ▁। ▁category : ... (+7 more)` | 17 |
+| 8k | `▁बेन िन ▁अफ्रिका ▁महाद्वीपमाई ▁रयाको ▁एक ▁देश ▁हो । ▁सन्दर्भ ... (+1 more)` | 11 |
+| 16k | `▁बेनिन ▁अफ्रिका ▁महाद्वीपमाई ▁रयाको ▁एक ▁देश ▁हो । ▁सन्दर्भ ▁देशअन` | 10 |
+| 32k | `▁बेनिन ▁अफ्रिका ▁महाद्वीपमाई ▁रयाको ▁एक ▁देश ▁हो । ▁सन्दर्भ ▁देशअन` | 10 |
+| 64k | `▁बेनिन ▁अफ्रिका ▁महाद्वीपमाई ▁रयाको ▁एक ▁देश ▁हो । ▁सन्दर्भ ▁देशअन` | 10 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.898x compression
-- **Lowest UNK Rate:** 8k with 0.0317% unknown tokens
+- **Best Compression:** 64k achieves 4.539x compression
+- **Lowest UNK Rate:** 8k with 0.1249% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -125,57 +139,111 @@ Category:बाजुरा जिल्ला
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 1,565 🏆 | 10.61 | 20,335 | 39.7% | 77.8% |
-| **2-gram** | 686 🏆 | 9.42 | 5,687 | 48.0% | 91.5% |
-| **3-gram** | 11,526 | 13.49 | 73,663 | 15.5% | 43.1% |
-| **3-gram** | 5,835 | 12.51 | 42,059 | 16.8% | 52.9% |
-| **4-gram** | 44,727 | 15.45 | 198,068 | 8.7% | 25.3% |
-| **4-gram** | 27,067 | 14.72 | 159,916 | 8.3% | 29.3% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 5,114 | 12.32 | 8,849 | 15.4% | 44.5% |
+| **2-gram** | Subword | 2,395 🏆 | 11.23 | 19,229 | 33.4% | 67.5% |
+| **3-gram** | Word | 5,204 | 12.35 | 8,802 | 15.6% | 43.7% |
+| **3-gram** | Subword | 18,338 | 14.16 | 76,407 | 10.5% | 33.0% |
+| **4-gram** | Word | 9,926 | 13.28 | 16,181 | 11.8% | 33.3% |
+| **4-gram** | Subword | 63,062 | 15.94 | 207,437 | 6.1% | 20.3% |
+| **5-gram** | Word | 7,716 | 12.91 | 12,232 | 12.4% | 36.5% |
+| **5-gram** | Subword | 95,990 | 16.55 | 239,024 | 4.9% | 15.8% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `सन्दर्भ सामग्रीअन` | 752 |
+| 2 | `गाउँ विकास` | 631 |
+| 3 | `वि सं` | 572 |
+| 4 | `सन् मी` | 549 |
+| 5 | `हो यो` | 514 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `सन्दर्भ सामग्रीअन भाइरा` | 305 |
+| 2 | `सामग्रीअन भाइरा लिङ्कअन` | 282 |
+| 3 | `विकास समिति हो` | 281 |
+| 4 | `यो लै हेर` | 276 |
+| 5 | `गाउँ विकास समिति` | 253 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `सन्दर्भ सामग्रीअन भाइरा लिङ्कअन` | 282 |
+| 2 | `गाउँ विकास समिति हो` | 232 |
+| 3 | `एक गाउँ विकास समिति` | 173 |
+| 4 | `रयाको एक देश हो` | 150 |
+| 5 | `सन्दर्भअन यिन लै हेरऽ` | 130 |
+
+**5-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `् र` | 30,133 |
-| 2 | `् य` | 27,109 |
-| 3 | `य ा` | 23,794 |
-| 4 | `क ा` | 22,745 |
-| 5 | `ा र` | 22,039 |
+| 1 | `एक गाउँ विकास समिति हो` | 173 |
+| 2 | `गाउँ विकास समितीन मध्येको एक` | 123 |
+| 3 | `मध्येको एक गाउँ विकास समिति` | 123 |
+| 4 | `समितीन मध्येको एक गाउँ विकास` | 123 |
+| 5 | `विकास समितीन मध्येको एक गाउँ` | 123 |
 
-**3-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `् य ा` | 15,112 |
-| 2 | `् र े` | 7,144 |
-| 3 | `त ् र` | 7,066 |
-| 4 | `ि क ा` | 6,720 |
-| 5 | `श ् र` | 6,006 |
+| 1 | `को _` | 29,200 |
+| 2 | `। _` | 25,775 |
+| 3 | `न _` | 25,224 |
+| 4 | `र _` | 22,897 |
+| 5 | `_ स` | 20,865 |
 
-**4-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `न े प ा` | 5,394 |
-| 2 | `श ् र े` | 5,064 |
-| 3 | `् र े ण` | 4,917 |
-| 4 | `र े ण ी` | 4,900 |
-| 5 | `े ण ी :` | 4,817 |
+| 1 | `_ । _` | 7,563 |
+| 2 | `_ रे _` | 7,379 |
+| 3 | `अ न _` | 5,308 |
+| 4 | `ला ई _` | 4,856 |
+| 5 | `_ उ न` | 4,051 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ स न्द र्भ` | 2,988 |
+| 2 | `_ ए क _` | 2,776 |
+| 3 | `_ ने पा ल` | 2,487 |
+| 4 | `_ हो । _` | 2,146 |
+| 5 | `स न्द र्भ _` | 2,025 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ स न्द र्भ _` | 2,024 |
+| 2 | `। _ स न्द र्भ` | 1,726 |
+| 3 | `_ च ल चि त्र` | 1,346 |
+| 4 | `_ हो _ । _` | 1,310 |
+| 5 | `_ उ न ले _` | 1,285 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 686
+- **Best Perplexity:** 2-gram (subword) with 2,395
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~29% of corpus
+- **Coverage:** Top-1000 patterns cover ~16% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -183,55 +251,86 @@ Category:बाजुरा जिल्ला
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.4932 | 1.408 | 5.12 | 23,936 | 50.7% |
-| **1** | 0.8448 | 1.796 | 8.27 | 1,664 | 15.5% |
-| **2** | 0.3997 | 1.319 | 2.75 | 122,540 | 60.0% |
-| **2** | 1.0565 | 2.080 | 6.51 | 13,756 | 0.0% |
-| **3** | 0.3302 | 1.257 | 1.99 | 337,074 | 67.0% |
-| **3** | 0.7861 | 1.724 | 3.55 | 89,517 | 21.4% |
-| **4** | 0.2386 🏆 | 1.180 | 1.53 | 672,221 | 76.1% |
-| **4** | 0.5021 🏆 | 1.416 | 2.20 | 317,836 | 49.8% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.6976 | 1.622 | 4.02 | 85,572 | 30.2% |
+| **1** | Subword | 0.8621 | 1.818 | 10.06 | 6,314 | 13.8% |
+| **2** | Word | 0.1550 | 1.113 | 1.27 | 343,062 | 84.5% |
+| **2** | Subword | 0.5671 | 1.482 | 3.71 | 63,513 | 43.3% |
+| **3** | Word | 0.0392 | 1.028 | 1.05 | 434,501 | 96.1% |
+| **3** | Subword | 0.4781 | 1.393 | 2.53 | 235,438 | 52.2% |
+| **4** | Word | 0.0141 🏆 | 1.010 | 1.02 | 456,418 | 98.6% |
+| **4** | Subword | 0.2801 | 1.214 | 1.62 | 594,541 | 72.0% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `रे सचिवमी नियुक्त गरेको थियो वि स न पा हरूसाविक वडा छन् यै जिल्लामा बसोबास गर्दा`
+2. `हो यी लै प्रजनन भारत सरकारले विवादित मौजाहरूको फिराद थियो हार्ट भल्भ हरु को केन्द्र बठे`
+3. `छ यद्यपि यै जिल्लामी धान नाच तमरा रूचिका विषयन्मी लेख नयाँ दिल्लीमी नानाजी देशमुखलाई गोलवलकरले उत्तर`
+
+**Context Size 2:**
+
+1. `सन्दर्भ सामग्रीअन बाह्य कडीअन माइस्पेस आधिकारिक पृष्ठ रङ्गशालाको वातावरण फिफा विश्वकपका रङ्गशालाअन य...`
+2. `गाउँ विकास समिति हो जनगणना अन्सारअ येइ ठउर को जनसङ्ख्या १६ ५८९ रह्याको थ्यो सन्दर्भ सामग्रीअन बाइल्ल...`
+3. `वि सं राणा शमशेर जङ्गबहादुर राणा सत्चित शमशेर जङ्गबहादुर राणा नर शमशेर जङ्गबहादुर राणा बमबहादुर राणा...`
 
-### Generated Text Samples
+**Context Size 3:**
+
+1. `सन्दर्भ सामग्रीअन भाइरा लिङ्कअन अभिनेताअन राजनीतिज्ञ`
+2. `यो लै हेर घनप्रसाद शर्मा सन्दर्भ सामग्रीअन पिडित नागरिक`
+3. `सामग्रीअन भाइरा लिङ्कअन कमंस कार्ल मार्क्स कार्ल मार्क्सको हो राष्ट्रधर्म चर्चित व्यक्तित्वअन`
+
+**Context Size 4:**
+
+1. `सन्दर्भ सामग्रीअन भाइरा लिङ्कअन यो लै हेर चलचित्र अभिनेत्रीअन मान्सु`
+2. `गाउँ विकास समिति हो विकास समितिअन`
+3. `एक गाउँ विकास समिति हो यी पन हेर्या जिल्ला विकास समितिअन`
 
-Below are text samples generated from each Markov chain model:
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `ा ग ी य ा न ् य थ ् ख ी हर ू र े`
-2. `् न ् दनहर ू तन ह ु अर ि न े ह ु स ि`
-3. `ि शम ा ल ो । यश ् बक ु न ् य े ई प`
+1. `_गण_यूक्त_भाइन्_'केसम्पर्क`
+2. `रपागरेकीय_पनिकक्ष_रै_अनु`
+3. `न_सयनले_स्रो,_विभिन्न_d_`
 
 **Context Size 2:**
 
-1. `् र तहम ै स ा ल ो र ा ज ु रस ् त ी ज`
-2. `् य ा ट ी क ो इत ि ह ु न ् छन ् । प`
-3. `य ा । प े श ह ो । सन ् दर ् भ स ा क`
+1. `को_पैल्ली_कि_लेप्चा_सम्बन्ध���त_अरे`
+2. `।_यै_प्रस्ताव_भारतका_दीबहिनी`
+3. `न_फुटबल"_आधुनिक_नसङ्ख्या_`
 
 **Context Size 3:**
 
-1. `् य ा र ् ग र ि म ि र ा ज ् यkingdom of thailand ราชอาณาจ`
-2. `् र े सक ो प ् र ा प ् द ै कय ौ ं प ्`
-3. `त ् र उपलब ् ध गर ि य ा हत ि य ा र े धनर ा`
+1. `_।_उप-अवधारणालाई_आज_स`
+2. `_रे_सम्बत_साफ्टवेयर_लिग_च्याम्पि`
+3. `अन_पिउने_विश्वामित्रो_जमघटका_`
 
 **Context Size 4:**
 
-1. `न े प ा ल ी मह ि ल ा क ो अ ं श म ा न ्`
-2. `श ् र े ण ी : स ा ङ ् ग ी त व ि क ा स`
-3. `् र े ण ी : अफ ् र ि क े टब ा ट य ा त ्`
+1. `_सन्दर्भहरू_माइ_विषेश_दिन_नजि`
+2. `_एक_अङ्ग_भङ्ग_गर्ने_अनुमति_दि`
+3. `_नेपाल_रेड्डी_निर्देशक_हिन्दी_सिनेमा`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 76.1% predictability
+- **Best Predictability:** Context-4 (word) with 98.6% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (317,836 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (594,541 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -247,64 +346,64 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 9,201 |
-| Total Tokens | 1,296,700 |
-| Mean Frequency | 140.93 |
-| Median Frequency | 4 |
-| Frequency Std Dev | 2143.27 |
+| Vocabulary Size | 32,797 |
+| Total Tokens | 456,553 |
+| Mean Frequency | 13.92 |
+| Median Frequency | 3 |
+| Frequency Std Dev | 85.63 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | र | 98,979 |
-| 2 | क | 74,013 |
-| 3 | न | 68,092 |
-| 4 | ल | 62,705 |
-| 5 | य | 57,617 |
-| 6 | म | 47,734 |
-| 7 | त | 47,517 |
-| 8 | स | 46,391 |
-| 9 | प | 40,691 |
-| 10 | द | 30,036 |
+| 1 | रे | 7,392 |
+| 2 | हो | 4,556 |
+| 3 | छ | 3,784 |
+| 4 | मी | 3,555 |
+| 5 | एक | 2,814 |
+| 6 | यो | 2,747 |
+| 7 | को | 2,624 |
+| 8 | र | 2,560 |
+| 9 | सन्दर्भ | 2,229 |
+| 10 | माइ | 2,088 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | खओ | 2 |
-| 2 | सरपछ | 2 |
-| 3 | diseases | 2 |
-| 4 | sexual | 2 |
-| 5 | pregnancy | 2 |
-| 6 | ररह | 2 |
-| 7 | pacifists | 2 |
-| 8 | she | 2 |
-| 9 | bc | 2 |
-| 10 | आङल | 2 |
+| 1 | पिक्सेल | 2 |
+| 2 | प्रयोगकर्ताहरूद्वारा | 2 |
+| 3 | महानिरीक्षकलाई | 2 |
+| 4 | महानिरीक्षकअन | 2 |
+| 5 | दुबधागो | 2 |
+| 6 | हार्बिनको | 2 |
+| 7 | अगुदा | 2 |
+| 8 | शाङ्जिङ | 2 |
+| 9 | प्रिफेक्चर | 2 |
+| 10 | लिआङले | 2 |
 
 ### Zipf's Law Analysis
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.5100 |
-| R² (Goodness of Fit) | 0.992734 |
+| Zipf Coefficient | 0.9878 |
+| R² (Goodness of Fit) | 0.989849 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 80.7% |
-| Top 1,000 | 95.5% |
-| Top 5,000 | 99.2% |
-| Top 10,000 | 0.0% |
+| Top 100 | 23.7% |
+| Top 1,000 | 52.9% |
+| Top 5,000 | 76.7% |
+| Top 10,000 | 85.9% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9927 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 80.7% of corpus
-- **Long Tail:** -799 words needed for remaining 100.0% coverage
+- **Zipf Compliance:** R²=0.9898 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 23.7% of corpus
+- **Long Tail:** 22,797 words needed for remaining 14.1% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -317,24 +416,115 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 13,106 | 32 | 3.703 | 0.820 | 0.8984 🏆 |
-| **mono_64d** | 13,106 | 64 | 4.116 | 0.757 | 0.7803 |
-| **mono_128d** | 13,106 | 128 | 4.294 | 0.756 | 0.3288 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.9032 🏆 | 0.3305 | N/A | N/A |
+| **mono_64d** | 64 | 0.7587 | 0.2622 | N/A | N/A |
+| **mono_128d** | 128 | 0.3039 | 0.2479 | N/A | N/A |
+| **aligned_32d** | 32 | 0.9032 | 0.3256 | 0.0040 | 0.0640 |
+| **aligned_64d** | 64 | 0.7587 | 0.2643 | 0.0060 | 0.0960 |
+| **aligned_128d** | 128 | 0.3039 | 0.2488 | 0.0220 | 0.1640 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8984 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 13,106 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.9032 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2799. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 2.2% R@1 in cross-lingual retrieval.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **1.309** | High formulaic/idiomatic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+| `-प्` | प्रयोगकर्ता, प्रवेशमी, प्रेसिडेन्ट |
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-ा` | क्षेत्तीमा, नैपालमा, पणया |
+| `-को` | ताराको, सिजनको, गोल्डकपको |
+| `-का` | भनेका, आदिका, फाराक्का |
+| `-ले` | ऋषिले, अहिले, देसाईले |
+| `-मी` | लडाइँमी, प्रवेशमी, आरक्षमी |
+| `-ाई` | जेफरसनलाई, पर्वलाई, कार्कीलाई |
+| `-या` | पणया, लगाइया, आत्महत्या |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+*No significant bound stems detected.*
+
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+| Prefix | Suffix | Frequency | Examples |
+|--------|--------|-----------|----------|
+| `-प्` | `-ा` | 27 words | प्रतिरक्षा, प्यासा |
+| `-प्` | `-को` | 26 words | प्रजाको, प्राणीको |
+| `-प्` | `-का` | 13 words | प्रियङ्का, प्रदर्शनका |
+| `-प्` | `-मी` | 10 words | प्रकृतिमी, प्रहरीमी |
+| `-प्` | `-ले` | 9 words | प्रकारले, प्रविधिले |
+| `-प्` | `-ाई` | 9 words | प्रधानमन्त्रीलाई, प्रचलनमाई |
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| संस्थानको | **`संस्थान-को`** | 4.5 | `संस्थान` |
+| संस्कारमी | **`संस्कार-मी`** | 4.5 | `संस्कार` |
+| सरस्वतीले | **`सरस्वती-ले`** | 4.5 | `सरस्वती` |
+| आन्दोलनको | **`आन्दोलन-को`** | 4.5 | `आन्दोलन` |
+| महिनाहरूको | **`महिनाहरू-को`** | 4.5 | `महिनाहरू` |
+| त्रिपाठीको | **`त्रिपाठी-को`** | 4.5 | `त्रिपाठी` |
+| पञ्चायतको | **`पञ्चायत-को`** | 4.5 | `पञ्चायत` |
+| सुर्मासरोवरको | **`सुर्मासरोवर-को`** | 4.5 | `सुर्मासरोवर` |
+| ब्राजिलले | **`ब्राजिल-ले`** | 4.5 | `ब्राजिल` |
+| हार्बिनको | **`हार्बिन-को`** | 4.5 | `हार्बिन` |
+| न्यायाधीशको | **`न्यायाधीश-को`** | 4.5 | `न्यायाधीश` |
+| अध्यक्षका | **`अध्यक्ष-का`** | 4.5 | `अध्यक्ष` |
+| सेमिफाइनलमी | **`सेमिफाइनल-मी`** | 4.5 | `सेमिफाइनल` |
+| संस्कृतिका | **`संस्कृति-का`** | 4.5 | `संस्कृति` |
+| सैनिकहरूको | **`सैनिकहरू-को`** | 4.5 | `सैनिकहरू` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language Dotyali shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -342,11 +532,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.90x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (686) |
-| Markov | **Context-4** | Highest predictability (76.1%) |
+| Tokenizer | **64k BPE** | Best compression (4.54x) |
+| N-gram | **2-gram** | Lowest perplexity (2,395) |
+| Markov | **Context-4** | Highest predictability (98.6%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -536,7 +727,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -552,7 +744,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-30 08:39:36*
+*Report Date: 2026-01-04 02:49:05*

models/embeddings/aligned/dty_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a6df023330e2195f800360c824bde8773548f727b69f55f71fefa7277db3f85
+size 1036935333

models/embeddings/aligned/dty_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "dty", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/dty_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3d207fd118785719cf664c19e5807f868e7802525c07d2bc8115f4ad28da9d5
+size 65664

models/embeddings/aligned/dty_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "dty",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 509,
+  "vocab_size": 12303
+}

models/embeddings/aligned/dty_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b532bfd26fd0d611e004f34565c248c86fd89a1df3af45f110674e15983c0451
+size 259486629

models/embeddings/aligned/dty_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "dty", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/dty_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19f5b91e26570d46339b263628bf08de8a797c79aea3ac13ac33e52a1f1bc2d5
+size 4224

models/embeddings/aligned/dty_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "dty",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 509,
+  "vocab_size": 12303
+}

models/embeddings/aligned/dty_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58c33d7e1814be9f2296f1d84aa16338a980c40219d858bccd193debfd3430db
+size 518636197

models/embeddings/aligned/dty_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "dty", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/dty_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82e38a5624b3ee9c2141d6447c888d0065a127bf30c96dc2b006422533c4e93e
+size 16512

models/embeddings/aligned/dty_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "dty",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 509,
+  "vocab_size": 12303
+}

models/embeddings/monolingual/dty_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:964ac2285f0e38d0803ee1c0d50f3fb3dd60e8a16f39a7cbda1241bc6de74635
-size 1037780541
+oid sha256:6a6df023330e2195f800360c824bde8773548f727b69f55f71fefa7277db3f85
+size 1036935333

models/embeddings/monolingual/dty_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 13106
+  "vocab_size": 12303
 }

models/embeddings/monolingual/dty_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:17ffb6d5f8f61eccd79dd844d5a1d9345f5be530f49ce8eb96ea1cbb7f784208
-size 259715133
+oid sha256:b532bfd26fd0d611e004f34565c248c86fd89a1df3af45f110674e15983c0451
+size 259486629

models/embeddings/monolingual/dty_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 13106
+  "vocab_size": 12303
 }

models/embeddings/monolingual/dty_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:72f1f007b40183626f203188e8d1a31a25c4698a554082b0b881babc3e1890f5
-size 519070269
+oid sha256:58c33d7e1814be9f2296f1d84aa16338a980c40219d858bccd193debfd3430db
+size 518636197

models/embeddings/monolingual/dty_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 13106
+  "vocab_size": 12303
 }

models/subword_markov/dty_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f5dddad653b07e339d9c8909e7ab6d396197b9da6ba28815641e69293f5a05bf
-size 107292
+oid sha256:5c4606e3097a1e7b7e5d045c48569856bd63a11c951ce8ddc81db29647e3c7e7
+size 430313

models/subword_markov/dty_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "dty",
-  "unique_contexts": 1664,
-  "total_transitions": 3708553
+  "unique_contexts": 6314,
+  "total_transitions": 2101190
 }

models/subword_markov/dty_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b91bebe7a5e47a8baa2ce9cf450bd83183e59cca09bf3283f1b1a4c3f17ebc35
-size 645221
+oid sha256:b58a6b0b7e539c06370b4dc01729003754b09ac8f566522be5fbdad6bba9b45b
+size 1857730

models/subword_markov/dty_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "dty",
-  "unique_contexts": 13756,
-  "total_transitions": 3704635
+  "unique_contexts": 63513,
+  "total_transitions": 2097548
 }

models/subword_markov/dty_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9b346a0fd02a8d547545dc9f1e3606f60bbffc6f4bf5c5d8b4f68ccb94ce1867
-size 2294843
+oid sha256:c664f3da30cf4762671b95ea4008dc1ffd480fe6908456e3045176ddc6ae7a46
+size 5439479

models/subword_markov/dty_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "dty",
-  "unique_contexts": 89517,
-  "total_transitions": 3700717
+  "unique_contexts": 235438,
+  "total_transitions": 2093906
 }

models/subword_markov/dty_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:56dea843402bbf323bea75f0fb852b61f045eef0c58feea816c63d1cd3e09861
-size 6457663
+oid sha256:539aec359bc1463def787e02b0f23e004d080b7a901f6b07a801b5f34c26b952
+size 11859716

models/subword_markov/dty_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "dty",
-  "unique_contexts": 317836,
-  "total_transitions": 3696799
+  "unique_contexts": 594541,
+  "total_transitions": 2090264
 }

models/subword_ngram/dty_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3859a187ae14a9ba34371fcc63a0e54952f7b111d8d6d7bea4f5938a43907515
-size 72956
+oid sha256:91059a3c0b3a0135f9e475a2821b30d7f4f889ea33e321ab06657ca0cdf3cdae
+size 282681

models/subword_ngram/dty_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "dty",
-  "unique_ngrams": 5687,
-  "total_ngrams": 3708553
+  "unique_ngrams": 19229,
+  "total_ngrams": 2101190
 }

models/subword_ngram/dty_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aa90f67b98d0785768f0ab92a0499cd2be8309aeebffc852ff93a7373f83d22b
-size 546029
+oid sha256:c3c3c678cbbadb4a67e4b0ec397979aabb37babee65ce814438a94edfdd3337f
+size 1123459

models/subword_ngram/dty_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "dty",
-  "unique_ngrams": 42059,
-  "total_ngrams": 3704635
+  "unique_ngrams": 76407,
+  "total_ngrams": 2097548
 }

models/subword_ngram/dty_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:97110596ec37c9fbc611c120615722130c0517af25276733be2dda48c710f3a0
-size 2018858
+oid sha256:6d8b34382178c2983d8d46b24d2d798a1e8771182547df5897c5d603e12d64e9
+size 3183973

models/subword_ngram/dty_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "dty",
-  "unique_ngrams": 159916,
-  "total_ngrams": 3700717
+  "unique_ngrams": 207437,
+  "total_ngrams": 2093906
 }

models/subword_ngram/dty_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dde08a0eac83ccb4af222f63d033396817e032ce57f261374d8c1965e4baa7f5
+size 3974790

models/subword_ngram/dty_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "dty",
+  "unique_ngrams": 239024,
+  "total_ngrams": 2090264
+}

models/tokenizer/dty_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:af3f58fc5ff9bcaf011ade4076073810abfcd819f4b4486ba55fd1017844a0d2
-size 629308
+oid sha256:f61a110035f023a615b3079e0b5e488eb1cb161f70edfa13063227d384131a25
+size 634336

models/tokenizer/dty_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2a716fc871e037889e6a3592fc2aa736634390782120c6d7451fdf9a8ce773e4
-size 1040508
+oid sha256:b4cf674f2960ea4d1535c519c207ceeeb67d1d6a709c91d852039347f60fde8a
+size 1064480

models/tokenizer/dty_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d6fa67e8e16413cf04cf2c7978d1ca1e2d35871cb67ae76783adfa1ef036b610
-size 1939075
+oid sha256:b0a026123a9e7792b2cd1d6318fd304f76cff9a4df713cdcab5ab10a0f463914
+size 1967380

models/tokenizer/dty_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8236a80310f925c5c1fcd0d7f3fce13195c910276821056c71ffbd35653a6c5b
-size 426991
+oid sha256:183b81e5c9a2a0b22f81a616029d1cf921f9a5756c30bcc4f1206c10f82cc0e9
+size 429380

models/vocabulary/dty_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3a9d806340037287a970008919bdfc6968895d6aa63373b131ce94cec857e1e8
-size 148910
+oid sha256:d47b7adc912f9c0df9912584fec20f1b1194798a7b7232389a2fd3bcc3b45f09
+size 632005

models/vocabulary/dty_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "dty",
-  "vocabulary_size": 9201,
+  "vocabulary_size": 32797,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.018026380217430103,
+    "type_token_ratio": 0.16816803258575846,
     "coverage": {
-      "top_100": 0.7984149598744293,
-      "top_1000": 0.9440133502754104,
-      "top_5000": 0.9815632879629389,
-      "top_10000": 0.9896006052775689
+      "top_100": 0.2125828139569122,
+      "top_1000": 0.47418363841586103,
+      "top_5000": 0.6877538401138539,
+      "top_10000": 0.7700426951955637
     },
-    "hapax_count": 14434,
-    "hapax_ratio": 0.6107044637190607,
-    "total_documents": 3918
+    "hapax_count": 52872,
+    "hapax_ratio": 0.6171660694066698,
+    "total_documents": 3642
   }
 }

models/word_markov/dty_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:11a53bfe41fbe64eeb6045e4672e71ca88d3f895a7bf3c13f336845342b033a9
-size 932289
+oid sha256:98eac32fe3c03cfb59984e22bfebb3a278222deeb0825539fda0c0a83766dcbd
+size 4402202

models/word_markov/dty_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "dty",
-  "unique_contexts": 23936,
-  "total_transitions": 2460213
+  "unique_contexts": 85572,
+  "total_transitions": 505783
 }

models/word_markov/dty_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f64fc4045548853e597a6cc2de667b4672e9ceddd1c6cc03461c4bf8f810b006
-size 2858396
+oid sha256:97680d178c288fe5131b5f36cc99d40f469cd24ce20c549a1170580d437a7c12
+size 9758496

models/word_markov/dty_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "dty",
-  "unique_contexts": 122540,
-  "total_transitions": 2456295
+  "unique_contexts": 343062,
+  "total_transitions": 502141
 }