Upload all models and assets for dsb (latest)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,6 +1,6 @@
 ---
 language: dsb
-language_name: DSB
+language_name: Lower Sorbian
 language_family: slavic_west
 tags:
   - wikilangs
@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-slavic_west
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,20 +33,20 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 4.294
+    value: 4.367
   - name: best_isotropy
     type: isotropy
-    value: 0.8252
+    value: 0.8231
   - name: vocabulary_size
     type: vocab
-    value: 32309
-generated: 2025-12-30
+    value: 0
+generated: 2026-01-04
 ---
 
-# DSB - Wikilangs Models
+# Lower Sorbian - Wikilangs Models
 ## Comprehensive Research Report & Full Ablation Study
 
-This repository contains NLP models trained and evaluated by Wikilangs, specifically on **DSB** Wikipedia data.
+This repository contains NLP models trained and evaluated by Wikilangs, specifically on **Lower Sorbian** Wikipedia data.
 We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
 
 ## 📋 Repository Contents
@@ -44,12 +54,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +70,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,55 +80,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 3.271x | 3.18 | 0.0992% | 351,825 |
-| **16k** | 3.624x | 3.52 | 0.1099% | 317,561 |
-| **32k** | 3.965x | 3.86 | 0.1202% | 290,267 |
-| **64k** | 4.294x 🏆 | 4.18 | 0.1302% | 268,009 |
+| **8k** | 3.295x | 3.30 | 0.1090% | 314,655 |
+| **16k** | 3.690x | 3.69 | 0.1221% | 280,957 |
+| **32k** | 4.049x | 4.05 | 0.1339% | 256,086 |
+| **64k** | 4.367x 🏆 | 4.37 | 0.1445% | 237,425 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Złocieniec jo město w Pólskej, w pódwjacoropomorskem wójwodstwje. Lažy w Pomorsk...`
+**Sample 1:** `Andrew Garfield (* 20. awgusta Los Angeles) jo amerikański grajaŕ. Eksterne wótk...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁zło cie niec ▁jo ▁město ▁w ▁pólskej , ▁w ▁pódwjacoro ... (+13 more)` | 23 |
-| 16k | `▁zło cie niec ▁jo ▁město ▁w ▁pólskej , ▁w ▁pódwjacoro ... (+13 more)` | 23 |
-| 32k | `▁zło cie niec ▁jo ▁město ▁w ▁pólskej , ▁w ▁pódwjacoro ... (+13 more)` | 23 |
-| 64k | `▁zło cieniec ▁jo ▁město ▁w ▁pólskej , ▁w ▁pódwjacoro pomorskem ... (+12 more)` | 22 |
-
-**Sample 2:** `Nowy Dwór Królewski jo wjas w Pólskej.
+| 8k | `▁andre w ▁gar fi el d ▁(* ▁ 2 0 ... (+12 more)` | 22 |
+| 16k | `▁andre w ▁gar fi eld ▁(* ▁ 2 0 . ... (+11 more)` | 21 |
+| 32k | `▁andrew ▁gar field ▁(* ▁ 2 0 . ▁awgusta ▁los ... (+9 more)` | 19 |
+| 64k | `▁andrew ▁garfield ▁(* ▁ 2 0 . ▁awgusta ▁los ▁angeles ... (+8 more)` | 18 |
 
-Kurów lažy mjazy městoma Chelmno a Torun...`
+**Sample 2:** `Pabianice jo město w Pólskej, w łódźskem wójwodstwje, we wokrejsu Pabianice. W l...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁nowy ▁dwór ▁k ró le wski ▁jo ▁wjas ▁w ▁pólskej ... (+21 more)` | 31 |
-| 16k | `▁nowy ▁dwór ▁kró le wski ▁jo ▁wjas ▁w ▁pólskej . ... (+19 more)` | 29 |
-| 32k | `▁nowy ▁dwór ▁króle wski ▁jo ▁wjas ▁w ▁pólskej . ▁kurów ... (+16 more)` | 26 |
-| 64k | `▁nowy ▁dwór ▁króle wski ▁jo ▁wjas ▁w ▁pólskej . ▁kurów ... (+15 more)` | 25 |
+| 8k | `▁pa bia nice ▁jo ▁město ▁w ▁pólskej , ▁w ▁łódźskem ... (+26 more)` | 36 |
+| 16k | `▁pa bia nice ▁jo ▁město ▁w ▁pólskej , ▁w ▁łódźskem ... (+26 more)` | 36 |
+| 32k | `▁pabianice ▁jo ▁město ▁w ▁pólskej , ▁w ▁łódźskem ▁wójwodstwje , ... (+22 more)` | 32 |
+| 64k | `▁pabianice ▁jo ▁město ▁w ▁pólskej , ▁w ▁łódźskem ▁wójwodstwje , ... (+22 more)` | 32 |
 
-**Sample 3:** `Janusz Gajos (* 23. septembra 1939) jo pólski grajaŕ, fotograf a pedagog.
-thumb
-...`
+**Sample 3:** `Żukowo (kaš. Żukòwò, nim. Zuckau) jo město w Pólskej, kótarež lažy w pomorskem w...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁jan usz ▁ga jo s ▁(* ▁ 2 3 . ... (+33 more)` | 43 |
-| 16k | `▁jan usz ▁ga jo s ▁(* ▁ 2 3 . ... (+32 more)` | 42 |
-| 32k | `▁janusz ▁ga jos ▁(* ▁ 2 3 . ▁septembra ▁ ... (+30 more)` | 40 |
-| 64k | `▁janusz ▁gajos ▁(* ▁ 2 3 . ▁septembra ▁ 1 ... (+29 more)` | 39 |
+| 8k | `▁ż u kowo ▁( kaš . ▁ż uk ò w ... (+22 more)` | 32 |
+| 16k | `▁ż u kowo ▁( kaš . ▁ż uk ò w ... (+22 more)` | 32 |
+| 32k | `▁żukowo ▁( kaš . ▁ż ukòwò , ▁nim . ▁zu ... (+17 more)` | 27 |
+| 64k | `▁żukowo ▁( kaš . ▁żukòwò , ▁nim . ▁zu ckau ... (+16 more)` | 26 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 4.294x compression
-- **Lowest UNK Rate:** 8k with 0.0992% unknown tokens
+- **Best Compression:** 64k achieves 4.367x compression
+- **Lowest UNK Rate:** 8k with 0.1090% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -125,57 +139,111 @@ thumb
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 5,225 🏆 | 12.35 | 14,685 | 19.8% | 49.5% |
-| **2-gram** | 508 🏆 | 8.99 | 4,003 | 51.9% | 96.7% |
-| **3-gram** | 10,031 | 13.29 | 21,515 | 13.6% | 36.7% |
-| **3-gram** | 4,632 | 12.18 | 29,259 | 18.2% | 55.2% |
-| **4-gram** | 19,104 | 14.22 | 37,295 | 10.0% | 27.6% |
-| **4-gram** | 23,026 | 14.49 | 127,625 | 9.2% | 29.7% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 4,470 | 12.13 | 8,572 | 17.8% | 48.2% |
+| **2-gram** | Subword | 446 🏆 | 8.80 | 3,440 | 54.0% | 97.7% |
+| **3-gram** | Word | 5,728 | 12.48 | 9,797 | 15.0% | 41.9% |
+| **3-gram** | Subword | 4,110 | 12.01 | 24,943 | 18.0% | 57.3% |
+| **4-gram** | Word | 10,398 | 13.34 | 16,574 | 10.9% | 31.5% |
+| **4-gram** | Subword | 21,363 | 14.38 | 109,172 | 8.1% | 29.6% |
+| **5-gram** | Word | 7,815 | 12.93 | 11,757 | 11.1% | 34.9% |
+| **5-gram** | Subword | 57,069 | 15.80 | 221,040 | 5.0% | 20.1% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `až do` | 933 |
+| 2 | `w lěśe` | 890 |
+| 3 | `jo był` | 874 |
+| 4 | `jo se` | 751 |
+| 5 | `w pólskej` | 720 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `jo město w` | 444 |
+| 2 | `w lěśe jo` | 408 |
+| 3 | `w pólskej w` | 301 |
+| 4 | `jo how bydliło` | 290 |
+| 5 | `město w pólskej` | 280 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `jo město w pólskej` | 278 |
+| 2 | `lěśe jo how bydliło` | 271 |
+| 3 | `w lěśe jo how` | 271 |
+| 4 | `město w pólskej w` | 265 |
+| 5 | `luźi galerija w pólskej` | 195 |
+
+**5-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `kategorija :` | 7,667 |
-| 2 | `) jo` | 2,004 |
-| 3 | `) .` | 1,655 |
-| 4 | `( *` | 1,443 |
-| 5 | `) ,` | 1,402 |
+| 1 | `w lěśe jo how bydliło` | 271 |
+| 2 | `jo město w pólskej w` | 264 |
+| 3 | `oslwokrejs górne błota łužyca bramborska` | 123 |
+| 4 | `spohn was blüht denn da` | 92 |
+| 5 | `bechtle spohn was blüht denn` | 92 |
 
-**3-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `kategorija : sedlišćo` | 991 |
-| 2 | `: sedlišćo w` | 874 |
-| 3 | `. kategorija :` | 760 |
-| 4 | `kategorija : roź` | 672 |
-| 5 | `: roź .` | 672 |
+| 1 | `a _` | 64,101 |
+| 2 | `e _` | 45,814 |
+| 3 | `_ w` | 44,765 |
+| 4 | `_ s` | 35,936 |
+| 5 | `o _` | 35,677 |
 
-**4-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `kategorija : sedlišćo w` | 874 |
-| 2 | `kategorija : roź .` | 672 |
-| 3 | `kategorija : wum .` | 395 |
-| 4 | `978 - 3 -` | 367 |
-| 5 | `isbn 978 - 3` | 359 |
+| 1 | `j o _` | 13,646 |
+| 2 | `_ j o` | 12,615 |
+| 3 | `_ a _` | 11,980 |
+| 4 | `n a _` | 11,930 |
+| 5 | `s k e` | 11,746 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ j o _` | 11,352 |
+| 2 | `s k i _` | 7,449 |
+| 3 | `s k e j` | 6,203 |
+| 4 | `_ w ó t` | 6,170 |
+| 5 | `s k a _` | 4,852 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ w ó t _` | 3,402 |
+| 2 | `s e r b s` | 3,221 |
+| 3 | `e r b s k` | 3,202 |
+| 4 | `_ s e r b` | 2,762 |
+| 5 | `a _ j o _` | 2,563 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 508
+- **Best Perplexity:** 2-gram (subword) with 446
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~30% of corpus
+- **Coverage:** Top-1000 patterns cover ~20% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -183,55 +251,86 @@ thumb
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.5742 | 1.489 | 3.58 | 83,705 | 42.6% |
-| **1** | 1.1222 | 2.177 | 9.59 | 1,062 | 0.0% |
-| **2** | 0.2165 | 1.162 | 1.49 | 299,024 | 78.3% |
-| **2** | 1.0246 | 2.034 | 5.91 | 10,184 | 0.0% |
-| **3** | 0.0847 | 1.061 | 1.15 | 446,488 | 91.5% |
-| **3** | 0.8420 | 1.793 | 3.87 | 60,186 | 15.8% |
-| **4** | 0.0411 🏆 | 1.029 | 1.07 | 514,008 | 95.9% |
-| **4** | 0.6005 🏆 | 1.516 | 2.44 | 232,636 | 39.9% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.6397 | 1.558 | 3.41 | 79,306 | 36.0% |
+| **1** | Subword | 1.0660 | 2.094 | 8.70 | 993 | 0.0% |
+| **2** | Word | 0.1672 | 1.123 | 1.33 | 269,674 | 83.3% |
+| **2** | Subword | 0.9899 | 1.986 | 5.80 | 8,629 | 1.0% |
+| **3** | Word | 0.0539 | 1.038 | 1.08 | 355,887 | 94.6% |
+| **3** | Subword | 0.8277 | 1.775 | 3.86 | 50,014 | 17.2% |
+| **4** | Word | 0.0234 🏆 | 1.016 | 1.03 | 383,185 | 97.7% |
+| **4** | Subword | 0.6176 | 1.534 | 2.51 | 193,064 | 38.2% |
+
+### Generated Text Samples (Word-based)
+
+Below are text samples generated from each word-based Markov chain model:
+
+**Context Size 1:**
+
+1. `a hiri słowo jo był historiski region region region iv december dartford engelska 6 kulojte až`
+2. `w pomorskem wójewódstwje we chicago homepage lfn english creoles spoken in 3 349 300 źiśi ze`
+3. `jo jano 13 v werner měškank serbski słownik za literaturu w pólskej w prien am nordrand`
+
+**Context Size 2:**
+
+1. `až do drjowku w lěśe jo how bydliło 2 467 luźi galerija w pólskej w kujawsko pomorskem`
+2. `w lěśe wóna jo była hanka krawcec cłonkojstwo domowinje pśisłušaju slědujuce towaristwa župy budyšyn...`
+3. `jo był dolnołužyska wjas pla chóśebuza wót lěta pśecej na pjerwjejšnych systemach by mógło se snaź d...`
 
-### Generated Text Samples
+**Context Size 3:**
+
+1. `jo město w pólskej w podkarpatskem wójwodstwje we wokrejsu chełmno w lěśe jo how bydliło 57 458 luźi`
+2. `w lěśe jo w sankt petersburgu jo był jaden z nejwuznamnjejšych zastupnikow tak pomjenjonego bergaŕsk...`
+3. `w pólskej w kujawsko pomorskem wójwodstwje we wokrejsu leżajsk w lěśe jo how bydliło 127 602 luźi ek...`
+
+**Context Size 4:**
+
+1. `jo město w pólskej w lublińskem wójwodstwje we wokrejsu hrubieszów w lěśe jo how bydliło 13 766 luźi...`
+2. `lěśe jo how bydliło 65 149 luźi historiski centrum jo na lisćinje unesco mě w drugich rěcach vilnius...`
+3. `w lěśe jo how bydliło 3 223 luźi galerija eksterne wótkaze biała rawska pól biała rawska pól w pólsk...`
 
-Below are text samples generated from each Markov chain model:
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. konwencionelna mutageneza pśi albańskej a litawskeju ( - 1962 : hugo gunckel lüer – nimski`
-2. `, 1 , leipzig palmenhaus auf der räuber hotzenplotz " kaž “ a twórje kupy ,`
-3. `: jazor 58 . aitingk , ale teke literarne myto ćišinskego kategorija : prizaŕske bórkowy amt`
+1. `_zojejost_łnja_s`
+2. `aropynderoveiin.`
+3. `epruroni_dpekaru`
 
 **Context Size 2:**
 
-1. `kategorija : sedlišćo w dolnej łužycy . wótkaze lisćina galiskich słowow pśi wordgumbo nastawk pśi i...`
-2. `) jo družyna droznow . samica jo brunocarna , mjaztym až se w šyrokem źělu pódpołnocneje afriki`
-3. `) . města nejwětše města su : santiago de cuba ) jo historiska slězyna japańskeje tragedije .`
+1. `a_kótka_źiw_mil_w`
+2. `e_da_kuchórbski_t`
+3. `_w_sertika_wu_re_`
 
 **Context Size 3:**
 
-1. `kategorija : sedlišćo w pólskej kategorija : pomorske wójwodstwo kategorija : sedlišćo w českej kate...`
-2. `: sedlišćo w baden - württembergskej kategorija : rěka w českej , ako na pśikład za staty ,`
-3. `. kategorija : sad kategorija : bomy kategorija : hybridy`
+1. `jo_spis_krěpojcne_`
+2. `_jo_septemata_kral`
+3. `_a_wótšy_pśeder_wi`
 
 **Context Size 4:**
 
-1. `kategorija : sedlišćo w argentinskej kategorija : stolica w europje kategorija : rěka , alfabetiski ...`
-2. `kategorija : roź . 1757 kategorija : wum . 1913`
-3. `kategorija : wum . 1985 kategorija : słowakski spiwaŕ kategorija : muž`
+1. `_jo_kupki_spisowaśe`
+2. `ski_casom_stiftung_`
+3. `_wótwezeł._pěś_žołt`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 95.9% predictability
+- **Best Predictability:** Context-4 (word) with 97.7% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (232,636 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (193,064 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -247,36 +346,36 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 32,309 |
-| Total Tokens | 432,054 |
-| Mean Frequency | 13.37 |
+| Vocabulary Size | 31,116 |
+| Total Tokens | 390,195 |
+| Mean Frequency | 12.54 |
 | Median Frequency | 3 |
-| Frequency Std Dev | 142.47 |
+| Frequency Std Dev | 136.48 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | a | 12,388 |
-| 2 | w | 12,290 |
-| 3 | jo | 11,552 |
-| 4 | kategorija | 7,674 |
-| 5 | na | 4,664 |
-| 6 | z | 4,262 |
-| 7 | se | 3,646 |
-| 8 | wót | 3,524 |
-| 9 | su | 2,927 |
-| 10 | do | 2,441 |
+| 1 | a | 12,373 |
+| 2 | w | 12,119 |
+| 3 | jo | 11,480 |
+| 4 | na | 4,655 |
+| 5 | z | 4,220 |
+| 6 | se | 3,637 |
+| 7 | wót | 3,522 |
+| 8 | su | 2,923 |
+| 9 | do | 2,438 |
+| 10 | za | 1,989 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | 1474wjerchojstwo | 2 |
-| 2 | wolgast5 | 2 |
-| 3 | 1478wjerchojstwo | 2 |
-| 4 | 1592 | 2 |
-| 5 | 1625wjerchojstwo | 2 |
+| 1 | wikowje | 2 |
+| 2 | kšace | 2 |
+| 3 | gotował | 2 |
+| 4 | moderěrował | 2 |
+| 5 | procowarjow | 2 |
 | 6 | zachdniego | 2 |
 | 7 | gdanskiego | 2 |
 | 8 | podzially | 2 |
@@ -287,24 +386,24 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 0.9658 |
-| R² (Goodness of Fit) | 0.995856 |
+| Zipf Coefficient | 0.9483 |
+| R² (Goodness of Fit) | 0.996724 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 30.4% |
-| Top 1,000 | 57.0% |
-| Top 5,000 | 77.2% |
-| Top 10,000 | 85.9% |
+| Top 100 | 30.7% |
+| Top 1,000 | 56.8% |
+| Top 5,000 | 76.7% |
+| Top 10,000 | 85.6% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9959 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 30.4% of corpus
-- **Long Tail:** 22,309 words needed for remaining 14.1% coverage
+- **Zipf Compliance:** R²=0.9967 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 30.7% of corpus
+- **Long Tail:** 21,116 words needed for remaining 14.4% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -317,24 +416,121 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 11,406 | 32 | 4.255 | 0.918 | 0.8252 🏆 |
-| **mono_64d** | 11,406 | 64 | 4.511 | 0.859 | 0.5783 |
-| **mono_128d** | 11,406 | 128 | 4.576 | 0.852 | 0.1767 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8231 🏆 | 0.3397 | N/A | N/A |
+| **mono_64d** | 64 | 0.5887 | 0.3131 | N/A | N/A |
+| **mono_128d** | 128 | 0.1790 | 0.3018 | N/A | N/A |
+| **aligned_32d** | 32 | 0.8231 | 0.3455 | 0.0460 | 0.2420 |
+| **aligned_64d** | 64 | 0.5887 | 0.3066 | 0.0660 | 0.3060 |
+| **aligned_128d** | 128 | 0.1790 | 0.3019 | 0.0860 | 0.3460 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8252 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 11,406 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8231 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.3181. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 8.6% R@1 in cross-lingual retrieval.
+- **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
-## 6. Summary & Recommendations
+## 6.  Morphological Analysis (Experimental)
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **0.741** | High formulaic/idiomatic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-a` | trilogija, rinetta, kenija |
+| `-e` | gališćinje, evidence, hercegowinje |
+| `-je` | gališćinje, hercegowinje, wótstoje |
+| `-ch` | reichenbach, proch, žurnalistiskich |
+| `-ka` | hypotetiska, francoska, wěrika |
+| `-ki` | monotypiski, wólšynki, keltiski |
+| `-ow` | dokusow, wunjow, basnikow |
+| `-nje` | gališćinje, hercegowinje, wótchylenje |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `šćin` | 1.95x | 41 contexts | češćinu, češćina, češćiny |
+| `jenj` | 1.71x | 62 contexts | jenje, mjenju, mjenja |
+| `ótar` | 2.17x | 19 contexts | kótara, kótaru, kótare |
+| `skej` | 1.53x | 56 contexts | českej, wuskej, irskej |
+| `měst` | 1.87x | 25 contexts | městy, města, město |
+| `rbsk` | 1.95x | 17 contexts | srbská, serbsku, serbsko |
+| `owan` | 1.70x | 26 contexts | głowan, cowanje, źěkowano |
+| `kóta` | 2.17x | 12 contexts | kótara, kótaru, kótare |
+| `iski` | 1.63x | 25 contexts | niski, bliski, leniski |
+| `iske` | 1.46x | 36 contexts | niske, aziske, bliske |
+| `erbs` | 1.90x | 14 contexts | herbst, serbsku, serbsko |
+| `imsk` | 1.72x | 16 contexts | nimska, nimsko, nimske |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+*No significant affix co-occurrences detected.*
+
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| wóznamjenjenje | **`wóznam-je-nje-nje`** | 7.5 | `wóznam` |
+| biologowka | **`biolog-ow-ka`** | 6.0 | `biolog` |
+| pósćonych | **`pósćony-ch`** | 4.5 | `pósćony` |
+| wótstojecych | **`wótstojecy-ch`** | 4.5 | `wótstojecy` |
+| pomorskeje | **`pomorske-je`** | 4.5 | `pomorske` |
+| halšterje | **`halšter-je`** | 4.5 | `halšter` |
+| nejlěpšych | **`nejlěpšy-ch`** | 4.5 | `nejlěpšy` |
+| kamjentnych | **`kamjentny-ch`** | 4.5 | `kamjentny` |
+| pódpołnocnje | **`pódpołnoc-nje`** | 4.5 | `pódpołnoc` |
+| spominanje | **`spomina-nje`** | 4.5 | `spomina` |
+| pódwjacorneje | **`pódwjacorne-je`** | 4.5 | `pódwjacorne` |
+| organiskeje | **`organiske-je`** | 4.5 | `organiske` |
+| wótpósłańcka | **`wótpósłańc-ka`** | 4.5 | `wótpósłańc` |
+| chinskeje | **`chinske-je`** | 4.5 | `chinske` |
+| twarjenjach | **`twarjenja-ch`** | 4.5 | `twarjenja` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language Lower Sorbian shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
+
+> **Note on Idiomaticity:** The high Idiomaticity Gap suggests a large number of frequent multi-word expressions or formulaic sequences that are statistically distinct from their component parts.
+
+---
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -342,11 +538,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (4.29x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (508) |
-| Markov | **Context-4** | Highest predictability (95.9%) |
+| Tokenizer | **64k BPE** | Best compression (4.37x) |
+| N-gram | **2-gram** | Lowest perplexity (446) |
+| Markov | **Context-4** | Highest predictability (97.7%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -536,7 +733,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -552,7 +750,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-30 08:33:54*
+*Report Date: 2026-01-04 02:35:27*

models/embeddings/aligned/dsb_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:44d5e48685a3f6634762f9d640be037f055cde481781bb0f0d0d7db242549891
+size 1034871950

models/embeddings/aligned/dsb_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "dsb", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/dsb_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:292973cad54ffbfb67fbdb195461d778020da4f9f4937d7a2029a168f7c5565b
+size 65664

models/embeddings/aligned/dsb_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "dsb",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 2894,
+  "vocab_size": 10440
+}

models/embeddings/aligned/dsb_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:219809436b8357d45d070c67fa33443687fa4b37032640fa3568d2294e0073b8
+size 258854030

models/embeddings/aligned/dsb_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "dsb", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/dsb_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be16301266ab3358f75636b55076ae6198c1bcb4a786fb822487c6991d261dd8
+size 4224

models/embeddings/aligned/dsb_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "dsb",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 2894,
+  "vocab_size": 10440
+}

models/embeddings/aligned/dsb_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38d8f796364be353447c2c77dd18cd77fbe362a7edc792dbef86324ea3dc2841
+size 517526670

models/embeddings/aligned/dsb_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "dsb", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/dsb_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dbcbd4592b3c876a2ba35cfa0bf8d8d6d29218bbb5e7c880a1d6584b55575d26
+size 16512

models/embeddings/aligned/dsb_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "dsb",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 2894,
+  "vocab_size": 10440
+}

models/embeddings/monolingual/dsb_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:69c984e87d1636c608606c4c5465d8002825ca2f54d231f3d0a0277f593edecb
-size 1035879476
+oid sha256:44d5e48685a3f6634762f9d640be037f055cde481781bb0f0d0d7db242549891
+size 1034871950

models/embeddings/monolingual/dsb_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 11406
+  "vocab_size": 10440
 }

models/embeddings/monolingual/dsb_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:081fb3a085c2c647fb4256975b1946eef7c9db9c3214387b4f47c18933cc0175
-size 259119668
+oid sha256:219809436b8357d45d070c67fa33443687fa4b37032640fa3568d2294e0073b8
+size 258854030

models/embeddings/monolingual/dsb_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 11406
+  "vocab_size": 10440
 }

models/embeddings/monolingual/dsb_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:811da793a1d91d94aecf05624b1782db2a0780ebddab0716c5a7449a1314bee9
-size 518039604
+oid sha256:38d8f796364be353447c2c77dd18cd77fbe362a7edc792dbef86324ea3dc2841
+size 517526670

models/embeddings/monolingual/dsb_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 11406
+  "vocab_size": 10440
 }

models/subword_markov/dsb_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e75991d06d216913d5349e487fb7f5dda5adf4a84e1112c7d75804a41f526651
-size 78473
+oid sha256:436c54ec357fb0763cb0fa733bc0d9b76ece24f29c3e7c54a45c45b049e148d5
+size 68396

models/subword_markov/dsb_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "dsb",
-  "unique_contexts": 1062,
-  "total_transitions": 3435229
+  "unique_contexts": 993,
+  "total_transitions": 3036522
 }

models/subword_markov/dsb_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d2d04158c387dad22bea4d026f6486e27063cbccf1b5b267c23067200f38df80
-size 474937
+oid sha256:cf8c66b2f77c86ec521fc7217c756384af0d18bbcf9b0d8a2730634967f26a23
+size 384677

models/subword_markov/dsb_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "dsb",
-  "unique_contexts": 10184,
-  "total_transitions": 3431780
+  "unique_contexts": 8629,
+  "total_transitions": 3033131
 }

models/subword_markov/dsb_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:65e073c2672b04e0cc6a210d14c210bd5f5b2d610e9df300eec4e645bb58b544
-size 1661127
+oid sha256:ec822035fb412125284dd467c486f26ea20024799da89ad39855500832b0909d
+size 1385595

models/subword_markov/dsb_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "dsb",
-  "unique_contexts": 60186,
-  "total_transitions": 3428331
+  "unique_contexts": 50014,
+  "total_transitions": 3029740
 }

models/subword_markov/dsb_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8e1e13ba8381c4b0aeedbfd268e3df569421c02c0509adce385934b61c9d4134
-size 4548151
+oid sha256:a2799c7de3f7a71ac3bd4755535b089956b3956e85f98305a147b00e57292674
+size 3794359

models/subword_markov/dsb_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "dsb",
-  "unique_contexts": 232636,
-  "total_transitions": 3424882
+  "unique_contexts": 193064,
+  "total_transitions": 3026349
 }

models/subword_ngram/dsb_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:54ab8a0b169ac29131c7497aab72ed0c6262588ee25933132b96a9ea893c6cab
-size 52880
+oid sha256:746b922edca237142efcbd8f0a2604d0bc113fc7492d8d2132521b3bfa37b718
+size 45593

models/subword_ngram/dsb_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "dsb",
-  "unique_ngrams": 4003,
-  "total_ngrams": 3435229
+  "unique_ngrams": 3440,
+  "total_ngrams": 3036522
 }

models/subword_ngram/dsb_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:491b077d6277a92677cbfb36b213f6c9845b6625d639c685caaf3c81b098ae71
-size 355310
+oid sha256:aafff3e6d0a5d49cc78d1782ea111c41a440abaac93567a6bf42e0d2b8ef7c96
+size 303856

models/subword_ngram/dsb_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "dsb",
-  "unique_ngrams": 29259,
-  "total_ngrams": 3431780
+  "unique_ngrams": 24943,
+  "total_ngrams": 3033131
 }

models/subword_ngram/dsb_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e0210a38e1f600c563d26ca91f697d6f95cf6c35633b5d062baf24f897ca3ea6
-size 1444626
+oid sha256:2abb409c94e0aeed705ed22a3fcd724bf52f441f36328a3429e4afdc66c159f4
+size 1242812

models/subword_ngram/dsb_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "dsb",
-  "unique_ngrams": 127625,
-  "total_ngrams": 3428331
+  "unique_ngrams": 109172,
+  "total_ngrams": 3029740
 }

models/subword_ngram/dsb_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de69022eb2dd2a5cdaebc53519477c8f8fc3713ae4325ba2d493b8abe4562b8b
+size 2555147

models/subword_ngram/dsb_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "dsb",
+  "unique_ngrams": 221040,
+  "total_ngrams": 3026349
+}

models/tokenizer/dsb_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:83223ed94450b083f445917ad19d45108f784776f35590e24106f870e89fc048
-size 509210
+oid sha256:f450890dd63280defd15857196d9e6c71c97ea5f5fd59cb2b9bba9938ab713c8
+size 512642

models/tokenizer/dsb_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:385d9c2bd426307bc0210de4e33f8ef09b6d82186b16307f29650deb8b0128a5
-size 785019
+oid sha256:a02834e9aaaec4da88204cbbcb026a0d36f29083c0a5a9feb0277ef838fc7254
+size 789886

models/tokenizer/dsb_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:be1039984143149d1392ae5a765816e6ff2af6dad234612968afce460c6ae6d9
-size 1380290
+oid sha256:ad041a92ae3179f3718bcb864966e84846b5f19fe718d3055ef75582461abc58
+size 1397025

models/tokenizer/dsb_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:230d8ebcf4cbf468966a99ee03e43588c6326a65ab777732705c0175ce721003
-size 372857
+oid sha256:654c08cb20974ff3a4b64ea86a20bd26d208687795ee4243654e854197027768
+size 373991

models/vocabulary/dsb_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:87ff7058d1987d1d5ac416a3afd9abb9662e91efa908990e180308c27a05e007
-size 548787
+oid sha256:b75512214fe0946039a8216e6100a74a504dc95745bbd72f667b8d1ea1f2729d
+size 537376

models/vocabulary/dsb_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "dsb",
-  "vocabulary_size": 32309,
+  "vocabulary_size": 31116,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.17291342948830685,
+    "type_token_ratio": 0.18104502121060073,
     "coverage": {
-      "top_100": 0.2721360928748627,
-      "top_1000": 0.5094544574264923,
-      "top_5000": 0.6904536774001329,
-      "top_10000": 0.7675997326806216
+      "top_100": 0.2729781508005291,
+      "top_1000": 0.5055808967750764,
+      "top_5000": 0.682700816494093,
+      "top_10000": 0.7613510924599736
     },
-    "hapax_count": 51263,
-    "hapax_ratio": 0.6133992246206863,
-    "total_documents": 3449
+    "hapax_count": 48265,
+    "hapax_ratio": 0.6080170317834243,
+    "total_documents": 3391
   }
 }

models/word_markov/dsb_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:578726324c9438985c16ae29b0fba1e4d2c95ea67256780dc13e4597c96325ed
-size 2811261
+oid sha256:e720612de97bb0cc8b9d9579b46438fbc250d4c04655167cbb4dda7aebbdc7b7
+size 2583382

models/word_markov/dsb_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "dsb",
-  "unique_contexts": 83705,
-  "total_transitions": 642857
+  "unique_contexts": 79306,
+  "total_transitions": 435069
 }

models/word_markov/dsb_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4d179629dada8b928138f75fd99ce884b747702d0d96c228aee1ff738dbbb6de
-size 5734522
+oid sha256:705c1a49a9dd1d5570181c4a3554a2b3af42707817a799b45653e480300f19b1
+size 5168511

models/word_markov/dsb_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "dsb",
-  "unique_contexts": 299024,
-  "total_transitions": 639409
+  "unique_contexts": 269674,
+  "total_transitions": 431678
 }