Upload all models and assets for cy (latest)

.gitattributes

@@ -39,3 +39,4 @@ visualizations/position_encoding_comparison.png filter=lfs diff=lfs merge=lfs -t
 visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
 visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
 visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_tsne_multilingual.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -10,11 +10,21 @@ tags:
   - n-gram
   - markov
   - wikipedia
+  - feature-extraction
+  - sentence-similarity
+  - tokenization
+  - n-grams
+  - markov-chain
+  - text-mining
+  - fasttext
+  - babelvec
+  - vocabulous
+  - vocabulary
   - monolingual
   - family-celtic_brythonic
 license: mit
 library_name: wikilangs
-pipeline_tag: feature-extraction
+pipeline_tag: text-generation
 datasets:
   - omarkamali/wikipedia-monthly
 dataset_info:
@@ -23,14 +33,14 @@ dataset_info:
 metrics:
   - name: best_compression_ratio
     type: compression
-    value: 3.377
+    value: 4.109
   - name: best_isotropy
     type: isotropy
-    value: 0.8061
+    value: 0.8420
   - name: vocabulary_size
     type: vocab
-    value: 389645
-generated: 2025-12-29
+    value: 0
+generated: 2026-01-04
 ---
 
 # Welsh - Wikilangs Models
@@ -44,12 +54,13 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 ### Models & Assets
 
 - Tokenizers (8k, 16k, 32k, 64k)
-- N-gram models (2, 3, 4-gram)
-- Markov chains (context of 1, 2, 3 and 4)
+- N-gram models (2, 3, 4, 5-gram)
+- Markov chains (context of 1, 2, 3, 4 and 5)
 - Subword N-gram and Markov chains
-- Embeddings in various sizes and dimensions
+- Embeddings in various sizes and dimensions (aligned and unaligned)
 - Language Vocabulary
 - Language Statistics
+
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
 ### Analysis and Evaluation
@@ -59,7 +70,8 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 - [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
 - [4. Vocabulary Analysis](#4-vocabulary-analysis)
 - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
-- [6. Summary & Recommendations](#6-summary--recommendations)
+- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
+- [7. Summary & Recommendations](#7-summary--recommendations)
 - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
 - [Visualizations Index](#visualizations-index)
 
@@ -68,55 +80,57 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 ![Tokenizer Compression](visualizations/tokenizer_compression.png)
 
+![Tokenizer Fertility](visualizations/tokenizer_fertility.png)
+
+![Tokenizer OOV](visualizations/tokenizer_oov.png)
+
+![Total Tokens](visualizations/tokenizer_total_tokens.png)
+
 ### Results
 
 | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
 |------------|-------------|---------------|----------|--------------|
-| **8k** | 2.890x | 2.85 | 0.0354% | 1,175,001 |
-| **16k** | 3.105x | 3.07 | 0.0380% | 1,093,811 |
-| **32k** | 3.264x | 3.22 | 0.0400% | 1,040,478 |
-| **64k** | 3.377x 🏆 | 3.33 | 0.0414% | 1,005,729 |
+| **8k** | 3.346x | 3.35 | 0.0427% | 894,556 |
+| **16k** | 3.678x | 3.68 | 0.0469% | 813,770 |
+| **32k** | 3.925x | 3.93 | 0.0501% | 762,670 |
+| **64k** | 4.109x 🏆 | 4.11 | 0.0524% | 728,422 |
 
 ### Tokenization Examples
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Roedd Despoena (hefyd Despina) yn nymff oedd yn ferch i Boseidon a Demeter ym my...`
+**Sample 1:** `Canwr opera o Ganada oedd Jonathan Stewart Vickers, CC (29 Hydref – 10 Gorffenna...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁roedd ▁des po ena ▁( h efyd ▁des p ina ... (+26 more)` | 36 |
-| 16k | `▁roedd ▁des po ena ▁( hefyd ▁des p ina ) ... (+23 more)` | 33 |
-| 32k | `▁roedd ▁des po ena ▁( hefyd ▁desp ina ) ▁yn ... (+20 more)` | 30 |
-| 64k | `▁roedd ▁des po ena ▁( hefyd ▁desp ina ) ▁yn ... (+17 more)` | 27 |
+| 8k | `▁canwr ▁opera ▁o ▁ganada ▁oedd ▁jonathan ▁stewart ▁v ick ers ... (+30 more)` | 40 |
+| 16k | `▁canwr ▁opera ▁o ▁ganada ▁oedd ▁jonathan ▁stewart ▁vick ers , ... (+27 more)` | 37 |
+| 32k | `▁canwr ▁opera ▁o ▁ganada ▁oedd ▁jonathan ▁stewart ▁vick ers , ... (+26 more)` | 36 |
+| 64k | `▁canwr ▁opera ▁o ▁ganada ▁oedd ▁jonathan ▁stewart ▁vickers , ▁cc ... (+22 more)` | 32 |
 
-**Sample 2:** `Gregg Township, Centre County, Pennsylvania
-Gregg Township, Union County, Pennsy...`
+**Sample 2:** `Pêl-droediwr o Japan yw (ganed 11 Rhagfyr Tîm Cenedlaethol Tîm cenedlaethol Dole...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁greg g ▁township , ▁cent re ▁county , ▁pennsylvania ▁greg ... (+7 more)` | 17 |
-| 16k | `▁greg g ▁township , ▁centre ▁county , ▁pennsylvania ▁greg g ... (+6 more)` | 16 |
-| 32k | `▁gregg ▁township , ▁centre ▁county , ▁pennsylvania ▁gregg ▁township , ... (+4 more)` | 14 |
-| 64k | `▁gregg ▁township , ▁centre ▁county , ▁pennsylvania ▁gregg ▁township , ... (+4 more)` | 14 |
-
-**Sample 3:** `Ceir sawl Swydd Butte yn yr Unol Daleithiau:
+| 8k | `▁pêl - droediwr ▁o ▁japan ▁yw ▁( ganed ▁ 1 ... (+15 more)` | 25 |
+| 16k | `▁pêl - droediwr ▁o ▁japan ▁yw ▁( ganed ▁ 1 ... (+15 more)` | 25 |
+| 32k | `▁pêl - droediwr ▁o ▁japan ▁yw ▁( ganed ▁ 1 ... (+15 more)` | 25 |
+| 64k | `▁pêl - droediwr ▁o ▁japan ▁yw ▁( ganed ▁ 1 ... (+15 more)` | 25 |
 
- Swydd Butte, Califfornia
- Swydd B...`
+**Sample 3:** `Clostridium tetani yw'r bacteria sy'n achosi Tetanws.`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ceir ▁sawl ▁swydd ▁butt e ▁yn ▁yr ▁unol ▁daleithiau : ... (+16 more)` | 26 |
-| 16k | `▁ceir ▁sawl ▁swydd ▁butt e ▁yn ▁yr ▁unol ▁daleithiau : ... (+16 more)` | 26 |
-| 32k | `▁ceir ▁sawl ▁swydd ▁butt e ▁yn ▁yr ▁unol ▁daleithiau : ... (+16 more)` | 26 |
-| 64k | `▁ceir ▁sawl ▁swydd ▁butte ▁yn ▁yr ▁unol ▁daleithiau : ▁swydd ... (+12 more)` | 22 |
+| 8k | `▁cl ost rid ium ▁t et ani ▁yw ' r ... (+13 more)` | 23 |
+| 16k | `▁cl ost rid ium ▁t et ani ▁yw ' r ... (+12 more)` | 22 |
+| 32k | `▁cl ost rid ium ▁tet ani ▁yw ' r ▁bacteria ... (+8 more)` | 18 |
+| 64k | `▁cl ost rid ium ▁tet ani ▁yw ' r ▁bacteria ... (+8 more)` | 18 |
 
 
 ### Key Findings
 
-- **Best Compression:** 64k achieves 3.377x compression
-- **Lowest UNK Rate:** 8k with 0.0354% unknown tokens
+- **Best Compression:** 64k achieves 4.109x compression
+- **Lowest UNK Rate:** 8k with 0.0427% unknown tokens
 - **Trade-off:** Larger vocabularies improve compression but increase model size
 - **Recommendation:** 32k vocabulary provides optimal balance for production use
 
@@ -125,57 +139,111 @@ Gregg Township, Union County, Pennsy...`
 
 ![N-gram Perplexity](visualizations/ngram_perplexity.png)
 
+![N-gram Unique](visualizations/ngram_unique.png)
+
 ![N-gram Coverage](visualizations/ngram_coverage.png)
 
 ### Results
 
-| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
-|--------|------------|---------|----------------|------------------|-------------------|
-| **2-gram** | 14,574 🏆 | 13.83 | 870,365 | 26.9% | 51.5% |
-| **2-gram** | 329 🏆 | 8.36 | 17,053 | 62.3% | 99.0% |
-| **3-gram** | 35,282 | 15.11 | 1,913,384 | 21.8% | 43.7% |
-| **3-gram** | 2,419 | 11.24 | 127,945 | 27.9% | 70.4% |
-| **4-gram** | 69,076 | 16.08 | 3,444,441 | 17.0% | 37.6% |
-| **4-gram** | 10,383 | 13.34 | 705,121 | 18.5% | 47.3% |
+| N-gram | Variant | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|---------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | Word | 17,960 | 14.13 | 742,720 | 26.5% | 49.5% |
+| **2-gram** | Subword | 266 🏆 | 8.05 | 14,977 | 67.6% | 99.3% |
+| **3-gram** | Word | 34,403 | 15.07 | 1,470,847 | 23.6% | 43.7% |
+| **3-gram** | Subword | 2,056 | 11.01 | 96,402 | 28.1% | 74.0% |
+| **4-gram** | Word | 58,140 | 15.83 | 2,520,966 | 20.5% | 39.2% |
+| **4-gram** | Subword | 9,573 | 13.22 | 505,960 | 17.0% | 48.0% |
+| **5-gram** | Word | 66,270 | 16.02 | 2,303,277 | 18.6% | 36.6% |
+| **5-gram** | Subword | 29,179 | 14.83 | 1,685,188 | 12.7% | 38.3% |
 
 ### Top 5 N-grams by Size
 
-**2-grams:**
+**2-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `unol daleithiau` | 486,479 |
+| 2 | `daleithiau america` | 459,399 |
+| 3 | `y ffilm` | 330,346 |
+| 4 | `y cyfarwyddwr` | 255,174 |
+| 5 | `o ffilmiau` | 249,770 |
+
+**3-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `unol daleithiau america` | 447,977 |
+| 2 | `daleithiau america saesneg` | 189,392 |
+| 3 | `gan y cyfarwyddwr` | 147,806 |
+| 4 | `gan gynnwys y` | 143,480 |
+| 5 | `gynnwys y canlynol` | 142,458 |
+
+**4-grams (Word):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `unol daleithiau america saesneg` | 183,879 |
+| 2 | `gan gynnwys y canlynol` | 142,457 |
+| 3 | `o ffilmiau gan gynnwys` | 141,034 |
+| 4 | `nifer o ffilmiau gan` | 141,018 |
+| 5 | `ffilmiau gan gynnwys y` | 141,004 |
+
+**5-grams (Word):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `categori :` | 1,494,932 |
-| 2 | `- 01` | 1,376,508 |
-| 3 | `: ffilmiau` | 1,056,825 |
-| 4 | `01 -` | 688,052 |
-| 5 | `' r` | 677,131 |
+| 1 | `nifer o ffilmiau gan gynnwys` | 141,016 |
+| 2 | `o ffilmiau gan gynnwys y` | 141,003 |
+| 3 | `ffilmiau gan gynnwys y canlynol` | 140,997 |
+| 4 | `y nodwyd cyhoeddwyd y ffilm` | 140,932 |
+| 5 | `fel y nodwyd cyhoeddwyd y` | 140,932 |
 
-**3-grams:**
+**2-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `categori : ffilmiau` | 1,056,809 |
-| 2 | `- 01 -` | 688,032 |
-| 3 | `01 - 01` | 668,773 |
-| 4 | `unol daleithiau america` | 448,243 |
-| 5 | `cyfeiriadau categori :` | 227,847 |
+| 1 | `n _` | 6,965,988 |
+| 2 | `d _` | 6,143,531 |
+| 3 | `_ y` | 5,977,485 |
+| 4 | `d d` | 5,740,307 |
+| 5 | `_ a` | 5,232,448 |
 
-**4-grams:**
+**3-grams (Subword):**
 
 | Rank | N-gram | Count |
 |------|--------|-------|
-| 1 | `- 01 - 01` | 668,772 |
-| 2 | `unol daleithiau america saesneg` | 152,032 |
-| 3 | `gan gynnwys y canlynol` | 142,457 |
-| 4 | `gynnwys y canlynol :` | 142,446 |
-| 5 | `o ffilmiau gan gynnwys` | 141,007 |
+| 1 | `y n _` | 2,646,508 |
+| 2 | `d d _` | 2,490,077 |
+| 3 | `_ y n` | 2,304,841 |
+| 4 | `w y d` | 2,285,145 |
+| 5 | `_ y _` | 2,240,041 |
+
+**4-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `_ y n _` | 2,171,280 |
+| 2 | `f i l m` | 1,586,546 |
+| 3 | `f f i l` | 1,571,751 |
+| 4 | `_ f f i` | 1,455,954 |
+| 5 | `i l m _` | 1,222,896 |
+
+**5-grams (Subword):**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `f f i l m` | 1,569,304 |
+| 2 | `_ f f i l` | 1,419,063 |
+| 3 | `f i l m _` | 1,222,863 |
+| 4 | `_ g a n _` | 924,207 |
+| 5 | `w y d _ y` | 781,315 |
 
 
 ### Key Findings
 
-- **Best Perplexity:** 2-gram with 329
+- **Best Perplexity:** 2-gram (subword) with 266
 - **Entropy Trend:** Decreases with larger n-grams (more predictable)
-- **Coverage:** Top-1000 patterns cover ~47% of corpus
+- **Coverage:** Top-1000 patterns cover ~38% of corpus
 - **Recommendation:** 4-gram or 5-gram for best predictive performance
 
 ---
@@ -183,55 +251,86 @@ Gregg Township, Union County, Pennsy...`
 
 ![Markov Entropy](visualizations/markov_entropy.png)
 
+![Markov Contexts](visualizations/markov_contexts.png)
+
 ![Markov Branching](visualizations/markov_branching.png)
 
 ### Results
 
-| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
-|---------|-------------|------------|------------------|-----------------|----------------|
-| **1** | 0.7241 | 1.652 | 7.40 | 841,116 | 27.6% |
-| **1** | 1.3771 | 2.597 | 9.31 | 5,659 | 0.0% |
-| **2** | 0.4190 | 1.337 | 2.47 | 6,216,866 | 58.1% |
-| **2** | 0.7856 | 1.724 | 5.09 | 52,646 | 21.4% |
-| **3** | 0.1820 | 1.134 | 1.44 | 15,354,414 | 81.8% |
-| **3** | 0.7948 | 1.735 | 4.45 | 267,808 | 20.5% |
-| **4** | 0.0927 🏆 | 1.066 | 1.20 | 22,160,397 | 90.7% |
-| **4** | 0.7045 🏆 | 1.630 | 3.44 | 1,190,900 | 29.5% |
+| Context | Variant | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | Word | 0.9977 | 1.997 | 9.84 | 671,269 | 0.2% |
+| **1** | Subword | 1.0862 | 2.123 | 6.65 | 8,673 | 0.0% |
+| **2** | Word | 0.3690 | 1.291 | 2.18 | 6,591,949 | 63.1% |
+| **2** | Subword | 0.6157 | 1.532 | 3.90 | 57,679 | 38.4% |
+| **3** | Word | 0.1502 | 1.110 | 1.34 | 14,351,859 | 85.0% |
+| **3** | Subword | 0.6340 | 1.552 | 3.78 | 225,011 | 36.6% |
+| **4** | Word | 0.0687 🏆 | 1.049 | 1.14 | 19,154,889 | 93.1% |
+| **4** | Subword | 0.6561 | 1.576 | 3.40 | 850,309 | 34.4% |
 
-### Generated Text Samples
+### Generated Text Samples (Word-based)
 
-Below are text samples generated from each Markov chain model:
+Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `. ymosododd ar ddeg ysgrif ) eu brest . derbyniad gweler hefyd cyhoeddodd carlos roffé ,`
-2. `- 01 ez naiz sbaen sbaeneg a ' w gael eu galw ' r ffilm gore`
-3. `y cyfarwyddwr ffilm ddrama gan gyfarwyddwyr ffilm maneater a chyfansoddwyd y deyrnas unedig saesneg ...`
+1. `y ffindir gweler hefyd cyhoeddodd nifer o r almaen almaenegno unknown value the white ship mutiny`
+2. `yn ystod eang derbyniad gweler hefyd rhestr goch yr enw tacson delwedd gwlad dyddiad a 22`
+3. `o leiaf 1 050 o ffilmiau gan nifer o fariau cul o awstria almaeneg cyfeiriadau gan`
 
 **Context Size 2:**
 
-1. `categori : ffilmiau 1968 categori : gwyddonwyr ffrengig categori : ffilmiau 2006 categori : ffilmiau...`
-2. `- 01 just in roseland . ar ôl i weld ei thad wedi cyfaddef mewn u od`
-3. `: ffilmiau dogfen categori : ffilmiau saesneg wedi gweld golau dydd . cyfarwyddwr ganwyd y cyfarwydd...`
+1. `unol daleithiau america rhamantaidd gyda llai na 10 o actorion lleisiol a olygwyd gan mogens skot ha...`
+2. `daleithiau america cyfeiriadau gan gyfarwyddwyr ffilm gwrywaidd saesneg du a gwyn o japan mud sydd a...`
+3. `y ffilm hon yw warner baxter stuart erwin edmund lowe cafodd ei ddanfon gan fyddin a adwaenid`
 
 **Context Size 3:**
 
-1. `categori : ffilmiau gyda llai na 10 o actorion lleisiol categori : ffilmiau comedi categori : dramâu...`
-2. `- 01 - 01 jenatsch ffraincyr almaeny swistir almaenegportiwgalegsaesnegffrangeg 2018 - 05 - 15 prese...`
-3. `01 - 01 glamour y deyrnas unedig saesneg 1950 - 01 - 01 zero india 2018 - 01`
+1. `unol daleithiau america in every womans life unol daleithiau america saesneg the boys from brazil a ...`
+2. `daleithiau america saesneg cyfeiriadau gan gyfarwyddwyr ffilm gwrywaidd tsieceg o tsiecoslofacia gyd...`
+3. `gan y cyfarwyddwr kevin billington yw the rise of the nazis stalingrad fernsehepisode y deyrnas uned...`
 
 **Context Size 4:**
 
-1. `- 01 - 01 la folie du doute ffrainc no / unknown value 1914 - 01 - 01 en`
-2. `unol daleithiau america saesneg 1977 - 01 - 01 who will marry me ? unol daleithiau america 1964 -`
-3. `gan gynnwys y canlynol : ffilm delwedd gwlad iaith wreiddiol dyddiad behind the scenes center | 100p...`
+1. `unol daleithiau america saesneg o unol daleithiau america arswyd o unol daleithiau america comedi gy...`
+2. `gan gynnwys y canlynol cyfeiriadau lliw lliw o sbaen rhamantaidd o sbaen sbaeneg o sbaen comedi gyda...`
+3. `o ffilmiau gan gynnwys y canlynol ffilm delwedd gwlad iaith wreiddiol dyddiad coyote summer unol dal...`
+
+
+### Generated Text Samples (Subword-based)
+
+Below are text samples generated from each subword-based Markov chain model:
+
+**Context Size 1:**
+
+1. `_o/uchomau_dcolm`
+2. `adankeegoeei'cho`
+3. `elm_seratir,_pae`
+
+**Context Size 2:**
+
+1. `n_gannwyddyd_gwed`
+2. `d_rasalanc_wr_pon`
+3. `_y_faraithia_cymg`
+
+**Context Size 3:**
+
+1. `yn_wreidd_gwyn_cyh`
+2. `dd_a_10,700_strwyd`
+3. `_yn_coln,_sy'n_alm`
+
+**Context Size 4:**
+
+1. `_yn_sydd_('cyfarwyd`
+2. `filmio_oeddwyd,_cyh`
+3. `ffilm_hon_walter,_j`
 
 
 ### Key Findings
 
-- **Best Predictability:** Context-4 with 90.7% predictability
+- **Best Predictability:** Context-4 (word) with 93.1% predictability
 - **Branching Factor:** Decreases with context size (more deterministic)
-- **Memory Trade-off:** Larger contexts require more storage (1,190,900 contexts)
+- **Memory Trade-off:** Larger contexts require more storage (850,309 contexts)
 - **Recommendation:** Context-3 or Context-4 for text generation
 
 ---
@@ -247,37 +346,37 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Vocabulary Size | 389,645 |
-| Total Tokens | 63,461,717 |
-| Mean Frequency | 162.87 |
+| Vocabulary Size | 360,120 |
+| Total Tokens | 54,213,529 |
+| Mean Frequency | 150.54 |
 | Median Frequency | 5 |
-| Frequency Std Dev | 8486.55 |
+| Frequency Std Dev | 7791.16 |
 
 ### Most Common Words
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | y | 2,296,979 |
-| 2 | yn | 2,179,510 |
-| 3 | o | 1,595,326 |
-| 4 | categori | 1,496,135 |
-| 5 | a | 1,393,254 |
-| 6 | 01 | 1,383,521 |
-| 7 | ffilmiau | 1,310,809 |
-| 8 | ffilm | 1,218,970 |
-| 9 | gan | 925,807 |
-| 10 | r | 732,675 |
+| 1 | y | 2,261,654 |
+| 2 | yn | 2,177,991 |
+| 3 | o | 1,594,538 |
+| 4 | a | 1,391,156 |
+| 5 | ffilm | 1,218,819 |
+| 6 | gan | 925,486 |
+| 7 | r | 723,127 |
+| 8 | i | 650,709 |
+| 9 | yr | 521,021 |
+| 10 | daleithiau | 501,348 |
 
 ### Least Common Words (from vocabulary)
 
 | Rank | Word | Frequency |
 |------|------|-----------|
-| 1 | bulac | 2 |
-| 2 | geirfaoedd | 2 |
-| 3 | volcabulaire | 2 |
-| 4 | ethnolog | 2 |
-| 5 | siculu | 2 |
-| 6 | metafonetig | 2 |
+| 1 | geirfaoedd | 2 |
+| 2 | volcabulaire | 2 |
+| 3 | ethnolog | 2 |
+| 4 | siculu | 2 |
+| 5 | metafonetig | 2 |
+| 6 | prano | 2 |
 | 7 | defynydd | 2 |
 | 8 | clwsterau | 2 |
 | 9 | ŋm | 2 |
@@ -287,24 +386,24 @@ Below are text samples generated from each Markov chain model:
 
 | Metric | Value |
 |--------|-------|
-| Zipf Coefficient | 1.2070 |
-| R² (Goodness of Fit) | 0.997118 |
+| Zipf Coefficient | 1.1638 |
+| R² (Goodness of Fit) | 0.998189 |
 | Adherence Quality | **excellent** |
 
 ### Coverage Analysis
 
 | Top N Words | Coverage |
 |-------------|----------|
-| Top 100 | 48.9% |
-| Top 1,000 | 73.9% |
-| Top 5,000 | 85.7% |
-| Top 10,000 | 89.6% |
+| Top 100 | 49.6% |
+| Top 1,000 | 72.6% |
+| Top 5,000 | 84.6% |
+| Top 10,000 | 88.7% |
 
 ### Key Findings
 
-- **Zipf Compliance:** R²=0.9971 indicates excellent adherence to Zipf's law
-- **High Frequency Dominance:** Top 100 words cover 48.9% of corpus
-- **Long Tail:** 379,645 words needed for remaining 10.4% coverage
+- **Zipf Compliance:** R²=0.9982 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 49.6% of corpus
+- **Long Tail:** 350,120 words needed for remaining 11.3% coverage
 
 ---
 ## 5. Word Embeddings Evaluation
@@ -317,24 +416,116 @@ Below are text samples generated from each Markov chain model:
 
 ![t-SNE Sentences](visualizations/tsne_sentences.png)
 
-### Model Comparison
 
-| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
-|-------|------------|-----------|----------|----------|----------|
-| **mono_32d** | 302,236 | 32 | 4.313 | 1.212 | 0.8061 🏆 |
-| **mono_64d** | 302,236 | 64 | 4.810 | 1.191 | 0.7892 |
-| **mono_128d** | 302,236 | 128 | 5.402 | 1.237 | 0.7570 |
-| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+### 5.1 Cross-Lingual Alignment
+
+![Alignment Quality](visualizations/embedding_alignment_quality.png)
+
+![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)
+
+
+### 5.2 Model Comparison
+
+| Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
+|-------|-----------|----------|------------------|---------------|----------------|
+| **mono_32d** | 32 | 0.8420 🏆 | 0.3264 | N/A | N/A |
+| **mono_64d** | 64 | 0.8198 | 0.2681 | N/A | N/A |
+| **mono_128d** | 128 | 0.7807 | 0.2230 | N/A | N/A |
+| **aligned_32d** | 32 | 0.8420 | 0.3314 | 0.2180 | 0.6520 |
+| **aligned_64d** | 64 | 0.8198 | 0.2651 | 0.3480 | 0.7540 |
+| **aligned_128d** | 128 | 0.7807 | 0.2238 | 0.5000 | 0.8640 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.8061 (more uniform distribution)
-- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
-- **Vocabulary Coverage:** All models cover 302,236 words
-- **Recommendation:** 100d for balanced semantic capture and efficiency
+- **Best Isotropy:** mono_32d with 0.8420 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2730. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 50.0% R@1 in cross-lingual retrieval.
+- **Recommendation:** 128d aligned for best cross-lingual performance
+
+---
+## 6.  Morphological Analysis (Experimental)
+
+This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.
+
+### 6.1 Productivity & Complexity
+
+| Metric | Value | Interpretation | Recommendation |
+|--------|-------|----------------|----------------|
+| Productivity Index | **5.000** | High morphological productivity | Reliable analysis |
+| Idiomaticity Gap | **-0.043** | Low formulaic content | - |
+
+### 6.2 Affix Inventory (Productive Units)
+
+These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.
+
+#### Productive Prefixes
+| Prefix | Examples |
+|--------|----------|
+
+#### Productive Suffixes
+| Suffix | Examples |
+|--------|----------|
+| `-er` | menschenfresser, spengler, giessler |
+| `-dd` | cwmnioedd, ailysgrifennodd, maswedd |
+| `-on` | cenawon, pittston, dimson |
+| `-au` | llinachau, rygiau, halennau |
+| `-en` | vorsitzenden, misshandlingen, ddiacen |
+
+### 6.3 Bound Stems (Lexical Roots)
+
+Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.
+
+| Stem | Cohesion | Substitutability | Examples |
+|------|----------|------------------|----------|
+| `iada` | 2.24x | 67 contexts | riada, viada, diada |
+| `efyd` | 2.21x | 69 contexts | hefyd, lefyd, efydd |
+| `ddio` | 1.98x | 84 contexts | addio, ddiog, ddios |
+| `feir` | 2.03x | 69 contexts | feiro, feira, sfeir |
+| `nnwy` | 2.19x | 46 contexts | annwyl, annwyd, gynnwy |
+| `leit` | 2.36x | 32 contexts | leite, fleit, leith |
+| `yddi` | 1.71x | 121 contexts | fyddi, byddi, dyddio |
+| `dwyd` | 2.14x | 43 contexts | nodwyd, ildwyd, codwyd |
+| `ithi` | 1.55x | 152 contexts | deithi, teithi, rithio |
+| `alei` | 2.30x | 26 contexts | dalei, malei, maleia |
+| `adau` | 2.02x | 40 contexts | badau, gadau, fadau |
+| `eddw` | 1.67x | 49 contexts | feddw, weddw, meddw |
+
+### 6.4 Affix Compatibility (Co-occurrence)
+
+This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.
+
+*No significant affix co-occurrences detected.*
+
+
+### 6.5 Recursive Morpheme Segmentation
+
+Using **Recursive Hierarchical Substitutability**, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).
+
+| Word | Suggested Split | Confidence | Stem |
+|------|-----------------|------------|------|
+| deiamwntau | **`deiamwnt-au`** | 4.5 | `deiamwnt` |
+| croniclau | **`cronicl-au`** | 4.5 | `cronicl` |
+| komödianten | **`komödiant-en`** | 4.5 | `komödiant` |
+| recruiter | **`recruit-er`** | 4.5 | `recruit` |
+| diffiniodd | **`diffinio-dd`** | 4.5 | `diffinio` |
+| catholicon | **`catholic-on`** | 4.5 | `catholic` |
+| telesgopau | **`telesgop-au`** | 4.5 | `telesgop` |
+| canlyniadau | **`canlyniad-au`** | 4.5 | `canlyniad` |
+| lluswydden | **`lluswy-dd-en`** | 3.0 | `lluswy` |
+| organeddau | **`organe-dd-au`** | 3.0 | `organe` |
+| chynffonau | **`chynff-on-au`** | 3.0 | `chynff` |
+| wastadeddau | **`wastade-dd-au`** | 3.0 | `wastade` |
+| ffilmymgyrchydd | **`ffilmymgyrchy-dd`** | 1.5 | `ffilmymgyrchy` |
+| stabilizer | **`stabiliz-er`** | 1.5 | `stabiliz` |
+| effeithiolrwydd | **`effeithiolrwy-dd`** | 1.5 | `effeithiolrwy` |
+
+### 6.6 Linguistic Interpretation
+
+> **Automated Insight:**
+The language Welsh shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.
 
 ---
-## 6. Summary & Recommendations
+## 7. Summary & Recommendations
 
 ![Performance Dashboard](visualizations/performance_dashboard.png)
 
@@ -342,11 +533,12 @@ Below are text samples generated from each Markov chain model:
 
 | Component | Recommended | Rationale |
 |-----------|-------------|-----------|
-| Tokenizer | **32k BPE** | Best compression (3.38x) with low UNK rate |
-| N-gram | **5-gram** | Lowest perplexity (329) |
-| Markov | **Context-4** | Highest predictability (90.7%) |
+| Tokenizer | **64k BPE** | Best compression (4.11x) |
+| N-gram | **2-gram** | Lowest perplexity (266) |
+| Markov | **Context-4** | Highest predictability (93.1%) |
 | Embeddings | **100d** | Balanced semantic capture and isotropy |
 
+
 ---
 ## Appendix: Metrics Glossary & Interpretation Guide
 
@@ -536,7 +728,8 @@ If you use these models in your research, please cite:
   author = {Kamali, Omar},
   title = {Wikilangs: Open NLP Models for Wikipedia Languages},
   year = {2025},
-  publisher = {HuggingFace},
+  doi = {10.5281/zenodo.18073153},
+  publisher = {Zenodo},
   url = {https://huggingface.co/wikilangs}
   institution = {Omneity Labs}
 }
@@ -552,7 +745,8 @@ MIT License - Free for academic and commercial use.
 - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
 - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
 - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
+- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2025-12-29 06:44:48*
+*Report Date: 2026-01-04 02:01:49*

models/embeddings/aligned/cy_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:71a9cc470933ca021e965ef59d5d35ae42cbf5d712bd69da16d2172f82b6e899
+size 1311006409

models/embeddings/aligned/cy_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "cy", "dim": 128, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/cy_128d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:253945786c2db421271a179761c1f204cf870db589c34c30603f0b648808fa14
+size 65664

models/embeddings/aligned/cy_128d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "cy",
+  "dimension": 128,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 112312,
+  "vocab_size": 275534
+}

models/embeddings/aligned/cy_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4b757fa77344292a77e59e092a2b18c17302b08200f8e2d1d392b9ed729eac94
+size 331396297

models/embeddings/aligned/cy_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "cy", "dim": 32, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/cy_32d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7cb68ea51508c5c1e3515f4866ba7792c4ee05d53dd13bf3760d3dfffdc67c94
+size 4224

models/embeddings/aligned/cy_32d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "cy",
+  "dimension": 32,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 112312,
+  "vocab_size": 275534
+}

models/embeddings/aligned/cy_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e60dc915534a1b716efbe68b36298dbef28c1788a964bd2fd7cfa92b82bd2316
+size 657933001

models/embeddings/aligned/cy_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "cy", "dim": 64, "max_seq_len": 512, "is_aligned": true}

models/embeddings/aligned/cy_64d.projection.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0415c21350d5c84213bf49b3054ba4ea03b8d3d19b37b76a0d288396784fc11f
+size 16512

models/embeddings/aligned/cy_64d_metadata.json

@@ -0,0 +1,8 @@
+{
+  "language": "cy",
+  "dimension": 64,
+  "version": "aligned",
+  "hub_language": "en",
+  "seed_vocab_size": 112312,
+  "vocab_size": 275534
+}

models/embeddings/monolingual/cy_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8032ea070b9a556d69743a6ba1afe7ceb90f929ed90f38a19a929db0cc5fbf9c
-size 1339038716
+oid sha256:71a9cc470933ca021e965ef59d5d35ae42cbf5d712bd69da16d2172f82b6e899
+size 1311006409

models/embeddings/monolingual/cy_128d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 128,
   "version": "monolingual",
   "training_params": {
-    "dim": 128,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 128
   },
-  "vocab_size": 302236
+  "vocab_size": 275534
 }

models/embeddings/monolingual/cy_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0dbe9dbf660202119679aac32782c94ce45f3981db2e982117998f8735eb0b66
-size 338921468
+oid sha256:4b757fa77344292a77e59e092a2b18c17302b08200f8e2d1d392b9ed729eac94
+size 331396297

models/embeddings/monolingual/cy_32d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 32,
   "version": "monolingual",
   "training_params": {
-    "dim": 32,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 32
   },
-  "vocab_size": 302236
+  "vocab_size": 275534
 }

models/embeddings/monolingual/cy_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2c29cde8e0a588f9cf6c2b2790bb7a742001e3649cf9e7c1459ff7be61d341c6
-size 672293884
+oid sha256:e60dc915534a1b716efbe68b36298dbef28c1788a964bd2fd7cfa92b82bd2316
+size 657933001

models/embeddings/monolingual/cy_64d_metadata.json

@@ -3,11 +3,13 @@
   "dimension": 64,
   "version": "monolingual",
   "training_params": {
-    "dim": 64,
+    "algorithm": "skipgram",
     "min_count": 5,
     "window": 5,
     "negative": 5,
-    "epochs": 5
+    "epochs": 5,
+    "encoding_method": "rope",
+    "dim": 64
   },
-  "vocab_size": 302236
+  "vocab_size": 275534
 }

models/subword_markov/cy_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e7be1214f6fb63a919f01f8997bc9434554ba27fdf4308fec51ca834b5c60d83
-size 364951
+oid sha256:77bfb515ff96720dfb9ea7b567149c487d1e1c5abf52aab5d3e9f32257e27c8d
+size 433622

models/subword_markov/cy_markov_ctx1_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "subword",
   "language": "cy",
-  "unique_contexts": 5659,
-  "total_transitions": 395609240
+  "unique_contexts": 8673,
+  "total_transitions": 331031071
 }

models/subword_markov/cy_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5dc3880004368f38d82df372289042ec774f9021cdd7a537c7d1040e36c2dba7
-size 2215561
+oid sha256:80874947d2f81f135d491086e3debb63549c17dec7ac004f27d0651a5c60d5bd
+size 1980045

models/subword_markov/cy_markov_ctx2_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "subword",
   "language": "cy",
-  "unique_contexts": 52646,
-  "total_transitions": 395325415
+  "unique_contexts": 57679,
+  "total_transitions": 330747308
 }

models/subword_markov/cy_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7fe74491901bd644df2ecce9a56a7f7443c39c955a3d40a7050cd6821f2cdc17
-size 10077791
+oid sha256:534097083e70adb36fae774d7d647a80b25c491927432795d131c8a1860db353
+size 7942804

models/subword_markov/cy_markov_ctx3_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 3,
   "variant": "subword",
   "language": "cy",
-  "unique_contexts": 267808,
-  "total_transitions": 395041590
+  "unique_contexts": 225011,
+  "total_transitions": 330463545
 }

models/subword_markov/cy_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2bff5e3485ba67f8fb246702a093233ac90553839e243a24c8399bf5c13888d1
-size 33314253
+oid sha256:89e7898de9bb4c0bc01b6fb2c90d51ae670425a19ae7cda1857524212cbbe9e5
+size 24550193

models/subword_markov/cy_markov_ctx4_subword_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 4,
   "variant": "subword",
   "language": "cy",
-  "unique_contexts": 1190900,
-  "total_transitions": 394757765
+  "unique_contexts": 850309,
+  "total_transitions": 330179782
 }

models/subword_ngram/cy_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f0fef0a9228eba40b4017257ab0c383dd88a8a64e506ed4585323df321064c59
-size 234591
+oid sha256:0794cdccf210ecd2c67a97dfecc22867aacb11b80bee254a78bf063eae65f55e
+size 208837

models/subword_ngram/cy_2gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 2,
   "variant": "subword",
   "language": "cy",
-  "unique_ngrams": 17053,
-  "total_ngrams": 395609240
+  "unique_ngrams": 14977,
+  "total_ngrams": 331031071
 }

models/subword_ngram/cy_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dca13224e4c4a821574aeb63066a78b395af4320995e437d830a9f29a02ec455
-size 1606115
+oid sha256:d846dcbd8d0c05875b602277cbe1d1e86250892e478be8b608df266cf2af545f
+size 1269256

models/subword_ngram/cy_3gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 3,
   "variant": "subword",
   "language": "cy",
-  "unique_ngrams": 127945,
-  "total_ngrams": 395325415
+  "unique_ngrams": 96402,
+  "total_ngrams": 330747308
 }

models/subword_ngram/cy_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c30610d2431ebfd59727ef6174314d6ed4de69340eabb205c85b4149c435ea29
-size 8286493
+oid sha256:ec9f3eced4fd4cf6f7f5df5fc2748b38d1a56ee9e5941496343545ebe192fb14
+size 6072142

models/subword_ngram/cy_4gram_subword_metadata.json

@@ -2,6 +2,6 @@
   "n": 4,
   "variant": "subword",
   "language": "cy",
-  "unique_ngrams": 705121,
-  "total_ngrams": 395041590
+  "unique_ngrams": 505960,
+  "total_ngrams": 330463545
 }

models/subword_ngram/cy_5gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:238eb12d1d1836fc4053d4fff63010bd520f61e606f9f9e8e0857673bbf3ec73
+size 20429696

models/subword_ngram/cy_5gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 5,
+  "variant": "subword",
+  "language": "cy",
+  "unique_ngrams": 1685188,
+  "total_ngrams": 330179782
+}

models/tokenizer/cy_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:54f1e149a19959f257fa6d275287ecee0003f2b90cac7ab3ac44c70ddd67bec3
-size 504780
+oid sha256:53fac0099672dfd14c84298634363bafca172cef8cf2886fcd05d536c3633680
+size 505264

models/tokenizer/cy_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e29cd5ea4c2b78e8e7182e943efa0282f70d1c58cc36390447563f0e71fe4388
-size 779387
+oid sha256:bf54a3516d63ce6d78f285ab430915d146623aec35336552551dff7df762780c
+size 779588

models/tokenizer/cy_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a53705a33984a063d3ea0ad404e8784046151432af2368b824d34685610e90c9
-size 1341924
+oid sha256:a3e47fd9fa704ae838dc06a7f6e1e241ef1b154e39c7031289aa1dfa10ac2ff7
+size 1344109

models/tokenizer/cy_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f133e678d0bcd9cee54f10e779a871317fabfe9f02dab45ef63562e0768302ab
-size 371729
+oid sha256:c57f61d9bc3e44c9b08254a66bd0b61d18d27253365c0e6a34709a0671ab5196
+size 372334

models/vocabulary/cy_vocabulary.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c77a77f2457dbcc78ea0ac032b77197d09e5fc107e8ff1a37fcc6a47715503f1
-size 5893708
+oid sha256:0c9f27db05545e54198e12ec55f76159742ebfd6709d06f6f0bd37c1e57d5e60
+size 5482460

models/vocabulary/cy_vocabulary_metadata.json

@@ -1,16 +1,17 @@
 {
   "language": "cy",
-  "vocabulary_size": 389645,
+  "vocabulary_size": 360120,
+  "variant": "full",
   "statistics": {
-    "type_token_ratio": 0.0131546528620712,
+    "type_token_ratio": 0.012317229272748727,
     "coverage": {
-      "top_100": 0.48536421243668115,
-      "top_1000": 0.7336080742357445,
-      "top_5000": 0.8512773563452204,
-      "top_10000": 0.889406496721323
+      "top_100": 0.4934650351070113,
+      "top_1000": 0.7220562249915204,
+      "top_5000": 0.8413792196556568,
+      "top_10000": 0.8820137131209119
     },
-    "hapax_count": 451106,
-    "hapax_ratio": 0.5365512500133809,
-    "total_documents": 283825
+    "hapax_count": 311477,
+    "hapax_ratio": 0.4637855737890431,
+    "total_documents": 283763
   }
 }

models/word_markov/cy_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:42d7b87be999fc860a768fb63310dce637913b0cdd04e1a5762efc6ffd96e12c
-size 55261051
+oid sha256:eae5083c326a617bc8f5f837e5019f5aed6318c160e43f0e2e84ebd1d60d928d
+size 55601762

models/word_markov/cy_markov_ctx1_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 1,
   "variant": "word",
   "language": "cy",
-  "unique_contexts": 841116,
-  "total_transitions": 77665759
+  "unique_contexts": 671269,
+  "total_transitions": 54241243
 }

models/word_markov/cy_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:df2e9a656905791c64c690564408ef31601ec967e17bec7c732dfa37774ffa8b
-size 164804333
+oid sha256:30817c15c44e32d480990853b784bc0cbeb3a42c9fb8612d7988d299a9335db9
+size 165351799

models/word_markov/cy_markov_ctx2_word_metadata.json

@@ -2,6 +2,6 @@
   "context_size": 2,
   "variant": "word",
   "language": "cy",
-  "unique_contexts": 6216866,
-  "total_transitions": 77381936
+  "unique_contexts": 6591949,
+  "total_transitions": 53957480
 }