tk / README.md

Upload all models and assets for tk (latest)

28ab55e verified 3 days ago

30.5 kB

	---
	language: tk
	language_name: Turkmen
	language_family: turkic_oghuz
	tags:
	- wikilangs
	- nlp
	- tokenizer
	- embeddings
	- n-gram
	- markov
	- wikipedia
	- feature-extraction
	- sentence-similarity
	- tokenization
	- n-grams
	- markov-chain
	- text-mining
	- fasttext
	- babelvec
	- vocabulous
	- vocabulary
	- monolingual
	- family-turkic_oghuz
	license: mit
	library_name: wikilangs
	pipeline_tag: text-generation
	datasets:
	- omarkamali/wikipedia-monthly
	dataset_info:
	name: wikipedia-monthly
	description: Monthly snapshots of Wikipedia articles across 300+ languages
	metrics:
	- name: best_compression_ratio
	type: compression
	value: 4.949
	- name: best_isotropy
	type: isotropy
	value: 0.8902
	- name: vocabulary_size
	type: vocab
	value: 0
	generated: 2026-01-11
	---

	# Turkmen - Wikilangs Models
	## Comprehensive Research Report & Full Ablation Study

	This repository contains NLP models trained and evaluated by Wikilangs, specifically on Turkmen Wikipedia data.
	We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.

	## 📋 Repository Contents

	### Models & Assets

	- Tokenizers (8k, 16k, 32k, 64k)
	- N-gram models (2, 3, 4, 5-gram)
	- Markov chains (context of 1, 2, 3, 4 and 5)
	- Subword N-gram and Markov chains
	- Embeddings in various sizes and dimensions (aligned and unaligned)
	- Language Vocabulary
	- Language Statistics

	![Performance Dashboard](visualizations/performance_dashboard.png)

	### Analysis and Evaluation

	- [1. Tokenizer Evaluation](#1-tokenizer-evaluation)
	- [2. N-gram Model Evaluation](#2-n-gram-model-evaluation)
	- [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
	- [4. Vocabulary Analysis](#4-vocabulary-analysis)
	- [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
	- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
	- [7. Summary & Recommendations](#7-summary--recommendations)
	- [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
	- [Visualizations Index](#visualizations-index)

	---
	## 1. Tokenizer Evaluation

	![Tokenizer Compression](visualizations/tokenizer_compression.png)

	![Tokenizer Fertility](visualizations/tokenizer_fertility.png)

	![Tokenizer OOV](visualizations/tokenizer_oov.png)

	![Total Tokens](visualizations/tokenizer_total_tokens.png)

	### Results

	\| Vocab Size \| Compression \| Avg Token Len \| UNK Rate \| Total Tokens \|
	\|------------\|-------------\|---------------\|----------\|--------------\|
	\| 8k \| 3.867x \| 3.87 \| 0.1563% \| 394,866 \|
	\| 16k \| 4.295x \| 4.30 \| 0.1736% \| 355,501 \|
	\| 32k \| 4.665x \| 4.67 \| 0.1885% \| 327,292 \|
	\| 64k \| 4.949x 🏆 \| 4.95 \| 0.2000% \| 308,505 \|

	### Tokenization Examples

	Below are sample sentences tokenized with each vocabulary size:

	Sample 1: `Wakalar Sebitler boýunça Tema boýunça <noinclude> Dünýä inenler Aradan çykanlar`

	\| Vocab \| Tokens \| Count \|
	\|-------\|--------\|-------\|
	\| 8k \| `▁wakalar ▁sebitler ▁boýunça ▁tema ▁boýunça ▁< noinclude > ▁dünýä ▁inenler ... (+2 more)` \| 12 \|
	\| 16k \| `▁wakalar ▁sebitler ▁boýunça ▁tema ▁boýunça ▁< noinclude > ▁dünýä ▁inenler ... (+2 more)` \| 12 \|
	\| 32k \| `▁wakalar ▁sebitler ▁boýunça ▁tema ▁boýunça ▁< noinclude > ▁dünýä ▁inenler ... (+2 more)` \| 12 \|
	\| 64k \| `▁wakalar ▁sebitler ▁boýunça ▁tema ▁boýunça ▁< noinclude > ▁dünýä ▁inenler ... (+2 more)` \| 12 \|

	Sample 2: `Wakalar Sebitler boýunça Tema boýunça <noinclude> Dünýä inenler Aradan çykanlar`

	\| Vocab \| Tokens \| Count \|
	\|-------\|--------\|-------\|
	\| 8k \| `▁wakalar ▁sebitler ▁boýunça ▁tema ▁boýunça ▁< noinclude > ▁dünýä ▁inenler ... (+2 more)` \| 12 \|
	\| 16k \| `▁wakalar ▁sebitler ▁boýunça ▁tema ▁boýunça ▁< noinclude > ▁dünýä ▁inenler ... (+2 more)` \| 12 \|
	\| 32k \| `▁wakalar ▁sebitler ▁boýunça ▁tema ▁boýunça ▁< noinclude > ▁dünýä ▁inenler ... (+2 more)` \| 12 \|
	\| 64k \| `▁wakalar ▁sebitler ▁boýunça ▁tema ▁boýunça ▁< noinclude > ▁dünýä ▁inenler ... (+2 more)` \| 12 \|

	Sample 3: `Seýdi etraby — Lebap welayatynyň bir etrabydyr. etraplary welaýaty welaýatyndaky...`

	\| Vocab \| Tokens \| Count \|
	\|-------\|--------\|-------\|
	\| 8k \| `▁seý di ▁etraby ▁— ▁lebap ▁welayat ynyň ▁bir ▁etraby dyr ... (+5 more)` \| 15 \|
	\| 16k \| `▁seýdi ▁etraby ▁— ▁lebap ▁welayat ynyň ▁bir ▁etraby dyr . ... (+4 more)` \| 14 \|
	\| 32k \| `▁seýdi ▁etraby ▁— ▁lebap ▁welayatynyň ▁bir ▁etrabydyr . ▁etraplary ▁welaýaty ... (+2 more)` \| 12 \|
	\| 64k \| `▁seýdi ▁etraby ▁— ▁lebap ▁welayatynyň ▁bir ▁etrabydyr . ▁etraplary ▁welaýaty ... (+2 more)` \| 12 \|


	### Key Findings

	- Best Compression: 64k achieves 4.949x compression
	- Lowest UNK Rate: 8k with 0.1563% unknown tokens
	- Trade-off: Larger vocabularies improve compression but increase model size
	- Recommendation: 32k vocabulary provides optimal balance for production use

	---
	## 2. N-gram Model Evaluation

	![N-gram Perplexity](visualizations/ngram_perplexity.png)

	![N-gram Unique](visualizations/ngram_unique.png)

	![N-gram Coverage](visualizations/ngram_coverage.png)

	### Results

	\| N-gram \| Variant \| Perplexity \| Entropy \| Unique N-grams \| Top-100 Coverage \| Top-1000 Coverage \|
	\|--------\|---------\|------------\|---------\|----------------\|------------------\|-------------------\|
	\| 2-gram \| Word \| 11,088 \| 13.44 \| 23,947 \| 14.6% \| 32.8% \|
	\| 2-gram \| Subword \| 355 🏆 \| 8.47 \| 4,493 \| 61.5% \| 98.3% \|
	\| 3-gram \| Word \| 7,047 \| 12.78 \| 19,707 \| 21.5% \| 35.2% \|
	\| 3-gram \| Subword \| 2,934 \| 11.52 \| 34,530 \| 22.8% \| 66.5% \|
	\| 4-gram \| Word \| 20,732 \| 14.34 \| 46,279 \| 14.6% \| 21.3% \|
	\| 4-gram \| Subword \| 14,717 \| 13.85 \| 159,071 \| 11.4% \| 36.9% \|
	\| 5-gram \| Word \| 15,656 \| 13.93 \| 36,681 \| 16.0% \| 22.7% \|
	\| 5-gram \| Subword \| 46,546 \| 15.51 \| 363,230 \| 6.8% \| 23.5% \|

	### Top 5 N-grams by Size

	2-grams (Word):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `ýa da` \| 2,786 \|
	\| 2 \| `aradan çykanlar` \| 2,220 \|
	\| 3 \| `tema boýunça` \| 2,220 \|
	\| 4 \| `dünýä inenler` \| 2,217 \|
	\| 5 \| `sebitler boýunça` \| 2,216 \|

	3-grams (Word):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `wakalar sebitler boýunça` \| 2,208 \|
	\| 2 \| `boýunça tema boýunça` \| 2,201 \|
	\| 3 \| `sebitler boýunça tema` \| 2,201 \|
	\| 4 \| `dünýä inenler aradan` \| 2,174 \|
	\| 5 \| `inenler aradan çykanlar` \| 2,174 \|

	4-grams (Word):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `sebitler boýunça tema boýunça` \| 2,201 \|
	\| 2 \| `wakalar sebitler boýunça tema` \| 2,196 \|
	\| 3 \| `dünýä inenler aradan çykanlar` \| 2,174 \|
	\| 4 \| `tema boýunça noinclude dünýä` \| 2,119 \|
	\| 5 \| `boýunça noinclude dünýä inenler` \| 2,119 \|

	5-grams (Word):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `wakalar sebitler boýunça tema boýunça` \| 2,196 \|
	\| 2 \| `tema boýunça noinclude dünýä inenler` \| 2,119 \|
	\| 3 \| `sebitler boýunça tema boýunça noinclude` \| 2,112 \|
	\| 4 \| `boýunça tema boýunça noinclude dünýä` \| 2,112 \|
	\| 5 \| `noinclude dünýä inenler aradan çykanlar` \| 2,085 \|

	2-grams (Subword):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `a r` \| 188,493 \|
	\| 2 \| `l a` \| 152,165 \|
	\| 3 \| `a n` \| 151,310 \|
	\| 4 \| `_ b` \| 146,537 \|
	\| 5 \| `a _` \| 138,776 \|

	3-grams (Subword):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `l a r` \| 82,667 \|
	\| 2 \| `a r y` \| 58,594 \|
	\| 3 \| `y ň _` \| 57,971 \|
	\| 4 \| `a n _` \| 55,883 \|
	\| 5 \| `r . _` \| 53,874 \|

	4-grams (Subword):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `l a r y` \| 41,638 \|
	\| 2 \| `n y ň _` \| 30,386 \|
	\| 3 \| `_ w e _` \| 29,297 \|
	\| 4 \| `y n d a` \| 26,755 \|
	\| 5 \| `l e r i` \| 26,718 \|

	5-grams (Subword):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `_ b i l e` \| 16,563 \|
	\| 2 \| `i l e n _` \| 16,493 \|
	\| 3 \| `y n d a _` \| 16,259 \|
	\| 4 \| `y n y ň _` \| 15,844 \|
	\| 5 \| `b i l e n` \| 14,698 \|


	### Key Findings

	- Best Perplexity: 2-gram (subword) with 355
	- Entropy Trend: Decreases with larger n-grams (more predictable)
	- Coverage: Top-1000 patterns cover ~23% of corpus
	- Recommendation: 4-gram or 5-gram for best predictive performance

	---
	## 3. Markov Chain Evaluation

	![Markov Entropy](visualizations/markov_entropy.png)

	![Markov Contexts](visualizations/markov_contexts.png)

	![Markov Branching](visualizations/markov_branching.png)

	### Results

	\| Context \| Variant \| Avg Entropy \| Perplexity \| Branching Factor \| Unique Contexts \| Predictability \|
	\|---------\|---------\|-------------\|------------\|------------------\|-----------------\|----------------\|
	\| 1 \| Word \| 0.8425 \| 1.793 \| 5.31 \| 167,857 \| 15.8% \|
	\| 1 \| Subword \| 1.0332 \| 2.047 \| 8.72 \| 1,227 \| 0.0% \|
	\| 2 \| Word \| 0.1779 \| 1.131 \| 1.35 \| 888,328 \| 82.2% \|
	\| 2 \| Subword \| 1.0291 \| 2.041 \| 6.32 \| 10,675 \| 0.0% \|
	\| 3 \| Word \| 0.0393 \| 1.028 \| 1.06 \| 1,193,586 \| 96.1% \|
	\| 3 \| Subword \| 0.8531 \| 1.806 \| 4.15 \| 67,431 \| 14.7% \|
	\| 4 \| Word \| 0.0110 🏆 \| 1.008 \| 1.01 \| 1,255,469 \| 98.9% \|
	\| 4 \| Subword \| 0.6220 \| 1.539 \| 2.69 \| 279,783 \| 37.8% \|

	### Generated Text Samples (Word-based)

	Below are text samples generated from each word-based Markov chain model:

	Context Size 1:

	1. `we hemişe eline düşüpdir aýallaryñ häkimlik edýär bangkokdaky ýurduň 12 15 eretriýadan hem de ýylyň ...`
	2. `bilen icc bütindünýä güni kyýamat gününi alada üns berilýär asteroidler ýaly düzüp ol birwagtlar zaý...`
	3. `hem satuwa çykaryldy awstro wengriýa bilen kagyz ýüzündeligine galdy ž gulart käbir bölekleriniň geç...`

	Context Size 2:

	1. `ýa da mikaýyl bin seljuk bin dükak ýylda mälik şa üçin jelaly kalendaryny hijri kalendaryny mysal hö...`
	2. `tema boýunça noinclude dünýä inenler aradan çykanlar kategoriýa`
	3. `dünýä inenler aradan çykanlar salgylanmalar`

	Context Size 3:

	1. `wakalar sebitler boýunça tema boýunça noinclude dünýä inenler aradan çykanlar 31`
	2. `boýunça tema boýunça noinclude dünýä inenler aradan çykanlar 104`
	3. `sebitler boýunça tema boýunça noinclude dünýä inenler aradan çykanlar 29`

	Context Size 4:

	1. `sebitler boýunça tema boýunça noinclude dünýä inenler aradan çykanlar 26`
	2. `wakalar sebitler boýunça tema boýunça noinclude dünýä inenler aradan çykanlar baýramçylyklar`
	3. `boýunça noinclude dünýä inenler aradan çykanlar towşan esenowa hydyr derýaýew kerim gurbannepesow`


	### Generated Text Samples (Subword-based)

	Below are text samples generated from each subword-based Markov chain model:

	Context Size 1:

	1. `_botaýärkmp),_öz`
	2. `a_gumgitdar_nyle`
	3. `ebury_der._ýasah`

	Context Size 2:

	1. `ar._oduşli_düşdir`
	2. `lar.ilbaşdyry,_ob`
	3. `an_emlündama_(ýar`

	Context Size 3:

	1. `laryň_daşly_şübhes`
	2. `ary_12-150-nji_mil`
	3. `yň_aýatynyň_keşler`

	Context Size 4:

	1. `lary_deňde_gölli,_o`
	2. `nyň_bolandygynda_ru`
	3. `_we_goşuny,_hassa_t`


	### Key Findings

	- Best Predictability: Context-4 (word) with 98.9% predictability
	- Branching Factor: Decreases with context size (more deterministic)
	- Memory Trade-off: Larger contexts require more storage (279,783 contexts)
	- Recommendation: Context-3 or Context-4 for text generation

	---
	## 4. Vocabulary Analysis

	![Zipf's Law](visualizations/zipf_law.png)

	![Top Words](visualizations/top20_words.png)

	![Coverage Curve](visualizations/vocab_coverage.png)

	### Statistics

	\| Metric \| Value \|
	\|--------\|-------\|
	\| Vocabulary Size \| 70,850 \|
	\| Total Tokens \| 1,266,247 \|
	\| Mean Frequency \| 17.87 \|
	\| Median Frequency \| 4 \|
	\| Frequency Std Dev \| 172.27 \|

	### Most Common Words

	\| Rank \| Word \| Frequency \|
	\|------\|------\|-----------\|
	\| 1 \| we \| 29,419 \|
	\| 2 \| bilen \| 14,593 \|
	\| 3 \| hem \| 9,723 \|
	\| 4 \| bu \| 9,296 \|
	\| 5 \| bir \| 7,148 \|
	\| 6 \| üçin \| 7,116 \|
	\| 7 \| da \| 6,676 \|
	\| 8 \| boýunça \| 6,346 \|
	\| 9 \| ol \| 6,099 \|
	\| 10 \| ýylda \| 5,569 \|

	### Least Common Words (from vocabulary)

	\| Rank \| Word \| Frequency \|
	\|------\|------\|-----------\|
	\| 1 \| halaçda \| 2 \|
	\| 2 \| byradarlygynyň \| 2 \|
	\| 3 \| halaja \| 2 \|
	\| 4 \| bakynyň \| 2 \|
	\| 5 \| esaslandyrylanlar \| 2 \|
	\| 6 \| ailəsi \| 2 \|
	\| 7 \| yörükler \| 2 \|
	\| 8 \| ýarymgoragçysy \| 2 \|
	\| 9 \| jizak \| 2 \|
	\| 10 \| kolhozçi \| 2 \|

	### Zipf's Law Analysis

	\| Metric \| Value \|
	\|--------\|-------\|
	\| Zipf Coefficient \| 0.9487 \|
	\| R² (Goodness of Fit) \| 0.992202 \|
	\| Adherence Quality \| excellent \|

	### Coverage Analysis

	\| Top N Words \| Coverage \|
	\|-------------\|----------\|
	\| Top 100 \| 22.4% \|
	\| Top 1,000 \| 47.7% \|
	\| Top 5,000 \| 70.0% \|
	\| Top 10,000 \| 79.1% \|

	### Key Findings

	- Zipf Compliance: R²=0.9922 indicates excellent adherence to Zipf's law
	- High Frequency Dominance: Top 100 words cover 22.4% of corpus
	- Long Tail: 60,850 words needed for remaining 20.9% coverage

	---
	## 5. Word Embeddings Evaluation

	![Embedding Isotropy](visualizations/embedding_isotropy.png)

	![Similarity Matrix](visualizations/embedding_similarity.png)

	![t-SNE Words](visualizations/tsne_words.png)

	![t-SNE Sentences](visualizations/tsne_sentences.png)


	### 5.1 Cross-Lingual Alignment

	![Alignment Quality](visualizations/embedding_alignment_quality.png)

	![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)


	### 5.2 Model Comparison

	\| Model \| Dimension \| Isotropy \| Semantic Density \| Alignment R@1 \| Alignment R@10 \|
	\|-------\|-----------\|----------\|------------------\|---------------\|----------------\|
	\| mono_32d \| 32 \| 0.8902 \| 0.2916 \| N/A \| N/A \|
	\| mono_64d \| 64 \| 0.8799 \| 0.2188 \| N/A \| N/A \|
	\| mono_128d \| 128 \| 0.6945 \| 0.1696 \| N/A \| N/A \|
	\| aligned_32d \| 32 \| 0.8902 🏆 \| 0.2952 \| 0.0120 \| 0.1680 \|
	\| aligned_64d \| 64 \| 0.8799 \| 0.2224 \| 0.0560 \| 0.2240 \|
	\| aligned_128d \| 128 \| 0.6945 \| 0.1700 \| 0.0840 \| 0.3140 \|

	### Key Findings

	- Best Isotropy: aligned_32d with 0.8902 (more uniform distribution)
	- Semantic Density: Average pairwise similarity of 0.2279. Lower values indicate better semantic separation.
	- Alignment Quality: Aligned models achieve up to 8.4% R@1 in cross-lingual retrieval.
	- Recommendation: 128d aligned for best cross-lingual performance

	---
	## 6. Morphological Analysis (Experimental)

	This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.

	### 6.1 Productivity & Complexity

	\| Metric \| Value \| Interpretation \| Recommendation \|
	\|--------\|-------\|----------------\|----------------\|
	\| Productivity Index \| 5.000 \| High morphological productivity \| Reliable analysis \|
	\| Idiomaticity Gap \| -0.035 \| Low formulaic content \| - \|

	### 6.2 Affix Inventory (Productive Units)

	These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.

	#### Productive Prefixes
	\| Prefix \| Examples \|
	\|--------\|----------\|
	\| `-a` \| awyny, andrýu, alta \|
	\| `-s` \| saklanýandyr, saýylmadyk, stories \|
	\| `-g` \| gyşy, guzlar, gallipoli \|
	\| `-b` \| beloklaryny, basílio, basylan \|
	\| `-m` \| meýi, maersk, mortier \|
	\| `-k` \| kekene, klisfeniň, kesil \|
	\| `-d` \| diskriminasiýa, deňlemek, dakylýar \|
	\| `-t` \| theodore, territoriýasyndaky, taýynlapdyr \|

	#### Productive Suffixes
	\| Suffix \| Examples \|
	\|--------\|----------\|
	\| `-ň` \| operasiýalaryň, klisfeniň, aşgabadyň \|
	\| `-r` \| saklanýandyr, guzlar, mortier \|
	\| `-y` \| beloklaryny, gyşy, awyny \|
	\| `-a` \| diskriminasiýa, alta, gatyşmagynda \|
	\| `-yň` \| operasiýalaryň, aşgabadyň, wahýyň \|
	\| `-n` \| humaýun, basylan, araçäkleşýän \|
	\| `-i` \| meýi, redmi, erişleri \|
	\| `-an` \| basylan, gan, barylýan \|

	### 6.3 Bound Stems (Lexical Roots)

	Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.

	\| Stem \| Cohesion \| Substitutability \| Examples \|
	\|------\|----------\|------------------\|----------\|
	\| `kmen` \| 3.11x \| 26 contexts \| rkmen, sökmen, çekmen \|
	\| `anla` \| 1.82x \| 155 contexts \| sanlar, panlar, hanlar \|
	\| `asyn` \| 1.76x \| 181 contexts \| ýasyn, masyn, gasyn \|
	\| `erin` \| 1.91x \| 103 contexts \| lerin, erine, yerin \|
	\| `rkme` \| 3.11x \| 14 contexts \| rkmen, türkmer, turkmen \|
	\| `tlar` \| 1.70x \| 133 contexts \| atlar, otlar, otlara \|
	\| `rler` \| 1.83x \| 86 contexts \| ärler, ÿrler, ýerler \|
	\| `nlar` \| 1.84x \| 79 contexts \| onlar, gunlar, hunlar \|
	\| `erle` \| 1.63x \| 96 contexts \| ýerler, ỳerler, gerlen \|
	\| `ylar` \| 1.63x \| 72 contexts \| lylar, kylar, sylar \|
	\| `rlar` \| 1.60x \| 76 contexts \| arlar, durlar, ýarlar \|
	\| `klar` \| 1.67x \| 63 contexts \| uklar, klark, oklar \|

	### 6.4 Affix Compatibility (Co-occurrence)

	This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.

	\| Prefix \| Suffix \| Frequency \| Examples \|
	\|--------\|--------\|-----------\|----------\|
	\| `-g` \| `-y` \| 123 words \| gaşy, gatnawy \|
	\| `-g` \| `-r` \| 121 words \| gaçypdyrlar, girilýär \|
	\| `-g` \| `-a` \| 96 words \| gidrogeologiýa, graflyklara \|
	\| `-b` \| `-r` \| 92 words \| bir, bazaar \|
	\| `-g` \| `-n` \| 89 words \| gaýtarylan, gelmeýän \|
	\| `-g` \| `-i` \| 88 words \| geçmegi, güýçli \|
	\| `-s` \| `-ň` \| 87 words \| sahypalaryň, süýümleriniň \|
	\| `-s` \| `-y` \| 80 words \| sostawyny, satmagy \|
	\| `-g` \| `-ň` \| 76 words \| goýumdarlarynyň, guramaklygyň \|
	\| `-b` \| `-y` \| 75 words \| bozulmagy, bidgatçy \|

	### 6.5 Recursive Morpheme Segmentation

	Using Recursive Hierarchical Substitutability, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).

	\| Word \| Suggested Split \| Confidence \| Stem \|
	\|------\|-----------------\|------------\|------\|
	\| slawýanlarda \| `slawýanl-ar-da` \| 7.5 \| `ar` \|
	\| görkezipdir \| `görkezip-di-r` \| 7.5 \| `di` \|
	\| oktýabrdan \| `oktýabr-da-n` \| 7.5 \| `da` \|
	\| balyklaryñ \| `balykl-ar-yñ` \| 7.5 \| `ar` \|
	\| sazandalary \| `sazandal-ar-y` \| 7.5 \| `ar` \|
	\| bolanlary \| `bolanl-ar-y` \| 7.5 \| `ar` \|
	\| garşydaşlary \| `garşydaşl-ar-y` \| 7.5 \| `ar` \|
	\| halykynyň \| `halyky-n-yň` \| 7.5 \| `n` \|
	\| manjurlaryň \| `manjurl-ar-yň` \| 7.5 \| `ar` \|
	\| mukdarlary \| `mukdarl-ar-y` \| 7.5 \| `ar` \|
	\| ybadatlarda \| `ybadatl-ar-da` \| 7.5 \| `ar` \|
	\| guşaklyklary \| `guşaklykl-ar-y` \| 7.5 \| `ar` \|
	\| amallaryň \| `amall-ar-yň` \| 7.5 \| `ar` \|
	\| ugurlarda \| `ugurl-ar-da` \| 7.5 \| `ar` \|
	\| ýakynlarda \| `ýakynl-ar-da` \| 7.5 \| `ar` \|

	### 6.6 Linguistic Interpretation

	> Automated Insight:
	The language Turkmen shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.

	---
	## 7. Summary & Recommendations

	![Performance Dashboard](visualizations/performance_dashboard.png)

	### Production Recommendations

	\| Component \| Recommended \| Rationale \|
	\|-----------\|-------------\|-----------\|
	\| Tokenizer \| 64k BPE \| Best compression (4.95x) \|
	\| N-gram \| 2-gram \| Lowest perplexity (355) \|
	\| Markov \| Context-4 \| Highest predictability (98.9%) \|
	\| Embeddings \| 100d \| Balanced semantic capture and isotropy \|


	---
	## Appendix: Metrics Glossary & Interpretation Guide

	This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report.

	### Tokenizer Metrics

	Compression Ratio
	> Definition: The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text.
	>
	> Intuition: Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average.
	>
	> What to seek: Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information.

	Average Token Length (Fertility)
	> Definition: Mean number of characters per token produced by the tokenizer.
	>
	> Intuition: Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length.
	>
	> What to seek: Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens.

	Unknown Token Rate (OOV Rate)
	> Definition: Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent.
	>
	> Intuition: Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences.
	>
	> What to seek: Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback.

	### N-gram Model Metrics

	Perplexity
	> Definition: Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction.
	>
	> Intuition: If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options.
	>
	> What to seek: Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size.

	Entropy
	> Definition: Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy.
	>
	> Intuition: High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character.
	>
	> What to seek: Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases.

	Coverage (Top-K)
	> Definition: Percentage of corpus occurrences explained by the top K most frequent n-grams.
	>
	> Intuition: High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage.
	>
	> What to seek: Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text.

	### Markov Chain Metrics

	Average Entropy
	> Definition: Mean entropy across all contexts, measuring average uncertainty in next-word prediction.
	>
	> Intuition: Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations).
	>
	> What to seek: Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions.

	Branching Factor
	> Definition: Average number of unique next tokens observed for each context.
	>
	> Intuition: High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive).
	>
	> What to seek: Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains.

	Predictability
	> Definition: Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are.
	>
	> Intuition: 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes.
	>
	> What to seek: Higher predictability for text generation quality, but too high (>98%) may produce repetitive output.

	### Vocabulary & Zipf's Law Metrics

	Zipf's Coefficient
	> Definition: The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1.
	>
	> Intuition: A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare.
	>
	> What to seek: Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text.

	R² (Coefficient of Determination)
	> Definition: Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1.
	>
	> Intuition: R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns.
	>
	> What to seek: R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora.

	Vocabulary Coverage
	> Definition: Cumulative percentage of corpus tokens accounted for by the top N words.
	>
	> Intuition: Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words.
	>
	> What to seek: Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary.

	### Word Embedding Metrics

	Isotropy
	> Definition: Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values.
	>
	> Intuition: High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness.
	>
	> What to seek: Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy.

	Average Norm
	> Definition: Mean magnitude (L2 norm) of word vectors in the embedding space.
	>
	> Intuition: Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained.
	>
	> What to seek: Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation).

	Cosine Similarity
	> Definition: Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction).
	>
	> Intuition: Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings.
	>
	> What to seek: Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7.

	t-SNE Visualization
	> Definition: t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization.
	>
	> Intuition: Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence.
	>
	> What to seek: Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure.

	### General Interpretation Guidelines

	1. Compare within model families: Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer).
	2. Consider trade-offs: Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate).
	3. Context matters: Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification.
	4. Corpus influence: All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature.
	5. Language-specific patterns: Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages.


	### Visualizations Index

	\| Visualization \| Description \|
	\|---------------\|-------------\|
	\| Tokenizer Compression \| Compression ratios by vocabulary size \|
	\| Tokenizer Fertility \| Average token length by vocabulary \|
	\| Tokenizer OOV \| Unknown token rates \|
	\| Tokenizer Total Tokens \| Total tokens by vocabulary \|
	\| N-gram Perplexity \| Perplexity by n-gram size \|
	\| N-gram Entropy \| Entropy by n-gram size \|
	\| N-gram Coverage \| Top pattern coverage \|
	\| N-gram Unique \| Unique n-gram counts \|
	\| Markov Entropy \| Entropy by context size \|
	\| Markov Branching \| Branching factor by context \|
	\| Markov Contexts \| Unique context counts \|
	\| Zipf's Law \| Frequency-rank distribution with fit \|
	\| Vocab Frequency \| Word frequency distribution \|
	\| Top 20 Words \| Most frequent words \|
	\| Vocab Coverage \| Cumulative coverage curve \|
	\| Embedding Isotropy \| Vector space uniformity \|
	\| Embedding Norms \| Vector magnitude distribution \|
	\| Embedding Similarity \| Word similarity heatmap \|
	\| Nearest Neighbors \| Similar words for key terms \|
	\| t-SNE Words \| 2D word embedding visualization \|
	\| t-SNE Sentences \| 2D sentence embedding visualization \|
	\| Position Encoding \| Encoding method comparison \|
	\| Model Sizes \| Storage requirements \|
	\| Performance Dashboard \| Comprehensive performance overview \|

	---
	## About This Project

	### Data Source

	Models trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) - a monthly snapshot of Wikipedia articles across 300+ languages.

	### Project

	A project by [Wikilangs](https://wikilangs.org) - Open-source NLP models for every Wikipedia language.

	### Maintainer

	[Omar Kamali](https://omarkamali.com) - [Omneity Labs](https://omneitylabs.com)

	### Citation

	If you use these models in your research, please cite:

	```bibtex
	@misc{wikilangs2025,
	author = {Kamali, Omar},
	title = {Wikilangs: Open NLP Models for Wikipedia Languages},
	year = {2025},
	doi = {10.5281/zenodo.18073153},
	publisher = {Zenodo},
	url = {https://huggingface.co/wikilangs}
	institution = {Omneity Labs}
	}
	```

	### License

	MIT License - Free for academic and commercial use.

	### Links

	- 🌐 Website: [wikilangs.org](https://wikilangs.org)
	- 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
	- 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
	- 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
	- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
	---
	Generated by Wikilangs Models Pipeline

	Report Date: 2026-01-11 01:05:04