--- language: azb language_name: AZB language_family: turkic_oghuz tags: - wikilangs - nlp - tokenizer - embeddings - n-gram - markov - wikipedia - monolingual - family-turkic_oghuz license: mit library_name: wikilangs pipeline_tag: feature-extraction datasets: - omarkamali/wikipedia-monthly dataset_info: name: wikipedia-monthly description: Monthly snapshots of Wikipedia articles across 300+ languages metrics: - name: best_compression_ratio type: compression value: 4.198 - name: best_isotropy type: isotropy value: 0.8242 - name: vocabulary_size type: vocab value: 317640 generated: 2025-12-27 --- # AZB - Wikilangs Models ## Comprehensive Research Report & Full Ablation Study This repository contains NLP models trained and evaluated by Wikilangs, specifically on **AZB** Wikipedia data. We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings. ## 📋 Repository Contents ### Models & Assets - Tokenizers (8k, 16k, 32k, 64k) - N-gram models (2, 3, 4-gram) - Markov chains (context of 1, 2, 3 and 4) - Subword N-gram and Markov chains - Embeddings in various sizes and dimensions - Language Vocabulary - Language Statistics ![Performance Dashboard](visualizations/performance_dashboard.png) ### Analysis and Evaluation - [1. Tokenizer Evaluation](#1-tokenizer-evaluation) - [2. N-gram Model Evaluation](#2-n-gram-model-evaluation) - [3. Markov Chain Evaluation](#3-markov-chain-evaluation) - [4. Vocabulary Analysis](#4-vocabulary-analysis) - [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation) - [6. Summary & Recommendations](#6-summary--recommendations) - [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide) - [Visualizations Index](#visualizations-index) --- ## 1. Tokenizer Evaluation ![Tokenizer Compression](visualizations/tokenizer_compression.png) ### Results | Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens | |------------|-------------|---------------|----------|--------------| | **8k** | 3.179x | 3.12 | 0.3165% | 385,493 | | **16k** | 3.558x | 3.49 | 0.3542% | 344,465 | | **32k** | 3.895x | 3.82 | 0.3877% | 314,660 | | **64k** | 4.198x 🏆 | 4.12 | 0.4179% | 291,946 | ### Tokenization Examples Below are sample sentences tokenized with each vocabulary size: **Sample 1:** `ماقنتیتی ( ) روسیه اؤلکه‌سینده یئر آلان بیر کند دیر و مورمانسک اوبلاستیندا یئرلش...` | Vocab | Tokens | Count | |-------|--------|-------| | 8k | `▁م اق ن تی تی ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ... (+19 more)` | 29 | | 16k | `▁ماق ن تی تی ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ▁یئر ... (+16 more)` | 26 | | 32k | `▁ماق نتی تی ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ▁یئر ▁آلان ... (+13 more)` | 23 | | 64k | `▁ماق نتی تی ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ▁یئر ▁آلان ... (+13 more)` | 23 | **Sample 2:** `سید ضیاء الدین طباطبائی یزدی (دوغوم:۱۲۸۶ شیراز -اؤلوم:۷ شهریور ۱۳۴۸ تهران) احمدش...` | Vocab | Tokens | Count | |-------|--------|-------| | 8k | `▁سید ▁ض یا ء ▁الدین ▁ط با ط با ئی ... (+34 more)` | 44 | | 16k | `▁سید ▁ضیا ء ▁الدین ▁ط با ط با ئی ▁ی ... (+29 more)` | 39 | | 32k | `▁سید ▁ضیا ء ▁الدین ▁ط باط با ئی ▁یز دی ... (+26 more)` | 36 | | 64k | `▁سید ▁ضیاء ▁الدین ▁طباطبا ئی ▁یزدی ▁( دوغوم : ۱۲ ... (+21 more)` | 31 | **Sample 3:** `ویکسن (اینگیلیسجه: Vixen, Louisiana) آمریکانین لوئیزیانا ایالتینده یئرلشن بیر یا...` | Vocab | Tokens | Count | |-------|--------|-------| | 8k | `▁ویک سن ▁( اینگیلیسجه : ▁v ix en , ▁louisiana ... (+15 more)` | 25 | | 16k | `▁ویک سن ▁( اینگیلیسجه : ▁v ix en , ▁louisiana ... (+15 more)` | 25 | | 32k | `▁ویک سن ▁( اینگیلیسجه : ▁v ix en , ▁louisiana ... (+15 more)` | 25 | | 64k | `▁ویک سن ▁( اینگیلیسجه : ▁vixen , ▁louisiana ) ▁آمریکانین ... (+13 more)` | 23 | ### Key Findings - **Best Compression:** 64k achieves 4.198x compression - **Lowest UNK Rate:** 8k with 0.3165% unknown tokens - **Trade-off:** Larger vocabularies improve compression but increase model size - **Recommendation:** 32k vocabulary provides optimal balance for production use --- ## 2. N-gram Model Evaluation ![N-gram Perplexity](visualizations/ngram_perplexity.png) ![N-gram Coverage](visualizations/ngram_coverage.png) ### Results | N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage | |--------|------------|---------|----------------|------------------|-------------------| | **2-gram** | 7,645 🏆 | 12.90 | 268,529 | 30.2% | 57.7% | | **2-gram** | 655 🏆 | 9.35 | 14,131 | 46.5% | 93.7% | | **3-gram** | 13,711 | 13.74 | 512,828 | 24.9% | 52.4% | | **3-gram** | 4,808 | 12.23 | 141,425 | 21.1% | 58.1% | | **4-gram** | 23,522 | 14.52 | 979,944 | 21.9% | 47.2% | | **4-gram** | 19,536 | 14.25 | 804,296 | 13.9% | 41.7% | ### Top 5 N-grams by Size **2-grams:** | Rank | N-gram | Count | |------|--------|-------| | 1 | `بؤلمه :` | 391,479 | | 2 | `او ْ` | 139,043 | | 3 | `‌ لار` | 137,076 | | 4 | `قایناقلار بؤلمه` | 125,059 | | 5 | `‌ نین` | 118,471 | **3-grams:** | Rank | N-gram | Count | |------|--------|-------| | 1 | `قایناقلار بؤلمه :` | 125,058 | | 2 | `( اینگیلیسجه :` | 112,489 | | 3 | `. قایناقلار بؤلمه` | 97,901 | | 4 | `قایناق ‌ لار` | 92,555 | | 5 | `ایشلدنلری طرفیندن یارانمیش` | 76,848 | **4-grams:** | Rank | N-gram | Count | |------|--------|-------| | 1 | `. قایناقلار بؤلمه :` | 97,901 | | 2 | `ایشلدنلری طرفیندن یارانمیش «` | 76,839 | | 3 | `ْ خلانیلیبدیر ) .` | 76,827 | | 4 | `» ، مقاله ‌` | 76,824 | | 5 | `، مقاله ‌ سیندن` | 76,824 | ### Key Findings - **Best Perplexity:** 2-gram with 655 - **Entropy Trend:** Decreases with larger n-grams (more predictable) - **Coverage:** Top-1000 patterns cover ~42% of corpus - **Recommendation:** 4-gram or 5-gram for best predictive performance --- ## 3. Markov Chain Evaluation ![Markov Entropy](visualizations/markov_entropy.png) ![Markov Branching](visualizations/markov_branching.png) ### Results | Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability | |---------|-------------|------------|------------------|-----------------|----------------| | **1** | 0.4591 | 1.375 | 4.28 | 1,005,885 | 54.1% | | **1** | 1.2059 | 2.307 | 10.74 | 2,934 | 0.0% | | **2** | 0.2308 | 1.173 | 1.71 | 4,302,287 | 76.9% | | **2** | 1.1061 | 2.153 | 8.32 | 31,498 | 0.0% | | **3** | 0.1101 | 1.079 | 1.28 | 7,369,809 | 89.0% | | **3** | 0.9393 | 1.918 | 5.09 | 262,151 | 6.1% | | **4** | 0.0637 🏆 | 1.045 | 1.14 | 9,387,722 | 93.6% | | **4** | 0.7222 🏆 | 1.650 | 3.21 | 1,333,406 | 27.8% | ### Generated Text Samples Below are text samples generated from each Markov chain model: **Context Size 1:** 1. `‌ ایللیین اووللرین ده یاشایان اینسانلار بؤلمه : bhumibol adulyadej crown , footballer , alchimie du` 2. `. قان معائناتی و زنگین ‌ نین ایشلدنلری طرفیندن یارانمیش « panama 2 episodes ایلآدی رو` 3. `: sergey ryzhikov ( ۱۹۴۸ ایلینده گونئی کارولینا ( تحصیلات ) رومنی ( لهیستانجا : 2` **Context Size 2:** 1. `بؤلمه : وورت بؤلگه ‌ سیاراک بؤلگه ‌ سی ‌ دیر گؤرونتولر قایناق ‌ لار اینگیلیسجه ویکی` 2. `او ْ یونچو . ۲۶ ژوئن ۱۹۵۷ میلادی تاریخیندا وفات ائدیب . قایناق ‌ لار اینگیلیسجه ویکی` 3. `‌ لار بؤلمه : آمریکا شهرلری بؤلمه : گرینزبورو دوغوملولار بؤلمه : ۲۰۱۹ - جو ایلینده افغانیستاندا` **Context Size 3:** 1. `قایناقلار بؤلمه : هیندوستان کندلری en : kahra` 2. `( اینگیلیسجه : louis prima ) آمریکالی موغنی ، یازیچی و او ْ ندان ایستیفاده ائتمیشدیر . روبنسین` 3. `. قایناقلار بؤلمه : هیندوستان کندلری en : kalawad` **Context Size 4:** 1. `. قایناقلار بؤلمه : هیندوستان کندلری en : balowal` 2. `ایشلدنلری طرفیندن یارانمیش « crabtree ' s catalyst » ، مقاله ‌ سیندن گؤتورولوبدور . ( ۷ سپتامبر ۲۰۱۷` 3. `، مقاله ‌ سیندن گؤتورولوبدور . ( ۱۲ ژوئن ۲۰۱۸ تاریخینده یو ْ خلانیلیبدیر ) . بؤلمه : ایستادیوملار` ### Key Findings - **Best Predictability:** Context-4 with 93.6% predictability - **Branching Factor:** Decreases with context size (more deterministic) - **Memory Trade-off:** Larger contexts require more storage (1,333,406 contexts) - **Recommendation:** Context-3 or Context-4 for text generation --- ## 4. Vocabulary Analysis ![Zipf's Law](visualizations/zipf_law.png) ![Top Words](visualizations/top20_words.png) ![Coverage Curve](visualizations/vocab_coverage.png) ### Statistics | Metric | Value | |--------|-------| | Vocabulary Size | 317,640 | | Total Tokens | 16,697,106 | | Mean Frequency | 52.57 | | Median Frequency | 3 | | Frequency Std Dev | 1499.80 | ### Most Common Words | Rank | Word | Frequency | |------|------|-----------| | 1 | بؤلمه | 391,840 | | 2 | و | 291,285 | | 3 | اینگیلیسجه | 188,551 | | 4 | بیر | 169,796 | | 5 | او | 145,694 | | 6 | قایناقلار | 143,368 | | 7 | سی | 139,966 | | 8 | لار | 138,388 | | 9 | دیر | 132,038 | | 10 | the | 130,277 | ### Least Common Words (from vocabulary) | Rank | Word | Frequency | |------|------|-----------| | 1 | سئندن | 2 | | 2 | ائششکین | 2 | | 3 | onager | 2 | | 4 | داشناکلارلا | 2 | | 5 | قۇلان | 2 | | 6 | آسینۇس | 2 | | 7 | هئمیو | 2 | | 8 | تاپؽلمیشدیر | 2 | | 9 | kulan | 2 | | 10 | کسا | 2 | ### Zipf's Law Analysis | Metric | Value | |--------|-------| | Zipf Coefficient | 1.1842 | | R² (Goodness of Fit) | 0.995333 | | Adherence Quality | **excellent** | ### Coverage Analysis | Top N Words | Coverage | |-------------|----------| | Top 100 | 36.8% | | Top 1,000 | 66.7% | | Top 5,000 | 81.2% | | Top 10,000 | 85.9% | ### Key Findings - **Zipf Compliance:** R²=0.9953 indicates excellent adherence to Zipf's law - **High Frequency Dominance:** Top 100 words cover 36.8% of corpus - **Long Tail:** 307,640 words needed for remaining 14.1% coverage --- ## 5. Word Embeddings Evaluation ![Embedding Isotropy](visualizations/embedding_isotropy.png) ![Similarity Matrix](visualizations/embedding_similarity.png) ![t-SNE Words](visualizations/tsne_words.png) ![t-SNE Sentences](visualizations/tsne_sentences.png) ### Model Comparison | Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy | |-------|------------|-----------|----------|----------|----------| | **mono_32d** | 134,712 | 32 | 5.098 | 1.243 | 0.8242 🏆 | | **mono_64d** | 134,712 | 64 | 5.525 | 1.165 | 0.7928 | | **mono_128d** | 134,712 | 128 | 6.056 | 1.052 | 0.7520 | | **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 | ### Key Findings - **Best Isotropy:** mono_32d with 0.8242 (more uniform distribution) - **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy - **Vocabulary Coverage:** All models cover 134,712 words - **Recommendation:** 100d for balanced semantic capture and efficiency --- ## 6. Summary & Recommendations ![Performance Dashboard](visualizations/performance_dashboard.png) ### Production Recommendations | Component | Recommended | Rationale | |-----------|-------------|-----------| | Tokenizer | **32k BPE** | Best compression (4.20x) with low UNK rate | | N-gram | **5-gram** | Lowest perplexity (655) | | Markov | **Context-4** | Highest predictability (93.6%) | | Embeddings | **100d** | Balanced semantic capture and isotropy | --- ## Appendix: Metrics Glossary & Interpretation Guide This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report. ### Tokenizer Metrics **Compression Ratio** > *Definition:* The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text. > > *Intuition:* Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average. > > *What to seek:* Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information. **Average Token Length (Fertility)** > *Definition:* Mean number of characters per token produced by the tokenizer. > > *Intuition:* Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length. > > *What to seek:* Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens. **Unknown Token Rate (OOV Rate)** > *Definition:* Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent. > > *Intuition:* Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences. > > *What to seek:* Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback. ### N-gram Model Metrics **Perplexity** > *Definition:* Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction. > > *Intuition:* If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options. > > *What to seek:* Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size. **Entropy** > *Definition:* Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy. > > *Intuition:* High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character. > > *What to seek:* Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases. **Coverage (Top-K)** > *Definition:* Percentage of corpus occurrences explained by the top K most frequent n-grams. > > *Intuition:* High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage. > > *What to seek:* Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text. ### Markov Chain Metrics **Average Entropy** > *Definition:* Mean entropy across all contexts, measuring average uncertainty in next-word prediction. > > *Intuition:* Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations). > > *What to seek:* Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions. **Branching Factor** > *Definition:* Average number of unique next tokens observed for each context. > > *Intuition:* High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive). > > *What to seek:* Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains. **Predictability** > *Definition:* Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are. > > *Intuition:* 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes. > > *What to seek:* Higher predictability for text generation quality, but too high (>98%) may produce repetitive output. ### Vocabulary & Zipf's Law Metrics **Zipf's Coefficient** > *Definition:* The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1. > > *Intuition:* A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare. > > *What to seek:* Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text. **R² (Coefficient of Determination)** > *Definition:* Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1. > > *Intuition:* R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns. > > *What to seek:* R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora. **Vocabulary Coverage** > *Definition:* Cumulative percentage of corpus tokens accounted for by the top N words. > > *Intuition:* Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words. > > *What to seek:* Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary. ### Word Embedding Metrics **Isotropy** > *Definition:* Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values. > > *Intuition:* High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness. > > *What to seek:* Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy. **Average Norm** > *Definition:* Mean magnitude (L2 norm) of word vectors in the embedding space. > > *Intuition:* Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained. > > *What to seek:* Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation). **Cosine Similarity** > *Definition:* Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction). > > *Intuition:* Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings. > > *What to seek:* Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7. **t-SNE Visualization** > *Definition:* t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization. > > *Intuition:* Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence. > > *What to seek:* Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure. ### General Interpretation Guidelines 1. **Compare within model families:** Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer). 2. **Consider trade-offs:** Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate). 3. **Context matters:** Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification. 4. **Corpus influence:** All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature. 5. **Language-specific patterns:** Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages. ### Visualizations Index | Visualization | Description | |---------------|-------------| | Tokenizer Compression | Compression ratios by vocabulary size | | Tokenizer Fertility | Average token length by vocabulary | | Tokenizer OOV | Unknown token rates | | Tokenizer Total Tokens | Total tokens by vocabulary | | N-gram Perplexity | Perplexity by n-gram size | | N-gram Entropy | Entropy by n-gram size | | N-gram Coverage | Top pattern coverage | | N-gram Unique | Unique n-gram counts | | Markov Entropy | Entropy by context size | | Markov Branching | Branching factor by context | | Markov Contexts | Unique context counts | | Zipf's Law | Frequency-rank distribution with fit | | Vocab Frequency | Word frequency distribution | | Top 20 Words | Most frequent words | | Vocab Coverage | Cumulative coverage curve | | Embedding Isotropy | Vector space uniformity | | Embedding Norms | Vector magnitude distribution | | Embedding Similarity | Word similarity heatmap | | Nearest Neighbors | Similar words for key terms | | t-SNE Words | 2D word embedding visualization | | t-SNE Sentences | 2D sentence embedding visualization | | Position Encoding | Encoding method comparison | | Model Sizes | Storage requirements | | Performance Dashboard | Comprehensive performance overview | --- ## About This Project ### Data Source Models trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) - a monthly snapshot of Wikipedia articles across 300+ languages. ### Project A project by **[Wikilangs](https://wikilangs.org)** - Open-source NLP models for every Wikipedia language. ### Maintainer [Omar Kamali](https://omarkamali.com) - [Omneity Labs](https://omneitylabs.com) ### Citation If you use these models in your research, please cite: ```bibtex @misc{wikilangs2025, author = {Kamali, Omar}, title = {Wikilangs: Open NLP Models for Wikipedia Languages}, year = {2025}, publisher = {HuggingFace}, url = {https://huggingface.co/wikilangs} institution = {Omneity Labs} } ``` ### License MIT License - Free for academic and commercial use. ### Links - 🌐 Website: [wikilangs.org](https://wikilangs.org) - 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs) - 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) - 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali) --- *Generated by Wikilangs Models Pipeline* *Report Date: 2025-12-27 22:57:16*