|
|
--- |
|
|
language: azb |
|
|
language_name: AZB |
|
|
language_family: turkic_oghuz |
|
|
tags: |
|
|
- wikilangs |
|
|
- nlp |
|
|
- tokenizer |
|
|
- embeddings |
|
|
- n-gram |
|
|
- markov |
|
|
- wikipedia |
|
|
- monolingual |
|
|
- family-turkic_oghuz |
|
|
license: mit |
|
|
library_name: wikilangs |
|
|
pipeline_tag: feature-extraction |
|
|
datasets: |
|
|
- omarkamali/wikipedia-monthly |
|
|
dataset_info: |
|
|
name: wikipedia-monthly |
|
|
description: Monthly snapshots of Wikipedia articles across 300+ languages |
|
|
metrics: |
|
|
- name: best_compression_ratio |
|
|
type: compression |
|
|
value: 4.198 |
|
|
- name: best_isotropy |
|
|
type: isotropy |
|
|
value: 0.8242 |
|
|
- name: vocabulary_size |
|
|
type: vocab |
|
|
value: 317640 |
|
|
generated: 2025-12-27 |
|
|
--- |
|
|
|
|
|
# AZB - Wikilangs Models |
|
|
## Comprehensive Research Report & Full Ablation Study |
|
|
|
|
|
This repository contains NLP models trained and evaluated by Wikilangs, specifically on **AZB** Wikipedia data. |
|
|
We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings. |
|
|
|
|
|
## 📋 Repository Contents |
|
|
|
|
|
### Models & Assets |
|
|
|
|
|
- Tokenizers (8k, 16k, 32k, 64k) |
|
|
- N-gram models (2, 3, 4-gram) |
|
|
- Markov chains (context of 1, 2, 3 and 4) |
|
|
- Subword N-gram and Markov chains |
|
|
- Embeddings in various sizes and dimensions |
|
|
- Language Vocabulary |
|
|
- Language Statistics |
|
|
 |
|
|
|
|
|
### Analysis and Evaluation |
|
|
|
|
|
- [1. Tokenizer Evaluation](#1-tokenizer-evaluation) |
|
|
- [2. N-gram Model Evaluation](#2-n-gram-model-evaluation) |
|
|
- [3. Markov Chain Evaluation](#3-markov-chain-evaluation) |
|
|
- [4. Vocabulary Analysis](#4-vocabulary-analysis) |
|
|
- [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation) |
|
|
- [6. Summary & Recommendations](#6-summary--recommendations) |
|
|
- [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide) |
|
|
- [Visualizations Index](#visualizations-index) |
|
|
|
|
|
--- |
|
|
## 1. Tokenizer Evaluation |
|
|
|
|
|
 |
|
|
|
|
|
### Results |
|
|
|
|
|
| Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens | |
|
|
|------------|-------------|---------------|----------|--------------| |
|
|
| **8k** | 3.179x | 3.12 | 0.3165% | 385,493 | |
|
|
| **16k** | 3.558x | 3.49 | 0.3542% | 344,465 | |
|
|
| **32k** | 3.895x | 3.82 | 0.3877% | 314,660 | |
|
|
| **64k** | 4.198x 🏆 | 4.12 | 0.4179% | 291,946 | |
|
|
|
|
|
### Tokenization Examples |
|
|
|
|
|
Below are sample sentences tokenized with each vocabulary size: |
|
|
|
|
|
**Sample 1:** `ماقنتیتی ( ) روسیه اؤلکهسینده یئر آلان بیر کند دیر و مورمانسک اوبلاستیندا یئرلش...` |
|
|
|
|
|
| Vocab | Tokens | Count | |
|
|
|-------|--------|-------| |
|
|
| 8k | `▁م اق ن تی تی ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ... (+19 more)` | 29 | |
|
|
| 16k | `▁ماق ن تی تی ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ▁یئر ... (+16 more)` | 26 | |
|
|
| 32k | `▁ماق نتی تی ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ▁یئر ▁آلان ... (+13 more)` | 23 | |
|
|
| 64k | `▁ماق نتی تی ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ▁یئر ▁آلان ... (+13 more)` | 23 | |
|
|
|
|
|
**Sample 2:** `سید ضیاء الدین طباطبائی یزدی (دوغوم:۱۲۸۶ شیراز -اؤلوم:۷ شهریور ۱۳۴۸ تهران) احمدش...` |
|
|
|
|
|
| Vocab | Tokens | Count | |
|
|
|-------|--------|-------| |
|
|
| 8k | `▁سید ▁ض یا ء ▁الدین ▁ط با ط با ئی ... (+34 more)` | 44 | |
|
|
| 16k | `▁سید ▁ضیا ء ▁الدین ▁ط با ط با ئی ▁ی ... (+29 more)` | 39 | |
|
|
| 32k | `▁سید ▁ضیا ء ▁الدین ▁ط باط با ئی ▁یز دی ... (+26 more)` | 36 | |
|
|
| 64k | `▁سید ▁ضیاء ▁الدین ▁طباطبا ئی ▁یزدی ▁( دوغوم : ۱۲ ... (+21 more)` | 31 | |
|
|
|
|
|
**Sample 3:** `ویکسن (اینگیلیسجه: Vixen, Louisiana) آمریکانین لوئیزیانا ایالتینده یئرلشن بیر یا...` |
|
|
|
|
|
| Vocab | Tokens | Count | |
|
|
|-------|--------|-------| |
|
|
| 8k | `▁ویک سن ▁( اینگیلیسجه : ▁v ix en , ▁louisiana ... (+15 more)` | 25 | |
|
|
| 16k | `▁ویک سن ▁( اینگیلیسجه : ▁v ix en , ▁louisiana ... (+15 more)` | 25 | |
|
|
| 32k | `▁ویک سن ▁( اینگیلیسجه : ▁v ix en , ▁louisiana ... (+15 more)` | 25 | |
|
|
| 64k | `▁ویک سن ▁( اینگیلیسجه : ▁vixen , ▁louisiana ) ▁آمریکانین ... (+13 more)` | 23 | |
|
|
|
|
|
|
|
|
### Key Findings |
|
|
|
|
|
- **Best Compression:** 64k achieves 4.198x compression |
|
|
- **Lowest UNK Rate:** 8k with 0.3165% unknown tokens |
|
|
- **Trade-off:** Larger vocabularies improve compression but increase model size |
|
|
- **Recommendation:** 32k vocabulary provides optimal balance for production use |
|
|
|
|
|
--- |
|
|
## 2. N-gram Model Evaluation |
|
|
|
|
|
 |
|
|
|
|
|
 |
|
|
|
|
|
### Results |
|
|
|
|
|
| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage | |
|
|
|--------|------------|---------|----------------|------------------|-------------------| |
|
|
| **2-gram** | 7,645 🏆 | 12.90 | 268,529 | 30.2% | 57.7% | |
|
|
| **2-gram** | 655 🏆 | 9.35 | 14,131 | 46.5% | 93.7% | |
|
|
| **3-gram** | 13,711 | 13.74 | 512,828 | 24.9% | 52.4% | |
|
|
| **3-gram** | 4,808 | 12.23 | 141,425 | 21.1% | 58.1% | |
|
|
| **4-gram** | 23,522 | 14.52 | 979,944 | 21.9% | 47.2% | |
|
|
| **4-gram** | 19,536 | 14.25 | 804,296 | 13.9% | 41.7% | |
|
|
|
|
|
### Top 5 N-grams by Size |
|
|
|
|
|
**2-grams:** |
|
|
|
|
|
| Rank | N-gram | Count | |
|
|
|------|--------|-------| |
|
|
| 1 | `بؤلمه :` | 391,479 | |
|
|
| 2 | `او ْ` | 139,043 | |
|
|
| 3 | ` لار` | 137,076 | |
|
|
| 4 | `قایناقلار بؤلمه` | 125,059 | |
|
|
| 5 | ` نین` | 118,471 | |
|
|
|
|
|
**3-grams:** |
|
|
|
|
|
| Rank | N-gram | Count | |
|
|
|------|--------|-------| |
|
|
| 1 | `قایناقلار بؤلمه :` | 125,058 | |
|
|
| 2 | `( اینگیلیسجه :` | 112,489 | |
|
|
| 3 | `. قایناقلار بؤلمه` | 97,901 | |
|
|
| 4 | `قایناق لار` | 92,555 | |
|
|
| 5 | `ایشلدنلری طرفیندن یارانمیش` | 76,848 | |
|
|
|
|
|
**4-grams:** |
|
|
|
|
|
| Rank | N-gram | Count | |
|
|
|------|--------|-------| |
|
|
| 1 | `. قایناقلار بؤلمه :` | 97,901 | |
|
|
| 2 | `ایشلدنلری طرفیندن یارانمیش «` | 76,839 | |
|
|
| 3 | `ْ خلانیلیبدیر ) .` | 76,827 | |
|
|
| 4 | `» ، مقاله ` | 76,824 | |
|
|
| 5 | `، مقاله سیندن` | 76,824 | |
|
|
|
|
|
|
|
|
### Key Findings |
|
|
|
|
|
- **Best Perplexity:** 2-gram with 655 |
|
|
- **Entropy Trend:** Decreases with larger n-grams (more predictable) |
|
|
- **Coverage:** Top-1000 patterns cover ~42% of corpus |
|
|
- **Recommendation:** 4-gram or 5-gram for best predictive performance |
|
|
|
|
|
--- |
|
|
## 3. Markov Chain Evaluation |
|
|
|
|
|
 |
|
|
|
|
|
 |
|
|
|
|
|
### Results |
|
|
|
|
|
| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability | |
|
|
|---------|-------------|------------|------------------|-----------------|----------------| |
|
|
| **1** | 0.4591 | 1.375 | 4.28 | 1,005,885 | 54.1% | |
|
|
| **1** | 1.2059 | 2.307 | 10.74 | 2,934 | 0.0% | |
|
|
| **2** | 0.2308 | 1.173 | 1.71 | 4,302,287 | 76.9% | |
|
|
| **2** | 1.1061 | 2.153 | 8.32 | 31,498 | 0.0% | |
|
|
| **3** | 0.1101 | 1.079 | 1.28 | 7,369,809 | 89.0% | |
|
|
| **3** | 0.9393 | 1.918 | 5.09 | 262,151 | 6.1% | |
|
|
| **4** | 0.0637 🏆 | 1.045 | 1.14 | 9,387,722 | 93.6% | |
|
|
| **4** | 0.7222 🏆 | 1.650 | 3.21 | 1,333,406 | 27.8% | |
|
|
|
|
|
### Generated Text Samples |
|
|
|
|
|
Below are text samples generated from each Markov chain model: |
|
|
|
|
|
**Context Size 1:** |
|
|
|
|
|
1. ` ایللیین اووللرین ده یاشایان اینسانلار بؤلمه : bhumibol adulyadej crown , footballer , alchimie du` |
|
|
2. `. قان معائناتی و زنگین نین ایشلدنلری طرفیندن یارانمیش « panama 2 episodes ایلآدی رو` |
|
|
3. `: sergey ryzhikov ( ۱۹۴۸ ایلینده گونئی کارولینا ( تحصیلات ) رومنی ( لهیستانجا : 2` |
|
|
|
|
|
**Context Size 2:** |
|
|
|
|
|
1. `بؤلمه : وورت بؤلگه سیاراک بؤلگه سی دیر گؤرونتولر قایناق لار اینگیلیسجه ویکی` |
|
|
2. `او ْ یونچو . ۲۶ ژوئن ۱۹۵۷ میلادی تاریخیندا وفات ائدیب . قایناق لار اینگیلیسجه ویکی` |
|
|
3. ` لار بؤلمه : آمریکا شهرلری بؤلمه : گرینزبورو دوغوملولار بؤلمه : ۲۰۱۹ - جو ایلینده افغانیستاندا` |
|
|
|
|
|
**Context Size 3:** |
|
|
|
|
|
1. `قایناقلار بؤلمه : هیندوستان کندلری en : kahra` |
|
|
2. `( اینگیلیسجه : louis prima ) آمریکالی موغنی ، یازیچی و او ْ ندان ایستیفاده ائتمیشدیر . روبنسین` |
|
|
3. `. قایناقلار بؤلمه : هیندوستان کندلری en : kalawad` |
|
|
|
|
|
**Context Size 4:** |
|
|
|
|
|
1. `. قایناقلار بؤلمه : هیندوستان کندلری en : balowal` |
|
|
2. `ایشلدنلری طرفیندن یارانمیش « crabtree ' s catalyst » ، مقاله سیندن گؤتورولوبدور . ( ۷ سپتامبر ۲۰۱۷` |
|
|
3. `، مقاله سیندن گؤتورولوبدور . ( ۱۲ ژوئن ۲۰۱۸ تاریخینده یو ْ خلانیلیبدیر ) . بؤلمه : ایستادیوملار` |
|
|
|
|
|
|
|
|
### Key Findings |
|
|
|
|
|
- **Best Predictability:** Context-4 with 93.6% predictability |
|
|
- **Branching Factor:** Decreases with context size (more deterministic) |
|
|
- **Memory Trade-off:** Larger contexts require more storage (1,333,406 contexts) |
|
|
- **Recommendation:** Context-3 or Context-4 for text generation |
|
|
|
|
|
--- |
|
|
## 4. Vocabulary Analysis |
|
|
|
|
|
 |
|
|
|
|
|
 |
|
|
|
|
|
 |
|
|
|
|
|
### Statistics |
|
|
|
|
|
| Metric | Value | |
|
|
|--------|-------| |
|
|
| Vocabulary Size | 317,640 | |
|
|
| Total Tokens | 16,697,106 | |
|
|
| Mean Frequency | 52.57 | |
|
|
| Median Frequency | 3 | |
|
|
| Frequency Std Dev | 1499.80 | |
|
|
|
|
|
### Most Common Words |
|
|
|
|
|
| Rank | Word | Frequency | |
|
|
|------|------|-----------| |
|
|
| 1 | بؤلمه | 391,840 | |
|
|
| 2 | و | 291,285 | |
|
|
| 3 | اینگیلیسجه | 188,551 | |
|
|
| 4 | بیر | 169,796 | |
|
|
| 5 | او | 145,694 | |
|
|
| 6 | قایناقلار | 143,368 | |
|
|
| 7 | سی | 139,966 | |
|
|
| 8 | لار | 138,388 | |
|
|
| 9 | دیر | 132,038 | |
|
|
| 10 | the | 130,277 | |
|
|
|
|
|
### Least Common Words (from vocabulary) |
|
|
|
|
|
| Rank | Word | Frequency | |
|
|
|------|------|-----------| |
|
|
| 1 | سئندن | 2 | |
|
|
| 2 | ائششکین | 2 | |
|
|
| 3 | onager | 2 | |
|
|
| 4 | داشناکلارلا | 2 | |
|
|
| 5 | قۇلان | 2 | |
|
|
| 6 | آسینۇس | 2 | |
|
|
| 7 | هئمیو | 2 | |
|
|
| 8 | تاپؽلمیشدیر | 2 | |
|
|
| 9 | kulan | 2 | |
|
|
| 10 | کسا | 2 | |
|
|
|
|
|
### Zipf's Law Analysis |
|
|
|
|
|
| Metric | Value | |
|
|
|--------|-------| |
|
|
| Zipf Coefficient | 1.1842 | |
|
|
| R² (Goodness of Fit) | 0.995333 | |
|
|
| Adherence Quality | **excellent** | |
|
|
|
|
|
### Coverage Analysis |
|
|
|
|
|
| Top N Words | Coverage | |
|
|
|-------------|----------| |
|
|
| Top 100 | 36.8% | |
|
|
| Top 1,000 | 66.7% | |
|
|
| Top 5,000 | 81.2% | |
|
|
| Top 10,000 | 85.9% | |
|
|
|
|
|
### Key Findings |
|
|
|
|
|
- **Zipf Compliance:** R²=0.9953 indicates excellent adherence to Zipf's law |
|
|
- **High Frequency Dominance:** Top 100 words cover 36.8% of corpus |
|
|
- **Long Tail:** 307,640 words needed for remaining 14.1% coverage |
|
|
|
|
|
--- |
|
|
## 5. Word Embeddings Evaluation |
|
|
|
|
|
 |
|
|
|
|
|
 |
|
|
|
|
|
 |
|
|
|
|
|
 |
|
|
|
|
|
### Model Comparison |
|
|
|
|
|
| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy | |
|
|
|-------|------------|-----------|----------|----------|----------| |
|
|
| **mono_32d** | 134,712 | 32 | 5.098 | 1.243 | 0.8242 🏆 | |
|
|
| **mono_64d** | 134,712 | 64 | 5.525 | 1.165 | 0.7928 | |
|
|
| **mono_128d** | 134,712 | 128 | 6.056 | 1.052 | 0.7520 | |
|
|
| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 | |
|
|
|
|
|
### Key Findings |
|
|
|
|
|
- **Best Isotropy:** mono_32d with 0.8242 (more uniform distribution) |
|
|
- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy |
|
|
- **Vocabulary Coverage:** All models cover 134,712 words |
|
|
- **Recommendation:** 100d for balanced semantic capture and efficiency |
|
|
|
|
|
--- |
|
|
## 6. Summary & Recommendations |
|
|
|
|
|
 |
|
|
|
|
|
### Production Recommendations |
|
|
|
|
|
| Component | Recommended | Rationale | |
|
|
|-----------|-------------|-----------| |
|
|
| Tokenizer | **32k BPE** | Best compression (4.20x) with low UNK rate | |
|
|
| N-gram | **5-gram** | Lowest perplexity (655) | |
|
|
| Markov | **Context-4** | Highest predictability (93.6%) | |
|
|
| Embeddings | **100d** | Balanced semantic capture and isotropy | |
|
|
|
|
|
--- |
|
|
## Appendix: Metrics Glossary & Interpretation Guide |
|
|
|
|
|
This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report. |
|
|
|
|
|
### Tokenizer Metrics |
|
|
|
|
|
**Compression Ratio** |
|
|
> *Definition:* The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text. |
|
|
> |
|
|
> *Intuition:* Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average. |
|
|
> |
|
|
> *What to seek:* Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information. |
|
|
|
|
|
**Average Token Length (Fertility)** |
|
|
> *Definition:* Mean number of characters per token produced by the tokenizer. |
|
|
> |
|
|
> *Intuition:* Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length. |
|
|
> |
|
|
> *What to seek:* Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens. |
|
|
|
|
|
**Unknown Token Rate (OOV Rate)** |
|
|
> *Definition:* Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent. |
|
|
> |
|
|
> *Intuition:* Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences. |
|
|
> |
|
|
> *What to seek:* Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback. |
|
|
|
|
|
### N-gram Model Metrics |
|
|
|
|
|
**Perplexity** |
|
|
> *Definition:* Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction. |
|
|
> |
|
|
> *Intuition:* If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options. |
|
|
> |
|
|
> *What to seek:* Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size. |
|
|
|
|
|
**Entropy** |
|
|
> *Definition:* Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy. |
|
|
> |
|
|
> *Intuition:* High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character. |
|
|
> |
|
|
> *What to seek:* Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases. |
|
|
|
|
|
**Coverage (Top-K)** |
|
|
> *Definition:* Percentage of corpus occurrences explained by the top K most frequent n-grams. |
|
|
> |
|
|
> *Intuition:* High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage. |
|
|
> |
|
|
> *What to seek:* Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text. |
|
|
|
|
|
### Markov Chain Metrics |
|
|
|
|
|
**Average Entropy** |
|
|
> *Definition:* Mean entropy across all contexts, measuring average uncertainty in next-word prediction. |
|
|
> |
|
|
> *Intuition:* Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations). |
|
|
> |
|
|
> *What to seek:* Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions. |
|
|
|
|
|
**Branching Factor** |
|
|
> *Definition:* Average number of unique next tokens observed for each context. |
|
|
> |
|
|
> *Intuition:* High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive). |
|
|
> |
|
|
> *What to seek:* Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains. |
|
|
|
|
|
**Predictability** |
|
|
> *Definition:* Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are. |
|
|
> |
|
|
> *Intuition:* 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes. |
|
|
> |
|
|
> *What to seek:* Higher predictability for text generation quality, but too high (>98%) may produce repetitive output. |
|
|
|
|
|
### Vocabulary & Zipf's Law Metrics |
|
|
|
|
|
**Zipf's Coefficient** |
|
|
> *Definition:* The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1. |
|
|
> |
|
|
> *Intuition:* A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare. |
|
|
> |
|
|
> *What to seek:* Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text. |
|
|
|
|
|
**R² (Coefficient of Determination)** |
|
|
> *Definition:* Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1. |
|
|
> |
|
|
> *Intuition:* R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns. |
|
|
> |
|
|
> *What to seek:* R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora. |
|
|
|
|
|
**Vocabulary Coverage** |
|
|
> *Definition:* Cumulative percentage of corpus tokens accounted for by the top N words. |
|
|
> |
|
|
> *Intuition:* Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words. |
|
|
> |
|
|
> *What to seek:* Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary. |
|
|
|
|
|
### Word Embedding Metrics |
|
|
|
|
|
**Isotropy** |
|
|
> *Definition:* Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values. |
|
|
> |
|
|
> *Intuition:* High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness. |
|
|
> |
|
|
> *What to seek:* Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy. |
|
|
|
|
|
**Average Norm** |
|
|
> *Definition:* Mean magnitude (L2 norm) of word vectors in the embedding space. |
|
|
> |
|
|
> *Intuition:* Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained. |
|
|
> |
|
|
> *What to seek:* Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation). |
|
|
|
|
|
**Cosine Similarity** |
|
|
> *Definition:* Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction). |
|
|
> |
|
|
> *Intuition:* Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings. |
|
|
> |
|
|
> *What to seek:* Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7. |
|
|
|
|
|
**t-SNE Visualization** |
|
|
> *Definition:* t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization. |
|
|
> |
|
|
> *Intuition:* Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence. |
|
|
> |
|
|
> *What to seek:* Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure. |
|
|
|
|
|
### General Interpretation Guidelines |
|
|
|
|
|
1. **Compare within model families:** Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer). |
|
|
2. **Consider trade-offs:** Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate). |
|
|
3. **Context matters:** Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification. |
|
|
4. **Corpus influence:** All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature. |
|
|
5. **Language-specific patterns:** Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages. |
|
|
|
|
|
|
|
|
### Visualizations Index |
|
|
|
|
|
| Visualization | Description | |
|
|
|---------------|-------------| |
|
|
| Tokenizer Compression | Compression ratios by vocabulary size | |
|
|
| Tokenizer Fertility | Average token length by vocabulary | |
|
|
| Tokenizer OOV | Unknown token rates | |
|
|
| Tokenizer Total Tokens | Total tokens by vocabulary | |
|
|
| N-gram Perplexity | Perplexity by n-gram size | |
|
|
| N-gram Entropy | Entropy by n-gram size | |
|
|
| N-gram Coverage | Top pattern coverage | |
|
|
| N-gram Unique | Unique n-gram counts | |
|
|
| Markov Entropy | Entropy by context size | |
|
|
| Markov Branching | Branching factor by context | |
|
|
| Markov Contexts | Unique context counts | |
|
|
| Zipf's Law | Frequency-rank distribution with fit | |
|
|
| Vocab Frequency | Word frequency distribution | |
|
|
| Top 20 Words | Most frequent words | |
|
|
| Vocab Coverage | Cumulative coverage curve | |
|
|
| Embedding Isotropy | Vector space uniformity | |
|
|
| Embedding Norms | Vector magnitude distribution | |
|
|
| Embedding Similarity | Word similarity heatmap | |
|
|
| Nearest Neighbors | Similar words for key terms | |
|
|
| t-SNE Words | 2D word embedding visualization | |
|
|
| t-SNE Sentences | 2D sentence embedding visualization | |
|
|
| Position Encoding | Encoding method comparison | |
|
|
| Model Sizes | Storage requirements | |
|
|
| Performance Dashboard | Comprehensive performance overview | |
|
|
|
|
|
--- |
|
|
## About This Project |
|
|
|
|
|
### Data Source |
|
|
|
|
|
Models trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) - a monthly snapshot of Wikipedia articles across 300+ languages. |
|
|
|
|
|
### Project |
|
|
|
|
|
A project by **[Wikilangs](https://wikilangs.org)** - Open-source NLP models for every Wikipedia language. |
|
|
|
|
|
### Maintainer |
|
|
|
|
|
[Omar Kamali](https://omarkamali.com) - [Omneity Labs](https://omneitylabs.com) |
|
|
|
|
|
### Citation |
|
|
|
|
|
If you use these models in your research, please cite: |
|
|
|
|
|
```bibtex |
|
|
@misc{wikilangs2025, |
|
|
author = {Kamali, Omar}, |
|
|
title = {Wikilangs: Open NLP Models for Wikipedia Languages}, |
|
|
year = {2025}, |
|
|
publisher = {HuggingFace}, |
|
|
url = {https://huggingface.co/wikilangs} |
|
|
institution = {Omneity Labs} |
|
|
} |
|
|
``` |
|
|
|
|
|
### License |
|
|
|
|
|
MIT License - Free for academic and commercial use. |
|
|
|
|
|
### Links |
|
|
|
|
|
- 🌐 Website: [wikilangs.org](https://wikilangs.org) |
|
|
- 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs) |
|
|
- 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) |
|
|
- 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali) |
|
|
--- |
|
|
*Generated by Wikilangs Models Pipeline* |
|
|
|
|
|
*Report Date: 2025-12-27 22:57:16* |
|
|
|