Upload all models and assets for ary (20251201)

.gitattributes

@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+visualizations/embedding_similarity.png filter=lfs diff=lfs merge=lfs -text
+visualizations/performance_dashboard.png filter=lfs diff=lfs merge=lfs -text
+visualizations/tsne_sentences.png filter=lfs diff=lfs merge=lfs -text
+visualizations/tsne_words.png filter=lfs diff=lfs merge=lfs -text
+visualizations/zipf_law.png filter=lfs diff=lfs merge=lfs -text

COMPREHENSIVE_RESEARCH_REPORT.md

@@ -0,0 +1,463 @@
+# Wikilangs Models: Comprehensive Research Report
+## ARY - Full Ablation Study
+
+This report presents a comprehensive evaluation of language models trained on ARY Wikipedia data.
+We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
+
+---
+## 1. Tokenizer Evaluation
+
+![Tokenizer Compression](visualizations/01_tokenizer_compression.png)
+
+### Results
+
+| Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
+|------------|-------------|---------------|----------|--------------|
+| **8k** | 3.134x | 3.09 | 0.0472% | 379,309 |
+| **16k** | 3.346x | 3.30 | 0.0504% | 355,311 |
+| **32k** | 3.535x | 3.49 | 0.0532% | 336,296 |
+| **64k** | 3.683x 🏆 | 3.64 | 0.0555% | 322,761 |
+
+### Tokenization Examples
+
+Below are sample sentences tokenized with each vocabulary size:
+
+**Sample 1:** `نينڭ بايزورا بنت الشيخ حمزة أولا نينڭ بايزورا هي مومتيلة وموغنية ماليزية.
+
+ مصاد...`
+
+| Vocab | Tokens | Count |
+|-------|--------|-------|
+| 8k | `▁ن ينڭ ▁باي ز ورا ▁بنت ▁الشيخ ▁حم زة ▁أولا ... (+32 more)` | 42 |
+| 16k | `▁ن ينڭ ▁باي ز ورا ▁بنت ▁الشيخ ▁حمزة ▁أولا ▁ن ... (+29 more)` | 39 |
+| 32k | `▁ن ينڭ ▁باي ز ورا ▁بنت ▁الشيخ ▁حمزة ▁أولا ▁ن ... (+29 more)` | 39 |
+| 64k | `▁ن ينڭ ▁باي ز ورا ▁بنت ▁الشيخ ▁حمزة ▁أولا ▁ن ... (+27 more)` | 37 |
+
+**Sample 2:** `هادي صفحة د التوضيح، كلمة بركان يمكن يكونو عندها هاد لمعاني:
+
+ بْرْكان: مدينة مغ...`
+
+| Vocab | Tokens | Count |
+|-------|--------|-------|
+| 8k | `▁هادي ▁صفحة ▁د ▁التوضيح ، ▁كلمة ▁بركان ▁يمكن ▁يكونو ▁عندها ... (+26 more)` | 36 |
+| 16k | `▁هادي ▁صفحة ▁د ▁التوضيح ، ▁كلمة ▁بركان ▁يمكن ▁يكونو ▁عندها ... (+25 more)` | 35 |
+| 32k | `▁هادي ▁صفحة ▁د ▁التوضيح ، ▁كلمة ▁بركان ▁يمكن ▁يكونو ▁عندها ... (+24 more)` | 34 |
+| 64k | `▁هادي ▁صفحة ▁د ▁التوضيح ، ▁كلمة ▁بركان ▁يمكن ▁يكونو ▁عندها ... (+22 more)` | 32 |
+
+**Sample 3:** `أسيل عمران (مزيودة ف 1989) هي مغنية و ممتلة سعودية كتعيش ف لإمارات.
+
+ مصادر 
+
+تص...`
+
+| Vocab | Tokens | Count |
+|-------|--------|-------|
+| 8k | `▁أس يل ▁عمر ان ▁( مزيودة ▁ف ▁ 1 9 ... (+36 more)` | 46 |
+| 16k | `▁أس يل ▁عمر ان ▁( مزيودة ▁ف ▁ 1 9 ... (+32 more)` | 42 |
+| 32k | `▁أس يل ▁عمران ▁( مزيودة ▁ف ▁ 1 9 8 ... (+28 more)` | 38 |
+| 64k | `▁أس يل ▁عمران ▁( مزيودة ▁ف ▁ 1 9 8 ... (+28 more)` | 38 |
+
+
+### Key Findings
+
+- **Best Compression:** 64k achieves 3.683x compression
+- **Lowest UNK Rate:** 8k with 0.0472% unknown tokens
+- **Trade-off:** Larger vocabularies improve compression but increase model size
+- **Recommendation:** 32k vocabulary provides optimal balance for production use
+
+---
+## 2. N-gram Model Evaluation
+
+![N-gram Perplexity](visualizations/05_ngram_perplexity.png)
+
+![N-gram Coverage](visualizations/07_ngram_coverage.png)
+
+### Results
+
+| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | 7,187 🏆 | 12.81 | 56,749 | 24.4% | 53.2% |
+| **2-gram** | 486 🏆 | 8.93 | 6,227 | 54.9% | 95.4% |
+| **3-gram** | 8,812 | 13.11 | 76,888 | 21.3% | 52.8% |
+| **3-gram** | 4,295 | 12.07 | 51,256 | 22.1% | 58.7% |
+| **4-gram** | 12,168 | 13.57 | 124,859 | 20.1% | 50.4% |
+| **4-gram** | 22,008 | 14.43 | 260,844 | 12.0% | 35.5% |
+
+### Top 5 N-grams by Size
+
+**2-grams:**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `تصنيف :` | 37,187 |
+| 2 | `، و` | 18,746 |
+| 3 | `ن ّ` | 10,639 |
+| 4 | `) :` | 10,185 |
+| 5 | `مصادر تصنيف` | 10,087 |
+
+**3-grams:**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `مصادر تصنيف :` | 10,087 |
+| 2 | `تصنيف : مقالات` | 7,001 |
+| 3 | `ن ّ اس` | 6,981 |
+| 4 | `ل ّ ي` | 6,914 |
+| 5 | `: دوار ف` | 5,007 |
+
+**4-grams:**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `تصنيف : دوار ف` | 5,005 |
+| 2 | `نسبة ن ّ اس` | 4,061 |
+| 3 | `. مصادر تصنيف :` | 3,827 |
+| 4 | `تصنيف : مقالات زادهوم` | 3,506 |
+| 5 | `: مقالات زادهوم داريجابوت` | 3,506 |
+
+
+### Key Findings
+
+- **Best Perplexity:** 2-gram with 486
+- **Entropy Trend:** Decreases with larger n-grams (more predictable)
+- **Coverage:** Top-1000 patterns cover ~35% of corpus
+- **Recommendation:** 4-gram or 5-gram for best predictive performance
+
+---
+## 3. Markov Chain Evaluation
+
+![Markov Entropy](visualizations/09_markov_entropy.png)
+
+![Markov Branching](visualizations/10_markov_branching.png)
+
+### Results
+
+| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | 0.7813 | 1.719 | 5.36 | 189,320 | 21.9% |
+| **1** | 1.1519 | 2.222 | 8.71 | 1,931 | 0.0% |
+| **2** | 0.2761 | 1.211 | 1.68 | 1,014,676 | 72.4% |
+| **2** | 0.9863 | 1.981 | 6.24 | 16,826 | 1.4% |
+| **3** | 0.0931 | 1.067 | 1.18 | 1,701,309 | 90.7% |
+| **3** | 0.8744 | 1.833 | 4.33 | 104,928 | 12.6% |
+| **4** | 0.0366 🏆 | 1.026 | 1.07 | 2,000,181 | 96.3% |
+| **4** | 0.6731 🏆 | 1.594 | 2.82 | 454,694 | 32.7% |
+
+### Generated Text Samples
+
+Below are text samples generated from each Markov chain model:
+
+**Context Size 1:**
+
+1. `. قرات لانفورماتيك ، وحسبوهم النسابون المسلمين ) غايب مجموعntnən ( قائم الزاوية هو ، الطابلو`
+2. `، كان ل 6 % ، ولكن ماكخون ( ولا لبيطاليين اللي سبق ليهوم خدمو )`
+3. `ف كتاب " ف جماعة قروية ف دوك لي جاو فالغرب د لكورة تا نتيجة لاندماج`
+
+**Context Size 2:**
+
+1. `تصنيف : عوام د تقويم لميلادي تصنيف : نهارات د لعام تصنيف : كتاتبيا مغاربا د لقرن`
+2. `، و صدرات منو أغنية rip , love . الديسك خرج رسميا ً paypal holdings inc .`
+3. `ن ّ اس ن ّ شيطين ( ل ّ ي قاريين فوق الليسي ( ليسي و جامعة`
+
+**Context Size 3:**
+
+1. `مصادر تصنيف : يناير تصنيف : نهارات د لعام تصنيف : مقالات فيها مصدر و 3000 بايت تصنيف`
+2. `تصنيف : مقالات زادهوم داريجابوت تصنيف : بلايص مسكونين ف إقليم برشيد ، جهة د ّ ار لبيضا`
+3. `ن ّ اس اللي خدامين ف د ّ ولة : 4 , 4 % إقتصاد نسبة ن ّ`
+
+**Context Size 4:**
+
+1. `تصنيف : دوار ف لمغريب تصنيف : دوار ف لمغريب تصنيف : دوار ف لمغريب تصنيف : دوار ف`
+2. `نسبة ن ّ اس ن ّ شيطين ( ل ّ ي يقدرو يخدمو ) : 50 , 2 %`
+3. `. مصادر تصنيف : عوام د تقويم لميلادي تصنيف : مقالات زادهوم داريجابوت تصنيف : عوام 380 قبل لميلاد`
+
+
+### Key Findings
+
+- **Best Predictability:** Context-4 with 96.3% predictability
+- **Branching Factor:** Decreases with context size (more deterministic)
+- **Memory Trade-off:** Larger contexts require more storage (454,694 contexts)
+- **Recommendation:** Context-3 or Context-4 for text generation
+
+---
+## 4. Vocabulary Analysis
+
+![Zipf's Law](visualizations/12_zipf_law.png)
+
+![Top Words](visualizations/14_top20_words.png)
+
+![Coverage Curve](visualizations/15_vocab_coverage.png)
+
+### Statistics
+
+| Metric | Value |
+|--------|-------|
+| Vocabulary Size | 81,712 |
+| Total Tokens | 2,308,873 |
+| Mean Frequency | 28.26 |
+| Median Frequency | 4 |
+| Frequency Std Dev | 559.90 |
+
+### Most Common Words
+
+| Rank | Word | Frequency |
+|------|------|-----------|
+| 1 | ف | 84,463 |
+| 2 | د | 69,201 |
+| 3 | و | 61,463 |
+| 4 | تصنيف | 37,231 |
+| 5 | ل | 34,076 |
+| 6 | ديال | 32,761 |
+| 7 | من | 29,612 |
+| 8 | على | 19,717 |
+| 9 | لي | 18,627 |
+| 10 | ب | 18,189 |
+
+### Least Common Words (from vocabulary)
+
+| Rank | Word | Frequency |
+|------|------|-----------|
+| 1 | بيتسي | 2 |
+| 2 | وصانعي | 2 |
+| 3 | وأهميتها | 2 |
+| 4 | بورديو | 2 |
+| 5 | بلومر | 2 |
+| 6 | مقترحة | 2 |
+| 7 | anchor | 2 |
+| 8 | الرسميةاللي | 2 |
+| 9 | بعصبة | 2 |
+| 10 | ماڭي | 2 |
+
+### Zipf's Law Analysis
+
+| Metric | Value |
+|--------|-------|
+| Zipf Coefficient | 1.0380 |
+| R² (Goodness of Fit) | 0.999162 |
+| Adherence Quality | **excellent** |
+
+### Coverage Analysis
+
+| Top N Words | Coverage |
+|-------------|----------|
+| Top 100 | 39.3% |
+| Top 1,000 | 63.8% |
+| Top 5,000 | 78.6% |
+| Top 10,000 | 84.8% |
+
+### Key Findings
+
+- **Zipf Compliance:** R²=0.9992 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 39.3% of corpus
+- **Long Tail:** 71,712 words needed for remaining 15.2% coverage
+
+---
+## 5. Word Embeddings Evaluation
+
+![Embedding Isotropy](visualizations/16_embedding_isotropy.png)
+
+![Similarity Matrix](visualizations/18_embedding_similarity.png)
+
+![t-SNE Words](visualizations/20_tsne_words.png)
+
+![t-SNE Sentences](visualizations/21_tsne_sentences.png)
+
+### Model Comparison
+
+| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
+|-------|------------|-----------|----------|----------|----------|
+| **mono_32d** | 37,528 | 32 | 4.010 | 1.183 | 0.8264 🏆 |
+| **mono_64d** | 37,528 | 64 | 4.579 | 1.040 | 0.8183 |
+| **mono_128d** | 37,528 | 128 | 5.112 | 0.875 | 0.7212 |
+| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+
+### Key Findings
+
+- **Best Isotropy:** mono_32d with 0.8264 (more uniform distribution)
+- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
+- **Vocabulary Coverage:** All models cover 37,528 words
+- **Recommendation:** 100d for balanced semantic capture and efficiency
+
+---
+## 6. Summary & Recommendations
+
+![Performance Dashboard](visualizations/24_performance_dashboard.png)
+
+### Production Recommendations
+
+| Component | Recommended | Rationale |
+|-----------|-------------|-----------|
+| Tokenizer | **32k BPE** | Best compression (3.68x) with low UNK rate |
+| N-gram | **5-gram** | Lowest perplexity (486) |
+| Markov | **Context-4** | Highest predictability (96.3%) |
+| Embeddings | **100d** | Balanced semantic capture and isotropy |
+
+---
+## Appendix: Metrics Glossary & Interpretation Guide
+
+This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report.
+
+### Tokenizer Metrics
+
+**Compression Ratio**
+> *Definition:* The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text.
+>
+> *Intuition:* Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average.
+>
+> *What to seek:* Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information.
+
+**Average Token Length (Fertility)**
+> *Definition:* Mean number of characters per token produced by the tokenizer.
+>
+> *Intuition:* Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length.
+>
+> *What to seek:* Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens.
+
+**Unknown Token Rate (OOV Rate)**
+> *Definition:* Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent.
+>
+> *Intuition:* Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences.
+>
+> *What to seek:* Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback.
+
+### N-gram Model Metrics
+
+**Perplexity**
+> *Definition:* Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction.
+>
+> *Intuition:* If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options.
+>
+> *What to seek:* Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size.
+
+**Entropy**
+> *Definition:* Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy.
+>
+> *Intuition:* High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character.
+>
+> *What to seek:* Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases.
+
+**Coverage (Top-K)**
+> *Definition:* Percentage of corpus occurrences explained by the top K most frequent n-grams.
+>
+> *Intuition:* High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage.
+>
+> *What to seek:* Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text.
+
+### Markov Chain Metrics
+
+**Average Entropy**
+> *Definition:* Mean entropy across all contexts, measuring average uncertainty in next-word prediction.
+>
+> *Intuition:* Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations).
+>
+> *What to seek:* Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions.
+
+**Branching Factor**
+> *Definition:* Average number of unique next tokens observed for each context.
+>
+> *Intuition:* High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive).
+>
+> *What to seek:* Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains.
+
+**Predictability**
+> *Definition:* Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are.
+>
+> *Intuition:* 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes.
+>
+> *What to seek:* Higher predictability for text generation quality, but too high (>98%) may produce repetitive output.
+
+### Vocabulary & Zipf's Law Metrics
+
+**Zipf's Coefficient**
+> *Definition:* The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1.
+>
+> *Intuition:* A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare.
+>
+> *What to seek:* Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text.
+
+**R² (Coefficient of Determination)**
+> *Definition:* Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1.
+>
+> *Intuition:* R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns.
+>
+> *What to seek:* R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora.
+
+**Vocabulary Coverage**
+> *Definition:* Cumulative percentage of corpus tokens accounted for by the top N words.
+>
+> *Intuition:* Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words.
+>
+> *What to seek:* Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary.
+
+### Word Embedding Metrics
+
+**Isotropy**
+> *Definition:* Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values.
+>
+> *Intuition:* High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness.
+>
+> *What to seek:* Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy.
+
+**Average Norm**
+> *Definition:* Mean magnitude (L2 norm) of word vectors in the embedding space.
+>
+> *Intuition:* Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained.
+>
+> *What to seek:* Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation).
+
+**Cosine Similarity**
+> *Definition:* Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction).
+>
+> *Intuition:* Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings.
+>
+> *What to seek:* Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7.
+
+**t-SNE Visualization**
+> *Definition:* t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization.
+>
+> *Intuition:* Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence.
+>
+> *What to seek:* Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure.
+
+### General Interpretation Guidelines
+
+1. **Compare within model families:** Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer).
+2. **Consider trade-offs:** Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate).
+3. **Context matters:** Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification.
+4. **Corpus influence:** All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature.
+5. **Language-specific patterns:** Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages.
+
+
+### Visualizations Index
+
+| # | Visualization | Description |
+|---|---------------|-------------|
+| 01 | Tokenizer Compression | Compression ratios by vocabulary size |
+| 02 | Tokenizer Fertility | Average token length by vocabulary |
+| 03 | Tokenizer OOV | Unknown token rates |
+| 04 | Tokenizer Tokens | Total tokens by vocabulary |
+| 05 | N-gram Perplexity | Perplexity by n-gram size |
+| 06 | N-gram Entropy | Entropy by n-gram size |
+| 07 | N-gram Coverage | Top pattern coverage |
+| 08 | N-gram Unique | Unique n-gram counts |
+| 09 | Markov Entropy | Entropy by context size |
+| 10 | Markov Branching | Branching factor by context |
+| 11 | Markov Contexts | Unique context counts |
+| 12 | Zipf's Law | Frequency-rank distribution with fit |
+| 13 | Vocab Frequency | Word frequency distribution |
+| 14 | Top 20 Words | Most frequent words |
+| 15 | Vocab Coverage | Cumulative coverage curve |
+| 16 | Embedding Isotropy | Vector space uniformity |
+| 17 | Embedding Norms | Vector magnitude distribution |
+| 18 | Similarity Matrix | Word similarity heatmap |
+| 19 | Nearest Neighbors | Similar words for key terms |
+| 20 | t-SNE Words | 2D word embedding visualization |
+| 21 | t-SNE Sentences | 2D sentence embedding visualization |
+| 22 | Position Encoding | Encoding method comparison |
+| 23 | Model Sizes | Storage requirements |
+| 24 | Dashboard | Comprehensive performance overview |
+
+---
+*Generated by Wikilangs Models Pipeline*
+
+*Report Date: 2025-12-27 03:37:35*

README.md

@@ -0,0 +1,527 @@
+---
+language: ary
+language_name: Moroccan Arabic
+language_family: arabic
+tags:
+  - wikilangs
+  - nlp
+  - tokenizer
+  - embeddings
+  - n-gram
+  - markov
+  - wikipedia
+  - monolingual
+  - family-arabic
+license: mit
+library_name: wikilangs
+pipeline_tag: feature-extraction
+datasets:
+  - omarkamali/wikipedia-monthly
+metrics:
+  - name: best_compression_ratio
+    type: compression
+    value: 3.683
+  - name: best_isotropy
+    type: isotropy
+    value: 0.8264
+  - name: vocabulary_size
+    type: vocab
+    value: 81712
+generated: 2025-12-27
+---
+
+# Moroccan Arabic - Wikilangs Models
+## Comprehensive Research Report & Full Ablation Study
+
+This report presents a comprehensive evaluation of language models trained on **Moroccan Arabic** Wikipedia data.
+We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.
+
+---
+## 1. Tokenizer Evaluation
+
+![Tokenizer Compression](visualizations/01_tokenizer_compression.png)
+
+### Results
+
+| Vocab Size | Compression | Avg Token Len | UNK Rate | Total Tokens |
+|------------|-------------|---------------|----------|--------------|
+| **8k** | 3.134x | 3.09 | 0.0472% | 379,309 |
+| **16k** | 3.346x | 3.30 | 0.0504% | 355,311 |
+| **32k** | 3.535x | 3.49 | 0.0532% | 336,296 |
+| **64k** | 3.683x 🏆 | 3.64 | 0.0555% | 322,761 |
+
+### Tokenization Examples
+
+Below are sample sentences tokenized with each vocabulary size:
+
+**Sample 1:** `لّبسة لْجوية د لْغطيس (ب نݣليزية Atmospheric diving suit) هوّا لبسة ل شخص واحد ك...`
+
+| Vocab | Tokens | Count |
+|-------|--------|-------|
+| 8k | `▁لّ ب سة ▁لْ ج وية ▁د ▁لْ غط يس ... (+44 more)` | 54 |
+| 16k | `▁لّ ب سة ▁لْ ج وية ▁د ▁لْ غط يس ... (+38 more)` | 48 |
+| 32k | `▁لّ ب سة ▁لْ ج وية ▁د ▁لْ غط يس ... (+35 more)` | 45 |
+| 64k | `▁لّبسة ▁لْج وية ▁د ▁لْ غط يس ▁( ب ▁نݣليزية ... (+30 more)` | 40 |
+
+**Sample 2:** `هادي صفحة د التوضيح، ناصر عربية سمية د دكر. هادو شخصيات سميتهوم ناصر:
+ ناصر لارݣ...`
+
+| Vocab | Tokens | Count |
+|-------|--------|-------|
+| 8k | `▁هادي ▁صفحة ▁د ▁التوضيح ، ▁ن اصر ▁عربية ▁سمية ▁د ... (+33 more)` | 43 |
+| 16k | `▁هادي ▁صفحة ▁د ▁التوضيح ، ▁ناصر ▁عربية ▁سمية ▁د ▁دكر ... (+24 more)` | 34 |
+| 32k | `▁هادي ▁صفحة ▁د ▁التوضيح ، ▁ناصر ▁عربية ▁سمية ▁د ▁دكر ... (+19 more)` | 29 |
+| 64k | `▁هادي ▁صفحة ▁د ▁التوضيح ، ▁ناصر ▁عربية ▁سمية ▁د ▁دكر ... (+19 more)` | 29 |
+
+**Sample 3:** `اتحاد سلا هي فرقة مغريبية من مدينة سلا. تأسسات فـ1937، وكتلعب فتيران حي الرحمة ا...`
+
+| Vocab | Tokens | Count |
+|-------|--------|-------|
+| 8k | `▁اتحاد ▁سلا ▁هي ▁فرقة ▁مغريبية ▁من ▁مدينة ▁سلا . ▁تأسسات ... (+30 more)` | 40 |
+| 16k | `▁اتحاد ▁سلا ▁هي ▁فرقة ▁مغريبية ▁من ▁مدينة ▁سلا . ▁تأسسات ... (+28 more)` | 38 |
+| 32k | `▁اتحاد ▁سلا ▁هي ▁فرقة ▁مغريبية ▁من ▁مدينة ▁سلا . ▁تأسسات ... (+28 more)` | 38 |
+| 64k | `▁اتحاد ▁سلا ▁هي ▁فرقة ▁مغريبية ▁من ▁مدينة ▁سلا . ▁تأسسات ... (+27 more)` | 37 |
+
+
+### Key Findings
+
+- **Best Compression:** 64k achieves 3.683x compression
+- **Lowest UNK Rate:** 8k with 0.0472% unknown tokens
+- **Trade-off:** Larger vocabularies improve compression but increase model size
+- **Recommendation:** 32k vocabulary provides optimal balance for production use
+
+---
+## 2. N-gram Model Evaluation
+
+![N-gram Perplexity](visualizations/05_ngram_perplexity.png)
+
+![N-gram Coverage](visualizations/07_ngram_coverage.png)
+
+### Results
+
+| N-gram | Perplexity | Entropy | Unique N-grams | Top-100 Coverage | Top-1000 Coverage |
+|--------|------------|---------|----------------|------------------|-------------------|
+| **2-gram** | 7,187 🏆 | 12.81 | 56,749 | 24.4% | 53.2% |
+| **2-gram** | 486 🏆 | 8.93 | 6,227 | 54.9% | 95.4% |
+| **3-gram** | 8,812 | 13.11 | 76,888 | 21.3% | 52.8% |
+| **3-gram** | 4,295 | 12.07 | 51,256 | 22.1% | 58.7% |
+| **4-gram** | 12,168 | 13.57 | 124,859 | 20.1% | 50.4% |
+| **4-gram** | 22,008 | 14.43 | 260,844 | 12.0% | 35.5% |
+
+### Top 5 N-grams by Size
+
+**2-grams:**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `تصنيف :` | 37,187 |
+| 2 | `، و` | 18,746 |
+| 3 | `ن ّ` | 10,639 |
+| 4 | `) :` | 10,185 |
+| 5 | `مصادر تصنيف` | 10,087 |
+
+**3-grams:**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `مصادر تصنيف :` | 10,087 |
+| 2 | `تصنيف : مقالات` | 7,001 |
+| 3 | `ن ّ اس` | 6,981 |
+| 4 | `ل ّ ي` | 6,914 |
+| 5 | `: د��ار ف` | 5,007 |
+
+**4-grams:**
+
+| Rank | N-gram | Count |
+|------|--------|-------|
+| 1 | `تصنيف : دوار ف` | 5,005 |
+| 2 | `نسبة ن ّ اس` | 4,061 |
+| 3 | `. مصادر تصنيف :` | 3,827 |
+| 4 | `تصنيف : مقالات زادهوم` | 3,506 |
+| 5 | `: مقالات زادهوم داريجابوت` | 3,506 |
+
+
+### Key Findings
+
+- **Best Perplexity:** 2-gram with 486
+- **Entropy Trend:** Decreases with larger n-grams (more predictable)
+- **Coverage:** Top-1000 patterns cover ~35% of corpus
+- **Recommendation:** 4-gram or 5-gram for best predictive performance
+
+---
+## 3. Markov Chain Evaluation
+
+![Markov Entropy](visualizations/09_markov_entropy.png)
+
+![Markov Branching](visualizations/10_markov_branching.png)
+
+### Results
+
+| Context | Avg Entropy | Perplexity | Branching Factor | Unique Contexts | Predictability |
+|---------|-------------|------------|------------------|-----------------|----------------|
+| **1** | 0.7813 | 1.719 | 5.36 | 189,320 | 21.9% |
+| **1** | 1.1519 | 2.222 | 8.71 | 1,931 | 0.0% |
+| **2** | 0.2761 | 1.211 | 1.68 | 1,014,676 | 72.4% |
+| **2** | 0.9863 | 1.981 | 6.24 | 16,826 | 1.4% |
+| **3** | 0.0931 | 1.067 | 1.18 | 1,701,309 | 90.7% |
+| **3** | 0.8744 | 1.833 | 4.33 | 104,928 | 12.6% |
+| **4** | 0.0366 🏆 | 1.026 | 1.07 | 2,000,181 | 96.3% |
+| **4** | 0.6731 🏆 | 1.594 | 2.82 | 454,694 | 32.7% |
+
+### Generated Text Samples
+
+Below are text samples generated from each Markov chain model:
+
+**Context Size 1:**
+
+1. `. والفيلم لاخر ف جماعة قروية ف الدور د أدالت سويم " . a . بمعنى`
+2. `، و الجاج و 3000 بايت تصنيف : جهة سوس تصنيف : منتج ؤ بني ملال`
+3. `ف خمس سنين ، gallagher , blaine d ݣروپ c . ديليم د لعمر عند الجواج`
+
+**Context Size 2:**
+
+1. `تصنيف : بلايص مسكونين ف إقليم تاونات تصنيف : زيادة 1564 تصنيف : عوام د تقويم لميلادي`
+2. `، و نسبة د الناس النشيطين ( اللي سموها العرب بـالنكبة . الدعم الجوي : تم ختيارو`
+3. `ن ّ اس ل ّ خر د لعام تصنيف : سياسي مغريبي ، من بعد ، مشا`
+
+**Context Size 3:**
+
+1. `مصادر تصنيف : فيلسوف روماني قديم تصنيف : كاتب ألماني تصنيف : رياضيين من أصل مغريبي تصنيف :`
+2. `تصنيف : مقالات فيها مصدر و 3000 بايت تصنيف : ناس حيين تصنيف : سياسي لامنتامي تصنيف :`
+3. `ن ّ اس اللي خدامين ف لپريڤي ( أولا البيطاليين اللي سبق ليهوم خدمو ) : 2 ,`
+
+**Context Size 4:**
+
+1. `تصنيف : دوار ف إقليم تارودانت تصنيف : مقالات زادهوم داريجابوت تصنيف : تاريخ ديال دابا تصنيف : لقرون`
+2. `نسبة ن ّ اس ن ّ شيطين ( ل ّ ي يقدرو يخدمو ) : 48 % نسبة لبطالة`
+3. `. مصادر تصنيف : تقويم تصنيف : لقرون تصنيف : لألفيات تصنيف : مقالات فيها مصدر و 3000 بايت`
+
+
+### Key Findings
+
+- **Best Predictability:** Context-4 with 96.3% predictability
+- **Branching Factor:** Decreases with context size (more deterministic)
+- **Memory Trade-off:** Larger contexts require more storage (454,694 contexts)
+- **Recommendation:** Context-3 or Context-4 for text generation
+
+---
+## 4. Vocabulary Analysis
+
+![Zipf's Law](visualizations/12_zipf_law.png)
+
+![Top Words](visualizations/14_top20_words.png)
+
+![Coverage Curve](visualizations/15_vocab_coverage.png)
+
+### Statistics
+
+| Metric | Value |
+|--------|-------|
+| Vocabulary Size | 81,712 |
+| Total Tokens | 2,308,873 |
+| Mean Frequency | 28.26 |
+| Median Frequency | 4 |
+| Frequency Std Dev | 559.90 |
+
+### Most Common Words
+
+| Rank | Word | Frequency |
+|------|------|-----------|
+| 1 | ف | 84,463 |
+| 2 | د | 69,201 |
+| 3 | و | 61,463 |
+| 4 | تصنيف | 37,231 |
+| 5 | ل | 34,076 |
+| 6 | ديال | 32,761 |
+| 7 | من | 29,612 |
+| 8 | على | 19,717 |
+| 9 | لي | 18,627 |
+| 10 | ب | 18,189 |
+
+### Least Common Words (from vocabulary)
+
+| Rank | Word | Frequency |
+|------|------|-----------|
+| 1 | بيتسي | 2 |
+| 2 | وصانعي | 2 |
+| 3 | وأهميتها | 2 |
+| 4 | بورديو | 2 |
+| 5 | بلومر | 2 |
+| 6 | مقترحة | 2 |
+| 7 | anchor | 2 |
+| 8 | الرسميةاللي | 2 |
+| 9 | بعصبة | 2 |
+| 10 | ماڭي | 2 |
+
+### Zipf's Law Analysis
+
+| Metric | Value |
+|--------|-------|
+| Zipf Coefficient | 1.0380 |
+| R² (Goodness of Fit) | 0.999162 |
+| Adherence Quality | **excellent** |
+
+### Coverage Analysis
+
+| Top N Words | Coverage |
+|-------------|----------|
+| Top 100 | 39.3% |
+| Top 1,000 | 63.8% |
+| Top 5,000 | 78.6% |
+| Top 10,000 | 84.8% |
+
+### Key Findings
+
+- **Zipf Compliance:** R²=0.9992 indicates excellent adherence to Zipf's law
+- **High Frequency Dominance:** Top 100 words cover 39.3% of corpus
+- **Long Tail:** 71,712 words needed for remaining 15.2% coverage
+
+---
+## 5. Word Embeddings Evaluation
+
+![Embedding Isotropy](visualizations/16_embedding_isotropy.png)
+
+![Similarity Matrix](visualizations/18_embedding_similarity.png)
+
+![t-SNE Words](visualizations/20_tsne_words.png)
+
+![t-SNE Sentences](visualizations/21_tsne_sentences.png)
+
+### Model Comparison
+
+| Model | Vocab Size | Dimension | Avg Norm | Std Norm | Isotropy |
+|-------|------------|-----------|----------|----------|----------|
+| **mono_32d** | 37,528 | 32 | 4.010 | 1.183 | 0.8264 🏆 |
+| **mono_64d** | 37,528 | 64 | 4.579 | 1.040 | 0.8183 |
+| **mono_128d** | 37,528 | 128 | 5.112 | 0.875 | 0.7212 |
+| **embeddings_enhanced** | 0 | 0 | 0.000 | 0.000 | 0.0000 |
+
+### Key Findings
+
+- **Best Isotropy:** mono_32d with 0.8264 (more uniform distribution)
+- **Dimension Trade-off:** Higher dimensions capture more semantics but reduce isotropy
+- **Vocabulary Coverage:** All models cover 37,528 words
+- **Recommendation:** 100d for balanced semantic capture and efficiency
+
+---
+## 6. Summary & Recommendations
+
+![Performance Dashboard](visualizations/24_performance_dashboard.png)
+
+### Production Recommendations
+
+| Component | Recommended | Rationale |
+|-----------|-------------|-----------|
+| Tokenizer | **32k BPE** | Best compression (3.68x) with low UNK rate |
+| N-gram | **5-gram** | Lowest perplexity (486) |
+| Markov | **Context-4** | Highest predictability (96.3%) |
+| Embeddings | **100d** | Balanced semantic capture and isotropy |
+
+---
+## Appendix: Metrics Glossary & Interpretation Guide
+
+This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report.
+
+### Tokenizer Metrics
+
+**Compression Ratio**
+> *Definition:* The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text.
+>
+> *Intuition:* Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average.
+>
+> *What to seek:* Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information.
+
+**Average Token Length (Fertility)**
+> *Definition:* Mean number of characters per token produced by the tokenizer.
+>
+> *Intuition:* Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length.
+>
+> *What to seek:* Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens.
+
+**Unknown Token Rate (OOV Rate)**
+> *Definition:* Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent.
+>
+> *Intuition:* Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences.
+>
+> *What to seek:* Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback.
+
+### N-gram Model Metrics
+
+**Perplexity**
+> *Definition:* Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction.
+>
+> *Intuition:* If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options.
+>
+> *What to seek:* Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size.
+
+**Entropy**
+> *Definition:* Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy.
+>
+> *Intuition:* High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character.
+>
+> *What to seek:* Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases.
+
+**Coverage (Top-K)**
+> *Definition:* Percentage of corpus occurrences explained by the top K most frequent n-grams.
+>
+> *Intuition:* High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage.
+>
+> *What to seek:* Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text.
+
+### Markov Chain Metrics
+
+**Average Entropy**
+> *Definition:* Mean entropy across all contexts, measuring average uncertainty in next-word prediction.
+>
+> *Intuition:* Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations).
+>
+> *What to seek:* Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions.
+
+**Branching Factor**
+> *Definition:* Average number of unique next tokens observed for each context.
+>
+> *Intuition:* High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive).
+>
+> *What to seek:* Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains.
+
+**Predictability**
+> *Definition:* Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are.
+>
+> *Intuition:* 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes.
+>
+> *What to seek:* Higher predictability for text generation quality, but too high (>98%) may produce repetitive output.
+
+### Vocabulary & Zipf's Law Metrics
+
+**Zipf's Coefficient**
+> *Definition:* The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1.
+>
+> *Intuition:* A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare.
+>
+> *What to seek:* Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text.
+
+**R² (Coefficient of Determination)**
+> *Definition:* Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1.
+>
+> *Intuition:* R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns.
+>
+> *What to seek:* R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora.
+
+**Vocabulary Coverage**
+> *Definition:* Cumulative percentage of corpus tokens accounted for by the top N words.
+>
+> *Intuition:* Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words.
+>
+> *What to seek:* Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary.
+
+### Word Embedding Metrics
+
+**Isotropy**
+> *Definition:* Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values.
+>
+> *Intuition:* High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness.
+>
+> *What to seek:* Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy.
+
+**Average Norm**
+> *Definition:* Mean magnitude (L2 norm) of word vectors in the embedding space.
+>
+> *Intuition:* Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained.
+>
+> *What to seek:* Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation).
+
+**Cosine Similarity**
+> *Definition:* Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction).
+>
+> *Intuition:* Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings.
+>
+> *What to seek:* Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7.
+
+**t-SNE Visualization**
+> *Definition:* t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization.
+>
+> *Intuition:* Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence.
+>
+> *What to seek:* Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure.
+
+### General Interpretation Guidelines
+
+1. **Compare within model families:** Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer).
+2. **Consider trade-offs:** Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate).
+3. **Context matters:** Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification.
+4. **Corpus influence:** All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature.
+5. **Language-specific patterns:** Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages.
+
+
+### Visualizations Index
+
+| # | Visualization | Description |
+|---|---------------|-------------|
+| 01 | Tokenizer Compression | Compression ratios by vocabulary size |
+| 02 | Tokenizer Fertility | Average token length by vocabulary |
+| 03 | Tokenizer OOV | Unknown token rates |
+| 04 | Tokenizer Tokens | Total tokens by vocabulary |
+| 05 | N-gram Perplexity | Perplexity by n-gram size |
+| 06 | N-gram Entropy | Entropy by n-gram size |
+| 07 | N-gram Coverage | Top pattern coverage |
+| 08 | N-gram Unique | Unique n-gram counts |
+| 09 | Markov Entropy | Entropy by context size |
+| 10 | Markov Branching | Branching factor by context |
+| 11 | Markov Contexts | Unique context counts |
+| 12 | Zipf's Law | Frequency-rank distribution with fit |
+| 13 | Vocab Frequency | Word frequency distribution |
+| 14 | Top 20 Words | Most frequent words |
+| 15 | Vocab Coverage | Cumulative coverage curve |
+| 16 | Embedding Isotropy | Vector space uniformity |
+| 17 | Embedding Norms | Vector magnitude distribution |
+| 18 | Similarity Matrix | Word similarity heatmap |
+| 19 | Nearest Neighbors | Similar words for key terms |
+| 20 | t-SNE Words | 2D word embedding visualization |
+| 21 | t-SNE Sentences | 2D sentence embedding visualization |
+| 22 | Position Encoding | Encoding method comparison |
+| 23 | Model Sizes | Storage requirements |
+| 24 | Dashboard | Comprehensive performance overview |
+
+---
+## About This Project
+
+### Data Source
+
+Models trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) - a monthly snapshot of Wikipedia articles across 300+ languages.
+
+### Project
+
+A project by **[Wikilangs](https://wikilangs.org)** - Open-source NLP models for every Wikipedia language.
+
+### Maintainer
+
+[Omar Kamali](https://omarkamali.com) - [Omneity Labs](https://omneitylabs.com)
+
+### Citation
+
+If you use these models in your research, please cite:
+
+```bibtex
+@misc{wikilangs2025,
+  author = {Kamali, Omar},
+  title = {Wikilangs: Open NLP Models for Wikipedia Languages},
+  year = {2025},
+  publisher = {HuggingFace},
+  url = {https://huggingface.co/wikilangs}
+  institution = {Omneity Labs}
+}
+```
+
+### License
+
+MIT License - Free for academic and commercial use.
+
+### Links
+
+- 🌐 Website: [wikilangs.org](https://wikilangs.org)
+- 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
+- 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
+---
+*Generated by Wikilangs Models Pipeline*
+
+*Report Date: 2025-12-27 04:02:58*

models/embeddings/monolingual/ary_128d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5cc5293cf8429fb5399c588bd22cb8909420d94a23901f2c928f46ae56e690f
+size 1063209927

models/embeddings/monolingual/ary_128d.meta.json

@@ -0,0 +1 @@
+{"lang": "ary", "dim": 128, "max_seq_len": 512, "is_aligned": false}

models/embeddings/monolingual/ary_128d_metadata.json

@@ -0,0 +1,13 @@
+{
+  "language": "ary",
+  "dimension": 128,
+  "version": "monolingual",
+  "training_params": {
+    "dim": 128,
+    "min_count": 5,
+    "window": 5,
+    "negative": 5,
+    "epochs": 5
+  },
+  "vocab_size": 37528
+}

models/embeddings/monolingual/ary_32d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2f8b934f3ef0b52a7ad968cdbdcf1f979c0b7a075cbb32fa213a893a5c78d2f
+size 266388423

models/embeddings/monolingual/ary_32d.meta.json

@@ -0,0 +1 @@
+{"lang": "ary", "dim": 32, "max_seq_len": 512, "is_aligned": false}

models/embeddings/monolingual/ary_32d_metadata.json

@@ -0,0 +1,13 @@
+{
+  "language": "ary",
+  "dimension": 32,
+  "version": "monolingual",
+  "training_params": {
+    "dim": 32,
+    "min_count": 5,
+    "window": 5,
+    "negative": 5,
+    "epochs": 5
+  },
+  "vocab_size": 37528
+}

models/embeddings/monolingual/ary_64d.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83735f6b6acbb5e848400813f89e7a33d7fc8d847ad66d17693f0e0573ec6fd0
+size 531995591

models/embeddings/monolingual/ary_64d.meta.json

@@ -0,0 +1 @@
+{"lang": "ary", "dim": 64, "max_seq_len": 512, "is_aligned": false}

models/embeddings/monolingual/ary_64d_metadata.json

@@ -0,0 +1,13 @@
+{
+  "language": "ary",
+  "dimension": 64,
+  "version": "monolingual",
+  "training_params": {
+    "dim": 64,
+    "min_count": 5,
+    "window": 5,
+    "negative": 5,
+    "epochs": 5
+  },
+  "vocab_size": 37528
+}

models/subword_markov/ary_markov_ctx1_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02d6d2bdc8d7dbce6c10868f4220e2d2d66cf1a06f3491e669371f1b65129d05
+size 127609

models/subword_markov/ary_markov_ctx1_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "context_size": 1,
+  "variant": "subword",
+  "language": "ary",
+  "unique_contexts": 1931,
+  "total_transitions": 13216554
+}

models/subword_markov/ary_markov_ctx2_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:322a9496ec76e3be61fdafb53d8b77843c56708cd0dbe0e62dcd5edd91548627
+size 835900

models/subword_markov/ary_markov_ctx2_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "context_size": 2,
+  "variant": "subword",
+  "language": "ary",
+  "unique_contexts": 16826,
+  "total_transitions": 13205631
+}

models/subword_markov/ary_markov_ctx3_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d5187cefb9bc3bda8ef32e536676e0c3b42f01c2c61bed6abaeb131d627b820
+size 3229260

models/subword_markov/ary_markov_ctx3_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "context_size": 3,
+  "variant": "subword",
+  "language": "ary",
+  "unique_contexts": 104928,
+  "total_transitions": 13194708
+}

models/subword_markov/ary_markov_ctx4_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd8baa92bb2fa6351748ab87b35f8ba19922364e0eae3c6cbd772e527604f163
+size 10673093

models/subword_markov/ary_markov_ctx4_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "context_size": 4,
+  "variant": "subword",
+  "language": "ary",
+  "unique_contexts": 454694,
+  "total_transitions": 13183785
+}

models/subword_ngram/ary_2gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62a74b4e9a08aa132c61be02cbdead1b65df56d23212d4d13b44eee1de72ca5c
+size 85260

models/subword_ngram/ary_2gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 2,
+  "variant": "subword",
+  "language": "ary",
+  "unique_ngrams": 6227,
+  "total_ngrams": 13216554
+}

models/subword_ngram/ary_3gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33a4ba5238a6f6fb79055b4420c426248fbd61bc3060e6af9de75c84a51cb747
+size 664928

models/subword_ngram/ary_3gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 3,
+  "variant": "subword",
+  "language": "ary",
+  "unique_ngrams": 51256,
+  "total_ngrams": 13205631
+}

models/subword_ngram/ary_4gram_subword.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a9c53578c3475bc96d000d9db9a4d19e545b73cbbe4948d2149968e06900585
+size 3262126

models/subword_ngram/ary_4gram_subword_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 4,
+  "variant": "subword",
+  "language": "ary",
+  "unique_ngrams": 260844,
+  "total_ngrams": 13194708
+}

models/tokenizer/ary_tokenizer_16k.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66f08427ec3757387ee07eb1bbb3518ed56da42a7b4a144381b3c9d0e2a75fd2
+size 550569

models/tokenizer/ary_tokenizer_32k.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dfbe42ab44ef4ba9c324b96993e42805c7a7def9a21e8af3f8467142eef0b615
+size 880065

models/tokenizer/ary_tokenizer_64k.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46f721a26f1f518a4334f4a8844591eb94957318f9a1d14d80668720663635cc
+size 1546284

models/tokenizer/ary_tokenizer_8k.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:21bb3111788de7f71f160230b53c90faec9e158982f1fc3936144c7bb95bc0a1
+size 391386

models/vocabulary/ary_vocabulary.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e9e3400acc6760b029fbf22dea48ff18cbf62eda489e8d001ff473945dde76d
+size 1446167

models/vocabulary/ary_vocabulary_metadata.json

@@ -0,0 +1,16 @@
+{
+  "language": "ary",
+  "vocabulary_size": 81712,
+  "statistics": {
+    "type_token_ratio": 0.07826179445772023,
+    "coverage": {
+      "top_100": 0.3755263295122797,
+      "top_1000": 0.6095531030989189,
+      "top_5000": 0.7510592808230233,
+      "top_10000": 0.8101195151850255
+    },
+    "hapax_count": 107389,
+    "hapax_ratio": 0.5678922903633508,
+    "total_documents": 10923
+  }
+}

models/word_markov/ary_markov_ctx1_word.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf369ad231704eef182a92e5d64241ae5294d89bedc2393e46ffa50e68be61c3
+size 10974758

models/word_markov/ary_markov_ctx1_word_metadata.json

@@ -0,0 +1,7 @@
+{
+  "context_size": 1,
+  "variant": "word",
+  "language": "ary",
+  "unique_contexts": 189320,
+  "total_transitions": 3013648
+}

models/word_markov/ary_markov_ctx2_word.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6feb87132c6a3a8881dab7c3af6268a3da4e594d21311b665e71b2e95a0af000
+size 28263486

models/word_markov/ary_markov_ctx2_word_metadata.json

@@ -0,0 +1,7 @@
+{
+  "context_size": 2,
+  "variant": "word",
+  "language": "ary",
+  "unique_contexts": 1014676,
+  "total_transitions": 3002725
+}

models/word_markov/ary_markov_ctx3_word.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:354a3b1b04e0bdb94e0c7a15c339df347f9c313c580fe88c039f721a8a1e4237
+size 41576319

models/word_markov/ary_markov_ctx3_word_metadata.json

@@ -0,0 +1,7 @@
+{
+  "context_size": 3,
+  "variant": "word",
+  "language": "ary",
+  "unique_contexts": 1701309,
+  "total_transitions": 2991803
+}

models/word_markov/ary_markov_ctx4_word.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5cc0395cfc1cc580368483ee202676a9d7b57cdbdf737c5c7422dfd39ecdf963
+size 50573861

models/word_markov/ary_markov_ctx4_word_metadata.json

@@ -0,0 +1,7 @@
+{
+  "context_size": 4,
+  "variant": "word",
+  "language": "ary",
+  "unique_contexts": 2000181,
+  "total_transitions": 2980882
+}

models/word_ngram/ary_2gram_word.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60dc5550877fc074d5df5a925b63f1f3975f796566e9a5a5de18357cd8485faa
+size 1017637

models/word_ngram/ary_2gram_word_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 2,
+  "variant": "word",
+  "language": "ary",
+  "unique_ngrams": 56749,
+  "total_ngrams": 3013648
+}

models/word_ngram/ary_3gram_word.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:450d7e949b8c457d9c72e4207cc5a90c1e77b46a5e5091df8c971c29082d1978
+size 1493910

models/word_ngram/ary_3gram_word_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 3,
+  "variant": "word",
+  "language": "ary",
+  "unique_ngrams": 76888,
+  "total_ngrams": 3002725
+}

models/word_ngram/ary_4gram_word.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ee2009d481c48c5635cbb99107a721e89fb3733f2698b8960ac7a70030bbabc
+size 2616301

models/word_ngram/ary_4gram_word_metadata.json

@@ -0,0 +1,7 @@
+{
+  "n": 4,
+  "variant": "word",
+  "language": "ary",
+  "unique_ngrams": 124859,
+  "total_ngrams": 2991803
+}