Update model_comparison_report.md

model_comparison_report.md

@@ -0,0 +1,97 @@
+# Tatar2Vec Model Comparison Report
+
+**Date:** 2026-03-04
+**Author:** Mullosharaf K. Arabov
+
+## Overview
+
+This report presents a comprehensive comparison of five word embedding models trained for the Tatar language:
+- Word2Vec CBOW (100 and 200 dimensions)
+- Word2Vec Skip-gram (100 dimensions)
+- FastText CBOW (100 and 200 dimensions)
+
+## Evaluation Methodology
+
+### Tasks
+1. **Word Analogies** – 5 semantic analogy pairs
+2. **Semantic Similarity** – Cosine similarity on 8 word pairs
+3. **OOV Handling** – Testing with 6 morphologically complex words
+4. **Nearest Neighbours** – Qualitative inspection
+5. **Dimensionality Reduction** – PCA visualization
+
+## Detailed Results
+
+### 1. Word Analogies
+
+| Model | Accuracy | Details |
+|-------|----------|---------|
+| w2v_cbow_100 | 60% | ✓ Moscow:Russia = Kazan:Tatarstan<br>✓ teacher:school = doctor:hospital<br>✓ father:mother = grandfather:grandmother |
+| All FastText | 0% | Failed on all analogy tasks |
+
+### 2. Semantic Similarity
+
+| Word Pair | Word2Vec (cbow100) | FastText (cbow100) |
+|-----------|-------------------|-------------------|
+| Казан-Мәскәү | 0.777 | 0.736 |
+| татар-башкорт | 0.793 | 0.823 |
+| мәктәп-университет | 0.565 | 0.621 |
+| укытучы-укучы | 0.742 | 0.771 |
+| **Average** | **0.568** | **0.582** |
+
+### 3. Nearest Neighbours Analysis
+
+#### Word2Vec (cbow100) – Clean semantic neighbours
+```
+татар → Татар, башкорт, урыс, татарның, рус
+Казан → Мәскәү, Чаллы, Алабуга, Чистай, Уфа
+```
+
+#### FastText (cbow100) – Noisy neighbours with punctuation
+```
+татар → милләттатар, дтатар, —татар, –татар, Ттатар
+Казан → »Казан, –Казан, .Казан, )Казан, -Казан
+```
+
+### 4. Training Efficiency
+
+| Model | Training Time | Vocabulary Size |
+|-------|--------------|-----------------|
+| w2v_cbow_100 | 1760s (29min) | 1,293,992 |
+| ft_cbow_100 | 3323s (55min) | 1,293,992 |
+
+## Key Findings
+
+1. **Word2Vec CBOW (100-dim)** is the best overall model, particularly strong on semantic analogies (60% accuracy)
+
+2. **FastText models** show slightly better raw similarity scores but suffer from noisy representations (punctuation artifacts)
+
+3. **All models** achieve 100% vocabulary coverage
+
+4. **Word2Vec trains almost 2x faster** than FastText
+
+## Recommendations
+
+| Use Case | Recommended Model |
+|----------|-------------------|
+| Semantic similarity, analogies | **w2v_cbow_100** |
+| Morphological analysis | ft_cbow_100 |
+| Maximum precision (if memory allows) | w2v_cbow_200 |
+| Rare word handling | ft_cbow_100 |
+
+## Visualizations
+
+### PCA Projection (Word2Vec)
+*[Here you can add a screenshot of the PCA plot]*
+
+### PCA Projection (FastText)
+*[Here you can add a screenshot of the PCA plot]*
+
+## Conclusion
+
+**Winner: Word2Vec CBOW (100 dimensions)**
+
+Despite FastText's slightly better semantic similarity scores, Word2Vec produces cleaner, more interpretable embeddings and significantly outperforms on analogy tasks. It is recommended as the default choice for most Tatar NLP applications.
+
+---
+
+*This report was generated automatically during model evaluation.*