codemalt / REPORT.md

Sarthak

chore: update README and REPORT with performance insights and dataset changes

0dbb356 8 months ago

17 kB

	# Code-Specialized Model2Vec Distillation Analysis

	## 🎯 Executive Summary

	This report presents a comprehensive analysis of Model2Vec distillation experiments using different teacher models for code-specialized embedding generation.

	### Evaluated Models Overview

	Simplified Distillation Models: 14
	Peer Comparison Models: 19
	Total Models Analyzed: 33

	### Best Performing Simplified Model: code_model2vec_all_mpnet_base_v2

	Overall CodeSearchNet Performance:
	- NDCG@10: 0.7387
	- Mean Reciprocal Rank (MRR): 0.7010
	- Recall@5: 0.8017
	- Mean Rank: 6.4

	## 📊 Comprehensive Model Comparison

	### All Simplified Distillation Models Performance

	\| Model \| Teacher \| NDCG@10 \| MRR \| Recall@5 \| Status \|
	\|-------\|---------\|---------\|-----\|----------\|--------\|
	\| code_model2vec_all_mpnet_base_v2 \| [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) \| 0.7387 \| 0.7010 \| 0.8017 \| 🥇 Best \|
	\| code_model2vec_all_MiniLM_L6_v2 \| [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) \| 0.7385 \| 0.7049 \| 0.7910 \| 🥈 2nd \|
	\| code_model2vec_jina_embeddings_v2_base_code \| [jina-embeddings-v2-base-code](https://huggingface.co/jina-embeddings-v2-base-code) \| 0.7381 \| 0.6996 \| 0.8130 \| 🥉 3rd \|
	\| code_model2vec_paraphrase_MiniLM_L6_v2 \| [sentence-transformers/paraphrase-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2) \| 0.7013 \| 0.6638 \| 0.7665 \| #4 \|
	\| code_model2vec_Reason_ModernColBERT \| [lightonai/Reason-ModernColBERT](https://huggingface.co/lightonai/Reason-ModernColBERT) \| 0.6598 \| 0.6228 \| 0.7260 \| #5 \|
	\| code_model2vec_all_mpnet_base_v2_fine_tuned \| [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) \| 0.6147 \| 0.5720 \| 0.6950 \| #6 \|
	\| code_model2vec_bge_m3 \| [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) \| 0.4863 \| 0.4439 \| 0.5514 \| #7 \|
	\| code_model2vec_jina_embeddings_v3 \| [jinaai/jina-embeddings-v3](https://huggingface.co/jinaai/jina-embeddings-v3) \| 0.4755 \| 0.4416 \| 0.5456 \| #8 \|
	\| code_model2vec_nomic_embed_text_v2_moe \| [nomic-ai/nomic-embed-text-v2-moe](https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe) \| 0.4532 \| 0.4275 \| 0.5094 \| #9 \|
	\| code_model2vec_gte_Qwen2_1.5B_instruct \| [Alibaba-NLP/gte-Qwen2-1.5B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct) \| 0.4238 \| 0.3879 \| 0.4719 \| #10 \|
	\| code_model2vec_Qodo_Embed_1_1.5B \| [Qodo/Qodo-Embed-1-1.5B](https://huggingface.co/Qodo/Qodo-Embed-1-1.5B) \| 0.4101 \| 0.3810 \| 0.4532 \| #11 \|
	\| code_model2vec_graphcodebert_base \| [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base) \| 0.3420 \| 0.3140 \| 0.3704 \| #12 \|
	\| code_model2vec_Linq_Embed_Mistral \| [Linq-AI-Research/Linq-Embed-Mistral](https://huggingface.co/Linq-AI-Research/Linq-Embed-Mistral) \| 0.2868 \| 0.2581 \| 0.3412 \| #13 \|
	\| code_model2vec_codebert_base \| [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base) \| 0.2779 \| 0.2534 \| 0.3136 \| #14 \|


	### 📊 Model Specifications Analysis

	Our distilled models exhibit consistent architectural characteristics across different teacher models:

	\| Model \| Vocabulary Size \| Parameters \| Embedding Dim \| Disk Size \|
	\|-------\|----------------\|------------\|---------------\|-----------\|
	\| all_mpnet_base_v2 \| 29,528 \| 7.6M \| 256 \| 14.4MB \|
	\| all_MiniLM_L6_v2 \| 29,525 \| 7.6M \| 256 \| 14.4MB \|
	\| jina_embeddings_v2_base_code \| 61,053 \| 15.6M \| 256 \| 29.8MB \|
	\| paraphrase_MiniLM_L6_v2 \| 29,525 \| 7.6M \| 256 \| 14.4MB \|
	\| Reason_ModernColBERT \| 50,254 \| 12.9M \| 256 \| 24.5MB \|
	\| all_mpnet_base_v2_fine_tuned \| 36,624 \| 9.4M \| 256 \| 35.8MB \|
	\| bge_m3 \| 249,999 \| 64.0M \| 256 \| 122.1MB \|
	\| jina_embeddings_v3 \| 249,999 \| 64.0M \| 256 \| 122.1MB \|
	\| nomic_embed_text_v2_moe \| 249,999 \| 64.0M \| 256 \| 122.1MB \|
	\| gte_Qwen2_1.5B_instruct \| 151,644 \| 38.8M \| 256 \| 74.0MB \|
	\| Qodo_Embed_1_1.5B \| 151,644 \| 38.8M \| 256 \| 74.0MB \|
	\| graphcodebert_base \| 50,262 \| 12.9M \| 256 \| 24.5MB \|
	\| Linq_Embed_Mistral \| 31,999 \| 8.2M \| 256 \| 15.6MB \|
	\| codebert_base \| 50,262 \| 12.9M \| 256 \| 24.5MB \|


	![Model Specifications](analysis_charts/model_specifications.png)

	Comprehensive analysis of our distilled models showing vocabulary size, parameter count, embedding dimensions, and storage requirements.

	#### Key Insights from Model Specifications:


	- Vocabulary Consistency: All models use vocabulary sizes ranging from 29,525 to 249,999 tokens (avg: 101,594)
	- Parameter Efficiency: Models range from 7.6M to 64.0M parameters (avg: 26.0M)
	- Storage Efficiency: Disk usage ranges from 14.4MB to 122.1MB (avg: 50.9MB)
	- Embedding Dimensions: Consistent 256 dimensions across all models (optimized for efficiency)


	### Key Findings


	- Best Teacher Model: code_model2vec_all_mpnet_base_v2 (NDCG@10: 0.7387)
	- Least Effective Teacher: code_model2vec_codebert_base (NDCG@10: 0.2779)
	- Performance Range: 62.4% difference between best and worst
	- Average Performance: 0.5248 NDCG@10


	## 🎯 Language Performance Radar Charts

	### Best Model vs Peer Models Comparison

	![Comparative Radar Chart](analysis_charts/comparative_radar.png)

	Comparative view showing how the best simplified distillation model performs against top peer models across programming languages.

	### Individual Model Performance by Language

	#### code_model2vec_all_mpnet_base_v2 (Teacher: [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)) - NDCG@10: 0.7387

	![code_model2vec_all_mpnet_base_v2 Radar Chart](analysis_charts/radar_code_model2vec_all_mpnet_base_v2.png)

	#### code_model2vec_all_MiniLM_L6_v2 (Teacher: [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)) - NDCG@10: 0.7385

	![code_model2vec_all_MiniLM_L6_v2 Radar Chart](analysis_charts/radar_code_model2vec_all_MiniLM_L6_v2.png)

	#### code_model2vec_jina_embeddings_v2_base_code (Teacher: [jina-embeddings-v2-base-code](https://huggingface.co/jina-embeddings-v2-base-code)) - NDCG@10: 0.7381

	![code_model2vec_jina_embeddings_v2_base_code Radar Chart](analysis_charts/radar_code_model2vec_jina_embeddings_v2_base_code.png)

	#### code_model2vec_paraphrase_MiniLM_L6_v2 (Teacher: [sentence-transformers/paraphrase-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2)) - NDCG@10: 0.7013

	![code_model2vec_paraphrase_MiniLM_L6_v2 Radar Chart](analysis_charts/radar_code_model2vec_paraphrase_MiniLM_L6_v2.png)

	#### code_model2vec_Reason_ModernColBERT (Teacher: [lightonai/Reason-ModernColBERT](https://huggingface.co/lightonai/Reason-ModernColBERT)) - NDCG@10: 0.6598

	![code_model2vec_Reason_ModernColBERT Radar Chart](analysis_charts/radar_code_model2vec_Reason_ModernColBERT.png)

	#### code_model2vec_all_mpnet_base_v2_fine_tuned (Teacher: [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)) - NDCG@10: 0.6147

	![code_model2vec_all_mpnet_base_v2_fine_tuned Radar Chart](analysis_charts/radar_code_model2vec_all_mpnet_base_v2_fine_tuned.png)

	#### code_model2vec_bge_m3 (Teacher: [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3)) - NDCG@10: 0.4863

	![code_model2vec_bge_m3 Radar Chart](analysis_charts/radar_code_model2vec_bge_m3.png)

	#### code_model2vec_jina_embeddings_v3 (Teacher: [jinaai/jina-embeddings-v3](https://huggingface.co/jinaai/jina-embeddings-v3)) - NDCG@10: 0.4755

	![code_model2vec_jina_embeddings_v3 Radar Chart](analysis_charts/radar_code_model2vec_jina_embeddings_v3.png)

	#### code_model2vec_nomic_embed_text_v2_moe (Teacher: [nomic-ai/nomic-embed-text-v2-moe](https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe)) - NDCG@10: 0.4532

	![code_model2vec_nomic_embed_text_v2_moe Radar Chart](analysis_charts/radar_code_model2vec_nomic_embed_text_v2_moe.png)

	#### code_model2vec_gte_Qwen2_1.5B_instruct (Teacher: [Alibaba-NLP/gte-Qwen2-1.5B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct)) - NDCG@10: 0.4238

	![code_model2vec_gte_Qwen2_1.5B_instruct Radar Chart](analysis_charts/radar_code_model2vec_gte_Qwen2_15B_instruct.png)

	#### code_model2vec_Qodo_Embed_1_1.5B (Teacher: [Qodo/Qodo-Embed-1-1.5B](https://huggingface.co/Qodo/Qodo-Embed-1-1.5B)) - NDCG@10: 0.4101

	![code_model2vec_Qodo_Embed_1_1.5B Radar Chart](analysis_charts/radar_code_model2vec_Qodo_Embed_1_15B.png)

	#### code_model2vec_graphcodebert_base (Teacher: [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base)) - NDCG@10: 0.3420

	![code_model2vec_graphcodebert_base Radar Chart](analysis_charts/radar_code_model2vec_graphcodebert_base.png)

	#### code_model2vec_Linq_Embed_Mistral (Teacher: [Linq-AI-Research/Linq-Embed-Mistral](https://huggingface.co/Linq-AI-Research/Linq-Embed-Mistral)) - NDCG@10: 0.2868

	![code_model2vec_Linq_Embed_Mistral Radar Chart](analysis_charts/radar_code_model2vec_Linq_Embed_Mistral.png)

	#### code_model2vec_codebert_base (Teacher: [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base)) - NDCG@10: 0.2779

	![code_model2vec_codebert_base Radar Chart](analysis_charts/radar_code_model2vec_codebert_base.png)



	## 🏆 Peer Model Comparison

	![Peer Comparison](analysis_charts/peer_comparison.png)

	Comparison with established code-specialized embedding models using actual evaluation results.

	### Complete Model Ranking

	\| Rank \| Model \| Type \| NDCG@10 \| MRR \| Recall@5 \|
	\|------\|-------\|------\|---------\|-----\|----------\|
	\| 1 \| Alibaba-NLP/gte-Qwen2-1.5B-instruct \| General \| 0.9729 \| 0.9676 \| 0.9825 \|
	\| 2 \| Qodo/Qodo-Embed-1-1.5B \| General \| 0.9715 \| 0.9659 \| 0.9875 \|
	\| 3 \| jina-embeddings-v2-base-code \| General \| 0.9677 \| 0.9618 \| 0.9849 \|
	\| 4 \| jinaai/jina-embeddings-v3 \| General \| 0.9640 \| 0.9573 \| 0.9839 \|
	\| 5 \| sentence-transformers/all-mpnet-base-v2 \| General \| 0.9477 \| 0.9358 \| 0.9732 \|
	\| 6 \| nomic-ai/nomic-embed-text-v2-moe \| General \| 0.9448 \| 0.9357 \| 0.9659 \|
	\| 7 \| sentence-transformers/all-MiniLM-L12-v2 \| General \| 0.9398 \| 0.9265 \| 0.9732 \|
	\| 8 \| BAAI/bge-m3 \| General \| 0.9383 \| 0.9295 \| 0.9643 \|
	\| 9 \| sentence-transformers/all-MiniLM-L6-v2 \| General \| 0.9255 \| 0.9099 \| 0.9642 \|
	\| 10 \| lightonai/Reason-ModernColBERT \| General \| 0.9188 \| 0.9036 \| 0.9486 \|
	\| 11 \| Linq-AI-Research/Linq-Embed-Mistral \| General \| 0.9080 \| 0.8845 \| 0.9650 \|
	\| 12 \| sentence-transformers/paraphrase-MiniLM-L6-v2 \| General \| 0.8297 \| 0.8016 \| 0.8828 \|
	\| 13 \| minishlab/potion-base-8M \| Model2Vec \| 0.8162 \| 0.7817 \| 0.8931 \|
	\| 14 \| minishlab/potion-retrieval-32M \| Model2Vec \| 0.8137 \| 0.7810 \| 0.8792 \|
	\| 15 \| code_model2vec_all_mpnet_base_v2 \| 🔥 Simplified Distillation \| 0.7387 \| 0.7010 \| 0.8017 \|
	\| 16 \| code_model2vec_all_MiniLM_L6_v2 \| 🔥 Simplified Distillation \| 0.7385 \| 0.7049 \| 0.7910 \|
	\| 17 \| code_model2vec_jina_embeddings_v2_base_code \| 🔥 Simplified Distillation \| 0.7381 \| 0.6996 \| 0.8130 \|
	\| 18 \| code_model2vec_paraphrase_MiniLM_L6_v2 \| 🔥 Simplified Distillation \| 0.7013 \| 0.6638 \| 0.7665 \|
	\| 19 \| code_model2vec_Reason_ModernColBERT \| 🔥 Simplified Distillation \| 0.6598 \| 0.6228 \| 0.7260 \|
	\| 20 \| code_model2vec_all_mpnet_base_v2_fine_tuned \| 🎓 Fine-tuned Distillation \| 0.6147 \| 0.5720 \| 0.6950 \|
	\| 21 \| potion-multilingual-128M \| Model2Vec \| 0.6124 \| 0.5683 \| 0.7017 \|
	\| 22 \| huggingface/CodeBERTa-small-v1 \| Code-Specific \| 0.5903 \| 0.5350 \| 0.6779 \|
	\| 23 \| Salesforce/codet5-base \| Code-Specific \| 0.4872 \| 0.4500 \| 0.5742 \|
	\| 24 \| code_model2vec_bge_m3 \| 🔥 Simplified Distillation \| 0.4863 \| 0.4439 \| 0.5514 \|
	\| 25 \| code_model2vec_jina_embeddings_v3 \| 🔥 Simplified Distillation \| 0.4755 \| 0.4416 \| 0.5456 \|
	\| 26 \| code_model2vec_nomic_embed_text_v2_moe \| 🔥 Simplified Distillation \| 0.4532 \| 0.4275 \| 0.5094 \|
	\| 27 \| code_model2vec_gte_Qwen2_1.5B_instruct \| 🔥 Simplified Distillation \| 0.4238 \| 0.3879 \| 0.4719 \|
	\| 28 \| code_model2vec_Qodo_Embed_1_1.5B \| 🔥 Simplified Distillation \| 0.4101 \| 0.3810 \| 0.4532 \|
	\| 29 \| microsoft/graphcodebert-base \| Code-Specific \| 0.4039 \| 0.3677 \| 0.4650 \|
	\| 30 \| code_model2vec_graphcodebert_base \| 🔥 Simplified Distillation \| 0.3420 \| 0.3140 \| 0.3704 \|
	\| 31 \| code_model2vec_Linq_Embed_Mistral \| 🔥 Simplified Distillation \| 0.2868 \| 0.2581 \| 0.3412 \|
	\| 32 \| code_model2vec_codebert_base \| 🔥 Simplified Distillation \| 0.2779 \| 0.2534 \| 0.3136 \|
	\| 33 \| microsoft/codebert-base \| Code-Specific \| 0.1051 \| 0.1058 \| 0.1105 \|


	## 📈 Performance Analysis

	### Multi-Model Comparison Charts

	![Model Comparison](analysis_charts/model_comparison.png)

	Comprehensive comparison across all evaluation metrics.

	### Language Performance Analysis

	![Language Heatmap](analysis_charts/language_heatmap.png)

	Performance heatmap showing how different models perform across programming languages.

	### Efficiency Analysis

	![Efficiency Analysis](analysis_charts/efficiency_analysis.png)

	Performance vs model size analysis showing the efficiency benefits of distillation.



	## ⚡ Operational Performance Analysis

	![Benchmark Performance](analysis_charts/benchmark_performance.png)

	Comprehensive performance benchmarking across multiple operational metrics.

	### Performance Scaling Analysis

	![Batch Size Scaling](analysis_charts/batch_size_scaling.png)

	How performance scales with different batch sizes for optimal throughput.

	![Memory Scaling](analysis_charts/memory_scaling.png)

	Memory usage patterns across different batch sizes.



	## 🔍 Language-Specific Analysis

	### Performance by Programming Language

	\| Language \| Best Model Performance \| Average Performance \| Language Difficulty \|
	\|----------\|------------------------\|--------------------\|--------------------\|
	\| Go \| 0.9780 \| 0.6960 \| Easy \|
	\| Java \| 0.9921 \| 0.6553 \| Easy \|
	\| Javascript \| 0.9550 \| 0.5850 \| Easy \|
	\| Php \| 1.0000 \| 0.6321 \| Easy \|
	\| Python \| 1.0000 \| 0.8623 \| Easy \|
	\| Ruby \| 0.9493 \| 0.6397 \| Easy \|


	## 🎯 Conclusions and Recommendations

	### Teacher Model Analysis

	Based on the evaluation results across all simplified distillation models:


	1. Best Teacher Model: sentence-transformers/all-MiniLM-L6-v2 (NDCG@10: 0.7385)
	2. Least Effective Teacher: microsoft/codebert-base (NDCG@10: 0.2779)
	3. Teacher Model Impact: Choice of teacher model affects performance by 62.4%

	### Recommendations

	- For Production: Use sentence-transformers/all-MiniLM-L6-v2 as teacher model for best performance
	- For Efficiency: Model2Vec distillation provides significant size reduction with competitive performance
	- For Code Tasks: Specialized models consistently outperform general-purpose models


	## 📄 Methodology

	### Evaluation Protocol
	- Dataset: CodeSearchNet test sets for 6 programming languages
	- Metrics: NDCG@k, MRR, Recall@k following CodeSearchNet methodology
	- Query Format: Natural language documentation strings
	- Corpus Format: Function code strings
	- Evaluation: Retrieval of correct code for each documentation query

	### Teacher Models Tested
	- [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) (proven baseline)
	- [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) (general purpose)
	- [sentence-transformers/paraphrase-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2) (paraphrase model)
	- [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base) (code-specialized)
	- [microsoft/graphcodebert-base](https://huggingface.co/microsoft/graphcodebert-base) (graph-aware code model)
	- [Alibaba-NLP/gte-Qwen2-1.5B-instruct](https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct) (instruction model)
	- [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) (multilingual model)
	- [jinaai/jina-embeddings-v3](https://huggingface.co/jinaai/jina-embeddings-v3) (modern embedding model)
	- [nomic-ai/nomic-embed-text-v2-moe](https://huggingface.co/nomic-ai/nomic-embed-text-v2-moe) (mixture of experts)
	- [Qodo/Qodo-Embed-1-1.5B](https://huggingface.co/Qodo/Qodo-Embed-1-1.5B) (code-specialized)
	- [lightonai/Reason-ModernColBERT](https://huggingface.co/lightonai/Reason-ModernColBERT) (ColBERT architecture)
	- [Linq-AI-Research/Linq-Embed-Mistral](https://huggingface.co/Linq-AI-Research/Linq-Embed-Mistral) (Mistral-based)
	- [BAAI/bge-code-v1](https://huggingface.co/BAAI/bge-code-v1) (code-specialized BGE)
	- [Salesforce/SFR-Embedding-Code-2B_R](https://huggingface.co/Salesforce/SFR-Embedding-Code-2B_R) (large code model)

	### Distillation Method
	- Technique: Model2Vec static embedding generation
	- Parameters: PCA dims=256, SIF coefficient=1e-3, Zipf weighting=True
	- Training Data: CodeSearchNet comment-code pairs
	- Languages: Python, JavaScript, Java, PHP, Ruby, Go

	---

	Report generated on 2025-06-01 08:04:06 using automated analysis pipeline.
	For questions about methodology or results, please refer to the CodeSearchNet documentation.