mentee-embed-v3

A 41M-parameter trilingual text embedding model trained entirely from scratch — no pretrained backbone, no BERT, no RoBERTa. Random initialization only.

Developed by Team MenteE AI (menteeai.org) as part of an ongoing research effort to build competitive multilingual embeddings from the ground up for Arabic, English, and Urdu.

---

Key Facts

Property	Value
Parameters	41M
Embedding dimension	384
Max sequence length	128 tokens
Languages	Arabic 🇸🇦 · English 🇬🇧 · Urdu 🇵🇰
Architecture	12-layer Transformer, custom BPE tokenizer (50K vocab)
Initialization	Random (trained from scratch)
Training data	~2.1M triplets (NLI + MS-MARCO + OPUS parallel + MIRACL)
Pooling	Mean pooling
Training objective	Relational distillation + InfoNCE contrastive (teacher: multilingual-e5-base)

---

What "From Scratch" Means

Most embedding models fine-tune an existing pretrained encoder (BERT, RoBERTa, MPNet). mentee-embed-v3 does not. We:

1. Trained a custom BPE tokenizer on Arabic, English, and Urdu text
2. Initialized a 12-layer Transformer with random weights
3. Ran masked language modeling pretraining on 2.1M+ sentences
4. Applied two-round contrastive distillation with hard negative mining

No pretrained checkpoint was used at any stage.

---

Benchmark Results

All baselines evaluated under identical conditions on the same hardware.

Protocol A — In-batch Retrieval (pool ≈ 97 candidates)

Format: acc@1 / R@5 / MRR@10

Avg MRR@10 computed over 5 datasets: MIRACL-EN, MIRACL-AR, MIRACL-UR, xling EN-UR, MS-MARCO (val excluded from avg).

Model	MIRACL-EN	MIRACL-AR	MIRACL-UR	xling EN↔UR	MS-MARCO	Avg MRR@10
mentee-embed-v3 (ours)	0.636/0.920/0.766	0.326/0.604/0.475	0.290/0.568/0.443	0.781/0.925/0.848	0.517/0.981/0.742	0.655
paraphrase-multilingual-mpnet-base-v2	0.864/1.000/0.931	0.722/0.975/0.839	0.686/0.950/0.806	0.831/0.937/0.880	0.665/0.998/0.830	0.857
paraphrase-multilingual-MiniLM-L12-v2	0.854/0.997/0.924	0.696/0.964/0.819	0.621/0.908/0.753	0.782/0.907/0.841	0.600/0.998/0.796	0.827
all-MiniLM-L6-v2	0.856/0.999/0.927	0.025/0.109/0.144	0.028/0.088/0.140	0.065/0.172/0.186	0.696/1.000/0.848	0.449

✅ mentee-embed-v3 beats all-MiniLM-L6-v2 (0.655 vs 0.449) on Protocol A avg MRR@10 — despite all-MiniLM being a pretrained model.

---

Protocol B — MIRACL Wikipedia Corpus Retrieval (full ranking, ~5K–15K passages)

Format: MRR@10 · R@5 · R@100

Model	EN	AR	UR	Avg MRR@10
paraphrase-multilingual-mpnet-base-v2	0.853 · 0.923 · 0.997	0.622 · 0.757 · 0.947	0.534 · 0.680 · 0.903	0.670
paraphrase-multilingual-MiniLM-L12-v2	0.840 · 0.933 · 0.990	0.591 · 0.710 · 0.943	0.469 · 0.557 · 0.847	0.633
all-MiniLM-L6-v2	0.867 · 0.967 · 1.000	0.100 · 0.000 · 0.010	0.106 · 0.007 · 0.013	0.358
mentee-embed-v3 (ours)	0.418 · 0.473 · 0.850	0.182 · 0.167 · 0.500	0.180 · 0.143 · 0.440	0.260

📌 Protocol B uses Wikipedia passages (MIRACL) — a challenging out-of-domain test for a model trained primarily on NLI and MS-MARCO data.

---

Protocol C — MS-MARCO Corpus Retrieval (10K passages, in-domain)

Model	MRR@10	R@5	R@100
all-MiniLM-L6-v2	0.951	0.993	1.000
paraphrase-multilingual-mpnet-base-v2	0.882	0.970	0.993
paraphrase-multilingual-MiniLM-L12-v2	0.839	0.910	0.997
mentee-embed-v3 (ours)	0.645	0.760	0.957

🔥 0.645 MRR@10 from a randomly initialized 41M model — trained on 2.1M triplets vs billions for the baselines.
Gap to MiniLM-multilingual: only 0.194.

---

Training Data

Source	Language	Triplets	Type
all-NLI (sentence-transformers)	EN	558K	NLI triplets
XNLI	AR	128K	NLI triplets
XNLI	UR	125K	NLI triplets
OPUS-100 EN↔UR	EN/UR	300K	Parallel translation
OPUS-100 AR↔EN	AR/EN	300K	Parallel translation
MS-MARCO BM25 triplets	EN	500K	Passage retrieval
MS-MARCO hard negatives	EN	200K	Hard retrieval
MIRACL	EN/AR/UR	~9K	Wikipedia retrieval
Total		~2.1M

---

Training Pipeline

Stage 1 — MLM Pretraining
  Random init → masked language modeling on 2.1M sentences
  8,000 steps · batch=32 · vocab=50K BPE

Stage 2 — Distillation Round 1 (no hard negatives)
  Teacher: intfloat/multilingual-e5-base (768-dim)
  InfoNCE contrastive + relational distillation
  4,000 steps · batch=512 · temp=0.05

Stage 3 — Hard Negative Mining
  GPU-accelerated top-5 mining across full 2.1M corpus

Stage 4 — Distillation Round 2 (with hard negatives)
  Same objective + mined hard negatives per anchor
  10,000 steps · batch=512 · temp=0.05

---

Usage

# pip install torch transformers tokenizers huggingface_hub
from transformers import AutoModel, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v3", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v3",   trust_remote_code=True)

sentences = [
    "Hello, how are you?",
    "مرحبا، كيف حالك؟",
    "ہیلو، آپ کیسے ہیں؟"
]

embeddings = model.encode(sentences, tokenizer=tok)
print(embeddings.shape)  # torch.Size([3, 384])

trust_remote_code=True is required — standard for custom-architecture models on HuggingFace. The code runs entirely on your machine.

Similarity search

from transformers import AutoModel, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v3", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v3",   trust_remote_code=True)

query    = model.encode(["What is machine learning?"], tokenizer=tok)
passages = model.encode([
    "Machine learning is a subset of artificial intelligence.",
    "The weather today is sunny.",
    "تعلم الآلة هو فرع من فروع الذكاء الاصطناعي.",
], tokenizer=tok)

scores = query @ passages.T
print(scores)  # tensor([[0.81, 0.60, 0.79]])

---

Limitations

• Protocol B (Wikipedia retrieval) scores are lower than pretrained baselines — the model was not trained on Wikipedia-style passages
• Arabic and Urdu lag behind English due to less retrieval-specific training data (mMARCO Arabic/Urdu was unavailable in a compatible format)
• Vocabulary limited to 50K tokens trained on ~2.1M sentences — rare scripts and dialects may tokenize poorly
• Not evaluated on MTEB full suite yet

---

Citation

@misc{mentee-embed-v3-2026,
  title   = {How Far Can Multilingual Text Embeddings Be Trained From Scratch?
             A Compute-Efficient Study of Arabic, English, and Urdu},
  author  = {Shah, Syed Syab Ahmad and Sania, Shakeel and Hamza, Rustam and Mahboob, Iqbal},
  year    = {2026},
  doi     = {10.5281/zenodo.22117673},
  url     = {https://doi.org/10.5281/zenodo.22117673},
  note    = {MenteE AI. Apache-2.0 License}
}

---

About MenteE AI

Built by Syed Syab Ahmad Shah and Team MenteE AI. 🌐 menteeai.org · 📧 syab@menteeai.org Research paper: 10.5281/zenodo.22117673