| --- |
| language: |
| - en |
| - ms |
| tags: |
| - sentence-transformers |
| - embedding |
| - retrieval |
| - rag |
| - malaysian |
| - manglish |
| - multilingual |
| license: mit |
| model_type: sentence-transformers |
| base_model: paraphrase-multilingual-mpnet-base-v2 |
| --- |
| |
| # Aranda-v1 |
|
|
| Aranda-v1 is a sentence embedding model specialized for **Malaysian text retrieval**, including Bahasa Malaysia, Manglish (Malaysian English code-switching), and cross-lingual BM↔English matching. It outperforms all tested baselines on overall retrieval Recall@1, Recall@5, Recall@10, and MRR, and is the best model on every individual language category (BM, Manglish, English, Cross-lingual). |
|
|
| ## Training |
|
|
| Two-phase contrastive curriculum: |
|
|
| **Phase 1 — Breadth:** MultipleNegativesRankingLoss on 1M Malaysian positive pairs (paraphrases, social media, news, QA). LR=1e-5, 1000 steps. |
|
|
| **Phase 2 — Discrimination:** Fine-tuned on 58K diverse hard-negative triplets (Lowyat, Twitter, Facebook, formal BM QA, English anchors, cross-lingual pairs) with explicit mined hard negatives. LR=2e-6, 2000 steps. |
|
|
| Base model: `paraphrase-multilingual-mpnet-base-v2` |
|
|
| ## Evaluation |
|
|
| Tested on 4,149 retrieval queries (BM, Manglish, English, Cross-lingual) with ~25 candidates per query, plus 4 additional eval sets (heuristic similarity, mined holdout, Mesolitica reranker, English STS). |
|
|
| ### Overall Retrieval (4,149 queries) |
|
|
| | Model | Recall@1 | Recall@5 | Recall@10 | MRR | Precision@10 | |
| |---|---:|---:|---:|---:|---:| |
| | mpnet-base | 0.8631 | 0.9937 | 0.9973 | 0.9200 | 0.1785 | |
| | multilingual-e5-base | 0.8470 | 0.9940 | **0.9988** | 0.9091 | 0.1792 | |
| | labse | 0.8009 | 0.9882 | 0.9986 | 0.8821 | 0.1780 | |
| | distilbert-multilingual-quora | 0.7973 | 0.9636 | 0.9858 | 0.8719 | 0.1728 | |
| | **Aranda-v1** | **0.8891** | **0.9961** | **0.9998** | **0.9364** | **0.1788** | |
|
|
| ### Per-Language Recall@1 |
|
|
| | Model | BM | Manglish | English | Cross-lingual | |
| |---|---:|---:|---:|---:| |
| | mpnet-base | 0.8200 | 0.8988 | 0.8577 | 0.8267 | |
| | multilingual-e5-base | 0.7938 | 0.9104 | 0.8658 | 0.6167 | |
| | labse | 0.7354 | 0.8993 | 0.6872 | 0.7300 | |
| | distilbert-multilingual-quora | 0.7584 | 0.8320 | 0.7866 | 0.7633 | |
| | **Aranda-v1** | **0.8431** | **0.9290** | **0.8792** | **0.8500** | |
|
|
| ### Heuristic Similarity 2K (per-language Spearman) |
|
|
| | Model | Overall | BM | Manglish | |
| |---|---:|---:|---:| |
| | mpnet-base | **0.4799** | **0.5535** | 0.3937 | |
| | multilingual-e5-base | 0.3582 | 0.2714 | 0.4680 | |
| | labse | 0.3249 | 0.2866 | 0.4397 | |
| | distilbert-multilingual-quora | 0.4585 | 0.4857 | 0.4682 | |
| | Aranda-v1 | 0.4532 | 0.4915 | **0.4536** | |
|
|
| ### Mined Holdout 2K (positive vs negative margins) |
|
|
| | Model | pos>neg % | mean margin | mean pos sim | mean neg sim | |
| |---|---:|---:|---:|---:| |
| | mpnet-base | 98.1% | **0.4880** | 0.6576 | **0.1696** | |
| | multilingual-e5-base | 98.15% | 0.1129 | **0.8805** | 0.7676 | |
| | labse | **98.3%** | 0.4193 | 0.5960 | 0.1768 | |
| | distilbert-multilingual-quora | 94.6% | 0.1041 | 0.9336 | 0.8295 | |
| | Aranda-v1 | 98.2% | 0.4318 | 0.7334 | 0.3016 | |
|
|
| ### Mesolitica Reranker Test (NDCG@10) |
|
|
| | Model | Overall NDCG@10 | |
| |---|---:| |
| | mpnet-base | 0.4699 | |
| | multilingual-e5-base | 0.4705 | |
| | labse | 0.4695 | |
| | distilbert-multilingual-quora | 0.4690 | |
| | **Aranda-v1** | **0.4706** | |
|
|
| ### English STS (mteb/stsbenchmark-sts) |
|
|
| | Model | Spearman | |
| |---|---:| |
| | mpnet-base | **0.8682** | |
| | multilingual-e5-base | 0.8420 | |
| | labse | 0.7225 | |
| | distilbert-multilingual-quora | 0.7866 | |
| | Aranda-v1 | 0.8369 | |
|
|
| ## Usage |
|
|
| ```python |
| from sentence_transformers import SentenceTransformer |
| |
| model = SentenceTransformer("rekabytes/Aranda-v1") |
| |
| # Encode texts |
| embeddings = model.encode([ |
| "macam mana nak renew lesen memandu", |
| "how to renew driving license", |
| "saya nak makan nasi lemak" |
| ], normalize_embeddings=True) |
| |
| # Cosine similarity |
| similarities = embeddings @ embeddings.T |
| print(similarities) |
| ``` |
|
|
| ### With RAG / vector search |
|
|
| ```python |
| # Encode your document corpus (do this once) |
| doc_embeddings = model.encode(documents, normalize_embeddings=True) |
| |
| # At query time |
| query_embedding = model.encode([query], normalize_embeddings=True) |
| scores = query_embedding @ doc_embeddings.T |
| top_k = scores.argsort()[0][-5:][::-1] |
| ``` |
|
|
| ## Intended Use |
|
|
| - **RAG context retrieval** for Malaysian applications |
| - **Semantic search** over BM/Manglish/English document corpora |
| - **Cross-lingual matching** (BM ↔ English) |
| - **Dense retrieval** in hybrid search pipelines (paired with BM25) |
|
|
| ## Limitations |
|
|
| - English STS performance is below the base mpnet model (0.8369 vs 0.8682) — the model specialized for Malaysian text |
| - Not a reranker — use a cross-encoder for second-stage reranking |
| - Tested on Malaysian web data; performance may vary on other Southeast Asian languages |
|
|
| ## Model Details |
|
|
| - **Architecture:** XLM-RoBERTa (base) |
| - **Embedding dimension:** 768 |
| - **Max sequence length:** 128 |
| - **Pooling:** Mean |
| - **Normalization:** L2 |
|
|
