File size: 4,903 Bytes
a707cd9 75c6892 a707cd9 75c6892 a707cd9 75c6892 a707cd9 75c6892 a707cd9 75c6892 a707cd9 75c6892 a707cd9 75c6892 a707cd9 75c6892 a707cd9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 | ---
language:
- en
- ms
tags:
- sentence-transformers
- embedding
- retrieval
- rag
- malaysian
- manglish
- multilingual
license: mit
model_type: sentence-transformers
base_model: paraphrase-multilingual-mpnet-base-v2
---
# Aranda-v1
Aranda-v1 is a sentence embedding model specialized for **Malaysian text retrieval**, including Bahasa Malaysia, Manglish (Malaysian English code-switching), and cross-lingual BM↔English matching. It outperforms all tested baselines on overall retrieval Recall@1, Recall@5, Recall@10, and MRR, and is the best model on every individual language category (BM, Manglish, English, Cross-lingual).
## Training
Two-phase contrastive curriculum:
**Phase 1 — Breadth:** MultipleNegativesRankingLoss on 1M Malaysian positive pairs (paraphrases, social media, news, QA). LR=1e-5, 1000 steps.
**Phase 2 — Discrimination:** Fine-tuned on 58K diverse hard-negative triplets (Lowyat, Twitter, Facebook, formal BM QA, English anchors, cross-lingual pairs) with explicit mined hard negatives. LR=2e-6, 2000 steps.
Base model: `paraphrase-multilingual-mpnet-base-v2`
## Evaluation
Tested on 4,149 retrieval queries (BM, Manglish, English, Cross-lingual) with ~25 candidates per query, plus 4 additional eval sets (heuristic similarity, mined holdout, Mesolitica reranker, English STS).
### Overall Retrieval (4,149 queries)
| Model | Recall@1 | Recall@5 | Recall@10 | MRR | Precision@10 |
|---|---:|---:|---:|---:|---:|
| mpnet-base | 0.8631 | 0.9937 | 0.9973 | 0.9200 | 0.1785 |
| multilingual-e5-base | 0.8470 | 0.9940 | **0.9988** | 0.9091 | 0.1792 |
| labse | 0.8009 | 0.9882 | 0.9986 | 0.8821 | 0.1780 |
| distilbert-multilingual-quora | 0.7973 | 0.9636 | 0.9858 | 0.8719 | 0.1728 |
| **Aranda-v1** | **0.8891** | **0.9961** | **0.9998** | **0.9364** | **0.1788** |
### Per-Language Recall@1
| Model | BM | Manglish | English | Cross-lingual |
|---|---:|---:|---:|---:|
| mpnet-base | 0.8200 | 0.8988 | 0.8577 | 0.8267 |
| multilingual-e5-base | 0.7938 | 0.9104 | 0.8658 | 0.6167 |
| labse | 0.7354 | 0.8993 | 0.6872 | 0.7300 |
| distilbert-multilingual-quora | 0.7584 | 0.8320 | 0.7866 | 0.7633 |
| **Aranda-v1** | **0.8431** | **0.9290** | **0.8792** | **0.8500** |
### Heuristic Similarity 2K (per-language Spearman)
| Model | Overall | BM | Manglish |
|---|---:|---:|---:|
| mpnet-base | **0.4799** | **0.5535** | 0.3937 |
| multilingual-e5-base | 0.3582 | 0.2714 | 0.4680 |
| labse | 0.3249 | 0.2866 | 0.4397 |
| distilbert-multilingual-quora | 0.4585 | 0.4857 | 0.4682 |
| Aranda-v1 | 0.4532 | 0.4915 | **0.4536** |
### Mined Holdout 2K (positive vs negative margins)
| Model | pos>neg % | mean margin | mean pos sim | mean neg sim |
|---|---:|---:|---:|---:|
| mpnet-base | 98.1% | **0.4880** | 0.6576 | **0.1696** |
| multilingual-e5-base | 98.15% | 0.1129 | **0.8805** | 0.7676 |
| labse | **98.3%** | 0.4193 | 0.5960 | 0.1768 |
| distilbert-multilingual-quora | 94.6% | 0.1041 | 0.9336 | 0.8295 |
| Aranda-v1 | 98.2% | 0.4318 | 0.7334 | 0.3016 |
### Mesolitica Reranker Test (NDCG@10)
| Model | Overall NDCG@10 |
|---|---:|
| mpnet-base | 0.4699 |
| multilingual-e5-base | 0.4705 |
| labse | 0.4695 |
| distilbert-multilingual-quora | 0.4690 |
| **Aranda-v1** | **0.4706** |
### English STS (mteb/stsbenchmark-sts)
| Model | Spearman |
|---|---:|
| mpnet-base | **0.8682** |
| multilingual-e5-base | 0.8420 |
| labse | 0.7225 |
| distilbert-multilingual-quora | 0.7866 |
| Aranda-v1 | 0.8369 |
## Usage
```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("rekabytes/Aranda-v1")
# Encode texts
embeddings = model.encode([
"macam mana nak renew lesen memandu",
"how to renew driving license",
"saya nak makan nasi lemak"
], normalize_embeddings=True)
# Cosine similarity
similarities = embeddings @ embeddings.T
print(similarities)
```
### With RAG / vector search
```python
# Encode your document corpus (do this once)
doc_embeddings = model.encode(documents, normalize_embeddings=True)
# At query time
query_embedding = model.encode([query], normalize_embeddings=True)
scores = query_embedding @ doc_embeddings.T
top_k = scores.argsort()[0][-5:][::-1]
```
## Intended Use
- **RAG context retrieval** for Malaysian applications
- **Semantic search** over BM/Manglish/English document corpora
- **Cross-lingual matching** (BM ↔ English)
- **Dense retrieval** in hybrid search pipelines (paired with BM25)
## Limitations
- English STS performance is below the base mpnet model (0.8369 vs 0.8682) — the model specialized for Malaysian text
- Not a reranker — use a cross-encoder for second-stage reranking
- Tested on Malaysian web data; performance may vary on other Southeast Asian languages
## Model Details
- **Architecture:** XLM-RoBERTa (base)
- **Embedding dimension:** 768
- **Max sequence length:** 128
- **Pooling:** Mean
- **Normalization:** L2
|