yahyaabd/paraphrase-multilingual-miniLM-L12-v2-mnrl-beir

SentenceTransformer based on sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

This is a sentence-transformers model finetuned from sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 on the statictable-triplets-all dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • statictable-triplets-all

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("yahyaabd/paraphrase-multilingual-miniLM-L12-v2-mnrl-beir")
# Run inference
sentences = [
    'Bagaimana tren ekspor teh Indonesia ke berbagai negara tahun 2008?',
    'Ekspor Teh Menurut Negara Tujuan Utama, 2000-2015',
    'Luas Kawasan Hutan dan Kawasan Konservasi Perairan Indonesia Berdasarkan Surat Keputusan Menteri Lingkungan Hidup dan Kehutanan, 2017-2022',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: bps-statictable-ir
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.3648
cosine_accuracy@5	0.6319
cosine_accuracy@10	0.7199
cosine_precision@1	0.3648
cosine_precision@5	0.159
cosine_precision@10	0.1088
cosine_recall@1	0.2757
cosine_recall@5	0.4928
cosine_recall@10	0.5744
cosine_ndcg@1	0.3648
cosine_ndcg@5	0.4404
cosine_ndcg@10	0.4644
cosine_mrr@1	0.3648
cosine_mrr@5	0.4639
cosine_mrr@10	0.4758
cosine_map@1	0.3648
cosine_map@5	0.3868
cosine_map@10	0.3942

Training Details

Training Dataset

statictable-triplets-all

• Dataset: statictable-triplets-all at 24979b4
• Size: 967,831 training samples
• Columns: query, pos, and neg
• Approximate statistics based on the first 1000 samples:

  	query	pos	neg
  type	string	string	string
  details	min: 5 tokens mean: 18.53 tokens max: 37 tokens	min: 4 tokens mean: 25.42 tokens max: 58 tokens	min: 4 tokens mean: 25.71 tokens max: 58 tokens

• Samples:

  query	pos	neg
  Data pendapatan per kapita, bedakan per golongan rumah tangga (ribu rupiah), 2008	Rata-rata Jumlah Pendapatan perkapita Menurut Golongan Rumah Tangga (ribu rupiah), 2000, 2005, 2008	Ringkasan Neraca Arus Dana, Triwulan II, 2007, (Miliar Rupiah)
  Berapa ribu ton impor Indonesia dari negara-negara utama tahun 2018?	Volume Impor Menurut Negara Asal Utama (Berat bersih:ribu ton), 2017-2023	Rata-Rata Pengeluaran Konsumsi per Kapita Menurut Golongan Rumah Tangga (ribu rupiah), 2000, 2005, dan 2008
  Kredit dari lembaga keuangan non-bank 2000-2016	Pemberian Kredit oleh Lembaga-Lembaga Keuangan Lainnya (miliar rupiah), 2000-2016	Angka Partisipasi Sekolah (APS) Penduduk Umur 7-18 Tahun Menurut Klasifikasi Desa, Jenis Kelamin, dan Kelompok Umur, 2009-2023

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

statictable-triplets-all

• Dataset: statictable-triplets-all at 24979b4
• Size: 967,831 evaluation samples
• Columns: query, pos, and neg
• Approximate statistics based on the first 1000 samples:

  	query	pos	neg
  type	string	string	string
  details	min: 5 tokens mean: 18.75 tokens max: 37 tokens	min: 4 tokens mean: 25.35 tokens max: 58 tokens	min: 4 tokens mean: 25.44 tokens max: 58 tokens

• Samples:

  query	pos	neg
  Bagaimana kebiasaan rumah tangga memisahkan sampah (mudah busuk vs tidak), per provinsi, 2021?	Persentase Rumah Tangga Menurut Provinsi dan Perlakuan Memilah Sampah Mudah Membusuk dan Tidak Mudah Membusuk, 2013-2014, 2021	Rata-Rata Upah/Gaji Bersih Sebulan (rupiah) Buruh/Karyawan/Pegawai menurut Provinsi dan Lapangan Pekerjaan Utama di 17 Sektor, 2023
  Bagaimana rincian pinjaman investasi (hanya Rupiah) dari bank umum ke berbagai sektor ekonomi pada tahun 2016?	Pinjaman Investasi Bank-Bank Umum dalam Rupiah Menurut Sektor Ekonomi (miliar rupiah), 2000 - 2016	Ringkasan Neraca Arus Dana, Triwulan IV, 2009, (Miliar Rupiah)
  Data rumah tangga per provinsi, adakah area serapan air? Ambil tahun 2013	Persentase Rumah Tangga Menurut Provinsi dan Keberadaan Area Resapan Air, 2013-2014	Sistem Neraca Sosial Ekonomi Indonesia Tahun 2022 (9 x 9)

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Logs

Epoch	Step	bps-statictable-ir_cosine_ndcg@10
0	0	0.4644

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.4.1
• Transformers: 4.48.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.3.0
• Datasets: 3.4.1
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}