Mursit-Embed-Qwen3-4B-TR

Model Description

Mursit-Embed-Qwen3-4B-TR is a Turkish embedding model converted from decoder-only architecture through decoder-to-encoder conversion. The model is based on Mecellem-Qwen3-4B-TR (continual pre-trained decoder model) and adapted for Turkish legal retrieval tasks.

Model Type: CLM-Embedding (Decoder-to-Encoder Converted)
Parameters: 4B
Base Model: newmindai/Mecellem-Qwen3-4B-TR
Embedding Dimension: 2,560
Max Sequence Length: 1,024 tokens

Architecture Conversion

The model underwent decoder-to-encoder conversion with the following modifications:

1. Removal of Language Modeling Head: The autoregressive lm_head layer was removed
2. Bidirectional Attention: Causal attention mask replaced with bidirectional attention
3. Mean Pooling: Fixed-size representations extracted using mean pooling over all token positions
4. Identity Projection: Projection layer initialized to maintain 2,560-dimensional embedding output

Training Details

Post-training for Embeddings:

• Dataset: MS MARCO-TR (920,106 triplets)
• Loss Function: CachedGISTEmbedLoss with BGE-M3 guide model (568M parameters)
• Training Framework: Sentence Transformers
• Optimizer: AdamW
• Learning Rate: 5×10⁻⁶
• Per-GPU Batch Size: 2
• Gradient Accumulation: 16
• Effective Batch Size: 128
• Max Sequence Length: 1,024 tokens
• Hardware: 4× H100 GPUs (single node, NVLink interconnect)
  • System: MareNostrum 5 ACC partition at Barcelona Supercomputing Center (BSC)
  • Node Configuration: Single node with 4× NVIDIA Hopper H100 64GB GPUs, 80 CPU cores, 512GB DDR5 memory
  • GPU Interconnect: NVLink for intra-node GPU communication (4 GPUs connected via NVLink)

Performance on MTEB-Turkish Benchmark

The following visualization shows the model's performance compared to other Turkish language models:

Model Performance Comparison: Legal Score vs. MTEB Score. CLM-Embedding models (orange squares) show competitive performance on general Turkish tasks.

This model was evaluated on the comprehensive MTEB-Turkish benchmark, which includes 17 tasks across 5 task types. The benchmark evaluates models on general Turkish NLP tasks as well as domain-specific legal retrieval tasks.

Comprehensive Benchmark Results

The following table presents comprehensive evaluation results across all models evaluated on the MTEB-Turkish benchmark. This model's results are highlighted in italics.

Model	MTEB	Legal	Cls.	Clus.	Pair	Ret.	STS	Cont.	Reg.	Case	Params	Type
embeddinggemma-300m	65.42	50.63	77.74	45.05	80.02	55.06	69.22	83.97	39.56	28.38	307M	Emb.
bge-m3	62.87	51.16	75.35	35.86	78.88	54.42	69.83	86.08	38.09	29.3	567M	Emb.
Mursit-Embed-Qwen3-1.7B-TR	56.84	34.76	68.46	42.22	59.67	50.1	63.77	70.22	17.94	16.11	1.7B	CLM-E.
Mursit-Large-TR-Retrieval	56.87	46.56	67.72	41.15	59.78	51.69	64.01	81.78	32.67	25.24	403M	Emb.
Mursit-Base-TR-Retrieval	55.86	47.52	66.25	39.75	61.31	50.07	61.9	80.4	34.1	28.07	155M	Emb.
Mursit-Embed-Qwen3-4B-TR	53.65	37.0	67.29	36.68	58.36	51.12	54.77	69.25	24.21	17.56	4B	CLM-E.
-------	------	-------	------	------	------	------	-----	-------	------	------	--------	------
bert-base-turkish-uncased	46.23	24.94	68.05	33.81	60.44	32.01	36.85	52.47	12.05	10.29	110M	MLM
turkish-large-bert-cased	45.3	19.12	67.43	34.24	60.11	28.68	36.04	47.57	5.93	3.85	337M	MLM
bert-base-turkish-cased	45.17	24.41	66.39	35.28	60.05	30.52	33.62	54.03	10.13	9.07	110M	MLM
BERTurk-Legal	42.02	32.63	60.61	26.24	59.51	25.8	37.94	61.4	15.51	20.99	184M	MLM
Mursit-Large	41.75	23.71	62.95	25.34	58.04	27.4	35.01	42.74	11.29	17.1	403M	MLM
turkish-base-bert-uncased	44.68	27.58	66.22	30.23	58.84	31.4	36.74	56.6	13.39	12.74	110M	MLM
Mursit-Base	40.23	17.93	59.78	25.48	58.65	20.82	36.45	36.0	7.4	10.4	155M	MLM
mmBERT-base	39.65	12.15	61.84	26.77	59.25	15.83	34.56	34.45	1.33	0.68	306M	MLM
TabiBERT	37.77	11.5	59.63	25.75	58.19	14.96	30.32	32.02	1.86	0.63	148M	MLM
ModernBERT-base	23.8	2.99	39.06	2.01	53.95	2.1	21.91	7.92	0.62	0.43	149M	MLM
ModernBERT-large	23.74	2.44	39.44	3.9	53.73	1.8	19.85	6.12	0.62	0.59	394M	MLM

Column abbreviations: MTEB = mean performance across task types; Legal = weighted average of Contracts, Regulation, Caselaw; Classification = accuracy on Turkish classification tasks; Clustering = V-measure on clustering tasks; Pair Classification = average precision on pair classification tasks like NLI; Retrieval = nDCG@10 on information retrieval tasks; Semantic Textual Similarity = Spearman correlation; Contracts = nDCG@10 on legal contract retrieval; Regulation = nDCG@10 on regulatory text retrieval; Caselaw = nDCG@10 on case law retrieval; Number of Parameters = number of model parameters; Model Type = model type (Embedding, CLM-Embedding, Masked Language Model). Bold values indicate the highest score in each column.

Key Findings:

• The model achieves competitive MTEB Score (53.65) on general Turkish tasks
• Shows improved Legal Score (37.00) compared to the 1.7B variant, indicating that larger models benefit from decoder-to-encoder conversion | Semantic Textual Similarity | Spearman Correlation | 54.77 |

Legal Domain Performance

Task	Metric	Score
Contracts Retrieval	nDCG@10	69.25
Regulation Retrieval	nDCG@10	24.21
Caselaw Retrieval	nDCG@10	17.56

Note: This larger model shows even greater degradation compared to the 1.7B variant, suggesting that increased model capacity alone does not compensate for the architectural mismatch between autoregressive generation and bidirectional embedding tasks.

Usage

Installation

pip install sentence-transformers

Basic Usage

from sentence_transformers import SentenceTransformer

# Load model
model = SentenceTransformer("newmindai/Mursit-Embed-Qwen3-4B-TR")

# Encode sentences
sentences = [
    "Türk hukuk sistemi medeni hukuk geleneğine dayanır",
    "Anayasa Türkiye Cumhuriyeti'nin temel hukuk belgesidir"
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # (2, 2560)

ONNX Model Inference

This script demonstrates how to use the ONNX model from Hugging Face for text embedding generation.

Exporting Model to ONNX

To export the model to ONNX format, use the optimum-cli command:

optimum-cli export onnx \
  -m newmindai/Mursit-Embed-Qwen3-4B-TR \
  --task feature-extraction \
  onnx/MursitEmbedQwen3-4b

This will create the model.onnx file (and model.onnx_data if the model is large) in the specified output directory.

Installation

pip install onnxruntime-gpu transformers huggingface_hub numpy

Usage

from huggingface_hub import snapshot_download
import onnxruntime as ort
from transformers import AutoTokenizer
import os

repo_id = "newmindai/Mursit-Embed-Qwen3-4B-TR"

local_dir = snapshot_download(
    repo_id=repo_id,
    allow_patterns=["model.onnx", "model.onnx_data"]
)

onnx_path = os.path.join(local_dir, "model.onnx")

tokenizer = AutoTokenizer.from_pretrained(repo_id)

sess = ort.InferenceSession(
    onnx_path,
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"]
)

texts = ["This is a test"]
inputs = tokenizer(
    texts,
    padding=True,
    truncation=True,
    return_tensors="np"
)

ort_inputs = {k: v for k, v in inputs.items()}

# Inference
outputs = sess.run(None, ort_inputs)

# Sentence embedding (last output)
sentence_embedding = outputs[-1]

print("Shape:", sentence_embedding.shape)
print("Providers:", sess.get_providers())
print(sentence_embedding[0][:10])

Features

• Automatic GPU/CPU selection: Uses CUDA if available, otherwise falls back to CPU
• Hugging Face integration: Downloads model files directly from Hugging Face Hub
• Simple API: Easy-to-use interface for text embedding generation

Use Cases

• Semantic search in Turkish documents
• Information retrieval tasks
• Text similarity and matching
• Question answering systems

Acknowledgments

This work was supported by the EuroHPC Joint Undertaking through project etur46 with access to the MareNostrum 5 supercomputer, hosted by Barcelona Supercomputing Center (BSC), Spain. MareNostrum 5 is owned by EuroHPC JU and operated by BSC. We are grateful to the BSC support team for their assistance with job scheduling, environment configuration, and technical guidance throughout the project.

The numerical calculations reported in this work were fully/partially performed at TÜBİTAK ULAKBİM, High Performance and Grid Computing Center (TRUBA resources). The authors gratefully acknowledge the know-how provided by the MINERVA Support for expert guidance and collaboration opportunities in HPC-AI integration.

References

If you use this model, please cite our paper:

@article{mecellem2026,
  title={Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain},
  author={Uğur, Özgür and Göksu, Mahmut and Çimen, Mahmut and Yılmaz, Musa and Şavirdi, Esra and Demir, Alp Talha and Güllüce, Rumeysa and Çetin, İclal and Sağbaş, Ömer Can},
  journal={arXiv preprint arXiv:2601.16018},
  year={2026},
  month={January},
  url={https://arxiv.org/abs/2601.16018},
  doi={10.48550/arXiv.2601.16018},
  eprint={2601.16018},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

Base Model References

@article{qwen2024,
  title={Qwen3: A Large Language Model Series},
  author={Qwen Team},
  journal={arXiv preprint arXiv:2409.00000},
  year={2024}
}