Update README.md

998d171 verified 16 days ago

10.4 kB

---
language:
  - fa
  - eng
license: apache-2.0
library_name: sentence-transformers
pipeline_tag: sentence-similarity
base_model: Qwen/Qwen3-Embedding-0.6B
base_model_relation: finetune
datasets:
  - Yaldat/Persian-Pairwise
  - Yaldat/Persian-Pairwise-Embedded
tags:
  - persian
  - text-embeddings
  - dense-retrieval
  - semantic-search
  - sentence-transformers
  - transformers
  - knowledge-distillation
  - feature-extraction
---
# Yalda Embedding

**Yalda Embedding** is an openly developed Persian text embedding model for dense retrieval
and semantic search. It specializes
[Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) using more than one
million Persian query-document pairs and relational knowledge distillation from
[F2LLM-v2-8B](https://huggingface.co/codefuse-ai/F2LLM-v2-8B).

With **595.8 million parameters** and **1,024-dimensional embeddings**, Yalda achieves
**state-of-the-art performance among the evaluated 0.6B models** on eight Persian retrieval
tasks from FaMTEB. Its reported mean is **0.5488**, compared with **0.4713** for its Qwen3-0.6B
initialization and **0.5563** for Qwen3-Embedding-4B. It offers competitive retrieval performance
against the 4B reference while retaining the 0.6B model's inference footprint.

The model weights, training data, teacher embeddings, and development notebooks are public:

- **Training and benchmark code:** [yaldat/yalda-embedding](https://github.com/yaldat/yalda-embedding)
- **Training data:** [Yaldat/Persian-Pairwise](https://huggingface.co/datasets/Yaldat/Persian-Pairwise)
- **Teacher-embedded data:** [Yaldat/Persian-Pairwise-Embedded](https://huggingface.co/datasets/Yaldat/Persian-Pairwise-Embedded)

## Model details

| Property | Value |
| --- | --- |
| Primary language | Persian / Farsi |
| Base model | Qwen3-Embedding-0.6B |
| Parameters | 595,776,512 |
| Embedding dimension | 1,024 |
| Architecture | Decoder-only Transformer, 28 blocks |
| Attention | Causal, grouped-query attention |
| Pooling | Last non-padding token |
| Training and evaluation input length | 512 tokens |
| Similarity | Cosine similarity; dot product for L2-normalized embeddings |
| Released weights | FP32 Safetensors |
| License | Apache 2.0 |

## Quick start

Install Sentence Transformers and its model dependencies:

```bash
pip install -U sentence-transformers transformers torch
```

Use the saved **query** prompt for search queries and the **document** prompt for passages.
Normalize embeddings so their dot product equals cosine similarity.

```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Yaldat/Yalda-Embedding")
model.tokenizer.padding_side = "left"
model.max_seq_length = 512
queries = [
    "پایتخت ایران کجاست؟",
    "چرا برگ درختان در پاییز تغییر رنگ می‌دهد؟",
]
documents = [
    "تهران پایتخت ایران و یکی از بزرگ‌ترین شهرهای این کشور است.",
    "در پاییز با کاهش نور خورشید، کلروفیل برگ‌ها تجزیه می‌شود و رنگدانه‌های دیگر نمایان می‌شوند.",
    "کتابخانه‌ها فضایی برای مطالعه و دسترسی به منابع علمی فراهم می‌کنند.",
]
query_embeddings = model.encode(
    queries,
    prompt_name="query",
    normalize_embeddings=True,
)
document_embeddings = model.encode(
    documents,
    prompt_name="document",
    normalize_embeddings=True,
)
scores = query_embeddings @ document_embeddings.T
for query, row in zip(queries, scores):
    best_index = int(row.argmax())
    print(f"Query: {query}")
    print(f"Best passage: {documents[best_index]}")
    print(f"Similarity: {row[best_index]:.4f}\n")
```

The model has no default prompt, so select `prompt_name="query"` explicitly for queries.
The saved query prompt is:

```text
Instruct: Given a web search query, retrieve relevant passages that answer the query
Query:
```

Documents receive no instruction. Pass raw query text to `encode`; the query prompt is
added automatically when selected. For larger collections, precompute document embeddings
and index them in a vector search system.

The example uses 512 tokens to match the training and evaluation setup. Longer inputs are
truncated under this setting. The inherited backbone's longer context configuration does
not establish retrieval quality beyond the evaluated input length.

## Evaluation

The results below are reported in **Yalda Embedding: An Engineering Effort to Reach State of
the Art**. Evaluation uses the test splits of eight Persian retrieval tasks from
[FaMTEB](https://aclanthology.org/2025.findings-emnlp.614/). The summary is the unweighted
macro-average of the reported task scores.

### Comparison with evaluated baselines

| Model | Mean retrieval score |
| --- | ---: |
| Qwen3-Embedding-4B | 0.5563 |
| **Yalda Embedding** | **0.5488** |
| F2LLM-v2-0.6B | 0.5063 |
| Dibachain/Diba-Embed | 0.4788 |
| Qwen3-Embedding-0.6B | 0.4713 |
| heydariAI/persian-embeddings | 0.4175 |
| Tooka-SBERT-V2-Large | 0.4150 |

Yalda improves over Qwen3-Embedding-0.6B by **0.0775 absolute** (**16.4% relative**) and
over F2LLM-v2-0.6B by **0.0425 absolute**. It exceeds Diba-Embed by **0.0700 absolute**
(**14.6% relative**). It scores higher than each of these three baselines on all eight
tasks at the reported precision.

Against Qwen3-Embedding-4B, Yalda scores higher on NFCorpus-Fa, SynPerQARetrieval,
PersianWebDocumentRetrieval, and HotpotQA-FaHardNegatives, and ties on FiQA2018-Fa.
Its overall mean is **0.0075 below** the 4B reference. The SOTA claim is scoped to the
evaluated 0.6B models and these eight retrieval tasks.

### Yalda's per-task scores

| Task | Reported score |
| --- | ---: |
| FiQA2018-Fa | 0.30 |
| NFCorpus-Fa | 0.33 |
| SynPerQARetrieval | 0.88 |
| PersianWebDocumentRetrieval | 0.57 |
| HotpotQA-FaHardNegatives | 0.58 |
| MSMARCO-FaHardNegatives | 0.65 |
| NQ-FaHardNegatives | 0.43 |
| SciFact-Fa | 0.65 |
| **Macro-average** | **0.5488** |

Student training uses query-positive-document pairs, in-batch negatives, and relational
distillation. It uses no explicitly mined or supplied hard negatives, while achieving the
reported scores on the three hard-negative evaluation tasks.

## Training recipe

The model is fine-tuned on **1,035,981 Persian query-positive-document pairs** assembled from
13 retrieval resources. Training data comes from the source-provided training splits;
evaluation uses their corresponding test splits. The empirical source mixture is shuffled
without resampling.

The shared encoder is optimized with query-to-document InfoNCE and forward KL divergence
between teacher and student distributions over the documents in each global batch:

```text
loss = (1 - alpha) * InfoNCE + alpha * KL(teacher || student)
alpha = 0.75
```

The final recipe weights InfoNCE by **0.25** and distillation by **0.75**. Teacher vectors
are precomputed offline, truncated from 4,096 to their first 1,024 Matryoshka dimensions,
and L2-normalized before constructing similarity distributions. The method transfers
ranking relationships rather than matching teacher and student coordinates. The 8B teacher
is needed for data preparation, not for student training or inference.

| Setting | Value |
| --- | --- |
| Hardware | Eight TPU v5e devices |
| Training framework | JAX 0.9.2 and Flax NNX 0.12.6 |
| Epochs | 1 |
| Global batch size | 512 |
| Pairs per device | 64 |
| Optimization steps | 2,023 |
| Optimizer | AdamW |
| Peak learning rate | 2e-5 |
| Warmup | First 10% of updates |
| Decay | Cosine to zero |
| Gradient clipping | Global norm 1.0 |
| Student / teacher temperature | 0.05 / 0.07 |
| KD weight | 0.75 |
| Sequence length | 512 tokens |
| Precision | BF16 projections with FP32 parameters, normalization, and losses |
| Memory strategy | Single-axis FSDP and activation rematerialization |

One epoch consumes 1,035,776 pairs; the incomplete final batch of 205 pairs is omitted.
The model seed is zero; the final experiment does not fix the data permutation seed.

### Distillation ablation

| Configuration | KD weight | Mean retrieval score |
| --- | ---: | ---: |
| Stabilized InfoNCE, batch size 512 | 0.00 | 0.4813 |
| Ablation 4 | 0.25 | 0.5125 |
| Ablation 5 | 0.50 | 0.5313 |
| **Ablation 6: final model** | **0.75** | **0.5488** |

The final distilled configuration improves all eight tasks over the stabilized,
non-distilled model. Increasing the KD weight from 0.50 to 0.75 adds **0.0175** to the mean.

## Open development and reproducibility

The public [GitHub repository](https://github.com/yaldat/yalda-embedding) contains:

- [Yalda-Embedding-Training.ipynb](https://github.com/yaldat/yalda-embedding/blob/main/Yalda-Embedding-Training.ipynb): TPU training, configurable distillation, checkpoint export, and Hugging Face upload.
- [Benchmark.ipynb](https://github.com/yaldat/yalda-embedding/blob/main/Benchmark.ipynb): MTEB evaluation on selected Persian retrieval test tasks.
- [ReadMe.md](https://github.com/yaldat/yalda-embedding/blob/main/ReadMe.md): experiment setup and instructions.

Set `KD_ALPHA = 0.75` in the training notebook to use the final paper configuration.
Keep the model revision, dependency versions, task selection, prompt settings, and exact
score field with each benchmark run. The notebooks expose the development procedure;
the tables above reproduce the paper's reported results.

## Scope and limitations

The reported evaluation covers Persian retrieval. Performance on other languages,
classification, clustering, and other embedding tasks has not been established by these
experiments. The eight-task comparison does not establish SOTA across all model sizes or
benchmarks. Longer-context retrieval quality is not evaluated here.

In-batch negatives can include semantically relevant passages, and distillation can transfer
the teacher's retrieval preferences and errors. Separate source splits are used, but broader
overlap auditing and repeated seeded runs remain areas for further evaluation.

## Acknowledgments

Yalda builds on [Qwen3 Embedding](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B), uses
[F2LLM-v2-8B](https://huggingface.co/codefuse-ai/F2LLM-v2-8B) as its offline teacher, and is
evaluated using [MTEB](https://github.com/embeddings-benchmark/mteb) and
[FaMTEB](https://aclanthology.org/2025.findings-emnlp.614/).

	---
	language:
	- fa
	- eng
	license: apache-2.0
	library_name: sentence-transformers
	pipeline_tag: sentence-similarity
	base_model: Qwen/Qwen3-Embedding-0.6B
	base_model_relation: finetune
	datasets:
	- Yaldat/Persian-Pairwise
	- Yaldat/Persian-Pairwise-Embedded
	tags:
	- persian
	- text-embeddings
	- dense-retrieval
	- semantic-search
	- sentence-transformers
	- transformers
	- knowledge-distillation
	- feature-extraction
	---

	# Yalda Embedding

	Yalda Embedding is an openly developed Persian text embedding model for dense retrieval
	and semantic search. It specializes
	[Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) using more than one
	million Persian query-document pairs and relational knowledge distillation from
	[F2LLM-v2-8B](https://huggingface.co/codefuse-ai/F2LLM-v2-8B).

	With 595.8 million parameters and 1,024-dimensional embeddings, Yalda achieves
	state-of-the-art performance among the evaluated 0.6B models on eight Persian retrieval
	tasks from FaMTEB. Its reported mean is 0.5488, compared with 0.4713 for its Qwen3-0.6B
	initialization and 0.5563 for Qwen3-Embedding-4B. It offers competitive retrieval performance
	against the 4B reference while retaining the 0.6B model's inference footprint.

	The model weights, training data, teacher embeddings, and development notebooks are public:

	- Training and benchmark code: [yaldat/yalda-embedding](https://github.com/yaldat/yalda-embedding)
	- Training data: [Yaldat/Persian-Pairwise](https://huggingface.co/datasets/Yaldat/Persian-Pairwise)
	- Teacher-embedded data: [Yaldat/Persian-Pairwise-Embedded](https://huggingface.co/datasets/Yaldat/Persian-Pairwise-Embedded)

	## Model details

	\| Property \| Value \|
	\| --- \| --- \|
	\| Primary language \| Persian / Farsi \|
	\| Base model \| Qwen3-Embedding-0.6B \|
	\| Parameters \| 595,776,512 \|
	\| Embedding dimension \| 1,024 \|
	\| Architecture \| Decoder-only Transformer, 28 blocks \|
	\| Attention \| Causal, grouped-query attention \|
	\| Pooling \| Last non-padding token \|
	\| Training and evaluation input length \| 512 tokens \|
	\| Similarity \| Cosine similarity; dot product for L2-normalized embeddings \|
	\| Released weights \| FP32 Safetensors \|
	\| License \| Apache 2.0 \|

	## Quick start

	Install Sentence Transformers and its model dependencies:

	```bash
	pip install -U sentence-transformers transformers torch
	```

	Use the saved query prompt for search queries and the document prompt for passages.
	Normalize embeddings so their dot product equals cosine similarity.

	```python
	from sentence_transformers import SentenceTransformer

	model = SentenceTransformer("Yaldat/Yalda-Embedding")
	model.tokenizer.padding_side = "left"
	model.max_seq_length = 512

	queries = [
	"پایتخت ایران کجاست؟",
	"چرا برگ درختان در پاییز تغییر رنگ می‌دهد؟",
	]
	documents = [
	"تهران پایتخت ایران و یکی از بزرگ‌ترین شهرهای این کشور است.",
	"در پاییز با کاهش نور خورشید، کلروفیل برگ‌ها تجزیه می‌شود و رنگدانه‌های دیگر نمایان می‌شوند.",
	"کتابخانه‌ها فضایی برای مطالعه و دسترسی به منابع علمی فراهم می‌کنند.",
	]

	query_embeddings = model.encode(
	queries,
	prompt_name="query",
	normalize_embeddings=True,
	)
	document_embeddings = model.encode(
	documents,
	prompt_name="document",
	normalize_embeddings=True,
	)

	scores = query_embeddings @ document_embeddings.T
	for query, row in zip(queries, scores):
	best_index = int(row.argmax())
	print(f"Query: {query}")
	print(f"Best passage: {documents[best_index]}")
	print(f"Similarity: {row[best_index]:.4f}\n")
	```

	The model has no default prompt, so select `prompt_name="query"` explicitly for queries.
	The saved query prompt is:

	```text
	Instruct: Given a web search query, retrieve relevant passages that answer the query
	Query:
	```

	Documents receive no instruction. Pass raw query text to `encode`; the query prompt is
	added automatically when selected. For larger collections, precompute document embeddings
	and index them in a vector search system.

	The example uses 512 tokens to match the training and evaluation setup. Longer inputs are
	truncated under this setting. The inherited backbone's longer context configuration does
	not establish retrieval quality beyond the evaluated input length.

	## Evaluation

	The results below are reported in **Yalda Embedding: An Engineering Effort to Reach State of
	the Art**. Evaluation uses the test splits of eight Persian retrieval tasks from
	[FaMTEB](https://aclanthology.org/2025.findings-emnlp.614/). The summary is the unweighted
	macro-average of the reported task scores.

	### Comparison with evaluated baselines

	\| Model \| Mean retrieval score \|
	\| --- \| ---: \|
	\| Qwen3-Embedding-4B \| 0.5563 \|
	\| Yalda Embedding \| 0.5488 \|
	\| F2LLM-v2-0.6B \| 0.5063 \|
	\| Dibachain/Diba-Embed \| 0.4788 \|
	\| Qwen3-Embedding-0.6B \| 0.4713 \|
	\| heydariAI/persian-embeddings \| 0.4175 \|
	\| Tooka-SBERT-V2-Large \| 0.4150 \|

	Yalda improves over Qwen3-Embedding-0.6B by 0.0775 absolute (16.4% relative) and
	over F2LLM-v2-0.6B by 0.0425 absolute. It exceeds Diba-Embed by 0.0700 absolute
	(14.6% relative). It scores higher than each of these three baselines on all eight
	tasks at the reported precision.

	Against Qwen3-Embedding-4B, Yalda scores higher on NFCorpus-Fa, SynPerQARetrieval,
	PersianWebDocumentRetrieval, and HotpotQA-FaHardNegatives, and ties on FiQA2018-Fa.
	Its overall mean is 0.0075 below the 4B reference. The SOTA claim is scoped to the
	evaluated 0.6B models and these eight retrieval tasks.

	### Yalda's per-task scores

	\| Task \| Reported score \|
	\| --- \| ---: \|
	\| FiQA2018-Fa \| 0.30 \|
	\| NFCorpus-Fa \| 0.33 \|
	\| SynPerQARetrieval \| 0.88 \|
	\| PersianWebDocumentRetrieval \| 0.57 \|
	\| HotpotQA-FaHardNegatives \| 0.58 \|
	\| MSMARCO-FaHardNegatives \| 0.65 \|
	\| NQ-FaHardNegatives \| 0.43 \|
	\| SciFact-Fa \| 0.65 \|
	\| Macro-average \| 0.5488 \|

	Student training uses query-positive-document pairs, in-batch negatives, and relational
	distillation. It uses no explicitly mined or supplied hard negatives, while achieving the
	reported scores on the three hard-negative evaluation tasks.

	## Training recipe

	The model is fine-tuned on 1,035,981 Persian query-positive-document pairs assembled from
	13 retrieval resources. Training data comes from the source-provided training splits;
	evaluation uses their corresponding test splits. The empirical source mixture is shuffled
	without resampling.

	The shared encoder is optimized with query-to-document InfoNCE and forward KL divergence
	between teacher and student distributions over the documents in each global batch:

	```text
	loss = (1 - alpha) * InfoNCE + alpha * KL(teacher \|\| student)
	alpha = 0.75
	```

	The final recipe weights InfoNCE by 0.25 and distillation by 0.75. Teacher vectors
	are precomputed offline, truncated from 4,096 to their first 1,024 Matryoshka dimensions,
	and L2-normalized before constructing similarity distributions. The method transfers
	ranking relationships rather than matching teacher and student coordinates. The 8B teacher
	is needed for data preparation, not for student training or inference.

	\| Setting \| Value \|
	\| --- \| --- \|
	\| Hardware \| Eight TPU v5e devices \|
	\| Training framework \| JAX 0.9.2 and Flax NNX 0.12.6 \|
	\| Epochs \| 1 \|
	\| Global batch size \| 512 \|
	\| Pairs per device \| 64 \|
	\| Optimization steps \| 2,023 \|
	\| Optimizer \| AdamW \|
	\| Peak learning rate \| 2e-5 \|
	\| Warmup \| First 10% of updates \|
	\| Decay \| Cosine to zero \|
	\| Gradient clipping \| Global norm 1.0 \|
	\| Student / teacher temperature \| 0.05 / 0.07 \|
	\| KD weight \| 0.75 \|
	\| Sequence length \| 512 tokens \|
	\| Precision \| BF16 projections with FP32 parameters, normalization, and losses \|
	\| Memory strategy \| Single-axis FSDP and activation rematerialization \|

	One epoch consumes 1,035,776 pairs; the incomplete final batch of 205 pairs is omitted.
	The model seed is zero; the final experiment does not fix the data permutation seed.

	### Distillation ablation

	\| Configuration \| KD weight \| Mean retrieval score \|
	\| --- \| ---: \| ---: \|
	\| Stabilized InfoNCE, batch size 512 \| 0.00 \| 0.4813 \|
	\| Ablation 4 \| 0.25 \| 0.5125 \|
	\| Ablation 5 \| 0.50 \| 0.5313 \|
	\| Ablation 6: final model \| 0.75 \| 0.5488 \|

	The final distilled configuration improves all eight tasks over the stabilized,
	non-distilled model. Increasing the KD weight from 0.50 to 0.75 adds 0.0175 to the mean.

	## Open development and reproducibility

	The public [GitHub repository](https://github.com/yaldat/yalda-embedding) contains:

	- [Yalda-Embedding-Training.ipynb](https://github.com/yaldat/yalda-embedding/blob/main/Yalda-Embedding-Training.ipynb): TPU training, configurable distillation, checkpoint export, and Hugging Face upload.
	- [Benchmark.ipynb](https://github.com/yaldat/yalda-embedding/blob/main/Benchmark.ipynb): MTEB evaluation on selected Persian retrieval test tasks.
	- [ReadMe.md](https://github.com/yaldat/yalda-embedding/blob/main/ReadMe.md): experiment setup and instructions.

	Set `KD_ALPHA = 0.75` in the training notebook to use the final paper configuration.
	Keep the model revision, dependency versions, task selection, prompt settings, and exact
	score field with each benchmark run. The notebooks expose the development procedure;
	the tables above reproduce the paper's reported results.

	## Scope and limitations

	The reported evaluation covers Persian retrieval. Performance on other languages,
	classification, clustering, and other embedding tasks has not been established by these
	experiments. The eight-task comparison does not establish SOTA across all model sizes or
	benchmarks. Longer-context retrieval quality is not evaluated here.

	In-batch negatives can include semantically relevant passages, and distillation can transfer
	the teacher's retrieval preferences and errors. Separate source splits are used, but broader
	overlap auditing and repeated seeded runs remain areas for further evaluation.

	## Acknowledgments

	Yalda builds on [Qwen3 Embedding](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B), uses
	[F2LLM-v2-8B](https://huggingface.co/codefuse-ai/F2LLM-v2-8B) as its offline teacher, and is
	evaluated using [MTEB](https://github.com/embeddings-benchmark/mteb) and
	[FaMTEB](https://aclanthology.org/2025.findings-emnlp.614/).