README.md

---
language:
  - zh
  - en
license: apache-2.0
library_name: sentence-transformers
tags:
  - text-embedding
  - sentence-transformers
  - feature-extraction
  - chinese
  - minimind
  - qwen3-embedding
  - matryoshka
base_model: jingyaogong/minimind-3
pipeline_tag: feature-extraction
---

<div align="center">

<img src="./images/banner.png" alt="MiniMind-Embedding" width="100%">

</div>


# MiniMind-Embedding-Dense

<div align="center">

![params](https://img.shields.io/badge/params-64M-blue)
![dim](https://img.shields.io/badge/dim-768-green)
![max_seq](https://img.shields.io/badge/max__seq-8192-orange)
![STS](https://img.shields.io/badge/C_MTEB%20STS-0.48-brightgreen)

</div>

A text embedding model built on the [MiniMind-3](https://huggingface.co/jingyaogong/minimind-3-pytorch) small language model (64M params), with the training pipeline fully aligned with [Qwen3-Embedding](https://arxiv.org/abs/2506.05176). Part of the [minimind-embedding](https://github.com/muzian666/minimind-embedding) project.

## 📌 Introduction

This model turns the 64M MiniMind-3 causal language model into a text encoder using three key techniques:

### 1. Last-Token Pooling

> 💡 *Think of a sentence as a queue of people. The last person (EOS) has "heard" everyone before them through causal attention, so they alone can summarize the whole sentence.*

<div align="center">

![Last-Token Pooling](./images/concept_last_token_pool.png)

</div>

Instead of BERT-style bidirectional attention + mean pooling, we keep **causal (unidirectional) attention** and take the hidden state at the appended EOS token. The last token aggregates information from all preceding tokens via causal attention, naturally serving as a sentence summary. This maximizes reuse of pretrained weights.

### 2. InfoNCE + False-Negative Masking

> 💡 *Contrastive learning is like a "find your friend" game: push the correct match closer, push strangers further away. But what if a "stranger" is actually your friend (false negative)? We detect and mask them!*

<div align="center">

![InfoNCE Contrastive Learning](./images/concept_infonce.png)

</div>

The core loss is InfoNCE (contrastive). Qwen3-Embedding adds a **false-negative mask**: if a candidate negative is *more* similar to the query than the positive (beyond margin=0.1), it's likely a false negative and gets masked out.

<div align="center">

![False-Negative Masking](./images/concept_false_neg_mask.png)

</div>

### 3. Matryoshka Representation Learning (MRL)

> 💡 *Like Russian nesting dolls — the big doll (768-dim) contains smaller dolls (512/256/128/64-dim), each fully functional at its size.*

<div align="center">

![Matryoshka MRL](./images/concept_mrl.png)

</div>

Trained so the vector works at any truncated dimension. Use 64-dim for fast coarse retrieval, 768-dim for precise re-ranking.

## 🏗️ Model Details

| Item | Value |
|------|-------|
| Architecture | MiniMind-3 (causal decoder-only Transformer) |
| Parameters | 64M |
| Hidden size | 768 |
| Layers | 8 |
| Attention heads | 8 (query) / 4 (KV, GQA) |
| Vocab size | 6400 |
| Embedding output dim | 768 (MRL: truncatable to 512/256/128/64) |
| Max sequence length | 8192 (backbone supports 32768) |
| Pooling | last-token |
| Features | L2-normalized; MRL multi-dim |

## 🎓 Training Details (3 Stages)

| Stage | Data | Loss | Steps | Final Loss | Status |
|-------|------|------|-------|------------|--------|
| **1. Weakly-supervised** | T2Ranking triplets (90k) | Standard InfoNCE | 1,413 | 0.43 | ✅ Done |
| **2. Supervised** | T2Ranking-15 (340k, 15 hard negs) | InfoNCE + false-neg mask + MRL | 63,768 (3 epochs) | 0.72 | ✅ Done |
| **3. Model merging** | Last 5 Stage-2 checkpoints | SLERP | — | — | ✅ Done |

<div align="center">

![Training Loss Curves](./images/full_training_loss.png)

*Stage 1 (left): 1.16→0.43. Stage 2 (right): 1.47→0.72, 3 epochs with epoch boundaries marked.*

</div>

| Hyperparameter | Value |
|----------------|-------|
| Temperature τ | 0.02 |
| False-neg margin | 0.1 |
| Hard negatives | 7 per query |
| Batch size | 16 |
| Learning rate | 1e-5 (Stage 2), 2e-4 (Stage 1) |
| MRL dims | [768, 512, 256, 128, 64] |

## 📊 Evaluation

<div align="center">

![C-MTEB STS Scores](./images/sts_scores.png)

</div>

| Task | Samples | Score |
|------|---------|:-----:|
| ATEC | 20,000 | 0.265 |
| BQ | 10,000 | 0.379 |
| LCQMC | 12,500 | 0.631 |
| STSB | 1,361 | 0.637 |
| **Average** | | **0.478** |

<div align="center">

![Model Comparison](./images/model_comparison.png)

</div>

| Model | Params | C-MTEB STS |
|-------|--------|:----------:|
| **This model** | **64M** | **0.478** |
| BGE-small-zh | 24M | ~0.60 |
| BGE-large-zh | 326M | ~0.66 |
| Qwen3-Embedding-0.6B | 600M | ~0.66 |

> **Honest note**: scores are bounded by the small vocab (6400) and limited Stage-1 data (90k vs Qwen3's 150M). The value is the **complete, reproducible** Qwen3-Embedding pipeline.

## 🚀 Usage

### sentence-transformers

```python
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer("Muzian/minimind-embedding-dense")

documents = ["天空呈蓝色是因为阳光散射。", "红烧肉的做法是..."]
doc_embs = model.encode(documents, normalize_embeddings=True)

queries = ["天空为什么是蓝色的"]
query_embs = model.encode(queries, prompt_name="query", normalize_embeddings=True)

scores = cos_sim(query_embs, doc_embs)
```

### Matryoshka (truncate dimensions)

```python
# Use only first 256 dims (smaller index, faster search)
emb_256 = embedding[:, :256]
emb_256 = F.normalize(emb_256, p=2, dim=1)
```

## ⚠️ Limitations

1. **Small vocabulary (6400)** — weak Chinese compression, primary bottleneck
2. **Limited Stage-1 data** (90k vs Qwen3's 150M) — scaling is the top improvement direction
3. SLERP merging shows minimal gain on small models (Stage2 ≈ Stage3)

## 🙏 Acknowledgements

- Base: [MiniMind](https://github.com/jingyaogong/minimind) by @jingyaogong
- Route: [Qwen3-Embedding](https://arxiv.org/abs/2506.05176)
- Data: [T2Ranking](https://huggingface.co/datasets/THUIR/T2Ranking)
- Eval: [MTEB](https://github.com/embeddings-benchmark/mteb)

## 📄 License

Apache 2.0