README.md

---
license: apache-2.0
language:
- en
- zh
- ru
- es
- fr
- de
- ar
- nl
- vi
- hi
- ko
- ja
- it
- id
- pt
- pl
- tr
- da
- th
- sv
- fa
- uk
- cs
- 'no'
- el
- ca
- ro
- fi
- bg
- tl
- gl
- my
- hy
- km
- ne
- hu
- eu
- he
- lo
- sw
- az
- lv
- si
- sk
- tg
- et
- lt
- ms
- hr
- is
- sl
- sr
- ur
- bn
- af
- ta
- ka
- te
- ml
- mn
- nn
- kk
- cy
- mr
- sq
- nb
- mk
- jv
- kn
- eo
- la
- gu
- uz
- am
- oc
- be
- mg
- vo
- pa
- lb
- ht
- br
- ga
- xh
- tt
- bs
- yo
base_model:
- codefuse-ai/F2LLM-v2-0.6B-Preview
pipeline_tag: feature-extraction
library_name: transformers
tags:
- sentence-transformers
datasets:
- codefuse-ai/F2LLM-v2
---

# ML-Embed-0.6B

**ML-Embed-0.6B** is a multilingual text embedding model developed by CodeFuse AI and trained from Qwen3-0.6B. It is part of the ML-Embed family introduced in the ICML 2026 paper [**ML-Embed: Inclusive and Efficient Embeddings for a Multilingual World**](https://arxiv.org/abs/2605.15081).

This model is designed to be:

- strong multilingual embedding model
- fully compatible with standard Qwen3 loading/inference
- efficient to deploy, thanks to:
  - **Matryoshka Layer Learning (MLL)** for layer truncation
  - **Matryoshka Embedding Learning (MEL)** for factorized embedding deployment
  - **Matryoshka Representation Learning (MRL)** for flexible embedding dimension truncation

The default released checkpoint is in **compatibility mode**, meaning it behaves like a standard Transformer embedding model and can be used directly with `sentence-transformers` or `transformers`.


## Model Highlights

- Base architecture: Qwen3-0.6B
- Embedding dimension: 1024
- Sequence embedding: EOS token representation
- Attention type: causal attention
- Trained in two stages (first-stage checkpiont: [codefuse-ai/F2LLM-v2-0.6B-Preview](https://huggingface.co/codefuse-ai/F2LLM-v2-0.6B-Preview))
- Supports multilingual retrieval and semantic similarity
- Supports efficient deployment with:
  - fewer transformer layers
  - factorized embedding matrices (`U.pth`, `V.pth`)


## Quick Start

### With Sentence Transformers

To encode text with the [Sentence Transformers](https://www.sbert.net/) library:

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "codefuse-ai/ML-Embed-0.6B",
    device="cuda:0",
    model_kwargs={"torch_dtype": "bfloat16"}
)

# Some sample query and documents
query = "What is ML-Embed used for?"
documents = [
    "ML-Embed is a family of multilingual embedding models for retrieval, semantic search, and other NLP tasks.",
    "ML-Embed is trained to produce text embeddings that work well across many languages.",
    "ML-Embed 是 CodeFuse AI 开源的多语言嵌入模型。",
    "ML-Embed — это многоязычная модель эмбеддингов для поиска и семантического сопоставления."
]

# Encode the query and documents separately. The encode_query method uses the query prompt
query_embedding = model.encode_query(query)
document_embeddings = model.encode_document(documents)

print(query_embedding.shape, document_embeddings.shape)
# (1024,) (4, 1024)

# Compute cosine similarity between the query and documents
similarity = model.similarity(query_embedding, document_embeddings)
print(similarity)
```

### With Transformers

Or directly with the [Transformers](https://huggingface.co/docs/transformers/index) library:

```python
from transformers import AutoModel, AutoTokenizer
import torch
import torch.nn.functional as F

model_path = "codefuse-ai/ML-Embed-0.6B"

tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModel.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map={"": 0}
)

query = "What is ML-Embed used for?"
query_prompt = "Instruct: Given a question, retrieve passages that can help answer the question.\nQuery: "

documents = [
    "ML-Embed is a family of multilingual embedding models for retrieval, semantic search, and other NLP tasks.",
    "ML-Embed is trained to produce text embeddings that work well across many languages.",
    "ML-Embed 是 CodeFuse AI 开源的多语言嵌入模型。",
    "ML-Embed — это многоязычная модель эмбеддингов для поиска и семантического сопоставления."
]

def encode(sentences):
    batch_size = len(sentences)
    tokenized_inputs = tokenizer(sentences, padding=True, return_tensors="pt").to(model.device)
    last_hidden_state = model(**tokenized_inputs).last_hidden_state
    eos_positions = tokenized_inputs.attention_mask.sum(dim=1) - 1
    embeddings = last_hidden_state[torch.arange(batch_size, device=model.device), eos_positions]
    embeddings = F.normalize(embeddings, p=2, dim=1)
    return embeddings

# Encode the query and documents
query_embedding = encode([query_prompt + query])
document_embeddings = encode(documents)

print(query_embedding.shape, document_embeddings.shape)
# torch.Size([1, 1024]) torch.Size([4, 1024])

# Compute cosine similarity between the query and documents
similarity = query_embedding @ document_embeddings.T
print(similarity)
```


## Prompts

The model supports custom instructions in the following format:

```text
Instruct: your_instruction
Query:
```

In general, for retrieval and reranking tasks:

- use the prompt for **queries**
- do **not** prepend the prompt to **documents/passages**

For symmetric tasks such as STS, clustering, and bitext mining, you can encode the documents either with or without prompts. The model is trained to support both scenarios.


## Efficient Deployment

ML-Embed-0.6B was trained with **3D Matryoshka Learning**, including:

- **MLL**: Matryoshka Layer Learning
- **MEL**: Matryoshka Embedding Learning
- **MRL**: Matryoshka Representation Learning

This enables multiple deployment modes.

### 1. Compatibility Mode

The default released checkpoint is a standard Qwen3-compatible model. You can load it normally with `AutoModel` or `SentenceTransformer` without any code changes (refer to the examples above).

This is the recommended option if you want the simplest integration.

### 2. Fewer-Layer Deployment with MLL

This model was trained so that **shallower versions remain useful**.  
If you want to save memory or compute, you can deploy a smaller model by editing:

- `num_hidden_layers`
- `max_window_layers`

in `config.json` to a value smaller than the current one.

For example, changing both values from `28` to `16` will make `transformers` load only the first 16 layers and ignore the remaining weights.

This works automatically with the Hugging Face `transformers` library.

> Note: make sure `num_hidden_layers` and `max_window_layers` stay consistent. If you are using transformers v5, you will also need to trucate `layer_types` in the config file according to the new layer count.

### 3. Factorized Embedding Deployment with MEL

This repository also provides two additional weight files:

- `U.pth`
- `V.pth`

These correspond to the factorized embedding layer.

Instead of using the full embedding matrix, you can reconstruct token embeddings from:

$$
E \approx U V
$$

This can reduce storage and memory usage for the embedding layer, and it also allows more aggressive low-rank deployment if desired.

Below is a simple inference example adapted from the training code.

#### Inference demo using `U.pth` and `V.pth`

```python
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
from huggingface_hub import hf_hub_download

model_path = "codefuse-ai/ML-Embed-0.6B"
dtype = torch.bfloat16
device = "cuda"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModel.from_pretrained(
    model_path,
    torch_dtype=dtype,
    device_map={"": 0}
)
model.eval()

# Load factorized embedding weights
u_path = hf_hub_download(repo_id=model_path, filename="U.pth")
v_path = hf_hub_download(repo_id=model_path, filename="V.pth")

U = torch.load(u_path, map_location="cpu").to(dtype).to(device)
V = torch.load(v_path, map_location="cpu").to(dtype).to(device)

# Optional: choose a smaller rank for more compression
# If rank is None, use the full factorized rank
rank = None

def encode_with_factorized_embedding(sentences, rank=None):
    tokenized = tokenizer(sentences, padding=True, return_tensors="pt").to(device)
    input_ids = tokenized["input_ids"]
    attention_mask = tokenized["attention_mask"]

    if rank is None:
        inputs_embeds = (U @ V)[input_ids]
    else:
        inputs_embeds = (U[:, :rank] @ V[:rank, :])[input_ids]

    outputs = model(
        inputs_embeds=inputs_embeds,
        attention_mask=attention_mask
    )

    last_hidden_state = outputs.last_hidden_state
    eos_positions = attention_mask.sum(dim=1) - 1
    embeddings = last_hidden_state[torch.arange(len(sentences), device=device), eos_positions]
    embeddings = F.normalize(embeddings, p=2, dim=1)
    return embeddings

query = "What is ML-Embed used for?"
query_prompt = "Instruct: Given a question, retrieve passages that can help answer the question.\nQuery: "

documents = [
    "ML-Embed is a family of multilingual embedding models for retrieval, semantic search, and other NLP tasks.",
    "ML-Embed is trained to produce text embeddings that work well across many languages.",
    "ML-Embed 是 CodeFuse AI 开源的多语言嵌入模型。",
    "ML-Embed — это многоязычная модель эмбеддингов для поиска и семантического сопоставления."
]

query_embedding = encode_with_factorized_embedding([query_prompt + query], rank=rank)
document_embeddings = encode_with_factorized_embedding(documents, rank=rank)

similarity = query_embedding @ document_embeddings.T
print(similarity)
```


## Matryoshka Representation Learning

This model was also trained with MRL, which means the embedding is trained to support prefix truncation.

The full embedding size is 1024, but users may also experiment with keeping only the first `d` dimensions. This can reduce storage and speed up vector search in downstream systems. The model is trained with a smallest Matryoshka dimension of 8.

Example:

```python
embedding = embedding[..., :512]
embedding = torch.nn.functional.normalize(embedding, p=2, dim=-1)
```

> Note: you need to apply normalization **after** trucation, not the other way around.

If you are using sentence transformer, you can also simply pass `truncate_dim=512` to the encode interface.

## Training

ML-Embed-0.6B is trained in two stages from Qwen3-0.6B.
- Stage 1 includes large-scale retrieval-focused data to build strong semantic embedding capability.
- Stage 2 uses instruction-aware fine-tuning on a diverse multilingual mixture of tasks.

The model is fully open:
- Training data is available at: [`codefuse-ai/F2LLM-v2`](https://huggingface.co/datasets/codefuse-ai/F2LLM-v2)
- Training code is available at: https://github.com/codefuse-ai/CodeFuse-Embeddings/tree/main/ML-Embed

## Intended Uses

ML-Embed-0.6B can be used for:

- semantic search
- dense retrieval
- RAG pipelines
- clustering
- classification
- duplicate detection
- cross-lingual retrieval
- multilingual similarity search


## Limitations

- Performance may vary across languages, domains, and task formats.
- Benchmark performance does not guarantee optimal behavior in all production retrieval or RAG systems.
- For best embedding quality, use appropriate prompts for queries.
- Efficient modes such as fewer-layer deployment and low-rank factorized embeddings involve trade-offs between quality and efficiency.

### Citation (to be updated after the conference)

```bibtex
@misc{zhang2026mlembedinclusiveefficientembeddings,
      title={ML-Embed: Inclusive and Efficient Embeddings for a Multilingual World}, 
      author={Ziyin Zhang and Zihan Liao and Hang Yu and Peng Di and Rui Wang},
      year={2026},
      eprint={2605.15081},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.15081}, 
}
```

## Related models

The baseline models are released and submitted to the MTEB leaderboard under the name [`F2LLM-v2`](https://huggingface.co/collections/codefuse-ai/f2llm):

| Model | Base                                                                                | Instruct                                                            |
| ----- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| 80M   |                                                                                     | [🤗F2LLM-v2-80M](https://huggingface.co/codefuse-ai/F2LLM-v2-80M)   |
| 160M  |                                                                                     | [🤗F2LLM-v2-160M](https://huggingface.co/codefuse-ai/F2LLM-v2-160M) |
| 330M  |                                                                                     | [🤗F2LLM-v2-330M](https://huggingface.co/codefuse-ai/F2LLM-v2-330M) |
| 0.6B  | [🤗F2LLM-v2-0.6B-Preview](https://huggingface.co/codefuse-ai/F2LLM-v2-0.6B-Preview) | [🤗F2LLM-v2-0.6B](https://huggingface.co/codefuse-ai/F2LLM-v2-0.6B) |
| 1.7B  | [🤗F2LLM-v2-1.7B-Preview](https://huggingface.co/codefuse-ai/F2LLM-v2-1.7B-Preview) | [🤗F2LLM-v2-1.7B](https://huggingface.co/codefuse-ai/F2LLM-v2-1.7B) |
| 4B    | [🤗F2LLM-v2-4B-Preview](https://huggingface.co/codefuse-ai/F2LLM-v2-4B-Preview)     | [🤗F2LLM-v2-4B](https://huggingface.co/codefuse-ai/F2LLM-v2-4B)     |
| 8B    | [🤗F2LLM-v2-8B-Preview](https://huggingface.co/codefuse-ai/F2LLM-v2-8B-Preview)     | [🤗F2LLM-v2-8B](https://huggingface.co/codefuse-ai/F2LLM-v2-8B)     |
| 14B   | [🤗F2LLM-v2-14B-Preview](https://huggingface.co/codefuse-ai/F2LLM-v2-14B-Preview)   | [🤗F2LLM-v2-14B](https://huggingface.co/codefuse-ai/F2LLM-v2-14B)   |