# M3-Embedding

## Overview

M3-Embedding is a versatile text embedding model distinguished by its capabilities in three key areas:

- **Multi-Functionality**: Simultaneously performs three retrieval functions - dense retrieval, multi-vector retrieval, and sparse retrieval
- **Multi-Linguality**: Supports over 100 working languages
- **Multi-Granularity**: Processes inputs ranging from short sentences to long documents up to 8192 tokens

## Model Information

| Model | Dimension | Sequence Length | Description |
|:----:|:---:|:---:|:---|
| M3-Embedding | 1024 | 8192 | Multilingual model with unified fine-tuning |

## Key Features

### Retrieval Methods

- **Dense Retrieval**: Maps text into a single embedding vector
- **Sparse Retrieval**: Generates token weights for lexical matching
- **Multi-Vector Retrieval**: Uses multiple vectors to represent text

## Usage

### Installation

```bash
pip install -U FlagEmbedding
```

### Generate Embeddings

**Dense Embedding:**

```python
from FlagEmbedding import BGEM3FlagModel

model = BGEM3FlagModel('BAAI/bge-m3', use_fp16=True)

sentences_1 = ["What is M3?", "Definition of BM25"]
sentences_2 = ["M3 is an embedding model supporting dense retrieval, lexical matching and multi-vector interaction.", 
               "BM25 is a bag-of-words retrieval function"]

embeddings_1 = model.encode(sentences_1, batch_size=12, max_length=8192)['dense_vecs']
embeddings_2 = model.encode(sentences_2)['dense_vecs']
similarity = embeddings_1 @ embeddings_2.T
```

**Sparse Embedding:**

```python
output_1 = model.encode(sentences_1, return_dense=True, return_sparse=True, return_colbert_vecs=False)
output_2 = model.encode(sentences_2, return_dense=True, return_sparse=True, return_colbert_vecs=False)

# View token weights
print(model.convert_id_to_token(output_1['lexical_weights']))

# Compute lexical matching scores
lexical_scores = model.compute_lexical_matching_score(output_1['lexical_weights'][0], output_2['lexical_weights'][0])
```

**Multi-Vector (ColBERT):**

```python
output_1 = model.encode(sentences_1, return_dense=True, return_sparse=True, return_colbert_vecs=True)
output_2 = model.encode(sentences_2, return_dense=True, return_sparse=True, return_colbert_vecs=True)

print(model.colbert_score(output_1['colbert_vecs'][0], output_2['colbert_vecs'][0]))
```

### Compute Scores for Text Pairs

```python
sentence_pairs = [[i,j] for i in sentences_1 for j in sentences_2]

results = model.compute_score(sentence_pairs, 
                            max_passage_length=128,
                            weights_for_different_modes=[0.4, 0.2, 0.4])
```

## Performance Highlights

- Top performer in multilingual embedding benchmarks
- Strong results across 100+ languages
- Excellent performance on both short and long document retrieval
- Competitive with BM25 for lexical matching tasks

## Training Approach

The model incorporates:

- **Self-Knowledge Distillation**: Combining multiple retrieval modes as reward signals
- **Efficient Batching**: Optimized for long text fine-tuning
- **MCLS**: Method for improving long text performance without fine-tuning