File size: 3,126 Bytes
5fc9ffd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | # M3-Embedding
## Overview
M3-Embedding is a versatile text embedding model distinguished by its capabilities in three key areas:
- **Multi-Functionality**: Simultaneously performs three retrieval functions - dense retrieval, multi-vector retrieval, and sparse retrieval
- **Multi-Linguality**: Supports over 100 working languages
- **Multi-Granularity**: Processes inputs ranging from short sentences to long documents up to 8192 tokens
## Model Information
| Model | Dimension | Sequence Length | Description |
|:----:|:---:|:---:|:---|
| M3-Embedding | 1024 | 8192 | Multilingual model with unified fine-tuning |
## Key Features
### Retrieval Methods
- **Dense Retrieval**: Maps text into a single embedding vector
- **Sparse Retrieval**: Generates token weights for lexical matching
- **Multi-Vector Retrieval**: Uses multiple vectors to represent text
## Usage
### Installation
```bash
pip install -U FlagEmbedding
```
### Generate Embeddings
**Dense Embedding:**
```python
from FlagEmbedding import BGEM3FlagModel
model = BGEM3FlagModel('BAAI/bge-m3', use_fp16=True)
sentences_1 = ["What is M3?", "Definition of BM25"]
sentences_2 = ["M3 is an embedding model supporting dense retrieval, lexical matching and multi-vector interaction.",
"BM25 is a bag-of-words retrieval function"]
embeddings_1 = model.encode(sentences_1, batch_size=12, max_length=8192)['dense_vecs']
embeddings_2 = model.encode(sentences_2)['dense_vecs']
similarity = embeddings_1 @ embeddings_2.T
```
**Sparse Embedding:**
```python
output_1 = model.encode(sentences_1, return_dense=True, return_sparse=True, return_colbert_vecs=False)
output_2 = model.encode(sentences_2, return_dense=True, return_sparse=True, return_colbert_vecs=False)
# View token weights
print(model.convert_id_to_token(output_1['lexical_weights']))
# Compute lexical matching scores
lexical_scores = model.compute_lexical_matching_score(output_1['lexical_weights'][0], output_2['lexical_weights'][0])
```
**Multi-Vector (ColBERT):**
```python
output_1 = model.encode(sentences_1, return_dense=True, return_sparse=True, return_colbert_vecs=True)
output_2 = model.encode(sentences_2, return_dense=True, return_sparse=True, return_colbert_vecs=True)
print(model.colbert_score(output_1['colbert_vecs'][0], output_2['colbert_vecs'][0]))
```
### Compute Scores for Text Pairs
```python
sentence_pairs = [[i,j] for i in sentences_1 for j in sentences_2]
results = model.compute_score(sentence_pairs,
max_passage_length=128,
weights_for_different_modes=[0.4, 0.2, 0.4])
```
## Performance Highlights
- Top performer in multilingual embedding benchmarks
- Strong results across 100+ languages
- Excellent performance on both short and long document retrieval
- Competitive with BM25 for lexical matching tasks
## Training Approach
The model incorporates:
- **Self-Knowledge Distillation**: Combining multiple retrieval modes as reward signals
- **Efficient Batching**: Optimized for long text fine-tuning
- **MCLS**: Method for improving long text performance without fine-tuning
|