Update model card slightly (#2)
Browse files- Update model card slightly (10c3f19872f1e5b090ea7ca44b33592bf34f08b3)
Co-authored-by: Tom Aarsen <tomaarsen@users.noreply.huggingface.co>
README.md
CHANGED
|
@@ -114,7 +114,7 @@ language:
|
|
| 114 |
# nomic-embed-text-v2-moe: Multilingual Mixture of Experts Text Embeddings
|
| 115 |
|
| 116 |
## Model Overview
|
| 117 |
-
nomic-embed-text-v2-moe is SoTA multilingual MoE text embedding model:
|
| 118 |
|
| 119 |
- **High Performance**: SoTA Multilingual performance compared to ~300M parameter models, competitive with models 2x in size
|
| 120 |
- **Multilinguality**: Supports ~100 languages and trained over 1.6B pairs
|
|
@@ -157,14 +157,15 @@ For best performance on GPU, please install
|
|
| 157 |
pip install torch transformers einops git+https://github.com/nomic-ai/megablocks.git
|
| 158 |
```
|
| 159 |
|
| 160 |
-
|
|
|
|
|
|
|
| 161 |
|
| 162 |
-
|
| 163 |
|
| 164 |
-
|
| 165 |
|
| 166 |
-
|
| 167 |
-
If using Transformers, **make sure to prepend the task instruction prefix**
|
| 168 |
|
| 169 |
```python
|
| 170 |
import torch
|
|
@@ -187,11 +188,17 @@ with torch.no_grad():
|
|
| 187 |
model_output = model(**encoded_input)
|
| 188 |
embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
|
| 189 |
embeddings = F.normalize(embeddings, p=2, dim=1)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 190 |
```
|
| 191 |
|
| 192 |
-
|
| 193 |
|
| 194 |
-
With SentenceTransformers, you can specify the prompt_name
|
| 195 |
|
| 196 |
```python
|
| 197 |
from sentence_transformers import SentenceTransformer
|
|
@@ -199,6 +206,12 @@ from sentence_transformers import SentenceTransformer
|
|
| 199 |
model = SentenceTransformer("nomic-ai/nomic-embed-text-v2-moe", trust_remote_code=True)
|
| 200 |
sentences = ["Hello!", "¡Hola!"]
|
| 201 |
embeddings = model.encode(sentences, prompt_name="passage")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 202 |
```
|
| 203 |
|
| 204 |
## Performance
|
|
@@ -221,7 +234,7 @@ nomic-embed-text-v2-moe performance on BEIR at 768 dimension and truncated to 25
|
|
| 221 |
## Limitations
|
| 222 |
- Performance may vary across different languages
|
| 223 |
- Resource requirements may be higher than traditional dense models due to MoE architecture
|
| 224 |
-
- Must
|
| 225 |
|
| 226 |
## Training Details
|
| 227 |
|
|
|
|
| 114 |
# nomic-embed-text-v2-moe: Multilingual Mixture of Experts Text Embeddings
|
| 115 |
|
| 116 |
## Model Overview
|
| 117 |
+
`nomic-embed-text-v2-moe` is SoTA multilingual MoE text embedding model:
|
| 118 |
|
| 119 |
- **High Performance**: SoTA Multilingual performance compared to ~300M parameter models, competitive with models 2x in size
|
| 120 |
- **Multilinguality**: Supports ~100 languages and trained over 1.6B pairs
|
|
|
|
| 157 |
pip install torch transformers einops git+https://github.com/nomic-ai/megablocks.git
|
| 158 |
```
|
| 159 |
|
| 160 |
+
> [!IMPORTANT]
|
| 161 |
+
> **Important!**
|
| 162 |
+
> The text prompt *must* include a *task instruction prefix*, instructing the model which task is being performed.
|
| 163 |
|
| 164 |
+
Please use `search_query: ` before your queries/questions, and `search_document: ` before your documents.
|
| 165 |
|
| 166 |
+
### Transformers
|
| 167 |
|
| 168 |
+
If using Transformers, **make sure to prepend the task instruction prefix**.
|
|
|
|
| 169 |
|
| 170 |
```python
|
| 171 |
import torch
|
|
|
|
| 188 |
model_output = model(**encoded_input)
|
| 189 |
embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
|
| 190 |
embeddings = F.normalize(embeddings, p=2, dim=1)
|
| 191 |
+
print(embeddings.shape)
|
| 192 |
+
# torch.Size([2, 768])
|
| 193 |
+
|
| 194 |
+
similarity = F.cosine_similarity(embeddings[0], embeddings[1])
|
| 195 |
+
print(similarity)
|
| 196 |
+
# tensor(0.9118)
|
| 197 |
```
|
| 198 |
|
| 199 |
+
### SentenceTransformers
|
| 200 |
|
| 201 |
+
With SentenceTransformers, you can specify the `prompt_name` as either `"query"` or `"passage"`, and the task instruction will be included automatically.
|
| 202 |
|
| 203 |
```python
|
| 204 |
from sentence_transformers import SentenceTransformer
|
|
|
|
| 206 |
model = SentenceTransformer("nomic-ai/nomic-embed-text-v2-moe", trust_remote_code=True)
|
| 207 |
sentences = ["Hello!", "¡Hola!"]
|
| 208 |
embeddings = model.encode(sentences, prompt_name="passage")
|
| 209 |
+
print(embeddings.shape)
|
| 210 |
+
# (2, 768)
|
| 211 |
+
|
| 212 |
+
similarity = model.similarity(embeddings[0], embeddings[1])
|
| 213 |
+
print(similarity)
|
| 214 |
+
# tensor([[0.9118]])
|
| 215 |
```
|
| 216 |
|
| 217 |
## Performance
|
|
|
|
| 234 |
## Limitations
|
| 235 |
- Performance may vary across different languages
|
| 236 |
- Resource requirements may be higher than traditional dense models due to MoE architecture
|
| 237 |
+
- Must use `trust_remote_code=True` when loading the model to use our custom architecture implementation
|
| 238 |
|
| 239 |
## Training Details
|
| 240 |
|