File size: 9,293 Bytes
f7bd684 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 | ---
language:
- de
- en
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- feature-extraction
- retrieval
- german
- mteb
base_model: nvidia/Nemotron-3-Embed-1B-BF16
license: cc-by-nc-4.0
---
# most-embed-de: A German Retrieval-Optimized Embedding Model
**most-embed-de** is a 1.1B-parameter German text-embedding model optimized for the **retrieval in German customer-support applications**. It acts as the
retriever behind a support RAG system, an FAQ/help-center search, or an agent that has to find the one
passage in a company's own documentation that resolves a customer's request.
It is a fine-tune of **[nvidia/Nemotron-3-Embed-1B-BF16](https://huggingface.co/nvidia/Nemotron-3-Embed-1B-BF16)**
(2048-dim embeddings, up to 32k context). Among models it can be deployed next to, it is:
- **#1 in its size class (≤1.5B) on the German retrieval + reranking cut of MTEB(deu)** — and it beats
the 7.6B `Qwen3-Embedding-8B` and the 14B `F2LLM-v2-14B` while being **~7–12× smaller**.
- **#1 in its size class on customer-support retrieval** (our SupportIR-DE benchmark), second only to the Nemotron-3-Embed
7.6B model and again ahead of `Qwen3-Embedding-8B` — at a fraction of the serving cost.
- Target language: **German**
- Base model languages: English, Arabic, Assamese, Bengali, Bulgarian, Chinese, Danish, Dutch, Finnish, French, Hindi, Hinglish, Indonesian, Italian, Japanese, Korean, Malay, Marathi, Nepali, Norwegian, Persian, Portuguese, Romanian, Russian, Spanish, Swahili, Swedish, Tamil, Telugu, Thai, Ukrainian, Urdu, Vietnamese.
At 1.1B parameters it runs comfortably on a single commodity GPU, so you get 7–14B-class German support
retrieval quality at ~1B-class latency and cost.
## Why this model exists
Public German retrieval benchmarks like the German slice of MTEB and RTEB are dominated by
**Wikipedia-style open-domain QA and legal question answering**. They are excellent academic tasks, but
they are **not** what many real world applications such a customer-support RAG systems need. A support retriever has to:
- work **within a single company's index** (a query about *your* return policy must not match a
*competitor's* return policy), and
- handle the real register of support queries — terse keyword searches, full questions, and statement-style
descriptions of a problem — across very different **industries** (telco, banking, insurance, e-commerce),
each with its own vocabulary and document conventions.
None of the public German benchmarks measure this. So we built one.
## Evaluation
### German Retrieval
German Retrieval is essentially the [**Retrieval + Reranking subset of MTEB(deu)**](https://mteb-leaderboard.hf.space/) — LegalQuAD,
GerDaLIR (small, for efficiency), GermanDPR, GermanQuAD, XMarket, and MIRACL(de) — scored with the
official MTEB methodology (mean of nDCG@10, MRR@5 for GermanQuAD). Baselines are the published leaderboard
numbers; most-embed-de is measured on the identical task suite.
### SupportIR-DE — customer-support retrieval
SupportIR-DE scores **company-scoped nDCG@10**: every query is ranked only against *its own company's*
documents — the index a real support RAG system actually searches — across four industries, **held out by
company in 5-fold cross-validation** (mean ± std over folds).
Because no public German benchmark reflects support retrieval, we designed **SupportIR-DE** as part of
this project:
- **Four industries** — telecommunications, retail banking & payments, insurance, and e-commerce /
omni-channel retail — each with real German help-center content from public Web pages and sourced from the [Common Crawl corpus](https://commoncrawl.org).
- **Company-scoped task** — every query is evaluated against only its own company's document index
(brand/company disambiguation is part of the task, exactly as in deployment). A pooled-industry variant
is reported as a diagnostic.
- **Multiple query registers** — natural questions, keyword searches, and statement-style problem
descriptions, so a model cannot look strong on one register alone.
- **Held out by company** (k-fold cross-validation over companies) with confidence intervals, plus a
human-authored FAQ gold slice for judge calibration.
SupportIR-DE is **not yet public**.
## How it was built — autonomous, agent-driven development
The distinguishing feature of this project is not a single training trick; it is the **development
process**. most-embed-de was produced by an **autonomous, agent-driven research pipeline** that carried the
work through the *entire* model-building lifecycle:
- **literature and landscape research** (what German retrieval benchmarks exist, where they fall short),
- **benchmark selection and design** (choosing the public evaluation cut; designing SupportIR-DE),
- **data curation** (sourcing and cleaning German support corpora; generating and quality-judging training
data; contamination and leakage guards),
- **experimental design** (choosing the base model, the training and retention mix, and the ablations),
- **scheduling and operating training + evaluation jobs on a GPU cluster**, and
- **evaluation and error analysis** (including catching a train/test leakage issue in a retention pool and
correcting the reported numbers).
The approach was inspired by [Andrej Karpathy's autoresearch project](https://github.com/karpathy/autoresearch). A human
directed the objectives and reviewed every decision gate; the agents did the research, engineering, and
analysis in between. The result is a stat
## Usage
The model can be used with all frameworks and inference systems that support Nemotron-3-Embed:
### Sentence Transformers
```python
import torch
from sentence_transformers import SentenceTransformer
MODEL_ID = "malteos/most-embed-de"
model = SentenceTransformer(
MODEL_ID,
device="cuda",
model_kwargs={
"dtype": torch.bfloat16,
"attn_implementation": "flash_attention_2", # optional
},
)
model.max_seq_length = 32768
QUERIES = [
"Write a Python function that counts the frequency of each element in a list of lists.",
"Write a function that orders a dictionary with tuple keys by the product of each key's tuple values.",
"What symptoms and common triggers help distinguish eczema from other inflammatory skin conditions?",
"How can someone reduce exposure to pollen during allergy season?",
]
DOCUMENTS = [
"def frequency_lists(list1):\n flattened = [item for sublist in list1 for item in sublist]\n counts = {}\n for item in flattened:\n if item in counts:\n counts[item] += 1\n else:\n counts[item] = 1\n return counts",
"def sort_dict_item(test_dict):\n return {key: test_dict[key] for key in sorted(test_dict.keys(), key=lambda ele: ele[0] * ele[1])}",
"Eczema commonly causes itchy, dry, inflamed patches of skin. The affected areas may look red, scaly, cracked, or darker than the surrounding skin depending on skin tone. Symptoms can flare after exposure to irritants, allergens, stress, or changes in weather.",
"People with pollen allergy can reduce exposure by staying indoors on dry, windy days, avoiding early-morning outdoor activity, and going outside after rain when pollen levels are lower. They should check pollen forecasts, close windows and doors when counts are high, and consider starting allergy medication before symptoms begin if high pollen is expected. After being outside, showering, changing clothes, avoiding outdoor laundry drying, and wearing a face mask for yard work can help limit pollen contact.",
]
query_embeddings = model.encode_query(QUERIES, batch_size=8, convert_to_tensor=True)
document_embeddings = model.encode_document(DOCUMENTS, batch_size=8, convert_to_tensor=True)
scores = model.similarity(query_embeddings, document_embeddings)
print("Similarity scores:")
print(f"{'':>4}" + "".join(f"d[{i}]".rjust(10) for i in range(scores.shape[1])))
for query_index, row in enumerate(scores):
print(f"q[{query_index}]" + "".join(f"{score.item():>10.4f}" for score in row))
```
### vLLM Online Serving
```bash
MODEL_ID=malteos/most-embed-de
vllm serve "$MODEL_ID"
```
vLLM defaults to port 8000. Add host and port when you need an explicit bind address or a non-default port:
```bash
vllm serve "$MODEL_ID" --host 0.0.0.0 --port 8000
```
For other frameworks, see the [usage section in the Nemotron model card](https://huggingface.co/nvidia/Nemotron-3-Embed-1B-BF16#usage).
## License
**CC-BY-NC-4.0** — free for research and non-commercial use with attribution. [Reach out for commercial use or extended support](https://ostendorff.org/contact/). |