README.md

---
base_model: intfloat/multilingual-e5-large
library_name: model2vec
license: mit
language:
- de
- multilingual
tags:
- embeddings
- static-embeddings
- sentence-transformers
- model2vec
- german
- education
---

# m2v-e5-large-edu — static embeddings for educational content

A fast, **torch-free static embedding model** tuned for **educational metadata**. It is a
[Model2Vec](https://github.com/MinishLab/model2vec) distillation of
**[intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large)**
enriched with a domain vocabulary of curated **education-domain keyword tags**, so that
subject terms and multi-word phrases (e.g. *"Quadratische Gleichung"*, *"Erneuerbare
Energien"*) are represented as atomic units.

The base model is a **multilingual** XLM-RoBERTa-large, so the model still embeds many
languages; the added domain vocabulary is **primarily German**, so the strongest gains are
on German educational text. Inference is **numpy-only — no PyTorch** — and fast: in our CPU
benchmark it encoded **~7000 texts/s** in a single process.

## How it was built

- **Base model:** [`intfloat/multilingual-e5-large`](https://huggingface.co/intfloat/multilingual-e5-large)
  (a strong multilingual retrieval encoder, XLM-RoBERTa-large).
- **Distillation:** [`model2vec.distill`](https://github.com/MinishLab/model2vec) with
  **mean pooling** (e5's native pooling — this choice has a large effect on quality),
  `pca_dims=512`, `float32`, and SIF weighting (`sif_coefficient=1e-4`, Zipf-based frequency
  down-weighting of common tokens).
- **Domain vocabulary:** ~70,000 cleaned keyword tags (document frequency ≥ 3) extracted
  from the keyword field of ~340k educational resources — mostly German subject terms —
  added on top of the base subword vocabulary. Subword fallback is kept for out-of-vocabulary text.

## Evaluation

Evaluated on a German **educational subject-classification** task and compared against other
embedding models.

**Methodology (identical across models — only the embedding model differs):** 25,068
educational items with 47 subject labels and a fixed train / validation / **held-out test**
split. Each embedding model is frozen; a small MLP classification head is trained on top
(Optuna hyperparameter search on validation), per-label decision thresholds are tuned on
validation, and F1 is reported on the **held-out test split** that is never used for training
or tuning (honest, non-optimistic evaluation).

| embedding model | dim | F1-macro | F1-micro | inference |
|---|---|---|---|---|
| `intfloat/multilingual-e5-small` | 384 | 0.587 | 0.777 | requires torch |
| [`m2v-bge-m3-edu`](https://huggingface.co/JanSchachtschabel/m2v-bge-m3-edu) (sibling) | 512 | 0.581 | 0.746 | torch-free (numpy) |
| **m2v-e5-large-edu** (this model) | 512 | 0.567 | 0.748 | torch-free (numpy) |
| `m2v-gte-multilingual-768` (generic static) | 768 | 0.559 | 0.747 | torch-free (numpy) |

**What this shows:** among static distillations of strong multilingual encoders, this model
lands in the same band (~0.56–0.60 macro) — the practical ceiling for frozen static
embeddings on this short, keyword-heavy metadata. A transformer (e5-small) stays slightly
ahead, as expected for static embeddings; the trade-off is that this model needs **no
PyTorch** and is far lighter and faster at inference. For this *specific* classification task
a plain TF-IDF + linear classifier scores higher still (macro ≈ 0.62), so the intended value
of this model is **fast, torch-free semantic embeddings for the education domain**, not being
the single best classifier for one dataset. These are **domain-specific benchmark** numbers,
not a general-purpose (e.g. MTEB) score.

## Intended use

Fast embedding of (mostly German) educational metadata — titles, descriptions, keywords —
for classification, clustering, semantic search and retrieval, especially in CPU-only or
low-resource deployments where a transformer is too heavy.

## Usage

```python
from model2vec import StaticModel

model = StaticModel.from_pretrained("JanSchachtschabel/m2v-e5-large-edu")
embeddings = model.encode(["Arbeitsblatt zur Bruchrechnung, Klasse 6"])
```

Or via Sentence Transformers:

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("JanSchachtschabel/m2v-e5-large-edu")
embeddings = model.encode(["Arbeitsblatt zur Bruchrechnung, Klasse 6"])
```

## How Model2Vec works

[Model2Vec](https://github.com/MinishLab/model2vec) passes a vocabulary through a
sentence-transformer, reduces dimensionality with PCA, and weights the token embeddings with
[SIF weighting](https://openreview.net/pdf?id=SyK00v5xx) (frequent tokens down-weighted). At
inference it takes the (weighted) mean of the static token vectors of a text — no transformer
forward pass.

## License

Inherits the base model's license (multilingual-e5-large: MIT).

## Citation

```
@article{minishlab2024model2vec,
  author = {Tulkens, Stephan and {van Dongen}, Thomas},
  title = {Model2Vec: Fast State-of-the-Art Static Embeddings},
  year = {2024},
  url = {https://github.com/MinishLab/model2vec}
}
```