---
language:
  - en
  - fr
  - de
license: mit
library_name: sentence-transformers
base_model: intfloat/multilingual-e5-small
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - job-titles
  - retrieval
  - contrastive-learning
  - matryoshka
  - cross-lingual
  - esco
  - onet
---

# job-title-normalizer-e5-small

The **lightweight variant** of
[job-title-normalizer-e5-base](https://huggingface.co/Misbahuddin/job-title-normalizer-e5-base):
a 118M-param sentence encoder fine-tuned to **normalize messy, multilingual job titles to a
canonical occupation taxonomy** (ESCO + O*NET), framed as retrieval over 4,055 canonical
occupations. ~2.5× smaller and faster than the e5-base variant at a modest accuracy cost —
the right choice for CPU serving and bulk backfills.

**Live demo:** <https://job-title-normalizer-525186107937.us-central1.run.app> (Cloud Run; may cold-start ~1 min)

## Results

Held-out test set of **15,248 real ESCO/O*NET titles**; split **by occupation** (zero
train/test occupation overlap, asserted at build time).

| Slice | Method | Recall@1 | Recall@5 | Recall@10 | MRR |
|---|---|---|---|---|---|
| Overall | BM25 (lexical) | 0.095 | 0.163 | 0.183 | 0.124 |
| Overall | zero-shot e5-small | 0.226 | 0.371 | 0.437 | 0.286 |
| Overall | **this model** | **0.367** | **0.539** | **0.601** | **0.441** |
| FR→EN | **this model** | **0.525** | **0.737** | **0.799** | **0.618** |
| DE→EN | **this model** | **0.523** | **0.745** | **0.815** | **0.619** |

BM25 scores **MRR 0.034** cross-lingually; this model reaches **0.619**. For the best
accuracy, use the [e5-base variant](https://huggingface.co/Misbahuddin/job-title-normalizer-e5-base)
(overall MRR 0.463 vs 0.441).

## Training

- **Base:** [`intfloat/multilingual-e5-small`](https://huggingface.co/intfloat/multilingual-e5-small) (mean pooling, `query:`/`passage:` prefixes, 384-dim).
- **Objective:** in-batch-negatives InfoNCE (`MultipleNegativesRankingLoss`, scale 20), wrapped in **MatryoshkaLoss** (dims 384/256/128/64).
- **Data:** 160,240 positive pairs (synonyms, alternate titles, cross-lingual label pairs) from ESCO (EN/FR/DE) + O*NET alternate titles; ≤50 pairs per occupation.
- **Setup:** 2 epochs, batch 128, lr 2e-5, warmup 10%, max_seq_len 64, fp16, `NoDuplicatesDataLoader`. Trained on an RTX 4060 Laptop (8 GB) in ~8 min.

## How to use

The e5 family needs **asymmetric prefixes**: titles as `query: …`, canonical labels as
`passage: …`. Forgetting them silently degrades accuracy.

```python
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer("Misbahuddin/job-title-normalizer-e5-small")

canonicals = [
    "passage: software developer",
    "passage: data scientist",
    "passage: nurse responsible for general care",
]
query = "query: Krankenpfleger"              # German → English canonical

q = model.encode(query, normalize_embeddings=True)
c = model.encode(canonicals, normalize_embeddings=True)
scores = cos_sim(q, c)[0]
print(canonicals[int(scores.argmax())], float(scores.max()))
```

Vectors are L2-normalized (cosine == dot product) — use FAISS `IndexFlatIP` for production.
Matryoshka truncation to 256/128/64 dims works after re-normalizing. **Calibrate an
abstention threshold**: out-of-taxonomy queries return low-score nearest neighbours rather
than failing.

## Intended use & limitations

- **In scope:** normalizing titles from resumes, postings, CRM/ATS/HRIS records to occupation
  IDs; semantic occupation search; FR/DE→EN cross-lingual lookup.
- **Not a hiring/screening/compensation decision system.** Keep a human in the loop for
  consequential uses.
- **Coverage bounded by ESCO + O*NET**; en/fr/de verified only. Short-text regime (≤64 tokens).

## Acknowledgements

- **ESCO** — © European Union; reused under Commission Decision 2011/833/EU. <https://esco.ec.europa.eu/>
- **O*NET** — by the National Center for O*NET Development for USDOL/ETA; CC BY 4.0. <https://www.onetcenter.org/>

> O*NET® is a trademark of USDOL/ETA. This model was produced using O*NET data but is not
> endorsed by USDOL/ETA. ESCO is a service of the European Commission; this model is not
> endorsed by the Commission.