| --- |
| language: |
| - en |
| license: apache-2.0 |
| library_name: sentence-transformers |
| tags: |
| - sentence-transformers |
| - sentence-similarity |
| - feature-extraction |
| - dense-retrieval |
| - information-retrieval |
| - job-skill-matching |
| - esco |
| - talentclef |
| - xlm-roberta |
| base_model: jjzha/esco-xlm-roberta-large |
| pipeline_tag: sentence-similarity |
| model-index: |
| - name: skillscout-large |
| results: |
| - task: |
| type: information-retrieval |
| name: Information Retrieval |
| dataset: |
| name: TalentCLEF 2026 Task B — Validation (304 queries, 9052 skills) |
| type: talentclef-2026-taskb-validation |
| metrics: |
| - type: cosine_ndcg_at_10 |
| value: 0.4830 |
| name: nDCG@10 |
| - type: cosine_map_at_100 |
| value: 0.1825 |
| name: MAP@100 |
| - type: cosine_mrr_at_10 |
| value: 0.6657 |
| name: MRR@10 |
| - type: cosine_accuracy_at_1 |
| value: 0.5099 |
| name: Accuracy@1 |
| - type: cosine_accuracy_at_10 |
| value: 0.9474 |
| name: Accuracy@10 |
| --- |
| |
| # SkillScout Large — Job-to-Skill Dense Retriever |
|
|
| **SkillScout Large** is a dense bi-encoder for retrieving relevant skills from a job title. |
| Given a job title (e.g., *"Data Scientist"*), it encodes it into a 1024-dimensional embedding and retrieves the most semantically relevant skills from the [ESCO](https://esco.ec.europa.eu/) skill gazetteer (9,052 skills) using cosine similarity. |
|
|
| This is **Stage 1** of the TalentGuide two-stage job-skill matching pipeline, trained for [TalentCLEF 2026 Task B](https://talentclef.github.io/). |
|
|
| > **Best pipeline result (TalentCLEF 2026 validation set):** |
| > nDCG@10 graded = **0.6896** · nDCG@10 binary = **0.7330** |
| > when combined with a fine-tuned cross-encoder re-ranker at blend α = 0.7. |
| > Bi-encoder alone: nDCG@10 graded = **0.3621** · MAP = **0.4545** |
|
|
| --- |
|
|
| ## Model Summary |
|
|
| | Property | Value | |
| |---|---| |
| | Base model | [`jjzha/esco-xlm-roberta-large`](https://huggingface.co/jjzha/esco-xlm-roberta-large) | |
| | Architecture | XLM-RoBERTa-large + mean pooling | |
| | Embedding dimension | 1024 | |
| | Max sequence length | 64 tokens | |
| | Training loss | Multiple Negatives Ranking (MNR) | |
| | Training pairs | 93,720 (ESCO job–skill pairs, essential + optional) | |
| | Epochs | 3 | |
| | Best checkpoint | Step 3500 (saved by validation nDCG@10) | |
| | Hardware | NVIDIA RTX 3070 8GB · fp16 AMP | |
|
|
| --- |
|
|
| ## What is TalentCLEF Task B? |
|
|
| **TalentCLEF 2026 Task B** is a graded information-retrieval shared task: |
|
|
| - **Query**: a job title (e.g., *"Electrician"*) |
| - **Corpus**: 9,052 ESCO skills (e.g., *"install electric switches"*, *"comply with electrical safety regulations"*) |
| - **Relevance levels**: |
| - `2` — Core skill (essential regardless of context) |
| - `1` — Contextual skill (depends on employer / industry) |
| - `0` — Non-relevant |
|
|
| **Primary metric**: nDCG with graded relevance (core=2, contextual=1) |
|
|
| --- |
|
|
| ## Usage |
|
|
| ### Installation |
|
|
| ```bash |
| pip install sentence-transformers faiss-cpu # or faiss-gpu |
| ``` |
|
|
| ### Encode & Compare |
|
|
| ```python |
| from sentence_transformers import SentenceTransformer |
| |
| model = SentenceTransformer("talentguide/skillscout-large") |
| |
| job = "Data Scientist" |
| skills = ["data science", "machine learning", "install electric switches"] |
| |
| embs = model.encode([job] + skills, normalize_embeddings=True) |
| scores = embs[0] @ embs[1:].T |
| |
| for skill, score in zip(skills, scores): |
| print(f"{score:.3f} {skill}") |
| # 0.872 data science |
| # 0.731 machine learning |
| # 0.112 install electric switches |
| ``` |
|
|
| ### Full Retrieval with FAISS (Recommended) |
|
|
| ```python |
| from sentence_transformers import SentenceTransformer |
| import faiss, numpy as np |
| |
| model = SentenceTransformer("talentguide/skillscout-large") |
| |
| # --- Build index once over your skill corpus --- |
| skill_texts = [...] # list of skill names / descriptions |
| |
| embs = model.encode(skill_texts, batch_size=128, |
| normalize_embeddings=True, |
| show_progress_bar=True).astype(np.float32) |
| |
| index = faiss.IndexFlatIP(embs.shape[1]) # inner product on L2-normed = cosine |
| index.add(embs) |
| |
| # --- Query at inference time --- |
| job_title = "Software Engineer" |
| q = model.encode([job_title], normalize_embeddings=True).astype(np.float32) |
| |
| scores, idxs = index.search(q, k=50) |
| for rank, (idx, score) in enumerate(zip(idxs[0], scores[0]), 1): |
| print(f"{rank:3d}. [{score:.4f}] {skill_texts[idx]}") |
| ``` |
|
|
| ### Demo Output |
|
|
| ``` |
| Software Engineer |
| 1. [0.942] define software architecture |
| 2. [0.938] software frameworks |
| 3. [0.935] create software design |
| |
| Data Scientist |
| 1. [0.951] data science |
| 2. [0.921] establish data processes |
| 3. [0.919] create data models |
| |
| Electrician |
| 1. [0.944] install electric switches |
| 2. [0.938] install electricity sockets |
| 3. [0.930] use electrical wire tools |
| ``` |
|
|
| --- |
|
|
| ## Two-Stage Pipeline Integration |
|
|
| SkillScout Large is designed as **Stage 1** — fast ANN retrieval. |
| For maximum ranking quality, pair it with a cross-encoder re-ranker: |
|
|
| ``` |
| Job title |
| │ |
| ▼ |
| [SkillScout Large] ← this model |
| │ top-200 candidates (FAISS ANN, ~40ms) |
| ▼ |
| [Cross-encoder re-ranker] |
| │ fine-grained re-scoring of top-200 |
| ▼ |
| Final ranked list (graded: core > contextual > irrelevant) |
| ``` |
|
|
| **Score blending** (best result at α = 0.7): |
|
|
| ```python |
| final_score = alpha * biencoder_score + (1 - alpha) * crossencoder_score |
| ``` |
|
|
| --- |
|
|
| ## Training Details |
|
|
| ### Data |
|
|
| Source: [ESCO occupational ontology](https://esco.ec.europa.eu/), TalentCLEF 2026 training split. |
|
|
| | | Count | |
| |---|---| |
| | Raw job–skill pairs (essential + optional) | 114,699 | |
| | ESCO jobs with aliases | 3,039 | |
| | ESCO skills with aliases | 13,939 | |
| | Training InputExamples (after canonical-pair inclusion) | **93,720** | |
| | Validation queries | 304 | |
| | Validation corpus (skills) | 9,052 | |
| | Validation relevance judgments | 56,417 | |
|
|
| Essential pairs are included in full; optional skill pairs are downsampled to 50% of the essential count to maintain class balance. |
|
|
| ### Hyperparameters |
|
|
| ``` |
| Loss : MultipleNegativesRankingLoss (scale=20, cos_sim) |
| Batch size : 64 → 63 in-batch negatives per anchor |
| Epochs : 3 |
| Warmup : 10% of total steps (~440 steps) |
| Optimizer : AdamW (fused), lr=5e-5, linear decay |
| Precision : fp16 (AMP) |
| Max seq length : 64 tokens |
| Best model saved : by cosine-nDCG@10 on validation (eval every 500 steps) |
| Seed : 42 |
| ``` |
|
|
| ### Training Curve |
|
|
| | Epoch | Step | Train Loss | nDCG@10 (val) | MAP@100 (val) | |
| |:---:|:---:|:---:|:---:|:---:| |
| | 0.34 | 500 | 2.9232 | 0.3430 | — | |
| | 0.68 | 1000 | 2.1179 | 0.3424 | — | |
| | 1.00 | 1465 | — | 0.3676 | 0.1758 | |
| | 1.37 | 2000 | 1.7070 | 0.3692 | — | |
| | 1.71 | 2500 | 1.6366 | 0.3744 | — | |
| | 2.00 | 2930 | — | 0.3717 | 0.1780 | |
| | 2.39 | **3500** ✓ | **1.4540** | **0.3769** | **0.1808** | |
|
|
| *Best checkpoint saved at step 3500.* |
|
|
| ### Validation Metrics (best checkpoint, binary relevance) |
|
|
| | Metric | Value | |
| |---|---| |
| | **nDCG@10** | **0.4830** | |
| | nDCG@50 | 0.4240 | |
| | nDCG@100 | 0.3769 | |
| | **MAP@100** | **0.1825** | |
| | **MRR@10** | **0.6657** | |
| | Accuracy@1 | 0.5099 | |
| | Accuracy@3 | 0.7993 | |
| | Accuracy@5 | 0.8914 | |
| | Accuracy@10 | **0.9474** | |
|
|
| *Evaluated with `sentence_transformers.evaluation.InformationRetrievalEvaluator` (binary: any qrel > 0 = relevant).* |
|
|
| ### Pipeline Results (graded nDCG, full 9052-skill ranking, server-side) |
|
|
| | Run | nDCG@10 graded | nDCG@10 binary | MAP | |
| |---|---|---|---| |
| | Zero-shot `jjzha/esco-xlm-roberta-large` | 0.2039 | 0.2853 | 0.2663 | |
| | **SkillScout Large (bi-encoder only)** | **0.3621** | **0.4830** | **0.4545** | |
| | SkillScout Large + cross-encoder (α=0.7) | **0.6896** | **0.7330** | 0.2481 | |
|
|
| --- |
|
|
| ## Competitive Context (TalentCLEF 2025 Task B) |
|
|
| | Team | MAP (test) | Approach | |
| |---|---|---| |
| | pjmathematician (winner 2025) | 0.36 | GTE 7B + contrastive + LLM-augmented data | |
| | NLPnorth (3rd of 14, 2025) | 0.29 | 3-class discriminative classification | |
| | **SkillScout Large (2026 val)** | **0.4545** | MNR fine-tuned bi-encoder (Stage 1 only) | |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - **English only** — trained on ESCO EN labels. |
| - **ESCO-domain** — optimised for the ESCO skill taxonomy; performance on other taxonomies (O*NET, custom) may vary without fine-tuning. |
| - **64-token cap** — long job descriptions should be reduced to a concise title before encoding. |
| - **Graded distinction** — the bi-encoder alone does not reliably separate core (2) from contextual (1) skills; a cross-encoder re-ranker is needed for strong graded nDCG. |
| |
| --- |
| |
| ## Citation |
| |
| ```bibtex |
| @misc{talentguide-skillscout-2026, |
| title = {SkillScout Large: Dense Job-to-Skill Retrieval for TalentCLEF 2026}, |
| author = {TalentGuide}, |
| year = {2026}, |
| url = {https://huggingface.co/talentguide/skillscout-large} |
| } |
| |
| @misc{talentclef2026taskb, |
| title = {TalentCLEF 2026 Task B: Job-Skill Matching}, |
| author = {TalentCLEF Organizers}, |
| year = {2026}, |
| url = {https://talentclef.github.io/} |
| } |
| ``` |
| |
| --- |
| |
| ## Framework Versions |
| |
| | Package | Version | |
| |---|---| |
| | Python | 3.12.10 | |
| | sentence-transformers | 5.3.0 | |
| | transformers | 5.5.0 | |
| | PyTorch | 2.11.0+cu128 | |
| | Accelerate | 1.13.0 | |
| | Tokenizers | 0.22.2 | |
| |
| --- |
| |
| ## License |
| |
| [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) |
| |
