sutd-reranker-aug98

Finetuned cross-encoder/ms-marco-MiniLM-L6-v2 for reranking retrieved SUTD modules in the SUTD Course Recommendation Chatbot (MLOps Group 9). This is the production reranker used in the final Hybrid pipeline.

Given a (job description, module passage) pair, this model predicts a relevance score used to rerank candidate modules before LLM answer generation.

Model Details

Property	Value
Base model	cross-encoder/ms-marco-MiniLM-L6-v2
Max sequence length	512
Output	Single relevance score
Loss	BinaryCrossEntropyLoss

Training Data

Finetuned on hard negatives mined by henreads/sutd-bge-large-aug98. For each of the 98 training jobs (67 hand-annotated + 31 augmented from MyCareersFuture), the top-30 retrieved modules (excluding ground-truth positives) were used as hard negatives. After expansion this produces 2685 training rows and 271 validation rows.

A version trained on the smaller 67-pair dataset is available at henreads/sutd-reranker-ft67.

Training Setup

• Hardware: Modal A10G (24 GB VRAM)
• Training time: ~30 seconds (0.008 GPU hours)
• Epochs: 5
• Batch size: 32
• Learning rate: 2e-5
• Hard negatives: top-30 per job via sutd-bge-large-aug98, excluding positives
• Train/val split: job-level 10% validation
• Tracking: Weights & Biases (sutd-mlops-reranker-finetune)

Evaluation

This model must be paired with its matched embedding model (sutd-bge-large-aug98). The reranker and embedding model must be co-designed around the same candidate distribution — swapping components changes results significantly.

Config	Chat Quality (Overall 1-5)
Hybrid, base reranker, k=25	3.865
Hybrid, sutd-reranker-ft67, aug98 BGE, k=25	3.865
Hybrid, sutd-reranker-aug98, aug98 BGE, k=25	4.06

The Hybrid pipeline (PE + RAG candidates merged, top-20 reranked) with this model achieves 4.06 / 5.0 on a Gemini LLM-as-judge evaluation across 50 job-description Q&A pairs, exceeding the project success criterion of 4.0.

Usage

from sentence_transformers import CrossEncoder

model = CrossEncoder("henreads/sutd-reranker-aug98")

job_description = "Data Scientist at GovTech. Build ML models with Python..."
module_passage = "50.007 Machine Learning — Topics: supervised learning, neural networks..."

score = model.predict([[job_description, module_passage]])
print(score)

Project

Part of the SUTD Course Recommendation Chatbot — MLOps Group 9.
Code: github.com/henreads/sutd-mlops-group9