README.md

---
license: mit
library_name: lightgbm
tags:
  - learning-to-rank
  - lightgbm
  - lambdarank
  - recruitment
  - candidate-ranking
---

# Intelligent Candidate Ranker (LightGBM LambdaRank)

**The ranking model from the Redrob Intelligent Candidate Discovery and Ranking Challenge submission.**

Given a 22-feature vector describing a candidate's fit against a job description, this model outputs a relevance score. It is trained on labels generated by 2,500 pairwise judgments from a local LLM (Gemma3) rather than hand-coded heuristics, specifically to avoid label circularity.

![Model Training Architecture](model_training_architecture.png)

Full pipeline (retrieval, feature engineering, consistency scoring, reasoning generation) lives in the GitHub repo:
https://github.com/Pranjal1342/Intelligent-Candidate-Discovery-Ranking-System

---

## This model's role in the pipeline

This model is one stage inside a larger offline candidate-ranking pipeline. It does not do retrieval, does not compute the input features itself, and does not produce the final rank on its own.

```
raw_score = this_model.predict(feature_vector)
final_score = raw_score * consistency_score   # applied by the host pipeline, not this model
```

`consistency_score` is a separate, multiplicative honeypot/data-integrity check computed by the host application — it is not part of this model's output.

---

## How to load

```python
from huggingface_hub import hf_hub_download
import lightgbm as lgb

model_path = hf_hub_download(
    repo_id="<your-username>/intelligent-candidate-ranker",
    filename="lgbm_model.txt"
)
model = lgb.Booster(model_file=model_path)

raw_score = model.predict(feature_vector)  # feature_vector: 22-dim float32
```

---

## Input / Output

- **Input:** a 22-feature float32 vector per candidate (exact feature order below)
- **Output:** a single raw relevance score (higher = more relevant). Not yet penalized for data-integrity issues — combine with a `consistency_score` downstream before final ranking.

### Feature vector (in order)

| # | Feature | Formula / Source |
|---|---|---|
| 1 | `bm25_score` | Stage 1 BM25 retrieval score (normalised) |
| 2 | `yoe` | `profile.years_of_experience` |
| 3 | `Param_A_Systems_Depth` | Fraction of career months in roles whose descriptions contain retrieval, search, or ranking keywords |
| 4 | `Param_B_Availability` | `(recruiter_response_rate + exp(-days_inactive / 90)) / 2` |
| 5 | `Param_C_Tenure` | `min(avg_tenure_months, 48) / 48`, rewards 3+ year tenures |
| 6 | `Param_D_Notice_Exp` | `exp(-max(0, days-30) / 30)`: 30d → 1.0, 60d → 0.37, 90d → 0.14, 150d → 0.006 |
| 7 | `Param_E_Credibility` | `advanced_claimed_count / max(1, assessed_count)`, higher means less credible |
| 8 | `Param_F_Consulting` | Fraction of career at IT-services consulting firms (`industry == "IT Services" AND size == "10001+"`) |
| 9 | `Param_G_Location` | Noida/Pune = 1.0, other India = 0.7, outside and willing to relocate = 0.3, outside and unwilling = 0.0 |
| 10 | `Param_H_GitHub` | `github_activity_score / 100`; 0.3 imputed when the field equals -1 (absent) |
| 11 | `title_ai_fraction` | Career-weighted fraction in AI, ML, or data roles via a static title taxonomy |
| 12 | `prod_signal_log` | Log-compressed production keyword count, -1.0 if academic-only |
| 13 | `consistency_score` | Multiplicative honeypot penalty, c1 × c2 × c3 × c4 × c5 (included as a training feature; also reapplied post-inference — see below) |
| 14 | `hard_req_coverage` | Fraction of JD hard requirements satisfied by the candidate's skill list |
| 15 | `flag_consulting_only` | `consulting_fraction > 0.95` |
| 16 | `flag_title_chaser` | `avg_tenure < 18 months` across 3+ jobs |
| 17 | `flag_langchain_dabbler` | LLM-era months > 12 and pre-LLM months == 0 |
| 18 | `flag_cv_specialist` | CV/speech months > 24 and NLP/IR months == 0 |
| 19 | `flag_title_desc_mismatch` | Domain-category mismatch fraction across career history |
| 20 | `flag_template_desc` | Max SequenceMatcher ratio against the template registry |
| 21 | `interaction_req_x_consistency` | `hard_req_coverage * consistency_score` |
| 22 | `interaction_yoe_x_prod` | `yoe * prod_signal_log` |

---

## Training

**Model configuration:**
- `objective: lambdarank`
- `eval_at: [5, 10, 50]`, explicitly optimising Precision@5
- Early stopping monitors NDCG@5, patience 30
- 200 boosting rounds

**Training labels — Gemma3 pairwise annotation (the key differentiator):**

Rather than a pure heuristic label, training labels are generated via 2,500 pairwise LLM comparisons using Gemma3:4b-it-q4_K_M running locally on Ollama, with zero external API calls and full reproducibility. A stratified sample of 500 candidates is drawn across three strata (top-100, boundary 101–300, and a broader pool with guaranteed low-consistency coverage), and each candidate receives roughly five matchups against random opponents.

For each pair, Gemma3 reads both candidates' full structured profiles alongside the JD requirements and disqualifiers, then produces a single verdict: `CANDIDATE_A`, `CANDIDATE_B`, or `TIE`. Win and loss tallies convert to Elo ratings via Laplace-smoothed win rates:

```python
win_rate = (wins + 0.5) / (total + 1)
elo = 400 * log10(win_rate / (1 - win_rate)) + 1500
```

Elo ratings are thresholded to 0–3 relevance labels by quartile, producing a balanced training set with roughly 125 candidates per label.

**Why this breaks circularity:** Gemma had no knowledge of the 22 engineered features, the BM25 scores, or the penalty weights. It learned independently that IR-specific skills (FAISS, BM25, Qdrant, Sentence Transformers) outrank generic ML skills, and that production-company backgrounds outrank consulting-only careers. LightGBM then learns how the 22 features correlate with these independent judgments, surfacing interactions that were never explicitly encoded.

---

## Model Comparison: Heuristic vs. Gemma-Trained

The competition provides no ground-truth relevance labels, so a standard NDCG@10 ablation against a labeled holdout set isn't possible to compute honestly. What is available, and what is reported here, is a direct head-to-head comparison between a LightGBM model trained on the original heuristic weak label and this model (trained on Gemma3 pairwise labels), run on the same candidate pool with the same feature vectors.

**Method:** both trained models score the full ~8,500-candidate retrieval pool. The same post-inference consistency multiplier is applied to both before ranking, so the comparison isolates the effect of the training label, not the honeypot suppression layer.

| Metric | Result |
|---|---|
| Top-10 overlap between the two models | 0 of 10 candidates in common |
| Spearman rank correlation (top-100) | 0.001 — statistically independent rankings |
| Honeypot leakage, heuristic-trained model | Required a hand-coded post-processing suppression list to keep keyword-stuffed non-technical profiles out of the top 100 |
| Honeypot leakage, Gemma-trained model (this model) | 0 of 100 candidates with `consistency_score < 0.25`, achieved with no post-processing suppression list |

**Qualitative before/after:** prior to the Gemma retrain, the heuristic-trained model's unsuppressed top-10 surfaced profiles such as Content Writer, Project Manager, and Sales Executive — each with AI-sounding skills listed but no underlying technical career history, because the heuristic label rewarded keyword coverage directly. After the Gemma retrain, the same pool's top-10 surfaced candidates with FAISS, BM25, Qdrant, Sentence Transformers, and Hugging Face Transformers in their skill history — sourced from a model that never saw `bm25_score` or `hard_req_coverage` during label generation and discovered the IR-relevance ordering independently from reading full candidate profiles.

The two models disagreeing almost completely (Spearman 0.001) is itself evidence of non-circularity: a model trained on labels derived from the same 22 features it predicts on would be expected to correlate strongly with a heuristic built from those same features, not diverge from it entirely.

---

## Intended Use & Limitations

- Built for a hackathon submission (Redrob Intelligent Candidate Discovery and Ranking Challenge); not validated for production hiring decisions.
- Expects the exact 22-feature schema above, computed by the host pipeline's `src/features.py`. Feeding hand-built or differently-ordered features will produce meaningless scores.
- Raw model output is **not** the final ranking score — it must be multiplied by a separately computed `consistency_score` before use.
- Trained on a synthetic/competition candidate dataset; label distribution and feature semantics may not generalize to other candidate pools without retraining.

---

## AI Tool Disclosure

Gemma3:4b-it-q4_K_M (Google DeepMind, running locally via Ollama) was used offline to generate 2,500 pairwise relevance judgments on a stratified sample of 500 candidates. These judgments served as independent, non-circular training labels for this model. No candidate data was transmitted to any external service at any point.