# Talentry AI - Methodology This is the document a **Stage-4 reviewer** should read. Every modelling decision is justified against a specific line of [`configs/job_description.txt`](../configs/job_description.txt) or [`configs/redrob_signals_doc.txt`](../configs/redrob_signals_doc.txt). --- ## 1. The mandate, reread The JD explicitly closes with: > The "right answer" to this JD is not "find candidates whose skills section > contains the most AI keywords." That's a trap we've explicitly built into > the dataset. … A candidate who has all the AI keywords listed as skills but > whose title is 'Marketing Manager' is not a fit, no matter how perfect > their skill list looks. … Your ranking system should also weigh behavioural > signals - a perfect-on-paper candidate who hasn't logged in for 6 months > and has a 5% recruiter response rate is, for hiring purposes, not actually > available. Down-weight them appropriately. Talentry AI's six scoring components map 1:1 onto these requirements. --- ## 2. The six components ### 2.1 Title alignment (weight 0.32) A free-text title is mapped onto a **role family** via the lexicon in `nlp/lexicons.py::ROLE_FAMILIES`. Scores range from `3.0` (ML / search-IR engineer) down to `-1.0` (Marketing Manager, etc.). Two trajectory amplifiers: * `-0.40` if every employer on the candidate's CV is a consulting firm (JD: *"People who have only worked at consulting firms in their entire career"*). * `-0.20` if the current title is a pure management track without "engineer / developer / scientist / architect" anywhere (JD: *"this role writes code"*). * `+0.10` if there is at least one product-company employer (JD: *"applied ML at product companies (not pure services)"*). ### 2.2 Semantic fit (weight 0.22) A hybrid retrieval score combining: * **BM25** (60%) - rare-term overlap; e.g. "FAISS", "RAG", "NDCG". * **TF-IDF cosine** (40%) - topical similarity that smooths over phrasing differences ("retrieval", "search", "ranking" should cluster together). We min-max normalise each to [0,1] before combining so the two stay comparable across query lengths. Why no dense embeddings? See [`architecture.md` §3](architecture.md#3-why-not-dense-embeddings). ### 2.3 Skill evidence (weight 0.28) - the anti-stuffer core Each claimed skill gets a **trust score** in [0, 1]: ``` trust = 0.40·proficiency + 0.20·endorsements + 0.10·duration + 0.30·assessment (where assessment is present; else 0.55·proficiency + 0.25·endorsements + 0.20·duration) ``` * Proficiency: `beginner=0.25 / intermediate=0.55 / advanced=0.85 / expert=1.0` * Endorsements saturate at 50 (per dataset max ≈ 60). * Duration saturates at 36 months. Trust scores are then aggregated per **skill cluster** (six clusters - `embeddings_retrieval`, `ranking_recsys`, `nlp_llm`, `ml_core`, `python_engineering`, `data_engineering`) normalised to [0,1] by dividing by `min(cluster_size, 4)`. A stuffer signal - `keyword_stuff_ratio` - counts AI-keyword surface claims that look padded (advanced/expert + ≤2 endorsements + ≤6 months + no assessment, *or* trust < 0.40). When the ratio ≥ 0.7 the cluster sum is discounted by 35%. A CV/speech dominance signal - if ≥ 55% of total trust is in CV-only or speech-only skills, the cluster sum is discounted by 45% (JD: *"primary expertise is computer vision, speech, or robotics … you'd be re-learning fundamentals here"*). ### 2.4 Experience band (weight 0.12) Triangular score, `1.0` inside `[min_years, max_years]`, soft-decaying outside. The JD: *"5-9 is a range, not a requirement … we'll seriously consider candidates outside the band if other signals are strong"* - hence the soft decay rather than a hard cutoff. ### 2.5 Location (weight 0.06) Hierarchical preference: | Location bucket | Score (willing/not) | | -------------------------------------------- | ------------------- | | Pune / Noida / Delhi NCR | 1.00 | | Other Tier-1 India (Bangalore, Hyd, …) | 0.85 / 0.75 | | Other India | 0.65 / 0.45 | | Outside India | 0.30 / 0.10 | ### 2.6 Behavioural multiplier × Honeypot penalty Multiplier components (sum + 1.0, clipped to [0.55, 1.20]): | Component | Contribution | | ------------------------ | ----------------------------------------------- | | Activity recency | +0.12 / +0.05 / -0.05 / -0.20 at 30/90/180/180+ | | Recruiter response rate | `(rr − 0.40) × 0.30` | | Interview completion | `(icr − 0.50) × 0.15` | | Open-to-work flag | +0.05 / -0.02 | | Verifications | +0.015 email, +0.015 phone, +0.02 linkedin | | Notice period | +0.05 / 0 / -0.04 / -0.08 at ≤30/60/90/90+ | | Saved by recruiters 30d | up to +0.05 | Honeypot penalty (subtracted, capped at 0.50): * +0.18 if career-month sum exceeds declared years by ≥ 24 mo; * +0.15 if ≥ 3 expert/advanced claims have zero endorsements **and** ≤ 2 mo of usage; * +0.10 if multiple `is_current=True` roles; * +0.08 if salary band is inverted (min > max); * +0.04 if `signup_date` is after `last_active_date`; * +0.03 if max education end-year is way after the candidate's current-role start year. --- ## 3. Reasoning composition (Stage-4 anti-hallucination) `ranker/reasoning.py` builds every per-row sentence from facts already in the candidate's profile. The structure is: ``` " - ; . Concerns: <…>." ``` The "strongest evidence" span is picked by preference order: 1. A career-history description that mentions retrieval / ranking / embedding / search / recommendation / RAG / vector / FAISS / Elasticsearch / LTR - *quoted with company name + tenure*. 2. The strongest skill cluster (only if ≥ 0.40 and not the AI-keyword surface). 3. A top assessed skill (proficiency + duration + assessment score). 4. Fallback: current title + employer. Concerns are emitted *only when present* - long notice, low activity, stuffer profile, CV/speech dominance, consulting-only career, or honeypot suspicion. Each concern phrase quotes a real numeric fact (e.g. `"long notice 120d"`). This satisfies Stage 4's six checks (specific facts, JD connection, honest concerns, no hallucination, variation, rank consistency) **by construction**. --- ## 4. Reproducibility checklist * `pyproject.toml` pins minimum versions of every dependency. * The pipeline is single-threaded and uses no random state. * `reference_date` for behavioural recency is overridable. * The CSV writer re-implements every `validate_submission.py` invariant and fails fast on violation. * Smoke run reproduces the same ranking for the same input. * `make docker-build && make docker-run` reproduces the full stack inside the same image we deploy to HuggingFace Spaces.