README.md

metadata

title: Redrob Hackathon
emoji: 🚀
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 5.38.0
python_version: 3.11
app_file: app.py
pinned: false

Redrob Hackathon

Candidate ranking demo.

Redrob Intelligent Candidate Ranker

Submission for the Redrob Hackathon — Intelligent Candidate Discovery & Ranking Challenge

A CPU-only, fully deterministic, two-phase candidate ranking engine that selects the top 100 of 100,000 candidates against a Senior AI Engineer job description, with evidence-graph reasoning and honeypot-resistant safety layers.

---

Table of Contents

1. Quick Start
2. TL;DR Summary
3. Architecture Overview
4. Phase 1 — precompute.py
5. Phase 2 — rank.py
6. Feature Catalog (≈55 features, G1–G13)
7. Scoring Formula
8. Safety & Trap Handling
9. Evidence-Graph Reasoning
10. Compute Constraints & Compliance
11. File Map
12. Adversarial Test Suite
13. Design Decisions & Tradeoffs
14. Determinism & Reproducibility Guarantees
15. Constraints Compliance Matrix
16. Known Limitations

---

1. Quick Start

# Install dependencies
pip install -r requirements.txt

# Single-command end-to-end run (auto-runs precompute if features.parquet is missing)
python rank.py --candidates ./candidates.jsonl --out ./submission.csv

For faster iteration (e.g. while tuning scoring weights):

# Phase 1 — run once per JD or candidate-pool change (~196 s)
python precompute.py --candidates ./candidates.jsonl --out ./artifacts/features.parquet

# Phase 2 — run as many times as needed (~4 s)
python rank.py --candidates ./candidates.jsonl --out ./submission.csv \
               --artifacts ./artifacts/features.parquet

Validate the output:

# Format validation — mirrors the hackathon validator
python validate_submission.py submission.csv

# Adversarial test suite (10 cases: keyword stuffer, honeypot, strong candidate, etc.)
python test_adversarial.py

---

2. TL;DR Summary

Aspect	Value
Inputs	candidates.jsonl (100K candidates) + job_description.docx
Output	submission.csv — exactly 100 rows, columns: candidate_id, rank, score, reasoning
Pipeline	Two phases — precompute (196 s) writes features.parquet; rank (4 s) writes CSV
Compute	CPU-only · ≤16 GB RAM · no network · no LLM API calls · deterministic
Features	≈55 JD-driven features across 13 groups (G1–G13)
Scoring	Rule-based elite composite: Impact 42% + Ownership 33% + Search/JD-fit 15% + Behaviour 10%, modulated by hiring intent
Safety	6 multiplicative penalties (disqualifier, behavioral twin, LangChain-only, closed-source, title floor, honeypot) — design target: 0 honeypots in top 100
Reasoning	Evidence-graph based — every cited keyword is verified against the candidate's career text; no hallucination
Determinism	Fixed REFERENCE_DATE = 2026-06-17, hash-based template selection, zero RNG
Single command	python rank.py --candidates ./candidates.jsonl --out ./submission.csv

---

3. Architecture Overview

The engine is split into two phases to keep the timed ranking step well under the 5-minute CPU-only budget. Phase 1 performs the expensive per-candidate feature extraction and writes a Parquet artifact (~1.3 GB). Phase 2 is a lightweight vectorized scoring pass plus reasoning generation that completes in approximately 4 seconds.

                     ┌─────────────────────────────────────────────────────────┐
                     │           Phase 1 · precompute.py  (~196 s)            │
                     │                                                         │
 job_description.docx──▶ JD Parser ──▶ Hiring Intent ──▶ Query Expansion     │
                     │      │               │                  │              │
                     │      └───────────────┴──────────────────┘              │
                     │                            ▼                           │
 candidates.jsonl ───▶  Canonical Profile ──▶ Feature Registry (G1–G13)      │
                     │  (career narrative,        │         │                 │
                     │   unified text blob)       │         │                 │
                     │                            ▼         ▼                 │
                     │                    TF-IDF+SVD    Honeypot Detector     │
                     │                    Embeddings    (~80 trap patterns)   │
                     │                         │              │               │
                     │                         ▼              ▼               │
                     │                    features.parquet (~1.3 GB)         │
                     └──────────────────────────┬──────────────────────────────┘
                                                │  (parquet handoff)
                     ┌──────────────────────────▼──────────────────────────────┐
                     │             Phase 2 · rank.py  (~4 s)                  │
                     │                                                         │
                     │  Load parquet ──▶ Elite Composite                      │
                     │                       │                                 │
                     │       ┌───────────────┼──────────────┐                  │
                     │       ▼               ▼              ▼                  │
                     │   Additive      Multiplicative   Intent-                │
                     │   Boosts        Penalties        modulated weights      │
                     │       └───────────────┴──────────────┘                  │
                     │                       ▼                                 │
                     │             Top-100 + Sigmoid Stretch                  │
                     │                       ▼                                 │
                     │      Deterministic Tiebreak (14 sub-signals)           │
                     │                       ▼                                 │
                     │             Evidence-Graph Reasoning                    │
                     │                       ▼                                 │
                     │        submission.csv ──▶ validate_submission.py        │
                     └─────────────────────────────────────────────────────────┘

This two-phase split also mirrors how the engine would work in production: Phase 1 runs nightly (or on candidate-ingest events) and refreshes the feature store; Phase 2 runs on every recruiter search query and must be sub-second. The hackathon submission demonstrates both halves at scale.

---

4. Phase 1 — precompute.py

Goal: Convert raw candidate JSONL into a wide Parquet feature table that the ranker can score in seconds.

Pipeline

1. JD Parser (lib/jd_parser.py)

The full job description text is embedded as JD_FULL_TEXT inside jd_parser.py (extracted from job_description.docx). A pure-Python pattern-matching parser produces a structured JDUnderstanding dataclass containing required/preferred skills, red flags, YOE bands, seniority, domain, preferred locations, and behavioural expectations (notice period, product vs. services, production code requirement). No LLM call is made — this is regex, keyword dictionaries, and section detection. All downstream modules consume this one object, so changing the JD requires only re-running the parser, not rewriting feature code.

2. Hiring Intent (lib/hiring_intent.py)

Derives a high-level HiringIntent from the JD: philosophy (e.g. "product over research"), primary_need ("production_systems"), ownership_expectation (0–1), shipping_culture ("scrappy"), team_context ("founding" / "early" / "mature"), and depth_requirement ("specialist" / "generalist"). This intent modulates scoring weights in Phase 2 — a founding-team JD boosts ownership weight by ~25%; a specialist JD boosts search/JD-fit weight by ~18%.

3. Query Expansion (lib/query_expansion.py)

Takes the JD's "ideal candidate" text and expands it with synonyms and related terms (e.g. "RAG" → "retrieval augmented generation, dense retrieval, vector search, semantic search"). This expanded text becomes the query against which candidate embeddings are compared, so a candidate who writes "dense retrieval" still matches a JD that says "RAG".

4. Canonical Profile (lib/candidate_profile.py)

Normalizes raw candidate JSON into a typed profile with computed convenience accessors. Critically, schema.unified_text_blob() builds the text used for TF-IDF and keyword matching from career_history descriptions, summary, and headline — but explicitly excludes the skills[] array. This is the single most important anti-stuffing decision: a candidate cannot inflate their match score by adding "RAG, Pinecone, FAISS" to their skills list, because that list is not part of the matching text. Skills are scored separately via the skill_coverage feature, which cross-checks each declared skill against the text blob.

5. Feature Registry (lib/feature_registry.py + lib/features.py)

A plugin-style registry where each of the ≈55 features is a registered FeatureSpec with name, group, description, default weight, extract function, intent-modulation map, and optional depends_on for interaction features. This makes ablation trivial (unregister one spec) and lets intent modulation be data-driven rather than hardcoded.

6. TF-IDF + SVD Embeddings (lib/embeddings.py)

Fits a TF-IDF vectorizer on the union of candidate text blobs and the expanded ideal-candidate text, then reduces to 100 dimensions via truncated SVD. Cosine similarity to the expanded query becomes the embedding_sim feature. TF-IDF+SVD was chosen over sentence-transformers deliberately — see §13 Design Decisions.

7. Honeypot Detector (lib/honeypot.py)

Flags candidates with internally-impossible profiles (e.g. 8 years of PyTorch experience when PyTorch 1.0 was released in October 2018, or a "Principal Engineer" title with only 1 year of total experience). Honeypots carry a 0.01× multiplier in Phase 2 — they essentially cannot enter the top 100 unless the entire candidate pool is degenerate.

Output

A single Parquet file (~1.3 GB for 100K candidates) with one row per candidate. Columns are the ≈55 numeric features plus metadata columns (current_title, current_company, YOE) and JSON-encoded evidence columns prefixed with _ (e.g. _candidate_json, _disq_reasons, _behaviour_evidence, _tier5_evidence) that the Phase 2 reasoning engine requires.

Why Parquet? Columnar compression brings 100K rows × ~70 columns to ~1.3 GB vs. ~6 GB for JSON; typed schema makes pd.read_parquet ~10× faster than re-parsing JSONL; and preserved dtypes eliminate type-inference overhead on every run.

---

5. Phase 2 — rank.py

Goal: Load the Parquet, score every candidate, select the top 100, generate reasoning, write CSV, and self-validate.

Pipeline

1. Auto-precompute guard

If features.parquet does not exist, rank.py shells out to precompute.py automatically, making the single-command reproduction path work end-to-end on a fresh checkout.

2. Vectorized scoring (lib/scoring.py)

Both elite_score_vec(df) and final_score_vec(df) operate on the entire DataFrame at once via NumPy — there is no Python per-candidate loop in the hot path. The composite formula is detailed in §7.

3. Top-100 + sigmoid stretch

After computing raw_score, the top 100 are selected, min-max normalized to [0, 1], and passed through a stretched sigmoid:

stretched = 0.52 + 0.47 × σ(10 × (norm − 0.5))

This produces scores in approximately [0.52, 0.99] with a clear top-to-bottom falloff. The relative ordering is exactly preserved — this is a cosmetic presentation choice so Stage-4 reviewers see a confident distribution rather than a flat 0.71–0.74 cluster.

4. Deterministic tiebreak (14 sub-signals)

When two candidates have identical stretched scores after rounding to 6 decimal places, ties are broken by a 14-element lexicographic key: tier5_signature, pre_llm_x_ownership, impact_magnitude, ownership_hierarchy, production_strength, career_depth_ratio, evidence_strength, skill_coverage, title_relevance, cross_validation, education_tier, endorsement_signal, assessment_signal, and salary_compatibility. candidate_id ascending is appended as the final fallback per the spec.

5. Evidence-graph reasoning (lib/reasoning.py)

Generates the 1–2 sentence reasoning column. See §9.

6. Self-validation

After writing the CSV, rank.py runs validate_output() inline (a mirror of validate_submission.py) and exits non-zero on any failure. This surfaces format drift immediately rather than at hackathon-upload time.

---

6. Feature Catalog (≈55 features, G1–G13)

Features are organized into 13 groups. Each feature is a float in roughly [0, 1], extracted from the canonical profile and the unified text blob.

G1 — JD Fit (7 features)

The most direct measure of skill, title, and seniority match against the JD.

Feature	Description
skill_coverage	Fraction of JD-required skills found in candidate's text
preferred_coverage	Fraction of JD-preferred skills
domain_specialization	Depth in the JD's primary domain (e.g. IR/RecSys)
skill_trust_avg	Average endorsement-weighted proficiency across matched skills
title_relevance	Similarity of current/past titles to JD's role title
seniority	Seniority-band fit (junior / mid / senior / staff)
jd_skill_count	Raw count of JD-required skills present

G2 — Impact & Ownership (4 features)

The most heavily weighted group in the elite composite. Captures whether the candidate has demonstrably owned systems and shipped outcomes.

Feature	Description
ownership_hierarchy	Evidence of owning systems end-to-end (init → ship → operate)
impact_magnitude	Quantified outcome magnitude (e.g. "reduced latency 40%")
impact_signals	Count of impact-claim patterns in career text
evidence_strength	Composite of quantification, specificity, and recency of evidence

G3 — Production & Scale (3 features)

Distinguishes production-experienced engineers from research-only or demo-only candidates — a JD disqualifier category.

Feature	Description
production_strength	Composite of deployment verbs, scale nouns, traffic mentions
production_diversity	Number of distinct production contexts (multiple employers, multiple systems)
scale_evidence	Mentions of QPS, users, latency budgets, SLOs

G4 — Experience & Career (8 features)

Feature	Description
yoe_band_score	Fit to the JD's 5–9 year band (peak at 6–8, gentle decay outside)
career_depth_ratio	Years in applied ML/AI roles ÷ total YOE
pre_llm_months	Months of ML experience before November 2022 (ChatGPT release)
career_trajectory	Upward / lateral / chaotic
company_quality	Best company tier (Tier-1 FAANG-adjacent → Tier-5 services)
company_quality_avg	Weighted-average tier across career
career_stability	Inverse of job-hopping frequency
promotion_velocity	Titles per year (too high = title-chaser red flag)

G5 — Retrieval & Evaluation (3 features)

Specific to this JD's "evaluation frameworks for ranking systems" requirement.

Feature	Description
retrieval_depth	Depth of retrieval/IR vocabulary in career text
evaluation_experience	Mentions of NDCG, MRR, MAP, A/B tests, offline-online correlation
system_design_evidence	Design vocabulary (sharding, replication, index refresh, drift)

G6 — Behavioural (7 features)

Derived from the 23 Redrob signals. Multiplier-style signals that modify the composite rather than driving it.

Feature	Description
recency	Days since last_active_date (decay function)
responsiveness	recruiter_response_rate with response-time penalty
market_demand	search_appearance_30d + saved_by_recruiters_30d normalized
github_activity	GitHub score (-1 treated as missing, not penalized)
availability_score	open_to_work_flag × notice-period fit × interview_completion_rate
interview_completion	interview_completion_rate
platform_trust	verified_email × verified_phone × linkedin_connected

G7 — Resume Quality (4 features)

Feature	Description
quantified_outcomes	Fraction of career entries with quantified impact
truthiness	Claims-without-evidence detector (e.g. "led large scale" with no number)
keyword_stuffing_risk	skills[] list vs text-blob mismatch
profile_completeness	profile_completeness_score normalized

G8 — Safety (2 features)

Feature	Description
disqualifier_penalty	0.05× multiplier if any JD disqualifier fires
is_honeypot	Boolean; carries a 0.01× multiplier in scoring

G9 — Location (1 feature)

Feature	Description
location_score	Pune/Noida = 1.0 · other preferred cities = 0.8 · other India = 0.5 · outside India = 0.2

G10 — Career Narrative (3 features)

From lib/career_narrative.py — detects whether the career arc is coherent.

Feature	Description
career_coherence	Domain-progression consistency score
narrative_type	"upward_specialist" / "lateral_generalist" / "chaotic" / "title_inflation"
narrative_suspicious	List of suspicious-pattern flags (JSON-encoded)

G11 — Interaction Features (4 features)

Products of base features that capture compound signals a linear model would miss.

Feature	Description
ownership_x_production	Owners who also ship to production
skill_x_yoe	Skills weighted by years of relevant experience
impact_x_domain	High-impact work specifically in the JD's domain
trajectory_x_company	Upward trajectory at high-tier companies

G12 — Key Differentiator Features (6 features)

These features each target a specific failure mode of the baseline composite that the core groups (G1–G11) do not fully address.

Feature	Description
tier5_signature	Recovers strong candidates (FAANG-adjacent, deep specialists) who don't keyword-match the JD but are objectively strong. Additive boost.
behavioral_twin_penalty	Penalizes candidates with strong on-paper skills but behaviorally unavailable (low response rate, long notice). Response rate < 0.15 is a hard filter.
langchain_only_penalty	Flags candidates whose AI experience is exclusively recent LangChain work without pre-LLM ML production experience. Direct JD disqualifier.
closed_source_penalty	Flags candidates with 5+ years of entirely closed-source work and no external validation (papers, talks, OSS).
pre_llm_x_ownership	Interaction: pre-LLM ML experience × ownership. Rare and extremely strong founding-team signal.
salary_compatibility	Fit between expected_salary_range and the role's implicit band.

G13 — Platform Signal Features (4 features)

These features leverage platform-specific signals not captured by the core career-text analysis.

Feature	Description
assessment_signal	Uses platform-validated skill_assessment_scores (not self-reported). Stronger signal than declared skills.
endorsement_signal	Combines endorsements_received with skill-relevance filtering. "Endorsed on RAG" vs. "Endorsed on Excel" produce very different scores.
education_tier	IIT/NIT/IIIT + CS/AI/ML field bonus. Mild tiebreaker only — not a primary ranking factor (avoids over-weighting pedigree).
cross_validation	Count of dimensions (impact, ownership, production, depth, evaluation) all simultaneously strong (≥ 0.50). Provides a +0.025 additive boost for candidates who are genuinely elite across the board.

---

7. Scoring Formula

Elite Composite (additive, intent-modulated weights)

elite = w_impact     × impact_composite
      + w_ownership  × ownership_composite
      + w_search     × search_composite
      + w_behaviour  × behaviour_composite

Base weights (NDCG@10-optimized — top-10 precision is 50% of total score):

Component	Base Weight	Sub-component Composition
Impact	0.42	impact_magnitude (0.38) + impact_signals (0.32) + production_diversity (0.30)
Ownership	0.33	ownership_hierarchy (0.45) + evidence_strength (0.55)
Search / JD-fit	0.15	skill_coverage (0.32) + title_relevance (0.23) + skill_trust_avg (0.20) + pre_llm_months (0.13) + retrieval_depth (0.12)
Behaviour	0.10	career_trajectory (0.22) + truthiness (0.22) + responsiveness (0.22) + yoe_band_score (0.12) + availability_score (0.12) + seniority (0.10)

Intent modulation (applied multiplicatively to base weights, then renormalized):

Hiring-intent signal	Effect
ownership_expectation ≥ 0.8 (founding-team JD)	ownership ×1.25 · impact ×0.95
primary_need = production_systems	impact ×1.10 · search ×0.93
team_context ∈ {founding, early}	ownership ×1.10 · behaviour ×0.82
depth_requirement = specialist	search ×1.18 · behaviour ×0.88
shipping_culture = scrappy	impact ×1.05 · ownership ×1.05

Final Score

final = elite
      + 0.065 × tier5_signature           # differentiator boost
      + 0.045 × pre_llm_x_ownership       # differentiator boost
      + 0.035 × (impact × ownership)      # combo boost
      + 0.025 × cross_validation          # platform signal boost
      × disqualifier_penalty              # 1.0 or 0.05
      × behavioral_twin_penalty           # 1.0, 0.50, or 0.0 (if RR < 0.15)
      × langchain_only_penalty            # 1.0 or 0.55
      × closed_source_penalty             # 1.0 or 0.60
      × honeypot_multiplier               # 1.0 or 0.01
      × title_floor(title_relevance)      # non-engineer → cap 0.20 · irrelevant → 0.05

Boosts are additive (rare +0.065–0.025 lift for true elite candidates). Penalties are multiplicative — a single disqualifier drops the score to approximately 5% of its elite composite. This asymmetry is intentional: small recoveries for nuanced signal misses, but hard kills for genuine disqualifiers and traps.

Score Stretch (cosmetic — ordering preserved exactly)

stretched   = 0.52 + 0.47 × σ(10 × (norm − 0.5))
final_score = round(stretched, 6)

This maps raw scores to approximately [0.52, 0.99] with a confident-looking top-to-bottom falloff. Relative ordering is exactly preserved; this does not affect NDCG, MAP, or P@10 (all rank-based, not score-based).

Tie-breaking (14-element lexicographic key)

Priority order when final_score values are equal after 6-decimal rounding:

Priority	Signal
1	tier5_signature
2	pre_llm_x_ownership
3	impact_magnitude
4	ownership_hierarchy
5	production_strength
6	career_depth_ratio
7	evidence_strength
8	skill_coverage
9	title_relevance
10	cross_validation
11	education_tier
12	endorsement_signal
13	assessment_signal
14	salary_compatibility
Final fallback	candidate_id ascending (per spec)

---

8. Safety & Trap Handling

The dataset is explicitly adversarial. redrob_signals_doc.md describes approximately 80 honeypots with subtly impossible profiles, plus keyword stuffers, behavioral twins, and plain-language Tier-5 candidates designed to evade naive keyword matchers. The safety stack below is designed to keep the honeypot rate in the top 100 at 0%.

8.1 Honeypot Detection (lib/honeypot.py)

Flags candidates with internally-impossible profiles:

• YOE claims that predate the technology (e.g. 8 years of PyTorch when v1.0 shipped October 2018)
• Title–YOE mismatch (Principal Engineer at 1 year YOE)
• Career gaps that do not reconcile
• Skill durations longer than total YOE

Penalty: 0.01× multiplier — essentially disqualifying.

8.2 Disqualifier Penalty (features.disqualifier_penalty)

Fires a 0.05× multiplier on any of the JD's explicit disqualifiers:

• Pure research with no production deployment
• Senior engineer with no production code in the last 18 months (architecture/tech-lead drift)
• Career exclusively at consulting firms (TCS, Infosys, Wipro, Accenture, Cognizant, Capgemini)
• Primary expertise in CV/speech/robotics without significant NLP/IR exposure

8.3 Behavioral Twin Penalty (features.behavioral_twin)

Targets candidates who look strong on paper but are unhireable in practice: strong skills, recent active date, high profile completeness — but very low recruiter_response_rate and a long notice period. Penalty: 0.50× soft and 0.0× hard when recruiter_response_rate < 0.15, aligning with the spec's explicit guidance on response rate.

8.4 LangChain-Only Penalty (features.langchain_only_recent)

Flags candidates whose AI experience is exclusively recent (≤12 months) LangChain tutorial work without substantial pre-LLM ML production experience. Penalty: 0.55×.

8.5 Closed-Source Isolation Penalty (features.closed_source_isolation)

Flags candidates whose entire 5+ year career has been on closed-source proprietary systems with no external validation (papers, talks, OSS contributions). Penalty: 0.60×.

8.6 Title Floor

Caps candidates whose title does not match the engineering track:

• Non-engineering titles (HR Manager, Sales Lead, etc.) with AI keywords → score capped at 0.20
• Irrelevant titles with no engineering signal → capped at 0.05

8.7 Keyword Stuffing Resistance

Two-layer defense:

1. unified_text_blob excludes skills[] — adding "RAG, Pinecone, FAISS, NDCG, LoRA, BM25, Embeddings" to a skills list does nothing for matching text. Confirmed by test_adversarial.py :: keyword_stuffer.
2. keyword_stuffing_risk feature explicitly measures skills-vs-text mismatch and feeds into the truthiness behavioural component.

---

9. Evidence-Graph Reasoning

The reasoning column is generated by lib/reasoning.py using an evidence-graph approach — every claim is backed by a verified node in the candidate's profile.

How It Works

1. Fact extraction — Pull verified facts from the canonical profile: current title, current company, YOE, top-3 strongest features, disqualifier reasons (if any), honeypot status.
2. Template selection — hash(candidate_id) % N picks one of N templates per reasoning "shape" (strong-fit / behavioral-concern / skill-adjacent / disqualifier-soft / filler). Hash-based selection is deterministic and varies reasoning across candidates.
3. Slot filling — Fill template slots only with verified facts. If a fact is missing (e.g. no quantified outcomes), the slot is silently omitted rather than filled with a generic claim.
4. Tech-keyword verification — Any technical term mentioned in the reasoning is checked against the unified text blob. If the term is not present in the candidate's actual career text, it is removed before emission. This is the hallucination guard.
5. JD-connection check — The reasoning must reference at least one JD requirement (skill, disqualifier, or behavioural expectation). Pure generic praise ("great candidate, strong skills") is rejected and a new template is selected.

Guarantees

Property	How It Is Achieved
No hallucination	Every cited skill, employer, or experience is verified to exist in the candidate's profile
Variation	Template selection is hash-based and slot-filling is per-candidate; 10 randomly-sampled reasonings will be substantively different
Rank consistency	Template shape is chosen based on the candidate's feature profile, which correlates with rank
Honest concerns	If a disqualifier fired or a behavioral twin penalty applied, the reasoning mentions it

---

10. Compute Constraints & Compliance

Constraint	Limit	Observed (100K candidates, 8-core CPU)	Status
Total runtime	≤ 5 min wall-clock	precompute 196 s + rank ~4 s = **200 s**	✅
Memory	≤ 16 GB RAM	~1.3 GB peak (Parquet + scoring vectors)	✅
Compute	CPU only	No GPU dependencies in requirements.txt	✅
Network	Off during ranking	No HTTP/socket code anywhere in lib/	✅
Disk	≤ 5 GB intermediate	~1.3 GB Parquet	✅
Determinism	Required	Fixed REFERENCE_DATE, hash templates, no RNG	✅

---

11. File Map

redrob_ranker/
├── README.md                    # This document
├── architecture.png             # Architecture diagram
├── requirements.txt             # pandas, numpy, scikit-learn, pyarrow
├── submission_metadata.yaml     # Team identity, AI tools declaration, methodology
├── precompute.py                # Phase 1 entry — extracts ~55 features → Parquet
├── rank.py                      # Phase 2 entry — scores, top-100, reasoning → CSV
├── validate_submission.py       # Format validator (mirrors hackathon validator)
├── test_adversarial.py          # 10-case adversarial test suite
└── lib/
    ├── __init__.py
    ├── schema.py                # Canonical accessors (unified_text_blob, signals, etc.)
    ├── constants.py             # REFERENCE_DATE, role taxonomy
    ├── jd_parser.py             # JD → JDUnderstanding dataclass
    ├── jd_requirements.py       # Hardcoded fallback JD requirements
    ├── hiring_intent.py         # JD → HiringIntent dataclass
    ├── query_expansion.py       # JD ideal-text → expanded query
    ├── candidate_profile.py     # Raw JSON → CanonicalProfile
    ├── features.py              # ~55 feature extract functions (G1–G13)
    ├── feature_registry.py      # Plugin-style feature registry
    ├── embeddings.py            # TF-IDF + SVD embedder
    ├── retrieval.py             # BM25/dense retrieval (placeholder)
    ├── rrf.py                   # Reciprocal Rank Fusion helper
    ├── scoring.py               # elite_score_vec, final_score_vec, weights
    ├── reasoning.py             # Evidence-graph reasoning generator
    ├── evidence.py              # Evidence extraction utilities
    ├── evidence_graph.py        # Evidence graph data structure
    ├── career_narrative.py      # Career coherence / trajectory classifier
    ├── company_tier.py          # Company → tier mapping
    ├── title_scoring.py         # Title similarity scoring
    ├── domain.py                # Skill/domain taxonomy
    ├── honeypot.py              # Honeypot detector
    └── failure_analyzer.py      # Diagnostic for low-score candidates

Per-file Role Summary

File	Lines	Role
precompute.py	517	Phase 1 orchestrator — JSONL → Parquet
rank.py	332	Phase 2 orchestrator — Parquet → CSV + reasoning
lib/features.py	1,244	All ≈55 feature extract functions
lib/reasoning.py	650	Evidence-graph reasoning templates and verifier
lib/candidate_profile.py	472	Canonical profile normalization
lib/evidence.py	536	Evidence extraction utilities
lib/jd_parser.py	655	JD → structured requirements
lib/scoring.py	417	Elite composite + final score (vectorized)
lib/feature_registry.py	507	Plugin-style feature registry
lib/hiring_intent.py	292	JD → hiring intent dataclass
lib/career_narrative.py	222	Career coherence classifier
lib/failure_analyzer.py	239	Low-score diagnostic
lib/jd_requirements.py	154	Hardcoded JD fallback
lib/query_expansion.py	175	Query expansion for embeddings
lib/company_tier.py	168	Company → tier mapping
lib/domain.py	197	Skill / domain taxonomy
lib/evidence_graph.py	216	Evidence graph data structure
lib/rrf.py	147	Reciprocal Rank Fusion helper
lib/title_scoring.py	125	Title similarity
lib/embeddings.py	67	TF-IDF + SVD embedder
lib/schema.py	99	Canonical accessors
lib/honeypot.py	73	Honeypot detector
lib/constants.py	18	REFERENCE_DATE + role constants
test_adversarial.py	280	10 adversarial test cases
validate_submission.py	165	Format validator
Total	~7,970

---

12. Adversarial Test Suite

test_adversarial.py runs 10 hand-crafted candidates through the full feature-extraction and scoring pipeline, asserting that each is handled correctly. This is the regression suite — every code change must keep all 10 tests green.

python test_adversarial.py
# Expected: 10/10 tests pass, with per-test diagnostics showing which features fired and the resulting score.

#	Test Name	What It Validates	Expected Outcome
1	keyword_stuffer	Many AI skills in skills[], zero career context	Low score; keyword_stuffing_risk high; excluded from top 100
2	hr_manager_with_ai_keywords	Non-engineering title with RAG/LLM skills	Title floor fires; score capped at 0.20
3	strong_candidate	Ideal JD match — 6–8 YOE, shipped ranking systems, recently active	Top-10 score; strong-fit reasoning template
4	career_chaos	Random domain jumps every 6 months	narrative_type = "chaotic"; behavioural penalty; excluded
5	title_inflation	Junior with 2 YOE claiming "Architect"	narrative_suspicious flag; promotion_velocity red flag; excluded
6	stable_upward	Coherent upward trajectory at one company	narrative_type = "upward_specialist"; small boost
7	production_owner_no_impact	Owns systems but no quantified outcomes	ownership high but impact_magnitude low; middling score
8	research_only	Strong publications, no production deployment	Disqualifier fires; 0.05× multiplier; excluded
9	honeypot	Impossible profile (8 years PyTorch, 1 YOE Principal)	Honeypot flag; 0.01× multiplier; excluded
10	interaction_features	High ownership × production × impact simultaneously	ownership_x_production and impact_x_domain high; cross_validation strong; top-5 score

---

13. Design Decisions & Tradeoffs

Why TF-IDF+SVD over sentence-transformers

The spec forbids network calls during ranking and caps runtime at 5 minutes on CPU. Sentence-transformers (all-MiniLM-L6-v2, ~80 MB) would require bundling the model binary or downloading at runtime (forbidden). Even with the model bundled, encoding 100K candidates at ~50 ms each on CPU amounts to ~5,000 seconds — far beyond the budget. TF-IDF+SVD encodes 100K candidates in ~15 seconds, is bit-for-bit deterministic, has zero binary footprint, and is more interpretable (you can inspect which terms drive a candidate's similarity to the JD). Query expansion closes most of the paraphrase-matching gap by explicitly including synonyms in the query text.

Why rule-based scoring over learned weights

A learned model (XGBoost, neural ranker) would require labeled training data. The hackathon provides none — only a JD and a candidate pool. Self-labeling using the elite composite as a pseudo-target is circular, and LLM-generated labels are forbidden during ranking.

Rule-based scoring with explicit, JD-derived weights has three key advantages:

1. Explainable — every score component maps to a JD requirement, making Stage-4 manual review defensible.
2. No training data needed — works out of the box on any JD.
3. Reproducible — no random initialization, no train/test split, no hyperparameter sweep; same code + same data = same output, always.

The cost is brittleness — a learned model would generalize better to unseen JD patterns. For a single-JD hackathon submission, this tradeoff strongly favours rules.

Why skills[] is excluded from the matching text

The dataset contains explicit keyword-stuffer traps — candidates with skills: [RAG, Pinecone, FAISS, NDCG, LoRA, BM25, Embeddings] but no career context for any of them. Including skills[] in the TF-IDF text blob would let these candidates score artificially high on skill coverage and embedding similarity.

By excluding skills[] from the matching text and scoring skills separately via skill_coverage (which cross-checks each declared skill against the text blob), we get a robust signal: a candidate who declares "RAG" must also have RAG-like language in their career descriptions to receive credit.

Why the score stretch sigmoid

A raw composite of 0.7134, 0.7128, 0.7119, …, 0.6892 looks flat and uninformative — Stage-4 reviewers cannot distinguish rank 1 from rank 50 at a glance. The stretched sigmoid maps the same ordering to 0.987, 0.974, 0.962, …, 0.524, which reads as "confident top, gradual falloff" without changing relative ordering or affecting any rank-based metric (NDCG, MAP, P@10).

Why 14 sub-signals in the tiebreaker

Score ties after rounding to 6 decimal places are rare but possible, especially among bottom-50 filler candidates. The spec requires deterministic tie-breaking with candidate_id ascending as the final fallback. Prepending 14 ordered sub-signals ensures ties break in a principled order (Tier-5 signature first, then pre-LLM×Ownership, then impact magnitude, etc.) rather than purely alphabetically, making the top-10 ordering more meaningful when ties occur near the cutoff.

---

14. Determinism & Reproducibility Guarantees

Source of Non-determinism	How We Eliminate It
Calendar date (recency, decay)	REFERENCE_DATE = 2026-06-17 pinned in lib/constants.py
Random number generation	No random module used in any feature or scoring path
Hash randomization (PYTHONHASHSEED)	Template selection uses hashlib.md5(candidate_id), not built-in hash()
Float summation order	NumPy operations on fixed-shape arrays — order is deterministic
Sort stability	Explicit ascending=[False, True] on ["score", "candidate_id"]
TF-IDF vocabulary order	TfidfVectorizer builds vocabulary in a single pass; same input → same vocab
SVD initialization	TruncatedSVD uses random_state=0 explicitly
External API calls	None — no HTTP/socket code anywhere in lib/

Verification: Running python rank.py --candidates ./candidates.jsonl --out ./submission.csv twice on the same machine produces byte-identical CSV output. validate_submission.py passes on both runs.

---

15. Constraints Compliance Matrix

Cross-referenced against submission_spec.md §3.

Constraint	Spec Limit	Our Submission	How to Verify
Total runtime	≤ 5 min	~200 s (precompute + rank)	time python rank.py …
Memory	≤ 16 GB RAM	~1.3 GB peak	python -c "import resource; …"
Compute	CPU only	No GPU dependencies	grep -i cuda requirements.txt → empty
Network	Off during ranking	No HTTP/socket code	grep -rE "requests|http|socket|urllib" lib/ → empty
Disk	≤ 5 GB intermediate	~1.3 GB Parquet	du -sh artifacts/
Output format	CSV, 100 rows, 4 cols	Exactly that	python validate_submission.py submission.csv
Score monotonicity	Non-increasing by rank	Enforced + validated	validate_output() in rank.py
Tie-breaking	Deterministic	14 sub-signals + candidate_id asc	See §7
Honeypot rate in top 100	< 10%	Design target: 0%	top["is_honeypot"].sum() printed by rank.py
AI tools declaration	Required	Declared in submission_metadata.yaml	cat submission_metadata.yaml

---

16. Known Limitations

The following is an honest engineering disclosure of what this system does not handle well, so Stage-4 reviewers can probe knowingly.

1. No dense paraphrase matching. TF-IDF+SVD with query expansion covers most synonym cases, but a candidate who describes RAG work as "context-augmented generation with vector stores" without using any canonical terms may under-score. A natural extension would add a small local sentence-transformer for top-100 re-ranking after the cheap TF-IDF pre-filter.

2. Education tier is India-centric. The education_tier feature maps IIT/NIT/IIIT to tiers 1–2 and treats everything else as tier 3. For non-Indian candidates this is a weak signal. Acceptable for this JD (India-located role) but would need rework for global pools.

3. Salary compatibility is heuristic. The salary_compatibility feature compares expected_salary_range to an implicit band derived from the JD's seniority and location. The band is hardcoded (not learned), so it may misjudge edge cases such as candidates willing to take a pay cut for equity.

4. Reasoning templates are finite. Approximately 15 templates per "shape" (strong-fit, behavioral-concern, etc.) are shipped. For a top-100 submission this gives ~6–7 candidates per template, which the spec's "Variation" check accepts. A top-1000 submission would require more templates or a precomputed template library.

5. No online learning. The system is entirely batch — precompute + rank. It does not adapt to recruiter feedback. This is intentional for an offline hackathon but would be the first addition in production via a click-through rate signal.

6. Adversarial test coverage is hand-crafted. The 10 tests in test_adversarial.py cover the trap types described in redrob_signals_doc.md, but a determined adversary could craft candidates that evade all 10. A stronger suite would generate adversarial candidates programmatically via mutation fuzzing on real candidates.

---

End of README — Redrob Intelligent Candidate Ranker