docs/offline-phase.md

# Offline Phase — Step by Step

> Everything here runs with internet on, no time limit. Output artifacts feed the
> 5-minute ranking step. See [architecture/spec.md](../architecture/spec.md) for the big picture.

## 5.1 Embedding (precompute_embeddings.py)

**Model:** `BAAI/bge-small-en-v1.5` — 33M params, 384 dims, ~4x faster than MiniLM-L6 on CPU,
MTEB scores competitive. Ships in ~130MB. Alternatively `minishlab/potion-base-8M` (Model2Vec)
for a pure-numpy fallback that embeds 100K in ~30 seconds.

**What to embed — per-role utterances, NOT whole profiles:**
Each candidate gets a list of text chunks, not one blob. Research finding from Malt (2026):
embedding per-utterance enables late-interaction scoring and per-requirement coverage computation.

```python
def candidate_to_chunks(c: dict) -> list[str]:
    chunks = []
    # Role descriptions — most signal-rich field
    for job in c.get("career_history", []):
        if job.get("description"):
            chunks.append(f"{job['title']} at {job['company']}: {job['description']}")
    # Summary if present
    if c["profile"].get("summary"):
        chunks.append(c["profile"]["summary"])
    # Headline
    if c["profile"].get("headline"):
        chunks.append(c["profile"]["headline"])
    return chunks or [c["profile"].get("current_title", "")]
```

Save:
- `candidate_embeddings.npy` — shape `(N_chunks_total, 384)`, float16 to halve disk
- `candidate_ids.json` — list of candidate_ids, one per chunk row (allows grouping back)
- At load time, group by candidate_id and pool (max or mean) for global candidate vector

**Runtime estimate:** bge-small + batch_size=512 → ~8–12 min for 100K on 8-core CPU.
This is pre-computation; no time limit.

## 5.2 Hypothetical Resume Generation (generate_hypothetical.py)

**Why:** ConFit v2 (ACL 2025) showed 17.5% nDCG improvement by embedding a hypothetical
ideal resume alongside the JD before scoring. The format asymmetry (discursive JD vs structured
resume) kills naive cosine similarity. Generating ideal resumes bridges the gap.

**Novel addition: anti-persona resumes.** Not in any paper. Generates negative anchors
matching exactly the failure modes Redrob planted in the dataset.

**Prompt for ideal resumes:**
```python
IDEAL_PERSONA_PROMPT = """
You are generating a realistic candidate profile for the following job description.
Create a profile that would be a STRONG FIT — a real person with consistent career history,
specific accomplishments, and natural language (no keyword stuffing).

Generate {n} different profiles. Each should be a different archetype:
1. The IR veteran — 8yr, built search/ranking at product company pre-LLM era, now adding modern ML
2. The startup ML shipper — 6yr, 2-3 startups, shipped RAG/rec-sys to real users, scrappy
3. The platform engineer — 7yr, vector DB + hybrid search infra, scale-focused
4. The applied researcher — 5yr, MSc/PhD but in industry, eval frameworks, A/B testing mindset
5. The product-ML hybrid — 6yr, ex-PM turned engineer, retrieval + ranking + product instincts

For each profile, write:
- headline (1 line)
- summary (3-4 sentences, natural language, no buzzwords)
- 2-3 job roles with title, company type, duration, description (50-100 words each)
- 5-7 skills with realistic experience durations

JOB DESCRIPTION:
{jd_text}

Return JSON array of profiles.
"""
```

**Prompt for anti-persona resumes (our novel contribution):**
```python
ANTI_PERSONA_PROMPT = """
Generate {n} profiles that would seem relevant on surface but are explicitly
disqualified by the following job description.

The JD explicitly says these are NOT wanted:
1. Keyword stuffer — Marketing Manager with every AI keyword in skills, but career is marketing
2. Pure researcher — academic lab career, never shipped to production users
3. Consulting lifer — entire career at TCS/Infosys/Wipro/Accenture, no product company
4. Framework enthusiast — only LangChain/OpenAI wrapper projects, no pre-LLM ML experience
5. Title chaser — avg tenure <18 months across 4+ jobs, optimizing for "Senior" → "Staff"

Each profile should look superficially plausible but fail the actual JD requirements.

JOB DESCRIPTION:
{jd_text}

Return JSON array of profiles.
"""
```

**Final query vectors:**
- Embed each ideal+anti-persona resume as chunks
- `jd_query_vectors.npy` — shape `(n_ideals + n_anti, 384)` with metadata flag (positive/negative)
- At retrieval time: `sim_positive = max(cosine(candidate, ideals))`,
  `sim_negative = max(cosine(candidate, anti_personas))`,
  `semantic_score = sim_positive - 0.4 * sim_negative`

## 5.3 Stratified Sampler (stratified_sampler.py)

Claude teacher labels are expensive (API cost) and take time. Sample smartly.
Want coverage across the full relevance spectrum, including hard cases.

```python
def stratified_sample(candidates, embeddings, n=2500):
    strata = {
        "top_retrieval_bm25":    200,  # top BM25 hits — high relevance likely
        "top_retrieval_dense":   200,  # top dense hits — captures plain-language Tier 5s
        "top_anti_persona_sim":  150,  # high sim to anti-personas — keyword stuffers
        "title_match_strong":    200,  # current_title contains engineer/ML/AI
        "title_mismatch":        150,  # high skill-match, wrong title (stuffer detection)
        "consulting_only":       100,  # all career at big 5 IT services
        "honeypot_flagged":      100,  # caught by consistency engine
        "high_behavioral":       150,  # top redrob_signals scores
        "low_behavioral":        150,  # poor behavioral signals, maybe good profile
        "tier1_education":       100,  # IIT/IIM/NIT tier_1 candidates
        "random":                1000, # uniform random for distributional coverage
    }
    # Returns list of (candidate_id, stratum_label)
```

## 5.4 Teacher Labeling (teacher_label.py)

**Model:** Claude (via Anthropic API). `claude-sonnet-4-6` or `claude-haiku-4-5-20251001`
for cost efficiency. Haiku is ~20x cheaper and sufficient for labeling.

**Malt's two techniques (both required for label quality):**

1. **Semantic rubric anchoring** — fixed scale baked into the prompt so scores mean
   the same thing across all batches:
```
0.0 — No relevant skills or experience. Completely unable to perform the job.
0.2 — Minor relevance. Some adjacent skills but fundamentally wrong profile.
0.4 — Moderate match. Some relevant skills, significant gaps on core requirements.
0.6 — Good match. Mostly relevant, can perform with some ramp-up. Meets most requirements.
0.8 — Strong match. Highly relevant skills and experience. Ready to perform well.
1.0 — Perfect match. Skills and experience fully aligned. Expert on the topic.
```

2. **Anchored batching** — always include 1 obvious 0.0 and 1 obvious 1.0 as anchors
   in every batch of 12 candidates. Forces consistent calibration across batches.

**Batch prompt structure:**
```python
TEACHER_PROMPT = """
You are an objective evaluator for a recruiting platform.

JOB DESCRIPTION:
{jd_text}

SCORING RUBRIC (use ONLY these values):
0.0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0
{rubric_text}

Below are {n} candidate profiles. Score each independently.
Profile 1 is a known PERFECT FIT (score must be 0.9-1.0).
Profile {n} is a known NON-FIT (score must be 0.0-0.1).
Score profiles 2 through {n-1} based solely on the rubric.

For each candidate, provide:
- score: float (0.0, 0.2, 0.4, 0.6, 0.8, or 1.0)
- rationale: 1 sentence citing specific evidence from their profile

Return JSON array with fields: candidate_id, score, rationale

CANDIDATES:
{candidate_profiles_json}
"""
```

**What to include per candidate for the teacher (keep minimal to save tokens):**
```python
def candidate_for_teacher(c: dict) -> dict:
    return {
        "candidate_id": c["candidate_id"],
        "current_title": c["profile"]["current_title"],
        "years_of_experience": c["profile"]["years_of_experience"],
        "summary": c["profile"]["summary"][:400],  # truncated
        "career": [
            {
                "title": j["title"],
                "company": j["company"],
                "industry": j["industry"],
                "company_size": j["company_size"],
                "duration_months": j["duration_months"],
                "description": j["description"][:200]
            }
            for j in c["career_history"][:4]
        ],
        "skills_top5": [
            {"name": s["name"], "proficiency": s["proficiency"],
             "endorsements": s["endorsements"], "duration_months": s.get("duration_months", 0)}
            for s in sorted(c.get("skills", []),
                           key=lambda x: x["endorsements"], reverse=True)[:5]
        ],
        "education": [
            {"degree": e["degree"], "field": e["field_of_study"],
             "institution": e["institution"], "tier": e.get("tier")}
            for e in c.get("education", [])[:2]
        ]
    }
```

Save `teacher_labels.csv` with columns: `candidate_id, score, rationale, stratum`.

**Quality check before training:** compute self-consistency by double-labeling 100 candidates
with a fresh prompt. If Pearson correlation of scores < 0.85, the rubric needs tightening.

## 5.5 Train LambdaMART (train_ranker.py)

**Why LambdaMART:** Directly optimizes NDCG (the competition metric). LinkedIn production
talent search uses LTR with embedding features. XGBoost `rank:ndcg` is the standard implementation.

```python
import xgboost as xgb

dtrain = xgb.DMatrix(X_train, label=y_train)
dtrain.set_group(group_sizes_train)  # required for LTR

params = {
    "objective": "rank:ndcg",
    "eval_metric": "ndcg@10",
    "eta": 0.05,
    "max_depth": 6,
    "min_child_weight": 5,
    "subsample": 0.8,
    "colsample_bytree": 0.8,
    "n_estimators": 500,
    "tree_method": "hist",
    # Monotonic constraints: behavioral signals should have monotone positive effect
    # Feature index must match column order in X
    "monotone_constraints": "(0,0,0,...,1,1,1,...)",  # fill after feature list is finalized
}

model = xgb.train(params, dtrain, evals=[(dval, "val")], early_stopping_rounds=30)
model.save_model("artifacts/ranker_model.json")
```

**Hold-out eval:** use 20% of teacher-labeled candidates as validation set.
Compute NDCG@10 locally. Run ablations (no behavioral signals, no anti-persona, etc.) —
this directly becomes your Stage 4 methodology and Stage 5 interview material.