pairwise_llm_check/README.md

experiments/pairwise_llm_check/

What This Experiment Does

This is an offline experiment that generates better LightGBM training labels by replacing the heuristic weak_label = hard_req_coverage × consistency_score × jd_penalty with LLM pairwise judgments on sampled Stage 1 candidates.

Pipeline Summary

1. Load Stage 1 BM25 retrieval pool.
2. Stratified sample of candidates weighted toward the current model's top and boundary regions.
3. Generate pairwise matchups; annotate with quantized LLaMA via Ollama.
4. Convert pairwise verdicts → Elo ratings → 0–3 integer relevance labels.
5. Retrain LightGBM on these labels using identical hyperparameters to precompute.py.
6. Save the new model as precomputed/lgbm_model_llm.pkl.
7. Print a comparison report: top-10 overlap, Spearman correlation, honeypot audit.