Create README.md

README.md

@@ -0,0 +1,232 @@
+---
+language:
+- he
+tags:
+- hebrew
+- semantic-retrieval
+- information-retrieval
+- dense-retrieval
+- reranking
+- rrf
+- sentence-transformers
+- competition
+pipeline_tag: sentence-similarity
+license: other
+---
+
+# Hebrew Semantic Retrieval — 2nd Place Solution
+
+**Competition:** Hebrew Semantic Retrieval Challenge by MAFAT DDR&D (Directorate of Defense Research & Development) in partnership with the **Israel National NLP Program**
+
+**Result:** 🥈 **2nd place** — nDCG@20 = **0.656792** (private test set) · **0.460408** (public test set)
+
+**Author:** itk77
+
+---
+
+## Overview
+
+This repository contains the complete inference code and fine-tuned models for the 2nd-place solution to the **Hebrew Semantic Retrieval Challenge**. The challenge tasked participants with ranking Hebrew paragraphs from a 127,731-passage corpus in response to natural-language Hebrew queries, evaluated by **NDCG@20**.
+
+Hebrew is a morphologically rich Semitic language written in an almost consonant-only script, creating significant lexical ambiguity and making retrieval substantially harder than for high-resource languages. The solution addresses this with a carefully engineered three-stage pipeline: sparse + dual-dense retrieval fused via Weighted Reciprocal Rank Fusion (WRRF), followed by a BGE cross-encoder reranker fine-tuned specifically on the challenge corpus, and a final conditional score blending step.
+
+---
+
+## The Challenge
+
+| Property | Detail |
+|---|---|
+| Organizer | MAFAT DDR&D + Israel National NLP Program |
+| Corpus size | 127,731 Hebrew paragraphs |
+| Data sources | Hebrew Wikipedia, Kol-Zchut (legal/civil-rights), Knesset committee protocols |
+| Evaluation metric | NDCG@20 |
+| Phase I | Public leaderboard (Codabench) |
+| Phase II | Private test set with additional human annotation of previously unseen retrievals |
+| Relevance scale | 0–4 (human annotated) |
+
+---
+
+## Solution Architecture
+
+The solution is a **three-stage pipeline**: sparse + dual-dense retrieval fused with Weighted RRF, cross-encoder reranking, and conditional score blending.
+
+```
+Query
+  │
+  ├─► [BM25  (k1=1.3, b=0.7, w=1.0)]  ──┐
+  ├─► [E5-large fine-tuned (w=1.2)]  ├─► WRRF Fusion (k=35)
+  └─► [multilingual-E5-large (w=1.4)]  ┘
+            │
+            ▼
+     Top-190 Candidates
+            │
+            ▼
+     [BGE Cross-Encoder Reranker]  (fine-tuned, max_len=640)
+            │
+            ▼
+     Conditional Score Blending
+            │
+            ▼
+     Final Top-20 Results
+```
+
+### Stage 1 — Weighted Reciprocal Rank Fusion (WRRF)
+
+Three independent rankers each produce a ranked list of up to 190 candidates. Their lists are fused using **Weighted Reciprocal Rank Fusion**:
+
+$$\text{WRRF}(d) = \frac{w_\text{BM25}}{k + r_\text{BM25}(d) + 1} + \frac{w_\text{E5-ft}}{k + r_\text{E5-ft}(d) + 1} + \frac{w_\text{E5-base}}{k + r_\text{E5-base}(d) + 1}$$
+
+with $k = 35$ (RRF smoothing constant).
+
+| Ranker | Model | Weight | Max Length | Notes |
+|---|---|---|---|---|
+| BM25 | Custom Hebrew BM25 (bm25s backend) | 1.0 | — | Strip nikkud, NFKC norm, prefix stripping |
+| E5 (fine-tuned) | `e5-large-ft_v6` | 1.2 | 512 tokens | Mean pooling + L2 norm, `query:` / `passage:` prefixes |
+| E5 (base) | `multilingual-e5-large` | 1.4 | 512 tokens | Via SentenceTransformers, BF16; labeled `GemmaEmbedder` in code but loads E5 |
+
+**Hebrew-specific tokenization (BM25):** Unicode NFKC normalization, nikkud stripping (`\u0591–\u05C7`), Hebrew prefix removal (`ו`,`ה`,`ב`,`ל`,`כ`,`מ`,`ש`) with both the stripped and original form indexed, and a custom Hebrew stopword list.
+
+### Stage 2 — BGE Cross-Encoder Reranking
+
+The top-190 WRRF candidates are reranked by `bge-reranker-hsrc-pairwise-rrf-V1.4`, a BGE cross-encoder fine-tuned on the challenge corpus using **pairwise training with RRF-mined triples**. Pairs are scored with a max sequence length of 640 tokens.
+
+### Stage 3 — Conditional Score Blending
+
+The final score uses a non-linear conditional boost that amplifies the WRRF signal where the reranker is uncertain:
+
+$$\text{score}_\text{final} = \hat{s}_\text{BGE} + (1 - w_\text{BGE}) \cdot \hat{s}_\text{WRRF} \cdot (1 - \hat{s}_\text{BGE})$$
+
+where $w_\text{BGE} = 0.07$, and both scores are **min-max normalized** to $[0, 1]$ over the candidate pool. When the reranker assigns a high score ($\hat{s}_\text{BGE} \approx 1$), the WRRF boost vanishes; when it is uncertain ($\hat{s}_\text{BGE} \approx 0$), the WRRF signal takes over.
+
+---
+
+## Included Models (fine-tuned)
+
+| Path in repo | Base model | Fine-tuning |
+|---|---|---|
+| `models/e5-large-ft_v6/` | `intfloat/multilingual-e5-large` | Fine-tuned on the challenge corpus (v6 checkpoint) |
+| `models/bge-reranker-hsrc-pairwise-rrf-V1.4/` | `BAAI/bge-reranker-v2-m3` | Fine-tuned on RRF-mined pairwise triples from the challenge corpus |
+| `models/multilingual-e5-large/` | `intfloat/multilingual-e5-large` | Off-the-shelf (no fine-tuning) |
+
+---
+
+## Repository Structure
+
+```
+model.py              ← Full inference pipeline (preprocess + predict)
+bm25_backends.py      ← Pluggable BM25 backends (bm25s / pure-Python fallback)
+text_utils.py         ← Hebrew normalization & tokenization utilities
+models/
+  e5-large-ft_v6/                          ← Fine-tuned E5 embedder ✨
+  bge-reranker-hsrc-pairwise-rrf-V1.4/     ← Fine-tuned BGE reranker ✨
+  multilingual-e5-large/                   ← Off-the-shelf secondary embedder
+```
+
+---
+
+## Usage
+
+The pipeline exposes two functions matching the competition API:
+
+```python
+from model import preprocess, predict
+
+# Build corpus index (run once)
+# corpus_dict: {doc_id: {"passage": "..."}, ...}
+preprocessed = preprocess(corpus_dict)
+
+# Query at inference time
+results = predict({"query": "מה הזכויות של שוכרי דירה?"}, preprocessed)
+# Returns: [{"paragraph_uuid": "...", "score": 0.87}, ...]  (top-20)
+```
+
+**Requirements:**
+```
+torch
+transformers
+sentence-transformers
+bm25s
+scikit-learn
+numpy
+```
+
+A CUDA-capable GPU is strongly recommended (two large encoder models + one cross-encoder are loaded simultaneously, all in BF16/FP16).
+
+---
+
+## Training Pipeline
+
+The full training pipeline is located in `repro/documentation/complete_pipeline/` and orchestrated by `pipeline.py`. It automates four sequential stages:
+
+| Stage | Script | Description |
+|---|---|---|
+| 1 | `finetune_e5_large.py` | Fine-tunes E5 on the challenge corpus (12 runs, 2 epochs, lr=2e-6, batch=4) |
+| 2 | `stage1_weight_sweep.py` | Offline grid sweep of WRRF weights (BM25, E5, Gemma) |
+| 3 | `train_bge_ce_pairwise_rrf.py` | Trains the BGE cross-encoder reranker (lr=2e-5, max_len=640, batch=4×accum=8) |
+| 4 | `sweep_final2_from_components.py` | Offline sweep for the final blending weight |
+
+### Reranker Training Modes
+
+The pipeline supports two parallel reranker training paths:
+
+- **Deterministic mode** (`--rr_det_runs`): trains from **pinned triples** (`repro/documentation/triples/triples.jsonl`), enabling reproducible results.
+- **Non-deterministic mining mode** (`--rr_nd_runs`): the first run mines fresh triples from the best E5 checkpoint; subsequent runs reuse them. ~1 in 7 runs matches submitted model quality.
+
+### Example Full Run Command
+
+```bash
+python3 repro/documentation/complete_pipeline/pipeline.py \
+  --e5_runs 12 --e5_seed0 45 --e5_seed_stride 0 \
+  --e5_epochs 2 --e5_batch 4 --e5_lr 2e-6 \
+  --stage1_w_bm25 1.0,2.0,0.1 \
+  --stage1_w_e5 1.0,2.0,0.1 \
+  --stage1_w_gm 1.0,2.0,0.1 \
+  --rr_det_runs 1 --rr_det_seed0 42 --rr_det_seed_stride 0 \
+  --rr_det_triples_in repro/documentation/triples/triples.jsonl \
+  --rr_nd_runs 15 --rr_nd_seed0 42 --rr_nd_seed_stride 0 \
+  --rr_bsz 4 --rr_accum 8 --rr_lr 2e-5 --rr_max_len 640 \
+  --rr_sweep_rounds 2000
+```
+
+**Hardware:** Original model trained on RTX 3080 Ti; reproducibility runs executed on L40S (~24 hours for the full pipeline with 12 E5 + 15 reranker runs).
+
+---
+
+## Evaluation Protocol
+
+- **Holdout set:** First 100 queries of the provided training file (fixed split, never changed during development).
+- **Local evaluation script:** `scripts/eval_std_final.py` — runs silently when `EVAL_STD_MODE=1`.
+- **Score discrepancy:** 7 of the 100 holdout queries have no labels > 0 (empty relevance). The local script does not ignore these by default, resulting in a local nDCG ~0.615 vs. the public leaderboard score. When empty-label queries are excluded, local scores align with the official leaderboard.
+
+---
+
+## Technical Notes
+
+- All models are loaded in **BF16** (E5, Gemma) or **FP16** (BGE reranker) to reduce GPU memory usage.
+- **Corpus embedding caching:** E5 and Gemma corpus embeddings can be cached to disk (keyed by SHA-1 of document IDs + model path + corpus size) to skip re-encoding on repeated runs.
+- **BM25 backend fallback chain:** `bm25_backends.py` → direct `bm25s` → pure-Python deterministic BM25 (guaranteed to work without external dependencies).
+- **Dominant source of non-determinism:** GPU FP16/SDPA kernel behavior. Deterministic kernels are available but increase runtime ~3.6× and may exceed GPU memory limits.
+
+---
+
+## Results
+
+| Phase | NDCG@20 | Rank |
+|---|---|---|
+| Public (Phase I) | **0.460408** | 🥈 2nd |
+| Private (Phase II) | **0.656792** | 🥈 2nd |
+
+> The large gap between public and private scores is expected: the private phase incorporated additional human annotation of previously un-annotated retrieved documents, significantly impacting NDCG for systems that retrieved relevant but un-annotated paragraphs.
+
+---
+
+## Citation
+
+If you use this solution or the models in this repository, please acknowledge the **Hebrew Semantic Retrieval Challenge** by MAFAT DDR&D and the Israel National NLP Program, and credit **itk77** as the solution author.
+
+---
+
+## Acknowledgements
+
+- MAFAT DDR&D and the **Israel National NLP Program** for organizing the challenge and providing the annotated Hebrew corpus.
+- The authors of `intfloat/multilingual-e5-large` and `BAAI/bge-reranker-v2-m3`.