README.md

---
license: apache-2.0
language:
- ko
base_model: dragonkue/bge-reranker-v2-m3-ko
pipeline_tag: text-ranking
library_name: sentence-transformers
tags:
- sentence-transformers
- cross-encoder
- reranker
- korean
- legal
- retrieval
---

# EVE-Rerank-1.0-Legal

Korean cross-encoder reranker for **legal precedent retrieval**, fine-tuned from
[dragonkue/bge-reranker-v2-m3-ko](https://huggingface.co/dragonkue/bge-reranker-v2-m3-ko).

Stage 2 of a retrieval pipeline: it reorders the top 20 candidates from
[EVE-Embed-1.1-Legal](https://huggingface.co/ysmeta/EVE-Embed-1.1-Legal).

| pipeline | graded nDCG@10 | P@1 (any relevant) |
|---|---|---|
| **EVE-Embed-1.1 + EVE-Rerank-1.0** | **0.7721** | **0.9355** |
| EVE-Embed-1.1 + dragonkue/bge-reranker-v2-m3-ko | 0.7506 | 0.9110 |
| EVE-Embed-1.1 alone | 0.7253 | 0.8940 |
| dragonkue/BGE-m3-ko alone | 0.6557 | 0.8214 |

Scored on [EVE-Bench-Legal-2.0](https://huggingface.co/datasets/ysmeta/EVE-Bench-Legal-2.0):
5,998 practitioner-style queries, retrieval over the full 59,786-precedent corpus, graded
multi-gold relevance labels.

**P@1 is the number that matters in use**: the pipeline puts a genuinely relevant
precedent first for **93.6%** of queries, against 89.4% for retrieval alone.

All gaps are significant under a paired bootstrap over the same queries (10,000 resamples):

| comparison | Δ nDCG@10 | 95% CI | p | win/loss |
|---|---|---|---|---|
| vs. the public reranker | **+0.0214** | [+0.0194, +0.0235] | <0.0001 | 3069 / 1628 |
| vs. retrieval alone | +0.0468 | [+0.0429, +0.0506] | <0.0001 | 3444 / 1870 |

## Usage

```python
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("ysmeta/EVE-Rerank-1.0-Legal", max_length=512)

query = "계약금을 이미 받았는데도 상대방이 계약을 취소할 수 있나요?"
candidates = [...]                      # top-20 docs from EVE-Embed-1.1-Legal

scores = reranker.predict([[query, d] for d in candidates])
ranked = [d for _, d in sorted(zip(scores, candidates), key=lambda x: -x[0])]
```

Reranking 20 candidates costs 20 cross-encoder forward passes per query. Going beyond
top-20 buys little: 93.6% of queries already have a relevant precedent at rank 1.

## What made this work: relevant candidates are positives, not noise

A first attempt (v1) trained against **verified hard negatives** — candidates that an LLM
judge confirmed do *not* answer the query — and discarded the ~30% it judged relevant.
That is the recipe that works for the [embedding model](https://huggingface.co/ysmeta/EVE-Embed-1.1-Legal),
where unverified negatives cost −36.5%. For a reranker it was optimising the wrong target.

Legal precedents cluster by 쟁점, so a query typically has **7.20 genuinely relevant
precedents**, not one. The task is to rank *all* of them above the irrelevant ones.
Discarding the relevant siblings meant v1 never trained on the decision it is scored on.
Its evaluation curve fell from its very first measurement while the loss went down.

v2 turns every judged-relevant candidate into its own training row sharing the query's
irrelevant candidates as negatives:

| run | training signal | graded nDCG@10 | vs public reranker |
|---|---|---|---|
| v1 | verified negatives, relevant candidates discarded | 0.7540 | +0.0034 |
| **v2 (this model)** | **every relevant candidate a positive** | **0.7721** | **+0.0214** |

Same base model, same data, same judge — a 6× larger margin from changing what counts
as a positive. The curve inverted too: v1 ran 0.8822 → 0.8335 (best at its first eval),
v2 ran 0.8152 → 0.8362 (best at its last).

## Training

- **Base**: dragonkue/bge-reranker-v2-m3-ko (Apache-2.0)
- **Data**: 23,600 queries → 41,706 rows (1.77 positives per query, 4 negatives each),
  from [EVE-Train-Legal-Rerank-1.0](https://huggingface.co/datasets/ysmeta/EVE-Train-Legal-Rerank-1.0)
- **Candidates** come from EVE-Embed-1.1's own top-15, judged by
  [tencent/Hy3](https://huggingface.co/tencent/Hy3). A reranker trained on some other
  retriever's candidates solves a different problem than the one it is deployed into.
- **Loss**: CachedMultipleNegativesRankingLoss (listwise softmax over 1 positive + 4 negatives)
- **LR**: 2e-6, 1 epoch, effective batch 32, max_length 512, bf16, 1,304 steps (~76 min)
- **Checkpoint selection**: `load_best_model_at_end` on a held-out graded slice, with
  `eval_on_start=True` so the untrained reference sits inside the curve
- **Hardware**: 1 × NVIDIA H200

v1 used lr 1e-5 and degraded monotonically from step 100 — too high for an
already-trained reranker.

## Limitations

- **Tuned for this pipeline.** Candidates come from EVE-Embed-1.1; behaviour on another
  retriever's candidate distribution is untested.
- **Relevance labels are LLM-generated**, not lawyer-reviewed, and inherit the judge's
  reading of what "answers" a legal question.
- **판례 only**, no statute text (법령 조문). Case types: 민사 46.3% · 형사 21.9% ·
  일반행정 14.1% · 세무 11.9% · 특허 4.3% · 가사 1.6%.
- Benchmark queries are LLM-generated from their source precedent and carry traces of it,
  so part of the measured score reflects trace-matching rather than legal understanding.
  See the [benchmark card](https://huggingface.co/datasets/ysmeta/EVE-Bench-Legal-2.0)
  for the evidence.
- If you would rather not add a second model, `dragonkue/bge-reranker-v2-m3-ko` gets you
  from 0.7253 to 0.7506 for free — 54% of this model's gain.
- Not legal advice. Ranking surfaces precedents; it does not interpret them.

## License

Apache-2.0, inherited from the base model.