GLInt: Geometry-Matched Hard Negatives for Late-Interaction Retrieval

A field report on multi-vector mining, MaxSim geometry, knowledge distillation, and the experiments that did not work.

I started with a narrow question: Do hard negatives mined by a multi-vector retriever produce a better late-interaction model than negatives mined by a dense retriever? Pffftt, Obviously!
I introduce GLInt, which reaches 57.43 mean nDCG@10 on the 15 BEIR Tasks, starting from the unsupervised LateOn checkpoint. It outperforms the 57.22 result of LateOn, making it the strongest retriever under 300M parameters in this comparison. It also sets the highest Quora and HotPotQA score among models below 7B parameters.

The model and training data are available here:

• 🤗 Model: GLInt
• 📚 Training data: GLInt-data
• 🛠️ Tools: PyLate · FastPLAID

Contents

1. Introduction
2. Mining in MaxSim space
3. Hard negatives, false negatives, and filtering
4. From SFT to knowledge distillation
5. Final results
6. Mining at scale
7. Appendix: experiments that did not work
8. References and resources

---

Introduction

Late-interaction retrievers such as ColBERT do not score documents the way conventional dense retrievers do. A dense retriever compresses the query and document into one vector each and compares those vectors directly. A late-interaction retriever keeps multiple token vectors and delays the interaction until scoring time.

That difference is usually discussed as an inference-time advantage. In this project, it turned out to matter just as much during data construction.

Hard-negative mining is itself a ranking problem. Given a query and a document the dataset says is relevant, the miner searches for other documents that look deceptively relevant and uses them as negatives during training. If the miner and the student disagree about similarity, the miner can spend training slots on examples the student finds easy while missing documents the student actually finds confusing.

Related independent work has reached the same first-order conclusion from a different setup. Athrael's MaxSim mining experiment reranks a dense candidate net with MaxSim and shows that geometry-matched selection can help a late-interaction student. My experiments started from the same broad question, but the more useful part turned out to be what happened after making the entire data pipeline multi-vector: the score distribution compressed, dense-space thresholds stopped transferring, false-negative exposure increased, filtering consumed shallow candidate pools, and the same geometry showed up again during distillation.

Before going further, four terms matter:

• A positive is a document that the training dataset explicitly labels as relevant to the query.
• A negative is a document treated as irrelevant for training.
• A hard negative is a negative that the retriever nevertheless scores highly, making it difficult to separate from the positive.
• A false negative is a document treated as a negative even though it is actually relevant, often because the dataset's relevance labels are incomplete.

The better a miner becomes at finding semantically relevant documents, the closer it gets to the boundary where "negative" itself becomes uncertain.

The high-level training progression¹ was:

---

1. Mining in MaxSim space

For a late-interaction retriever, the query and document are represented by sets of token vectors. Following the ColBERT scoring formulation, a simplified MaxSim score is

$$S(q,d)=\sum _i\underset{j}{\max }\cos (q_i,d_j)\mathrm{.}S(q,d)\; =\; \backslash sum\_i\; \backslash max\_j\; \backslash cos(q\_i,\; d\_j).$$

Each query token selects its best matching document token, and those matches are summed.

The measurement that made the geometry tangible was the spread of scores inside a retrieved candidate pool.

The entire MaxSim candidate pool occupied a window of only about 1.4% around a large positive baseline. Every retrieved document matched many query tokens reasonably well, so every document received a substantial score floor. The useful differences were compressed into a narrow band.

That becomes important when positive-aware hard-negative mining enters the picture.

NV-Retriever popularized positive-aware mining rules that compare the score of a candidate against the score of the known positive. In simplified form, a candidate is rejected when

score(query, candidate) > threshold × score(query, positive)

The idea is sensible: if a mined candidate scores suspiciously close to the labelled positive, it may itself be relevant and should not be used as a negative.

The hidden assumption is that the ratio between candidate and positive scores is a stable quantity.

In my MaxSim pool, it was not.

At 0.95, the cutoff fell below even the worst retrieved candidate, so every candidate was rejected. At 1.02, every candidate survived. A movement of only a few percentage points changed the rule from "remove everything" to "remove nothing."

This is more important than a threshold that merely needed retuning. The statistic itself had become poorly conditioned for the score geometry. MaxSim is a sum over token-level maxima, so retrieved documents accumulate a substantial baseline score. The informative differences between hard candidates are compressed into a small region above that floor.

A threshold is not geometry-independent just because it is expressed as a ratio.

The replacement had to be rank-relative rather than ratio-relative: score the known positives and mined candidates with an independent MaxSim judge, then ask whether a candidate ranks above the best known positive.

That preserved the idea behind positive-aware mining while changing the decision rule to match the geometry in which the candidates were actually being judged.

In MaxSim space, re-derive every constant. Thresholds, temperatures, score scales, and filtering rules calibrated for dense retrieval do not transfer automatically.

---

2. Hard negatives, false negatives, and filtering

The first direct audit compared multi-vector and dense negatives for the same (query, positive) pair, scored by the same independent reranker.

The multi-vector negatives were 0.057 harder, consistently across every source. They also contained roughly twice as many false negatives.

Hardness here is the mean relevance score the reranker assigns to a source's negatives, scored in the same listwise pass as the positive; the false-negative rate is the fraction of those negatives that outscore the positive outright. (This is an absolute reranker score, distinct from the mining band, which is expressed relative to each pair's own positive.)

Those are not separate phenomena.

A token-level retriever is better at finding passages that line up with the query's evidence structure. That is exactly how it finds a difficult distractor and also how it finds an unannotated passage that answers the query.

Harder negatives and false negatives are two sides of the same behavior. A stronger miner moves closer to the boundary where the label "negative" becomes uncertain.

A second audit with an independent late-interaction judge found a weighted false-negative rate of 17.6% in the top of the original mined pool. The problem was especially severe in Natural Questions and MS MARCO, where the highest-ranked mined candidate often scored above the annotated positive.

The eventual filtering pipeline used GTE-ModernColBERT-v1 as the late-interaction judge. For each query, it scored the known positives and mined candidates with MaxSim, rejected any candidate ranked above the best known positive, and sampled negatives from the score band immediately below that boundary.

To test whether this actually reduced false negatives, I used a separate model, jina-reranker-v3.5, as an independent post-hoc auditor. Jina was not involved in constructing the filtered dataset. It was used only to compare the original and geometry-matched pools.

That is a 69% reduction in false-negative exposure.

The model improved in the expected direction: the damage relative to the unsupervised base was cut from −4.79 to −2.66 on the diagnostic evaluation.

Filtering was necessary, but it created the next problem.

My first mining run retained only the top 80 candidates per query. The LateOn dense recipe mined to a depth of 2,048, a 25-fold difference. Once candidates that outranked the positive were vetoed, the shallow multi-vector pool often had very little useful material left, especially for MS MARCO and NQ.

Re-mining to depth 2,048 while holding the training recipe fixed reduced the diagnostic damage from −2.66 to −1.02. It was the largest improvement from any negative-selection change.

The pattern was clearer than I expected.

Harder was not monotonically better. Deliberately spreading samples across harder ranks made transfer worse. Hardness had an interior optimum.

Once the pool was deep enough and passed through the same judge, the exact sampling rule mattered much less. Even adding a judged BM25 stream, which contributed very different candidates before filtering, changed the result by only −0.02.

And most importantly, comparisons between mining methods are not meaningful unless pool depth is controlled. A top-80 miner and a top-2,048 miner are not being tested under the same conditions.

Once filtering is geometry-aware, mining depth becomes part of the training recipe.

Deeper mining solved the per-query shortage. It did not solve the imbalance across datasets.

The judge veto removed very different fractions from different sources, leaving the surviving SFT pool heavily skewed: MS MARCO accounted for 37.9% of all usable pairs, while FiQA contributed just 0.3%. Training on those raw proportions would mostly preserve the imbalance.

So I weighted each source using

$$w_s=\sqrt{{\text{usable}}_s}(1-{\text{FN risk}}_s)\mathrm{.}w\_s\; =\; \backslash sqrt\{\backslash text\{usable\}\_s\}\backslash ,(1-\backslash text\{FN\; risk\}\_s).$$

The two terms do different jobs.

The square root is standard $\alpha=0.5$ smoothing, following the same convention as the dense recipe. It gives diminishing returns to sheer dataset size: a source with four times as many surviving pairs gets twice the size weight rather than four times the weight.

The second term accounts for something count-based smoothing cannot see: not every surviving pair is equally trustworthy. A source with a high measured false-negative rate should contribute less, even if it has plenty of usable rows.

NQ and SQuAD v2 make the difference concrete. They contain almost the same number of usable pairs after filtering:

• NQ: 112,900
• SQuAD v2: 113,579

With $\alpha=0.5$ smoothing alone, they receive essentially identical targets, a ratio of 1.003.

But their measured false-negative rates are completely different. NQ is at 32.6%, while SQuAD v2 is only 2.8%. Treating those pools as equivalent would mean spending a large fraction of NQ's training budget teaching the model to push down documents that may actually answer the query.

Multiplying by (1 − FN risk) moves the ratio to 1.45, so SQuAD v2 ends with 44% more rows despite starting from almost the same usable pool.

The risk term is deliberately outside the square root:

$$\sqrt{\text{usable}}(1-\text{FN risk})\backslash sqrt\{\backslash text\{usable\}\}\backslash ,(1-\backslash text\{FN\; risk\})$$

rather than

$$\sqrt{\text{usable}(1-\text{FN risk})}\mathrm{.}\backslash sqrt\{\backslash text\{usable\}(1-\backslash text\{FN\; risk\})\}.$$

The square root should damp size, not quality. If contamination is placed inside the square root, NQ's 32.6% false-negative rate is softened along with dataset size. The NQ/SQuAD ratio drops back to about 1.21, and the quality correction stops doing much.

Dataset size has diminishing returns. A defect rate does not.

I also considered simply dropping the dirtiest sources. That threw away too much. NQ at 32.6% risk is still mostly clean, it represents its own retrieval distribution, and later experiments showed that breadth was one of the strongest levers in the whole project. Down-weighting kept the coverage while paying less for the contamination.

Both the smoothing exponent and risk calibration were fixed using FiQA alone. No BEIR evaluation group was used to tune the recipe.

MS MARCO is the only source that is actually downsampled. Everything else receives some degree of upsampling, but for two different reasons.

The square-root term pulls small sources upward. BiCA goes from just 1.0% of the surviving pool to 4.6% of the final SFT mixture, despite having only 12,069 usable pairs.

The risk term stops dirty sources from rising with them. NQ and SQuAD v2 begin with almost identical usable counts, yet SQuAD v2 receives 44% more rows because its measured false-negative rate is roughly twelve times lower.

Upsampling here is not literal row duplication. Each time a (query, positive) pair is emitted, I randomly resample 7 negatives from its surviving candidate band. FiQA's 8.71× therefore does not mean the model sees the exact same example nine times. It means the same query-positive pair appears against roughly nine different negative sets.

There was also a warning hidden in the table that I did not appreciate until later. BiCA started with 19,997 raw rows, but 7,928 were discarded because fewer than seven negatives survived the judge veto. Its candidate pools were naturally shallow.

At seven negatives per SFT row, that was survivable.

At 32 candidates per distillation row, it was not.

---

3. From SFT to knowledge distillation

The same compressed score geometry that broke positive-aware filtering appeared again when I moved from supervised fine-tuning to knowledge distillation.

PyLate's default distillation path min-max normalized each row of teacher scores before applying a softmax. With 32 candidates, the resulting teacher distribution had entropy 3.437, compared with 3.466 for a perfectly uniform distribution. The target was 99.2% of the way to uniform.

Adding a teacher temperature of τ = 0.3 to re-sharpen the target improved the model by +0.36 on the diagnostic suite.

There was a second compression point inside the student objective. The late-interaction score needs to be summed over query tokens. Averaging the token contributions compresses the score range again, so a softmax over 32 candidates at temperature 1 becomes nearly uniform.

The lesson from the mining section carried straight into distillation: the same numerical operation can behave very differently once the underlying score geometry changes.

I initially expected the reranker used as the distillation teacher to matter a lot. In practice, it barely moved the result.

Keeping the Jina teacher and the same objective, but expanding the distillation data from MS MARCO alone to seven sources, raised the score from 54.58 to 57.33, a gain of +2.75.

The seven-source mixture covered MS MARCO, Natural Questions, SQuAD v2, FEVER, HotpotQA, TriviaQA, and FiQA. The teacher did not change. The objective did not change. What changed was the range of hard ranking problems the student saw.

That was roughly a twenty-to-one difference in effect size: teacher choice was almost a wash, while data breadth accounted for nearly the entire improvement.

The distribution you distill over can matter far more than the model producing the scores.

BiCA made the boundary of that conclusion much clearer.

I first added the biomedical citation dataset as an eighth SFT source. BiCA naturally supplied enough judged negatives for the seven-negative SFT rows, so the data could be used without inventing extra candidates. Once the seven-source KD stage was applied on top, the effect was roughly neutral: +0.10 on the full BEIR mean. It did not add much beyond the breadth already provided by the KD mixture, but it also did not damage the model. The final GLInt checkpoint uses this BiCA-augmented SFT checkpoint.

Adding BiCA as an eighth knowledge-distillation source was a completely different result. The diagnostic mean fell from 55.46 to 54.03, a loss of −1.43, with seven of eight datasets getting worse. The largest drops were on TREC-COVID, NFCorpus, and SciFact, the scientific and biomedical datasets BiCA was supposed to help.

The problem was not biomedical data. It was the fixed 32-way KD pool.

BiCA supplied only about 7.5 real citation negatives per query, but the listwise distillation format required 32 candidates. The remaining slots were padded with random documents from the corpus. In total, 489,936 of 639,904 candidate slots, or 77%, were random padding.

Under a sharpened listwise objective, those random documents were not harmless filler. They became part of the ranking target, teaching the student to separate trivially easy negatives instead of resolving meaningful ambiguities between hard candidates.

The SFT result showed that BiCA itself was usable. The failure came from forcing a naturally shallow source into a 32-way ranking format and pretending that 7.5 hard negatives plus 24.5 random documents formed a useful listwise task.

The useful unit of data breadth is not the number of sources. It is the number of hard ranking problems they contribute.

BiCA stayed in the SFT path used by GLInt, but it was excluded from the final seven-source KD mixture.

---

4. Final results

The final model is therefore not the result of one mining trick. It is the endpoint of a pipeline in which the mining geometry, false-negative filtering, candidate depth, mixture construction, and distillation setup were all adjusted around late-interaction scoring.

BEIR-15. The table below reports mean nDCG@10 over the standard 15 BEIR groups, with CQADupStack averaged across its subforums.

GLInt reaches 57.43 mean nDCG@10, compared with 57.22 for LateOn. It improves on LateOn on ArguAna, DBPedia, FEVER, HotpotQA, MS MARCO, Natural Questions, Quora, SciFact, and TREC-COVID, as well as on the overall average.

BEIR-Decontaminated. Following LateOn, I also evaluated GLInt on BEIR-Decontaminated, which removes known contamination and provides a second view of generalization across 14 BEIR groups.

GLInt reaches an average of 62.50 on this evaluation, compared with 61.4 for LateOn in the same table.

---

5. Mining at scale

Everything above assumes that full multi-vector mining is computationally practical. The engineering question was therefore how to retrieve deep MaxSim pools without turning data construction into a multi-day bottleneck.

I compared exact brute-force MaxSim with PLAID through PyLate's Rust fast-plaid implementation and WARP. The benchmark used 250,000 documents and exact MaxSim as the reference.

At first glance, WARP's recall looks poor: it misses more than one-third of the exact top 50. For hard-negative mining, however, set overlap was not the metric I ultimately cared about.

Because the MaxSim candidate distribution is so compressed, ranks 50 and 200 can be near-ties. Recall@50 heavily penalizes swapping one near-tied document for another even when both are equally useful as training negatives. The candidates WARP returned in the rank band that actually entered training had nearly the same MaxSim score as the exact candidates.

That made WARP roughly four times faster than fast-plaid: about four hours of wall-clock time instead of sixteen for the full job.

The score comparison used each backend's self-reported scores, which are not perfectly comparable across configurations. A rigorous exact comparison would take the document IDs returned by every backend and rescore all of them with exact MaxSim. I did not run that exhaustive rescore, so I treat the score-equivalence result as an engineering signal rather than a precision benchmark.

For mining, score-equivalent near-ties mattered more than exact top-50 ID overlap.

The practical split was:

• WARP for mining, where near-tie substitutions are acceptable and throughput dominates.
• fast-plaid for evaluation, where exact top-rank identity matters for nDCG and where I wanted consistency with the evaluation setup used for other late-interaction models.

---

Appendix: Experiments that did not work

After the final model converged, I tried a series of post-training and continuation ideas intended to extract another improvement. None of them worked overall.

The failures fell into three useful categories.

Adding query content changed what MaxSim was scoring. Inference-time [MASK] expansion, trained expansion vectors, LLM-generated evidence, and corpus-grounded feedback all added vectors to the query. This family is related in spirit to query and pseudo-document expansion methods such as HyDE, but under a sum-based MaxSim scorer the added vectors are not harmless context: they receive their own best document-token matches and alter the score budget.

The retriever began answering the expansion rather than the original query. Short-query datasets such as TREC-COVID, Quora, and FiQA were hit hardest because the added vectors made up a larger fraction of the final query representation.

Reweighting a converged scorer offered little headroom. Static IDF weighting, a learned token gate, exact candidate rescoring, and model averaging produced small or negative changes. The checkpoint had already converged under its native scorer. Reweighting that geometry after the fact mostly disturbed a solution that was already internally consistent.

The model-soup experiment followed the idea of averaging checkpoints from Model Soups, but in this case the uniform soup with the Nomic continuation checkpoint was still −0.09.

Exact rescoring deserves a specific clarification: the experiment rescored PLAID's existing top 100. It could change their order but could not recover an exact-MaxSim document that PLAID never returned. The negative result rejects the cheap reranking trick, not exact exhaustive retrieval.

Self-derived targets can collapse without an external anchor. The optimal-transport experiment used the model's own token-similarity matrix to create a detached transport target. The goal was to encourage query tokens to distribute evidence across the document rather than collapse onto a few strong matches.

The loss decreased smoothly, but retrieval collapsed to a mean of 0.25.

Detaching the target prevented gradients from flowing through it, but it did not prevent the target from changing with the student. Once the model's scores became uniform, its transport targets also became uniform, creating a self-consistent degenerate solution. Self-distillation methods such as DINO and SwAV use additional mechanisms to avoid this kind of collapse. My objective had no gold-positive, frozen teacher, reference model, centering mechanism, or cross-view anchor, so it could minimize itself while learning to retrieve nothing.

---

References and resources

Models and datasets

• LateOn and LateOn-unsupervised: the supervised reference and unsupervised starting checkpoint.
• MS MARCO BGE-Gemma scores: the native knowledge-distillation dataset and teacher scores.
• jina-reranker-v3.5: the alternative listwise teacher and independent post-hoc auditor used where described.
• GTE-ModernColBERT-v1: the late-interaction judge used to construct the filtered SFT data.
• BiCA: the biomedical citation data source.
• LightOn-embeddings-fine-tune: the main seven-source supervised fine-tuning data.
• PyLate: the late-interaction training and indexing library used throughout.

Methods and references

• Khattab and Zaharia, Efficient and Effective Passage Search via Contextualized Late Interaction over BERT: ColBERT and MaxSim late interaction.
• Thakur et al., BEIR benchmark and paper.
• Moreira et al., NV-Retriever: positive-aware hard-negative mining.
• Santhanam et al., PLAID: An Efficient Engine for Late Interaction Retrieval.
• Scheerer et al., WARP: An Efficient Engine for Multi-Vector Retrieval, SIGIR 2025.
• Athrael, Mine the Way Your Model Scores: MaxSim Hard-Negative Mining for a Late-Interaction Student.
• Gao et al., HyDE: hypothetical document embeddings.
• Wortsman et al., Model Soups.
• DINO and SwAV: relevant anti-collapse precedents for self-distillation.

Huge thanks to the amazing people at LightOn whose numbers and evaluations I can blindly trust, making my life a tad bit easier.

¹ Most ablations in this post use an eight-dataset diagnostic suite consisting of SCIDOCS, SciFact, FiQA, TREC-COVID, ArguAna, Touché-2020, Quora, and NFCorpus. These datasets are substantially smaller and faster to evaluate than the full BEIR suite, which made them practical for rapid iteration across many experimental arms. Unless otherwise stated, ablation results refer to this diagnostic suite; final model comparisons are reported on all 15 BEIR groups.