Upload folder using huggingface_hub

313c369 verified about 2 months ago

8.25 kB

---
license: apache-2.0
language:
- en
base_model:
- zeroentropy/zerank-2-reranker
base_model_relation: quantized
pipeline_tag: text-ranking
library_name: mlx
tags:
- mlx
- reranker
- cross-encoder
- apple-silicon
- finance
- legal
- code
- stem
- medical
---
# zerank-2-reranker — MLX (q8)

[`zeroentropy/zerank-2-reranker`](https://huggingface.co/zeroentropy/zerank-2-reranker)
converted to MLX for Apple silicon. Both the transformer body and the tied token embedding/readout matrix use **8-bit** quantization.

On-disk size: **4.3 GB**. For the highest-fidelity MLX conversion, use
[`fcmeyer/zerank-2-reranker-MLX-bf16`](https://huggingface.co/fcmeyer/zerank-2-reranker-MLX-bf16).

zerank-2 is a cross-encoder reranker built on Qwen3-4B. It reads a query and one
document together and returns a single relevance score, which is more accurate than
comparing independent embeddings but requires one forward pass per candidate
document.

## Usage

Download [`rerank.py`](./rerank.py) from this repo, then:

```bash
pip install mlx-lm transformers
```

```python
from rerank import Reranker, probability
reranker = Reranker("fcmeyer/zerank-2-reranker-MLX-8bit")
ranked = reranker.rank(
    "What causes the northern lights?",
    [
        "Auroras occur when charged particles from the solar wind are funnelled "
        "along Earth's magnetic field lines into the upper atmosphere.",
        "Iceland is a popular destination for tourists hoping to photograph them.",
        "The Earth's magnetic field reverses polarity every few hundred thousand years.",
    ],
)
for document, score in ranked:
    print(f"{score:+.3f}  p={probability(score):.3f}  {document[:60]}")
```

`rank()` returns documents best-first. `score()` takes explicit
`(query, document)` pairs and preserves your input order. Both batch internally.

## How scoring works

The model does not have a classification head. A pair is rendered through the
chat template as `system` = query, `user` = document, followed by an assistant
generation prompt; the score is the logit of token id **9454** (`"Yes"`) at the
final position. That token id comes from `1_LogitScore/config.json` in this repo.

The raw logit is the score, and it is what you rank by. `sigmoid(score / 5)`
maps it to the 0–1 range shown on the upstream model card. Scores are comparable
within a single query, not across queries.

## Verification

Every score below is a "Yes" logit measured on the same **486
query/document pairs** used to verify the unquantized conversion. PyTorch float32
is the reference; **MLX bf16** is the verified lossless conversion, and **MLX
q8** is this lossy build.

| vs. PyTorch float32 | MLX bf16 | **MLX q8** |
|---|---:|---:|
| mean abs. score delta | 0.0548 | **0.0723** |
| p99 abs. score delta | 0.1848 | **0.2119** |
| max abs. score delta | 0.2172 | **0.2736** |
| Spearman correlation | 0.999654 | **0.999494** |
| mean per-query Kendall τ | 0.978934 | **0.977836** |
| ranking inversions | 24 | **29** |
| resolvable ranking inversion rate | 0 / 4719 (0.0000%) | **0 / 4719 (0.0000%)** |
| relevance verdict flips | 0 | **0** |

Per-query Kendall τ is computed over each query's own candidate documents. An
inversion is resolvable only when the two documents' float32 scores differ by more
than 0.1930, the measured PyTorch bfloat16 p99
error. The rate therefore uses all 4719 same-query document pairs
that float32 resolved as its denominator. A relevance verdict flip crosses the
`sigmoid(score/5) = 0.5` boundary relative to float32.

### Independent BEIR verification

The independent fixture has **1449 pairs** from 210 complete
query pools sampled from FiQA, SciFact, and NFCorpus.

| vs. PyTorch float32 (BEIR sample) | MLX bf16 | **MLX q8** |
|---|---:|---:|
| mean abs. score delta | 0.0508 | **0.0807** |
| p99 abs. score delta | 0.1727 | **0.2861** |
| max abs. score delta | 0.2904 | **0.4682** |
| Spearman correlation | 0.999930 | **0.999875** |
| mean per-query Kendall τ | 0.988963 | **0.986915** |
| ranking inversions | 10 | **13** |
| resolvable ranking inversion rate | 0 / 4168 (0.0000%) | **1 / 4168 (0.0240%)** |
| resolvable relevance verdict flips | 0 | **0** |

This build has **1 resolvable ranking inversion
among 4168 float32-resolved same-query document pairs** on BEIR. It
does not meet the zero-resolvable-inversion bar that the MLX bf16 build met. It
has **0 resolvable relevance verdict flips**.

### NDCG@10 sanity check

| implementation | NDCG@10 against BEIR qrels |
|---|---:|
| PyTorch float32 | 0.954794 |
| MLX bf16 | 0.955648 |
| **MLX q8** | **0.955104** |

These are complete pools of only 6–8 documents, and most queries have a single
positive. NDCG@10 is therefore a sanity check, not the sharp metric for the
quantization damage; per-query Kendall τ and resolvable inversions see reorderings
among all documents.

### Quantization ablation on BEIR

| build | on-disk size | body / embedding bits | mean abs. delta | p99 abs. delta | resolvable inversion rate | resolvable verdict flips | mean per-query Kendall τ | NDCG@10 | NDCG@10 − float32 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| q4 | 2.28 GB | 4 / 4 | 1.5064 | 3.2907 | 127 / 4168 (3.0470%) | 13 | 0.902443 | 0.957950 | +0.003155 |
| q4-e8 | 2.47 GB | 4 / 8 | 1.3038 | 2.9973 | 123 / 4168 (2.9511%) | 10 | 0.903406 | 0.957289 | +0.002495 |
| q4-e8-mixed46 | 2.60 GB | 4 / 8 | 1.2570 | 2.8377 | 99 / 4168 (2.3752%) | 9 | 0.917504 | 0.957501 | +0.002706 |
| q6 | 3.28 GB | 6 / 6 | 0.1451 | 0.5718 | 9 / 4168 (0.2159%) | 2 | 0.969801 | 0.954977 | +0.000183 |
| q8 | 4.29 GB | 8 / 8 | 0.0807 | 0.2861 | 1 / 4168 (0.0240%) | 0 | 0.986915 | 0.955104 | +0.000310 |

The tied embedding is also the readout matrix: the score is
`dot(hidden_state, embed_tokens[9454])`. The 4-bit experiment therefore tested
whether holding that matrix at 8 bits would protect the score. It was directionally
right and practically useless. Relative to q4, q4-e8 removes
**23%** of the systematic bias (+1.319 to
+1.015 logits) and **13%** of mean
absolute error, but residual noise is essentially unchanged
(0.586 to 0.583) and mean
per-query Kendall τ is statistically indistinguishable
(0.9024 to
0.9034).
Protecting the readout row protects the ruler; 4-bit quantization also corrupts
the hidden state being measured, and that is what reorders documents.

The decomposition below is computed directly from the BEIR score vectors. Bias
is the mean signed error. Noise is the mean absolute residual after a least-squares
linear fit of each build's scores against float32.

| build | mean abs. error | mean signed error (bias) | mean abs. fitted residual (noise) | negative→positive flips | positive→negative flips |
|---|---:|---:|---:|---:|---:|
| q4 | 1.506 | +1.319 | 0.586 | 10 | 3 |
| q4-e8 | 1.304 | +1.015 | 0.583 | 6 | 4 |
| q4-e8-mixed46 | 1.257 | +1.007 | 0.501 | 7 | 2 |
| q6 | 0.145 | -0.007 | 0.145 | 0 | 2 |
| q8 | 0.081 | +0.031 | 0.074 | 0 | 0 |

The verdict directions corroborate the distinction: q4 produces
10 negative-to-positive flips against
3 the other way, the expected signature of its positive
logit bias at the 0.5 boundary. q6 has only 0 and
2 respectively, so its two flips run the other way and
are noise rather than systematic positive drift.

## Differences from the source repo

Only the weight format. This repo carries the full upstream file set, including
`1_LogitScore/config.json`, `modules.json`, `sentence_bert_config.json`,
`config_sentence_transformers.json`, and the BPE `vocab.json` / `merges.txt` —
`mlx_lm.convert` copies none of those on its own, and without
`1_LogitScore/config.json` there is no record in the repo of which token id the
score comes from.

## Reproducing

```bash
pip install mlx-lm
mlx_lm.convert --hf-path zeroentropy/zerank-2-reranker \
    --mlx-path zerank-2-mlx-q8 --quantize --q-bits 8 \
    --q-group-size 64
```

then copy the sentence-transformers metadata listed above across from the source repo.

## License and attribution

Apache 2.0, inherited from the source model. All credit for the model itself goes
to [ZeroEntropy](https://www.zeroentropy.dev); see their
[technical report](https://arxiv.org/abs/2509.12541). This repo is a format
conversion and adds no training.

	---
	license: apache-2.0
	language:
	- en
	base_model:
	- zeroentropy/zerank-2-reranker
	base_model_relation: quantized
	pipeline_tag: text-ranking
	library_name: mlx
	tags:
	- mlx
	- reranker
	- cross-encoder
	- apple-silicon
	- finance
	- legal
	- code
	- stem
	- medical
	---

	# zerank-2-reranker — MLX (q8)

	[`zeroentropy/zerank-2-reranker`](https://huggingface.co/zeroentropy/zerank-2-reranker)
	converted to MLX for Apple silicon. Both the transformer body and the tied token embedding/readout matrix use 8-bit quantization.

	On-disk size: 4.3 GB. For the highest-fidelity MLX conversion, use
	[`fcmeyer/zerank-2-reranker-MLX-bf16`](https://huggingface.co/fcmeyer/zerank-2-reranker-MLX-bf16).

	zerank-2 is a cross-encoder reranker built on Qwen3-4B. It reads a query and one
	document together and returns a single relevance score, which is more accurate than
	comparing independent embeddings but requires one forward pass per candidate
	document.

	## Usage

	Download [`rerank.py`](./rerank.py) from this repo, then:

	```bash
	pip install mlx-lm transformers
	```

	```python
	from rerank import Reranker, probability

	reranker = Reranker("fcmeyer/zerank-2-reranker-MLX-8bit")

	ranked = reranker.rank(
	"What causes the northern lights?",
	[
	"Auroras occur when charged particles from the solar wind are funnelled "
	"along Earth's magnetic field lines into the upper atmosphere.",
	"Iceland is a popular destination for tourists hoping to photograph them.",
	"The Earth's magnetic field reverses polarity every few hundred thousand years.",
	],
	)

	for document, score in ranked:
	print(f"{score:+.3f} p={probability(score):.3f} {document[:60]}")
	```

	`rank()` returns documents best-first. `score()` takes explicit
	`(query, document)` pairs and preserves your input order. Both batch internally.

	## How scoring works

	The model does not have a classification head. A pair is rendered through the
	chat template as `system` = query, `user` = document, followed by an assistant
	generation prompt; the score is the logit of token id 9454 (`"Yes"`) at the
	final position. That token id comes from `1_LogitScore/config.json` in this repo.

	The raw logit is the score, and it is what you rank by. `sigmoid(score / 5)`
	maps it to the 0–1 range shown on the upstream model card. Scores are comparable
	within a single query, not across queries.

	## Verification

	Every score below is a "Yes" logit measured on the same **486
	query/document pairs** used to verify the unquantized conversion. PyTorch float32
	is the reference; MLX bf16 is the verified lossless conversion, and **MLX
	q8** is this lossy build.

	\| vs. PyTorch float32 \| MLX bf16 \| MLX q8 \|
	\|---\|---:\|---:\|
	\| mean abs. score delta \| 0.0548 \| 0.0723 \|
	\| p99 abs. score delta \| 0.1848 \| 0.2119 \|
	\| max abs. score delta \| 0.2172 \| 0.2736 \|
	\| Spearman correlation \| 0.999654 \| 0.999494 \|
	\| mean per-query Kendall τ \| 0.978934 \| 0.977836 \|
	\| ranking inversions \| 24 \| 29 \|
	\| resolvable ranking inversion rate \| 0 / 4719 (0.0000%) \| 0 / 4719 (0.0000%) \|
	\| relevance verdict flips \| 0 \| 0 \|

	Per-query Kendall τ is computed over each query's own candidate documents. An
	inversion is resolvable only when the two documents' float32 scores differ by more
	than 0.1930, the measured PyTorch bfloat16 p99
	error. The rate therefore uses all 4719 same-query document pairs
	that float32 resolved as its denominator. A relevance verdict flip crosses the
	`sigmoid(score/5) = 0.5` boundary relative to float32.

	### Independent BEIR verification

	The independent fixture has 1449 pairs from 210 complete
	query pools sampled from FiQA, SciFact, and NFCorpus.

	\| vs. PyTorch float32 (BEIR sample) \| MLX bf16 \| MLX q8 \|
	\|---\|---:\|---:\|
	\| mean abs. score delta \| 0.0508 \| 0.0807 \|
	\| p99 abs. score delta \| 0.1727 \| 0.2861 \|
	\| max abs. score delta \| 0.2904 \| 0.4682 \|
	\| Spearman correlation \| 0.999930 \| 0.999875 \|
	\| mean per-query Kendall τ \| 0.988963 \| 0.986915 \|
	\| ranking inversions \| 10 \| 13 \|
	\| resolvable ranking inversion rate \| 0 / 4168 (0.0000%) \| 1 / 4168 (0.0240%) \|
	\| resolvable relevance verdict flips \| 0 \| 0 \|

	This build has **1 resolvable ranking inversion
	among 4168 float32-resolved same-query document pairs** on BEIR. It
	does not meet the zero-resolvable-inversion bar that the MLX bf16 build met. It
	has 0 resolvable relevance verdict flips.

	### NDCG@10 sanity check

	\| implementation \| NDCG@10 against BEIR qrels \|
	\|---\|---:\|
	\| PyTorch float32 \| 0.954794 \|
	\| MLX bf16 \| 0.955648 \|
	\| MLX q8 \| 0.955104 \|

	These are complete pools of only 6–8 documents, and most queries have a single
	positive. NDCG@10 is therefore a sanity check, not the sharp metric for the
	quantization damage; per-query Kendall τ and resolvable inversions see reorderings
	among all documents.

	### Quantization ablation on BEIR

	\| build \| on-disk size \| body / embedding bits \| mean abs. delta \| p99 abs. delta \| resolvable inversion rate \| resolvable verdict flips \| mean per-query Kendall τ \| NDCG@10 \| NDCG@10 − float32 \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| q4 \| 2.28 GB \| 4 / 4 \| 1.5064 \| 3.2907 \| 127 / 4168 (3.0470%) \| 13 \| 0.902443 \| 0.957950 \| +0.003155 \|
	\| q4-e8 \| 2.47 GB \| 4 / 8 \| 1.3038 \| 2.9973 \| 123 / 4168 (2.9511%) \| 10 \| 0.903406 \| 0.957289 \| +0.002495 \|
	\| q4-e8-mixed46 \| 2.60 GB \| 4 / 8 \| 1.2570 \| 2.8377 \| 99 / 4168 (2.3752%) \| 9 \| 0.917504 \| 0.957501 \| +0.002706 \|
	\| q6 \| 3.28 GB \| 6 / 6 \| 0.1451 \| 0.5718 \| 9 / 4168 (0.2159%) \| 2 \| 0.969801 \| 0.954977 \| +0.000183 \|
	\| q8 \| 4.29 GB \| 8 / 8 \| 0.0807 \| 0.2861 \| 1 / 4168 (0.0240%) \| 0 \| 0.986915 \| 0.955104 \| +0.000310 \|

	The tied embedding is also the readout matrix: the score is
	`dot(hidden_state, embed_tokens[9454])`. The 4-bit experiment therefore tested
	whether holding that matrix at 8 bits would protect the score. It was directionally
	right and practically useless. Relative to q4, q4-e8 removes
	23% of the systematic bias (+1.319 to
	+1.015 logits) and 13% of mean
	absolute error, but residual noise is essentially unchanged
	(0.586 to 0.583) and mean
	per-query Kendall τ is statistically indistinguishable
	(0.9024 to
	0.9034).
	Protecting the readout row protects the ruler; 4-bit quantization also corrupts
	the hidden state being measured, and that is what reorders documents.

	The decomposition below is computed directly from the BEIR score vectors. Bias
	is the mean signed error. Noise is the mean absolute residual after a least-squares
	linear fit of each build's scores against float32.

	\| build \| mean abs. error \| mean signed error (bias) \| mean abs. fitted residual (noise) \| negative→positive flips \| positive→negative flips \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| q4 \| 1.506 \| +1.319 \| 0.586 \| 10 \| 3 \|
	\| q4-e8 \| 1.304 \| +1.015 \| 0.583 \| 6 \| 4 \|
	\| q4-e8-mixed46 \| 1.257 \| +1.007 \| 0.501 \| 7 \| 2 \|
	\| q6 \| 0.145 \| -0.007 \| 0.145 \| 0 \| 2 \|
	\| q8 \| 0.081 \| +0.031 \| 0.074 \| 0 \| 0 \|

	The verdict directions corroborate the distinction: q4 produces
	10 negative-to-positive flips against
	3 the other way, the expected signature of its positive
	logit bias at the 0.5 boundary. q6 has only 0 and
	2 respectively, so its two flips run the other way and
	are noise rather than systematic positive drift.

	## Differences from the source repo

	Only the weight format. This repo carries the full upstream file set, including
	`1_LogitScore/config.json`, `modules.json`, `sentence_bert_config.json`,
	`config_sentence_transformers.json`, and the BPE `vocab.json` / `merges.txt` —
	`mlx_lm.convert` copies none of those on its own, and without
	`1_LogitScore/config.json` there is no record in the repo of which token id the
	score comes from.

	## Reproducing

	```bash
	pip install mlx-lm
	mlx_lm.convert --hf-path zeroentropy/zerank-2-reranker \
	--mlx-path zerank-2-mlx-q8 --quantize --q-bits 8 \
	--q-group-size 64
	```

	then copy the sentence-transformers metadata listed above across from the source repo.

	## License and attribution

	Apache 2.0, inherited from the source model. All credit for the model itself goes
	to [ZeroEntropy](https://www.zeroentropy.dev); see their
	[technical report](https://arxiv.org/abs/2509.12541). This repo is a format
	conversion and adds no training.