Duplicate from Voodisss/Qwen3-Reranker-4B-GGUF-llama_cpp

Co-authored-by: VooDisss <Voodisss@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,44 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+Qwen3-Reranker-4B-f16.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Reranker-4B.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Reranker-4B-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Reranker-4B-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Reranker-4B-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Reranker-4B-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Reranker-4B-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Reranker-4B-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Qwen3-Reranker-4B-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text

Qwen3-Reranker-4B-Q2_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:351cb7e1be302c7a94d3e055f3d0a139467d38ed204f64eb4384994e74ad0780
+size 1668935136

Qwen3-Reranker-4B-Q3_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2bc1029151b819db3958e54f15df17b6ad8ee48b70e9a8201491cd48f80e9759
+size 2075054048

Qwen3-Reranker-4B-Q4_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9fd3d5669d2caf0763dd060f320c688f5ed832abdba7f66dede6bd7f8623c473
+size 2368983680

Qwen3-Reranker-4B-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a693b22cc8122353562c2d261a3894301c14c6eecce7304bc29968f8994258b
+size 2496717440

Qwen3-Reranker-4B-Q5_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a840078d2b83d816c4bab931ae14eca0a318efd6f63621c7388a6e3503ac2152
+size 2823148800

Qwen3-Reranker-4B-Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13950913d05a320df83190fa2367872faccbe93e7beee2cc2ca2b9f345e8847c
+size 2888951040

Qwen3-Reranker-4B-Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e54c8d0d8002f852048a8b46b40c54cd5349cdc0b0128ea11ff1df974105dd88
+size 3305699264

Qwen3-Reranker-4B-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4de2e3e4179d5bca95a2e960e07d225a565018e3bbb5e073f1777809091f117
+size 8049922912

Qwen3-Reranker-4B.Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:102ac400d68f02877c2fdfecf0732872652298448352b6faf28ec7f8dca8c913
+size 4279678912

README.md

@@ -0,0 +1,104 @@
+---
+base_model: Qwen/Qwen3-Reranker-4B
+library_name: gguf
+license: apache-2.0
+pipeline_tag: text-ranking
+tags:
+  - reranker
+  - gguf
+  - llama.cpp
+  - qwen3
+  - text-ranking
+---
+
+# Qwen3-Reranker-4B — GGUF (llama.cpp)
+
+Working GGUF of [Qwen/Qwen3-Reranker-4B](https://huggingface.co/Qwen/Qwen3-Reranker-4B) for [llama.cpp](https://github.com/ggml-org/llama.cpp). Converted 2025-03-09 with the official `convert_hf_to_gguf.py`.
+
+> **Other sizes:** [0.6B](https://huggingface.co/Voodisss/Qwen3-Reranker-0.6B-GGUF-llama_cpp) · [4B (this)](https://huggingface.co/Voodisss/Qwen3-Reranker-4B-GGUF-llama_cpp) · [8B](https://huggingface.co/Voodisss/Qwen3-Reranker-8B-GGUF-llama_cpp)
+
+## Quantization quality comparison (Qwen3-Reranker-4B)
+
+Benchmarked on [MTEB AskUbuntuDupQuestions](https://huggingface.co/datasets/mteb/AskUbuntuDupQuestions) (361 queries) via llama-server `/v1/rerank` on RTX 3090. All quants produced from the same F16 source using `llama-quantize`.
+
+| Quant  | Size    | NDCG@10 | MAP@10 | MRR@10 | Δ NDCG@10 |
+| ------ | ------- | ------- | ------ | ------ | --------- |
+| F16    | 7.50 GB | 0.7003  | 0.5530 | 0.7711 | baseline  |
+| Q8_0   | 3.99 GB | 0.6985  | 0.5514 | 0.7670 | -0.3%     |
+| Q6_K   | 3.08 GB | 0.7016  | 0.5548 | 0.7722 | +0.2%     |
+| Q5_K_M | 2.69 GB | 0.7009  | 0.5517 | 0.7699 | +0.1%     |
+| Q5_0   | 2.63 GB | 0.6995  | 0.5532 | 0.7676 | -0.1%     |
+| Q4_K_M | 2.33 GB | 0.7058  | 0.5596 | 0.7746 | +0.8%     |
+| Q4_0   | 2.21 GB | 0.6930  | 0.5426 | 0.7623 | -1.1%     |
+| Q3_K_M | 1.93 GB | 0.7040  | 0.5555 | 0.7828 | +0.5%     |
+| Q2_K   | 1.55 GB | 0.6691  | 0.5079 | 0.7401 | **-4.5%**     |
+
+**Takeaway:** All quants from Q8_0 down to Q3_K_M are within ±1% of F16 — pick based on your VRAM budget. Q4_K_M (2.33 GB) is the sweet spot: 3.2x smaller than F16 with no measurable quality loss. **Avoid Q2_K** — it's the only quant with real degradation.
+## Does it work?
+
+Yes. Most community GGUFs of Qwen3-Reranker produce garbage scores (`4.5e-23`) because they're missing reranker-specific tensors. See [llama.cpp #16407](https://github.com/ggml-org/llama.cpp/issues/16407). This one works:
+
+```
+Doc 0 (relevant):   relevance_score = 0.999966
+Doc 1 (irrelevant): relevance_score = 0.000069
+```
+
+## Quick start
+
+```bash
+llama-server -m Qwen3-Reranker-4B-f16.gguf --reranking --pooling rank --embedding --port 8081
+```
+
+```bash
+curl http://localhost:8081/v1/rerank \
+  -H "Content-Type: application/json" \
+  -d '{
+    "query": "employment termination notice period",
+    "documents": [
+      "The Labour Code requires 30 calendar days written notice.",
+      "Corporate tax rates for small enterprises."
+    ]
+  }'
+```
+
+Use **`/v1/rerank`**, not `/v1/embeddings`. The embeddings endpoint returns zeros for reranker models.
+
+## What's different about this GGUF?
+
+The official `convert_hf_to_gguf.py` detects Qwen3-Reranker and does things naive converters skip:
+
+- Extracts `cls.output.weight` (the yes/no classifier) from `lm_head`
+- Sets `pooling_type = RANK` metadata
+- Bakes in the rerank chat template
+- Sets `classifier.output_labels = ["yes", "no"]`
+
+Without these, llama-server has nothing to compute scores from.
+
+## Known broken GGUFs
+
+- [DevQuasar/Qwen.Qwen3-Reranker-4B-GGUF](https://huggingface.co/DevQuasar/Qwen.Qwen3-Reranker-4B-GGUF) — confirmed broken with llama.cpp
+
+## models.ini example
+
+```ini
+[Qwen3-Reranker-4B-f16]
+model = /path/to/Qwen3-Reranker-4B-f16.gguf
+reranking = true
+pooling = rank
+embedding = true
+ctx-size = 32768
+```
+
+For a full multi-model setup guide (embedding + reranking + chat on one server), see the **[llama-server Qwen3 guide](https://gist.github.com/VooDisss/42bce4eb5c76d3c325633886c5e348ee)**.
+
+## Convert it yourself
+
+```bash
+pip install huggingface_hub gguf torch safetensors sentencepiece
+python -c "from huggingface_hub import snapshot_download; snapshot_download('Qwen/Qwen3-Reranker-4B', local_dir='Qwen3-Reranker-4B-src')"
+python convert_hf_to_gguf.py --outtype f16 --outfile Qwen3-Reranker-4B-f16.gguf Qwen3-Reranker-4B-src/
+```
+
+## License
+
+Apache 2.0 — same as the [original model](https://huggingface.co/Qwen/Qwen3-Reranker-4B).