zerank-2-reranker-FP8

本模型是 zeroentropy/zerank-2-reranker 的 FP8 量化版本，使用 llm-compressor 工具进行校准与导出。

模型支持直接通过 vLLM 进行高并发在线服务，或通过 Transformers (compressed-tensors) 在本地直接加载推理。

---

🚀 量化详情

• 基础模型: zeroentropy/zerank-2-reranker
• 量化框架: llm-compressor (oneshot PTQ)
• 量化格式: FP8 (Compressed-Tensors)
• 保护层: lm_head 与 model.embed_tokens 保持原始精度未被量化
• 校准配置: 512 条中文短文本对进行数据校准

🛠️ 快速上手

方案 1：使用 vLLM 启动高并发推理服务（推荐）

由于本模型属于 CausalLM 架构并以输出 "Yes" 的 Logit 作为打分，在启动 vLLM 时需要传入 --chat-template，--convert classify 与 --hf-overrides：

vllm serve liudev/zerank-2-reranker-fp8 \
  --dtype bfloat16 \
  --served-model-name zerank-2-reranker-fp8 \
  --host 0.0.0.0 \
  --port 8000 \
  --runner pooling \
  --convert classify \
  --hf-overrides '{"method":"no_post_processing","classifier_from_token":["Yes"],"num_labels":1,"use_sep_token":false}' \
  --chat-template $(python3 -c "from huggingface_hub import hf_hub_download; print(hf_hub_download('liudev/zerank-2-reranker-fp8', 'chat_template.jinja'))") \
  --default-chat-template-kwargs '{"add_generation_prompt":true}' \
  --quantization compressed-tensors \
  --max-model-len 1024 \
  --gpu-memory-utilization 0.5 \
  --max-num-seqs 128

# --chat-template 指定chat_template.jinja文件所在路径
# 例如：/root/.cache/huggingface/hub/models--liudev--zerank-2-reranker-fp8/snapshots/2060ec28526cc9f70f9310b33477c99e09aaca34/chat_template.jinja

客户端异步调用示例 (/v1/rerank)

服务启动后，可直接使用标准兼容接口发起重排：

import asyncio
import aiohttp

URL = "http://127.0.0.1:8000/v1/rerank"

async def rerank_demo():
    payload = {
        "model": "zerank-2-reranker-fp8",
        "query": "什么是大模型量化？",
        "documents": [
            "大模型量化是一种将浮点权重转换为低精度表示的技术，旨在加速推理并降低显存。",
            "今天的天气晴空万里，非常适合户外运动。"
        ],
        "use_activation": False,
        "chat_template_kwargs": {"add_generation_prompt": True},
    }
    
    async with aiohttp.ClientSession() as session:
        async with session.post(URL, json=payload) as resp:
            data = await resp.json()
            for item in data["results"]:
                idx = item["index"]
                score = item["relevance_score"]
                print(f"Doc {idx} 得分: {score:.4f} | 内容: {payload['documents'][idx]}")

asyncio.run(rerank_demo())
# 第一次运行
# Doc 0 得分: 15.5660 | 内容: 大模型量化是一种将浮点权重转换低精度表示的技术，旨在加速推理并降低显存。
# Doc 1 得分: -16.6288 | 内容: 今天的天气晴空万里，非常适合户运动。

# 第二次运行
# Doc 0 得分: 15.5375 | 内容: 大模型量化是一种将浮点权重转换低精度表示的技术，旨在加速推理并降低显存。
# Doc 1 得分: -16.6288 | 内容: 今天的天气晴空万里，非常适合户运动。

# 第三次运行
# Doc 0 得分: 15.5375 | 内容: 大模型量化是一种将浮点权重转换低精度表示的技术，旨在加速推理并降低显存。
# Doc 1 得分: -16.6288 | 内容: 今天的天气晴空万里，非常适合户运动。

---

方案 2：Python 本地直接调用 (Transformers)

无需启动外部服务，在 Python 单进程中即可直接执行：

pip install torch transformers accelerate compressed-tensors

运行环境

pip show torch
python --version
nvcc --version

Name: torch
Version: 2.13.0
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org
Author: 
Author-email: PyTorch Team <packages@pytorch.org>
License: 
Location: /usr/local/lib/python3.13/dist-packages
Requires: cuda-bindings, cuda-toolkit, filelock, fsspec, jinja2, networkx, nvidia-cudnn-cu13, nvidia-cusparselt-cu13, nvidia-nccl-cu13, nvidia-nvshmem-cu13, setuptools, sympy, triton, typing-extensions
Required-by: accelerate, auto-round, compressed-tensors, fastai, flashinfer-python, humming-kernels, instanttensor, llmcompressor, peft, quack-kernels, sentence-transformers, tilelang, timm, tokenspeed-mla, torch_c_dlpack_ext, torchdata, torchvision, vllm, xgrammar
Python 3.13.15
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Aug_20_01:58:59_PM_PDT_2025
Cuda compilation tools, release 13.0, V13.0.88
Build cuda_13.0.r13.0/compiler.36424714_0

安装flash_attn

wget https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.47/flash_attn-2.8.3+cu130torch2.13-cp313-cp313-linux_x86_64.whl
uv pip install flash_attn-2.8.3+cu130torch2.13-cp313-cp313-linux_x86_64.whl --no-build-isolation

import flash_attn;
print('成功安装，版本号为:', flash_attn.__version__)
# 成功安装，版本号为: 2.8.3

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

class ZerankLocalReranker:
    def __init__(self, model_path="zeroentropy/zerank-2-reranker", device="cuda"):
        self.device = device
        print(f"正在加载原模型: {model_path} ...")
        self.tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
        self.model = AutoModelForCausalLM.from_pretrained(
            model_path,
            dtype=torch.bfloat16,
            device_map=device,
            attn_implementation="flash_attention_2",
            trust_remote_code=True
        )
        self.model.eval()
        # 9454 是 "Yes" 对应的 token_id
        self.yes_token_id = self.tokenizer.encode("Yes", add_special_tokens=False)

    @torch.inference_mode()
    def predict(self, pairs):
        if not pairs:
            return []

        prompts = [
            self.tokenizer.apply_chat_template(
                [{"role": "query", "content": q}, {"role": "document", "content": d}],
                tokenize=False,
                add_generation_prompt=True
            )
            for q, d in pairs
        ]
        
        # 1. 逐条编码，完全不启用 padding
        encoded = [
            self.tokenizer.encode(p, truncation=True, max_length=1024) 
            for p in prompts
        ]
        lengths = [len(seq) for seq in encoded]
        
        # 2. 扁平拼接为一条 1D 序列 (batch_size = 1)
        flat_input_ids = [token for seq in encoded for token in seq]
        input_ids = torch.tensor([flat_input_ids], dtype=torch.long, device=self.device)
        
        # 3. 构造各自重置的 position_ids (每条样本在自己的区间内都是 0, 1, 2... 递增)
        # FlashAttention-2 会依据 position_ids 中的重置突变点，自动识别样本边界并调用 varlen 算子切分
        position_ids = torch.cat([
            torch.arange(length, device=self.device) for length in lengths
        ]).unsqueeze(0)
        
        # 4. 注意：切勿传入 attention_mask (设为 None)
        outputs = self.model(
            input_ids=input_ids,
            position_ids=position_ids,
            attention_mask=None
        )
        
        # 5. 按照每条样本的实际长度，定位各自最后一个 token 的位置索引
        end_indices = torch.tensor(lengths, device=self.device).cumsum(dim=0) - 1
        
        # 提取各个样本对应 "Yes" 的 Logit，保持原有的输出维度 [[score1], [score2], ...]
        last_token_logits = outputs.logits[0, end_indices][:, self.yes_token_id]
        return last_token_logits.cpu().tolist()


reranker = ZerankLocalReranker('liudev/zerank-2-reranker-fp8')

query_documents = [
    ("What is 2+2?", "4"),
    ("What is 2+2?", "The answer is definitely 1 million"),
]

scores = reranker.predict(query_documents)
print(scores)
# [[6.375], [-4.0625]]

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# Convert to a 1D NumPy array, then back to a list if needed
scores = np.array(scores).flatten()

probabilities = sigmoid(scores / 5)
print(probabilities)
# [0.78159746 0.30735802]

query = "什么是大模型量化？"
documents = [
    "大模型量化是一种将浮点权重转换为低精度表示的技术，旨在加速推理并降低显存。",
    "今天的天气晴空万里，非常适合户外运动。"
]
query_documents = [(query, doc) for doc in documents]
scores = reranker.predict(query_documents)
print(scores)
# [[15.625], [-16.25]]

---

引用与感谢

• 原模型：zeroentropy/zerank-2-reranker
• 量化工具：vLLM Project / llm-compressor

---

Releasing zeroentropy/zerank-2

In search engines, rerankers are crucial for improving the accuracy of your retrieval system.

However, SOTA rerankers are closed-source and proprietary. At ZeroEntropy, we've trained a SOTA reranker outperforming closed-source competitors, and we're launching our model here on HuggingFace.

This reranker outperforms proprietary rerankers such as cohere-rerank-v3.5 and gemini-2.5-flash across a wide variety of domains, including finance, legal, code, STEM, medical, and conversational data.

At ZeroEntropy we've developed an innovative multi-stage pipeline that models query-document relevance scores as adjusted Elo ratings. See our Technical Report (https://arxiv.org/abs/2509.12541 ) for more details.

This model is released under the Apache License 2.0.

Model Details

Property	Value
Parameters	4B
Context Length	32,768 tokens (32k)
Base Model	Qwen/Qwen3-4B
License	Apache-2.0

How to Use

Breaking change (May 2026): model.predict() now returns raw "Yes" logits instead of sigmoid'd probabilities in [0, 1]. Rankings are unchanged. To recover the previous 0-1 score, apply (scores / 5).sigmoid() — see the example below. Loading no longer requires trust_remote_code=True; passing it is harmless.

Using Sentence Transformers

Install Sentence Transformers:

pip install sentence_transformers

Then load the model and score query/document pairs. model.predict returns the raw "Yes" logit per pair; rankings can be used directly. To map the logits to a 0-1 score range, apply a temperature-scaled sigmoid: sigmoid(score / 5).

from sentence_transformers import CrossEncoder

model = CrossEncoder("zeroentropy/zerank-2")

query_documents = [
    ("What is 2+2?", "4"),
    ("What is 2+2?", "The answer is definitely 1 million"),
]

scores = model.predict(query_documents, convert_to_tensor=True)
print(scores)
# tensor([ 5.4062, -4.5000], device='cuda:0', dtype=torch.bfloat16)

# Optional: convert to 0-1 probabilities
probabilities = (scores / 5).sigmoid()
print(probabilities)
# tensor([0.7461, 0.2891], device='cuda:0', dtype=torch.bfloat16)

You can also use model.rank to score and sort a list of documents for a single query:

rankings = model.rank(
    "What is 2+2?",
    ["4", "The answer is definitely 1 million"],
)
for r in rankings:
    print(r)
# {'corpus_id': 0, 'score': np.float32(5.40625)}
# {'corpus_id': 1, 'score': np.float32(-4.5)}

The model can also be inferenced using ZeroEntropy's /models/rerank endpoint, and on AWS Marketplace.

Evaluations

NDCG@10 scores between zerank-2 and competing closed-source proprietary rerankers. Since we are evaluating rerankers, OpenAI's text-embedding-3-small is used as an initial retriever for the Top 100 candidate documents.

Domain	OpenAI embeddings	ZeroEntropy zerank-2	ZeroEntropy zerank-1	Gemini 2.5 Flash (Listwise)	Cohere rerank-3.5
Web	0.3819	0.6346	0.6069	0.5765	0.5594
Conversational	0.4305	0.6140	0.5801	0.6021	0.5648
STEM & Logic	0.3744	0.6521	0.6283	0.5447	0.5418
Code	0.4582	0.6528	0.6310	0.6128	0.5364
Legal	0.4101	0.6644	0.6222	0.5565	0.5257
Biomedical	0.4783	0.7217	0.6967	0.5371	0.6246
Finance	0.6232	0.7600	0.7539	0.7694	0.7402
Average	0.4509	0.6714	0.6456	0.5999	0.5847

License

This model is licensed under the Apache License 2.0.