SkillReason-reranker-4b

SkillReason-reranker-4b scores the relevance of a candidate skill document to a natural-language task request. It is initialized from Qwen3-Reranker-4B and is intended for the second stage of a retrieve-then-rerank pipeline.

Model Details

Property	Value
Parameters	4B
Primary use	Agent skill reranking
Input	Query and candidate skill document
Output	logit("yes") - logit("no")
Recommended dtype	BF16 on supported GPUs
Released pipeline depth	Top 20 candidates

Quick Start

Use the official toolkit to preserve the released prompt and long-document truncation behavior:

git clone https://github.com/donghong1/SkillReason.git
cd SkillReason
pip install -e .

skillreason-download --artifact reranker-4b --output-dir artifacts

skillreason-rerank \
  --model artifacts/models/SkillReason-reranker-4b \
  --candidates outputs/retrieval/predictions.jsonl \
  --corpus examples/skills.jsonl \
  --output-dir outputs/reranked \
  --devices 0 \
  --top-n 20

The candidate file is the predictions.jsonl produced by skillreason-retrieve. The output contains the reranked skill IDs and scores, plus metadata describing the resolved scoring protocol.

Transformers Usage

The following example scores one query-document pair. For batching and long documents, use the official toolkit so the answer suffix is preserved during truncation.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "donghongjiang/SkillReason-reranker-4b"
tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
).eval()

system = (
    "Judge whether the Document meets the requirements based on the Query and "
    'the Instruct provided. Note that the answer can only be "yes" or "no".'
)
instruction = (
    "Given a task description, judge whether the skill document is relevant "
    "and useful for completing the task"
)
body = (
    f"<Instruct>: {instruction}\n"
    "<Query>: <YOUR_USER_REQUEST>\n"
    "<Document>: <SKILL_NAME> | <SKILL_DESCRIPTION> | <SKILL_DOCUMENT>"
)
prompt = (
    f"<|im_start|>system\n{system}<|im_end|>\n"
    f"<|im_start|>user\n{body}<|im_end|>\n"
    "<|im_start|>assistant\n<think>\n\n</think>\n\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
yes_id = tokenizer.encode("yes", add_special_tokens=False)[-1]
no_id = tokenizer.encode("no", add_special_tokens=False)[-1]

with torch.no_grad():
    logits = model(**inputs, use_cache=False).logits[:, -1, :]

score = logits[:, yes_id].float() - logits[:, no_id].float()
print(score.item())

Evaluation

The released protocol reranks the top 20 candidates returned by a first-stage retriever. The SkillReason toolkit provides evaluation adapters for SkillReason-Bench, SRA-Bench, and SkillRet, and records predictions, metrics, prompts, precision, and batch settings for each run.

Related Resources

• SkillReason-embedding-4b
• SkillReason-reranker-0.6b
• SkillReason-Bench
• Inference and evaluation toolkit

License

The checkpoint is released under the Apache License 2.0. Users are responsible for following the licenses and terms of the skill documents they rerank.