Commerce-Rerank-Qwen3-8B

🇰🇷 한국어 카드: MODEL_CARD.ko.md

A product-search reranker. Given a shopping query and candidate product titles, it scores each title so the list can be re-ordered. It is Qwen/Qwen3-Reranker-8B fine-tuned with a LoRA adapter on the Marqo-GS-10M shopping ranking data, then merged back into full bf16 weights. Same architecture, same prompt, same size as the base model.

On held-out queries it never saw in training, it ranks better than the strongest public zero-shot reranker we tested, which was the base model itself: nDCG@10 0.6968 vs 0.6089 (+8.79 pp, 95% CI [+8.60, +8.98]). The data is English Google Shopping results, so the claim is limited to that distribution.

How to use

The model scores exactly like the base Qwen3-Reranker (last-position yes/no logits). It was trained and evaluated with the commerce instruction in inference.py; other instructions are untested.

# pip install "transformers>=4.51" torch huggingface_hub
from huggingface_hub import hf_hub_download
import importlib.util
p = hf_hub_download("ThakiCloud/Commerce-Rerank-Qwen3-8B", "inference.py")
spec = importlib.util.spec_from_file_location("inference", p); mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)

rr = mod.CommerceReranker("ThakiCloud/Commerce-Rerank-Qwen3-8B")      # bf16, GPU if available
rr.rerank("waterproof trail running shoes men", [
    "Men's Gore-Tex Trail Running Shoe, Waterproof",
    "Women's Road Running Shoe, Breathable Mesh",
    "Shoe Waterproofing Spray 300ml",
], top_k=3)
# -> [(index, score), ...] highest first. Scores order candidates within one query; they are not probabilities.

Score = logit(yes) − logit(no), bf16, left padding. If you need a probability-like number, apply a sigmoid, but we did not calibrate it.

Results

All numbers come from our own runs on one B200, transformers 5.19.0, bf16, same candidate lists and same code for every model. Metric: nDCG@10 per query, gain linear in the original Google Shopping position (101 − position) / 100. CIs are paired query-level bootstrap (2,000 resamples).

Sealed test (scored once, after the model was frozen)

Split	Queries	Qwen3-Reranker-8B (zero-shot)	This model	Δ [95% CI]
novel_query (unseen queries, seen products) — primary	19,647	0.6089	0.6968	+8.79 pp [+8.60, +8.98]
zero_shot (unseen queries and unseen products) — reported only	19,647	0.6071	0.6918	+8.47 pp [+8.27, +8.67]

The released checkpoint is seed 1, chosen before the test run as the seed-median by dev nDCG@10. The test split was opened once for this model; no hyper-parameter or seed was changed afterwards.

Dev (in-domain held-out queries, 1,029 queries / 51,416 pairs)

Model	nDCG@10
Qwen3-Reranker-0.6B (commerce instruction)	0.5890
Qwen3-Reranker-4B (commerce instruction)	0.5791
Qwen3-Reranker-8B (commerce instruction)	0.5904
Qwen3-Reranker-8B (default instruction) — strongest public baseline	0.5989
mxbai-rerank-large-v2	0.5939
This recipe, seed 0 / 1 / 2	0.6970 / 0.6943 / 0.6929

All three seeds beat the baseline with the 95% CI lower bound above zero (+8.98, +8.69, +8.55 pp).

Cost

Item	Value
Checkpoint	16.39 GB (merged bf16 safetensors; the LoRA adapter alone was 174.7 MB)
Inference GPU memory (peak, bf16, ~50 candidates per call)	17.24 GB on one B200
Inference speed (B200, bf16, one query's ~50 candidates per call)	145 ms per query (345 pairs/s)
Sealed-test scoring throughput (B200, token-budget batching)	~468 pairs/s (981,164 pairs in 2,095 GPU-s)
Training	1 epoch, ~63 min per seed on one B200, peak 26.7–27.3 GB

Training recipe

LoRA r=16, α=32 on Qwen3-Reranker-8B; ListNet loss over groups of 16 candidates (target ∝ 1/position); lr 1e-4, 1 epoch, 8 queries per step, max length 512; adapter merged into bf16 weights, and the merged weights are what was evaluated and released. Training data: 19,026 in-domain queries (948,584 pairs) from 100 shards of Marqo-GS-10M; dev = queries with sha1(query_id) % 20 == 0, excluded from training.

Limitations

• The target is the Google Shopping result order, not human relevance labels. The model learned to recover that order from (query, title). It can inherit whatever that ranking rewards, including popularity or ads.
• English only. Product titles only (no descriptions, images, price or stock). Not tested on Korean queries.
• One test split per claim. We did not compare against fine-tuned rerankers from other teams or commercial APIs.
• bf16 merged weights are not bit-identical to "base + adapter"; we measured the released file, not the adapter.
• Marqo-GS-10M is distributed with an Apache-2.0 license on Hugging Face. The dataset is derived from Google Shopping content; the applicability of any underlying third-party content rights has not been independently verified.

Data and license

• Training data: Marqo-GS-10M (Apache-2.0). No data is redistributed in this repo.
• Base model: Qwen/Qwen3-Reranker-8B (Apache-2.0).
• This model: Apache-2.0.