Instructions to use inference-optimization/Qwen3-VL-Reranker-0.6B-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/Qwen3-VL-Reranker-0.6B-small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="inference-optimization/Qwen3-VL-Reranker-0.6B-small")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("inference-optimization/Qwen3-VL-Reranker-0.6B-small")
model = AutoModelForMultimodalLM.from_pretrained("inference-optimization/Qwen3-VL-Reranker-0.6B-small", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/Qwen3-VL-Reranker-0.6B-small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/Qwen3-VL-Reranker-0.6B-small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3-VL-Reranker-0.6B-small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/Qwen3-VL-Reranker-0.6B-small

SGLang

How to use inference-optimization/Qwen3-VL-Reranker-0.6B-small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/Qwen3-VL-Reranker-0.6B-small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3-VL-Reranker-0.6B-small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/Qwen3-VL-Reranker-0.6B-small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Qwen3-VL-Reranker-0.6B-small",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/Qwen3-VL-Reranker-0.6B-small with Docker Model Runner:
```
docker model run hf.co/inference-optimization/Qwen3-VL-Reranker-0.6B-small
```

Qwen3-VL-Reranker-0.6B-small / README.md

kylesayrs

Upload folder using huggingface_hub

5735fee verified 6 days ago

preview code

Raw History Blame Contribute Delete

4.86 kB

---
license: apache-2.0
base_model:
- Qwen/Qwen3-VL-Reranker-2B
library_name: transformers
---
# Qwen3-VL-Reranker-0.6B-small

This is a small version of [Qwen/Qwen3-VL-Reranker-2B](https://huggingface.co/Qwen/Qwen3-VL-Reranker-2B) created for testing and development. It preserves the full architecture characteristics of the base (Qwen3-VL multimodal: dense text tower with QK-norms and interleaved mrope, vision tower with deepstack feature injection, spatial-merge projector, tied embeddings) with fewer layers.

## Model Details

- **Base Model**: Qwen/Qwen3-VL-Reranker-2B
- **Architecture**: qwen3_vl (Qwen3VLForConditionalGeneration)
- **Total Parameters**: 0.59B (591,990,784; dense model — activated == total)
## Configuration Changes
The following parameters were reduced from the original model (these are the ONLY changes — all other hyperparameters are identical to the base):
| Parameter | Original | Small |
|-----------|----------|-------|
| `text_config.num_hidden_layers` | 28 | 4 |
| `vision_config.depth` | 24 | 2 |
| `vision_config.deepstack_visual_indexes` | [5, 11, 17] | [1] |

The deepstack index list was trimmed consistently with the new vision depth (index 1 is a valid block index for depth 2), so the deepstack visual feature injection into the first text layers stays exercised (one deepstack merger, as in the base's per-index merger list).

## Checkpoint Structure

Single-file `model.safetensors` (bf16, 85 tensors). Key naming, per-layer key sets, and tensor shapes match the base checkpoint exactly modulo layer counts (verified against the base checkpoint's safetensors header: 0 extra/missing tensor patterns; text layers 28→4, vision blocks 24→2, deepstack mergers 3→1). `lm_head` is tied to the embedding (`tie_word_embeddings=true`) and absent from the checkpoint, as in the base.

## Usage

```python
from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained(
    "inference-optimization/Qwen3-VL-Reranker-0.6B-small",
    device_map="auto",
    dtype="bfloat16",
)
processor = AutoProcessor.from_pretrained("inference-optimization/Qwen3-VL-Reranker-0.6B-small")
inputs = processor(text="According to all known laws", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=20)
print(processor.decode(output[0], skip_special_tokens=True))
```

Multimodal (image + text) inputs are supported through the standard Qwen3-VL vision markup (`<|vision_start|>...<|vision_end|>`); the image+text path is smoke-tested below.

## Creation Process

This model was created using the llm-compressor `create-tiny-model` skill:

1. Base config inspected; the base config JSON was edited directly (text layer count, vision depth, deepstack index list), then reloaded and verified equivalent to the base config modulo the intended changes.
2. Model built with `skip_weights_download` (no base weights downloaded), re-initialized, NaN fixup, bf16 cast.
3. Fine-tuned (text-only) on the skill's copypasta toy dataset (bf16, lr 5e-5, batch 2, early-stopped) to perplexity 1.0.
4. Checkpoint structure validated against the base checkpoint header (see above).
5. Perplexity + generation + multimodal forward validated.

## Validation Output

```
Success: perplexity 1.0007 <= 10.0
Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to fly. Its wings are too small to get its fat little body off the ground. The bee, of course
Multimodal forward smoke test:
inputs: {'input_ids': (1, 22), 'attention_mask': (1, 22), 'mm_token_type_ids': (1, 22), 'pixel_values': (64, 1536), 'image_grid_thw': (1, 3)}
logits shape: (1, 22, 151936) finite: True
multimodal forward OK
```

## Notes

- All hyperparameters other than the table above are identical to the base (text hidden 2048, intermediate 6144, 16 heads / 8 kv heads, head_dim 128, interleaved mrope section [24, 20, 20], rope_theta 5e6, vocab 151936, tied embeddings, vision hidden 1024 / out 2048, patch 16, spatial merge 2, axial vision rope).
- The base is a generative reranker: its `Qwen3VLForConditionalGeneration` weights are a standard causal LM; reranker scoring compares the logits of the `true`/`false` tokens (ids 9693/2152 in the base's `1_LogitScore` sentence-transformers module). The sentence-transformers wrapper files (`modules.json`, `1_LogitScore/`, `config_sentence_transformers.json`, `scripts/`) from the base repo are not part of the transformers checkpoint and are not included; the default and reranker chat templates are included (`chat_template.jinja`, `additional_chat_templates/reranker.jinja`).
- `use_cache` stays `false` in the text config, matching the base.
- This model was fine-tuned to memorize a toy copypasta dataset; it is not intended to produce meaningful relevance scores or content.

	---
	license: apache-2.0
	base_model:
	- Qwen/Qwen3-VL-Reranker-2B
	library_name: transformers
	---

	# Qwen3-VL-Reranker-0.6B-small

	This is a small version of [Qwen/Qwen3-VL-Reranker-2B](https://huggingface.co/Qwen/Qwen3-VL-Reranker-2B) created for testing and development. It preserves the full architecture characteristics of the base (Qwen3-VL multimodal: dense text tower with QK-norms and interleaved mrope, vision tower with deepstack feature injection, spatial-merge projector, tied embeddings) with fewer layers.

	## Model Details

	- Base Model: Qwen/Qwen3-VL-Reranker-2B
	- Architecture: qwen3_vl (Qwen3VLForConditionalGeneration)
	- Total Parameters: 0.59B (591,990,784; dense model — activated == total)

	## Configuration Changes

	The following parameters were reduced from the original model (these are the ONLY changes — all other hyperparameters are identical to the base):

	\| Parameter \| Original \| Small \|
	\|-----------\|----------\|-------\|
	\| `text_config.num_hidden_layers` \| 28 \| 4 \|
	\| `vision_config.depth` \| 24 \| 2 \|
	\| `vision_config.deepstack_visual_indexes` \| [5, 11, 17] \| [1] \|

	The deepstack index list was trimmed consistently with the new vision depth (index 1 is a valid block index for depth 2), so the deepstack visual feature injection into the first text layers stays exercised (one deepstack merger, as in the base's per-index merger list).

	## Checkpoint Structure

	Single-file `model.safetensors` (bf16, 85 tensors). Key naming, per-layer key sets, and tensor shapes match the base checkpoint exactly modulo layer counts (verified against the base checkpoint's safetensors header: 0 extra/missing tensor patterns; text layers 28→4, vision blocks 24→2, deepstack mergers 3→1). `lm_head` is tied to the embedding (`tie_word_embeddings=true`) and absent from the checkpoint, as in the base.

	## Usage

	```python
	from transformers import AutoModelForImageTextToText, AutoProcessor

	model = AutoModelForImageTextToText.from_pretrained(
	"inference-optimization/Qwen3-VL-Reranker-0.6B-small",
	device_map="auto",
	dtype="bfloat16",
	)
	processor = AutoProcessor.from_pretrained("inference-optimization/Qwen3-VL-Reranker-0.6B-small")

	inputs = processor(text="According to all known laws", return_tensors="pt").to(model.device)
	output = model.generate(**inputs, max_new_tokens=20)
	print(processor.decode(output[0], skip_special_tokens=True))
	```

	Multimodal (image + text) inputs are supported through the standard Qwen3-VL vision markup (`<\|vision_start\|>...<\|vision_end\|>`); the image+text path is smoke-tested below.

	## Creation Process

	This model was created using the llm-compressor `create-tiny-model` skill:

	1. Base config inspected; the base config JSON was edited directly (text layer count, vision depth, deepstack index list), then reloaded and verified equivalent to the base config modulo the intended changes.
	2. Model built with `skip_weights_download` (no base weights downloaded), re-initialized, NaN fixup, bf16 cast.
	3. Fine-tuned (text-only) on the skill's copypasta toy dataset (bf16, lr 5e-5, batch 2, early-stopped) to perplexity 1.0.
	4. Checkpoint structure validated against the base checkpoint header (see above).
	5. Perplexity + generation + multimodal forward validated.

	## Validation Output

	```
	Success: perplexity 1.0007 <= 10.0

	Generating sample text:
	According to all known laws of aviation, there is no way a bee should be able to fly. Its wings are too small to get its fat little body off the ground. The bee, of course

	Multimodal forward smoke test:
	inputs: {'input_ids': (1, 22), 'attention_mask': (1, 22), 'mm_token_type_ids': (1, 22), 'pixel_values': (64, 1536), 'image_grid_thw': (1, 3)}
	logits shape: (1, 22, 151936) finite: True
	multimodal forward OK
	```

	## Notes

	- All hyperparameters other than the table above are identical to the base (text hidden 2048, intermediate 6144, 16 heads / 8 kv heads, head_dim 128, interleaved mrope section [24, 20, 20], rope_theta 5e6, vocab 151936, tied embeddings, vision hidden 1024 / out 2048, patch 16, spatial merge 2, axial vision rope).
	- The base is a generative reranker: its `Qwen3VLForConditionalGeneration` weights are a standard causal LM; reranker scoring compares the logits of the `true`/`false` tokens (ids 9693/2152 in the base's `1_LogitScore` sentence-transformers module). The sentence-transformers wrapper files (`modules.json`, `1_LogitScore/`, `config_sentence_transformers.json`, `scripts/`) from the base repo are not part of the transformers checkpoint and are not included; the default and reranker chat templates are included (`chat_template.jinja`, `additional_chat_templates/reranker.jinja`).
	- `use_cache` stays `false` in the text config, matching the base.
	- This model was fine-tuned to memorize a toy copypasta dataset; it is not intended to produce meaningful relevance scores or content.