README.md

---
library_name: transformers
pipeline_tag: image-text-to-text
base_model: Qwen/Qwen3-VL-Reranker-2B
tags:
  - tiny
  - testing
  - random-weights
  - reranker
---

This is a tiny version of Qwen/Qwen3-VL-Reranker-2B created for testing and development.

## Intended Use

A small, fast stand-in for the `qwen3_vl` reranker architecture, useful for:

- Inference / CI testing where a real 2B checkpoint is too large to download or run
- Exercising the vLLM reranker -> `Qwen3VLForSequenceClassification` path via `hf_overrides`
- Quantization & compression pipeline smoke tests (llm-compressor, compressed-tensors)
- Offloaded / distributed loading tests (see below)

Weights are random (then briefly fine-tuned on a toy corpus), so scores/generations
are not meaningful, this model is for plumbing, not output quality.

## Model Details

- **Base Model:** Qwen/Qwen3-VL-Reranker-2B
- **Architecture:** qwen3_vl (Qwen3VLForConditionalGeneration), used in vLLM as the base for Qwen3VLForSequenceClassification
- **Total Parameters:** 0.099B
- **Activated Parameters:** 0.099B

## Configuration Changes

The following parameters were reduced from the original model:

| Parameter | Original | Tiny |
|---|---|---|
| text_config.num_hidden_layers | 28 | 4 |
| text_config.hidden_size | 2048 | 512 |
| text_config.intermediate_size | 6144 | 1024 |
| text_config.num_attention_heads | 16 | 8 |
| text_config.num_key_value_heads | 8 | 2 |
| vision_config.depth | 24 | 4 |
| vision_config.hidden_size | 1024 | 256 |
| vision_config.intermediate_size | 4096 | 512 |
| vision_config.num_heads | 16 | 4 |
| vision_config.out_hidden_size | 2048 | 512 |
| vision_config.deepstack_visual_indexes | [5, 11, 17] | [0, 1, 2] |

Attention head_dim is kept at 128, and the full 151,936-token vocabulary is retained.

## Checkpoint Structure

Single safetensors file (`model.safetensors`). Key naming matches the original
checkpoint format (`model.language_model.*`, `model.visual.*`). Module-path
structure was verified equal to the base checkpoint's safetensors header.

## Usage

Load as a sequence-classification reranker in vLLM (as with the full
Qwen3-VL-Reranker-2B):

```python
from vllm import LLM

llm = LLM(
    model="soyrsoyr/Qwen3-VL-Reranker-0.1B-tiny",
    hf_overrides={
        "architectures": ["Qwen3VLForSequenceClassification"],
        "classifier_from_token": ["no", "yes"],
        "is_original_qwen3_reranker": True,
    },
)
```

Or as a plain generative model in transformers:

```python
from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "soyrsoyr/Qwen3-VL-Reranker-0.1B-tiny", device_map="auto"
)
processor = AutoProcessor.from_pretrained("soyrsoyr/Qwen3-VL-Reranker-0.1B-tiny")
input_ids = processor.tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
print(processor.tokenizer.decode(model.generate(input_ids, max_new_tokens=20)[0]))
```

### Offloaded / distributed loading (compressed-tensors)

This is a multimodal `...ForConditionalGeneration` model, so `AutoModelForCausalLM`
does **not** resolve it. Pass `AutoModelForImageTextToText` to
`load_offloaded_model` — the class you pass must match the class you call, since
that is where `device_map="auto_offload"` support is injected:

```python
from transformers import AutoModelForImageTextToText
from compressed_tensors.offload import load_offloaded_model
from compressed_tensors.distributed import init_dist

init_dist()
with load_offloaded_model(model_class=AutoModelForImageTextToText):
    model = AutoModelForImageTextToText.from_pretrained(
        "soyrsoyr/Qwen3-VL-Reranker-0.1B-tiny",
        device_map="auto_offload",                 # weights on CPU/disk, GPU for activations
    )
```

## Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

- Config inspected via `inspect_config.py`
- Tiny model created via a modified `save_tiny_model.py`, adapted for the
  multimodal class (`AutoModelForImageTextToText.from_config`); the text tower
  and vision tower were shrunk and any all-zero / non-finite / extreme param was
  fixed after `init_weights()`
- Fine-tuned on the copypasta dataset; reached training perplexity 1.00
  (target: ≤3.0) at lr=5e-4 (CPU, Adafactor)
- Checkpoint structure validated against the original HuggingFace safetensors
  header (module-path match)
- Inference validated via `validate_tiny_model.py`

## Notes

- **Saved as Qwen3VLForConditionalGeneration** (matching Qwen3-VL-Reranker-2B).
  vLLM converts it to `Qwen3VLForSequenceClassification` at load time via the
  `hf_overrides` shown above, so this tiny model exercises the reranker →
  sequence-classification path.
- **Projector alignment.** `vision_config.out_hidden_size` is set to the text
  hidden size (512) so the visual merger projects into the text tower;
  `deepstack_visual_indexes` is remapped to valid indices for the reduced
  4-layer vision tower.
- `tie_word_embeddings=True`: `lm_head` shares `embed_tokens` and is not stored
  as a separate tensor.

Validation output: `Success: 1.003219485282898 <= 10.0`