---
license: apache-2.0
base_model: Qwen/Qwen2.5-0.5B-Instruct
tags: [vision-language-model, vqa, image-captioning, lora, siglip2, qwen2.5]
---

# tinyvlm-vqa

A compact LLaVA-style VLM: SigLIP2 vision encoder -> pixel-shuffle -> MLP
projector -> Qwen2.5-0.5B-Instruct with LoRA.

| | |
|---|---|
| Vision encoder | `google/siglip2-base-patch16-224` (frozen) |
| LLM | `Qwen/Qwen2.5-0.5B-Instruct` (LoRA: q,k,v,o,gate,up,down) |
| Visual tokens | 49 per image (14x14 patches, 2x2 pixel-shuffle) |
| Image placeholder | `<|image_pad|>` -- an existing Qwen token, so no vocab resize |
| Stage 1 | projector alignment on Flickr8k + Flickr30k captions |
| Stage 2 | LoRA instruction tuning on VQAv2 + LLaVA-ReCap + captions |

## Answer length is prompt-controlled

Stage 2 mixes one-word VQA answers with multi-sentence descriptions, and VQA
examples carry an explicit hint. Append it for short answers, omit it for prose:

```
What color is the bus?
Answer the question using a single word or phrase.     -> "red"

What color is the bus?                                 -> "The bus is red."
```

## Files

- `projector.pt` -- projector weights (`pool_stride` is recoverable from the tensor shapes)
- `lora_adapter/` -- PEFT adapter for the LLM

## Usage

Needs the companion code from the training repo (`Model/model.py`,
`Model/dataset.py`, `Inference/inference.py`):

```python
from huggingface_hub import hf_hub_download, snapshot_download
from inference import load_model, answer

projector = hf_hub_download(repo_id="dhruvpatel93/tinyvlm-vqa", filename="projector.pt")
lora = snapshot_download(repo_id="dhruvpatel93/tinyvlm-vqa", allow_patterns=["lora_adapter/*"]) + "/lora_adapter"

model = load_model(projector, lora, device="cuda")
print(answer(model, "photo.jpg", "What is in this image?"))
print(answer(model, "photo.jpg", "What color is the car?", short_answer=True))
```

## Limitations

224x224 input, 49 visual tokens, and a 0.5B LLM. Expect
everyday-scene captioning and simple VQA -- not OCR, fine detail, counting, or
multi-step visual reasoning.

## Training

- optimizer steps: 10518
- final held-out loss: 0.9565
- batch size 4 x 4 accum, lr 0.0002