README.md

---
license: apache-2.0
base_model: unsloth/Qwen3.5-2B
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - vision-language-model
  - vehicle-recognition
  - make-model-recognition
  - fine-grained-classification
  - lora
  - unsloth
  - qwen
language:
  - en
---

# MakeModel-VLM

**MakeModel-VLM** is a compact vision-language model fine-tuned to recognize
vehicles from images and return a structured description of their
**make/model**, **class**, and **color**. It is built on top of
[`unsloth/Qwen3.5-2B`](https://huggingface.co/unsloth/Qwen3.5-2B) and specialized
for **vehicles commonly seen on Indian roads**.

The model is trained to respond with a single JSON object, making it easy to plug
into downstream pipelines (ANPR/ITMS systems, traffic analytics, fleet
monitoring, dashcam processing, etc.).

## What it does

Given a vehicle image, the model outputs:

```json
{"make_model": "Maruti Suzuki Swift", "class": "car", "color": "White"}
```

- **make_model** — the manufacturer and model (e.g. `Hero Honda Splendor`,
  `Tata Ace`, `Mahindra Bolero`, `Auto Rickshaw`).
- **class** — the vehicle category: `car`, `bike`, `auto`, `truck`, `bus`,
  `van`, or `train`.
- **color** — the dominant color of the vehicle.

## Highlights

- **Works well on cropped vehicle images.** It was trained primarily on tight
  crops of individual vehicles, so it performs best when the vehicle fills most
  of the frame — the typical output of an upstream object detector.
- **Tuned for the Indian vehicle landscape** — two-wheelers, three-wheeler
  auto-rickshaws, compact cars, and a wide range of commercial trucks/buses.
- **Structured JSON output** for zero-parsing integration.
- **Small and fast** — a 2B-parameter backbone that serves comfortably on a
  single modern GPU.

## Intended use

MakeModel-VLM is designed to sit **after a vehicle detector** in a pipeline: the
detector localizes vehicles, and this model classifies each crop. It is well
suited to:

- Automatic vehicle attribute tagging in traffic/surveillance feeds
- Fleet and parking analytics
- Enriching detection outputs with make/model/color metadata

## Usage

### Serving with vLLM (OpenAI-compatible API)

```bash
vllm serve sanskar003/MakeModel-VLM \
    --served-model-name sanskar003/MakeModel-VLM \
    --max-model-len 4096 \
    --dtype bfloat16 \
    --trust-remote-code
```

Then query it like any OpenAI vision chat endpoint:

```python
from openai import OpenAI
import base64

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

with open("vehicle_crop.jpg", "rb") as f:
    img = base64.b64encode(f.read()).decode()

resp = client.chat.completions.create(
    model="sanskar003/MakeModel-VLM",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": (
                "You are a vehicle recognition expert. Look at the vehicle in the "
                "image and identify it. Respond ONLY with a single JSON object with "
                'exactly these keys: {"make_model": string, "class": string, '
                '"color": string}. No extra text.'
            )},
            {"type": "image_url",
             "image_url": {"url": f"data:image/jpeg;base64,{img}"}},
        ],
    }],
    temperature=0.0,
    max_tokens=128,
)
print(resp.choices[0].message.content)
```

### Inference with Transformers

```python
from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "sanskar003/MakeModel-VLM", torch_dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained("sanskar003/MakeModel-VLM")

image = Image.open("vehicle_crop.jpg").convert("RGB")
instruction = (
    "You are a vehicle recognition expert. Look at the vehicle in the image and "
    "identify it. Respond ONLY with a single JSON object with exactly these keys: "
    '{"make_model": string, "class": string, "color": string}. No extra text.'
)
messages = [{"role": "user", "content": [
    {"type": "image"}, {"type": "text", "text": instruction}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Prompt format

For best results, use the exact instruction the model was trained with:

> You are a vehicle recognition expert. Look at the vehicle in the image and
> identify it. Respond ONLY with a single JSON object with exactly these keys:
> {"make_model": string, "class": string, "color": string}. No extra text.

## Training

- **Base model:** `unsloth/Qwen3.5-2B`
- **Method:** LoRA fine-tuning (vision + language layers) via
  [Unsloth](https://github.com/unslothai/unsloth), merged to 16-bit for serving.
- **Data:** a curated dataset of Indian road vehicle images with make/model,
  class, and color labels.
- **Precision:** bfloat16.

## Limitations

- Best on **cropped, reasonably clear** vehicle images; performance drops on wide
  scenes, heavy occlusion, extreme angles, or very low resolution.
- **make/model** is the hardest attribute — visually near-identical models or
  trim variants can be confused. **class** and **color** are more reliable.
- Optimized for **Indian-market vehicles**; models rarely seen in that market may
  be misidentified.
- Occasionally the identified make/model is a close but not exact match; treat
  low-confidence cases accordingly.

## License

Released under the Apache-2.0 license, consistent with the base model.