README.md

---
license: apache-2.0
language:
- en
tags:
- video
- video-question-answering
- multimodal
- vision-language
- qwen3-vl
- inference-time
- frame-selection
- clip
base_model: Qwen/Qwen3-VL-2B-Instruct
pipeline_tag: video-text-to-text
library_name: transformers
---

# DW-KhotTaeVL-2B-QueryFrames

**Built on [Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) (Apache 2.0).**

A query-aware frame selection wrapper around stock Qwen3-VL-2B-Instruct
for video multiple-choice / decision-style question answering. **No model
weights are modified** — this method ships a CLIP-ViT-L/14-driven frame
selector plus an optional task-type-aware uniform-fallback policy as a
wrapper around the stock model.

On Video-MME mini at 8-frame budget, this recovers **~44 % of the
8-frame → 64-frame stock baseline gap in MCQ mode, and ~56 % in
task-aware MCQ mode**, with zero training, zero parameter changes, and
~+0.4 s overhead per question.

## Scope

This release evaluates query-aware frame selection in a video
multiple-choice / decision-style QA setting. The selector may use
both the question text and the answer options as its CLIP query.
This is appropriate for Video-MME-style MCQ benchmarks and for
operational triage workflows where the system chooses among
predefined actions or alert categories (e.g. *normal passage /
restricted-zone entry / staff activity / false alarm*). It should
**not** be read as an open-ended video-understanding benchmark claim.

## Motivation

This work started from CCTV / video-security R&D, where only a small
number of frames can be sent to a VLM under latency and compute
constraints. The released artifact is a general-purpose query-aware
frame selector for video MCQ / decision-style video QA — not a
product-specific CCTV model.

## TL;DR

| Method | trainable params | Video-MME mini 300 Q (8 frames) | Δ vs stock |
|---|---:|---:|---:|
| Stock Qwen3-VL-2B (uniform 8 f) | 0 | 57.0 % | 0 |
| **QueryFrames — MCQ mode** (no task_type) | 0 | **64.3 %** | **+7.3 pp** |
| **QueryFrames — Task-aware MCQ mode** (task_type from dataset) | 0 | **66.3 %** | **+9.3 pp** |
| Stock Qwen3-VL-2B (uniform 64 f) — ceiling | 0 | 73.7 % | +16.7 pp |

**12 of 12 task buckets non-negative; 8 strongly positive (≥ 5 pp);
0 regressions** in task-aware MCQ mode (task_type from Video-MME dataset).

> **Scope note.** This method targets short-clip, low-frame-budget
> video QA. The 300 Q numbers above are inside that design envelope.
> On the full 2700 Q split, overall Δ is **+0.22 pp** — see
> [Scope on the full Video-MME mini (2700 Q)](#scope-on-the-full-video-mme-mini-2700-q) below.

## Why it works

Stock Qwen3-VL-2B at 8 frames lags itself at 64 frames by ~17 pp.
The gap is *by definition* a frame-coverage problem (same model, same
prompt, only frame budget changes). The bottleneck is **which 8
frames you give the model**, not the model itself.

DW-KhotTaeVL-2B-QueryFrames picks the 8 frames *that match the
question* via CLIP-ViT-L/14 cosine similarity. For two task types
where 64-frame stock does *not* outperform 8-frame stock (Object
Reasoning and Temporal Reasoning per the Video-MME taxonomy), the
hybrid policy reverts to uniform sampling — frame coverage is not
the bottleneck for those questions, and CLIP scoring can mis-pick.

## Pipeline

```
For each (video, question, options[A,B,C,D]):
    1. Sample 32 uniformly-spaced candidate frames.
    2. Encode question text with CLIP-ViT-L/14 → 768-d text vector.
    3. Encode candidate frames → 768-d image vectors.
    4. Cosine similarity → pick top-8 (or uniform-8 if task is
       Object Reasoning / Temporal Reasoning, when task_type is given).
    5. Sort selected 8 frames by original temporal index.
    6. Pass 8 frames + MCQ to stock Qwen3-VL-2B-Instruct.
    7. Extract letter from output.
```

## Usage

### Install dependencies

```bash
pip install torch transformers pillow decord huggingface_hub
```

### Minimal example

```python
from dw_queryframes import QueryFrames

fv = QueryFrames(device="auto")  # auto-resolves to cuda / mps / cpu

result = fv.answer_mcq(
    video_path="cooking.mp4",
    question="What does the chef do after pouring the oil into the pot?",
    options=[
        "Chops fresh green herbs",
        "Pours broth into the pot",
        "Stirs the oil in the pot",
        "Adds salt to the pot",
    ],
    task_type=None,  # or e.g. "Action Recognition" for task-aware MCQ mode
)
print(result["pred"])              # e.g. 'B'
print(result["frames_used"])       # 'query_aware' or 'uniform_fallback'
print(result["latency_clip_s"])    # ~0.4 s
print(result["latency_gen_s"])     # ~3 s on Apple M4 MPS
```

### Two operating modes

| Mode | Input | Use | Acc 300 Q |
|---|---|---|---:|
| **MCQ mode** (no task_type) | video + question + answer options | Video-MCQ / decision-style QA without task taxonomy | **64.3 %** |
| **Task-aware MCQ mode** | + `task_type` string | benchmark or controlled workflows where task taxonomy is supplied | **66.3 %** |

Pass any of the Video-MME task labels (e.g. `"Action Recognition"`,
`"Object Reasoning"`, `"Counting Problem"`) to `task_type`. Two values
trigger the uniform-fallback path: `"Object Reasoning"` and
`"Temporal Reasoning"`. All other task strings (or `None`) use the
query-aware path.

> **MCQ mode without task_type (64.3 %, +7.3 pp)** is the default
> reported setting: it uses only the video, question, and answer
> options, with no task taxonomy.
>
> **Task-aware MCQ mode (66.3 %, +9.3 pp)** uses the `task_type`
> label supplied by Video-MME to route Object Reasoning and Temporal
> Reasoning questions to uniform sampling. This is a benchmark /
> controlled-workflow setting and is reported separately from default
> MCQ mode.

## Per-task accuracy on Video-MME mini 300 Q

| Task | n | Stock 8 f | QueryFrames | Δ |
|---|---:|---:|---:|---:|
| Action Reasoning      |  9 | 0.444 | 0.667 | **+0.222** ⭐ |
| Action Recognition    | 45 | 0.489 | 0.644 | **+0.156** ⭐ |
| Attribute Perception  | 37 | 0.730 | 0.811 | **+0.081** ⭐ |
| Counting Problem      | 34 | 0.265 | 0.353 | **+0.088** ⭐ |
| Information Synopsis  | 30 | 0.800 | 0.800 |  +0.000  |
| OCR Problems          | 23 | 0.391 | 0.609 | **+0.217** ⭐ |
| Object Reasoning      | 36 | 0.722 | 0.722 |  +0.000  |
| Object Recognition    | 51 | 0.588 | 0.667 | **+0.078** ⭐ |
| Spatial Perception    | 10 | 0.600 | 0.700 | **+0.100** ⭐ |
| Spatial Reasoning     |  9 | 0.778 | 1.000 | **+0.222** ⭐ |
| Temporal Perception   |  8 | 0.625 | 0.750 | **+0.125** ⭐ |
| Temporal Reasoning    |  8 | 0.250 | 0.250 |  +0.000  |

(Task-aware MCQ mode shown — task_type provided by Video-MME dataset.
⭐ = Δ ≥ 5 pp.)

## What this is NOT

- It is **not** a fine-tuned model. Qwen3-VL-2B-Instruct weights are
  unchanged. You can verify with the standard Hugging Face model
  hash check.
- It is **not** a leaderboard submission claim. The numbers above are
  on the publicly-available Video-MME mini split (300 Q, filtered to
  videos available locally via the standard mini chunks).
- It is **not** a replacement for fine-tuning when you have abundant
  domain data. For domain-shifted deployments (e.g. surveillance
  video), training-based adaptation may be required.

## Hardware

Runs on:

| Device | Notes |
|---|---|
| Apple M4 Max / M3 Pro (MPS, ≥ 32 GB RAM) | tested; ~3-4 s/q at 8 frames |
| NVIDIA A100 / H100 (CUDA) | works; faster |
| CPU (BF16-capable) | works but slow |

VRAM / unified memory needed: ~6-8 GB at 262 144 max-pixels with
8 frames. Lower `max_pixels` (e.g. to 153 600) if memory-constrained.

## Reproducibility

All numbers in this card are reproducible from a fresh clone of this
repo, using the [official Video-MME parquet](https://huggingface.co/datasets/lmms-lab/Video-MME)
(filtered to its `videos_chunked_01.zip` mini split).

The shipped scripts (`eval_videomme.py` and `build_hybrid.py`) are
**self-contained** — they have no external project dependencies beyond
the local `dw_queryframes.py` module and standard Python /
Hugging Face / PyTorch packages.

### Three-command reproduction recipe

```bash
# Install deps
pip install torch transformers pillow decord huggingface_hub pandas pyarrow

# 1. Reproduce stock-uniform-8f baseline (writes stock_uniform_300q.json)
python eval_videomme.py --mode stock-uniform --n-questions 300 \
    --out-json stock_uniform_300q.json

# 2. Reproduce QA-mode (no task_type) (writes wild_300q.json)
python eval_videomme.py --mode wild --n-questions 300 \
    --out-json wild_300q.json

# 3. Combine into task-aware MCQ mode via the hybrid policy
python build_hybrid.py \
    --wild-json wild_300q.json \
    --stock-uniform-json stock_uniform_300q.json \
    --out-json hybrid_300q.json
```

Expected results at 300 Q (greedy decoding, `do_sample=False`,
`max_pixels=262144`):

| Output | Accuracy | Δ vs stock |
|---|---:|---:|
| `stock_uniform_300q.json` | 0.5700 | — |
| `wild_300q.json` (MCQ mode) | 0.6433 | +7.3 pp |
| `hybrid_300q.json` (task-aware MCQ mode) | 0.6633 | +9.3 pp |

This artifact is **fully deterministic** at greedy decoding —
re-running on the same 300 questions reproduces the same 199 / 300 = 66.3 %
in task-aware MCQ mode.

> **Caveat — sample size and split.** The 300 Q numbers above are on
> the `videos_chunked_01.zip` mini subset, which happens to be mostly
> short clips. For full-split numbers on Video-MME mini 2700 Q
> (balanced short / medium / long), see
> [Scope on the full Video-MME mini (2700 Q)](#scope-on-the-full-video-mme-mini-2700-q)
> below. This release is not a leaderboard submission.

## Scope on the full Video-MME mini (2700 Q)

After the 300 Q release, the eval was extended to the full 2700 Q
split (MCQ mode without `task_type`). Stock 53.11 %, QueryFrames
53.33 %, **Δ +0.22 pp**.

This method targets short-clip, low-frame-budget video QA. The
2700 Q split is balanced across short / medium / long-form clips;
averaging across that range dilutes the gain to roughly neutral.

## Acknowledgements / Related Work

This project builds on Qwen3-VL-2B-Instruct and uses a simple
CLIP-based query-aware frame selection policy at inference time.

Query-aware and adaptive frame selection for Video-LLMs is an active
research direction. This release is an independent, simple CLIP-based
inference-time implementation focused on small-model video MCQ /
decision-style video QA under tight frame budgets.

## License

| Component | License | Source |
|---|---|---|
| This wrapper code | Apache 2.0 | this repo |
| Base model (Qwen3-VL-2B-Instruct) | Apache 2.0 | https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct |
| Frame scorer (CLIP-ViT-L/14) | MIT | https://huggingface.co/openai/clip-vit-large-patch14 |
| Eval data (Video-MME mini) | as published by lmms-lab | https://huggingface.co/datasets/lmms-lab/Video-MME |

When using or citing this work, please credit the base model:

> Built on Qwen3-VL-2B-Instruct (Apache 2.0).
> Frame selector: CLIP-ViT-L/14 (Radford et al. 2021, OpenAI, MIT).

## Citation

```bibtex
@misc{dw-khottaevl-2b-queryframes-2026,
  author = {Deaw},
  title  = {DW-KhotTaeVL-2B-QueryFrames: Query-Aware Frame Selection
            for Video MCQ on Qwen3-VL-2B-Instruct},
  year   = {2026},
  publisher = {Hugging Face},
  url    = {https://huggingface.co/commandeaw/DW-KhotTaeVL-2B-QueryFrames}
}

@misc{qwen3vl2025,
  title  = {Qwen3-VL: Multilingual Vision-Language Models},
  author = {Qwen Team},
  year   = {2025},
}

@inproceedings{radford2021clip,
  title  = {Learning Transferable Visual Models From Natural Language Supervision},
  author = {Radford, Alec and Kim, Jong Wook and others},
  booktitle = {ICML},
  year   = {2021},
}

@misc{videomme2024,
  title  = {Video-MME: The First-Ever Comprehensive Evaluation Benchmark
            of Multi-modal LLMs in Video Analysis},
  author = {Fu, Chaoyou and others},
  year   = {2024},
}
```

## Author

**Deaw** ([@commandeaw](https://huggingface.co/commandeaw)) — independent
ML practitioner. Personal research release.

Issues / questions: open an issue on the model repo.