README.md

---
library_name: transformers
license: apache-2.0
base_model: interlive/video-ma2mba-1.3b
tags:
- video-understanding
- frame-selection
- long-video
- reinforcement-learning
- policy-gradient
- mamba
- multimodal
- plug-and-play
language:
- en
pipeline_tag: video-classification
model-index:
- name: ReFoCUS-1.3B
  results:
  - task:
      type: video-classification
      name: Query-Conditioned Frame Selection
    dataset:
      type: custom
      name: Video-MME (w/o sub)
    metrics:
    - type: accuracy
      value: 68.5
      name: Overall (w/ Qwen3-VL-8B)
    - type: accuracy
      value: 62.6
      name: Overall (w/ LLaVA-OneVision-7B)
  - task:
      type: video-classification
      name: Query-Conditioned Frame Selection
    dataset:
      type: custom
      name: LongVideoBench
    metrics:
    - type: accuracy
      value: 63.3
      name: Acc. val (w/ Qwen3-VL-8B)
    - type: accuracy
      value: 61.0
      name: Acc. val (w/ LLaVA-OneVision-7B)
  - task:
      type: video-classification
      name: Query-Conditioned Frame Selection
    dataset:
      type: custom
      name: MLVU
    metrics:
    - type: accuracy
      value: 72.5
      name: m-avg (w/ Qwen3-VL-8B)
    - type: accuracy
      value: 68.5
      name: m-avg (w/ LLaVA-OneVision-7B)
  - task:
      type: video-classification
      name: Query-Conditioned Frame Selection
    dataset:
      type: custom
      name: Video-MMMU
    metrics:
    - type: accuracy
      value: 61.1
      name: Overall (w/ Qwen3-VL-8B)
    - type: accuracy
      value: 35.7
      name: Overall (w/ LLaVA-OneVision-7B)
---

# ReFoCUS-1.3B

**ReFoCUS** (**Re**inforcement-guided **F**rame **O**ptimization for **C**ontextual **U**nder**S**tanding) is a lightweight frame-selection model for long-video understanding.
Given a video and a question, it identifies the frames that are **truly essential for answering the question** and hands them to a downstream Video-LLM, replacing uniform sampling with query-conditioned visual evidence.

<p align="center">
  <a href="https://arxiv.org/abs/2506.01274"><img src="https://img.shields.io/badge/arXiv-2506.01274-b31b1b" alt="arXiv"></a>
  <a href="https://interlive-team.github.io/ReFoCUS/"><img src="https://img.shields.io/badge/Project-Page-blue" alt="Project Page"></a>
  <a href="https://github.com/interlive-team/ReFoCUS"><img src="https://img.shields.io/badge/GitHub-Code-black" alt="GitHub"></a>
  <a href="https://huggingface.co/interlive"><img src="https://img.shields.io/badge/%F0%9F%A4%97-Model_Collection-yellow" alt="HF"></a>
</p>

> **Paper**: *ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding*
>
> Hosu Lee\*, Junho Kim\*, Hyunjun Kim, Yong Man Ro
>
> KAIST · UIUC

## What is ReFoCUS?

ReFoCUS is a framework for finding the visual evidence scattered along a video's temporal axis that is *truly essential* for answering a given query. The evidence needed to answer a question is usually sparse and unevenly distributed, yet most video-LLMs still consume a fixed set of uniformly sampled frames, and heuristic or retrieval-based selection is never jointly optimized with the model's own reasoning, so it often conflates raw visual dynamics with true semantic relevance.

ReFoCUS is the **first framework to integrate online policy-gradient reinforcement learning into frame-level optimization for video-LLMs**. Instead of aligning the *textual outputs* of an LMM with preferences, it optimizes the *visual inputs* the model attends to:

- **Policy model (this model)**: reads the dense video together with the question and autoregressively picks the frames that best support answering, each choice conditioned on the query and the frames already chosen.
- **Reward model (frozen)**: a reference video-LLM scores each selected frame set by how confidently it favors the correct answer, capturing its internal utility of that visual evidence.
- **Reinforcement learning, no frame labels**: the policy learns from these reward signals alone; frame-level supervision is never required, and semantically and temporally coherent frame compositions emerge implicitly.

### Plug-and-Play Pipeline

```
Video (4 fps, up to 512 frames) + Question
  │
  ▼
[ReFoCUS Policy Model]     ← this model
  │
  │ 32 query-relevant frames (temporally sorted)
  ▼
[Downstream Video-LLM]     ← frozen, any off-the-shelf
  │
  ▼
Answer
```

Starting from the special `<|startofframe|>` token, the policy attends over the pool of frame embeddings and samples frames one at a time without replacement, so every choice is conditioned on the question and all prior selections. Its Mamba-2 (state-space) backbone keeps computation and memory **linear** in video length, and the whole selection runs in a single feed-forward autoregressive pass, with no iterative evaluation or post-processing.

## Results

### Video QA Benchmarks

| Model | LLM Size | Video-MME | LongVideoBench | MLVU | Video-MMMU |
|---|:---:|:---:|:---:|:---:|:---:|
| *Closed Source* | | | | | |
| Gemini 2.5 Flash | - | 66.0 | 47.9 | 52.8 | 40.6 |
| **+ ReFoCUS** | - | 69.5 (+3.5) | 50.9 (+3.0) | 58.0 (+5.2) | 45.6 (+5.0) |
| GPT-4o | - | 58.8 | 49.5 | 58.7 | 62.9 |
| **+ ReFoCUS** | - | 60.8 (+2.0) | 52.9 (+3.4) | 65.1 (+6.4) | 62.1 (-0.8) |
| *Open Source* | | | | | |
| LLaVA-OneVision | 0.5B | 43.5 | 44.7 | 44.8 | 17.3 |
| **+ ReFoCUS** | 0.5B | 47.1 (+3.6) | 48.7 (+4.0) | 50.3 (+5.5) | 19.4 (+2.1) |
| InternVL3 | 1B | 50.0 | 47.6 | 54.0 | 27.7 |
| **+ ReFoCUS** | 1B | 53.6 (+3.6) | 50.6 (+3.0) | 58.9 (+4.9) | 29.3 (+1.6) |
| VideoLLaMA 3 | 2B | 43.1 | 48.8 | 46.8 | 28.7 |
| **+ ReFoCUS** | 2B | 47.1 (+4.0) | 53.7 (+4.9) | 50.2 (+3.4) | 29.2 (+0.5) |
| InternVL3 | 2B | 58.4 | 50.9 | 62.7 | 38.3 |
| **+ ReFoCUS** | 2B | 60.7 (+2.3) | 54.9 (+4.0) | 68.0 (+5.3) | 39.3 (+1.0) |
| InternVL3.5 | 4B | 62.7 | 57.7 | 66.6 | 52.0 |
| **+ ReFoCUS** | 4B | 65.9 (+3.2) | 62.6 (+4.9) | 71.5 (+4.9) | 53.3 (+1.3) |
| Qwen3-VL | 4B | 62.1 | 57.4 | 63.1 | 54.0 |
| **+ ReFoCUS** | 4B | 66.4 (+4.3) | 61.9 (+4.5) | 71.9 (+8.8) | 56.4 (+2.4) |
| VideoLLaMA 3 | 7B | 59.0 | 54.8 | 52.9 | 32.8 |
| **+ ReFoCUS** | 7B | 62.2 (+3.2) | 57.0 (+2.2) | 59.8 (+6.9) | 34.4 (+1.6) |
| LLaVA-OneVision | 7B | 58.4 | 55.0 | 63.7 | 34.1 |
| **+ ReFoCUS** | 7B | 62.6 (+4.2) | 61.0 (+6.0) | 68.5 (+4.8) | 35.7 (+1.6) |
| InternVL3 | 8B | 64.3 | 57.8 | 68.1 | 49.3 |
| **+ ReFoCUS** | 8B | 67.0 (+2.7) | 62.0 (+4.2) | 72.7 (+4.6) | 50.6 (+1.3) |
| InternVL3.5 | 8B | 64.4 | 59.7 | 67.3 | 50.0 |
| **+ ReFoCUS** | 8B | 66.7 (+2.3) | 64.1 (+4.4) | 70.6 (+3.3) | 53.2 (+3.2) |
| Qwen3-VL | 8B | 65.0 | 56.6 | 63.0 | 59.1 |
| **+ ReFoCUS** | 8B | 68.5 (+3.5) | 63.3 (+6.7) | 72.5 (+9.5) | 61.1 (+2.0) |

### Open-Ended Video QA

| Model | LLM Size | NExT-QA (WUPS) | ActivityNet-QA Score | Video-ChatGPT Score |
|---|:---:|:---:|:---:|:---:|
| VideoLLaMA 3 | 7B | 25.8 | 70.2 | 60.3 |
| **+ ReFoCUS** | 7B | 26.5 (+0.7) | 72.2 (+2.0) | 62.7 (+2.4) |
| LLaVA-OneVision | 7B | 16.2 | 68.8 | 59.9 |
| **+ ReFoCUS** | 7B | 16.4 (+0.2) | 69.7 (+0.9) | 61.4 (+1.5) |
| InternVL3 | 8B | 26.6 | 69.4 | 59.4 |
| **+ ReFoCUS** | 8B | 26.8 (+0.2) | 70.4 (+1.0) | 61.0 (+1.6) |
| InternVL3.5 | 8B | 24.3 | 67.7 | 59.5 |
| **+ ReFoCUS** | 8B | 24.7 (+0.4) | 68.8 (+1.1) | 60.8 (+1.3) |
| Qwen3-VL | 8B | 25.3 | 65.9 | 60.3 |
| **+ ReFoCUS** | 8B | 25.7 (+0.4) | 67.2 (+1.3) | 62.3 (+2.0) |

## Usage

The selected frame indices are temporally sorted and can be forwarded to **any** off-the-shelf video-LLM.
For the full selection/inference pipeline and evaluation scripts, please refer to the [ReFoCUS GitHub repository](https://github.com/interlive-team/ReFoCUS).

## Citation

```bibtex
@InProceedings{Lee_2026_CVPR,
    author    = {Lee, Hosu and Kim, Junho and Kim, Hyunjun and Ro, Yong Man},
    title     = {ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings},
    month     = {June},
    year      = {2026},
    pages     = {8291-8302}
}
```

## License

This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).