| --- |
| library_name: transformers |
| license: apache-2.0 |
| base_model: interlive/video-ma2mba-0.3b |
| tags: |
| - video-understanding |
| - frame-selection |
| - long-video |
| - reinforcement-learning |
| - policy-gradient |
| - mamba |
| - multimodal |
| - plug-and-play |
| language: |
| - en |
| pipeline_tag: video-classification |
| --- |
| |
| # ReFoCUS-0.3B |
|
|
| **ReFoCUS** (**Re**inforcement-guided **F**rame **O**ptimization for **C**ontextual **U**nder**S**tanding) is a lightweight frame-selection model for long-video understanding. |
| Given a video and a question, it identifies the frames that are **truly essential for answering the question** and hands them to a downstream Video-LLM, replacing uniform sampling with query-conditioned visual evidence. |
|
|
| <p align="center"> |
| <a href="https://arxiv.org/abs/2506.01274"><img src="https://img.shields.io/badge/arXiv-2506.01274-b31b1b" alt="arXiv"></a> |
| <a href="https://interlive-team.github.io/ReFoCUS/"><img src="https://img.shields.io/badge/Project-Page-blue" alt="Project Page"></a> |
| <a href="https://github.com/interlive-team/ReFoCUS"><img src="https://img.shields.io/badge/GitHub-Code-black" alt="GitHub"></a> |
| <a href="https://huggingface.co/interlive"><img src="https://img.shields.io/badge/%F0%9F%A4%97-Model_Collection-yellow" alt="HF"></a> |
| </p> |
|
|
| > **Paper**: *ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding* |
| > |
| > Hosu Lee\*, Junho Kim\*, Hyunjun Kim, Yong Man Ro |
| > |
| > KAIST · UIUC |
|
|
| ## What is ReFoCUS? |
|
|
| ReFoCUS is a framework for finding the visual evidence scattered along a video's temporal axis that is *truly essential* for answering a given query. The evidence needed to answer a question is usually sparse and unevenly distributed, yet most video-LLMs still consume a fixed set of uniformly sampled frames, and heuristic or retrieval-based selection is never jointly optimized with the model's own reasoning, so it often conflates raw visual dynamics with true semantic relevance. |
|
|
| ReFoCUS is the **first framework to integrate online policy-gradient reinforcement learning into frame-level optimization for video-LLMs**. Instead of aligning the *textual outputs* of an LMM with preferences, it optimizes the *visual inputs* the model attends to: |
|
|
| - **Policy model (this model)**: reads the dense video together with the question and autoregressively picks the frames that best support answering, each choice conditioned on the query and the frames already chosen. |
| - **Reward model (frozen)**: a reference video-LLM scores each selected frame set by how confidently it favors the correct answer, capturing its internal utility of that visual evidence. |
| - **Reinforcement learning, no frame labels**: the policy learns from these reward signals alone; frame-level supervision is never required, and semantically and temporally coherent frame compositions emerge implicitly. |
|
|
| ### Plug-and-Play Pipeline |
|
|
| ``` |
| Video (4 fps, up to 512 frames) + Question |
| │ |
| ▼ |
| [ReFoCUS Policy Model] ← this model |
| │ |
| │ 32 query-relevant frames (temporally sorted) |
| ▼ |
| [Downstream Video-LLM] ← frozen, any off-the-shelf |
| │ |
| ▼ |
| Answer |
| ``` |
|
|
| Starting from the special `<|startofframe|>` token, the policy attends over the pool of frame embeddings and samples frames one at a time without replacement, so every choice is conditioned on the question and all prior selections. Its Mamba-2 (state-space) backbone keeps computation and memory **linear** in video length, and the whole selection runs in a single feed-forward autoregressive pass, with no iterative evaluation or post-processing. |
|
|
| ## Usage |
|
|
| The selected frame indices are temporally sorted and can be forwarded to **any** off-the-shelf video-LLM. |
| For the full selection/inference pipeline and evaluation scripts, please refer to the [ReFoCUS GitHub repository](https://github.com/interlive-team/ReFoCUS). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @InProceedings{Lee_2026_CVPR, |
| author = {Lee, Hosu and Kim, Junho and Kim, Hyunjun and Ro, Yong Man}, |
| title = {ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding}, |
| booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings}, |
| month = {June}, |
| year = {2026}, |
| pages = {8291-8302} |
| } |
| ``` |
|
|
| ## License |
|
|
| This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0). |
|
|
