README.md

---
library_name: transformers
license: apache-2.0
base_model:
- state-spaces/mamba2-370m
- google/siglip-so400m-patch14-384
tags:
- video-understanding
- long-video
- state-space-model
- mamba
- mamba-2
- linear-complexity
- multimodal
- video-question-answering
language:
- en
pipeline_tag: video-text-to-text
model-index:
- name: video-ma2mba-0.3b
  results:
  - task:
      type: video-text-to-text
      name: Long-Form Video Understanding
    dataset:
      type: custom
      name: Video-MME (w/o sub)
    metrics:
    - type: accuracy
      value: 33.1
      name: Overall
  - task:
      type: video-text-to-text
      name: Long-Form Video Understanding
    dataset:
      type: custom
      name: LongVideoBench
    metrics:
    - type: accuracy
      value: 34.0
      name: Val
  - task:
      type: video-text-to-text
      name: Video Question Answering
    dataset:
      type: custom
      name: ActivityNet-QA
    metrics:
    - type: accuracy
      value: 43.8
      name: Accuracy
  - task:
      type: video-text-to-text
      name: Video Question Answering
    dataset:
      type: custom
      name: MVBench
    metrics:
    - type: accuracy
      value: 41.1
      name: Accuracy
---

# video-ma2mba-0.3b

**Video-Ma²mba** is an efficient long-form video understanding model that replaces Transformer attention with **State Space Models (SSMs)** in the **Mamba-2** framework, achieving **linear** time and memory scaling in sequence length. A single GPU can therefore process video sequences equivalent to **millions of tokens — over two hours of video at 1 FPS**.

<p align="center">
  <a href="https://arxiv.org/abs/2411.19460"><img src="https://img.shields.io/badge/arXiv-2411.19460-b31b1b" alt="arXiv"></a>
  <a href="https://huggingface.co/papers/2411.19460"><img src="https://img.shields.io/badge/%F0%9F%A4%97-Paper-yellow" alt="HF Paper"></a>
  <a href="https://ivy-lvlm.github.io/Video-MA2MBA/"><img src="https://img.shields.io/badge/Project-Page-blue" alt="Project Page"></a>
  <a href="https://github.com/IVY-LVLM/Video-MA2MBA"><img src="https://img.shields.io/badge/GitHub-Code-black" alt="GitHub"></a>
  <a href="https://huggingface.co/interlive"><img src="https://img.shields.io/badge/%F0%9F%A4%97-Model_Collection-yellow" alt="HF"></a>
</p>

> **Paper**: *Look Every Frame All at Once: Video-Ma²mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing*
>
> Hosu Lee, Junho Kim, Hyunjun Kim, Yong Man Ro
>
> Integrated Vision and Language Lab, KAIST

## What is Video-Ma²mba?

Processing long videos with Transformer-based video-LLMs is expensive: self-attention scales **quadratically** with the number of tokens, so memory and compute explode as videos grow to thousands of frames. Video-Ma²mba addresses this at the architecture level.

- **State-space backbone.** The language model is built on **Mamba-2**, replacing attention with selective state-space layers whose cost grows **linearly** with sequence length. A long video is consumed as one continuous stream rather than a truncated window.
- **Multi-Axis Gradient Checkpointing (MA-GC).** A training-time memory technique that checkpoints activations along multiple axes, sharply reducing the memory footprint versus standard gradient checkpointing and making long-sequence training feasible.
- **Streaming-scale context.** Together these let the model ingest sequences equivalent to **millions of tokens — 2+ hours of video at 1 FPS — on a single GPU**.

### Architecture

```
Video (sampled at 1 FPS)
  │
  ▼
[SigLIP-so400m Vision Encoder]  →  per-frame visual tokens
  │
  ▼
[MLP Projector]                 →  language-model embedding space
  │
  ▼
[Mamba-2 Language Model]        ←  linear-time SSM · 48 layers · d=1024
  │
  ▼
Text response
```

| Component | Specification |
|---|---|
| Architecture | `LlavaMambaForCausalLM` (`llava_mamba`) |
| Language model | Mamba-2 · 48 layers · hidden size 1024 · ~370M params |
| Vision encoder | [`google/siglip-so400m-patch14-384`](https://huggingface.co/google/siglip-so400m-patch14-384) |
| Projector | 2-layer MLP with GELU |
| Precision | bfloat16 |
| Sampling | 1 FPS with time-instruction conditioning |
| Max context | up to ~1M tokens |

## Results

Scores are **as reported in the paper** for the Video-Ma²mba family. The row marked **(this)** corresponds to this checkpoint.

### Video-MME (w/o subtitles)

| Model | Total Size | Short | Medium | Long | Overall |
|---|:---:|:---:|:---:|:---:|:---:|
| **Video-Ma²mba-0.3B (this)** | 0.7B | 37.4 | 35.0 | 26.8 | **33.1** |
| Video-Ma²mba-1.3B | 1.8B | 49.4 | 39.2 | 31.9 | 40.3 |
| Video-Ma²mba-2.7B | 3.1B | 57.6 | 42.7 | 35.4 | 45.2 |

### LongVideoBench

| Model | Total Size | Val | Test |
|---|:---:|:---:|:---:|
| **Video-Ma²mba-0.3B (this)** | 0.7B | **34.0** | 34.2 |
| Video-Ma²mba-1.3B | 1.8B | 38.0 | 39.8 |
| Video-Ma²mba-2.7B | 3.1B | 43.0 | 44.2 |

### General Video Understanding

| Model | Total Size | ActivityNet-QA | Video-ChatGPT | MVBench |
|---|:---:|:---:|:---:|:---:|
| **Video-Ma²mba-0.3B (this)** | 0.7B | 43.8 | 2.69 | 41.1 |
| Video-Ma²mba-1.3B | 1.8B | 50.0 | 2.76 | 44.4 |
| Video-Ma²mba-2.7B | 3.1B | 51.7 | 3.03 | 48.3 |

## Model Variants

| Model | LLM (Mamba-2) | Total |
|---|:---:|:---:|
| [video-ma2mba-0.3b](https://huggingface.co/interlive/video-ma2mba-0.3b) **(this)** | 370M | 0.7B |
| [video-ma2mba-1.3b](https://huggingface.co/interlive/video-ma2mba-1.3b) | 1.3B | 1.8B |
| [video-ma2mba-2.7b](https://huggingface.co/interlive/video-ma2mba-2.7b) | 2.7B | 3.1B |

## Usage

Video-Ma²mba uses a custom `LlavaMambaForCausalLM` architecture. For the inference pipeline, video preprocessing, and evaluation scripts, please refer to the official [Video-Ma²mba GitHub repository](https://github.com/IVY-LVLM/Video-MA2MBA).

## Citation

```bibtex
@article{lee2024videoma2mba,
  title={Look Every Frame All at Once: Video-Ma2mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing},
  author={Lee, Hosu and Kim, Junho and Kim, Hyunjun and Ro, Yong Man},
  journal={arXiv preprint arXiv:2411.19460},
  year={2024}
}
```

## License

This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).