Add Video-Ma2mba-0.3B weights, tokenizer, config, and model card

07ee872 17 days ago

6.28 kB

library_name: transformers
license: apache-2.0
base_model:
  - state-spaces/mamba2-370m
  - google/siglip-so400m-patch14-384
tags:
  - video-understanding
  - long-video
  - state-space-model
  - mamba
  - mamba-2
  - linear-complexity
  - multimodal
  - video-question-answering
language:
  - en
pipeline_tag: video-text-to-text
model-index:
  - name: video-ma2mba-0.3b
    results:
      - task:
          type: video-text-to-text
          name: Long-Form Video Understanding
        dataset:
          type: custom
          name: Video-MME (w/o sub)
        metrics:
          - type: accuracy
            value: 33.1
            name: Overall
      - task:
          type: video-text-to-text
          name: Long-Form Video Understanding
        dataset:
          type: custom
          name: LongVideoBench
        metrics:
          - type: accuracy
            value: 34
            name: Val
      - task:
          type: video-text-to-text
          name: Video Question Answering
        dataset:
          type: custom
          name: ActivityNet-QA
        metrics:
          - type: accuracy
            value: 43.8
            name: Accuracy
      - task:
          type: video-text-to-text
          name: Video Question Answering
        dataset:
          type: custom
          name: MVBench
        metrics:
          - type: accuracy
            value: 41.1
            name: Accuracy

video-ma2mba-0.3b

Video-Ma²mba is an efficient long-form video understanding model that replaces Transformer attention with State Space Models (SSMs) in the Mamba-2 framework, achieving linear time and memory scaling in sequence length. A single GPU can therefore process video sequences equivalent to millions of tokens — over two hours of video at 1 FPS.

Paper: Look Every Frame All at Once: Video-Ma²mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing

Hosu Lee, Junho Kim, Hyunjun Kim, Yong Man Ro

Integrated Vision and Language Lab, KAIST

What is Video-Ma²mba?

Processing long videos with Transformer-based video-LLMs is expensive: self-attention scales quadratically with the number of tokens, so memory and compute explode as videos grow to thousands of frames. Video-Ma²mba addresses this at the architecture level.

State-space backbone. The language model is built on Mamba-2, replacing attention with selective state-space layers whose cost grows linearly with sequence length. A long video is consumed as one continuous stream rather than a truncated window.
Multi-Axis Gradient Checkpointing (MA-GC). A training-time memory technique that checkpoints activations along multiple axes, sharply reducing the memory footprint versus standard gradient checkpointing and making long-sequence training feasible.
Streaming-scale context. Together these let the model ingest sequences equivalent to millions of tokens — 2+ hours of video at 1 FPS — on a single GPU.

Architecture

Video (sampled at 1 FPS)
  │
  ▼
[SigLIP-so400m Vision Encoder]  →  per-frame visual tokens
  │
  ▼
[MLP Projector]                 →  language-model embedding space
  │
  ▼
[Mamba-2 Language Model]        ←  linear-time SSM · 48 layers · d=1024
  │
  ▼
Text response

Component	Specification
Architecture	`LlavaMambaForCausalLM` (`llava_mamba`)
Language model	Mamba-2 · 48 layers · hidden size 1024 · ~370M params
Vision encoder	`google/siglip-so400m-patch14-384`
Projector	2-layer MLP with GELU
Precision	bfloat16
Sampling	1 FPS with time-instruction conditioning
Max context	up to ~1M tokens

Results

Scores are as reported in the paper for the Video-Ma²mba family. The row marked (this) corresponds to this checkpoint.

Video-MME (w/o subtitles)

Model	Total Size	Short	Medium	Long	Overall
Video-Ma²mba-0.3B (this)	0.7B	37.4	35.0	26.8	33.1
Video-Ma²mba-1.3B	1.8B	49.4	39.2	31.9	40.3
Video-Ma²mba-2.7B	3.1B	57.6	42.7	35.4	45.2

LongVideoBench

Model	Total Size	Val	Test
Video-Ma²mba-0.3B (this)	0.7B	34.0	34.2
Video-Ma²mba-1.3B	1.8B	38.0	39.8
Video-Ma²mba-2.7B	3.1B	43.0	44.2

General Video Understanding

Model	Total Size	ActivityNet-QA	Video-ChatGPT	MVBench
Video-Ma²mba-0.3B (this)	0.7B	43.8	2.69	41.1
Video-Ma²mba-1.3B	1.8B	50.0	2.76	44.4
Video-Ma²mba-2.7B	3.1B	51.7	3.03	48.3

Model Variants

Model	LLM (Mamba-2)	Total
video-ma2mba-0.3b (this)	370M	0.7B
video-ma2mba-1.3b	1.3B	1.8B
video-ma2mba-2.7b	2.7B	3.1B

Usage

Video-Ma²mba uses a custom LlavaMambaForCausalLM architecture. For the inference pipeline, video preprocessing, and evaluation scripts, please refer to the official Video-Ma²mba GitHub repository.

Citation

@article{lee2024videoma2mba,
  title={Look Every Frame All at Once: Video-Ma2mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing},
  author={Lee, Hosu and Kim, Junho and Kim, Hyunjun and Ro, Yong Man},
  journal={arXiv preprint arXiv:2411.19460},
  year={2024}
}

License

This model is released under the Apache 2.0 License.