README.md

---
license: apache-2.0
language:
- en
- zh
library_name: transformers
pipeline_tag: video-text-to-text
base_model: OpenMOSS-Team/MOSS-VL-Realtime
tags:
- MOSS-VL
- realtime
- streaming
- video-understanding
- FP8
- compressed-tensors
- HQQ
- quantized
- custom_code
---

<p align="center">
  <img src="assets/logo.png" width="300" alt="MOSS-VL"/>
</p>

<p align="center">
  English | <a href="https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime-FP8/blob/main/README_zh.md">中文</a>
</p>

# MOSS-VL-Realtime FP8 Dynamic + Transformers KV8

This is the Transformers FP8 release of
[MOSS-VL-Realtime](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime).
It preserves the timestamp-aware streaming interface for frame-by-frame video
inference. This checkpoint is not an SGLang release.

## Architecture

<p align="center">
  <img src="assets/architecture.png" alt="MOSS-VL architecture" width="100%"/>
</p>

## Quantization profile

| Component | Format |
| --- | --- |
| 252 self-attention/MLP Linear layers in 36 non-cross language layers | compressed-tensors FP8 E4M3 weights with channel-wise static scales and per-token dynamic FP8 input activations |
| 12 cross-attention language layers | BF16 |
| Vision encoder and merger | BF16 |
| Embeddings, norms and `lm_head` | BF16 |
| Transformers KV cache | HQQ INT8, group size 64, BF16 residual length 128 |
| Attention backend | FlashAttention 2 |

`generation_config.json` enables HQQ KV8 automatically. Load the checkpoint
directly and do not pass a second quantization configuration or replace its
generation config with the BF16 source file.

## Quantization benchmark

The final evaluation compares the original BF16 model with all four release
profiles on their corresponding benchmark suites. For this streaming FP8
checkpoint, the scores are 70.66 on OVOBench Avg, 62.93 on StreamingBench Avg,
and 65.50 on OmniMMI PA, compared with 70.86, 62.42, and 66.00 for BF16.

<p align="center">
  <img src="assets/mossvl_quantization_benchmark_comparison_final_v3_en_4k.png" alt="MOSS-VL quantization benchmark comparison" width="100%"/>
</p>

## Hardware requirements

The validated 30-frame streaming test peaked at 25,522 MiB of process VRAM and
26,249 MiB total GPU memory, including a 727 MiB baseline. Use an NVIDIA GPU
with more than 26 GiB available memory, or allow Transformers to shard the
model across multiple GPUs with `device_map="auto"`.

## Environment

### Installation

```bash
git clone https://github.com/OpenMOSS/MOSS-VL.git
cd MOSS-VL

conda create -n moss_vl_quant python=3.12 pip -y
conda activate moss_vl_quant
pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt
pip install -i https://pypi.org/simple \
  compressed-tensors==0.14.0.1 \
  hqq==0.2.8.post1
python -m pip check
```

Validated core versions:

| Package | Version |
| --- | --- |
| Python | 3.12.8 |
| PyTorch | 2.8.0 + CUDA 12.8 |
| Transformers | 4.57.1 |
| Accelerate | 1.12.0 |
| FlashAttention | 2.8.1 |
| compressed-tensors | 0.14.0.1 |
| HQQ | 0.2.8.post1 |

Video decoding also requires FFmpeg in `PATH`.

### Load the model

```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor

checkpoint = "OpenMOSS-Team/MOSS-VL-Realtime-FP8"

processor = AutoProcessor.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    frame_extract_num_threads=1,
)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
model.eval()
```

## Realtime streaming inference

Supply PIL-compatible frames with non-decreasing timestamps. One model
instance supports one active realtime session.

```python
import time
from PIL import Image

session = model.create_realtime_session(
    processor,
    initial_prompt=(
        "Describe important changes in the video as they happen. "
        "Stay silent when there is no meaningful update."
    ),
    frame_queue_size=1,
    max_tokens_per_turn=12,
    max_new_tokens=4096,
    do_sample=False,
)

frame_paths = [
    "data/frame_0001.jpg",
    "data/frame_0002.jpg",
    "data/frame_0003.jpg",
]

try:
    session.start()
    for index, frame_path in enumerate(frame_paths):
        image = Image.open(frame_path).convert("RGB")
        session.push_frame(image, timestamp=float(index))
        while True:
            chunk = session.poll_output(timeout=0.0)
            if chunk is None:
                break
            print(chunk, end="", flush=True)
        time.sleep(1.0)
finally:
    session.close()
```

The model may emit control tokens such as `<|silence|>`, `<|round_start|>`,
and `<|round_end|>`; applications should filter or render them according to
their protocol.

## Validated reproduction

The fixed validation used a Xinjiang aerial video at 1 FPS with 30 timestamped
frames. It completed 30/30 frames and produced a relevant Chinese tour-guide
description.

```bash
source /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/venv/bin/activate

/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/venv/bin/python \
  /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/benchmark_mossvl_quant.py \
  --label streaming_fp8_reproduce \
  --checkpoint /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/final_release/quant/MOSS-VL-0708-Streaming-FP8-Dynamic-KV8-HQQ \
  --gpu 0 \
  --frames 30 \
  --attention-backend flash_attention_2 \
  --timeout 300
```

Full inputs, commands and raw logs:

```text
/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/transformers_validation_20260811
```

## Configuration files

- `config.json`: model and FP8 weight/activation configuration.
- `generation_config.json`: Transformers HQQ KV8 configuration.
- `recipe.yaml`: compressed-tensors quantization recipe.
- `modeling_moss_vl.py`: checkpoint-local streaming and quantized-cache code.