README_zh.md

---
license: apache-2.0
language:
- en
- zh
library_name: transformers
pipeline_tag: video-text-to-text
base_model: OpenMOSS-Team/MOSS-VL-Instruct-0708
tags:
- MOSS-VL
- image-understanding
- video-understanding
- bitsandbytes
- NF4
- quantized
- custom_code
---

<p align="center">
  <img src="assets/logo.png" width="300" alt="MOSS-VL"/>
</p>

<p align="center">
  <a href="https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0708-NF4/blob/main/README.md">English</a> | 中文
</p>

# MOSS-VL-Instruct-0708 W4A16 NF4

这是 [MOSS-VL-Instruct-0708](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0708)
的 Transformers NF4 发布版本，支持 MOSS-VL 标准离线图片和视频推理。
该 checkpoint 不是 SGLang 发布版本。

## 模型架构

<p align="center">
  <img src="assets/architecture.png" alt="MOSS-VL 模型架构" width="100%"/>
</p>

## 量化方法

| 模块 | 格式 |
| --- | --- |
| 第 4-43 层中的 240 个可量化 Linear | bitsandbytes NF4 weight-only，启用 double quantization，BF16 计算 |
| 前四层和后四层语言模型层 | BF16 |
| Cross-attention projection | BF16 |
| 视觉编码器和 merger | BF16 |
| Embedding、norm 和 `lm_head` | BF16 |
| Transformers KV Cache | BF16 |
| Attention 后端 | FlashAttention 2 |

checkpoint 已包含 bitsandbytes 配置，加载时不要再传入第二份运行时量化配置。
该版本没有启用 HQQ KV8，`generation_config.json` 使用标准 BF16 KV Cache。

## 量化 Benchmark

最终测评在各自对应的 benchmark 上对比原始 BF16 模型与四个量化发布配置。
该离线 NF4 checkpoint 的 DocVQA、VideoMME、MLVU_dev 分别为 89.53、
67.30、75.86；三个 TimeLens 子集为 51.00、48.17、59.33，VSIBench 为 61.76。

<p align="center">
  <img src="assets/mossvl_quantization_benchmark_comparison_final_v3_zh_4k.png" alt="MOSS-VL 量化配置 benchmark 对比" width="100%"/>
</p>

## 硬件要求

图片实测进程峰值显存为 12,494 MiB；1 FPS、最多 32 帧的视频实测峰值为
16,708 MiB。单张 24 GB 显存的 NVIDIA GPU 可以运行已验证配置。

## 环境安装

```bash
git clone https://github.com/OpenMOSS/MOSS-VL.git
cd MOSS-VL

conda create -n moss_vl_quant python=3.12 pip -y
conda activate moss_vl_quant
pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt
pip install -i https://pypi.org/simple bitsandbytes==0.49.2
python -m pip check
```

已验证的主要环境版本：

| 依赖 | 版本 |
| --- | --- |
| Python | 3.12.8 |
| PyTorch | 2.8.0 + CUDA 12.8 |
| Transformers | 4.57.1 |
| Accelerate | 1.12.0 |
| FlashAttention | 2.8.1 |
| bitsandbytes | 0.49.2 |

视频解码还需要确保 FFmpeg 已加入 `PATH`。

## 加载模型

```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor

checkpoint = "OpenMOSS-Team/MOSS-VL-Instruct-0708-NF4"

processor = AutoProcessor.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    frame_extract_num_threads=1,
)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
model.eval()
```

## 图片推理

```python
text = model.offline_image_generate(
    processor,
    prompt="请描述这张图片。",
    image="data/example_image.jpg",
    max_new_tokens=256,
    do_sample=False,
    vision_chunked_length=64,
)
print(text)
```

## 视频推理

```python
text = model.offline_video_generate(
    processor,
    prompt="请描述这段视频。",
    video="data/example_video.mp4",
    video_fps=1.0,
    min_frames=1,
    max_frames=32,
    max_new_tokens=256,
    do_sample=False,
    vision_chunked_length=64,
)
print(text)
```

## 完整复测命令

官方 runner 已通过收据图片和 1 FPS 星巴克视频测试：

```bash
source /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/venv/bin/activate

/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/venv/bin/python \
  /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/final_release/mossvl-github/MOSS-VL/inference/run_inference.py \
  --checkpoint /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/final_release/quant/MOSS-VL-0708-Offline-NF4-Keep4-KV16 \
  --mode image \
  --input /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/transformers_validation_20260811/inputs/offline_image.json \
  --output /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/transformers_validation_20260811/results/offline_nf4_image_output.json \
  --timeout-seconds 300

/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/venv/bin/python \
  /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/final_release/mossvl-github/MOSS-VL/inference/run_inference.py \
  --checkpoint /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/final_release/quant/MOSS-VL-0708-Offline-NF4-Keep4-KV16 \
  --mode video \
  --input /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/transformers_validation_20260811/inputs/offline_video.json \
  --output /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/transformers_validation_20260811/results/offline_nf4_video_output.json \
  --timeout-seconds 300
```

完整输入、命令和原始结果：

```text
/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/transformers_validation_20260811
```