metadata
license: apache-2.0
language:
- en
- zh
library_name: transformers
pipeline_tag: video-text-to-text
base_model: OpenMOSS-Team/MOSS-VL-Instruct-0708
tags:
- MOSS-VL
- image-understanding
- video-understanding
- bitsandbytes
- NF4
- quantized
- custom_code
English | 中文
MOSS-VL-Instruct-0708 W4A16 NF4
这是 MOSS-VL-Instruct-0708 的 Transformers NF4 发布版本，支持 MOSS-VL 标准离线图片和视频推理。 该 checkpoint 不是 SGLang 发布版本。
模型架构
量化方法
| 模块 | 格式 |
|---|---|
| 第 4-43 层中的 240 个可量化 Linear | bitsandbytes NF4 weight-only，启用 double quantization，BF16 计算 |
| 前四层和后四层语言模型层 | BF16 |
| Cross-attention projection | BF16 |
| 视觉编码器和 merger | BF16 |
Embedding、norm 和 lm_head |
BF16 |
| Transformers KV Cache | BF16 |
| Attention 后端 | FlashAttention 2 |
checkpoint 已包含 bitsandbytes 配置，加载时不要再传入第二份运行时量化配置。
该版本没有启用 HQQ KV8，generation_config.json 使用标准 BF16 KV Cache。
量化 Benchmark
最终测评在各自对应的 benchmark 上对比原始 BF16 模型与四个量化发布配置。 该离线 NF4 checkpoint 的 DocVQA、VideoMME、MLVU_dev 分别为 89.53、 67.30、75.86；三个 TimeLens 子集为 51.00、48.17、59.33，VSIBench 为 61.76。
硬件要求
图片实测进程峰值显存为 12,494 MiB；1 FPS、最多 32 帧的视频实测峰值为 16,708 MiB。单张 24 GB 显存的 NVIDIA GPU 可以运行已验证配置。
环境安装
git clone https://github.com/OpenMOSS/MOSS-VL.git
cd MOSS-VL
conda create -n moss_vl_quant python=3.12 pip -y
conda activate moss_vl_quant
pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt
pip install -i https://pypi.org/simple bitsandbytes==0.49.2
python -m pip check
已验证的主要环境版本：
| 依赖 | 版本 |
|---|---|
| Python | 3.12.8 |
| PyTorch | 2.8.0 + CUDA 12.8 |
| Transformers | 4.57.1 |
| Accelerate | 1.12.0 |
| FlashAttention | 2.8.1 |
| bitsandbytes | 0.49.2 |
视频解码还需要确保 FFmpeg 已加入 PATH。
加载模型
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
checkpoint = "OpenMOSS-Team/MOSS-VL-Instruct-0708-NF4"
processor = AutoProcessor.from_pretrained(
checkpoint,
trust_remote_code=True,
frame_extract_num_threads=1,
)
model = AutoModelForCausalLM.from_pretrained(
checkpoint,
trust_remote_code=True,
device_map="auto",
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
)
model.eval()
图片推理
text = model.offline_image_generate(
processor,
prompt="请描述这张图片。",
image="data/example_image.jpg",
max_new_tokens=256,
do_sample=False,
vision_chunked_length=64,
)
print(text)
视频推理
text = model.offline_video_generate(
processor,
prompt="请描述这段视频。",
video="data/example_video.mp4",
video_fps=1.0,
min_frames=1,
max_frames=32,
max_new_tokens=256,
do_sample=False,
vision_chunked_length=64,
)
print(text)
完整复测命令
官方 runner 已通过收据图片和 1 FPS 星巴克视频测试：
source /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/venv/bin/activate
/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/venv/bin/python \
/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/final_release/mossvl-github/MOSS-VL/inference/run_inference.py \
--checkpoint /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/final_release/quant/MOSS-VL-0708-Offline-NF4-Keep4-KV16 \
--mode image \
--input /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/transformers_validation_20260811/inputs/offline_image.json \
--output /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/transformers_validation_20260811/results/offline_nf4_image_output.json \
--timeout-seconds 300
/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/venv/bin/python \
/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/final_release/mossvl-github/MOSS-VL/inference/run_inference.py \
--checkpoint /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/final_release/quant/MOSS-VL-0708-Offline-NF4-Keep4-KV16 \
--mode video \
--input /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/transformers_validation_20260811/inputs/offline_video.json \
--output /inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/transformers_validation_20260811/results/offline_nf4_video_output.json \
--timeout-seconds 300
完整输入、命令和原始结果：
/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/transformers_validation_20260811