OpenMOSS-Team/MOSS-VL-Instruct-0708-FP8

English | 中文

MOSS-VL-Instruct-0708 FP8 Dynamic + Transformers KV8

This is the 24 GiB quantized release of MOSS-VL-Instruct-0708. The same checkpoint directory is supported by both the standard Transformers inference path and the native SGLang MOSS-VL backend.

Quantization profile

Component	Format
Most language layers	FP8 weights + per-token dynamic FP8 input activations
Cross-attention, vision modules and lm_head	BF16
Transformers KV cache	HQQ INT8
SGLang KV cache	Native SGLang BF16 cache

The FP8 weights are shared across both engines. Transformers reads the HQQ KV8 configuration from generation_config.json; SGLang loads the same weights with model_impl=sglang and uses its own cache implementation.

Quantization benchmark

Across the selected benchmarks, the quantized models remain close to their non-quantized BF16 counterparts, showing that overall model quality is largely preserved after quantization.

Hardware requirements

The model is designed to run with the Transformers inference path on a single NVIDIA GPU with 24 GB of VRAM. SGLang memory requirements depend on its server configuration and KV cache allocation.

Transformers environment

Installation

Start from the standard MOSS-VL release environment, then install the FP8 checkpoint loader and HQQ cache backend:

git clone https://github.com/OpenMOSS/MOSS-VL.git
cd MOSS-VL

conda create -n moss_vl_quant python=3.12 pip -y
conda activate moss_vl_quant
pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt
pip install -i https://pypi.org/simple \
  compressed-tensors==0.14.0 \
  hqq==0.2.8.post1
python -m pip check

Core versions used by the standard Transformers path:

Package	Version
Python	3.12
PyTorch	2.8.0 + CUDA 12.8
Transformers	4.57.1
Accelerate	1.12.0
FlashAttention	2.8.1
compressed-tensors checkpoint format	0.14.0
HQQ	0.2.8.post1

Video decoding also requires FFmpeg to be available in PATH.

Load the model

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

checkpoint = "/path/to/MOSS-VL-0708-Instruct-FP8-Dynamic-KV8-HQQ"

processor = AutoProcessor.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    frame_extract_num_threads=1,
)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
model.eval()

generation_config.json enables HQQ KV8 automatically. Keep that file beside the checkpoint and do not pass a conflicting cache configuration.

Image inference

text = model.offline_image_generate(
    processor,
    prompt="Describe this image.",
    image="data/example_image.jpg",
    shortest_edge=4096,
    longest_edge=16777216,
    multi_image_max_pixels=201326592,
    patch_size=16,
    temporal_patch_size=1,
    merge_size=2,
    image_mean=[0.5, 0.5, 0.5],
    image_std=[0.5, 0.5, 0.5],
    max_new_tokens=256,
    temperature=1.0,
    top_k=50,
    top_p=1.0,
    repetition_penalty=1.0,
    do_sample=False,
    vision_chunked_length=64,
)
print(text)

Video inference

text = model.offline_video_generate(
    processor,
    prompt="Describe this video.",
    video="data/example_video.mp4",
    shortest_edge=4096,
    longest_edge=16777216,
    video_max_pixels=201326592,
    patch_size=16,
    temporal_patch_size=1,
    merge_size=2,
    video_fps=1.0,
    min_frames=1,
    max_frames=256,
    num_extract_threads=4,
    image_mean=[0.5, 0.5, 0.5],
    image_std=[0.5, 0.5, 0.5],
    max_new_tokens=256,
    temperature=1.0,
    top_k=50,
    top_p=1.0,
    repetition_penalty=1.0,
    do_sample=False,
    vision_chunked_length=64,
)
print(text)

Repository inference runner

The standard repository runner accepts JSON/JSONL queries with image or video content and loads this directory directly:

python inference/run_inference.py \
  --checkpoint /path/to/MOSS-VL-0708-Instruct-FP8-Dynamic-KV8-HQQ \
  --mode offline \
  --input inference/video_queries.json \
  --output video_results.json

SGLang environment

Use a separate environment for SGLang. The native backend in the mossvl_sglang release directory pins:

Package	Version
Python	3.12.8
SGLang	0.5.11
sglang-kernel	0.4.2
PyTorch	2.11.0 + CUDA 13.0
Transformers	5.6.0
compressed-tensors	0.17.1

Create the environment from its complete lock file:

cd /path/to/mossvl_sglang
env -u INDEX_URL -u PIP_INDEX_URL conda create -y -n mossvl-sglang-0511 \
  --override-channels \
  -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main \
  python=3.12.8 pip=24.2
conda activate mossvl-sglang-0511
env -u INDEX_URL -u PIP_INDEX_URL python -m pip install -r requirements.txt
python -m pip check

Start the OpenAI-compatible service with the quantized directory as the model path:

conda activate mossvl-sglang-0511
cd /path/to/mossvl_sglang
GPU_IDS=0 TP_SIZE=1 MEM_FRACTION_STATIC=0.35 \
  ./start_sglang_moss_vl.sh \
  --model-path /path/to/MOSS-VL-0708-Instruct-FP8-Dynamic-KV8-HQQ

The service listens on http://127.0.0.1:30000 by default. Keep MODEL_IMPL=sglang; do not force the Transformers remote implementation. SGLang intentionally uses its native BF16 KV cache rather than the HQQ cache in generation_config.json.

Configuration files

• config.json: model and FP8 weight/activation configuration.
• generation_config.json: Transformers HQQ KV8 configuration.
• modeling_moss_vl.py: checkpoint-local cross-attention/QuantizedCache code.