MOSS-VL-Base-0708

Introduction

MOSS-VL-Base-0708 is the foundation checkpoint of the MOSS-VL 0708 release, part of the OpenMOSS ecosystem for open visual understanding.

Built through multimodal pretraining only, this checkpoint serves as a high-capacity offline multimodal base model. It provides strong general-purpose visual-language representations across image and video inputs, and is intended primarily as the base model for supervised fine-tuning, alignment, and domain adaptation.

The 0708 release keeps the MOSS-VL cross-attention design and a 256K text context window while refreshing the data and pretraining recipe for stronger offline multimodal foundations.

Specifically, the pretraining pipeline follows four progressive stages:

Stage 1: Vision-language alignment
Stage 2: Large-scale multimodal pretraining
Stage 3: High-quality multimodal pretraining
Stage 4: Annealing and long-context extension

Highlights

Strong foundation model: provides general visual-language representations for image, video, and text inputs.
Native dynamic resolution: processes images and video frames at their original aspect ratios and resolutions.
Native interleaved image and video inputs: supports mixed image/video/text sequences in a unified pipeline.
Open base checkpoint: designed for continued pretraining, supervised fine-tuning, alignment, and domain adaptation.

Model Architecture

MOSS-VL-Base-0708 adopts a cross-attention-based vision-language architecture that decouples visual encoding from language reasoning. The model processes images, videos, and text in a unified pipeline and uses cross-attention layers to connect language tokens with visual representations.

MOSS-VL Architecture

Key configuration details:

Item	Value
Parameters	11B
Tensor type	BF16
Context length	256K
Vision patch size	16
Temporal patch size	1
Default video FPS	1.0
Default max video frames	256

Absolute Timestamps

For video inputs, MOSS-VL injects absolute timestamps alongside sampled frames. This helps the base model learn event order, duration, pacing, and temporal localization instead of relying only on frame order.

Cross-attention RoPE (XRoPE)

MOSS-VL uses Cross-attention Rotary Position Embedding (XRoPE), which maps text tokens and visual patches into a unified three-dimensional coordinate space defined by Time (t), Height (h), and Width (w). This gives the model a consistent positional representation for image and video understanding.

Model Performance

MOSS-VL-Base-0708 is intended as a pretrained foundation checkpoint for offline multimodal understanding and model adaptation. Detailed benchmark tables for the 0708 release will be maintained in the MOSS-VL project resources.

MOSS-VL Offline Benchmark

For the previous public base checkpoint, see MOSS-VL-Base-0408.

Quickstart

Installation

Clone the MOSS-VL repository and install the project requirements:

git clone https://github.com/OpenMOSS/MOSS-VL.git
cd MOSS-VL
conda create -n moss_vl python=3.12 pip -y
conda activate moss_vl
pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt

Load Model

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

checkpoint = "OpenMOSS-Team/MOSS-VL-Base-0708"

processor = AutoProcessor.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    frame_extract_num_threads=1,
)
model = AutoModelForCausalLM.from_pretrained(
    checkpoint,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)

Run Inference

Single-image Inference

image_path = "data/example_image.jpg"

text = model.offline_image_generate(
    processor,
    prompt="",
    image=image_path,
    shortest_edge=4096,
    longest_edge=16777216,
    multi_image_max_pixels=201326592,
    patch_size=16,
    temporal_patch_size=1,
    merge_size=2,
    image_mean=[0.5, 0.5, 0.5],
    image_std=[0.5, 0.5, 0.5],
    max_new_tokens=256,
    temperature=1.0,
    top_k=50,
    top_p=1.0,
    repetition_penalty=1.0,
    do_sample=False,
    vision_chunked_length=64,
)

print(text)

Single-video Inference

video_path = "data/example_video.mp4"

text = model.offline_video_generate(
    processor,
    prompt="",
    video=video_path,
    shortest_edge=4096,
    longest_edge=16777216,
    video_max_pixels=201326592,
    patch_size=16,
    temporal_patch_size=1,
    merge_size=2,
    video_fps=1.0,
    min_frames=1,
    max_frames=256,
    num_extract_threads=4,
    image_mean=[0.5, 0.5, 0.5],
    image_std=[0.5, 0.5, 0.5],
    max_new_tokens=256,
    temperature=1.0,
    top_k=50,
    top_p=1.0,
    repetition_penalty=1.0,
    do_sample=False,
    vision_chunked_length=64,
)

print(text)

Batched Offline Inference

offline_batch_generate accepts independent image/video/text queries. Queries in the same batch should share the same media_kwargs and generate_kwargs.

queries = [
    {
        "images": ["data/sample_a.jpg"],
        "videos": [],
        "generate_kwargs": {
            "temperature": 1.0,
            "top_k": 50,
            "top_p": 1.0,
            "max_new_tokens": 256,
            "repetition_penalty": 1.0,
            "do_sample": False,
        },
    },
    {
        "images": [],
        "videos": ["data/sample_b.mp4"],
        "media_kwargs": {
            "video_fps": 1.0,
            "min_frames": 8,
            "max_frames": 256,
        },
        "generate_kwargs": {
            "temperature": 1.0,
            "top_k": 50,
            "top_p": 1.0,
            "max_new_tokens": 256,
            "repetition_penalty": 1.0,
            "do_sample": False,
        },
    },
]

with torch.no_grad():
    result = model.offline_batch_generate(
        processor,
        queries,
        vision_chunked_length=64,
    )

texts = [item["text"] for item in result["results"]]
print(texts)

Related Checkpoints

Model	Parameters	Context	Usage	Hugging Face
MOSS-VL-Base-0708	11B	256K	Continued pretraining and fine-tuning	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0708
MOSS-VL-Instruct-0708	11B	256K	Offline multimodal instruction following	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0708
MOSS-VL-Base-0408	11B	256K	Previous base checkpoint	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0408
MOSS-VL-Instruct-0408	11B	256K	Previous instruction-tuned checkpoint	https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0408

Limitations and Future Work

MOSS-VL-Base-0708 is a pretrained base checkpoint. It is not instruction-tuned, so applied use cases should generally fine-tune or align it before using it as an assistant-style model.

We are continuing to improve OCR and document understanding, extremely long video understanding, mathematical reasoning, code reasoning, RL post-training, and broader task-specific evaluations for future MOSS-VL releases.

Citation

@misc{moss_vl_2026,
  title         = {{MOSS-VL Technical Report}},
  author        = {OpenMOSS Team},
  year          = {2026},
  howpublished  = {\url{https://github.com/OpenMOSS/MOSS-VL}},
  note          = {GitHub repository}
}