---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: video-text-to-text
tags:
- moss_vl
- feature-extraction
- Base
- Video-Understanding
- Image-Understanding
- MOSS-VL
- OpenMOSS
- multimodal
- video
- vision-language
- custom_code
---
# MOSS-VL-Base-0708
## Introduction
MOSS-VL-Base-0708 is the foundation checkpoint of the MOSS-VL 0708 release, part of the OpenMOSS ecosystem for open visual understanding.
Built through multimodal pretraining only, this checkpoint serves as a high-capacity offline multimodal base model. It provides strong general-purpose visual-language representations across image and video inputs, and is intended primarily as the base model for supervised fine-tuning, alignment, and domain adaptation.
The 0708 release keeps the MOSS-VL cross-attention design and a 256K text context window while refreshing the data and pretraining recipe for stronger offline multimodal foundations.
Specifically, the pretraining pipeline follows four progressive stages:
- Stage 1: Vision-language alignment
- Stage 2: Large-scale multimodal pretraining
- Stage 3: High-quality multimodal pretraining
- Stage 4: Annealing and long-context extension
## Highlights
- Strong foundation model: provides general visual-language representations for image, video, and text inputs.
- Native dynamic resolution: processes images and video frames at their original aspect ratios and resolutions.
- Native interleaved image and video inputs: supports mixed image/video/text sequences in a unified pipeline.
- Open base checkpoint: designed for continued pretraining, supervised fine-tuning, alignment, and domain adaptation.
## Model Architecture
MOSS-VL-Base-0708 adopts a cross-attention-based vision-language architecture that decouples visual encoding from language reasoning. The model processes images, videos, and text in a unified pipeline and uses cross-attention layers to connect language tokens with visual representations.
Key configuration details:
| Item | Value |
| --- | --- |
| Parameters | 11B |
| Tensor type | BF16 |
| Context length | 256K |
| Vision patch size | 16 |
| Temporal patch size | 1 |
| Default video FPS | 1.0 |
| Default max video frames | 256 |
## Absolute Timestamps
For video inputs, MOSS-VL injects absolute timestamps alongside sampled frames. This helps the base model learn event order, duration, pacing, and temporal localization instead of relying only on frame order.
## Cross-attention RoPE (XRoPE)
MOSS-VL uses Cross-attention Rotary Position Embedding (XRoPE), which maps text tokens and visual patches into a unified three-dimensional coordinate space defined by Time (t), Height (h), and Width (w). This gives the model a consistent positional representation for image and video understanding.
## Model Performance
MOSS-VL-Base-0708 is intended as a pretrained foundation checkpoint for offline multimodal understanding and model adaptation. Detailed benchmark tables for the 0708 release will be maintained in the MOSS-VL project resources.
For the previous public base checkpoint, see [MOSS-VL-Base-0408](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0408).
## Quickstart
### Installation
Clone the MOSS-VL repository and install the project requirements:
```bash
git clone https://github.com/OpenMOSS/MOSS-VL.git
cd MOSS-VL
conda create -n moss_vl python=3.12 pip -y
conda activate moss_vl
pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt
```
### Load Model
```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
checkpoint = "OpenMOSS-Team/MOSS-VL-Base-0708"
processor = AutoProcessor.from_pretrained(
checkpoint,
trust_remote_code=True,
frame_extract_num_threads=1,
)
model = AutoModelForCausalLM.from_pretrained(
checkpoint,
trust_remote_code=True,
device_map="auto",
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
)
```
### Run Inference
Single-image Inference
```python
image_path = "data/example_image.jpg"
text = model.offline_image_generate(
processor,
prompt="",
image=image_path,
shortest_edge=4096,
longest_edge=16777216,
multi_image_max_pixels=201326592,
patch_size=16,
temporal_patch_size=1,
merge_size=2,
image_mean=[0.5, 0.5, 0.5],
image_std=[0.5, 0.5, 0.5],
max_new_tokens=256,
temperature=1.0,
top_k=50,
top_p=1.0,
repetition_penalty=1.0,
do_sample=False,
vision_chunked_length=64,
)
print(text)
```
Single-video Inference
```python
video_path = "data/example_video.mp4"
text = model.offline_video_generate(
processor,
prompt="",
video=video_path,
shortest_edge=4096,
longest_edge=16777216,
video_max_pixels=201326592,
patch_size=16,
temporal_patch_size=1,
merge_size=2,
video_fps=1.0,
min_frames=1,
max_frames=256,
num_extract_threads=4,
image_mean=[0.5, 0.5, 0.5],
image_std=[0.5, 0.5, 0.5],
max_new_tokens=256,
temperature=1.0,
top_k=50,
top_p=1.0,
repetition_penalty=1.0,
do_sample=False,
vision_chunked_length=64,
)
print(text)
```
Batched Offline Inference
`offline_batch_generate` accepts independent image/video/text queries. Queries in the same batch should share the same `media_kwargs` and `generate_kwargs`.
```python
queries = [
{
"images": ["data/sample_a.jpg"],
"videos": [],
"generate_kwargs": {
"temperature": 1.0,
"top_k": 50,
"top_p": 1.0,
"max_new_tokens": 256,
"repetition_penalty": 1.0,
"do_sample": False,
},
},
{
"images": [],
"videos": ["data/sample_b.mp4"],
"media_kwargs": {
"video_fps": 1.0,
"min_frames": 8,
"max_frames": 256,
},
"generate_kwargs": {
"temperature": 1.0,
"top_k": 50,
"top_p": 1.0,
"max_new_tokens": 256,
"repetition_penalty": 1.0,
"do_sample": False,
},
},
]
with torch.no_grad():
result = model.offline_batch_generate(
processor,
queries,
vision_chunked_length=64,
)
texts = [item["text"] for item in result["results"]]
print(texts)
```
## Related Checkpoints
| Model | Parameters | Context | Usage | Hugging Face |
| --- | ---: | ---: | --- | --- |
| MOSS-VL-Realtime | 11B | 256K | Realtime streaming video interaction | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime |
| MOSS-VL-Instruct | 11B | 256K | Offline multimodal instruction following | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0708 |
| MOSS-VL-Base | 11B | 256K | Continued pretraining and fine-tuning | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0708 |
| MOSS-VL-Instruct-0408 | 11B | 256K | Previous instruction-tuned checkpoint | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0408 |
| MOSS-VL-Base-0408 | 11B | 256K | Previous base checkpoint | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0408 |
## Limitations and Future Work
MOSS-VL-Base-0708 is a pretrained base checkpoint. It is not instruction-tuned, so applied use cases should generally fine-tune or align it before using it as an assistant-style model.
We are continuing to improve OCR and document understanding, extremely long video understanding, mathematical reasoning, code reasoning, RL post-training, and broader task-specific evaluations for future MOSS-VL releases.
## Citation
```bibtex
@misc{moss_vl_2026,
title = {{MOSS-VL Technical Report}},
author = {OpenMOSS Team},
year = {2026},
howpublished = {\url{https://github.com/OpenMOSS/MOSS-VL}},
note = {GitHub repository}
}
```