| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: video-text-to-text |
| tags: |
| - moss_vl |
| - feature-extraction |
| - Base |
| - Video-Understanding |
| - Image-Understanding |
| - MOSS-VL |
| - OpenMOSS |
| - multimodal |
| - video |
| - vision-language |
| - custom_code |
| --- |
| |
| <p align="center"> |
| <img src="assets/logo.png" width="300" alt="MOSS-VL"/> |
| </p> |
|
|
| # MOSS-VL-Base-0708 |
|
|
| ## Introduction |
|
|
| MOSS-VL-Base-0708 is the foundation checkpoint of the MOSS-VL 0708 release, part of the OpenMOSS ecosystem for open visual understanding. |
|
|
| Built through multimodal pretraining only, this checkpoint serves as a high-capacity offline multimodal base model. It provides strong general-purpose visual-language representations across image and video inputs, and is intended primarily as the base model for supervised fine-tuning, alignment, and domain adaptation. |
|
|
| The 0708 release keeps the MOSS-VL cross-attention design and a 256K text context window while refreshing the data and pretraining recipe for stronger offline multimodal foundations. |
|
|
| Specifically, the pretraining pipeline follows four progressive stages: |
|
|
| - Stage 1: Vision-language alignment |
| - Stage 2: Large-scale multimodal pretraining |
| - Stage 3: High-quality multimodal pretraining |
| - Stage 4: Annealing and long-context extension |
|
|
| ## Highlights |
|
|
| - Strong foundation model: provides general visual-language representations for image, video, and text inputs. |
| - Native dynamic resolution: processes images and video frames at their original aspect ratios and resolutions. |
| - Native interleaved image and video inputs: supports mixed image/video/text sequences in a unified pipeline. |
| - Open base checkpoint: designed for continued pretraining, supervised fine-tuning, alignment, and domain adaptation. |
|
|
| ## Model Architecture |
|
|
| MOSS-VL-Base-0708 adopts a cross-attention-based vision-language architecture that decouples visual encoding from language reasoning. The model processes images, videos, and text in a unified pipeline and uses cross-attention layers to connect language tokens with visual representations. |
|
|
| <p align="center"> |
| <img src="assets/architecture.png" alt="MOSS-VL Architecture" width="100%"/> |
| </p> |
|
|
| Key configuration details: |
|
|
| | Item | Value | |
| | --- | --- | |
| | Parameters | 11B | |
| | Tensor type | BF16 | |
| | Context length | 256K | |
| | Vision patch size | 16 | |
| | Temporal patch size | 1 | |
| | Default video FPS | 1.0 | |
| | Default max video frames | 256 | |
|
|
| ## Absolute Timestamps |
|
|
| For video inputs, MOSS-VL injects absolute timestamps alongside sampled frames. This helps the base model learn event order, duration, pacing, and temporal localization instead of relying only on frame order. |
|
|
| ## Cross-attention RoPE (XRoPE) |
|
|
| MOSS-VL uses Cross-attention Rotary Position Embedding (XRoPE), which maps text tokens and visual patches into a unified three-dimensional coordinate space defined by Time (t), Height (h), and Width (w). This gives the model a consistent positional representation for image and video understanding. |
|
|
| ## Model Performance |
|
|
| MOSS-VL-Base-0708 is intended as a pretrained foundation checkpoint for offline multimodal understanding and model adaptation. Detailed benchmark tables for the 0708 release will be maintained in the MOSS-VL project resources. |
|
|
| <p align="center"> |
| <img src="assets/benchmark-offline.png" alt="MOSS-VL Offline Benchmark" width="100%"/> |
| </p> |
|
|
| For the previous public base checkpoint, see [MOSS-VL-Base-0408](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0408). |
|
|
| ## Quickstart |
|
|
| ### Installation |
|
|
| Clone the MOSS-VL repository and install the project requirements: |
|
|
| ```bash |
| git clone https://github.com/OpenMOSS/MOSS-VL.git |
| cd MOSS-VL |
| conda create -n moss_vl python=3.12 pip -y |
| conda activate moss_vl |
| pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt |
| ``` |
|
|
| ### Load Model |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoProcessor |
| |
| checkpoint = "OpenMOSS-Team/MOSS-VL-Base-0708" |
| |
| processor = AutoProcessor.from_pretrained( |
| checkpoint, |
| trust_remote_code=True, |
| frame_extract_num_threads=1, |
| ) |
| model = AutoModelForCausalLM.from_pretrained( |
| checkpoint, |
| trust_remote_code=True, |
| device_map="auto", |
| torch_dtype=torch.bfloat16, |
| attn_implementation="flash_attention_2", |
| ) |
| ``` |
|
|
| ### Run Inference |
|
|
| <details> |
| <summary><b>Single-image Inference</b></summary> |
|
|
| ```python |
| image_path = "data/example_image.jpg" |
| |
| text = model.offline_image_generate( |
| processor, |
| prompt="", |
| image=image_path, |
| shortest_edge=4096, |
| longest_edge=16777216, |
| multi_image_max_pixels=201326592, |
| patch_size=16, |
| temporal_patch_size=1, |
| merge_size=2, |
| image_mean=[0.5, 0.5, 0.5], |
| image_std=[0.5, 0.5, 0.5], |
| max_new_tokens=256, |
| temperature=1.0, |
| top_k=50, |
| top_p=1.0, |
| repetition_penalty=1.0, |
| do_sample=False, |
| vision_chunked_length=64, |
| ) |
| |
| print(text) |
| ``` |
|
|
| </details> |
|
|
| <details> |
| <summary><b>Single-video Inference</b></summary> |
|
|
| ```python |
| video_path = "data/example_video.mp4" |
| |
| text = model.offline_video_generate( |
| processor, |
| prompt="", |
| video=video_path, |
| shortest_edge=4096, |
| longest_edge=16777216, |
| video_max_pixels=201326592, |
| patch_size=16, |
| temporal_patch_size=1, |
| merge_size=2, |
| video_fps=1.0, |
| min_frames=1, |
| max_frames=256, |
| num_extract_threads=4, |
| image_mean=[0.5, 0.5, 0.5], |
| image_std=[0.5, 0.5, 0.5], |
| max_new_tokens=256, |
| temperature=1.0, |
| top_k=50, |
| top_p=1.0, |
| repetition_penalty=1.0, |
| do_sample=False, |
| vision_chunked_length=64, |
| ) |
| |
| print(text) |
| ``` |
|
|
| </details> |
|
|
| <details> |
| <summary><b>Batched Offline Inference</b></summary> |
|
|
| `offline_batch_generate` accepts independent image/video/text queries. Queries in the same batch should share the same `media_kwargs` and `generate_kwargs`. |
|
|
| ```python |
| queries = [ |
| { |
| "images": ["data/sample_a.jpg"], |
| "videos": [], |
| "generate_kwargs": { |
| "temperature": 1.0, |
| "top_k": 50, |
| "top_p": 1.0, |
| "max_new_tokens": 256, |
| "repetition_penalty": 1.0, |
| "do_sample": False, |
| }, |
| }, |
| { |
| "images": [], |
| "videos": ["data/sample_b.mp4"], |
| "media_kwargs": { |
| "video_fps": 1.0, |
| "min_frames": 8, |
| "max_frames": 256, |
| }, |
| "generate_kwargs": { |
| "temperature": 1.0, |
| "top_k": 50, |
| "top_p": 1.0, |
| "max_new_tokens": 256, |
| "repetition_penalty": 1.0, |
| "do_sample": False, |
| }, |
| }, |
| ] |
| |
| with torch.no_grad(): |
| result = model.offline_batch_generate( |
| processor, |
| queries, |
| vision_chunked_length=64, |
| ) |
| |
| texts = [item["text"] for item in result["results"]] |
| print(texts) |
| ``` |
|
|
| </details> |
|
|
| ## Related Checkpoints |
|
|
| | Model | Parameters | Context | Usage | Hugging Face | |
| | --- | ---: | ---: | --- | --- | |
| | MOSS-VL-Realtime | 11B | 256K | Realtime streaming video interaction | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime | |
| | MOSS-VL-Instruct | 11B | 256K | Offline multimodal instruction following | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0708 | |
| | MOSS-VL-Base | 11B | 256K | Continued pretraining and fine-tuning | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0708 | |
| | MOSS-VL-Instruct-0408 | 11B | 256K | Previous instruction-tuned checkpoint | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0408 | |
| | MOSS-VL-Base-0408 | 11B | 256K | Previous base checkpoint | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0408 | |
|
|
| ## Limitations and Future Work |
|
|
| MOSS-VL-Base-0708 is a pretrained base checkpoint. It is not instruction-tuned, so applied use cases should generally fine-tune or align it before using it as an assistant-style model. |
|
|
| We are continuing to improve OCR and document understanding, extremely long video understanding, mathematical reasoning, code reasoning, RL post-training, and broader task-specific evaluations for future MOSS-VL releases. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{moss_vl_2026, |
| title = {{MOSS-VL Technical Report}}, |
| author = {OpenMOSS Team}, |
| year = {2026}, |
| howpublished = {\url{https://github.com/OpenMOSS/MOSS-VL}}, |
| note = {GitHub repository} |
| } |
| ``` |
|
|
|
|
