Video-Text-to-Text
Transformers
Safetensors
English
Chinese
mllama
text-generation
multimodal
video
vision-language
custom_code
text-generation-inference
Instructions to use OpenMOSS-Team/moss-video-preview-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use OpenMOSS-Team/moss-video-preview-base with Transformers:
# Load model directly from transformers import AutoProcessor, AutoModelForCausalLM processor = AutoProcessor.from_pretrained("OpenMOSS-Team/moss-video-preview-base", trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained("OpenMOSS-Team/moss-video-preview-base", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -174,54 +174,7 @@ print(processor.decode(output_ids[0], skip_special_tokens=False))
|
|
| 174 |
|
| 175 |
For full environment setup (including optional FlashAttention2 extras), see the top-level repository `README.md`.
|
| 176 |
|
| 177 |
-
## 🛠️ Key Technical Features
|
| 178 |
|
| 179 |
-
- **Native Streaming Architecture**: Supports continuous input and processing of video frames rather than discrete frame sampling.
|
| 180 |
-
- **Unified Position Encoding**: Shared synchronization mechanism for position encoding across both visual and textual modalities.
|
| 181 |
-
- **Efficient Pooling Strategy**: Employs `average` pooling with `stride=4` to balance computational efficiency and feature preservation.
|
| 182 |
-
- **Flash Attention 2**: Full support for FA2 acceleration to optimize memory usage during long-sequence training.
|
| 183 |
-
|
| 184 |
-
## 🏗️ Model Architecture
|
| 185 |
-
|
| 186 |
-
The architecture of MOSS-Video-Preview is designed for maximum scalability and efficiency in processing multimodal temporal data. For more detailed information, please refer to the official repository: [fnlp-vision/MOSS-Video-Preview](https://github.com/fnlp-vision/MOSS-Video-Preview)
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
## 📥 Model Usage
|
| 190 |
-
|
| 191 |
-
For detailed usage instructions, please refer to the official repository: [fnlp-vision/MOSS-Video-Preview](https://github.com/fnlp-vision/MOSS-Video-Preview)
|
| 192 |
-
|
| 193 |
-
## 🛠️ Usage
|
| 194 |
-
|
| 195 |
-
This repository provides inference entry points under `inference/`. For end-to-end usage examples and detailed instructions, please refer to the official repository:
|
| 196 |
-
|
| 197 |
-
- [fnlp-vision/MOSS-Video-Preview](https://github.com/fnlp-vision/MOSS-Video-Preview)
|
| 198 |
-
|
| 199 |
-
## 🚀 Training Stages (for context)
|
| 200 |
-
|
| 201 |
-
Although this directory is a **pretrain-only base checkpoint**, the overall model family is produced via multiple stages:
|
| 202 |
-
|
| 203 |
-
### 1. Stage 1: Vision-Language Alignment (PT1)
|
| 204 |
-
- **Objective**: Establish initial alignment between visual features and the language model, enabling basic visual understanding of video frames.
|
| 205 |
-
- **Configuration**:
|
| 206 |
-
- **Frozen Parameters**: Language Model (LLM) and Vision Tower.
|
| 207 |
-
- **Trainable Parameters**: Vision Projector.
|
| 208 |
-
- **Data**: Large-scale image-text pairs and short video clips.
|
| 209 |
-
- **Key Feature**: Introduces `mllama_add_video_position_encoding` to provide temporal position information for video frames.
|
| 210 |
-
|
| 211 |
-
### 2. Stage 2: Full Spatio-Temporal Pretraining (PT2)
|
| 212 |
-
- **Objective**: Enhance the model's understanding of long videos and complex temporal relationships.
|
| 213 |
-
- **Configuration**:
|
| 214 |
-
- **Method**: Full Parameter Fine-tuning.
|
| 215 |
-
- **Trainable Parameters**: All modules (Vision Tower, Projector, and LLM) are unfrozen.
|
| 216 |
-
- **Data**: Video data with longer durations (supporting 256+ frames).
|
| 217 |
-
- **Key Feature**: Uses `mllama_use_full_attn` to enable full attention mechanisms, improving cross-frame modeling.
|
| 218 |
-
|
| 219 |
-
### 3. Stage 3: Supervised Fine-Tuning (SFT)
|
| 220 |
-
- **Objective**: Enable the model to follow complex instructions for real-time streaming video dialogue and task processing.
|
| 221 |
-
- **Configuration**:
|
| 222 |
-
- **Template**: Uses the `mllama` instruction template.
|
| 223 |
-
- **Data**: High-quality video instruction-following datasets (e.g., real-time description, action recognition, video Q&A).
|
| 224 |
-
- **Optimization**: Optimized for streaming inference to produce coherent textual responses with low latency.
|
| 225 |
|
| 226 |
|
| 227 |
## ⚠️ Notes
|
|
@@ -233,7 +186,7 @@ Although this directory is a **pretrain-only base checkpoint**, the overall mode
|
|
| 233 |
|
| 234 |
```bibtex
|
| 235 |
@misc{moss_video_2026,
|
| 236 |
-
title = {MOSS-Video-Preview: Towards
|
| 237 |
author = {OpenMOSS Team},
|
| 238 |
year = {2026},
|
| 239 |
publisher = {GitHub},
|
|
|
|
| 174 |
|
| 175 |
For full environment setup (including optional FlashAttention2 extras), see the top-level repository `README.md`.
|
| 176 |
|
|
|
|
| 177 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 178 |
|
| 179 |
|
| 180 |
## ⚠️ Notes
|
|
|
|
| 186 |
|
| 187 |
```bibtex
|
| 188 |
@misc{moss_video_2026,
|
| 189 |
+
title = {MOSS-Video-Preview: Towards Real-Time Video Understanding},
|
| 190 |
author = {OpenMOSS Team},
|
| 191 |
year = {2026},
|
| 192 |
publisher = {GitHub},
|