findcard12138 commited on
Commit
c6635f2
·
verified ·
1 Parent(s): d6ed2a9

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +1 -48
README.md CHANGED
@@ -174,54 +174,7 @@ print(processor.decode(output_ids[0], skip_special_tokens=False))
174
 
175
  For full environment setup (including optional FlashAttention2 extras), see the top-level repository `README.md`.
176
 
177
- ## 🛠️ Key Technical Features
178
 
179
- - **Native Streaming Architecture**: Supports continuous input and processing of video frames rather than discrete frame sampling.
180
- - **Unified Position Encoding**: Shared synchronization mechanism for position encoding across both visual and textual modalities.
181
- - **Efficient Pooling Strategy**: Employs `average` pooling with `stride=4` to balance computational efficiency and feature preservation.
182
- - **Flash Attention 2**: Full support for FA2 acceleration to optimize memory usage during long-sequence training.
183
-
184
- ## 🏗️ Model Architecture
185
-
186
- The architecture of MOSS-Video-Preview is designed for maximum scalability and efficiency in processing multimodal temporal data. For more detailed information, please refer to the official repository: [fnlp-vision/MOSS-Video-Preview](https://github.com/fnlp-vision/MOSS-Video-Preview)
187
-
188
-
189
- ## 📥 Model Usage
190
-
191
- For detailed usage instructions, please refer to the official repository: [fnlp-vision/MOSS-Video-Preview](https://github.com/fnlp-vision/MOSS-Video-Preview)
192
-
193
- ## 🛠️ Usage
194
-
195
- This repository provides inference entry points under `inference/`. For end-to-end usage examples and detailed instructions, please refer to the official repository:
196
-
197
- - [fnlp-vision/MOSS-Video-Preview](https://github.com/fnlp-vision/MOSS-Video-Preview)
198
-
199
- ## 🚀 Training Stages (for context)
200
-
201
- Although this directory is a **pretrain-only base checkpoint**, the overall model family is produced via multiple stages:
202
-
203
- ### 1. Stage 1: Vision-Language Alignment (PT1)
204
- - **Objective**: Establish initial alignment between visual features and the language model, enabling basic visual understanding of video frames.
205
- - **Configuration**:
206
- - **Frozen Parameters**: Language Model (LLM) and Vision Tower.
207
- - **Trainable Parameters**: Vision Projector.
208
- - **Data**: Large-scale image-text pairs and short video clips.
209
- - **Key Feature**: Introduces `mllama_add_video_position_encoding` to provide temporal position information for video frames.
210
-
211
- ### 2. Stage 2: Full Spatio-Temporal Pretraining (PT2)
212
- - **Objective**: Enhance the model's understanding of long videos and complex temporal relationships.
213
- - **Configuration**:
214
- - **Method**: Full Parameter Fine-tuning.
215
- - **Trainable Parameters**: All modules (Vision Tower, Projector, and LLM) are unfrozen.
216
- - **Data**: Video data with longer durations (supporting 256+ frames).
217
- - **Key Feature**: Uses `mllama_use_full_attn` to enable full attention mechanisms, improving cross-frame modeling.
218
-
219
- ### 3. Stage 3: Supervised Fine-Tuning (SFT)
220
- - **Objective**: Enable the model to follow complex instructions for real-time streaming video dialogue and task processing.
221
- - **Configuration**:
222
- - **Template**: Uses the `mllama` instruction template.
223
- - **Data**: High-quality video instruction-following datasets (e.g., real-time description, action recognition, video Q&A).
224
- - **Optimization**: Optimized for streaming inference to produce coherent textual responses with low latency.
225
 
226
 
227
  ## ⚠️ Notes
@@ -233,7 +186,7 @@ Although this directory is a **pretrain-only base checkpoint**, the overall mode
233
 
234
  ```bibtex
235
  @misc{moss_video_2026,
236
- title = {MOSS-Video-Preview: Towards Synchronized Streaming Video Understanding},
237
  author = {OpenMOSS Team},
238
  year = {2026},
239
  publisher = {GitHub},
 
174
 
175
  For full environment setup (including optional FlashAttention2 extras), see the top-level repository `README.md`.
176
 
 
177
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
178
 
179
 
180
  ## ⚠️ Notes
 
186
 
187
  ```bibtex
188
  @misc{moss_video_2026,
189
+ title = {MOSS-Video-Preview: Towards Real-Time Video Understanding},
190
  author = {OpenMOSS Team},
191
  year = {2026},
192
  publisher = {GitHub},