Upload 2 files

README.md

@@ -1,233 +1,186 @@
 ---
+pipeline_tag: image-text-to-text
 language:
-- en
-- zh
-library_name: diffusers
-license: mit
-pipeline_tag: text-to-video
+- multilingual
 tags:
-- transformers
-- diffusers
-- image-to-video
-- video-continuation
+- deepseek
+- vision-language
+- ocr
+- custom_code
+license: mit
+library_name: transformers
 ---
-
-# LongCat-Video
-
 <div align="center">
-  <img src="assets/longcat_logo.svg" width="45%" alt="LongCat-Video" />
+  <img src="https://github.com/deepseek-ai/DeepSeek-V2/blob/main/figures/logo.svg?raw=true" width="60%" alt="DeepSeek AI" />
 </div>
 <hr>
+<div align="center">
+  <a href="https://www.deepseek.com/" target="_blank">
+    <img alt="Homepage" src="https://github.com/deepseek-ai/DeepSeek-V2/blob/main/figures/badge.svg?raw=true" />
+  </a>
+  <a href="https://huggingface.co/deepseek-ai/DeepSeek-OCR" target="_blank">
+    <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DeepSeek%20AI-ffc107?color=ffc107&logoColor=white" />
+  </a>
 
-<div align="center" style="line-height: 1;">
-  <a href='https://meituan-longcat.github.io/LongCat-Video/'><img src='https://img.shields.io/badge/Project-Page-green'></a>
-  <a href='https://huggingface.co/papers/2510.22200'><img src='https://img.shields.io/badge/Paper-HuggingFace-red'></a>
-  <a href='https://huggingface.co/meituan-longcat/LongCat-Video'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue'></a>
-</div>
-
-<div align="center" style="line-height: 1;">
-  <a href='https://github.com/meituan-longcat/LongCat-Flash-Chat/blob/main/figures/wechat_official_accounts.png'><img src='https://img.shields.io/badge/WeChat-LongCat-brightgreen?logo=wechat&logoColor=white'></a>  
-  <a href='https://x.com/Meituan_LongCat'><img src='https://img.shields.io/badge/Twitter-LongCat-white?logo=x&logoColor=white'></a>
-</div>
-
-<div align="center" style="line-height: 1;">
-  <a href='https://huggingface.co/meituan-longcat/LongCat-Video/blob/main/LICENSE'><img src='https://img.shields.io/badge/License-MIT-f5de53?&color=f5de53'></a>
 </div>
 
-## Model Introduction
-We introduce LongCat-Video, a foundational video generation model with 13.6B parameters, delivering strong performance across *Text-to-Video*, *Image-to-Video*, and *Video-Continuation* generation tasks. It particularly excels in efficient and high-quality long video generation, representing our first step toward world models.
-
-### Key Features
-- 🌟 **Unified architecture for multiple tasks**: LongCat-Video unifies *Text-to-Video*, *Image-to-Video*, and *Video-Continuation* tasks within a single video generation framework. It natively supports all these tasks with a single model and consistently delivers strong performance across each individual task.
-- 🌟 **Long video generation**: LongCat-Video is natively pretrained on *Video-Continuation* tasks, enabling it to produce minutes-long videos without color drifting or quality degradation.
-- 🌟 **Efficient inference**: LongCat-Video generates $720p$, $30fps$ videos within minutes by employing a coarse-to-fine generation strategy along both the temporal and spatial axes. Block Sparse Attention further enhances efficiency, particularly at high resolutions
-- 🌟 **Strong performance with multi-reward RLHF**: Powered by multi-reward Group Relative Policy Optimization (GRPO), comprehensive evaluations on both internal and public benchmarks demonstrate that LongCat-Video achieves performance comparable to leading open-source video generation models as well as the latest commercial solutions.
-
-For more detail, please refer to the comprehensive [***LongCat-Video Technical Report***](https://huggingface.co/papers/2510.22200).
+<div align="center">
 
-## 🎥 Teaser Video
+  <a href="https://discord.gg/Tc7c45Zzu5" target="_blank">
+    <img alt="Discord" src="https://img.shields.io/badge/Discord-DeepSeek%20AI-7289da?logo=discord&logoColor=white&color=7289da" />
+  </a>
+  <a href="https://twitter.com/deepseek_ai" target="_blank">
+    <img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-deepseek_ai-white?logo=x&logoColor=white" />
+  </a>
 
-<div align="center">
-  <video src="https://github.com/user-attachments/assets/00fa63f0-9c4e-461a-a79e-c662ad596d7d" width="2264" height="384"> </video>
 </div>
 
-## Quick Start
 
-### Installation
 
-Clone the repo:
+<p align="center">
+  <a href="https://github.com/deepseek-ai/DeepSeek-OCR"><b>🌟 Github</b></a> |
+  <a href="https://huggingface.co/deepseek-ai/DeepSeek-OCR"><b>📥 Model Download</b></a> |
+  <a href="https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSeek_OCR_paper.pdf"><b>📄 Paper Link</b></a> |
+  <a href="https://arxiv.org/abs/2510.18234"><b>📄 Arxiv Paper Link</b></a> |
+</p>
+<h2>
+<p align="center">
+  <a href="https://huggingface.co/papers/2510.18234">DeepSeek-OCR: Contexts Optical Compression</a>
+</p>
+</h2>
+<p align="center">
+<img src="assets/fig1.png" style="width: 1000px" align=center>
+</p>
+<p align="center">
+<a href="https://huggingface.co/papers/2510.18234">Explore the boundaries of visual-text compression.</a>       
+</p>
+
+## Usage
+Inference using Huggingface transformers on NVIDIA GPUs. Requirements tested on python 3.12.9 + CUDA11.8：
 
-```shell
-git clone https://github.com/meituan-longcat/LongCat-Video
-cd LongCat-Video
 ```
-
-Install dependencies:
-
-```shell
-# create conda environment
-conda create -n longcat-video python=3.10
-conda activate longcat-video
-
-# install torch (configure according to your CUDA version)
-pip install torch==2.6.0+cu124 torchvision==0.21.0+cu124 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
-
-# install flash-attn-2
-pip install ninja 
-pip install psutil 
-pip install packaging 
-pip install flash_attn==2.7.4.post1
-
-# install other requirements
-pip install -r requirements.txt
+torch==2.6.0
+transformers==4.46.3
+tokenizers==0.20.3
+einops
+addict 
+easydict
+pip install flash-attn==2.7.3 --no-build-isolation
 ```
 
-FlashAttention-2 is enabled in the model config by default; you can also change the model config to use FlashAttention-3 or xformers.
+```python
+from transformers import AutoModel, AutoTokenizer
+import torch
+import os
+os.environ["CUDA_VISIBLE_DEVICES"] = '0'
+model_name = 'deepseek-ai/DeepSeek-OCR'
 
-### Model Download
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+model = AutoModel.from_pretrained(model_name, _attn_implementation='flash_attention_2', trust_remote_code=True, use_safetensors=True)
+model = model.eval().cuda().to(torch.bfloat16)
 
-| Models | Download Link |
-| --- | --- |
-| LongCat-Video | 🤗 [Huggingface](https://huggingface.co/meituan-longcat/LongCat-Video) |
+# prompt = "<image>\nFree OCR. "
+prompt = "<image>\n<|grounding|>Convert the document to markdown. "
+image_file = 'your_image.jpg'
+output_path = 'your/output/dir'
 
-Download models using huggingface-cli:
-```shell
-pip install "huggingface_hub[cli]"
-huggingface-cli download meituan-longcat/LongCat-Video --local-dir ./weights/LongCat-Video
-```
+# infer(self, tokenizer, prompt='', image_file='', output_path = ' ', base_size = 1024, image_size = 640, crop_mode = True, test_compress = False, save_results = False):
 
-### Run Text-to-Video
+# Tiny: base_size = 512, image_size = 512, crop_mode = False
+# Small: base_size = 640, image_size = 640, crop_mode = False
+# Base: base_size = 1024, image_size = 1024, crop_mode = False
+# Large: base_size = 1280, image_size = 1280, crop_mode = False
 
-```shell
-# Single-GPU inference
-torchrun run_demo_text_to_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile
+# Gundam: base_size = 1024, image_size = 640, crop_mode = True
 
-# Multi-GPU inference
-torchrun --nproc_per_node=2 run_demo_text_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
+res = model.infer(tokenizer, prompt=prompt, image_file=image_file, output_path = output_path, base_size = 1024, image_size = 640, crop_mode=True, save_results = True, test_compress = True)
 ```
 
-### Run Image-to-Video
+## vLLM
+Refer to [🌟GitHub](https://github.com/deepseek-ai/DeepSeek-OCR/) for guidance on model inference acceleration and PDF processing, etc.<!--  -->
 
+[2025/10/23] 🚀🚀🚀 DeepSeek-OCR is now officially supported in upstream [vLLM](https://docs.vllm.ai/projects/recipes/en/latest/DeepSeek/DeepSeek-OCR.html#installing-vllm).
 ```shell
-# Single-GPU inference
-torchrun run_demo_image_to_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile
-
-# Multi-GPU inference
-torchrun --nproc_per_node=2 run_demo_image_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
+uv venv
+source .venv/bin/activate
+# Until v0.11.1 release, you need to install vLLM from nightly build
+uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
 ```
 
-### Run Video-Continuation
-
-```shell
-# Single-GPU inference
-torchrun run_demo_video_continuation.py --checkpoint_dir=./weights/LongCat-Video --enable_compile
-
-# Multi-GPU inference
-torchrun --nproc_per_node=2 run_demo_video_continuation.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
+```python
+from vllm import LLM, SamplingParams
+from vllm.model_executor.models.deepseek_ocr import NGramPerReqLogitsProcessor
+from PIL import Image
+
+# Create model instance
+llm = LLM(
+    model="deepseek-ai/DeepSeek-OCR",
+    enable_prefix_caching=False,
+    mm_processor_cache_gb=0,
+    logits_processors=[NGramPerReqLogitsProcessor]
+)
+
+# Prepare batched input with your image file
+image_1 = Image.open("path/to/your/image_1.png").convert("RGB")
+image_2 = Image.open("path/to/your/image_2.png").convert("RGB")
+prompt = "<image>\nFree OCR."
+
+model_input = [
+    {
+        "prompt": prompt,
+        "multi_modal_data": {"image": image_1}
+    },
+    {
+        "prompt": prompt,
+        "multi_modal_data": {"image": image_2}
+    }
+]
+
+sampling_param = SamplingParams(
+            temperature=0.0,
+            max_tokens=8192,
+            # ngram logit processor args
+            extra_args=dict(
+                ngram_size=30,
+                window_size=90,
+                whitelist_token_ids={128821, 128822},  # whitelist: <td>, </td>
+            ),
+            skip_special_tokens=False,
+        )
+# Generate output
+model_outputs = llm.generate(model_input, sampling_param)
+
+# Print output
+for output in model_outputs:
+    print(output.outputs[0].text)
 ```
 
-### Run Long-Video Generation
 
-```shell
-# Single-GPU inference
-torchrun run_demo_long_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile
+## Visualizations
+<table>
+<tr>
+<td><img src="assets/show1.jpg" style="width: 500px"></td>
+<td><img src="assets/show2.jpg" style="width: 500px"></td>
+</tr>
+<tr>
+<td><img src="assets/show3.jpg" style="width: 500px"></td>
+<td><img src="assets/show4.jpg" style="width: 500px"></td>
+</tr>
+</table>
 
-# Multi-GPU inference
-torchrun --nproc_per_node=2 run_demo_long_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
-```
 
-### Run Interactive Video Generation
+## Acknowledgement
 
-```shell
-# Single-GPU inference
-torchrun run_demo_interactive_video.py --checkpoint_dir=./weights/LongCat-Video --enable_compile
+We would like to thank [Vary](https://github.com/Ucas-HaoranWei/Vary/), [GOT-OCR2.0](https://github.com/Ucas-HaoranWei/GOT-OCR2.0/), [MinerU](https://github.com/opendatalab/MinerU), [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR), [OneChart](https://github.com/LingyvKong/OneChart), [Slow Perception](https://github.com/Ucas-HaoranWei/Slow-Perception) for their valuable models and ideas.
 
-# Multi-GPU inference
-torchrun --nproc_per_node=2 run_demo_interactive_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video --enable_compile
-```
-
-### Run Streamlit
-
-```shell
-# Single-GPU inference
-streamlit run ./run_streamlit.py --server.fileWatcherType none --server.headless=false
-```
-
-
-
-## Evaluation Results
-
-### Text-to-Video
-The *Text-to-Video* MOS evaluation results on our internal benchmark.
-
-| **MOS score** | **Veo3** | **PixVerse-V5** | **Wan 2.2-T2V-A14B** | **LongCat-Video** |
-|---------------|-------------------|--------------------|-------------|-------------|
-| **Accessibility** | Proprietary | Proprietary | Open Source | Open Source |
-| **Architecture** | - | - | MoE | Dense |
-| **# Total Params** | - | - | 28B | 13.6B |
-| **# Activated Params** | - | - | 14B | 13.6B |
-| Text-Alignment↑ | 3.99 | 3.81 | 3.70 | 3.76 |
-| Visual Quality↑ | 3.23 | 3.13 | 3.26 | 3.25 |
-| Motion Quality↑ | 3.86 | 3.81 | 3.78 | 3.74 |
-| Overall Quality↑ | 3.48 | 3.36 | 3.35 | 3.38 |
-
-### Image-to-Video
-The *Image-to-Video* MOS evaluation results on our internal benchmark.
-
-| **MOS score** | **Seedance 1.0** | **Hailuo-02** | **Wan 2.2-I2V-A14B** | **LongCat-Video** |
-|---------------|-------------------|--------------------|-------------|-------------|
-| **Accessibility** | Proprietary | Proprietary | Open Source | Open Source |
-| **Architecture** | - | - | MoE | Dense |
-| **# Total Params** | - | - | 28B | 13.6B |
-| **# Activated Params** | - | - | 14B | 13.6B |
-| Image-Alignment↑ | 4.12 | 4.18 | 4.18 | 4.04 |
-| Text-Alignment↑ | 3.70 | 3.85 | 3.33 | 3.49 |
-| Visual Quality↑ | 3.22 | 3.18 | 3.23 | 3.27 |
-| Motion Quality↑ | 3.77 | 3.80 | 3.79 | 3.59 |
-| Overall Quality↑ | 3.35 | 3.27 | 3.26 | 3.17 |
-
-## Community Works
-
-Community works are welcome! Please PR or inform us in Issue to add your work.
-
-- [CacheDiT](https://github.com/vipshop/cache-dit) offers Fully Cache Acceleration support for LongCat-Video with DBCache and TaylorSeer, achieved nearly 1.7x speedup without obvious loss of precision. Visit their [example](https://github.com/vipshop/cache-dit/blob/main/examples/pipeline/run_longcat_video.py) for more details.
+We also appreciate the benchmarks: [Fox](https://github.com/ucaslcl/Fox), [OminiDocBench](https://github.com/opendatalab/OmniDocBench).
 
 
-## License Agreement
-
-The **model weights** are released under the **MIT License**. 
-
-Any contributions to this repository are licensed under the MIT License, unless otherwise stated. This license does not grant any rights to use Meituan trademarks or patents. 
-
-See the [LICENSE](https://huggingface.co/meituan-longcat/LongCat-Video/blob/main/LICENSE) file for the full license text.
-
-## Usage Considerations 
-This model has not been specifically designed or comprehensively evaluated for every possible downstream application. 
-
-Developers should take into account the known limitations of large language models, including performance variations across different languages, and carefully assess accuracy, safety, and fairness before deploying the model in sensitive or high-risk scenarios. 
-It is the responsibility of developers and downstream users to understand and comply with all applicable laws and regulations relevant to their use case, including but not limited to data protection, privacy, and content safety requirements. 
-
-Nothing in this Model Card should be interpreted as altering or restricting the terms of the MIT License under which the model is released. 
-
 ## Citation
-We kindly encourage citation of our work if you find it useful.
-
-```
-@misc{meituanlongcatteam2025longcatvideotechnicalreport,
-      title={LongCat-Video Technical Report}, 
-      author={Meituan LongCat Team and Xunliang Cai and Qilong Huang and Zhuoliang Kang and Hongyu Li and Shijun Liang and Liya Ma and Siyu Ren and Xiaoming Wei and Rixu Xie and Tong Zhang},
-      year={2025},
-      eprint={2510.22200},
-      archivePrefix={arXiv},
-      primaryClass={cs.CV},
-      url={https://arxiv.org/abs/2510.22200}, 
-}
-```
-
-## Acknowledgements
-
-We would like to thank the contributors to the [Wan](https://huggingface.co/Wan-AI), [UMT5-XXL](https://huggingface.co/google/umt5-xxl), [Diffusers](https://github.com/huggingface/diffusers) and [HuggingFace](https://huggingface.co) repositories, for their open research.
-
-
-## Contact
-Please contact us at <a href="mailto:longcat-team@meituan.com">longcat-team@meituan.com</a> or join our WeChat Group if you have any questions.
+```bibtex
+@article{wei2025deepseek,
+  title={DeepSeek-OCR: Contexts Optical Compression},
+  author={Wei, Haoran and Sun, Yaofeng and Li, Yukun},
+  journal={arXiv preprint arXiv:2510.18234},
+  year={2025}
+}

config.json

@@ -1,3 +1,118 @@
 {
-    "model_name": "LongCat-Video"
+ "_name_or_path": "deepseek-ai/DeepSeek-OCR",
+  "candidate_resolutions": [
+  [
+   1024,
+   1024
+  ]
+ ],
+ "global_view_pos": "head",
+  "architectures": [
+      "DeepseekOCRForCausalLM"
+    ],
+  "auto_map": {
+    "AutoConfig": "modeling_deepseekocr.DeepseekOCRConfig",
+    "AutoModel": "modeling_deepseekocr.DeepseekOCRForCausalLM"
+  },
+ "language_config": {
+    "architectures": [
+      "DeepseekV2ForCausalLM"
+    ],
+    "auto_map": {
+      "AutoConfig": "configuration_deepseekv2.DeepseekV2Config",
+      "AutoModel": "modeling_deepseek.DeepseekV2Model",
+      "AutoModelForCausalLM": "modeling_deepseek.DeepseekV2ForCausalLM"
+    },
+    "bos_token_id": 0,
+    "eos_token_id": 1,
+    "first_k_dense_replace": 1,
+    "hidden_size": 1280,
+    "intermediate_size": 6848,
+    "kv_lora_rank": null,
+    "lm_head": true,
+    "max_position_embeddings": 8192,
+    "moe_intermediate_size": 896,
+    "n_group": 1,
+    "n_routed_experts": 64,
+    "n_shared_experts": 2,
+    "num_attention_heads": 10,
+    "num_experts_per_tok": 6,
+    "num_hidden_layers": 12,
+    "num_key_value_heads": 10,
+    "q_lora_rank": null,
+    "qk_nope_head_dim": 0,
+    "qk_rope_head_dim": 0,
+    "rm_head": false,
+    "topk_group": 1,
+    "topk_method": "greedy",
+    "torch_dtype": "bfloat16",
+    "use_mla": false,
+    "v_head_dim": 0,
+    "vocab_size": 129280
+  },
+ "model_type": "deepseek_vl_v2",
+ "projector_config": {
+  "input_dim": 2048,
+  "model_type": "mlp_projector",
+  "n_embed": 1280,
+  "projector_type": "linear"
+ },
+ "tile_tag": "2D",
+ "torch_dtype": "bfloat16",
+ "transformers_version": "4.46.3",
+ "vision_config": {
+  "image_size": 1024,
+  "mlp_ratio": 3.7362,
+  "model_name": "deeplip_b_l",
+  "model_type": "vision",
+  "width": {
+   "clip-l-14-224": {
+    "heads": 16,
+    "image_size": 224,
+    "layers": 24,
+    "patch_size": 14,
+    "width": 1024
+   },
+   "sam_vit_b": {
+    "downsample_channels": [
+     512,
+     1024
+    ],
+    "global_attn_indexes": [
+     2,
+     5,
+     8,
+     11
+    ],
+    "heads": 12,
+    "layers": 12,
+    "width": 768
+   }
+  }
+ },
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "first_k_dense_replace": 1,
+  "hidden_size": 1280,
+  "intermediate_size": 6848,
+  "kv_lora_rank": null,
+  "lm_head": true,
+  "max_position_embeddings": 8192,
+  "moe_intermediate_size": 896,
+  "n_group": 1,
+  "n_routed_experts": 64,
+  "n_shared_experts": 2,
+  "num_attention_heads": 10,
+  "num_experts_per_tok": 6,
+  "num_hidden_layers": 12,
+  "num_key_value_heads": 10,
+  "q_lora_rank": null,
+  "qk_nope_head_dim": 0,
+  "qk_rope_head_dim": 0,
+  "rm_head": false,
+  "topk_group": 1,
+  "topk_method": "greedy",
+  "use_mla": false,
+  "v_head_dim": 0,
+  "vocab_size": 129280
 }