Instructions to use rednote-hilab/dots.ocr with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use rednote-hilab/dots.ocr with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="rednote-hilab/dots.ocr", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("rednote-hilab/dots.ocr", trust_remote_code=True, dtype="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use rednote-hilab/dots.ocr with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "rednote-hilab/dots.ocr" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "rednote-hilab/dots.ocr", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/rednote-hilab/dots.ocr
- SGLang
How to use rednote-hilab/dots.ocr with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "rednote-hilab/dots.ocr" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "rednote-hilab/dots.ocr", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "rednote-hilab/dots.ocr" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "rednote-hilab/dots.ocr", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use rednote-hilab/dots.ocr with Docker Model Runner:
docker model run hf.co/rednote-hilab/dots.ocr
Fix: Resolve TypeError for video_processor during model loading.
Browse files### **Subject: Fix: Resolve `TypeError` for `video_processor` during model loading**
**Description:**
This pull request addresses a `TypeError` that occurs when loading the `dots-ocr` model with the latest versions of the `transformers` library. The error message, "Received a NoneType for argument 'video_processor', but a BaseVideoProcessor was expected," is triggered because the `DotsVLProcessor` class inherits from a processor that now expects a `video_processor` attribute.
**The Problem:**
The current implementation of `DotsVLProcessor` does not explicitly handle the `video_processor` argument in its constructor. As the base classes in the `transformers` library have evolved, this argument has become a required part of the processor's initialization, leading to a `NoneType` being passed and causing the `TypeError`.
**The Solution:**
This has been resolved by making a minor but critical addition to the `DotsVLProcessor` class. By adding `video_processor=None` to the `__init__` method, we explicitly initialize the video processor as `None`, satisfying the requirements of the parent class without altering the model's core OCR functionality.
The change is as follows:
```python
class DotsVLProcessor(Qwen2_5_VLProcessor):
attributes = ["image_processor", "tokenizer"]
def __init__(self, image_processor=None, tokenizer=None, video_processor=None, chat_template=None, **kwargs):
super().__init__(image_processor, tokenizer, chat_template=chat_template)
self.image_token = "<|imgpad|>" if not hasattr(tokenizer, "image_token") else tokenizer.image_token
self.image_token_id = 151665 if not hasattr(tokenizer, "image_token_id") else tokenizer.image_token_id
```
This ensures that the model remains compatible with recent library updates and can be loaded without error.
## The updated implementation with `transformers==4.57.1` is as follows:
HF Space: https://huggingface.co/spaces/prithivMLmods/Multimodal-OCR3
<table>
<tr>
<td>
<img src="https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/MicSVxaKNLCj-Ih2iNM5n.png" alt="Screenshot 1" width="400"/>
</td>
<td>
<img src="https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/1dUvHL5V7RQ_jGihongWP.png" alt="Screenshot 2" width="400"/>
</td>
</tr>
</table>
- configuration_dots.py +4 -3
|
@@ -52,8 +52,8 @@ class DotsVisionConfig(PretrainedConfig):
|
|
| 52 |
|
| 53 |
class DotsOCRConfig(Qwen2Config):
|
| 54 |
model_type = "dots_ocr"
|
| 55 |
-
def __init__(self,
|
| 56 |
-
image_token_id = 151665,
|
| 57 |
video_token_id = 151656,
|
| 58 |
vision_config: Optional[dict] = None, *args, **kwargs):
|
| 59 |
super().__init__(*args, **kwargs)
|
|
@@ -67,6 +67,7 @@ class DotsOCRConfig(Qwen2Config):
|
|
| 67 |
|
| 68 |
|
| 69 |
class DotsVLProcessor(Qwen2_5_VLProcessor):
|
|
|
|
| 70 |
def __init__(self, image_processor=None, tokenizer=None, chat_template=None, **kwargs):
|
| 71 |
super().__init__(image_processor, tokenizer, chat_template=chat_template)
|
| 72 |
self.image_token = "<|imgpad|>" if not hasattr(tokenizer, "image_token") else tokenizer.image_token
|
|
@@ -74,4 +75,4 @@ class DotsVLProcessor(Qwen2_5_VLProcessor):
|
|
| 74 |
|
| 75 |
|
| 76 |
AutoProcessor.register("dots_ocr", DotsVLProcessor)
|
| 77 |
-
CONFIG_MAPPING.register("dots_ocr", DotsOCRConfig)
|
|
|
|
| 52 |
|
| 53 |
class DotsOCRConfig(Qwen2Config):
|
| 54 |
model_type = "dots_ocr"
|
| 55 |
+
def __init__(self,
|
| 56 |
+
image_token_id = 151665,
|
| 57 |
video_token_id = 151656,
|
| 58 |
vision_config: Optional[dict] = None, *args, **kwargs):
|
| 59 |
super().__init__(*args, **kwargs)
|
|
|
|
| 67 |
|
| 68 |
|
| 69 |
class DotsVLProcessor(Qwen2_5_VLProcessor):
|
| 70 |
+
attributes = ["image_processor", "tokenizer"]
|
| 71 |
def __init__(self, image_processor=None, tokenizer=None, chat_template=None, **kwargs):
|
| 72 |
super().__init__(image_processor, tokenizer, chat_template=chat_template)
|
| 73 |
self.image_token = "<|imgpad|>" if not hasattr(tokenizer, "image_token") else tokenizer.image_token
|
|
|
|
| 75 |
|
| 76 |
|
| 77 |
AutoProcessor.register("dots_ocr", DotsVLProcessor)
|
| 78 |
+
CONFIG_MAPPING.register("dots_ocr", DotsOCRConfig)
|