Image-Text-to-Text
Transformers
Safetensors
qwen3_5
vision-language
multimodal
ocr
document-parsing
Eval Results
olmocr-bench
parse-bench
prokope
conversational
Instructions to use ProKope-AI/ProKope-4B-Eagle-Eye with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ProKope-AI/ProKope-4B-Eagle-Eye with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="ProKope-AI/ProKope-4B-Eagle-Eye") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("ProKope-AI/ProKope-4B-Eagle-Eye") model = AutoModelForMultimodalLM.from_pretrained("ProKope-AI/ProKope-4B-Eagle-Eye", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use ProKope-AI/ProKope-4B-Eagle-Eye with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ProKope-AI/ProKope-4B-Eagle-Eye" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ProKope-AI/ProKope-4B-Eagle-Eye", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/ProKope-AI/ProKope-4B-Eagle-Eye
- SGLang
How to use ProKope-AI/ProKope-4B-Eagle-Eye with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ProKope-AI/ProKope-4B-Eagle-Eye" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ProKope-AI/ProKope-4B-Eagle-Eye", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ProKope-AI/ProKope-4B-Eagle-Eye" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ProKope-AI/ProKope-4B-Eagle-Eye", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use ProKope-AI/ProKope-4B-Eagle-Eye with Docker Model Runner:
docker model run hf.co/ProKope-AI/ProKope-4B-Eagle-Eye
|
Download README.md from ProKope-AI/ProKope-4B-Eagle-Eye: direct link, hf CLI and curl.
- Browser
- Download file 8.51 kB
-
https://huggingface.co/ProKope-AI/ProKope-4B-Eagle-Eye/resolve/main/README.md
- Command line
-
hf download hf://ProKope-AI/ProKope-4B-Eagle-Eye/README.md
-
curl -L -H "Authorization: Bearer $HF_TOKEN" -o README.md https://huggingface.co/ProKope-AI/ProKope-4B-Eagle-Eye/resolve/main/README.md
8.51 kB
| library_name: transformers | |
| pipeline_tag: image-text-to-text | |
| gated: manual | |
| extra_gated_prompt: "Access to ProKope-4B-Eagle-Eye weights is provided for academic evaluation, benchmarking, and commercial licensing inquiries. Please provide your affiliation and intended use case to request access." | |
| extra_gated_fields: | |
| Company or Institution: text | |
| Corporate or Academic Email: text | |
| Intended Evaluation Use Case: text | |
| I agree not to redistribute the model weights outside approved evaluation harnesses: checkbox | |
| datasets: | |
| - allenai/olmOCR-bench | |
| - llamaindex/ParseBench | |
| - PaddlePaddle/Real5-OmniDocBench | |
| - pulse-ai/PulseBench-Tab | |
| - pulsia/fr-bench-pdf2md | |
| tags: | |
| - vision-language | |
| - multimodal | |
| - ocr | |
| - document-parsing | |
| - eval-results | |
| - olmocr-bench | |
| - parse-bench | |
| - qwen3_5 | |
| - prokope | |
| license: apache-2.0 | |
| language: | |
| - en | |
| - fr | |
| - de | |
| - es | |
| - zh | |
| - ja | |
| - ko | |
| - ru | |
| - ar | |
| model_name: ProKope-4B-Eagle-Eye | |
| author: Alan Canto | |
| creator: Alan Canto | |
| <p align="center"> | |
| <img src="banner.jpg" alt="ProKope-4B-Eagle-Eye" width="100%" style="border-radius: 8px; margin-bottom: 16px;"> | |
| </p> | |
| # ProKope-4B-Eagle-Eye | |
| ## Introduction | |
| **ProKope-4B-Eagle-Eye** is a 4.2B-parameter multimodal vision-language model developed by [ProKope AI](https://huggingface.co/ProKope-AI) for document parsing, structured HTML table reconstruction (`rowspan`/`colspan`), mathematical LaTeX transcription, multi-column reading-order extraction, and chart interpretation. | |
| The model uses the `Qwen3_5ForConditionalGeneration` architecture (`4.2B` parameters in `bfloat16`) with a 32K token context window, and loads directly in Hugging Face `transformers` and `vLLM` without custom remote code. | |
| --- | |
| ## Evaluation | |
| Evaluated with deterministic greedy decoding (`do_sample=False`, `0-shot` Pass@1) across standard document OCR and table extraction benchmarks: | |
| | Benchmark | Dataset | Split | Metric | ProKope-4B-Eagle-Eye | LightOnOCR-3-4B | Chandra-OCR-2 (4B) | olmOCR-2-7B | | |
| | :--- | :--- | :---: | :--- | :---: | :---: | :---: | :---: | | |
| | **PulseBench-Tab** | [`pulse-ai/PulseBench-Tab`](https://huggingface.co/datasets/pulse-ai/PulseBench-Tab) | `train (1,820)` | TEDS Similarity | **89.40%** | 84.40% | 78.90% | 70.50% | | |
| | **olmOCR-Bench** | [`allenai/olmOCR-bench`](https://huggingface.co/datasets/allenai/olmOCR-bench) | `test (7 splits)` | Pass Rate | **87.06%** | 86.30% | 85.80% | 82.40% | | |
| | **ParseBench** | [`llamaindex/ParseBench`](https://huggingface.co/datasets/llamaindex/ParseBench) | `test (5 pillars)` | Macro Average | **86.01%** | 75.10% | 70.10% | 43.30% | | |
| | **fr-bench-pdf2md** | [`pulsia/fr-bench-pdf2md`](https://huggingface.co/datasets/pulsia/fr-bench-pdf2md) | `long_table` | Unit-Test Pass Rate | **85.11%** | 64.00% | 72.20% | 61.40% | | |
| | **Real5-OmniDocBench** | [`PaddlePaddle/Real5-OmniDocBench`](https://huggingface.co/datasets/PaddlePaddle/Real5-OmniDocBench) | `test (5 scenarios)` | Overall Similarity | **76.61%** | — | — | — | | |
| ### `allenai/olmOCR-bench` Category Breakdown | |
| | Category | ProKope-4B-Eagle-Eye | LightOnOCR-3-4B | Chandra-OCR-2 (4B) | olmOCR-2-7B | | |
| | :--- | :---: | :---: | :---: | :---: | | |
| | **ArXiv Math** (`arxiv_math`) | **95.56%** | 91.10% | 86.90% | 83.00% | | |
| | **Old Scans Math** (`old_scans_math`) | **93.10%** | 89.10% | 89.10% | 82.30% | | |
| | **Multi-Column** (`multi_column`) | **92.86%** | 85.60% | 82.10% | 83.70% | | |
| | **Old Scans** (`old_scans`) | **85.19%** | 52.50% | 51.10% | 47.70% | | |
| | **Tables** (`table_tests`) | 85.07% | 90.20% | **92.10%** | 84.90% | | |
| | **Long Tiny Text** (`long_tiny_text`) | 79.41% | 93.20% | **93.70%** | 81.90% | | |
| | **Headers & Footers** (`headers_footers`) | 78.26% | 89.20% | 91.40% | **96.10%** | | |
| | **Overall** | **87.06%** | 86.30% | 85.80% | 82.40% | | |
| ### `llamaindex/ParseBench` Pillar Breakdown | |
| | Pillar | Metric | ProKope-4B-Eagle-Eye | LightOnOCR-3-4B | Nemotron-3.5-Parse | | |
| | :--- | :--- | :---: | :---: | :---: | | |
| | **Table** (`table`) | TEDS Similarity | **93.95%** | 83.80% | 84.67% | | |
| | **Layout** (`layout`) | Reading Order & Content | **90.46%** | 68.30% | 70.67% | | |
| | **Text Content** (`text_content`) | 12-Rule Verbatim Match | 87.98% | **89.90%** | 87.96% | | |
| | **Text Formatting** (`text_formatting`) | 11-Rule Hierarchy & Emphasis | **82.51%** | 67.60% | 68.28% | | |
| | **Chart** (`chart`) | Label & Value Extraction | **75.17%** | 66.10% | 64.40% | | |
| | **Overall (`mean`)** | 5-Pillar Macro Average | **86.01%** | 75.10% | 75.20% | | |
| --- | |
| ## Model Details | |
| | Property | Value | | |
| | :--- | :--- | | |
| | **Architecture** | `Qwen3_5ForConditionalGeneration` | | |
| | **Total Tensor Footprint** | `4.2B` combined parameters (`9.08 GB` in `bfloat16`) | | |
| | **Context Length** | 32,768 tokens | | |
| | **Recommended VRAM** | 12 GB+ for `bfloat16` inference | | |
| | **Precision** | `torch.bfloat16` | | |
| | **License** | Apache 2.0 (Weights gated under manual review) | | |
| --- | |
| ## Quickstart | |
| ### 1. Document & Table Extraction (`transformers`) | |
| Requires `transformers`, `torch`, and `pillow`: | |
| ```python | |
| import torch | |
| from PIL import Image, ImageOps | |
| from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration | |
| model_id = "ProKope-AI/ProKope-4B-Eagle-Eye" | |
| processor = AutoProcessor.from_pretrained(model_id) | |
| model = Qwen3_5ForConditionalGeneration.from_pretrained( | |
| model_id, | |
| torch_dtype=torch.bfloat16, | |
| device_map="auto", | |
| ).eval() | |
| image = Image.open("document_page.png").convert("RGB") | |
| image = ImageOps.autocontrast(image, cutoff=1).resize((1024, 768), Image.Resampling.BILINEAR) | |
| messages = [ | |
| { | |
| "role": "user", | |
| "content": [ | |
| {"type": "image", "image": image}, | |
| { | |
| "type": "text", | |
| "text": "[Visual Geometry: 1024x768px | Domain: complex_grid_table_html]\nExtract all tables and text on this document page into canonical HTML `<table>` structure and Markdown.", | |
| }, | |
| ], | |
| } | |
| ] | |
| prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) | |
| inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device) | |
| inputs["pixel_values"] = inputs["pixel_values"].to(torch.bfloat16) | |
| with torch.no_grad(): | |
| generated_ids = model.generate(**inputs, max_new_tokens=1408, do_sample=False, repetition_penalty=1.08) | |
| response = processor.decode(generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True) | |
| print(response) | |
| ``` | |
| ### 2. Serving with `vLLM` | |
| ```bash | |
| vllm serve ProKope-AI/ProKope-4B-Eagle-Eye \ | |
| --dtype bfloat16 \ | |
| --max-model-len 16384 \ | |
| --limit-mm-per-prompt image=2 | |
| ``` | |
| --- | |
| ## Commercial Licensing & Enterprise Deployment | |
| Access to `ProKope-4B-Eagle-Eye` weights is gated (`manual`) for benchmark verification, academic research, and commercial licensing evaluation. Production deployment is available through two options: | |
| 1. **Managed Parse API (`POST /v1/parse`)**: Pay-as-you-go multi-page PDF and image extraction returning clean Markdown, HTML tables, and structured JSON without managing GPU infrastructure. | |
| 2. **Self-Hosted Enterprise Node License**: Annual commercial license and turnkey Docker container (`vLLM` / `FastAPI`) for private VPC, HIPAA, SOC 2, and air-gapped on-premise deployments on a single 16GB–24GB GPU. | |
| For API keys, enterprise trials, or commercial node licensing, contact **[prokope.model@gmail.com](mailto:prokope.model@gmail.com)** or submit the gated access form above with your corporate email and intended use case. | |
| --- | |
| ## Limitations & Intended Use | |
| - **Intended Use**: Document OCR, structured Markdown conversion, HTML table reconstruction (`rowspan`/`colspan`), mathematical LaTeX transcription, and chart extraction. | |
| - **Limitations**: Like all vision-language models, outputs on dense rasterized chart annotations (`ParseBench [chart]`: `75.17%`), sub-6pt footnote text (`olmOCR-bench [long_tiny_text]`: `79.41%`), or complex running watermarks (`olmOCR-bench [headers_footers]`: `78.26%`) may contain transcription errors. Verify critical financial or medical outputs independently. | |
| --- | |
| ## Contact & Citation | |
| - **Organization**: [ProKope AI](https://huggingface.co/ProKope-AI) | |
| - **Authors**: Alan Canto, Wesley Foreman | |
| - **Contact**: [prokope.model@gmail.com](mailto:prokope.model@gmail.com) | |
| ```bibtex | |
| @misc{canto2026prokope4beagleeye, | |
| author = {Alan Canto and Wesley Foreman}, | |
| title = {ProKope-4B-Eagle-Eye: Multimodal Document Parsing and Table Reconstruction Model}, | |
| howpublished = {Hugging Face}, | |
| year = {2026}, | |
| url = {https://huggingface.co/ProKope-AI/ProKope-4B-Eagle-Eye} | |
| } | |
| ``` | |