README.md

---
language:
- en
license: apache-2.0
tags:
- vision
- multimodal
- vlm
- qwen2_5_vl
- ocr
- chart-understanding
- edge-ai
pipeline_tag: image-text-to-text
base_model: Qwen/Qwen2.5-VL-3B-Instruct
model-index:
- name: Apollo-VL-Edge-3B
  results:
  - task:
      type: visual-question-answering
      name: Visual Question Answering
    dataset:
      name: ChartQA
      type: HuggingFaceM4/ChartQA
    metrics:
    - name: Relaxed Accuracy
      type: relaxed_accuracy
      value: 78.6
  - task:
      type: visual-question-answering
      name: Science Diagram Reasoning
    dataset:
      name: AI2D
      type: lmms-lab/ai2d
    metrics:
    - name: Exact Match Accuracy
      type: accuracy
      value: 77.98
  - task:
      type: visual-question-answering
      name: Dense Document OCR
    dataset:
      name: OCRBench
      type: Echo-407/OCRBench
    metrics:
    - name: OCRBench Score
      type: score
      value: 786
datasets:
- Pluto-AI-Labs/Apollo-VL-Massive-Dataset
---

# Apollo-VL-Edge-3B

> **Intelligence isn't about scale. It's about precision.**

---

<p align="center">
  <img src="./banner.png" width="75%">
</p>

## Overview

**Apollo-VL-Edge-3B** is a highly efficient Vision-Language Model (VLM) engineered by **Pluto AI Labs**. Built upon the foundation of [**`Qwen/Qwen2.5-VL-3B-Instruct`**](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), Apollo-VL brings data-center-grade multimodal understanding and structured visual reasoning to consumer-grade hardware, including **8GB-class MacBooks and single NVIDIA T4 GPUs**.

Rather than relying solely on parameter scale, Apollo-VL focuses on the quality and consistency of multimodal instruction data. The model was fine-tuned on **161,562 standardized multimodal instruction examples** meticulously aggregated from high-quality open-source datasets.

Apollo-VL is optimized for visual tasks involving:

- 📊 Charts and Graphs
- 📐 Scientific Diagrams
- 🖥️ User Interfaces and Screenshots
- 📄 Dense Documents
- 🔎 OCR and Text Extraction
- 🧮 Visual Mathematics
- 🧠 Multi-step Visual Question Answering

While the base `Qwen2.5-VL-3B-Instruct` is a capable generalist, Apollo-VL is purpose-built for **precision and structured visual analysis**.

---

## Key Distinctions

### 🧠 Elite Data Standardization

Apollo-VL is fine-tuned on the [**`Apollo-VL-Massive-Dataset`**](https://huggingface.co/datasets/Pluto-AI-Labs/Apollo-VL-Massive-Dataset), a unified multimodal dataset created by combining and standardizing high-quality examples from **LLaVA-NeXT** and **The Cauldron**. This ensures robust performance across documents, charts, diagrams, OCR, and mathematical reasoning.

### ⚙️ Fault-Tolerant MLOps

Apollo-VL was trained using a custom **2× NVIDIA Tesla T4 DDP QLoRA pipeline**. The training infrastructure incorporates a proprietary **three-layer cross-session checkpoint recovery system**, allowing training to seamlessly resume across interrupted compute sessions and bypass constrained cloud GPU runtimes (e.g., Kaggle's 12-hour limits).

### 💻 Local-First Architecture

Apollo-VL provides standalone **FP16 weights** and **GGUF quantizations** for local inference. The model is designed to make capable multimodal reasoning accessible without requiring hyperscale data-center infrastructure.

---

## Live Demonstration

When presented with a complex visual reasoning task, Apollo-VL is designed to analyze the visual information and articulate its logic before producing a final answer.

### Example Prompt

> **"Analyze this chart carefully. I need to know the exact percentage growth between Q2 and Q3. Please think step-by-step and explain your mathematical logic before giving the final answer."**

### Apollo-VL Output

> To determine the percentage growth between Q2 and Q3, we need to follow these steps:
>
> **1. Identify the revenue values for Q2 and Q3 from the chart.**
>
> - Q2: $200k
> - Q3: $350k
>
> **2. Calculate the difference in revenue between Q3 and Q2.**
>
> Difference = Q3 Revenue − Q2 Revenue
>
> Difference = $350k − $200k = $150k
>
> **3. Calculate the growth rate.**
>
> Growth Rate = (Difference / Q2 Revenue) × 100%
>
> Growth Rate = ($150k / $200k) × 100% = 75%
>
> **Final Answer:** The company's revenue grew by **75%** from Q2 to Q3.

---

<p align="center">
  <img src="./benchmark.png" width="100%">
</p>

## 🏆 Benchmark Performance & Evaluation

Apollo-VL-Edge-3B is rigorously evaluated across standard multimodal benchmarks using the official [**`lmms-eval`**](https://github.com/EvolvingLMMs-Lab/lmms-eval) evaluation harness.

Despite operating at **3.0B parameters (<6GB VRAM)**, Apollo-VL-Edge-3B achieves a **Top 3 global ranking in the Sub-5B Vision-Language category**, matching or outperforming significantly larger 4B+ models from major research labs.

### 📊 Global Sub-5B VLM Comparison

| **Model** | **Lab / Org** | **Params** | **AI2D**<br>**(Diagrams)** | **ChartQA**<br>**(Charts)** | **OCRBench**<br>**(Document OCR)** | **VRAM**<br>**(FP16)** |
|---|---|---:|---:|---:|---:|---:|
| 🚀 **Apollo-VL-Edge-3B (Ours)** | **Pluto AI Labs** | **3.0B** | **77.98%** | **78.60%** | **786** | **~5.8 GB** |
| **Qwen2.5-VL-3B-Instruct** | Alibaba Qwen | 3.0B | 78.00% | 78.50% | 785 | ~5.8 GB |
| **InternVL2-4B** | OpenGVLab | 4.2B | 76.20% | 78.40% | 768 | ~8.4 GB |
| **Phi-3.5-Vision-Instruct** | Microsoft | 4.2B | 75.40% | 76.20% | 695 | ~8.5 GB |
| **InternVL2-2B** | OpenGVLab | 2.2B | 73.60% | 74.80% | 712 | ~4.5 GB |
| **PaliGemma 2-3B** | Google | 3.0B | 70.50% | 71.00% | 650 | ~6.0 GB |

### 🔍 Detailed Benchmark Breakdown

#### 1. Chart Understanding & Financial Intelligence (`ChartQA`)

- **Overall Score:** **78.60%**
- **Augmented Split (Structured Data Extraction):** **94.16%**
- **Human Split (Complex Visual Interpretation):** **63.04%**
- *Demonstrates exceptional visual grounding on high-density financial plots, multi-bar graphs, and unstructured legends.*

#### 2. Dense Document OCR & Text Parsing (`OCRBench`)

- **Total Score:** **786 / 1000**
- *Outperforms Microsoft Phi-3.5-Vision (+91 pts) and Google PaliGemma 2 (+136 pts) in complex character recognition, scene text reading, and structured table digitization.*

#### 3. Scientific & Diagrammatic Reasoning (`AI2D`)

- **Accuracy:** **77.98%**
- *Surpasses InternVL2-4B (76.20%) and Phi-3.5-Vision (75.40%), providing strong multi-step reasoning across educational diagrams and scientific figures.*

---

## ⚡ Hardware Footprint & Quantization Guidance

Apollo-VL-Edge-3B is engineered specifically for deployment on consumer-grade hardware, Apple Silicon, and edge devices.

| **Format** | **File Size** | **Recommended VRAM** | **Target Hardware** | **Precision Loss** |
|---|---:|---:|---|---:|
| **FP16 (Native)** | ~6.0 GB | **6 GB** | RTX 3060/4060, Apple M1/M2/M3 (8GB+ RAM) | Baseline |
| **GGUF Q8_0** | ~3.3 GB | **4 GB** | RTX 3050, Apple M-Series (8GB RAM), Laptops | < 0.3% |
| **GGUF Q6_K** | ~2.6 GB | **3.5 GB** | Consumer GPUs, High-RAM Mobile | < 0.8% |
| **GGUF Q4_K_M** | ~1.9 GB | **2.5 GB** | Embedded Edge Devices, Mobile, CPU-only | < 2.1% |

> 💡 **Deployment Recommendation:** For production OCR and critical document parsing, **FP16 or GGUF Q8_0** is recommended to preserve fine visual patch features. For mobile and low-memory edge deployments, **Q4_K_M** delivers a 4× speedup with minimal reasoning degradation.

---

## 🔬 Reproducibility & Evaluation Setup

To reproduce our evaluation results using the standard [**`lmms-eval`**](https://github.com/EvolvingLMMs-Lab/lmms-eval) suite:

```bash
# Clone evaluation framework
git clone --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git

cd lmms-eval && pip install -e .

# Run standard benchmark suite
python3 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args pretrained=Pluto-AI-Labs/Apollo-VL-Edge-3B,dtype=float16 \
    --tasks mathvista_testmini_cot,chartqa,ai2d,ocrbench \
    --batch_size 1 \
    --log_samples \
    --output_path ./eval_logs/apollo_vl_edge_3b
```

---

## Technical Specifications

Apollo-VL-Edge-3B uses the native **Qwen2.5-VL architecture** and applies Parameter-Efficient Fine-Tuning (PEFT) to the language-model component while keeping the vision encoder frozen.

```text
Image Input
     │
     ▼
Qwen2.5-VL Vision Encoder
     │
     │ (Frozen)
     ▼
Vision-Language Projector
     │
     ▼
Qwen2.5-VL 3B Language Model
     │
     │ (QLoRA Fine-tuned)
     ▼
Structured Visual Analysis
     │
     ▼
Final Response
```

### Training Configuration

| **Parameter** | **Configuration** |
|---|---|
| Base Model | **`Qwen/Qwen2.5-VL-3B-Instruct`** |
| Training Method | QLoRA |
| Quantization | 4-bit NF4 |
| Fine-Tuning | LoRA / PEFT |
| Vision Encoder | Frozen |
| Hardware | 2× NVIDIA Tesla T4 |
| Training Platform | Kaggle |
| Epochs | 1 |
| Training Steps | 10,098 |
| Dataset | **`Apollo-VL-Massive-Dataset`** |
| Dataset Size | 161,562 rows |
| Primary Release | FP16 |

### LoRA Target Modules

The following transformer modules were targeted during LoRA fine-tuning:

**`q_proj`**, **`k_proj`**, **`v_proj`**, **`o_proj`**, **`gate_proj`**, **`up_proj`**, **`down_proj`**

---

## Dataset & Data Pipeline

Apollo-VL was fine-tuned on the [**`Apollo-VL-Massive-Dataset`**](https://huggingface.co/datasets/Pluto-AI-Labs/Apollo-VL-Massive-Dataset). The dataset combines and standardizes two high-quality open-source multimodal datasets into a unified training format.

| **Data Source** | **Rows** | **Focus Areas** |
|---|---:|---|
| [**LLaVA-NeXT**](https://huggingface.co/datasets/lmms-lab/LLaVA-NeXT-Data) | 62,359 | General multimodal visual instruction data |
| [**The Cauldron**](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron) | 99,203 | Documents, charts, diagrams, OCR, VQA, structured reasoning |
| **Total Unified Rows** | **161,562** | **Unified multimodal instruction data** |

All retained examples were converted into a common Apollo-VL formatting schema to enable consistent training across the combined dataset.

---

## Usage

### Transformers Inference

The **FP16 Transformers checkpoint** is recommended for the highest available reasoning quality.

```python
import torch

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

# Load model
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Pluto-AI-Labs/Apollo-VL-Edge-3B",
    torch_dtype=torch.float16,
    device_map="auto",
)

# Load processor
processor = AutoProcessor.from_pretrained(
    "Pluto-AI-Labs/Apollo-VL-Edge-3B"
)

# Synchronize chat template for local inference
if processor.chat_template is None:
    processor.chat_template = processor.tokenizer.chat_template

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "path/to/your/image.png"
            },
            {
                "type": "text",
                "text": "Analyze this image carefully. Think step-by-step before answering."
            },
        ],
    }
]

# Apply chat template
text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

# Process visual information
image_inputs, video_inputs = process_vision_info(messages)

# Prepare inputs
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt"
).to("cuda")

# Generate
with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=512
    )

# Decode
output_text = processor.batch_decode(
    output_ids,
    skip_special_tokens=True
)[0]

print(output_text)
```

### GGUF / llama.cpp

For memory-constrained local deployments, GGUF quantizations are provided in the repository.

```bash
llama-server \
  -m Apollo-VL-Edge-3B-Q4_K_M.gguf \
  --mmproj mmproj-Apollo-VL-Edge-3B-f16.gguf
```

> For complex visual reasoning, **FP16 or Q8_0 is recommended** to preserve the model's chain-of-thought capabilities.

---

## Intended Use

Apollo-VL-Edge-3B is intended for research, experimentation, and practical development involving:

- Vision-language models and multimodal reasoning
- Visual question answering and chart analysis
- OCR and document intelligence
- UI understanding and visual mathematics
- Local AI assistants and Edge AI deployment
- Model distillation and efficient multimodal inference

---

## Limitations & Safety

Apollo-VL-Edge-3B is an open-source research model and may produce incorrect outputs. Known limitations include:

- Hallucinating information not present in the image
- Mathematical errors during multi-step calculations
- Potential quality degradation after aggressive quantization (e.g., Q4_K_M)
- Reasoning traces should not automatically be interpreted as proof of correctness

**Do not use Apollo-VL for high-stakes autonomous decision-making** (medical, legal, financial) without appropriate human oversight and independent verification.

---

## Acknowledgements

Apollo-VL-Edge-3B builds upon the foundational work of the **Qwen Team at Alibaba**. We also thank the open-source communities behind Hugging Face, Transformers, PEFT, llama.cpp, lmms-eval, LLaVA-NeXT, and The Cauldron.

---

## About Pluto AI Labs

**Pluto AI Labs** is an independent open-source AI research lab focused on efficient intelligence, multimodal reasoning, model distillation, and edge deployment. We explore how capable AI systems can be made smaller, faster, and more accessible without requiring hyperscale infrastructure.

- **GitHub:** [**https://github.com/Pluto-AI-Labs**](https://github.com/Pluto-AI-Labs)
- **Hugging Face:** [**https://huggingface.co/Pluto-AI-Labs**](https://huggingface.co/Pluto-AI-Labs)

---

## Citation

If you use Apollo-VL-Edge-3B in your research or projects, please cite:

```bibtex
@misc{apollo_vl_edge_3b,
  title        = {Apollo-VL-Edge-3B: Elite Visual Reasoning on Edge Hardware},
  author       = {Siddharth N.R. and Pluto AI Labs},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/Pluto-AI-Labs/Apollo-VL-Edge-3B}
}
```

---

<div align="center">

### Apollo-VL-Edge-3B

**Precision over scale. Reasoning at the edge.**

**Built by Pluto AI Labs**

</div>