Update README.md

3b110b4 verified about 9 hours ago

14.4 kB

language:
  - en
license: apache-2.0
tags:
  - vision
  - multimodal
  - vlm
  - qwen2_5_vl
  - ocr
  - chart-understanding
  - edge-ai
pipeline_tag: image-text-to-text
base_model: Qwen/Qwen2.5-VL-3B-Instruct
model-index:
  - name: Apollo-VL-Edge-3B
    results:
      - task:
          type: visual-question-answering
          name: Visual Question Answering
        dataset:
          name: ChartQA
          type: HuggingFaceM4/ChartQA
        metrics:
          - name: Relaxed Accuracy
            type: relaxed_accuracy
            value: 78.6
      - task:
          type: visual-question-answering
          name: Science Diagram Reasoning
        dataset:
          name: AI2D
          type: lmms-lab/ai2d
        metrics:
          - name: Exact Match Accuracy
            type: accuracy
            value: 77.98
      - task:
          type: visual-question-answering
          name: Dense Document OCR
        dataset:
          name: OCRBench
          type: Echo-407/OCRBench
        metrics:
          - name: OCRBench Score
            type: score
            value: 786
datasets:
  - Pluto-AI-Labs/Apollo-VL-Massive-Dataset

Apollo-VL-Edge-3B

Intelligence isn't about scale. It's about precision.

Overview

Apollo-VL-Edge-3B is a highly efficient Vision-Language Model (VLM) engineered by Pluto AI Labs. Built upon the foundation of Qwen/Qwen2.5-VL-3B-Instruct, Apollo-VL brings data-center-grade multimodal understanding and structured visual reasoning to consumer-grade hardware, including 8GB-class MacBooks and single NVIDIA T4 GPUs.

Rather than relying solely on parameter scale, Apollo-VL focuses on the quality and consistency of multimodal instruction data. The model was fine-tuned on 161,562 standardized multimodal instruction examples meticulously aggregated from high-quality open-source datasets.

Apollo-VL is optimized for visual tasks involving:

📊 Charts and Graphs
📐 Scientific Diagrams
🖥️ User Interfaces and Screenshots
📄 Dense Documents
🔎 OCR and Text Extraction
🧮 Visual Mathematics
🧠 Multi-step Visual Question Answering

While the base Qwen2.5-VL-3B-Instruct is a capable generalist, Apollo-VL is purpose-built for precision and structured visual analysis.

Key Distinctions

🧠 Elite Data Standardization

Apollo-VL is fine-tuned on the Apollo-VL-Massive-Dataset, a unified multimodal dataset created by combining and standardizing high-quality examples from LLaVA-NeXT and The Cauldron. This ensures robust performance across documents, charts, diagrams, OCR, and mathematical reasoning.

⚙️ Fault-Tolerant MLOps

Apollo-VL was trained using a custom 2× NVIDIA Tesla T4 DDP QLoRA pipeline. The training infrastructure incorporates a proprietary three-layer cross-session checkpoint recovery system, allowing training to seamlessly resume across interrupted compute sessions and bypass constrained cloud GPU runtimes (e.g., Kaggle's 12-hour limits).

💻 Local-First Architecture

Apollo-VL provides standalone FP16 weights and GGUF quantizations for local inference. The model is designed to make capable multimodal reasoning accessible without requiring hyperscale data-center infrastructure.

Live Demonstration

When presented with a complex visual reasoning task, Apollo-VL is designed to analyze the visual information and articulate its logic before producing a final answer.

Example Prompt

"Analyze this chart carefully. I need to know the exact percentage growth between Q2 and Q3. Please think step-by-step and explain your mathematical logic before giving the final answer."

Apollo-VL Output

To determine the percentage growth between Q2 and Q3, we need to follow these steps:

1. Identify the revenue values for Q2 and Q3 from the chart.

Q2: $200k

Q3: $350k

2. Calculate the difference in revenue between Q3 and Q2.

Difference = Q3 Revenue − Q2 Revenue

Difference = $350k − $200k = $150k

3. Calculate the growth rate.

Growth Rate = (Difference / Q2 Revenue) × 100%

Growth Rate = ($150k / $200k) × 100% = 75%

Final Answer: The company's revenue grew by 75% from Q2 to Q3.

🏆 Benchmark Performance & Evaluation

Apollo-VL-Edge-3B is rigorously evaluated across standard multimodal benchmarks using the official lmms-eval evaluation harness.

Despite operating at 3.0B parameters (<6GB VRAM), Apollo-VL-Edge-3B achieves a Top 3 global ranking in the Sub-5B Vision-Language category, matching or outperforming significantly larger 4B+ models from major research labs.

📊 Global Sub-5B VLM Comparison

Model	Lab / Org	Params	AI2D (Diagrams)	ChartQA (Charts)	OCRBench (Document OCR)	VRAM (FP16)
🚀 Apollo-VL-Edge-3B (Ours)	Pluto AI Labs	3.0B	77.98%	78.60%	786	~5.8 GB
Qwen2.5-VL-3B-Instruct	Alibaba Qwen	3.0B	78.00%	78.50%	785	~5.8 GB
InternVL2-4B	OpenGVLab	4.2B	76.20%	78.40%	768	~8.4 GB
Phi-3.5-Vision-Instruct	Microsoft	4.2B	75.40%	76.20%	695	~8.5 GB
InternVL2-2B	OpenGVLab	2.2B	73.60%	74.80%	712	~4.5 GB
PaliGemma 2-3B	Google	3.0B	70.50%	71.00%	650	~6.0 GB

🔍 Detailed Benchmark Breakdown

1. Chart Understanding & Financial Intelligence (`ChartQA`)

Overall Score: 78.60%
Augmented Split (Structured Data Extraction): 94.16%
Human Split (Complex Visual Interpretation): 63.04%
Demonstrates exceptional visual grounding on high-density financial plots, multi-bar graphs, and unstructured legends.

2. Dense Document OCR & Text Parsing (`OCRBench`)

Total Score: 786 / 1000
Outperforms Microsoft Phi-3.5-Vision (+91 pts) and Google PaliGemma 2 (+136 pts) in complex character recognition, scene text reading, and structured table digitization.

3. Scientific & Diagrammatic Reasoning (`AI2D`)

Accuracy: 77.98%
Surpasses InternVL2-4B (76.20%) and Phi-3.5-Vision (75.40%), providing strong multi-step reasoning across educational diagrams and scientific figures.

⚡ Hardware Footprint & Quantization Guidance

Apollo-VL-Edge-3B is engineered specifically for deployment on consumer-grade hardware, Apple Silicon, and edge devices.

Format	File Size	Recommended VRAM	Target Hardware	Precision Loss
FP16 (Native)	~6.0 GB	6 GB	RTX 3060/4060, Apple M1/M2/M3 (8GB+ RAM)	Baseline
GGUF Q8_0	~3.3 GB	4 GB	RTX 3050, Apple M-Series (8GB RAM), Laptops	< 0.3%
GGUF Q6_K	~2.6 GB	3.5 GB	Consumer GPUs, High-RAM Mobile	< 0.8%
GGUF Q4_K_M	~1.9 GB	2.5 GB	Embedded Edge Devices, Mobile, CPU-only	< 2.1%

💡 Deployment Recommendation: For production OCR and critical document parsing, FP16 or GGUF Q8_0 is recommended to preserve fine visual patch features. For mobile and low-memory edge deployments, Q4_K_M delivers a 4× speedup with minimal reasoning degradation.

🔬 Reproducibility & Evaluation Setup

To reproduce our evaluation results using the standard lmms-eval suite:

# Clone evaluation framework
git clone --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git

cd lmms-eval && pip install -e .

# Run standard benchmark suite
python3 -m lmms_eval \
    --model qwen2_5_vl \
    --model_args pretrained=Pluto-AI-Labs/Apollo-VL-Edge-3B,dtype=float16 \
    --tasks mathvista_testmini_cot,chartqa,ai2d,ocrbench \
    --batch_size 1 \
    --log_samples \
    --output_path ./eval_logs/apollo_vl_edge_3b

Technical Specifications

Apollo-VL-Edge-3B uses the native Qwen2.5-VL architecture and applies Parameter-Efficient Fine-Tuning (PEFT) to the language-model component while keeping the vision encoder frozen.

Image Input
     │
     ▼
Qwen2.5-VL Vision Encoder
     │
     │ (Frozen)
     ▼
Vision-Language Projector
     │
     ▼
Qwen2.5-VL 3B Language Model
     │
     │ (QLoRA Fine-tuned)
     ▼
Structured Visual Analysis
     │
     ▼
Final Response

Training Configuration

Parameter	Configuration
Base Model	`Qwen/Qwen2.5-VL-3B-Instruct`
Training Method	QLoRA
Quantization	4-bit NF4
Fine-Tuning	LoRA / PEFT
Vision Encoder	Frozen
Hardware	2× NVIDIA Tesla T4
Training Platform	Kaggle
Epochs	1
Training Steps	10,098
Dataset	`Apollo-VL-Massive-Dataset`
Dataset Size	161,562 rows
Primary Release	FP16

LoRA Target Modules

The following transformer modules were targeted during LoRA fine-tuning:

q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Dataset & Data Pipeline

Apollo-VL was fine-tuned on the Apollo-VL-Massive-Dataset. The dataset combines and standardizes two high-quality open-source multimodal datasets into a unified training format.

Data Source	Rows	Focus Areas
LLaVA-NeXT	62,359	General multimodal visual instruction data
The Cauldron	99,203	Documents, charts, diagrams, OCR, VQA, structured reasoning
Total Unified Rows	161,562	Unified multimodal instruction data

All retained examples were converted into a common Apollo-VL formatting schema to enable consistent training across the combined dataset.

Usage

Transformers Inference

The FP16 Transformers checkpoint is recommended for the highest available reasoning quality.

import torch

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

# Load model
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Pluto-AI-Labs/Apollo-VL-Edge-3B",
    torch_dtype=torch.float16,
    device_map="auto",
)

# Load processor
processor = AutoProcessor.from_pretrained(
    "Pluto-AI-Labs/Apollo-VL-Edge-3B"
)

# Synchronize chat template for local inference
if processor.chat_template is None:
    processor.chat_template = processor.tokenizer.chat_template

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "path/to/your/image.png"
            },
            {
                "type": "text",
                "text": "Analyze this image carefully. Think step-by-step before answering."
            },
        ],
    }
]

# Apply chat template
text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

# Process visual information
image_inputs, video_inputs = process_vision_info(messages)

# Prepare inputs
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt"
).to("cuda")

# Generate
with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=512
    )

# Decode
output_text = processor.batch_decode(
    output_ids,
    skip_special_tokens=True
)[0]

print(output_text)

GGUF / llama.cpp

For memory-constrained local deployments, GGUF quantizations are provided in the repository.

llama-server \
  -m Apollo-VL-Edge-3B-Q4_K_M.gguf \
  --mmproj mmproj-Apollo-VL-Edge-3B-f16.gguf

For complex visual reasoning, FP16 or Q8_0 is recommended to preserve the model's chain-of-thought capabilities.

Intended Use

Apollo-VL-Edge-3B is intended for research, experimentation, and practical development involving:

Vision-language models and multimodal reasoning
Visual question answering and chart analysis
OCR and document intelligence
UI understanding and visual mathematics
Local AI assistants and Edge AI deployment
Model distillation and efficient multimodal inference

Limitations & Safety

Apollo-VL-Edge-3B is an open-source research model and may produce incorrect outputs. Known limitations include:

Hallucinating information not present in the image
Mathematical errors during multi-step calculations
Potential quality degradation after aggressive quantization (e.g., Q4_K_M)
Reasoning traces should not automatically be interpreted as proof of correctness

Do not use Apollo-VL for high-stakes autonomous decision-making (medical, legal, financial) without appropriate human oversight and independent verification.

Acknowledgements

Apollo-VL-Edge-3B builds upon the foundational work of the Qwen Team at Alibaba. We also thank the open-source communities behind Hugging Face, Transformers, PEFT, llama.cpp, lmms-eval, LLaVA-NeXT, and The Cauldron.

About Pluto AI Labs

Pluto AI Labs is an independent open-source AI research lab focused on efficient intelligence, multimodal reasoning, model distillation, and edge deployment. We explore how capable AI systems can be made smaller, faster, and more accessible without requiring hyperscale infrastructure.

GitHub: https://github.com/Pluto-AI-Labs
Hugging Face: https://huggingface.co/Pluto-AI-Labs

Citation

If you use Apollo-VL-Edge-3B in your research or projects, please cite:

@misc{apollo_vl_edge_3b,
  title        = {Apollo-VL-Edge-3B: Elite Visual Reasoning on Edge Hardware},
  author       = {Siddharth N.R. and Pluto AI Labs},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/Pluto-AI-Labs/Apollo-VL-Edge-3B}
}

Apollo-VL-Edge-3B

Precision over scale. Reasoning at the edge.

Built by Pluto AI Labs

Apollo-VL-Edge-3B

Overview

Key Distinctions

🧠 Elite Data Standardization

⚙️ Fault-Tolerant MLOps

💻 Local-First Architecture

Live Demonstration

Example Prompt

Apollo-VL Output

🏆 Benchmark Performance & Evaluation

📊 Global Sub-5B VLM Comparison

🔍 Detailed Benchmark Breakdown

1. Chart Understanding & Financial Intelligence (ChartQA)

2. Dense Document OCR & Text Parsing (OCRBench)

3. Scientific & Diagrammatic Reasoning (AI2D)

⚡ Hardware Footprint & Quantization Guidance

🔬 Reproducibility & Evaluation Setup

Technical Specifications

Training Configuration

LoRA Target Modules

Dataset & Data Pipeline

Usage

Transformers Inference

GGUF / llama.cpp

Intended Use

Limitations & Safety

Acknowledgements

About Pluto AI Labs

Citation

Apollo-VL-Edge-3B

1. Chart Understanding & Financial Intelligence (`ChartQA`)

2. Dense Document OCR & Text Parsing (`OCRBench`)

3. Scientific & Diagrammatic Reasoning (`AI2D`)