File size: 4,944 Bytes
52b71bc 0c2cb03 52b71bc 0c2cb03 4de7dd6 0c2cb03 52b71bc 4de7dd6 0c2cb03 4de7dd6 0c2cb03 4de7dd6 b12e33b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 | ---
library_name: peft
license: apache-2.0
base_model: ibm-granite/granite-vision-4.1-4b
pipeline_tag: image-text-to-text
tags:
- pytorch
- lora
- chart-understanding
---
<a id="top"></a>
<div align="center">
<h1>🚀 ChartLens @ CVPR 2026 DataMFM Chart Understanding Challenge</h1>
<p>
<b>Hao Liu</b><sup>1</sup>
<b>Ruping Cao</b><sup>1</sup>
<b>Kun Wang</b><sup>1</sup>
<b>Zhiran Li</b><sup>1</sup>
<b>Fan Liu</b><sup>2</sup>
<b>Yupeng Hu</b><sup>1</sup>
<b>Liqiang Nie</b><sup>3</sup>
</p>
<p>
<sup>1</sup>Shandong University<br>
<sup>2</sup>Southeast University<br>
<sup>3</sup>Harbin Institute of Technology (Shenzhen)
</p>
</div>
These are the official implementation resources, model weights, and prediction files for **ChartLens**, the champion solution for **DataMFM Challenge Track 2: Chart Understanding** at CVPR 2026.
🔗 **Paper:** [ChartLens: A Dual-Branch Framework for Chart Data Correction and Factual Summary Refinement](https://huggingface.co/papers/2606.10640)
🔗 **GitHub Repository:** [iLearnLab/CVPRW26-ChartLens](https://github.com/iLearnLab/CVPRW26-ChartLens)
🔗 **Challenge Page:** [DataMFM Challenge](https://datamfm.github.io/challenge.html)
---
## 📌 Model Information
### 1. Model Name
**ChartLens: A Dual-Branch Framework for Chart Data Correction and Factual Summary Refinement**
### 2. Task Type & Applicable Tasks
- **Task Type:** Chart Understanding / Multimodal Document Understanding
- **Applicable Tasks:** Chart-to-CSV extraction and chart-to-summary generation from chart images.
### 3. Project Introduction
Chart understanding requires models to recover structured chart data and generate faithful natural-language summaries from chart images. **ChartLens** addresses these complementary goals with a dual-branch, verification-guided correction framework.
> 💡 **Method Highlight:** ChartLens combines Granite-Vision-4.1-4B LoRA adaptation with two correction branches: **Structure-Aware CSV Verification and Correction (SAVC)** for reliable table recovery, and **Text-Retention-Guided Summary Refinement (TRSR)** for OCR-assisted factual summary repair. SAVC checks structure, completeness, and numerical accuracy, while TRSR preserves visible chart text such as titles, legends, annotations, sources, and numerical evidence.
### 4. Training Data Source
- Released ChartNet-based training data for LoRA adaptation.
- DataMFM Challenge chart understanding splits, including `real` and `synthetic` chart images.
### 5. Challenge Results
| Method | CSV Numeric F1 | CSV Structural Score | Summary ROUGE-L | Summary Numeric Fact F1 | Overall |
|--------|---------------:|---------------------:|----------------:|------------------------:|--------:|
| **ChartLens (Ours)** | **80.62** | **75.66** | **45.57** | **74.55** | **69.10** |
ChartLens ranked **1st place** on DataMFM Challenge Track 2.
---
## 🚀 Usage & Basic Inference
### Step 1: Prepare the Environment
Clone the GitHub repository and set up the Conda environment:
```bash
git clone https://github.com/iLearnLab/CVPRW26-ChartLens.git
cd CVPRW26-ChartLens
```
```bash
conda create -n chartlens python=3.10 -y
conda activate chartlens
pip install -r requirements.txt
```
### Step 2: Data & Weights Preparation
1. **Challenge Data:** Use the datasets and splits released by the [DataMFM Challenge](https://datamfm.github.io/challenge.html). The chart understanding track contains `real` and `synthetic` splits.
2. **ChartLens Checkpoints:** Download the model weights from this Hugging Face repository.
3. **Granite Vision Backbone:** Prepare the Granite-Vision-4.1-4B backbone and update the local `--model_path` argument when running inference.
To prepare ChartNet SFT data for LoRA training:
```bash
python code/load_chartnet_500.py \
--out_dir Fine-tuning/Dataset/raw \
--num_samples 500
python code/build_chartnet_sft.py \
--gt_path Fine-tuning/Dataset/raw/gt.jsonl \
--image_dir Fine-tuning/Dataset/raw/images \
--out_dir Fine-tuning/Dataset/sft \
--csv_repeat 2 \
--summary_repeat 1
```
### Step 3: Run Granite Vision + LoRA Inference
```bash
python code/infer_granite_with_lora.py \
--image_root /path/to/data \
--out_root /path/to/output \
--model_path /path/to/granite-vision-4.1-4b \
--lora_path /path/to/chartlens_lora \
--gpu_id 0 \
--splits real synthetic
```
Use `code/infer_chartnet_granite.py` for base Granite Vision inference without a LoRA adapter.
---
## 📝⭐️ Citation
If you find this project useful for your research, please consider citing:
```bibtex
@article{liu2026chartlens,
title={ChartLens: A Dual-Branch Framework for Chart Data Correction and Factual Summary Refinement},
author={Liu, Hao and Cao, Ruping and Wang, Kun and Li, Zhiran and Liu, Fan and Hu, Yupeng and Nie, Liqiang},
journal={arXiv preprint arXiv:2606.10640},
year={2026}
}
``` |