Instructions to use iLearn-Lab/CVPRW26-ChartLens with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use iLearn-Lab/CVPRW26-ChartLens with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("/data2/caoruping/DataMFM/models/granite-vision-4.1-4b") model = PeftModel.from_pretrained(base_model, "iLearn-Lab/CVPRW26-ChartLens") - Notebooks
- Google Colab
- Kaggle
Add pipeline tag, library name and link to paper
#1
by nielsr HF Staff - opened
README.md
CHANGED
|
@@ -1,8 +1,12 @@
|
|
| 1 |
---
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
-
|
|
|
|
| 4 |
tags:
|
| 5 |
- pytorch
|
|
|
|
|
|
|
| 6 |
---
|
| 7 |
|
| 8 |
<a id="top"></a>
|
|
@@ -26,9 +30,9 @@ tags:
|
|
| 26 |
</p>
|
| 27 |
</div>
|
| 28 |
|
| 29 |
-
These are the official implementation resources, model weights, and prediction files for **ChartLens**,
|
| 30 |
|
| 31 |
-
π **Paper:** [
|
| 32 |
π **GitHub Repository:** [iLearnLab/CVPRW26-ChartLens](https://github.com/iLearnLab/CVPRW26-ChartLens)
|
| 33 |
π **Challenge Page:** [DataMFM Challenge](https://datamfm.github.io/challenge.html)
|
| 34 |
|
|
@@ -114,104 +118,6 @@ python code/infer_granite_with_lora.py \
|
|
| 114 |
|
| 115 |
Use `code/infer_chartnet_granite.py` for base Granite Vision inference without a LoRA adapter.
|
| 116 |
|
| 117 |
-
### Step 4: SAVC CSV Correction
|
| 118 |
-
|
| 119 |
-
```bash
|
| 120 |
-
export OPENAI_API_KEY="..."
|
| 121 |
-
|
| 122 |
-
python code/calibrate_baseline_with_ai.py \
|
| 123 |
-
--split all \
|
| 124 |
-
--baseline_root /path/to/baseline_predictions \
|
| 125 |
-
--image_root /path/to/data \
|
| 126 |
-
--output_root /path/to/savc_output \
|
| 127 |
-
--base_url "https://your-openai-compatible-endpoint" \
|
| 128 |
-
--model gemini-3.5-flash \
|
| 129 |
-
--threshold 85
|
| 130 |
-
```
|
| 131 |
-
|
| 132 |
-
`--baseline_root` should contain split directories such as `real/` and `synthetic/`, each with `chart2csv_predictions.jsonl` and `chart2summary_predictions.jsonl`.
|
| 133 |
-
|
| 134 |
-
### Step 5: TRSR Summary Refinement
|
| 135 |
-
|
| 136 |
-
```bash
|
| 137 |
-
python code/ocr.py \
|
| 138 |
-
--real_images /path/to/data/real/images \
|
| 139 |
-
--synthetic_images /path/to/data/synthetic/images \
|
| 140 |
-
--real_summary /path/to/baseline/real/chart2summary_predictions.jsonl \
|
| 141 |
-
--synthetic_summary /path/to/baseline/synthetic/chart2summary_predictions.jsonl \
|
| 142 |
-
--output_dir /path/to/ocr_text_copy_coverage \
|
| 143 |
-
--threshold 0.8
|
| 144 |
-
|
| 145 |
-
export AIGCBEST_API_KEY="..."
|
| 146 |
-
|
| 147 |
-
python code/repair_summary.py \
|
| 148 |
-
--split all \
|
| 149 |
-
--workers 20 \
|
| 150 |
-
--ocr_eval_root /path/to/ocr_text_copy_coverage \
|
| 151 |
-
--output_root /path/to/trsr_output
|
| 152 |
-
```
|
| 153 |
-
|
| 154 |
-
### Step 6: Training (Optional)
|
| 155 |
-
|
| 156 |
-
Train the LoRA adapter on the prepared ChartNet SFT data:
|
| 157 |
-
|
| 158 |
-
```bash
|
| 159 |
-
python code/train_lora_chartnet.py \
|
| 160 |
-
--model_path /path/to/granite-vision-4.1-4b \
|
| 161 |
-
--train_jsonl Fine-tuning/Dataset/sft/train.jsonl \
|
| 162 |
-
--val_jsonl Fine-tuning/Dataset/sft/val.jsonl \
|
| 163 |
-
--output_dir Fine-tuning/FT/model/granite_chartnet_lora_bs2 \
|
| 164 |
-
--gpu_id 0 \
|
| 165 |
-
--epochs 2 \
|
| 166 |
-
--batch_size 1 \
|
| 167 |
-
--grad_accum 8
|
| 168 |
-
```
|
| 169 |
-
|
| 170 |
-
---
|
| 171 |
-
|
| 172 |
-
## π¦ Submission Format
|
| 173 |
-
|
| 174 |
-
For DataMFM Track 2, organize the final predictions as:
|
| 175 |
-
|
| 176 |
-
```bash
|
| 177 |
-
submission.zip
|
| 178 |
-
βββ real/
|
| 179 |
-
β βββ chart2csv_predictions.jsonl
|
| 180 |
-
β βββ chart2summary_predictions.jsonl
|
| 181 |
-
βββ synthetic/
|
| 182 |
-
βββ chart2csv_predictions.jsonl
|
| 183 |
-
βββ chart2summary_predictions.jsonl
|
| 184 |
-
```
|
| 185 |
-
|
| 186 |
-
Each CSV prediction line:
|
| 187 |
-
|
| 188 |
-
```json
|
| 189 |
-
{"imagename": "example.png", "predicted_csv": "Header A,Header B\nA,1\nB,2"}
|
| 190 |
-
```
|
| 191 |
-
|
| 192 |
-
Each summary prediction line:
|
| 193 |
-
|
| 194 |
-
```json
|
| 195 |
-
{"imagename": "example.png", "predicted_summary": "One paragraph summary grounded in the chart."}
|
| 196 |
-
```
|
| 197 |
-
|
| 198 |
-
---
|
| 199 |
-
|
| 200 |
-
## β οΈ Limitations & Notes
|
| 201 |
-
|
| 202 |
-
**Disclaimer:** This framework and its model weights are intended for **academic research purposes only**.
|
| 203 |
-
|
| 204 |
-
- Chart-to-CSV extraction may still struggle with dense layouts, asymmetric legends, or adjacent semantic-column misalignment.
|
| 205 |
-
- Summary refinement depends on OCR quality; OCR errors can affect text-retention scoring and repair decisions.
|
| 206 |
-
- GPU execution is expected for Granite Vision inference and LoRA training.
|
| 207 |
-
- API-backed correction scripts require valid credentials and an OpenAI-compatible endpoint.
|
| 208 |
-
|
| 209 |
-
---
|
| 210 |
-
|
| 211 |
-
## π€ Acknowledgements & Contact
|
| 212 |
-
|
| 213 |
-
- **Contact:** If you have any questions or encounter issues, feel free to contact Hao Liu at liuh90210@gmail.com or Ruping Cao at caoruping657@gmail.com.
|
| 214 |
-
|
| 215 |
---
|
| 216 |
|
| 217 |
## πβοΈ Citation
|
|
|
|
| 1 |
---
|
| 2 |
+
library_name: peft
|
| 3 |
license: apache-2.0
|
| 4 |
+
base_model: ibm-granite/granite-vision-4.1-4b
|
| 5 |
+
pipeline_tag: image-text-to-text
|
| 6 |
tags:
|
| 7 |
- pytorch
|
| 8 |
+
- lora
|
| 9 |
+
- chart-understanding
|
| 10 |
---
|
| 11 |
|
| 12 |
<a id="top"></a>
|
|
|
|
| 30 |
</p>
|
| 31 |
</div>
|
| 32 |
|
| 33 |
+
These are the official implementation resources, model weights, and prediction files for **ChartLens**, the champion solution for **DataMFM Challenge Track 2: Chart Understanding** at CVPR 2026.
|
| 34 |
|
| 35 |
+
π **Paper:** [ChartLens: A Dual-Branch Framework for Chart Data Correction and Factual Summary Refinement](https://huggingface.co/papers/2606.10640)
|
| 36 |
π **GitHub Repository:** [iLearnLab/CVPRW26-ChartLens](https://github.com/iLearnLab/CVPRW26-ChartLens)
|
| 37 |
π **Challenge Page:** [DataMFM Challenge](https://datamfm.github.io/challenge.html)
|
| 38 |
|
|
|
|
| 118 |
|
| 119 |
Use `code/infer_chartnet_granite.py` for base Granite Vision inference without a LoRA adapter.
|
| 120 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
---
|
| 122 |
|
| 123 |
## πβοΈ Citation
|