Instructions to use Junteng/Chart_CLIP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Junteng/Chart_CLIP with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-feature-extraction", model="Junteng/Chart_CLIP")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Junteng/Chart_CLIP", dtype="auto") - Notebooks
- Google Colab
- Kaggle
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -1,3 +1,33 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# CLIP Model for Chart Understanding
|
| 2 |
+
|
| 3 |
+
This repository contains the CLIP model implementation from our paper "[On the Perception Bottleneck of VLMs for Chart Understanding](https://arxiv.org/abs/2503.18435)".
|
| 4 |
+
|
| 5 |
+
## Overview
|
| 6 |
+
|
| 7 |
+
This CLIP model is specifically trained to address the perception bottleneck in Vision Language Models (VLMs) when processing and understanding charts and visualizations. Our work explores and aims to improve how CLIP effect its LVLMs.
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
## Model Details
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
- Model architecture: trained from openai/clip-vit-large-patch14-336
|
| 14 |
+
- Training data: from our collected and synthetic hard negatives chart data([Vision4Chart Dataset](https://huggingface.co/datasets/Junteng/Vision4Chart))
|
| 15 |
+
- Training method: NegCLIP Training
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
## Citation
|
| 19 |
+
|
| 20 |
+
If you find this model useful in your research, please consider citing our paper:
|
| 21 |
+
|
| 22 |
+
```bibtex
|
| 23 |
+
@misc{liu2025perceptionbottleneckvlmschart,
|
| 24 |
+
title={On the Perception Bottleneck of VLMs for Chart Understanding},
|
| 25 |
+
author={Junteng Liu and Weihao Zeng and Xiwen Zhang and Yijun Wang and Zifei Shan and Junxian He},
|
| 26 |
+
year={2025},
|
| 27 |
+
eprint={2503.18435},
|
| 28 |
+
archivePrefix={arXiv},
|
| 29 |
+
primaryClass={cs.CV},
|
| 30 |
+
url={https://arxiv.org/abs/2503.18435},
|
| 31 |
+
}
|
| 32 |
+
```
|
| 33 |
+
|