README.md

metadata

pipeline_tag: image-text-to-text
library_name: transformers
license: cc-by-nc-4.0
tags:
  - code-generation

ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation (ACL 2025 Main)

This repository is the official implementation of ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation.

About ChartCoder

ChartCoder is the first dedicated multimodal large language model (MLLM) designed for chart-to-code generation. It leverages Code LLMs as its language backbone to significantly enhance the executability of generated code. This model addresses two key challenges in chart interpretation:

1. Low executability and poor detail restoration in generated code from existing MLLMs.
2. Lack of large-scale and diverse training data for chart-to-code tasks.

To overcome these, ChartCoder introduces:

• Chart2Code-160k: The first large-scale and diverse dataset for chart-to-code generation.
• Snippet-of-Thought (SoT): A method that transforms direct chart-to-code generation into a step-by-step process.

With only 7B parameters, ChartCoder surpasses existing open-source MLLMs on chart-to-code benchmarks, achieving superior chart restoration and code executability.

Notes

1. ChartCoder is tested on the new version of Chartmimic, which contains 600 samples. The iclr version of ChartMimic is https://huggingface.co/datasets/ChartMimic/ChartMimic/blob/main/dataset-iclr.tar.gz.
2. The code we utilize for evaluating is the Supplementary Material of https://openreview.net/forum?id=sGpCzsfd1K.

All the results (including the baseline and our models) in the Table 3 in the paper is evaluated based on above two settings. When conducting the assessment in other settings, there may be performance differences. If you want to replicate the performance in the paper, it is recommended to achieve it under the aforementioned settings.

News

[2025.5.17] ChartCoder has been accepted by ACL 2025 Main.

[2025.3.13] We have upload our dataset Chart2Code-160k(HF) to Huggingface.

[2025.2.19] We have released our dataset Chart2Code-160k to ModelScope.

[2025.1.16] We have updated our data generation code data_generator, built on Multi-modal-Self-instruct. Please follow their instructions and our code to generate the <chart, code> data pairs.

Overview

Installation

To get started with ChartCoder, clone the repository and set up the environment:

git clone https://github.com/thunlp/ChartCoder.git
cd ChartCoder
conda create -n chartcoder python=3.10 -y
conda activate chartcoder
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

For training, additional packages are required:

pip install -e ".[train]"
pip install flash-attn --no-build-isolation

Models

Model	Download Link
MLP Connector	projector
ChartCoder	ChartCoder

The MLP Connector is our pre-trained MLP weights, which you could directly use for SFT.

Data

Dataset	Download Link
Chart2Code-160k	HuggingFace
Chart2Code-160k	ModelScope

Training

The whole training process consists of two stages. To train the ChartCoder, siglip-so400m-patch14-384 and deepseek-coder-6.7b-instruct should be downloaded first.

For Pre-training, run:

bash scripts/train/pretrain_siglip.sh

For SFT, run:

bash scripts/train/finetune_siglip_a4.sh

Please change the model path to your local path. See the corresponding .sh file for details. We also provide other training scripts, such as using CLIP _clip and multiple machines _m. See scripts/train for further information.

Inference (Sample Usage)

You can easily use ChartCoder with the Hugging Face transformers library. Ensure you have transformers and torch installed.

from transformers import AutoProcessor, AutoModelForCausalLM
import torch
from PIL import Image
import requests
from io import BytesIO

# Load model and processor
model_id = "xxxllz/ChartCoder" # The model's Hugging Face ID
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Example image (replace with a real chart image path or URL)
# For demonstration, let's use a placeholder image. In a real scenario, load your chart image.
# Example: image = Image.open("path/to/your/chart_image.png").convert("RGB")
# Or from a URL:
image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/chart_example.png"
image = Image.open(BytesIO(requests.get(image_url).content)).convert("RGB")

# Define your prompt for chart-to-code generation
prompt = "Generate Python code to recreate the given chart. Provide only the code, no explanations."

# Prepare messages in the chat template format
messages = [
    {"role": "user", "content": f"<image>\
{prompt}"}
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# Move inputs to GPU if available
inputs = processor(text=text, images=image, return_tensors="pt").to(model.device)

# Generate code
output_ids = model.generate(**inputs, max_new_tokens=512)
output_text = processor.batch_decode(output_ids, skip_special_tokens=True)[0].strip()

# Print the generated code
print(output_text)

Results

Please refer to our paper for detailed performance on ChartMimic, Plot2Code and ChartX benchmarks. Thanks for these contributions to the chart-to-code field.

Contact

For any questions, you can contact 2429527z@gmail.com.

Citation

If you find this work useful, consider giving this repository a star ⭐️ and citing 📝 our paper as follows:

@misc{zhao2025chartcoderadvancingmultimodallarge,
      title={ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation}, 
      author={Xuanle Zhao and Xianzhen Luo and Qi Shi and Chi Chen and Shuo Wang and Wanxiang Che and Zhiyuan Liu and Maosong Sun},
      year={2025},
      eprint={2501.06598},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2501.06598}, 
}

Acknowledgement

The code is based on the LLaVA-NeXT. Thanks for these great works and open sourcing!