metadata
license: apache-2.0
language:
- ko
- en
pipeline_tag: visual-question-answering
tags:
- text2text-generation
base_model: google/deplot
Ko-Deplot
Ko-Deplot is a korean Visual-QA model based on the Google's Pix2Struct architecture. It was fine-tuned from Deplot, using korean chart image-text pairs.
Ko-Deplot은 Google의 Pix2Struct 구조를 기반으로 한 한국어 Visual-QA 모델입니다. Deplot 모델을 한국어 차트 이미지-텍스트 쌍 데이터셋을 이용하여 파인튜닝하였습니다.
- Developed by: NUUA
- Model type: Visual Question Answering
- License: apache-2.0
- Finetuned from model: google/deplot
Model Usage
You can run a prediction by querying an input image together with a question as follows:
from transformers import Pix2StructProcessor, Pix2StructForConditionalGeneration
from PIL import Image
processor = Pix2StructProcessor.from_pretrained('nuua/Ko-Deplot')
model = Pix2StructForConditionalGeneration.from_pretrained('nuua/Ko-Deplot')
IMAGE_PATH = "LOCAL_PATH_TO_IMAGE"
image = Image.open(IMAGE_PATH)
inputs = processor(images=image, text="Generate underlying data table of the figure below:", return_tensors="pt")
predictions = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(predictions[0], skip_special_tokens=True))
Training Details
Training Data
Synthetic chart data from three libraries were used:
Training Procedure
The model was first exposed to a short warmup stage, following its original paper. It was then trained using the chart data for 50,000 steps.
Technical Specifications
Hardware
Ko-Deplot was trained by using A100 80G.
Contact
Any questions and suggestions, please use the discussion tab. If you want to contact us directly, email robin@nuua.ai.