| | --- |
| | license: apache-2.0 |
| | --- |
| | # Intruduction |
| | We introduce Xmodel-VLM, a cutting-edge multimodal vision language model. It is designed for efficient deployment on consumer GPU servers. |
| | Our work directly confronts a pivotal industry issue by grappling with the prohibitive service costs that hinder the broad adoption of large-scale multimodal systems. |
| |
|
| | Refer to [our paper](https://arxiv.org/pdf/2405.09215) and [github](https://github.com/XiaoduoAILab/XmodelVLM) for more details! |
| |
|
| | To use Xmodel_VLM for the inference, all you need to do is to input a few lines of codes as demonstrated below. **However, please make sure that you are using the latest code and related virtual environments.** |
| | |
| | ## Inference example |
| | ``` |
| | import sys |
| | import torch |
| | import argparse |
| | from PIL import Image |
| | from pathlib import Path |
| | import time |
| | sys.path.append(str(Path(__file__).parent.parent.resolve())) |
| | |
| | from xmodelvlm.model.xmodelvlm import load_pretrained_model |
| | from xmodelvlm.conversation import conv_templates, SeparatorStyle |
| | from xmodelvlm.utils import disable_torch_init, process_images, tokenizer_image_token, KeywordsStoppingCriteria |
| | from xmodelvlm.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN |
| | |
| | def inference_once(args): |
| | disable_torch_init() |
| | model_name = args.model_path.split('/')[-1] |
| | tokenizer, model, image_processor, context_len = load_pretrained_model(args.model_path, args.load_8bit, args.load_4bit) |
| | |
| | images = [Image.open(args.image_file).convert("RGB")] |
| | images_tensor = process_images(images, image_processor, model.config).to(model.device, dtype=torch.float16) |
| | |
| | conv = conv_templates[args.conv_mode].copy() |
| | conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\n" + args.prompt) |
| | conv.append_message(conv.roles[1], None) |
| | prompt = conv.get_prompt() |
| | stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2 |
| | # Input |
| | input_ids = (tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).cuda()) |
| | stopping_criteria = KeywordsStoppingCriteria([stop_str], tokenizer, input_ids) |
| | # Inference |
| | with torch.inference_mode(): |
| | start_time = time.time() |
| | output_ids = model.generate( |
| | input_ids, |
| | images=images_tensor, |
| | do_sample=True if args.temperature > 0 else False, |
| | temperature=args.temperature, |
| | top_p=args.top_p, |
| | num_beams=args.num_beams, |
| | max_new_tokens=args.max_new_tokens, |
| | use_cache=True, |
| | stopping_criteria=[stopping_criteria], |
| | ) |
| | end_time = time.time() |
| | execution_time = end_time-start_time |
| | print("the execution time (secend): ", execution_time) |
| | # Result-Decode |
| | input_token_len = input_ids.shape[1] |
| | n_diff_input_output = (input_ids != output_ids[:, :input_token_len]).sum().item() |
| | if n_diff_input_output > 0: |
| | print(f"[Warning] {n_diff_input_output} output_ids are not the same as the input_ids") |
| | outputs = tokenizer.batch_decode(output_ids[:, input_token_len:], skip_special_tokens=True)[0] |
| | outputs = outputs.strip() |
| | if outputs.endswith(stop_str): |
| | outputs = outputs[: -len(stop_str)] |
| | print(f"🚀 {model_name}: {outputs.strip()}\n") |
| | if __name__ == '__main__': |
| | model_path = "XiaoduoAILab/Xmodel_VLM" # model weight file |
| | image_file = "assets/demo.jpg" # image file |
| | prompt_str = "Who is the author of this book?\nAnswer the question using a single word or phrase." |
| | # (or) What is the title of this book? |
| | # (or) Is this book related to Education & Teaching? |
| | |
| | args = type('Args', (), { |
| | "model_path": model_path, |
| | "image_file": image_file, |
| | "prompt": prompt_str, |
| | "conv_mode": "v1", |
| | "temperature": 0, |
| | "top_p": None, |
| | "num_beams": 1, |
| | "max_new_tokens": 512, |
| | "load_8bit": False, |
| | "load_4bit": False, |
| | })() |
| | |
| | inference_once(args) |
| | ``` |
| | |
| | <center> |
| | |
| | Prompt: **Who is the author of this book?\nAnswer the question using a single word or phrase.** |
| |  |
| | Author: **Susan Wise Bauer** |
| |
|
| | </center> |
| |
|
| | ## Evaluation |
| |
|
| | We evaluate the multimodal performance across a variety of datasets: **VizWiz**, **SQA<sup>I</sup>**, **VQA<sup>T</sup>**, **POPE**, **GQA**, **MMB**, **MMB<sup>CN</sup>** |
| | , **MM-Vet**, and **MME**. Our analysis, as depicted In the following table. |
| |
|
| | | Method | LLM | Res. | VizWiz | SQA | VQA | POPE | GQA | MMB | MMB<sup>CN</sup> | MM-Vet | MME | |
| | |:--------------:|:----------------:|:----:|:------:|:----:|:----:|:----:|:----:|:----:|:--------:|:------:|:------:| |
| | | Openflamingo | MPT-7B | 336 | - | - | 33.6 | - | - | 4.6 | - | - | - | |
| | | BLIP-2 | Vicuna-13B | 224 | - | 61.0 | 42.5 | 85.3 | 41.0 | - | - | - | 1293.8 | |
| | | MiniGPT-4 | Vicuna-7B | 224 | - | - | - | - | 32.2 | 23.0 | - | - | 581.7 | |
| | | InstructBLIP | Vicuna-7B | 224 | - | 60.5 | 50.1 | - | 49.2 | - | - | - | - | |
| | | InstructBLIP | Vicuna-13B | 224 | - | 63.1 | 50.7 | 78.9 | 49.5 | - | - | - | 1212.8 | |
| | | Shikra | Vicuna-13B | 224 | - | - | - | - | - | 58.8 | - | - | - | |
| | | Qwen-VL | Qwen-7B | 448 | - | 67.1 | 63.8 | - | 59.3 | 38.2 | - | - | 1487.6 | |
| | | MiniGPT-v2 | LLaMA-7B | 448 | - | - | - | - | 60.3 | 12.2 | - | - | - | |
| | | LLaVA-v1.5-13B | Vicuna-13B | 336 | 53.6 | 71.6 | 61.3 | 85.9 | 63.3 | 67.7 | 63.6 | 35.4 | 1531.3 | |
| | | MobileVLM 1.7 | MobileLLaMA 1.4B | 336 | 26.3 | 54.7 | 41.5 | 84.5 | 56.1 | 53.2 | 16.67 | 21.7 | 1196.2 | |
| | | **Xmodel-VLM** | **Xmodel-LM 1.1B** | **336** | **41.7** | **53.3** | **39.9** | **85.9** | **58.3** | **52.0** | **45.7** | **21.8** | **1250.7** | |
| |
|
