--- base_model: - Qwen/Qwen2-7B - google/siglip-so400m-patch14-384 license: apache-2.0 ---

Logo Beyond Filtering:
Adaptive Image-Text Quality Enhancement for MLLM Pretraining

arxiv: https://arxiv.org/abs/2410.16166 github: https://github.com/hanhuang22/AITQE [2024.10.12] Release the inference code and pre-trained model of AITQE. We propose the **A**daptive **I**mage-**T**ext **Q**uality **E**nhancer, **AITQE**, a model that dynamically assesses and enhances the quality of image-text pairs. The conventional method (a) discards low-quality samples in raw data, reducing the amount of pretraining data, while our AITQE (b) enhances low-quality samples, retaining the same volume of data for MLLMs pretraining. illus Specifically, for pairs exhibiting low quality-such as low semantic similarity between modalities or subpar linguistic quality, AITQE performs text rewriting, generating high-quality text based on the input image and the raw low-quality text. Use the code from github: ```bash python inference.py \ --model_path /path/to/AITQE \ --output_all --gpu_id 0 \ --image_path ./figs/test.png \ --caption "Some random text to the image like this is a test" ``` and get the following output: 
{"Recaption": "A man stands in front of a checklist of customer service questions, including 'Do you take each customer seriously?' and 'Do you qualify customers properly?'", "Overall Score": "2", "Overall Explanation": "The caption is vague and does not accurately describe the image or its content. It lacks detail and relevance to the checklist shown in the image.", "Text Quality Score": 3, "Text Quality Explanation": "The caption is grammatically correct but lacks clarity and relevance to the image. It is vague and does not provide a meaningful description.", "Image-Text Matching Score": 2, "Image-Text Matching Explanation": "The caption does not accurately describe the image, which features a checklist of customer service questions. The caption is unrelated to the content of the image.", "Object Detail Score": 2, "Object Detail Explanation": "The caption does not provide any details about the objects in the image, such as the checklist or the person in the background.", "Semantic Understanding Score": 2, "Semantic Understanding Explanation": "The caption fails to convey any understanding of the image's context or purpose, which is about customer service evaluation.", "Text/Chart Description Score": 2, "Text/Chart Description Explanation": "The caption does not describe the text in the image, which is a checklist of customer service questions."}