Improve model card: Add pipeline tag, library, project page, and usage example

README.md

@@ -1,8 +1,21 @@
 ---
 license: apache-2.0
+pipeline_tag: image-text-to-text
+library_name: transformers
 ---
+
 # VPP-LLaVA Model Card
 
+## Abstract
+
+Although Multimodal Large Language Models (MLLMs) excel at various image-related tasks, they encounter challenges in precisely aligning coordinates with spatial information within images, particularly in position-aware tasks such as visual grounding. This limitation arises from two key factors. First, MLLMs lack explicit spatial references, making it difficult to associate textual descriptions with precise image locations. Second, their feature extraction processes prioritize global context over fine-grained spatial details, leading to weak localization capability. To address these issues, we introduce VPP-LLaVA, an MLLM enhanced with Visual Position Prompt (VPP) to improve its grounding capability. VPP-LLaVA integrates two complementary mechanisms: the global VPP overlays a learnable, axis-like tensor onto the input image to provide structured spatial cues, while the local VPP incorporates position-aware queries to support fine-grained localization. To effectively train our model with spatial guidance, we further introduce VPP-SFT, a curated dataset of 0.6M high-quality visual grounding samples. Designed in a compact format, it enables efficient training and is significantly smaller than datasets used by other MLLMs (e.g., ~21M samples in MiniGPT-v2), yet still provides a strong performance boost. The resulting model, VPP-LLaVA, not only achieves state-of-the-art results on standard visual grounding benchmarks but also demonstrates strong zero-shot generalization to challenging unseen datasets. The code and dataset are available at [this GitHub repository](https://github.com/WayneTomas/VPP-LLaVA).
+
+## Links
+
+*   **Paper**: [Visual Position Prompt for MLLM based Visual Grounding](https://arxiv.org/abs/2503.15426)
+*   **Code (GitHub)**: [https://github.com/WayneTomas/VPP-LLaVA](https://github.com/WayneTomas/VPP-LLaVA)
+*   **Project Page**: [https://osatlas.github.io/](https://osatlas.github.io/)
+
 ## Model Details
 
 **Model Type**: VPP-LLaVA is an enhanced multimodal model built upon the LLaVA architecture. It is designed to improve visual grounding capabilities by incorporating Visual Position Prompts (VPP) into the original LLaVA model. LLaVA itself is an open-source chatbot trained by fine-tuning LLaMA/Vicuna on GPT-generated multimodal instruction-following data. It is an auto-regressive language model based on the transformer architecture.
@@ -11,7 +24,7 @@ license: apache-2.0
 
 **Paper or Resources for More Information**:
 - Original LLaVA: [LLaVA: Large Language and Vision Assistant](https://llava-vl.github.io/)
-- VPP-LLaVA Enhancements: [Visual Position Prompt for MLLM based Visual Grounding](https://arxiv.org/pdf/2503.15426)
+- VPP-LLaVA Enhancements: [Visual Position Prompt for MLLM based Visual Grounding](https://arxiv.org/abs/2503.15426)
 
 ## License
 
@@ -27,6 +40,53 @@ For questions or comments about VPP-LLaVA, please refer to the GitHub repository
 
 **Primary Intended Users**: The primary intended users of VPP-LLaVA are researchers and hobbyists in the fields of computer vision, natural language processing, machine learning, and artificial intelligence, who are interested in exploring advanced multimodal models and improving visual grounding performance.
 
+## Usage
+
+You can use `VPP-LLaVA-13b` with the `transformers` library. The model is designed for visual grounding tasks.
+
+```python
+from transformers import AutoProcessor, AutoModelForCausalLM
+from PIL import Image
+import requests
+import torch
+
+# Load model and processor.
+# Note: trust_remote_code=True is required for custom model architectures.
+model_id = "wayneicloud/VPP-LLaVA-13b" # Replace with the actual model ID on Hugging Face
+processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16, # Adjust dtype based on your hardware and model's config
+    device_map="auto",
+    trust_remote_code=True
+)
+
+# Prepare an example image
+image_url = "https://llava-vl.github.io/static/images/a-man-and-a-woman.jpg"
+image = Image.open(requests.get(image_url, stream=True).raw).convert('RGB')
+
+# Define the text instruction for visual grounding
+prompt_text = "What is the bounding box of the woman? Use <box> and </box> tokens."
+messages = [
+    {"role": "user", "content": prompt_text}
+]
+
+# Process inputs (image + text)
+# The processor automatically handles the image token insertion and chat templating.
+inputs = processor(text=messages, images=image, return_tensors="pt").to(model.device)
+
+# Generate response
+with torch.no_grad():
+    output_ids = model.generate(**inputs, max_new_tokens=100)
+
+# Decode and print the generated text
+generated_text = processor.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
+print(f"Generated response: {generated_text}")
+# Expected output might be in the format like:
+# Generated response: The bounding box of the woman is <box> [x1, y1, x2, y2] </box>
+
+```
+
 ## Training Dataset
 
 The training dataset for VPP-LLaVA is the VPP-SFT dataset, which is available on Hugging Face: [VPP-SFT](https://huggingface.co/datasets/wayneicloud/VPP-SFT/tree/main). This dataset contains about 0.6M high-quality visual grounding samples, designed to efficiently train the model for improved visual grounding tasks. Please refer to our [VPP-LLaVA](https://github.com/WayneTomas/VPP-LLaVA) for more details.
@@ -42,7 +102,7 @@ The evaluation dataset for VPP-LLaVA includes the following benchmarks:
 
 ## Model Enhancements
 
-VPP-LLaVA introduces Visual Position Prompts (VPP) to the original LLaVA architecture to enhance visual grounding capabilities. The enhancements are based on the research presented in the paper [Visual Position Prompt for MLLM based Visual Grounding](https://arxiv.org/pdf/2503.15426). The VPP mechanism includes:
+VPP-LLaVA introduces Visual Position Prompts (VPP) to the original LLaVA architecture to enhance visual grounding capabilities. The enhancements are based on the research presented in the paper [Visual Position Prompt for MLLM based Visual Grounding](https://arxiv.org/abs/2503.15426). The VPP mechanism includes:
 - **Global VPP**: Provides a global position reference by overlaying learnable, axis-like embeddings onto the input image.
 - **Local VPP**: Focuses on fine-grained localization by incorporating position-aware queries that suggest probable object locations.
 
@@ -52,4 +112,23 @@ These enhancements enable VPP-LLaVA to achieve state-of-the-art performance in v
 
 VPP-LLaVA demonstrates remarkable zero-shot performance on unseen datasets, particularly in challenging scenarios involving part-object and multi-object situations. This capability is crucial for real-world applications where the model may encounter previously unseen objects or complex scenes. The model's ability to generalize and accurately ground visual references in these scenarios highlights its robustness and adaptability.
 
-VPP-LLaVA paper link: https://arxiv.org/abs/2503.15426
+VPP-LLaVA paper link: https://arxiv.org/abs/2503.15426
+
+## Acknowledgements
+This repo is changed from [LLaVA v1.5](https://github.com/haotian-liu/LLaVA). The repo also benifits form [ChatterBox (AAAI 2025)](https://github.com/sunsmarterjie/ChatterBox) and [Genixer (ECCV 2024)](https://github.com/zhaohengyuan1/Genixer)
+
+Thanks for their wonderful works.
+
+## Citation
+
+```bibtex
+@misc{tang2025visualpositionpromptmllm,
+      title={Visual Position Prompt for MLLM based Visual Grounding}, 
+      author={Wei Tang and Yanpeng Sun and Qinying Gu and Zechao Li},
+      year={2025},
+      eprint={2503.15426},
+      archivePrefix={arXiv},
+      primaryClass={cs.CV},
+      url={https://arxiv.org/abs/2503.15426}, 
+}
+```