Improve model card for LLaVA-OneVision-1.5 with paper, code, project links, and sample usage

README.md

@@ -1,52 +1,130 @@
 ---
-license: apache-2.0
-datasets:
-- lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M
 base_model:
 - Qwen/Qwen3-8B-Base
 - DeepGlint-AI/rice-vit-large-patch14-560
-pipeline_tag: image-text-to-text
+datasets:
+- lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M
 library_name: transformers
+license: apache-2.0
+pipeline_tag: image-text-to-text
 ---
+
 # LLaVA-OneVision-1.5: Fully Open-Source State-of-the-Art VLM Model
 
-**LLaVA-OneVision1.5** introduces a novel family of **fully open-source** Large Multimodal Models (LMMs) that achieves **state-of-the-art performance**  with substantially **lower cost** through training on **native resolution** images.
+**LLaVA-OneVision1.5** introduces a novel family of **fully open-source** Large Multimodal Models (LMMs) that achieves **state-of-the-art performance** with substantially **lower cost** through training on **native resolution** images.
+
+**Paper**: [LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training](https://huggingface.co/papers/2509.23661)
+**Code**: [https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5](https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5)
+**Project Page (Models & Datasets Collection)**: [https://huggingface.co/collections/lmms-lab/llava-onevision-15-68d385fe73b50bd22de23713](https://huggingface.co/collections/lmms-lab/llava-onevision-15-68d385fe73b50bd22de23713)
+**Demo**: [https://huggingface.co/spaces/lmms-lab/LLaVA-OneVision-1.5](https://huggingface.co/spaces/lmms-lab/LLaVA-OneVision-1.5)
+
+-   **Superior Performance**
+    A family of fully open-source large multimodal models demonstrating
+    -   Superior performance across multiple multimodal benchmarks
+    -   outperforming **Qwen2.5-VL** in most evaluation tasks.
+
+-   **High-Quality Data at Scale**
+    Meticulously curated **pre-training and SFT data** with rigorous filtering and quality control, achieving **superior data efficiency** with only **64B tokens**.
+    -   Concept-balanced, highly diverse, high-quality caption data
+    -   Comprehensive instruction fine-tuning data covering a wide range of tasks
+
+-   **Ultra-Efficient Training Framework** Complete end-to-end training framework designed for maximum efficiency:
+    -   $16000 total budget for full model training on A100 GPUs ($0.6 per GPU/Hour)
+    -   45% HFU efficiency in 8k context length
+    -   Built on **MegatronLM** with support for **MoE**, **FP8**, and **long sequence parallelization**
+    -   Optimized codebase for cost-effective scaling
 
-- **Superior Performance**
-A family of fully open-source large multimodal models demonstrating 
-    - Superior performance across multiple multimodal benchmarks
-    - outperforming **Qwen2.5-VL** in most evaluation tasks.
 
-- **High-Quality Data at Scale**
-Meticulously curated **pre-training and SFT data** with rigorous filtering and quality control, achieving **superior data efficiency** with only **64B tokens**.
-    - Concept-balanced, highly diverse, high-quality caption data
-    - Comprehensive instruction fine-tuning data covering a wide range of tasks
+-   **Fully Open Framework** for community access and reproducibility:
+    -   High-quality pre-training & SFT data
+    -   Complete training framework & code
+    -   Training recipes & configurations
+    -   Comprehensive training logs & metrics
 
-- **Ultra-Efficient Training Framework** Complete end-to-end training framework designed for maximum efficiency:
-    - $16000 total budget for full model training on A100 GPUs  ($0.6 per GPU/Hour)
-    - 45% HFU efficiency in 8k context length
-    - Built on **MegatronLM** with support for **MoE**, **FP8**, and **long sequence parallelization**
-    - Optimized codebase for cost-effective scaling
+## Quick Start with HuggingFace
 
+```python
+from transformers import AutoTokenizer, AutoProcessor, AutoModelForCausalLM
+from qwen_vl_utils import process_vision_info
+model_path = "lmms-lab/LLaVA-OneVision-1.5-8B-Instruct"
 
-- **Fully Open Framework** for community access and reproducibility:
-    - High-quality pre-training & SFT data
-    - Complete training framework & code
-    - Training recipes & configurations
-    - Comprehensive training logs & metrics
+# default: Load the model on the available device(s)
+model = AutoModelForCausalLM.from_pretrained(
+    model_path, torch_dtype="auto", device_map="auto", trust_remote_code=True
+)
+
+# default processer
+processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
+            },
+            {"type": "text", "text": "Describe this image."},
+        ],
+    }
+]
+
+# Preparation for inference
+text = processor.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True
+)
+image_inputs, video_inputs = process_vision_info(messages)
+inputs = processor(
+    text=[text],
+    images=image_inputs,
+    videos=video_inputs,
+    padding=True,
+    return_tensors="pt",
+)
+inputs = inputs.to("cuda")
+
+# Inference: Generation of the output
+generated_ids = model.generate(**inputs, max_new_tokens=1024)
+generated_ids_trimmed = [
+    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
+]
+output_text = processor.batch_decode(
+    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
+)
+print(output_text)
+
+```
+
+## Evaluation Results
+
+
+All evaluations were conducted using lmms_eval.
+
+![](asset/performance.png)
 
 ## Citation
 
 If you find *LLaVA-OneVision-1.5* useful in your research, please consider to cite the following related papers:
 
 ```
-@misc{an2025llavaonevision15fullyopenframework,
-      title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training}, 
-      author={Xiang An and Yin Xie and Kaicheng Yang and Wenkang Zhang and Xiuwei Zhao and Zheng Cheng and Yirui Wang and Songcen Xu and Changrui Chen and Chunsheng Wu and Huajie Tan and Chunyuan Li and Jing Yang and Jie Yu and Xiyao Wang and Bin Qin and Yumeng Wang and Zizhen Yan and Ziyong Feng and Ziwei Liu and Bo Li and Jiankang Deng},
-      year={2025},
-      eprint={2509.23661},
-      archivePrefix={arXiv},
-      primaryClass={cs.CV},
-      url={https://arxiv.org/abs/2509.23661}, 
+@inproceedings{LLaVA-OneVision-1.5,
+  title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
+  author={An, Xiang and Xie, Yin and Yang, Kaicheng and Zhang, Wenkang and Zhao, Xiuwei and Cheng, Zheng and Wang, Yirui and Xu, Songcen and Chen, Changrui and Wu, Chunsheng and Tan, Huajie and Li, Chunyuan and Yang, Jing and Jie Yu and Wang, Xiyao and Qin, Bin and Wang, Yumeng and Yan, Zizhen and Feng, Ziyong and Liu, Ziwei and Li, Bo and Deng, Jiankang},
+  booktitle={arxiv},  
+  year={2025}
+ }
+
+@inproceedings{xie2025region,
+  title={Region-based Cluster Discrimination for Visual Representation Learning},
+  author={Xie, Yin and Yang, Kaicheng and An, Xiang and Wu, Kun and Zhao, Yongle and Deng, Weimo and Ran, Zimin and Wang, Yumeng and Feng, Ziyong and Miles, Roy and Elezi, Ismail and Deng, Jiankang},
+  booktitle={ICCV},
+  year={2025}
+}
+
+@article{lillava,
+  title={LLaVA-OneVision: Easy Visual Task Transfer},
+  author={Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan},
+  journal={Transactions on Machine Learning Research}
+  year={2024}
 }
 ```