Update README.md

README.md

@@ -2,7 +2,10 @@
 
 # SD-VLM-8B
 
-The SD-VLM architecture enhances a standard Vision-Language Model (VLM) with 3D spatial awareness through a minimal yet effective modification. Its core structure is built upon three key components:
+[**🌐 Homepage**](https://cpystan.github.io/SD_VLM_pages/)  | [**🤗 Dataset**](https://huggingface.co/datasets/cpystan/MSMU) | [**📖 arXiv**](https://arxiv.org/abs/2509.17664) | [**GitHub**](https://github.com/cpystan/SD-VLM)
+
+
+🎯 **The SD-VLM architecture enhances a standard Vision-Language Model (VLM) with 3D spatial awareness through a minimal yet effective modification.**
 
 **1. Base VLM**: Utilizes the LLaVA-1.5-7B framework, consisting of a CLIP-ViT vision encoder, a Vicuna large language model (LLM), and a linear projector connecting them.
 
@@ -12,4 +15,93 @@ This simple addition injects explicit 3D spatial priors into the model without a
 
 **3. Training Approach**: The model is efficiently fine-tuned on the MSMU spatial dataset for one epoch using LoRA, keeping the vision encoder frozen. This allows the LLM and projector to learn how to interpret the depth-enhanced visual features for quantitative reasoning.
 
-In essence, SD-VLM's structure is defined by a streamlined integration: it upgrades a standard VLM to understand 3D space by fusing depth information into visual features through a parameter-free additive operation, all trained efficiently on targeted data.
+In essence, SD-VLM's structure is defined by a streamlined integration: it upgrades a standard VLM to understand 3D space by fusing depth information into visual features through a parameter-free additive operation, all trained efficiently on targeted data.
+
+### Model Framework
+
+
+<img src="https://huggingface.co/spaces/cpystan/images/resolve/main/framework.png" 
+     width="56%" />
+
+### Quick Start!
+
+```
+from llava.model.builder import load_pretrained_model
+from llava.mm_utils import get_model_name_from_path
+from llava.eval.run_llava import eval_model
+from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN, DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN
+import copy
+
+model_path = "cpystan/SD-VLM-7B"
+
+tokenizer, model, image_processor, context_len = load_pretrained_model(
+    model_path=model_path,
+    model_base=None,
+    model_name=get_model_name_from_path(model_path)
+)
+
+input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).cuda()
+image = Image.open(os.path.join(image_folder, image_file)).convert('RGB')
+ori_img = copy.deepcopy(image)
+image_tensor = process_images([image], image_processor, model.config)[0]
+
+with torch.inference_mode():
+    output_ids = model.generate(
+        input_ids,
+        images=image_tensor.unsqueeze(0).half().to(input_ids.device),
+        image_sizes=[image.size],
+        do_sample=True if temperature > 0 else False,
+        temperature=0.2,
+        top_p=None,
+        num_beams=1,
+        ori_imgs = [ori_img],
+        max_new_tokens=1024,
+        use_cache=True,)
+response= tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
+
+```
+
+
+## 🏆 Mini-Leaderboard
+We show a mini-leaderboard here. It shows the results of each sub-category and the overall performance.
+
+# Results on MSMU-Bench
+
+| Model | Existence | Object<br>Counting | Scale<br>Est. | Grounding | Relative<br>Position | Absolute<br>Distance | Scale<br>Comparison | Ref. Object<br>Est. | Average |
+| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
+| **Large Language Models (LLMs): Text only** |||||||||
+| GPT-4-Turbo | 12.76 | 5.21 | 13.51 | 12.64 | 24.84 | 7.50 | 36.79 | 12.04 | 15.66 |
+| Qwen2.5 | 4.25 | 0.00 | 0.78 | 13.79 | 0.62 | 0.00 | 16.04 | 1.57 | 4.63 |
+| DeepSeek-V3 | 0.00 | 5.24 | 1.54 | 6.90 | 10.56 | 0.00 | 25.47 | 5.24 | 7.39 |
+| **Vision-Language Models (VLMs): Image + Text** |||||||||
+| GPT-4o | 44.68 | 41.67 | 3.86 | 27.59 | 67.08 | 20.00 | 54.72 | 2.09 | 32.28 |
+| Gemini-2 | 38.30 | 43.75 | 23.94 | 19.54 | 54.66 | 12.50 | 69.81 | 18.85 | 35.17 |
+| Qwen2.5-VL-72B | 59.57 | 35.42 | 1.54 | 13.79 | 57.76 | 2.50 | 66.04 | 9.95 | 30.82 |
+| Qwen2.5-VL-32B | 29.79 | 41.67 | 10.81 | 18.39 | 60.25 | 2.50 | 46.23 | 10.99 | 27.59 |
+| Qwen2.5-VL-7B | 12.76 | 4.17 | 0.00 | 1.15 | 1.24 | 0.00 | 5.66 | 0.52 | 3.19 |
+| Intern-VL3-78B | 47.62 | 42.71 | 6.47 | 26.32 | 56.94 | 13.33 | 64.10 | 16.46 | 33.63 |
+| Intern-VL3-8B | 36.17 | 41.67 | 4.63 | 18.39 | 60.25 | 2.50 | 49.06 | 8.38 | 28.54 |
+| LLaVA-1.5-7B | 1.54 | 36.46 | 5.02 | 20.69 | 42.86 | 5.00 | 38.68 | 0.52 | 19.45 |
+| **Depth-encoded VLMs: Image + Depth + Text** |||||||||
+| SpatialBot | 10.64 | 46.88 | 15.83 | 28.74 | 66.46 | 5.00 | 50.94 | 8.90 | 29.17 |
+| SpatialRGPT | 10.64 | 36.46 | 20.08 | 17.24 | 60.25 | 15.00 | 62.26 | 9.95 | 28.98 |
+| **SD-VLM-8B** |87.23 | 47.92 | 51.35 | 42.53 | 75.16 | 40.00 | 55.66 | 46.07 | **56.31** |
+
+# Examples
+
+<img src="https://huggingface.co/spaces/cpystan/images/resolve/main/result_vis.png" 
+     width="56%" />
+
+
+## Citation
+
+**BibTeX:**
+```bibtex
+@misc{chen2025sdvlm,
+      title={SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models}, 
+      author={Pingyi Chen and Yujing Lou and Shen Cao and Jinhui Guo and Lubin Fan and Yue Wu and Lin Yang and Lizhuang Ma and Jieping Ye},
+      year={2025},
+      url={https://arxiv.org/abs/2509.17664}, 
+}
+```
+