| --- |
| license: other |
| license_name: gemma-terms-of-use |
| license_link: https://ai.google.dev/gemma/terms |
| tags: |
| - spatial-understanding |
| - 3d-vision |
| - vision-language-model |
| - paligemma2 |
| base_model: google/paligemma2-3b-pt-448 |
| pipeline_tag: visual-question-answering |
| --- |
| |
| # HiSpatial-3B |
|
|
| Official model weights for **[HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models](https://arxiv.org/abs/2603.25411)** (CVPR 2026). |
|
|
| - **Paper:** [arXiv:2603.25411](https://arxiv.org/abs/2603.25411) |
| - **Code:** [github.com/microsoft/HiSpatial](https://github.com/microsoft/HiSpatial) |
| - **Project Page:** [microsoft.github.io/HiSpatial](https://microsoft.github.io/HiSpatial/) |
|
|
| ## Model Description |
|
|
| HiSpatial-3B is a vision-language model for hierarchical 3D spatial understanding. It extends [PaliGemma2-3B](https://huggingface.co/google/paligemma2-3b-pt-448) with a depth encoder (Conv2dForXYZ) that fuses RGB image features from SigLIP with 3D point cloud features through a combined multi-modal projector. |
|
|
| | Component | Detail | |
| |-----------|--------| |
| | Backbone | PaliGemma2-3B (448px) | |
| | Vision Encoder | SigLIP-SO400M | |
| | Language Model | Gemma2-2B | |
| | Depth Encoder | Conv2dForXYZ | |
| | Projector | CombinedMultiModalProjector | |
| | Checkpoint Format | PyTorch state_dict (`weights.pt`, ~14GB) | |
| |
| ## Usage |
| |
| ```python |
| from hispatial.inference import MoGeProcessor, HiSpatialPredictor |
| |
| # Initialize MoGe depth estimator and HiSpatial predictor |
| moge = MoGeProcessor(device_name="cuda") |
| predictor = HiSpatialPredictor(model_load_path="lhzzzzzy/HiSpatial-3B") |
|
|
| # Load an image |
| image = "example.jpg" |
|
|
| # Estimate 3D point cloud from the image |
| xyz_values = moge.apply_transform(image) |
|
|
| # Ask a spatial question |
| answer = predictor.query( |
| image=image, |
| prompt="Which object is closer to the camera, the chair or the table?", |
| xyz_values=xyz_values, |
| ) |
| print(answer) |
| ``` |
| |
| See the [GitHub repo](https://github.com/microsoft/HiSpatial) for full installation and evaluation instructions. |
|
|
| ## License |
|
|
| This model is a derivative of [PaliGemma2](https://huggingface.co/google/paligemma2-3b-pt-448), which is based on [Gemma](https://ai.google.dev/gemma). Use of this model is subject to the [Gemma Terms of Use](https://ai.google.dev/gemma/terms), including the [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{liang2026hispatial, |
| title={HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models}, |
| author={Liang, Huizhi and Shen, Yichao and Deng, Yu and Xu, Sicheng and Feng, Zhiyuan and Zhang, Tong and Liang, Yaobo and Yang, Jiaolong}, |
| booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, |
| year={2026} |
| } |
| ``` |
|
|
