File size: 2,991 Bytes
bb8ca5a d7da855 bb8ca5a 4a5972d bb8ca5a 4a5972d e38e85b 4a5972d d7da855 4a5972d e38e85b 4a5972d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | ---
language:
- en
license: mit
pipeline_tag: video-text-to-text
---
# Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models
This is the model checkpoint of the GeoVR, a paradigm to restructure MLLM’s intrinsic representations with geometric awareness using purely 2D videos for Spatial Intelligence.
- **GitHub:** [https://github.com/WHB139426/GeoVR-MLLM](https://github.com/WHB139426/GeoVR-MLLM)
- **Paper:** [https://arxiv.org/abs/2606.05833](https://arxiv.org/abs/2606.05833)
## Sample Usage
To use this model, you need to clone the [official repository](https://github.com/WHB139426/GeoVR-MLLM) to access the custom modeling files.
```python
import torch
from utils.utils import *
from transformers import AutoProcessor
from models.qwen3vl_geo import Qwen3VLForConditionalGeneration
device = 'cuda:0'
model_id = "WHB139426/GeoVR-VGGT-Qwen3-VL-2B"
model = Qwen3VLForConditionalGeneration.from_pretrained(
model_id,
geometry_encoder_path=None,
metric_model_path=None,
dtype=torch.bfloat16,
attn_implementation="flash_attention_2",
add_camera=False,
add_scale=False,
add_depth=False,
distill_geometry_feature=False,
)
model.load_geometric_weights(model_id)
model.to(device)
num_frames = 32
processor = AutoProcessor.from_pretrained(model_id)
processor.video_processor.size = {"longest_edge": 384*num_frames*32*32, "shortest_edge": 4*num_frames*32*32}
messages = [
{
"role": "user",
"content": [
{"type": "video", "video": './assets/scene0111_02.mp4',},
{"type": "text", "text": "Measuring distance from the nearest points, select the closest object (trash bin, door, table, refrigerator) to the tv. If multiple exist, use the nearest instance.
Options:
A. trash bin
B. door
C. table
D. refrigerator
Answer with the option's letter from the given choices directly."},
],
}
]
generation_kwargs = {
'do_sample': True,
'top_p': 0.8,
'top_k': 20,
'temperature': 0.7,
'repetition_penalty': 1.0,
'max_new_tokens': 32*1024,
}
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt",
num_frames=num_frames,
fps=None,
enable_thinking=False,
).to(model.device)
with torch.cuda.amp.autocast(enabled=True, dtype=torch.bfloat16):
with torch.inference_mode():
generated_ids = model.generate(**inputs, **generation_kwargs)
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0].strip()
print(output_text)
```
## Citation
If you find this work useful, please consider citing:
```bibtex
@article{wang2026learning,
title={Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models},
author={Wang, Haibo and Huang, Lifu},
journal={arXiv preprint arXiv:2606.05833},
year={2026}
}
``` |