Image-Text-to-Text
Transformers
Safetensors
English
Chinese
qwen3_vl
text-generation
Vision-Language-Model
Vision-Language-Action
conversational
Instructions to use zzzrw/GEM-2B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use zzzrw/GEM-2B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="zzzrw/GEM-2B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoProcessor, AutoModelForSeq2SeqLM processor = AutoProcessor.from_pretrained("zzzrw/GEM-2B") model = AutoModelForSeq2SeqLM.from_pretrained("zzzrw/GEM-2B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use zzzrw/GEM-2B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "zzzrw/GEM-2B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "zzzrw/GEM-2B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/zzzrw/GEM-2B
- SGLang
How to use zzzrw/GEM-2B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "zzzrw/GEM-2B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "zzzrw/GEM-2B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "zzzrw/GEM-2B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "zzzrw/GEM-2B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use zzzrw/GEM-2B with Docker Model Runner:
docker model run hf.co/zzzrw/GEM-2B
Add library_name and improve model card
#1
by nielsr HF Staff - opened
README.md
CHANGED
|
@@ -1,13 +1,14 @@
|
|
| 1 |
---
|
| 2 |
-
|
|
|
|
| 3 |
datasets:
|
| 4 |
- zzzrw/GEM-250K
|
| 5 |
language:
|
| 6 |
- en
|
| 7 |
- zh
|
| 8 |
-
|
| 9 |
-
- Qwen/Qwen3-VL-2B-Instruct
|
| 10 |
pipeline_tag: image-text-to-text
|
|
|
|
| 11 |
---
|
| 12 |
|
| 13 |
<div align="center">
|
|
@@ -33,6 +34,9 @@ pipeline_tag: image-text-to-text
|
|
| 33 |
<sup>1</sup>Tsinghua University,
|
| 34 |
<sup>2</sup>Tencent Hunyuan
|
| 35 |
</p>
|
|
|
|
|
|
|
|
|
|
| 36 |
<a href='https://zhaorw02.github.io/GEM/'><img src="https://img.shields.io/badge/Project-Page-Green" alt="Project Page"></a>
|
| 37 |
<a href="https://arxiv.org/abs/2605.28548"><img src="https://img.shields.io/badge/Paper-Arxiv-red?logo=arxiv" alt="Paper"></a>
|
| 38 |
<a href="https://github.com/zhaorw02/GEM/"><img src="https://img.shields.io/badge/GitHub-Repo-181717?logo=github&logoColor=white" alt="GitHub"></a>
|
|
@@ -40,12 +44,35 @@ pipeline_tag: image-text-to-text
|
|
| 40 |
<a href="https://huggingface.co/datasets/zzzrw/GEM-250K/"><img src="https://img.shields.io/badge/Dataset-HuggingFace-yellow?logo=huggingface" alt="Dataset"></a>
|
| 41 |
|
| 42 |
</div>
|
|
|
|
| 43 |
<div align="center">
|
| 44 |
<video src="https://huggingface.co/datasets/zzzrw/GEM-250K/resolve/main/assets/GEM-demo.mp4" controls autoplay muted loop width="85%"></video>
|
| 45 |
</div>
|
| 46 |
|
| 47 |
-
|
| 48 |
|
| 49 |
<div align="center">
|
| 50 |
-
<img src="assets/overview.png" alt="GEM
|
| 51 |
-
</div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
base_model:
|
| 3 |
+
- Qwen/Qwen3-VL-2B-Instruct
|
| 4 |
datasets:
|
| 5 |
- zzzrw/GEM-250K
|
| 6 |
language:
|
| 7 |
- en
|
| 8 |
- zh
|
| 9 |
+
license: mit
|
|
|
|
| 10 |
pipeline_tag: image-text-to-text
|
| 11 |
+
library_name: transformers
|
| 12 |
---
|
| 13 |
|
| 14 |
<div align="center">
|
|
|
|
| 34 |
<sup>1</sup>Tsinghua University,
|
| 35 |
<sup>2</sup>Tencent Hunyuan
|
| 36 |
</p>
|
| 37 |
+
|
| 38 |
+
[**Project Page**](https://zhaorw02.github.io/GEM/) | [**Paper**](https://arxiv.org/abs/2605.28548) | [**GitHub**](https://github.com/zhaorw02/GEM/)
|
| 39 |
+
|
| 40 |
<a href='https://zhaorw02.github.io/GEM/'><img src="https://img.shields.io/badge/Project-Page-Green" alt="Project Page"></a>
|
| 41 |
<a href="https://arxiv.org/abs/2605.28548"><img src="https://img.shields.io/badge/Paper-Arxiv-red?logo=arxiv" alt="Paper"></a>
|
| 42 |
<a href="https://github.com/zhaorw02/GEM/"><img src="https://img.shields.io/badge/GitHub-Repo-181717?logo=github&logoColor=white" alt="GitHub"></a>
|
|
|
|
| 44 |
<a href="https://huggingface.co/datasets/zzzrw/GEM-250K/"><img src="https://img.shields.io/badge/Dataset-HuggingFace-yellow?logo=huggingface" alt="Dataset"></a>
|
| 45 |
|
| 46 |
</div>
|
| 47 |
+
|
| 48 |
<div align="center">
|
| 49 |
<video src="https://huggingface.co/datasets/zzzrw/GEM-250K/resolve/main/assets/GEM-demo.mp4" controls autoplay muted loop width="85%"></video>
|
| 50 |
</div>
|
| 51 |
|
| 52 |
+
**GEM** (Generative-supervised Embodied vision-language Model) is designed to bridge the gap between high-level semantic focus and low-level spatial/physical knowledge critical for robot execution. By integrating a depth map generation task directly into the VLM pre-training phase, GEM achieves substantial improvements in embodied intelligence, enhancing both semantic understanding and physical operation capabilities.
|
| 53 |
|
| 54 |
<div align="center">
|
| 55 |
+
<img src="https://huggingface.co/zzzrw/GEM-2B/resolve/main/assets/overview.png" alt="GEM Overview" width="85%">
|
| 56 |
+
</div>
|
| 57 |
+
|
| 58 |
+
## Highlights
|
| 59 |
+
- **Generative Supervision**: Proposes integrating a depth map generation task directly into VLM pre-training.
|
| 60 |
+
- **GEM-4M Dataset**: A comprehensive large-scale dataset featuring grounding, reasoning, and planning data paired with high-quality depth supervision.
|
| 61 |
+
- **SOTA Results**: Achieves state-of-the-art results across diverse embodied benchmarks and real-world evaluations.
|
| 62 |
+
|
| 63 |
+
## Usage
|
| 64 |
+
For details on training and inference, please refer to the [official GitHub repository](https://github.com/zhaorw02/GEM).
|
| 65 |
+
|
| 66 |
+
## Citation
|
| 67 |
+
If you find our work helpful, please consider citing:
|
| 68 |
+
```bibtex
|
| 69 |
+
@misc{zhao2026gemgenerativesupervisionhelps,
|
| 70 |
+
title={GEM: Generative Supervision Helps Embodied Intelligence},
|
| 71 |
+
author={Ruowen Zhao and Bangguo Li and Zuyan Liu and Yinan Liang and Junliang Ye and Fangfu Liu and Diankun Wu and Zhengyi Wang and Xumin Yu and Yongming Rao and Han Hu and Jun Zhu},
|
| 72 |
+
year={2026},
|
| 73 |
+
eprint={2605.28548},
|
| 74 |
+
archivePrefix={arXiv},
|
| 75 |
+
primaryClass={cs.CV},
|
| 76 |
+
url={https://arxiv.org/abs/2605.28548},
|
| 77 |
+
}
|
| 78 |
+
```
|