Upload folder using huggingface_hub

README.md

@@ -0,0 +1,104 @@
+---
+license: apache-2.0
+language:
+  - en
+library_name: transformers
+base_model:
+  - Qwen/Qwen3-VL-4B-Thinking
+pipeline_tag: image-text-to-text
+tags:
+  - visual-grounding
+  - multimodal
+  - qwen3-vl
+  - reinforcement-learning
+  - grpo
+---
+
+# EGM-Qwen3-VL-4B
+
+<p align="center">
+  <a href="https://nvlabs.github.io/EGM">[Project Page]</a> &nbsp;
+  <a href="https://github.com/NVlabs/EGM">[Code]</a> &nbsp;
+</p>
+
+## Model Summary
+
+**EGM-Qwen3-VL-4B** is an efficient visual grounding model from the [EGM (Efficient Visual Grounding Language Models)](https://nvlabs.github.io/EGM) family. It is built on top of [Qwen3-VL-4B-Thinking](https://huggingface.co/Qwen/Qwen3-VL-4B-Thinking) and trained with a two-stage pipeline: supervised fine-tuning (SFT) followed by reinforcement learning (RL) using GRPO (Group Relative Policy Optimization).
+
+EGM demonstrates that by increasing test-time computation, small vision-language models can **outperform much larger models** in visual grounding tasks while being significantly faster at inference.
+
+## Key Results
+
+- **90.9 average IoU** on the RefCOCO benchmark (vs. 87.2 for the base Qwen3-VL-4B-Thinking)
+- **+3.7 IoU improvement** over the base model
+- Outperforms Qwen3-VL-235B-A22B-Instruct (88.2 avg IoU) while being dramatically faster
+
+### RefCOCO Benchmark Results
+
+| Model | RefCOCO val | RefCOCO test-A | RefCOCO test-B | RefCOCO+ val | RefCOCO+ test-A | RefCOCO+ test-B | RefCOCOg val | RefCOCOg test | Avg |
+|---|---|---|---|---|---|---|---|---|---|
+| Qwen3-VL-4B-Thinking | 90.0 | 92.7 | 85.6 | 85.2 | 89.5 | 79.3 | 87.0 | 87.7 | 87.2 |
+| **EGM-Qwen3-VL-4B** | **93.5** | **95.1** | **90.0** | **89.7** | **93.1** | **84.9** | **90.4** | **90.8** | **90.9** |
+
+## How It Works
+
+VLMs of different sizes often share the same visual encoder. Small models fall behind large models primarily due to a gap in **text understanding** capabilities — 62.8% of small model errors stem from complex prompts with multiple relational descriptions. EGM mitigates this gap by generating many mid-quality tokens (from small models) to match the performance of large VLMs that produce fewer but more expensive tokens.
+
+### Training Pipeline
+
+1. **SFT Stage**: A proprietary VLM generates detailed chain-of-thought reasoning steps for visual grounding training data. The base model is fine-tuned on this data. The SFT checkpoint is available as [nvidia/EGM-4B-SFT](https://huggingface.co/nvidia/EGM-4B-SFT).
+2. **RL Stage**: GRPO is applied with a reward function combining IoU and task success metrics, further improving grounding accuracy.
+
+## Quickstart
+
+### Download
+
+```bash
+pip install -U huggingface_hub
+huggingface-cli download nvidia/EGM-4B --local-dir ./models/EGM-4B
+```
+
+### Evaluation
+
+```bash
+pip install sglang==0.5.5
+
+export BASE_DIR=$(pwd)
+export MODEL_PATH="${BASE_DIR}/models/EGM-4B"
+export DATA_JSON="${BASE_DIR}/data/EGM_Datasets/metadata/eval/refcoco+_testA.jsonl"
+export OUTPUT_DIR="${BASE_DIR}/result/"
+export BASE_IMG_DIR="${BASE_DIR}"
+
+cd verl
+bash scripts/sglang_infer.sh
+```
+
+## Model Architecture
+
+| Component | Details |
+|---|---|
+| Architecture | Qwen3VLForConditionalGeneration |
+| Text Hidden Size | 2560 |
+| Text Layers | 36 |
+| Attention Heads | 32 (8 KV heads) |
+| Text Intermediate Size | 9728 |
+| Vision Hidden Size | 1024 |
+| Vision Layers | 24 |
+| Patch Size | 16 x 16 |
+| Max Position Embeddings | 262,144 |
+| Vocabulary Size | 151,936 |
+
+## Citation
+
+```bibtex
+@article{zhan2026EGM,
+    author = {Zhan, Guanqi and Li, Changye and Liu, Zhijian and Lu, Yao and Wu, Yi and Han, Song and Zhu, Ligeng},
+    title = {EGM: Efficient Visual Grounding Language Models},
+    booktitle = {arXiv},
+    year = {2026}
+}
+```
+
+## Acknowledgment
+
+This repository benefits from [Qwen3-VL](https://github.com/QwenLM/Qwen3-VL), [InternVL](https://github.com/OpenGVLab/InternVL), [verl](https://github.com/volcengine/verl) and [verl-internvl](https://github.com/Weiyun1025/verl-internvl).

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ff9358cdaba7f4b406e242984e518e1743c65f619abce26f53a73517a8ecd28
+size 4997642072

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e14deec41f0199909ffd5633b9fb6f3a4989e199a851ef15aa8e70fff3acceab
+size 4974280088

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0194be3bb08510621893c00e8a3e1fbf5800e82c0f60074f74ebb9c9fea3f3c9
+size 4924892304

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:588b15fa129ec3dc3067fe3ef98350136c5919fc41ba0d7eba394f63e49cfaf5
+size 2637524984