Instructions to use ACIDE/User-VLM-3B-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ACIDE/User-VLM-3B-base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="ACIDE/User-VLM-3B-base")# Load model directly from transformers import AutoProcessor, AutoModelForImageTextToText processor = AutoProcessor.from_pretrained("ACIDE/User-VLM-3B-base") model = AutoModelForImageTextToText.from_pretrained("ACIDE/User-VLM-3B-base") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use ACIDE/User-VLM-3B-base with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ACIDE/User-VLM-3B-base" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ACIDE/User-VLM-3B-base", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/ACIDE/User-VLM-3B-base
- SGLang
How to use ACIDE/User-VLM-3B-base with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ACIDE/User-VLM-3B-base" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ACIDE/User-VLM-3B-base", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ACIDE/User-VLM-3B-base" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ACIDE/User-VLM-3B-base", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use ACIDE/User-VLM-3B-base with Docker Model Runner:
docker model run hf.co/ACIDE/User-VLM-3B-base
Update README.md
Browse files
README.md
CHANGED
|
@@ -12,6 +12,7 @@ base_model:
|
|
| 12 |
pipeline_tag: image-text-to-text
|
| 13 |
---
|
| 14 |
# User-VLM 360°
|
|
|
|
| 15 |
|
| 16 |
## Overview
|
| 17 |
**User-VLM 360°** is a series of personalized Vision-Language Models (VLMs) designed for social human-robot interactions. The model introduces **User-aware tuning**, addressing the **semantic gap** that arises from the misalignment between user queries and the observed scene as captured by a robot's camera. Unlike traditional instruction tuning, which introduces latency and reduces performance, **User-VLM 360°** enables **real-time, robust adaptation** in dynamic robotic environments by inherently aligning cross-modal user representations.
|
|
@@ -21,6 +22,8 @@ This model allows for **customization of open-weight VLMs** to produce **persona
|
|
| 21 |
## Training Details
|
| 22 |
**Base Model:** User-VLM 360° is built on **PaliGemma 2**, which consists of a **SigLIP vision encoder** and **Gemma 2 as the language model**.
|
| 23 |
|
|
|
|
|
|
|
| 24 |
### Fine-tuning Process:
|
| 25 |
1. **Base Model Tuning:**
|
| 26 |
- Tuned the **MLP layer** to provide **user and scene descriptions** over **1 epoch**.
|
|
|
|
| 12 |
pipeline_tag: image-text-to-text
|
| 13 |
---
|
| 14 |
# User-VLM 360°
|
| 15 |
+

|
| 16 |
|
| 17 |
## Overview
|
| 18 |
**User-VLM 360°** is a series of personalized Vision-Language Models (VLMs) designed for social human-robot interactions. The model introduces **User-aware tuning**, addressing the **semantic gap** that arises from the misalignment between user queries and the observed scene as captured by a robot's camera. Unlike traditional instruction tuning, which introduces latency and reduces performance, **User-VLM 360°** enables **real-time, robust adaptation** in dynamic robotic environments by inherently aligning cross-modal user representations.
|
|
|
|
| 22 |
## Training Details
|
| 23 |
**Base Model:** User-VLM 360° is built on **PaliGemma 2**, which consists of a **SigLIP vision encoder** and **Gemma 2 as the language model**.
|
| 24 |
|
| 25 |
+

|
| 26 |
+
|
| 27 |
### Fine-tuning Process:
|
| 28 |
1. **Base Model Tuning:**
|
| 29 |
- Tuned the **MLP layer** to provide **user and scene descriptions** over **1 epoch**.
|