Instructions to use vikhyatk/moondream2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use vikhyatk/moondream2 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="vikhyatk/moondream2", trust_remote_code=True)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("vikhyatk/moondream2", trust_remote_code=True, dtype="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use vikhyatk/moondream2 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "vikhyatk/moondream2" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "vikhyatk/moondream2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/vikhyatk/moondream2
- SGLang
How to use vikhyatk/moondream2 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "vikhyatk/moondream2" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "vikhyatk/moondream2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "vikhyatk/moondream2" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "vikhyatk/moondream2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use vikhyatk/moondream2 with Docker Model Runner:
docker model run hf.co/vikhyatk/moondream2
Aspect ratio bug
Browse filesApologies if I haven't understood how resizing works with PIL. With this the following bit of the (original) code, it seems you set `aspect_ratio = width / height`, but when comparing, you swap the two. Reason I have swapped the two is because in the first line you have `width, height = image.size`
```python
width, height = image.size
max_dim = max(width, height)
if max_dim < 512:
im_size = (378, 378)
else:
aspect_ratio = width / height
im_size = min(
self.supported_sizes,
key=lambda size: (
abs((size[1] / size[0]) - aspect_ratio),
abs(size[0] - width) + abs(size[1] - height),
),
)
```
- vision_encoder.py +1 -1
|
@@ -233,7 +233,7 @@ class VisionEncoder(nn.Module):
|
|
| 233 |
im_size = min(
|
| 234 |
self.supported_sizes,
|
| 235 |
key=lambda size: (
|
| 236 |
-
abs((size[
|
| 237 |
abs(size[0] - width) + abs(size[1] - height),
|
| 238 |
),
|
| 239 |
)
|
|
|
|
| 233 |
im_size = min(
|
| 234 |
self.supported_sizes,
|
| 235 |
key=lambda size: (
|
| 236 |
+
abs((size[0] / size[1]) - aspect_ratio),
|
| 237 |
abs(size[0] - width) + abs(size[1] - height),
|
| 238 |
),
|
| 239 |
)
|