Instructions to use convergence-ai/proxy-lite-3b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use convergence-ai/proxy-lite-3b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="convergence-ai/proxy-lite-3b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("convergence-ai/proxy-lite-3b")
model = AutoModelForMultimodalLM.from_pretrained("convergence-ai/proxy-lite-3b", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use convergence-ai/proxy-lite-3b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "convergence-ai/proxy-lite-3b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "convergence-ai/proxy-lite-3b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/convergence-ai/proxy-lite-3b

SGLang

How to use convergence-ai/proxy-lite-3b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "convergence-ai/proxy-lite-3b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "convergence-ai/proxy-lite-3b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "convergence-ai/proxy-lite-3b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "convergence-ai/proxy-lite-3b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use convergence-ai/proxy-lite-3b with Docker Model Runner:
```
docker model run hf.co/convergence-ai/proxy-lite-3b
```

add AIBOM

by sabato-nocera - opened Jul 21, 2025

base: refs/heads/main

←

from: refs/pr/5

Discussion Files changed

+69

-0

Files changed (1) hide show

convergence-ai_proxy-lite-3b.json +69 -0

convergence-ai_proxy-lite-3b.json ADDED Viewed

	@@ -0,0 +1,69 @@

+{
+    "bomFormat": "CycloneDX",
+    "specVersion": "1.6",
+    "serialNumber": "urn:uuid:6b5bd733-ebd1-46f4-abc0-cf84d3257714",
+    "version": 1,
+    "metadata": {
+        "timestamp": "2025-07-14T10:40:19.271236+00:00",
+        "component": {
+            "type": "machine-learning-model",
+            "bom-ref": "convergence-ai/proxy-lite-3b-045537c4-097d-5992-b759-dd2c48581631",
+            "name": "convergence-ai/proxy-lite-3b",
+            "externalReferences": [
+                {
+                    "url": "https://huggingface.co/convergence-ai/proxy-lite-3b",
+                    "type": "documentation"
+                }
+            ],
+            "modelCard": {
+                "modelParameters": {
+                    "task": "image-to-text",
+                    "architectureFamily": "qwen2_5_vl",
+                    "modelArchitecture": "Qwen2_5_VLForConditionalGeneration"
+                },
+                "properties": [
+                    {
+                        "name": "library_name",
+                        "value": "transformers"
+                    },
+                    {
+                        "name": "base_model",
+                        "value": "Qwen/Qwen2.5-VL-3B-Instruct"
+                    }
+                ],
+                "consideration": {
+                    "useCases": "<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->Proxy Lite is designed and trained to complete automated tasks in a web browser.Full code for running the model is available in the [github repository](https://github.com/convergence-ai/proxy-lite).This includes a CLI tool for running the model, as well as a streamlit app.You can use this [endpoint](https://huggingface.co/spaces/convergence-ai/demo-api) for small-scale testing.---"
+                }
+            },
+            "authors": [
+                {
+                    "name": "convergence-ai"
+                }
+            ],
+            "licenses": [
+                {
+                    "license": {
+                        "id": "CC-BY-NC-4.0",
+                        "url": "https://spdx.org/licenses/CC-BY-NC-4.0.html"
+                    }
+                }
+            ],
+            "description": "<!-- Provide a longer summary of what this model is. -->- **Developed by:** Convergence AI- **Model type:** 3B Vision-Language Model- **Agent type**: Web-browsing Agent- **License:** CC-BY-NC-4.0- **Finetuned from model:** Qwen/Qwen2.5-VL-3B-Instruct- [Running the agent](https://github.com/convergence-ai/proxy-lite)",
+            "tags": [
+                "transformers",
+                "safetensors",
+                "qwen2_5_vl",
+                "image-to-text",
+                "agent",
+                "action",
+                "vlm",
+                "base_model:Qwen/Qwen2.5-VL-3B-Instruct",
+                "base_model:finetune:Qwen/Qwen2.5-VL-3B-Instruct",
+                "license:cc-by-nc-4.0",
+                "text-generation-inference",
+                "endpoints_compatible",
+                "region:us"
+            ]
+        }
+    }
+}