Instructions to use bowmanslayer/VeriLoop-E2-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bowmanslayer/VeriLoop-E2-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bowmanslayer/VeriLoop-E2-W4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("bowmanslayer/VeriLoop-E2-W4A16")
model = AutoModelForMultimodalLM.from_pretrained("bowmanslayer/VeriLoop-E2-W4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bowmanslayer/VeriLoop-E2-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bowmanslayer/VeriLoop-E2-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bowmanslayer/VeriLoop-E2-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bowmanslayer/VeriLoop-E2-W4A16

SGLang

How to use bowmanslayer/VeriLoop-E2-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bowmanslayer/VeriLoop-E2-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bowmanslayer/VeriLoop-E2-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bowmanslayer/VeriLoop-E2-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bowmanslayer/VeriLoop-E2-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use bowmanslayer/VeriLoop-E2-W4A16 with Docker Model Runner:
```
docker model run hf.co/bowmanslayer/VeriLoop-E2-W4A16
```

VeriLoop-E2-W4A16

File size: 1,202 Bytes

711ed9f

{
  "vllm": "0.29.0",
  "torch": "2.13.0",
  "transformers": "5.17.0",
  "triton": "3.7.1",
  "llama_cpp": {
    "reported_version": "0.1.2-dev (build 0, commit unknown)",
    "compiler": "GNU 13.3.0 Linux x86_64",
    "sha256": {
      "llama-server": "151ca33e963f9189a7b80ea136f0bc607282611f63947131800478f93f7f7698",
      "libllama-server-impl.so": "55b0d74acb08709488a81b16e115e53946f87ffa5ef36a3e28999c924b1f6fb9",
      "libllama.so": "e0f23a9843348eb2b05d646589a06ef376bb3b4421de862fc01164dc017e480f",
      "libllama-common.so": "0d82c013ad7f009ea6b7b83520a5d4282dcf735dc5029a1005b3b0e5411d99fc",
      "libggml.so": "81320042a7ac223a556e1468c19b9032906a07bc0e11bd376c8a9f33c5f271b8",
      "libggml-base.so": "d6c426af13a8747990370b588f0fb2367e3601683890a24a892d53a1307af065",
      "libggml-cpu.so": "e2172c751f093b7cf86749931d8b8787123310af5c7baab4f0133f8f388078e6",
      "libggml-cuda.so": "0bef149d752c8637782fe53d38a8496bd8dfeb8fcc85dbe461f15bb9a1006c95"
    }
  },
  "local_fa2_fp8kv_sha256": {
    "fa2_fp8kv.so": "9ff7300a21221a3547f777843c097dc2473086e05a97890d5ab1067a40a462e9",
    "fa2_fp8kv_prefill.so": "21ccdb26dc1333fb3f16a3a22b1d1ce7401f0233d518f1b7b6af659ef7de5942"
  }
}