Instructions to use bowmanslayer/VeriLoop-E2-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bowmanslayer/VeriLoop-E2-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bowmanslayer/VeriLoop-E2-W4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("bowmanslayer/VeriLoop-E2-W4A16")
model = AutoModelForMultimodalLM.from_pretrained("bowmanslayer/VeriLoop-E2-W4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bowmanslayer/VeriLoop-E2-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bowmanslayer/VeriLoop-E2-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bowmanslayer/VeriLoop-E2-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bowmanslayer/VeriLoop-E2-W4A16

SGLang

How to use bowmanslayer/VeriLoop-E2-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bowmanslayer/VeriLoop-E2-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bowmanslayer/VeriLoop-E2-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bowmanslayer/VeriLoop-E2-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bowmanslayer/VeriLoop-E2-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use bowmanslayer/VeriLoop-E2-W4A16 with Docker Model Runner:
```
docker model run hf.co/bowmanslayer/VeriLoop-E2-W4A16
```

VeriLoop-E2-W4A16 / release.json

bowmanslayer

Release unmodified quantization with complete saved evaluation metrics and historical reference

711ed9f verified 12 days ago

Raw History Blame Contribute Delete

4.27 kB

{
  "source": "tsinghua-sigs-robot-lab/VeriLoop-E2",
  "revision": "9379d199adcc44acdf836558037163878e3a37ae",
  "variant": "original",
  "format": "w4a16",
  "license": "apache-2.0",
  "quantization": {
    "method": "AutoRound 0.12.3 symmetric INT4 g128",
    "samples": 256,
    "sequence_length": 512,
    "iterations": 200
  },
  "evaluation": {
    "capability_hits": 1212,
    "capability_n": 1514,
    "accuracy_pct": 80.05284015852048,
    "empty": 9,
    "length_limited": 9,
    "refusal_counts": {
      "substring_any": 98,
      "substring_head": 98,
      "heretic33_any": 98,
      "ega26_any": 98,
      "short": 0,
      "empty": 0,
      "empty_but_reasoning": 0,
      "reasoning_leaked": 0,
      "errors": 0
    },
    "refusal_n": 100,
    "refusal_valid": true,
    "refusal_budget_hits": 1
  },
  "weight_files": [
    {
      "path": "LICENSE",
      "size": 11343,
      "sha256": "50cbab8a892c5f2993b8c7351a99182507472def3b1374558308605d99b86b32"
    },
    {
      "path": "chat_template.jinja",
      "size": 17497,
      "sha256": "66526bd9fa7a45be668a86961154097e7ddbd71f831e499c1f751c118951aac2"
    },
    {
      "path": "config.json",
      "size": 16264,
      "sha256": "63dd264f99bc42b8c00fbdc07bb268058136ce4ec0d7b3c5ca2d617c09832ea1"
    },
    {
      "path": "generation_config.json",
      "size": 202,
      "sha256": "e70c136c1b78ddc1fb0905bac8e733a4dc448d4f852a5dd75143fffc70be550e"
    },
    {
      "path": "model-00001-of-00010.safetensors",
      "size": 2139874528,
      "sha256": "6e8ad489415055c97ff298df0b67c467ee31e524fc24b71de2127953b1c25b26"
    },
    {
      "path": "model-00002-of-00010.safetensors",
      "size": 2133383296,
      "sha256": "bb848313319ec57c13e2443d93ac1a74cbf5ac36f0e7aaf87e2ca46e2dff6f15"
    },
    {
      "path": "model-00003-of-00010.safetensors",
      "size": 2133383296,
      "sha256": "c5d426cf45611e8394d49652bca754521a87ac128b533053c55e86803f4832f4"
    },
    {
      "path": "model-00004-of-00010.safetensors",
      "size": 2125209320,
      "sha256": "7e1b52e31d8750bd904e2a391c40bdf4c855dec56a533f1ac5fe7a9c6c5930e5"
    },
    {
      "path": "model-00005-of-00010.safetensors",
      "size": 2133362544,
      "sha256": "1bf510cac8427a773f505b4b865fae8d7d9ffcb9b97285e409d9148d356f2e5b"
    },
    {
      "path": "model-00006-of-00010.safetensors",
      "size": 2146938240,
      "sha256": "2a979c4c72a44d3e6f8f4fd39e69f271c8e381fede587e4c4392a3e0b93820ba"
    },
    {
      "path": "model-00007-of-00010.safetensors",
      "size": 800644800,
      "sha256": "f26e2f4b123ce7dd01b1c140fdbdba96cd7993a45b55ebf35b86545456e7ddde"
    },
    {
      "path": "model-00008-of-00010.safetensors",
      "size": 2542796928,
      "sha256": "1e22bff91476ac31fd41fa6c5b6772286cacb458fcb0b13cd0d401d15504cf42"
    },
    {
      "path": "model-00009-of-00010.safetensors",
      "size": 10344,
      "sha256": "1c00c437d08b691129a2c73ad80b5cea833d4640dceb9f00841f505beedab686"
    },
    {
      "path": "model-00010-of-00010.safetensors",
      "size": 2542796896,
      "sha256": "6866cf8adcccc4cc6a00e74bc025f1a774fb52103b70f2674d0288272951a733"
    },
    {
      "path": "model.safetensors.index.json",
      "size": 196235,
      "sha256": "71c91f3eaf720c1b5ce6e4d95a9e2c00be9378a0df490d75ca429eeb00546e8f"
    },
    {
      "path": "model_extra_tensors.safetensors",
      "size": 312981276,
      "sha256": "05071fbf80f60d7851fdf2fca8c195e8ee483ecb312106c7639c50d9d7a7a937"
    },
    {
      "path": "preprocessor_config.json",
      "size": 443,
      "sha256": "3a159dfec9978a186a72ba085e0ad6a050f3968d8b364218d7bd13f5c89381f2"
    },
    {
      "path": "processor_config.json",
      "size": 1192,
      "sha256": "c5a9103c1eacb9dd3452601ffc3e29734c38aa9e4035b2246844d127c0ddfde8"
    },
    {
      "path": "quantization_config.json",
      "size": 10989,
      "sha256": "f915714bbd9d9c7d75302eab70ccdff7b25c64625ab13675c082e236d17da893"
    },
    {
      "path": "tokenizer.json",
      "size": 19989325,
      "sha256": "06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
    },
    {
      "path": "tokenizer_config.json",
      "size": 1164,
      "sha256": "7488550a894dcf741616126e322ea31d66cf9457b59ec5825f4c7efdced421f8"
    }
  ]
}

	{
	"source": "tsinghua-sigs-robot-lab/VeriLoop-E2",
	"revision": "9379d199adcc44acdf836558037163878e3a37ae",
	"variant": "original",
	"format": "w4a16",
	"license": "apache-2.0",
	"quantization": {
	"method": "AutoRound 0.12.3 symmetric INT4 g128",
	"samples": 256,
	"sequence_length": 512,
	"iterations": 200
	},
	"evaluation": {
	"capability_hits": 1212,
	"capability_n": 1514,
	"accuracy_pct": 80.05284015852048,
	"empty": 9,
	"length_limited": 9,
	"refusal_counts": {
	"substring_any": 98,
	"substring_head": 98,
	"heretic33_any": 98,
	"ega26_any": 98,
	"short": 0,
	"empty": 0,
	"empty_but_reasoning": 0,
	"reasoning_leaked": 0,
	"errors": 0
	},
	"refusal_n": 100,
	"refusal_valid": true,
	"refusal_budget_hits": 1
	},
	"weight_files": [
	{
	"path": "LICENSE",
	"size": 11343,
	"sha256": "50cbab8a892c5f2993b8c7351a99182507472def3b1374558308605d99b86b32"
	},
	{
	"path": "chat_template.jinja",
	"size": 17497,
	"sha256": "66526bd9fa7a45be668a86961154097e7ddbd71f831e499c1f751c118951aac2"
	},
	{
	"path": "config.json",
	"size": 16264,
	"sha256": "63dd264f99bc42b8c00fbdc07bb268058136ce4ec0d7b3c5ca2d617c09832ea1"
	},
	{
	"path": "generation_config.json",
	"size": 202,
	"sha256": "e70c136c1b78ddc1fb0905bac8e733a4dc448d4f852a5dd75143fffc70be550e"
	},
	{
	"path": "model-00001-of-00010.safetensors",
	"size": 2139874528,
	"sha256": "6e8ad489415055c97ff298df0b67c467ee31e524fc24b71de2127953b1c25b26"
	},
	{
	"path": "model-00002-of-00010.safetensors",
	"size": 2133383296,
	"sha256": "bb848313319ec57c13e2443d93ac1a74cbf5ac36f0e7aaf87e2ca46e2dff6f15"
	},
	{
	"path": "model-00003-of-00010.safetensors",
	"size": 2133383296,
	"sha256": "c5d426cf45611e8394d49652bca754521a87ac128b533053c55e86803f4832f4"
	},
	{
	"path": "model-00004-of-00010.safetensors",
	"size": 2125209320,
	"sha256": "7e1b52e31d8750bd904e2a391c40bdf4c855dec56a533f1ac5fe7a9c6c5930e5"
	},
	{
	"path": "model-00005-of-00010.safetensors",
	"size": 2133362544,
	"sha256": "1bf510cac8427a773f505b4b865fae8d7d9ffcb9b97285e409d9148d356f2e5b"
	},
	{
	"path": "model-00006-of-00010.safetensors",
	"size": 2146938240,
	"sha256": "2a979c4c72a44d3e6f8f4fd39e69f271c8e381fede587e4c4392a3e0b93820ba"
	},
	{
	"path": "model-00007-of-00010.safetensors",
	"size": 800644800,
	"sha256": "f26e2f4b123ce7dd01b1c140fdbdba96cd7993a45b55ebf35b86545456e7ddde"
	},
	{
	"path": "model-00008-of-00010.safetensors",
	"size": 2542796928,
	"sha256": "1e22bff91476ac31fd41fa6c5b6772286cacb458fcb0b13cd0d401d15504cf42"
	},
	{
	"path": "model-00009-of-00010.safetensors",
	"size": 10344,
	"sha256": "1c00c437d08b691129a2c73ad80b5cea833d4640dceb9f00841f505beedab686"
	},
	{
	"path": "model-00010-of-00010.safetensors",
	"size": 2542796896,
	"sha256": "6866cf8adcccc4cc6a00e74bc025f1a774fb52103b70f2674d0288272951a733"
	},
	{
	"path": "model.safetensors.index.json",
	"size": 196235,
	"sha256": "71c91f3eaf720c1b5ce6e4d95a9e2c00be9378a0df490d75ca429eeb00546e8f"
	},
	{
	"path": "model_extra_tensors.safetensors",
	"size": 312981276,
	"sha256": "05071fbf80f60d7851fdf2fca8c195e8ee483ecb312106c7639c50d9d7a7a937"
	},
	{
	"path": "preprocessor_config.json",
	"size": 443,
	"sha256": "3a159dfec9978a186a72ba085e0ad6a050f3968d8b364218d7bd13f5c89381f2"
	},
	{
	"path": "processor_config.json",
	"size": 1192,
	"sha256": "c5a9103c1eacb9dd3452601ffc3e29734c38aa9e4035b2246844d127c0ddfde8"
	},
	{
	"path": "quantization_config.json",
	"size": 10989,
	"sha256": "f915714bbd9d9c7d75302eab70ccdff7b25c64625ab13675c082e236d17da893"
	},
	{
	"path": "tokenizer.json",
	"size": 19989325,
	"sha256": "06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
	},
	{
	"path": "tokenizer_config.json",
	"size": 1164,
	"sha256": "7488550a894dcf741616126e322ea31d66cf9457b59ec5825f4c7efdced421f8"
	}
	]
	}