Instructions to use Vishva007/Muse-Glimmer-30B-W4A16-AutoRound with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Vishva007/Muse-Glimmer-30B-W4A16-AutoRound with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Vishva007/Muse-Glimmer-30B-W4A16-AutoRound")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Vishva007/Muse-Glimmer-30B-W4A16-AutoRound")
model = AutoModelForMultimodalLM.from_pretrained("Vishva007/Muse-Glimmer-30B-W4A16-AutoRound", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Vishva007/Muse-Glimmer-30B-W4A16-AutoRound with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Vishva007/Muse-Glimmer-30B-W4A16-AutoRound"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vishva007/Muse-Glimmer-30B-W4A16-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Vishva007/Muse-Glimmer-30B-W4A16-AutoRound

SGLang

How to use Vishva007/Muse-Glimmer-30B-W4A16-AutoRound with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Vishva007/Muse-Glimmer-30B-W4A16-AutoRound" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vishva007/Muse-Glimmer-30B-W4A16-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Vishva007/Muse-Glimmer-30B-W4A16-AutoRound" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vishva007/Muse-Glimmer-30B-W4A16-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Vishva007/Muse-Glimmer-30B-W4A16-AutoRound with Docker Model Runner:
```
docker model run hf.co/Vishva007/Muse-Glimmer-30B-W4A16-AutoRound
```

Muse Glimmer 30B - W4A16 AutoRound / GPTQ

This repository contains the W4A16 quantized versions of Meta's Muse Glimmer 30B, optimized for efficient local inference. The model was quantized using Intel AutoRound to 4-bit precision while explicitly keeping the Vision Encoder in its original precision (BF16) to preserve multimodal and visual agentic capabilities without degradation.

Two formats are provided across the repositories:

AutoRound Format: Vishva007/Muse-Glimmer-30B-W4A16-AutoRound
AutoGPTQ Format: Vishva007/Muse-Glimmer-30B-W4A16-AutoRound-GPTQ

Quantization Details

The quantization was performed with high-accuracy calibration settings tailored for production-grade inference:

Scheme: W4A16 (4-bit weights, 16-bit activations)
Group Size: 64
Symmetric: True
Vision Module: Kept unquantized (quant_nontext_module=False) to ensure maximum vision-language alignment.
Calibration: 512 samples, 800 iterations, sequence length of 2048.

By compressing the language model to under 20 GB, this model fits comfortably within 24GB or 32GB VRAM environments alongside its KV cache and perception encoder.

Usage with vLLM

For optimal performance, including agentic function calling and reasoning parsing, use the vllm/vllm-openai:muse-glimmer image or the latest vLLM build supporting Muse Glimmer.

Run the following command to serve the model (adjust the model path to the AutoRound or GPTQ repo as needed):

vllm serve Vishva007/Muse-Glimmer-30B-W4A16-AutoRound \
  --served-model-name muse-glimmer \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --max-num-seqs 128 \
  --generation-config auto \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer

Key vLLM Flags Explained

--max-model-len 32768: Sets the context window to 32k tokens.
--enable-auto-tool-choice: Enables agentic tool selection.
--tool-call-parser / --reasoning-parser: Configures the endpoint to natively handle Muse Glimmer's multi-step thinking and function schema outputs.

About the Base Model (Muse Glimmer 30B)

Muse Glimmer is a 30-billion-parameter causal language model with a dedicated perception encoder (~1.8B param ViT-G/14), distilled from Muse Spark by Meta Superintelligence Lab. It is purpose-built for autonomous agentic tasks on consumer hardware.

Key Capabilities:

End-to-end Agentic Task Completion: High success rates on DeepSearch QA, MCP-Atlas, 𝛕3-Bench, and SWE-Bench.
Multimodal Reasoning: Interprets interleaved text and images (charts, documents, screenshots).
Reliable Tool Use & Failure Recovery: Handles complex tool schemas and automatically diagnoses/retries failed calls.
Controllable Effort: Supports customizable reasoning strengths (low/medium/high/xhigh) via system prompts.
Multilingual: Trained on data from more than 100 languages.

License: Apache 2.0

Downloads last month: 18

Safetensors

Model size

5B params

Tensor type

BF16

I32

F16

Model tree for Vishva007/Muse-Glimmer-30B-W4A16-AutoRound

Base model

meta-models/Muse-Glimmer-30B

Quantized

(115)

this model

Collection including Vishva007/Muse-Glimmer-30B-W4A16-AutoRound

Meta's Muse Glimmer

Collection

Quantized Muse Glimmer models for efficient image-text understanding (AutoRound W4A16 and W2A16). • 4 items • Updated 2 days ago