Instructions to use microsoft/Mage-VL with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use microsoft/Mage-VL with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="microsoft/Mage-VL", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained("microsoft/Mage-VL", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use microsoft/Mage-VL with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "microsoft/Mage-VL"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Mage-VL",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/microsoft/Mage-VL

SGLang

How to use microsoft/Mage-VL with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "microsoft/Mage-VL" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Mage-VL",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "microsoft/Mage-VL" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Mage-VL",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use microsoft/Mage-VL with Docker Model Runner:
```
docker model run hf.co/microsoft/Mage-VL
```

Mage-VL / neural_codec /README.md

Xinjie-Q

Upload Mage-VL: unified codec-native streaming VLM (image+video understanding + proactive gate)

12acbba verified about 20 hours ago

preview code

Raw

History Blame Contribute Delete

3.25 kB

neural_codec — DCVC-RT patch selection

Internal package for DCVC-RT neural-codec patch selection: the codec's per-frame bit-cost map decides which video patches to feed the VLM (regions the codec spends more bits on — motion / new detail — are kept; predictable background is dropped), as the neural alternative to the traditional HEVC (cv-preinfer) path.

Usage, environment setup, and the controllable codec.dcvc parameters are documented in the top-level ../README.md. This file is an internal file reference.

Files

File	Role
`dcvc_rt_engine.py`	Loads DCVC-RT intra/inter nets; `DMCIBitmap` / `DMCBitmap` add `compute_bitmap` (per-frame `(H/16, W/16)` bit-cost map) via a streaming `reset_sequence` / `step` API. Loads the bundled DCVC-RT source from `DCVC/` (no env var); checkpoints default to `dcvc_rt_intra.tar` / `dcvc_rt_inter.tar` in this dir (`DCVC_INTRA_TAR` / `DCVC_INTER_TAR` to override).
`codec_dcvc_config.py`	Single source of truth — reads the `codec.dcvc` block of `../processor/preprocessor_config.json`.
`dcvc_readiness_gen.py`	Config-driven generator: runs the readiness pipeline (`codec_tools/`) with DCVC bit-cost as the score source. Invoked by the model's codec path (`../processor/codec_video_processing_magevl.py::_run_dcvc_rt`).
`reproduce_bench.py`	Reproduce the evaluated selection for one video (config-driven; presets `cap12` / `b50` / `s95_b50`).
`codec_loader.py`	Load precomputed assets → model inputs via the release codec helpers.
`infer_dcvc_rt.py`	Standalone end-to-end demo (`--asset_dir` or `--video`).
`precompute_dcvc_rt.py`	Standalone, CLI-flag-driven batch precompute (video(s) → assets).
`canvas_assembler.py`	Top-k patch selection + canvas packing used by the standalone precompute path.
`codec_tools/`	Vendored readiness pipeline (frame sampling, grouping, 2×2-block selection, canvas packing).
`DCVC/`	Bundled DCVC-RT source (MIT, microsoft/DCVC): the `src/` package the engine imports + the CUDA-kernel source under `src/layers/extensions/inference/`. No external checkout / `DCVC_RT_ROOT` needed.
`dcvc_rt_intra.tar` / `dcvc_rt_inter.tar`	DCVC-RT checkpoints.

Notes / limitations

The bit-cost map is the summed y-bits estimate from DCVC-RT's Gaussian entropy model (the dominant, spatially-resolved term); the small hyperprior z term is omitted (it is a ranking signal). It is not run through the RANS arithmetic coder.
patch=16 is mandatory — it must match the image processor (preprocessor_config.json: patch_size=16, merge_size=2). The codec.patch=14 field is a separate cv-preinfer internal and does not apply here.
Canvases are square; non-16:9 videos are letterboxed, so wide videos waste some budget on padding. Tune codec.dcvc.max_pixels if needed.
DCVC-RT decodes every frame 0..max(sampled) to keep temporal references valid, so long videos are slow — use codec.dcvc.max_side and multiple GPUs.
The DCVC CUDA kernels fall back to pytorch when the compiled extension is unavailable (slower but numerically fine, and deterministic on the fallback path).