JohnTdi/Mage-VL-GGUF

Mage-VL GGUF

Community GGUF conversion of microsoft/Mage-VL, including Q8_0 and Q4_K_M language backbones plus F16 and Q8_0 Mage-ViT projector weights and native Q8 StreamMind sidecars.

Docker runtime, patches and launch instructions: JohnTDI-cpu/mage-vl-gguf on GitHub

Mage-VL Studio showing a selected time range, static-text OCR, native runtime metrics and representative full-frame Highlights.

Development disclosure: code assistance and review were provided by OpenAI GPT-5.6 Sol. Final integration, testing and release decisions were made and verified by the repository maintainer.

The Hugging Face repository contains the model artifacts. The linked GitHub repository contains the pinned llama.cpp Docker builds for Vulkan and CUDA, conversion patches, automatic MP4 gateway and complete usage instructions.

Files

File	Size	Purpose
mage-vl-backbone-Q8_0.gguf	4.69 GB	Highest-quality quantized language backbone
mage-vl-backbone-Q4_K_M.gguf	2.72 GB	Smaller backbone and faster token generation
mage-vit-mmproj-F16.gguf	661 MB	Maximum-fidelity vision weights
mage-vit-mmproj-Q8_0.gguf	353 MB	Compact vision weights
mage-streammind-epfe-Q8_0.gguf	96.5 MB	Stateful live-stream memory
mage-streammind-cls-Q8_0.gguf	512.6 MB	Silent/speak gate classifier
mage-dcvc-rt-intra-F16.gguf	91.3 MB	First/reset frame codec graph
mage-dcvc-rt-inter-F16.gguf	41.4 MB	Stateful inter-frame codec graph

Important runtime status

The Qwen3 backbone and Mage-ViT projector run together in the patched/pinned llama.cpp build on Vulkan; CUDA and CPU builds are also provided. The runtime patch implements Mage's temporal patching, 2x2 spatial merger and exact 3D RoPE. Both files remain in their GGUF storage types during inference—there is no BF16 reconstruction or second Transformers model in memory.

Docker images, conversion patch and exact instructions: JohnTDI-cpu/mage-vl-gguf.

Quality summary

Evaluation	Microsoft BF16 reference	Q8 + vision Q8 (ours)	Q4 + vision Q8 (ours)	Q8 + vision F16 (ours)
MMBench EN dev CircularEval	84.19%	84.36%	83.76%	84.28%
Video-MME tc32, no subtitles	64.00%	63.33%	61.96%	63.22%

WikiText-2 perplexity was 7.6592 for BF16, 7.6710 for Q8_0 (+0.154%) and 7.8739 for Q4_K_M (+2.803%). Nine-image vision-output cosine similarity against BF16 was 0.999821 for F16 and 0.999354 for Q8_0.

Video-MME used all 2,700 questions, the official prompt, 32-frame traditional codec input without subtitles, and codec-video-prep==0.2.5. MMBench used all 4,329 CircularEval records / 1,164 base questions.

The BF16 column contains Microsoft's reported reference values. The remaining columns are our measurements of the released GGUF variants. Runtime inference uses the patched native llama.cpp graph and keeps the backbone and Mage-ViT in their GGUF storage types.

Measured Vulkan speed

Radeon AI PRO R9700, llama.cpp a52077c, RADV, full offload, Flash Attention:

Backbone	pp1024	tg128
BF16	1,380 tok/s	70.84 tok/s
Q8_0	5,751 tok/s	118.03 tok/s
Q4_K_M	5,482 tok/s	178.41 tok/s

These speeds cover the language backbone, not codec preprocessing or Mage-ViT.

Native Docker measurement for a 50.64-second 1080x1920 H.264 clip with 8,099 prompt tokens at the shipped ctx=16384 setting:

Backbone + vision	Peak VRAM	Video prefill	Decode	Request wall
Q8 + Q8	7.56 GiB	2,691 tok/s	80.66 tok/s	4.16 s

The shipped 16 GiB profile uses one slot, a 16,384-token context, at most 256 sampled video frames and 150,000 pixels per codec canvas. Exact memory and speed depend on the prompt, canvas count, driver and GPU power state.

The GitHub runtime also includes native stateful StreamMind monitoring for RTSP/RTMP, HTTP/HLS, localhost and supported web-page sources such as YouTube. Mage-ViT encodes each transport segment once and shares its embeddings with the gate and triggered Qwen response. Temporary MP4/canvas/MAGECV1 data is deleted after consumption; see the linked GitHub live instructions.

Native validation

The current Vulkan Docker build passes ten images and ten H.264 videos for every GGUF release combination: Q4+vision Q8, Q8+vision Q8, Q4+vision F16 and Q8+vision F16—80/80 semantic checks. The end-to-end path also covers JPEG, PNG, H.264, HEVC, the public gateway and native live streaming. These execution checks complement MMBench and Video-MME.

Download and run

The shortest supported setup is the guided installer in the GitHub repository:

git clone https://github.com/JohnTDI-cpu/mage-vl-gguf.git
cd mage-vl-gguf
./install.sh

It detects Vulkan/CUDA and available VRAM, selects an 8/16/24–32 GB profile, downloads the required files, creates .env and starts Docker. Manual model download remains available below.

python3 -m venv .hf-venv
.hf-venv/bin/pip install "huggingface_hub>=0.34"
.hf-venv/bin/hf download JohnTdi/Mage-VL-GGUF \
  mage-vl-backbone-Q8_0.gguf mage-vit-mmproj-Q8_0.gguf \
  mage-streammind-epfe-Q8_0.gguf mage-streammind-cls-Q8_0.gguf \
  mage-dcvc-rt-intra-F16.gguf mage-dcvc-rt-inter-F16.gguf \
  --local-dir models

Continue with the GitHub Docker instructions. The linked Docker guide applies the native runtime patch and launches both GGUF files.

After startup, open http://localhost:8080 and select an MP4, or use the API below. H.264/HEVC are processed directly; AV1, VP9, MPEG-4 Part 2 and other FFmpeg-readable codecs are converted to H.264 automatically. No manual MAGECV1 preparation is required, and audio tracks are ignored:

curl --fail http://localhost:8080/v1/video/analyze \
  -F video=@./video.mp4 \
  -F 'prompt=Describe the important events in temporal order.' \
  -F max_tokens=256

The gateway runs codec-video-prep==0.2.5, packs MAGECV1, caches the result by content hash and invokes native patched llama.cpp. JPEG/PNG images use /v1/image/analyze. The public service binds to 127.0.0.1 by default.

The included Mage-VL Studio browser panel provides drag-and-drop upload, a local player with an analysis-range selector, a five-level Speed/Detail control, and an optional dedicated OCR pass for static captions and UI. Its three evidence views show (1) the exact patch mosaics sent to Mage-ViT, (2) representative full-frame highlights, and (3) every source frame whose patches contributed to inference, all with exact timestamps.

Advanced settings can restart the local model with a different context, batch/micro-batch, and F16/Q8/Q4 KV cache after explicit confirmation. The resource panel reports combined gateway+llama resident RAM. VRAM uses per-process nvidia-smi on NVIDIA and a clearly labeled isolated-DRM-device delta from the pre-load baseline on AMD/Vulkan.

The browser path was exercised end-to-end on Firefox, Chromium and WebKit, including upload, range clipping, all evidence views, frame seeking, OCR, settings confirmation/restart, invalid inputs and responsive layout. The corresponding Playwright smoke test is included in the GitHub runtime repo.

Demonstration using a randomly selected public YouTube live stream; the source was chosen only to exercise the live-analysis path and is not an endorsement.

Measured live-stream capacity

On the test Radeon AI PRO R9700, native DCVC-RT plus canvas construction measured 14.30 fps at 854x480, 6.34 fps at 1280x720 and 1.55 fps at 1920x1080. The 16 GiB Docker profile therefore accepts a 1080p/4K source but downsizes it to at most 480 pixels high and analyzes 8 fps. A warmed container processed 17 sampled portrait-480p frames in 0.787 s. An empty Vulkan shader cache adds a one-time 15-17 second cold start.

The live path is native FFmpeg -> DCVC-RT GGUF -> canvas -> Mage-ViT -> StreamMind -> Qwen; it creates no intermediate MP4/MAGECV1 files and does not run Transformers. Codec, prompt/generation length, driver, power state and content change results. The GitHub repository includes the raw-method description and a reusable tools/live_benchmark.py JSON/Markdown harness.

Analyzed FPS is a user setting; this release does not automatically benchmark the GPU or change FPS during a session. The bounded queue can drop stale windows to avoid unlimited delay. Tune FPS and the live height limit from the p95 RTF and queue telemetry as documented in the GitHub runtime guide.

SHA-256

e842701e4a1b8363b49bcd5c96baea6707960b1d11ecfa468d6ce4b0a4ae9343  mage-vl-backbone-Q8_0.gguf
e8959bb666872d10cd826e0f76542a89e39e81dec18b2563ec3d281abfdd9fb0  mage-vl-backbone-Q4_K_M.gguf
5a0773a458eae3a75a8d9bc18ccd13d5d3042052e73e1e125b66a6d3653257cb  mage-vit-mmproj-F16.gguf
8a9d784e666f8178bc805b5a157ac56e52a66be9eef5a73035c746461df1d231  mage-vit-mmproj-Q8_0.gguf
a812b812ee33614e44a150410d55f41c394eb8dc3749e5ee12734e3e0c046589  mage-streammind-epfe-Q8_0.gguf
2b642c25a6d5e66242440a813c1cb229138ac3a638122970ed8ee3bcb13c79c3  mage-streammind-cls-Q8_0.gguf
4e54c52153242bbeebeb412aca8375a7e9b8ed0419475549fdbf5a9572f19bfa  mage-dcvc-rt-intra-F16.gguf
8355bc95d66e7198cd2f76917ab0c999373bbcdaac3af72ea388dd349bc1a659  mage-dcvc-rt-inter-F16.gguf

Test system

AMD EPYC 7543 (32 cores / 64 threads), Radeon AI PRO R9700 32 GiB, Ubuntu 24.04, Mesa 25.2.8, Vulkan/RADV and llama.cpp commit a52077c4cabb4f3c0298329c9d2dd1324d5604cb.

License

Mage-VL is released under Apache-2.0. Dataset licenses and terms remain independent and their datasets are not redistributed here.