ysharma/North-Micro-Vision-Instruct-GGUF

North-Micro-Vision-Instruct — first GGUFs

First GGUF conversions of CohereLabs/North-Micro-Vision-Instruct (2.48B, cohere_compass, Apache-2.0).

Files

File	Size	Use
north-micro-vision-instruct-f16.gguf	3.9 GB	F16 reference (≈ BF16 quality), the baseline other quants are compared to
north-micro-vision-instruct-q8_0.gguf	2.1 GB	Q8_0 — near-lossless, ~47% smaller
north-micro-vision-instruct-q4_k_m.gguf	1.3 GB	Q4_K_M — smallest practical, ~67% smaller
mmproj-north-f16.gguf	1.1 GB	Vision encoder (mmproj), required with every text file

Usage

llama-mtmd-cli -m north-micro-vision-instruct-q8_0.gguf \
    --mmproj mmproj-north-f16.gguf \
    --image photo.jpg -p "Describe this image." -n 128

Or with llama-server / any OpenAI-compatible client, passing --mmproj alongside the model.

How these were built (important caveat)

At conversion time, cohere_compass support did not exist in upstream llama.cpp (the widely-shared announcement that Cohere's models "work with llama.cpp" refers to PR #30062, which is cohere2 vision for Command A+, a different architecture). These GGUFs were produced with a community port (full patch, against llama.cpp commit 785322d17f): a new cohere_compass arch in the GGUF converter + C++ graph, reusing the cohere2 decoder structure and the Qwen3-VL mmproj/deepstack machinery (cohere_compass_vision is structurally a SigLIP2-dimensioned Qwen3-VL tower).

Verified against the HF transformers implementation (5.22.1) on CPU:

• Text: exact — greedy decoding matches transformers token-for-token (28/28 tokens incl. EOS, two independent prompts + forced-continuation probes) after fixing the interleaved M-RoPE wiring (rope.dimension_sections [24,20,20,0], full 128-dim rotation, θ 50000).
• Image: smoke-tested — the F16 and Q8_0/Q4_K_M files load the mmproj, encode images and produce coherent descriptions (e.g. a correct Volkswagen Beetle description of the standard car.jpg fixture). A token-level image oracle was not completed before publish, so treat vision quality as unverified-at-quantization-level until upstream lands.

This port is not merged upstream. When ggml-org/llama.cpp adds cohere_compass support, prefer official conversions.

Recommendation

Quant	Accuracy	Speed / size	Pick it when
F16	reference	3.9 GB	you want the unquantized baseline
Q8_0	≈ F16	2.1 GB	default choice — near-lossless at half the size
Q4_K_M	good	1.3 GB	tight memory; expect a small quality drop

(A quantitative accuracy/speed table against BF16 over 5 benchmarks was scoped for this task but not funded — see the benchmark-phase note in the port thread.)

License

Model weights and these conversions: Apache-2.0, © Cohere Labs. Port patch: MIT-style per llama.cpp license.