README.md

---
library_name: mlx
license: apache-2.0
pipeline_tag: image-text-to-text
base_model: CohereLabs/North-Micro-Vision-Instruct
language:
- en
- de
- fr
- es
- it
- pt
- hi
- ja
- ko
- zh
- ar
tags:
- mlx
- mlx-vlm
- openmed
- openmedkit
- apple-silicon
- on-device
- vision
- multimodal
- clinical
- medical
- privacy
- native-resolution
---

# North Micro Vision Instruct — OpenMed MLX family

On-device, native-resolution vision-language inference for Apple Silicon,
prepared for OpenMed's privacy-first clinical document workflows and the
future Cohere Compass runtime in [OpenMedKit](https://github.com/maziyarpanahi/openmed).

These repositories contain MLX conversions of
[`CohereLabs/North-Micro-Vision-Instruct`](https://huggingface.co/CohereLabs/North-Micro-Vision-Instruct),
a compact 2.4B-parameter vision-language model released by Cohere under
Apache 2.0. The same byte-identical README is used across the five precision
variants; the repository name and `config.json` identify the precision.

## Choose a precision

| Repository | Weight payload | Intended trade-off |
| --- | ---: | --- |
| [`OpenMed/North-Micro-Vision-Instruct-4bit-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-4bit-mlx) | 2.02 GiB | Smallest affine variant; validate quality on your exact document set |
| [`OpenMed/North-Micro-Vision-Instruct-5bit-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-5bit-mlx) | 2.25 GiB | Compact middle ground |
| [`OpenMed/North-Micro-Vision-Instruct-6bit-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-6bit-mlx) | 2.48 GiB | Recommended first on-device quality/size trial |
| [`OpenMed/North-Micro-Vision-Instruct-8bit-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-8bit-mlx) | 2.93 GiB | Higher-fidelity quantized variant |
| [`OpenMed/North-Micro-Vision-Instruct-bf16-mlx`](https://huggingface.co/OpenMed/North-Micro-Vision-Instruct-bf16-mlx) | 4.63 GiB | Full converted precision and quality reference |

The 4/5/6/8-bit repositories use 64-element affine weight groups. Their
vision tower remains in source precision; eligible language-model layers are
quantized. This matters for document and OCR work because visual feature
extraction is not compressed just to reduce the decoder footprint.

## Why this model is interesting for OpenMedKit

North Micro Vision accepts interleaved text and images, preserves native image
aspect ratios, and was trained for OCR, document understanding, charts,
grounding, captioning, multilingual prompts, and multi-image conversations.
That combination maps naturally to an OpenMedKit document pipeline:

1. Capture or import a page locally on iPhone, iPad, or Mac.
2. Keep image normalization, OCR/VLM inference, PII handling, and structured
   extraction on the user's device.
3. Validate generated fields against the source page before they enter a
   clinical record or workflow.
4. Redact or pseudonymize with OpenMedKit policies before any explicitly
   authorized export.
5. Never auto-trigger diagnosis, treatment, disclosure, or another
   consequential clinical decision from model output.

OpenMedKit's design defaults remain important even when the model is local:
no telemetry by default, no cloud fallback for PHI, no raw clinical text in
logs or analytics, synthetic fixtures in committed tests, and human review for
consequential use.

## Runtime status: read this before using Swift

The model weights use the standard MLX-VLM artifact layout and include the
source tokenizer, chat template, native-resolution image processor metadata,
and Cohere Compass configuration. They are validated today with the pinned
Python MLX-VLM Compass runtime on Apple Silicon.

The released OpenMedKit Swift package does **not yet contain a native Cohere
Compass VLM implementation**. Do not assume that downloading one of these
repositories through `OpenMedModelStore` is sufficient for iOS inference.
These artifacts are deliberately packaged without conversion-time Python
dependencies so a future OpenMedKit Compass loader can consume the same weight
payloads, but native Swift/iOS support must still land and pass physical-device
memory, thermal, image, and text parity gates.

Until that runtime ships, use MLX-VLM on an Apple Silicon Mac for the model
itself. OpenMedKit can still own capture, privacy policy, redaction, structured
validation, and the surrounding app workflow. Do not send raw PHI from an iOS
device to a Mac or service unless the user and deployment policy explicitly
authorize that transfer.

## Install the validated MLX runtime

Cohere Compass support is pinned to the exact MLX-VLM port used for conversion
and validation:

```bash
python -m pip install -U \
  "git+https://github.com/Blaizzy/mlx-vlm.git@dd79a5d8caf3edafd6fa9e6326d7ce4977ddcbfc"
```

The validated environment uses MLX 0.32.0, MLX-LM 0.31.3, MLX-VLM 0.6.10
from that commit, and Transformers 5.15.0. Once Cohere Compass support is in a
released MLX-VLM package, a normal `pip install -U mlx-vlm` can replace the
commit pin after you rerun your own parity tests.

## Image + text example

```python
from mlx_vlm import apply_chat_template, generate, load

model_id = "OpenMed/North-Micro-Vision-Instruct-6bit-mlx"
model, processor = load(model_id)

prompt = apply_chat_template(
    processor,
    model.config,
    "Read this synthetic document and list the visible medication and dose.",
    num_images=1,
)
result = generate(
    model,
    processor,
    prompt,
    image="synthetic-clinical-note.png",
    max_tokens=128,
    temperature=0.0,
)
print(result.text)
```

Use synthetic or properly authorized content while prototyping. For a real
clinical app, treat both the image and generated text as sensitive until the
OpenMedKit privacy policy has been applied.

## Text-only example

```python
from mlx_vlm import apply_chat_template, generate, load

model, processor = load(
    "OpenMed/North-Micro-Vision-Instruct-6bit-mlx"
)
prompt = apply_chat_template(
    processor,
    model.config,
    "Explain in one sentence why local processing can improve document privacy.",
    num_images=0,
)
result = generate(
    model,
    processor,
    prompt,
    max_tokens=80,
    temperature=0.0,
)
print(result.text)
```

System prompts are not recommended by the source model authors. Keep
instructions concise and place the task in the user message.

## OpenMedKit integration gates

Before these weights are advertised as a native OpenMedKit/iOS runtime, the
Swift implementation must demonstrate all of the following on a physical
iPhone/iPad as well as Apple Silicon macOS:

- strict weight loading for bf16 and every affine bit width;
- tokenizer and chat-template parity with the validated Python path;
- native-resolution image preprocessing and visual-token count parity;
- coherent deterministic text-only generation;
- correct OCR/document answers on synthetic fixtures;
- bounded memory, launch latency, sustained thermals, and cancellation;
- no network fallback, telemetry, or plaintext PHI logging;
- provenance, model revision, and privacy-policy visibility in the app UI.

That is the bar for “OpenMedKit-ready”; an MLX file extension alone is not.

## Validation performed before publication

Every OpenMed variant is loaded independently with strict tensor matching and
must pass the same deterministic suite before any repository is made public:

- coherent text-only privacy explanation;
- exact fact extraction from a short synthetic text note;
- exact extraction of name, synthetic record ID, medication, dose, frequency,
  and allergy from a generated non-PHI clinical document image;
- exact category/value extraction from a generated chart image;
- model type, processor assets, quantization metadata, and weight-size checks;
- bf16 SHA-256 payload parity with an independently published Cohere-linked
  conversion made from the same pinned source and runtime port;
- quantized payload-size parity plus an explicit nonzero token-embedding gate
  (whole-file hashes are recorded but are not treated as a correctness proxy).

The fixtures are synthetic and are not clinical-quality evidence. Passing
these checks establishes conversion/runtime coherence, not safety or accuracy
for a particular medical workflow. Evaluate the exact document types,
languages, device targets, and failure modes your app will encounter.

## Source model scope and limitations

The source checkpoint supports native-resolution and multiple images,
multilingual conversations, visual question answering, grounding, OCR, and
chart/document understanding. Its language backbone supports a long context,
but Cohere reports multimodal training and validation up to 8K tokens; do not
silently claim longer multimodal reliability.

The source model is not a reasoning model, has limited math and code ability,
does not support tool calling or agentic workflows, and is intended as a
compact foundation for prototyping and specialization. Native-resolution
images can materially increase memory use and latency.

This model and OpenMedKit are not medical devices. Outputs can be incomplete,
incorrect, or fabricated. A qualified human must verify consequential use.

## Reproducibility and credits

- Source: [`CohereLabs/North-Micro-Vision-Instruct`](https://huggingface.co/CohereLabs/North-Micro-Vision-Instruct)
- Pinned source revision: `373bda96ac70bf89f99f7048f420cf00dc07c149`
- MLX-VLM Compass port: [`dd79a5d8caf3edafd6fa9e6326d7ce4977ddcbfc`](https://github.com/Blaizzy/mlx-vlm/commit/dd79a5d8caf3edafd6fa9e6326d7ce4977ddcbfc)
- OpenMed / OpenMedKit: [github.com/maziyarpanahi/openmed](https://github.com/maziyarpanahi/openmed)

Thank you to Cohere for releasing North Micro Vision and to Prince Canuma and
the MLX-VLM contributors for the Apple Silicon runtime and Compass port.

The converted weights retain the source model's Apache 2.0 license. OpenMed's
SDK source is separately licensed under Apache 2.0.