README.md

---
license: apache-2.0
library_name: mlx
pipeline_tag: image-text-to-text
tags:
- mlx
- ocr
- document-parsing
- multimodal
- vision-language
base_model: ATH-MaaS/OvisOCR2
---

# OvisOCR2-4bit

4-bit MLX quantization of
[`ATH-MaaS/OvisOCR2`](https://huggingface.co/ATH-MaaS/OvisOCR2), an 853M-parameter
OCR / document-parsing VLM. Converted with `mlx-vlm` 0.6.8 (`mlx` 0.32.0) for
inference on Apple Silicon.

Fidelity was measured against the unquantized bf16 source, not assumed. The
numbers below and the method that produced them are in the tables that follow.

```bash
pip install mlx-vlm
python -m mlx_vlm generate \
    --model mlx-community/OvisOCR2-4bit \
    --image document.png \
    --prompt "Transcribe this document to markdown." \
    --max-tokens 512
```

## Quantization

| | |
|---|---|
| Bits | 4 |
| Group size | 64 |
| Mode | affine |
| Effective bits/weight | **5.863** |
| Size on disk | 0.63 GB |
| Quantized tensors | 187 (language model) |
| Unquantized tensors | 153 (vision tower, bf16) |

Effective bits per weight exceeds the nominal 4 because `mlx-vlm` quantizes
only the language model and leaves the vision tower in bf16 by design. Verified,
not assumed: of 153 vision tensors, **zero** carry a `.scales` entry.

Note this model ties its input and output embeddings, and the embedding is 33.8%
of all quantized parameters, a much larger share than in a typical 7B. Weight
error is therefore dominated by the embedding rather than the attention layers.

## Measured fidelity

Weight-level metrics compare dequantized weights against the bf16 source tensor
by tensor, streamed from disk. No prompts, no sampling, hardware-independent.

| Variant | bpw | SNR | Relative L2 | Cosine | CER vs bf16 | Decode tok/s | Peak RAM |
|---|---|---|---|---|---|---|---|
| 4-bit | 5.863 | 20.58 dB | 9.35% | 0.995632 | 0.3311 | 220.3 | 0.86 GB |
| 6-bit | 7.626 | 32.84 dB | 2.28% | 0.999740 | 0.0000 | 183.5 | 1.08 GB |
| 8-bit | 9.389 | 42.67 dB | 0.74% | 0.999973 | 0.0000 | 160.9 | 1.31 GB |
| bf16 (source) | 16 | n/a | n/a | n/a | 0.0000 | 91.0 | 1.83 GB |

Measured on an M2 Pro / 32 GB, greedy decoding.

## OCR accuracy vs ground truth

Because OCR has *ground truth*, transcription can be scored directly rather
than inferred from token distributions. Five documents were rendered locally with exactly known content
(invoice, clinical lab report, shipping label, receipt, motor-controller
datasheet), covering tabular line items, currency, dates and identifiers.

| Variant | Field accuracy | Content accuracy | Numeric recall | CER vs bf16 |
|---|---|---|---|---|
| 4-bit | 0.9677 | 1.0000 | 1.0000 | 0.3311 |
| 6-bit | 1.0000 | 1.0000 | 1.0000 | 0.0000 |
| 8-bit | 1.0000 | 1.0000 | 1.0000 | 0.0000 |
| bf16 (source) | 1.0000 | 1.0000 | 1.0000 | 0.0000 |

- **Field accuracy**: required strings present, exact match (format-sensitive).
- **Content accuracy**: the same, ignoring markup and whitespace.
- **Numeric recall**: ground-truth numbers present, ignoring markup. For an
  invoice or a datasheet the numbers *are* the payload.

**Every variant preserves 100% of content and 100% of the 77 ground-truth
numbers.** At 6 and 8 bits the transcriptions are **character-identical to
bf16** (CER 0.0000 across all five documents), not merely close.

### Why 4-bit shows CER 0.331 but loses no content

The 4-bit variant scores 0.968 on strict field accuracy and 0.331 CER, which
looks like degradation. It is not. At 4 bits the model emits an **HTML table**
where bf16 emits markdown:

```
bf16 : Widget A 2 12.00 24.00
4-bit: <tr><td>WidgetA</td><td>2</td><td>12.00</td><td>24.00</td></tr>
```

Every figure is correct. The only strict-match failures in all five documents
are `WidgetA` against `Widget A`, a single dropped space. Content accuracy and
numeric recall are both 1.000.

CER against bf16 treats the bf16 output as ground truth, so it penalises a valid
alternative rendering the same way BLEU penalises paraphrase. Reported here for
continuity, but it measures *drift*, not quality. If you post-process into
structured data, 4-bit is usable and is the fastest of the three; if you need
byte-stable markdown, use 6-bit.

## What was not measured

- **No distributional metrics (perplexity, top-1 agreement, KL).** They were
  run and are deliberately not reported. This model is an OCR specialist, and on
  general chat text the bf16 source itself degenerates (asked for the capital of
  Japan it emits `Tokyo / Nara / Nara / Nara ...`). Its perplexity spans 49.3x
  across passages of ordinary prose, so an aggregate over them measures domain
  mismatch in the source, not quantization error. Per-passage ratios do improve
  monotonically from 4 to 8 bits.
- **No standard OCR benchmark.** No OmniDocBench, no DocVQA, no CER against a
  public dataset. The five documents are synthetic and rendered with a default
  font: exact ground truth, but not photographs, not handwriting, not skewed
  scans, and not non-Latin scripts.
- **No judged quality**, no layout-structure scoring (reading order, cell
  spanning), no throughput figures for any chip other than an M2 Pro.

## Sibling variants

| Variant | bpw | Character-identical to bf16 |
|---|---|---|
| [OvisOCR2-4bit](https://huggingface.co/mlx-community/OvisOCR2-4bit) | 5.863 | no, formatting differs |
| [OvisOCR2-6bit](https://huggingface.co/mlx-community/OvisOCR2-6bit) | 7.626 | yes |
| [OvisOCR2-8bit](https://huggingface.co/mlx-community/OvisOCR2-8bit) | 9.389 | yes |

## Credits

All credit for the underlying model belongs to
[ATH-MaaS](https://huggingface.co/ATH-MaaS). This is a format conversion and
quantization; no training or fine-tuning was performed. See the
[source model card](https://huggingface.co/ATH-MaaS/OvisOCR2) for license,
intended use and limitations.