README.md

---
license: apache-2.0
base_model: stepfun-ai/GOT-OCR2_0
tags:
- gguf
- ocr
- crispembed
- got-ocr2
library_name: crispembed
---

GGUF conversion of [stepfun-ai/GOT-OCR2_0](https://huggingface.co/stepfun-ai/GOT-OCR2_0) for use with [CrispEmbed](https://github.com/CrispStrobe/CrispEmbed).

## Architecture

- **Vision**: SAM ViT-B (12 layers, 768d, 12 heads, 16×16 patches, 1024×1024 input)
  - Windowed attention (ws=14) with global attention at layers [2, 5, 8, 11]
  - Decomposed relative position encoding
  - Neck: Conv(768→256) → LN2d → Conv(256→256) → LN2d
  - Downsample: Conv(256→512→1024, stride 2) → 256 vision tokens
  - Projector: Linear(1024, 1024)
- **LLM**: Qwen2-0.5B (24 layers, 1024d, MHA 16/16, SiLU SwiGLU, RoPE θ=1M)
- **Tokenizer**: tiktoken (151860 vocab)
- **Total**: ~0.7B parameters

## Files

| File | Precision | Size | Notes |
|------|-----------|------|-------|
| `got-ocr2-q4_k.gguf` | **Q4_K** | 445 MB | **Recommended / default.** Correct OCR, fastest decode on Apple Silicon |
| `got-ocr2-q8_0.gguf` | Q8_0 | 599 MB | Correct OCR; on M1 the Q8_0 `mul_mv` path is slower per-token than Q4_K, so Q4_K is preferred |
| `got-ocr2-f16.gguf` | F16 | 1.44 GB | Full precision baseline |

## Precision & parity

The Qwen2-0.5B decoder quantizes **cleanly to Q4_K and Q8_0** — all three
builds above produce identical, correct OCR. Verified against the real HF model
(transformers `GotOcr2`) plus a Python f32 reference:

- **Vision** (ViT layers, neck, downsample, projector): cos ≥ 0.998 vs HF.
- **LLM decoder** (per-layer, Q8_0 weights vs f32 reference): **cos ≥ 0.99996**.

Per-token decode speed on an M1 (256 vision tokens spliced into the prompt):

| Build | Decode | 
|------|--------|
| Q4_K | ~20 ms/tok |
| F16  | ~38 ms/tok |
| Q8_0 | ~42 ms/tok |

Q4_K is ~2× faster to decode than F16 and 3× smaller, so it is the default.

> **Note on earlier builds.** A prior version of this repo shipped an
> F16-decoder build and claimed the 0.5B decoder was "catastrophically
> sensitive to quantization" (`llm_layer_0` cos ≈ 0.936 at Q8_0). That number
> was a **measurement artifact** of a per-row bug in the diff harness (it used
> the token count as the row length), not real quant sensitivity. With the
> corrected harness the Q8_0/Q4_K decoder matches f32 at cos ≥ 0.99996 and OCR
> output is identical to F16. See CrispEmbed issue #25.

## Usage

```bash
crispembed --ocr got-ocr2 image.png
```

## Reproducing the quants

```bash
crispembed-quantize got-ocr2-f16.gguf got-ocr2-q4_k.gguf q4_k
crispembed-quantize got-ocr2-f16.gguf got-ocr2-q8_0.gguf q8_0
```

(The quantizer also has an optional `--decoder-f16` flag that keeps the decoder
weights at F16; it is **not** needed for correctness and is retained only for
diagnostic / comparison use.)

## License

Apache-2.0 (same as upstream model)