File size: 7,456 Bytes

---
license: mit
base_model: zai-org/GLM-OCR
language:
- en
pipeline_tag: image-to-text
tags:
- ocr
- latex
- mathematics
- lora
- peft
---

# GLM-OCR: Fine-tuned for Handwritten Math OCR → LaTeX

LoRA fine-tunes of [zai-org/GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) (0.9B) for transcribing
handwritten university-level math answer sheets directly into compilable LaTeX.

**Code**: [github.com/realgauravvyas/ocr2tex](https://github.com/realgauravvyas/ocr2tex) — full data
pipeline, GLM-OCR + Baidu OCR training scripts, benchmark harness, and the live dashboard.

Two adapter versions are included in this repo:

| Version | Path | Notes |
|---|---|---|
| **v3.1** | [`v3.1/`](./v3.1) | Trained on 4,672 pages |
| **v4.1** | [`v4.1/`](./v4.1) | Trained on 12,575 pages; warm-started from the v4 adapter |

## Results (held out, 700 pages)

CER is measured against **silver labels from a teacher VLM**, not human-verified ground truth —
treat absolute CER as teacher-agreement, not true transcription accuracy. Relative comparisons
across rows share the same reference and remain sound.

Results are grouped by comparison type, since mixing them in one flat table obscures which rows
are actually comparable:

**Out-of-the-box baselines** (neither system fine-tuned on this project's data)

| System | Mean CER | Med CER | Compile % | Struct % | BLEU-4 | chrF | MathF1 | Latency (s) |
|---|---|---|---|---|---|---|---|---|
| Base GLM-OCR (frozen) | 0.5151 | 0.4609 | 0.0 | 0.0 | 0.4583 | 0.5370 | 0.7031 | 8.38 |
| Baidu OCR (stock) | 0.7176 | 0.5768 | 42.7 | 77.6 | 0.3113 | 0.4792 | 0.6264 | 40.79 |

**Fine-tuned on the same 12,575-page training corpus** (`workspace/9_split`) — controlled comparison

| System | Mean CER | Med CER | Compile % | Struct % | BLEU-4 | chrF | MathF1 | Latency (s) |
|---|---|---|---|---|---|---|---|---|
| Baidu OCR (fine-tuned) | 0.4071 | 0.3523 | 62.7 | 71.7 | 0.5872 | 0.7070 | 0.7959 | 24.93 |
| Baidu OCR (fine-tuned v2) | 0.4258 | 0.3501 | 64.9 | 71.9 | 0.5958 | 0.7176 | 0.7983 | 29.00 |
| GLM-OCR v4 (ours, intermediate) | 0.3837 | 0.2834 | 81.3 | 90.4 | 0.6489 | 0.7504 | 0.8277 | 15.62 |
| **GLM-OCR v3.1 (ours)** | 0.3971 | 0.2957 | **88.9** | **91.4** | 0.6180 | 0.7093 | 0.8171 | 14.12 |
| **GLM-OCR v4.1 (ours)** | **0.3816** | **0.2783** | 82.4 | 90.9 | **0.6513** | **0.7531** | **0.8272** | 13.44 |

Note: v3.1 was trained on a 4,672-page subset of this corpus, not the full 12,575 (see Training
configuration below) — it's grouped here because it's still a same-source fine-tune comparison,
not an out-of-the-box baseline.

- Base compile rate is **exactly** 0.0% — 0 of 700 pages produced a compiling PDF without fine-tuning.
- v4.1 has the lowest CER and best BLEU/chrF/MathF1, but **v3.1 compiles more often** (88.9% vs 82.4%).
  Scaling the training set 4,672 → 12,575 pages bought only ~0.016 CER — evidence the task is
  annotation-bound, not capacity-bound. The compile-rate regression tracks a shift in average
  output length (0.98x → 1.07x of reference length in v4.1).
- Both Baidu fine-tunes were trained on the identical page corpus used for GLM-OCR v4/v4.1, so the
  gap between them and v3.1/v4/v4.1 reflects the base models and training recipes, not a data
  advantage on either side.
- Headline: mean CER 0.515 → 0.382, a 25.9% relative reduction over the frozen base model.

## Training configuration

This repo ships the two versions selected from a broader set of experiments (v3, v3.1, v4, v4.1)
run over the course of the project. Full training configs are documented below for **v3.1 and
v4.1 only** — the two adapters actually released here. v4 appears in the results table above as
an intermediate checkpoint for context, but its run wasn't logged with the same rigor as the two
released versions and isn't included as a downloadable adapter in this repo.

Both released versions share the same LoRA setup — r=32, alpha=64, dropout=0.05, targets
q_proj/k_proj/v_proj/o_proj/gate_proj/up_proj/down_proj — and were trained on a single RTX 3060 12GB
(no cloud compute) with Python 3.11.9, PyTorch 2.10.0+cu130, Transformers 5.9.0, PEFT 0.18.1,
epochs=2, batch=1, grad_accum=8 (effective batch 8), bf16, seed=42. They differ as follows:

| | v3.1 | v4.1 |
|---|---|---|
| Training pages | 4,672 | 12,575 |
| Full validation partition | 584 | 698 |
| Periodic in-training eval subset | 200 (capped, for speed) | 200 (capped, for speed) |
| Learning rate | 2e-5 | 1e-5 |
| Warm start | from the v3 adapter | from the v4 adapter |
| Steps | 1,168 | 3,144 |
| Best val loss | 0.108 | 0.164 |

- max_length=3584, max_image_tokens=1536
- Decoding (benchmark): greedy (`do_sample=False`), max_new_tokens=2048, repetition_penalty=1.0
- Training time: v3.1 ~5 hours (1,168 steps) · v4.1 ~12.7 hours (3,144 steps, run resumed across
  multiple sessions on shared personal hardware)

## Data

Raw scans → blank/printed filter → deskew/prepare → teacher-VLM annotation → automated quality
review → pdflatex-validity filter → train/val/test split (12,575 / 698 / 700).

## Usage

Load either version by pointing `subfolder` at `v3.1` or `v4.1`:

```python
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel

base = "zai-org/GLM-OCR"
processor = AutoProcessor.from_pretrained(base, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(base, trust_remote_code=True)

# v4.1 — lower CER, best BLEU/chrF/MathF1
model = PeftModel.from_pretrained(model, "ctogaurav/GLM_OCR", subfolder="v4.1")

# v3.1 — higher compile rate (88.9% vs 82.4%)
# model = PeftModel.from_pretrained(model, "ctogaurav/GLM_OCR", subfolder="v3.1")
```

## Running locally (LM Studio / Ollama / llama.cpp)

**v4.1 is available as a ready-to-download GGUF**, verified working end-to-end (not just
"conversion succeeded" — output checked against this project's own validated ground truth):
[ctogaurav/GLM_OCR-GGUF](https://huggingface.co/ctogaurav/GLM_OCR-GGUF). Download both
`GLM-OCR-v4.1-Q8_0.gguf` and `mmproj-GLM-OCR-v4.1-Q8_0.gguf` (~1.2GB total) into the same folder
in LM Studio, or run directly with `llama.cpp`'s `llama-mtmd-cli`.

Producing this required a small local patch to llama.cpp's own conversion code — the public
`zai-org/GLM-OCR` checkpoint doesn't ship the weights its conversion path expects for an unused
auxiliary layer, causing a hard load failure otherwise. Full details, including the fix, are in
the [GGUF repo's README](https://huggingface.co/ctogaurav/GLM_OCR-GGUF).

v3.1 isn't converted yet — the same process applies (merge with `PeftModel.merge_and_unload()`,
convert with a patched `llama.cpp`), just not done for this version yet.

## License

Both adapters are released under **MIT**, matching the base model's license. The base
[zai-org/GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) model card notes that Apache-2.0-licensed
components (PP-DocLayoutV3) are used in some parts of its *broader* published pipeline — this
project doesn't use PP-DocLayoutV3 anywhere, so only MIT applies here. You're free to merge,
quantize, redistribute, and use these adapters commercially, including as GGUF conversions.

## Limitations

- Evaluation CER is against teacher-VLM silver labels, not human-verified transcriptions.
- Trained on university-level handwritten math; performance on other domains (e.g. chemistry
  notation, diagrams, non-English handwriting) is untested.