| --- |
| license: mit |
| base_model: zai-org/GLM-OCR |
| language: |
| - en |
| pipeline_tag: image-to-text |
| tags: |
| - ocr |
| - latex |
| - mathematics |
| - lora |
| - peft |
| --- |
| |
| # GLM-OCR: Fine-tuned for Handwritten Math OCR → LaTeX |
|
|
| LoRA fine-tunes of [zai-org/GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) (0.9B) for transcribing |
| handwritten university-level math answer sheets directly into compilable LaTeX. |
|
|
| **Code**: [github.com/realgauravvyas/ocr2tex](https://github.com/realgauravvyas/ocr2tex) — full data |
| pipeline, GLM-OCR + Baidu OCR training scripts, benchmark harness, and the live dashboard. |
|
|
| Two adapter versions are included in this repo: |
|
|
| | Version | Path | Notes | |
| |---|---|---| |
| | **v3.1** | [`v3.1/`](./v3.1) | Trained on 4,672 pages | |
| | **v4.1** | [`v4.1/`](./v4.1) | Trained on 12,575 pages; warm-started from the v4 adapter | |
|
|
| ## Results (held out, 700 pages) |
|
|
| CER is measured against **silver labels from a teacher VLM**, not human-verified ground truth — |
| treat absolute CER as teacher-agreement, not true transcription accuracy. Relative comparisons |
| across rows share the same reference and remain sound. |
|
|
| Results are grouped by comparison type, since mixing them in one flat table obscures which rows |
| are actually comparable: |
|
|
| **Out-of-the-box baselines** (neither system fine-tuned on this project's data) |
|
|
| | System | Mean CER | Med CER | Compile % | Struct % | BLEU-4 | chrF | MathF1 | Latency (s) | |
| |---|---|---|---|---|---|---|---|---| |
| | Base GLM-OCR (frozen) | 0.5151 | 0.4609 | 0.0 | 0.0 | 0.4583 | 0.5370 | 0.7031 | 8.38 | |
| | Baidu OCR (stock) | 0.7176 | 0.5768 | 42.7 | 77.6 | 0.3113 | 0.4792 | 0.6264 | 40.79 | |
|
|
| **Fine-tuned on the same 12,575-page training corpus** (`workspace/9_split`) — controlled comparison |
|
|
| | System | Mean CER | Med CER | Compile % | Struct % | BLEU-4 | chrF | MathF1 | Latency (s) | |
| |---|---|---|---|---|---|---|---|---| |
| | Baidu OCR (fine-tuned) | 0.4071 | 0.3523 | 62.7 | 71.7 | 0.5872 | 0.7070 | 0.7959 | 24.93 | |
| | Baidu OCR (fine-tuned v2) | 0.4258 | 0.3501 | 64.9 | 71.9 | 0.5958 | 0.7176 | 0.7983 | 29.00 | |
| | GLM-OCR v4 (ours, intermediate) | 0.3837 | 0.2834 | 81.3 | 90.4 | 0.6489 | 0.7504 | 0.8277 | 15.62 | |
| | **GLM-OCR v3.1 (ours)** | 0.3971 | 0.2957 | **88.9** | **91.4** | 0.6180 | 0.7093 | 0.8171 | 14.12 | |
| | **GLM-OCR v4.1 (ours)** | **0.3816** | **0.2783** | 82.4 | 90.9 | **0.6513** | **0.7531** | **0.8272** | 13.44 | |
|
|
| Note: v3.1 was trained on a 4,672-page subset of this corpus, not the full 12,575 (see Training |
| configuration below) — it's grouped here because it's still a same-source fine-tune comparison, |
| not an out-of-the-box baseline. |
|
|
| - Base compile rate is **exactly** 0.0% — 0 of 700 pages produced a compiling PDF without fine-tuning. |
| - v4.1 has the lowest CER and best BLEU/chrF/MathF1, but **v3.1 compiles more often** (88.9% vs 82.4%). |
| Scaling the training set 4,672 → 12,575 pages bought only ~0.016 CER — evidence the task is |
| annotation-bound, not capacity-bound. The compile-rate regression tracks a shift in average |
| output length (0.98x → 1.07x of reference length in v4.1). |
| - Both Baidu fine-tunes were trained on the identical page corpus used for GLM-OCR v4/v4.1, so the |
| gap between them and v3.1/v4/v4.1 reflects the base models and training recipes, not a data |
| advantage on either side. |
| - Headline: mean CER 0.515 → 0.382, a 25.9% relative reduction over the frozen base model. |
|
|
| ## Training configuration |
|
|
| This repo ships the two versions selected from a broader set of experiments (v3, v3.1, v4, v4.1) |
| run over the course of the project. Full training configs are documented below for **v3.1 and |
| v4.1 only** — the two adapters actually released here. v4 appears in the results table above as |
| an intermediate checkpoint for context, but its run wasn't logged with the same rigor as the two |
| released versions and isn't included as a downloadable adapter in this repo. |
|
|
| Both released versions share the same LoRA setup — r=32, alpha=64, dropout=0.05, targets |
| q_proj/k_proj/v_proj/o_proj/gate_proj/up_proj/down_proj — and were trained on a single RTX 3060 12GB |
| (no cloud compute) with Python 3.11.9, PyTorch 2.10.0+cu130, Transformers 5.9.0, PEFT 0.18.1, |
| epochs=2, batch=1, grad_accum=8 (effective batch 8), bf16, seed=42. They differ as follows: |
|
|
| | | v3.1 | v4.1 | |
| |---|---|---| |
| | Training pages | 4,672 | 12,575 | |
| | Full validation partition | 584 | 698 | |
| | Periodic in-training eval subset | 200 (capped, for speed) | 200 (capped, for speed) | |
| | Learning rate | 2e-5 | 1e-5 | |
| | Warm start | from the v3 adapter | from the v4 adapter | |
| | Steps | 1,168 | 3,144 | |
| | Best val loss | 0.108 | 0.164 | |
|
|
| - max_length=3584, max_image_tokens=1536 |
| - Decoding (benchmark): greedy (`do_sample=False`), max_new_tokens=2048, repetition_penalty=1.0 |
| - Training time: v3.1 ~5 hours (1,168 steps) · v4.1 ~12.7 hours (3,144 steps, run resumed across |
| multiple sessions on shared personal hardware) |
| |
| ## Data |
| |
| Raw scans → blank/printed filter → deskew/prepare → teacher-VLM annotation → automated quality |
| review → pdflatex-validity filter → train/val/test split (12,575 / 698 / 700). |
| |
| ## Usage |
| |
| Load either version by pointing `subfolder` at `v3.1` or `v4.1`: |
| |
| ```python |
| from transformers import AutoProcessor, AutoModelForImageTextToText |
| from peft import PeftModel |
| |
| base = "zai-org/GLM-OCR" |
| processor = AutoProcessor.from_pretrained(base, trust_remote_code=True) |
| model = AutoModelForImageTextToText.from_pretrained(base, trust_remote_code=True) |
| |
| # v4.1 — lower CER, best BLEU/chrF/MathF1 |
| model = PeftModel.from_pretrained(model, "ctogaurav/GLM_OCR", subfolder="v4.1") |
| |
| # v3.1 — higher compile rate (88.9% vs 82.4%) |
| # model = PeftModel.from_pretrained(model, "ctogaurav/GLM_OCR", subfolder="v3.1") |
| ``` |
| |
| ## Running locally (LM Studio / Ollama / llama.cpp) |
| |
| **v4.1 is available as a ready-to-download GGUF**, verified working end-to-end (not just |
| "conversion succeeded" — output checked against this project's own validated ground truth): |
| [ctogaurav/GLM_OCR-GGUF](https://huggingface.co/ctogaurav/GLM_OCR-GGUF). Download both |
| `GLM-OCR-v4.1-Q8_0.gguf` and `mmproj-GLM-OCR-v4.1-Q8_0.gguf` (~1.2GB total) into the same folder |
| in LM Studio, or run directly with `llama.cpp`'s `llama-mtmd-cli`. |
| |
| Producing this required a small local patch to llama.cpp's own conversion code — the public |
| `zai-org/GLM-OCR` checkpoint doesn't ship the weights its conversion path expects for an unused |
| auxiliary layer, causing a hard load failure otherwise. Full details, including the fix, are in |
| the [GGUF repo's README](https://huggingface.co/ctogaurav/GLM_OCR-GGUF). |
| |
| v3.1 isn't converted yet — the same process applies (merge with `PeftModel.merge_and_unload()`, |
| convert with a patched `llama.cpp`), just not done for this version yet. |
| |
| ## License |
| |
| Both adapters are released under **MIT**, matching the base model's license. The base |
| [zai-org/GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) model card notes that Apache-2.0-licensed |
| components (PP-DocLayoutV3) are used in some parts of its *broader* published pipeline — this |
| project doesn't use PP-DocLayoutV3 anywhere, so only MIT applies here. You're free to merge, |
| quantize, redistribute, and use these adapters commercially, including as GGUF conversions. |
| |
| ## Limitations |
| |
| - Evaluation CER is against teacher-VLM silver labels, not human-verified transcriptions. |
| - Trained on university-level handwritten math; performance on other domains (e.g. chemistry |
| notation, diagrams, non-English handwriting) is untested. |
| |
