--- license: apache-2.0 base_model: xiaomi-research/Xiaomi-OCR-0 base_model_relation: finetune library_name: transformers pipeline_tag: image-text-to-text language: - zh tags: - ocr - mlx - rare-characters - glyphwiki - palaeography --- # OpenGlyph-OCR-0 读古文字学论著里**排印的行**:一行图进去,文字出来。生僻字照 Unicode 写;没有码位的字形写成 ``,字形名就是它在 [GlyphWiki](https://glyphwiki.org) 上的名字。 *An OCR model for single lines of printed Chinese scholarly text that is dense with rare characters. It is Xiaomi-OCR-0 (0.8B) with LoRA, a retrained vision tail and 159,052 added output tokens: every CJK extension character and 78,060 GlyphWiki glyphs that have no code point, which it writes as ``. It loads with `transformers` (PyTorch) or with MLX. An English summary is at the end.* ## 它读什么,不读什么 - **读**:论文、著作里排印出来的一行横排文字(正文、脚注),包括其中的扩展区汉字和隶定字、造字。 - **不读整页**:要先把页面切成行。测量用的行图高约 100–130 像素,宽到 2,800 像素。 - **不读原始材料**:甲骨、金文、简帛的照片和拓片不在它的范围里,它读的是论著里印出来的字。 - 竖排、手写没有测过。 ## 怎么用 有两种装法:用 `transformers`(PyTorch),或者在 Apple 芯片的 Mac 上用 [MLX](https://github.com/ml-explore/mlx)。两种装法的权重相同。 ### 用 transformers(PyTorch) ```bash pip install "transformers==5.18.0" torch torchvision pillow ``` ```python import torch from transformers import AutoModelForImageTextToText, AutoProcessor repo = "OpenGlyph/OpenGlyph-OCR-0" model = AutoModelForImageTextToText.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval() processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True) out = model.read_line(processor, "line.png") print(out["text"]) # 输出的样子:其字作 ``` - **要 `trust_remote_code=True`。** 新增的记号在输出层各有一个偏置,`transformers` 自带的 Qwen3.5 类没有这一项,所以仓库里带一个几十行的模型类(`modeling_openglyph_ocr.py`);除输出层之外用的都是自带的实现。 - **按 float32 加载。** 训练改动过的权重按 32 位存放,其余是底模原来的 16 位浮点;按 float32 加载,每个权重都是训练出来的原值。按 16 位浮点加载会把训练出来的部分舍入,那样没有测过。 - **要装 torchvision。** 图像是靠它预处理的,装了才和测量时逐像素相同。不装时 `transformers` 改用 PIL 缩放图像,像素值会差到两个灰度级,那样没有测过。 - **下面的成绩是用 MLX 版量的。** 两种装法权重相同,算术不同:MLX 版每一步都把中间结果舍入到 16 位,PyTorch 版按 32 位算。发布之前,用 PyTorch 版把测试里的 43 行重读了一遍(CPU,float32):39 行读出的记号与 MLX 版逐个相同,4 行不同。这 4 行两版都没有读对,只是错处不同;按离答案差几个字算,PyTorch 版更近的 1 行,MLX 版更近的 1 行,一样远的 2 行(合计各差 20 字)。不同的 4 行里,3 行出自扩展区字的专门测试(该项共读 8 行),1 行出自论文里含难字的行(该项共读 9 行)。各表里的数字本身没有用 PyTorch 版重新量过。 - **读得快慢。** 在同一台 Mac 的 CPU 上,PyTorch 版每行约 38 秒,加载约 15 秒,内存最多占到约 8 GB。这个速度偏慢:当时机器上还有别的计算在跑,而且没有装 `causal_conv1d` 和 `flash-linear-attention`,`transformers` 用的是它自带的慢的参考实现。GPU 上没有测过。 ### 用 MLX(Apple 芯片) `base/` 里是没有改动的底模,训练出来的部分另存,由仓库里的 `openglyph_ocr.py` 拼起来。只取 MLX 用得着的文件: ```bash hf download OpenGlyph/OpenGlyph-OCR-0 --local-dir OpenGlyph-OCR-0 \ --include "base/*" --include "openglyph-ocr-0.safetensors" --include "tokens.json" \ --include "openglyph_ocr.py" --include "requirements.txt" cd OpenGlyph-OCR-0 pip install -r requirements.txt python openglyph_ocr.py line.png ``` ```python from openglyph_ocr import load, read_line, atoms ocr = load() # 本文件所在的目录 out = read_line(ocr, "line.png") print(out["text"]) # 输出的样子:其字作 print(atoms(out["text"])) # [('text', '其'), …, ('glyph', 'u2ff0-u79c0-u6208')] ``` 版本要照 `requirements.txt` 装(mlx 0.32.3、mlx-vlm 0.3.12、transformers 5.18.0、tokenizers 0.23.2)。底模是 16 位浮点,换了版本算出来的数会有细微出入,下面的成绩就不一定原样重现。`requirements.txt` 里的 torch 和 torchvision 只用来预处理图像,理由同上。 ### 读出来的字形名 `` 里的 NAME 可以直接到 `https://glyphwiki.org/wiki/NAME` 查。以 `u2ff0-` 之类开头的是按 IDS 命名的字形(`u2ff0-u79c0-u6208` 即 ⿰秀戈),`cdp-`、`cdpo-`、`cdpn-` 开头的是中研院 CDP 字,`ebag_` 开头的是 GlyphWiki 上 ebag 各系列的字形。 ## 成绩 以下都是 2026 年 11 月用这一套权重、MLX 版量的。“原版”指 Xiaomi-OCR-0 原样、用它自己的提示词去读同样的图。 **四篇论文的 398 行整行**(这四篇没有一行进过训练;每行中位数 36 字) | | 原版 Xiaomi-OCR-0 | OpenGlyph-OCR-0 | |---|---|---| | 整行一字不差 | 261 行(65.6%) | **324 行(81.4%)** | | 字错率 | 3.71% | **0.90%** | | 含难字的 208 行里整行对的 | 104 | 148 | | 不含难字的 190 行里整行对的 | 157 | 176 | “难字”指基本区里被分词器拆成两三个记号的字,和扩展区的字。 **按字的种类分**(同四篇论文里含难字的 313 行) | | 原版 | OpenGlyph-OCR-0 | |---|---|---| | 常用字 | 1995 / 2002 | 1994 / 2002 | | 基本区里拆成两个记号的字 | 294 / 350 | 315 / 350 | | 扩展区的字 | 0 / 33 | 11 / 33 | **扩展区字的专门测试**(程序排的句子,每句嵌几个扩展区字) | | 原版 | OpenGlyph-OCR-0 | |---|---|---| | 训练没用过的字体,849 个扩展区字 | 0 | 396(46.6%) | | 训练用过的字体,327 个 | 0 | 201(61.5%) | **GlyphWiki 字形的专门测试**(程序排的句子,每组 1,000 个没有码位的字形;原版没有这些记号,无从写出) | | 写对 | |---|---| | 笔画加粗、训练没见过的字样 | 607 / 1000 | | 训练时的字样 | 619 / 1000 | 第一组没写对的 393 个里,280 个被读成一个有码位的字,112 个被读成另一个字形,1 个漏掉。 **一本书的四页扫描**(93 行) | | 原版 | OpenGlyph-OCR-0 | |---|---|---| | 不含无码位字形的 66 行里整行对的 | 36 | 42 | | 同上,不计标点、数字、西文的全角半角之别 | 54 | 59 | | 37 处无码位的字形里写对的 | 0 | 9 | ## 毛病 - **扩展区的字会被读成形近的 GlyphWiki 字形。** 加进那七万多个字形之前,上面两项扩展区测试是 498 和 241,加进之后退到 396 和 201;这两组测试里一共凭空写出 190 个字形记号。在真实论文的 398 行里,凭空写出的字形记号只有 2 处。 - **没有码位的字形只认得 GlyphWiki 已经收了的。** 作者自己新造、GlyphWiki 没有的字,它会读成一个形近的字或字形。 - **字形的成绩主要来自程序排的句子。** 真实印刷品里这类字形只在那四页扫描上量过 37 处。 - 常用字基本没有变化(真实行里 1995 对 1994),提高来自难字。 - 读得快慢(MLX 版):在一台 24 GB 内存的 Apple 芯片 Mac 上,全部 1,639 行测试共用 1,051 秒,平均每行约 0.6 秒。 ## 它是怎么来的 1. 底模 [Xiaomi-OCR-0](https://huggingface.co/xiaomi-research/Xiaomi-OCR-0)(0.8B,Qwen3.5 架构)原样不动。 2. 在最后六层语言层的每个线性映射上加秩为 8 的 LoRA。 3. 视觉塔的最后四层和合并层重新训练过(32 位浮点)。 4. 新增 159,052 个输出记号,各有自己的输出行和偏置: - 229 个 ebag_s 目录字形; - 80,992 个扩展区汉字,即扩展 A 至 J 区里 Jigmo 字体画得出的每一个字(A 6,592、B 42,720、C 4,160、D 222、E 5,774、F 7,473、G 4,939、H 4,192、I 622、J 4,298); - 77,831 个没有码位的 GlyphWiki 字形(按 IDS 命名的 55,341 个,ebag 各系列 21,275 个,CDP 1,215 个)。 这些记号的输入行没有训练,全是同一个向量。 训练用的行主要是两类:用十几种字体和 GlyphWiki 字形排出来的句子,占大多数;从 26 篇古文字学论文的电子版里裁出的真实行。测试用的四篇论文和那本书都不在训练材料里。本仓库不含任何训练或测试数据。 ## 仓库里有什么 给 `transformers` 用的(`from_pretrained` 只取这些): | 文件 | 内容 | |---|---| | `model.safetensors` | 整个模型,PyTorch 的排法:底模的权重原样照抄,LoRA 并进了它所在的 46 个权重,视觉末四层与合并层、输出层按训练出来的值,这几部分按 32 位存放 | | `config.json`、`generation_config.json` | 配置 | | `modeling_openglyph_ocr.py`、`configuration_openglyph_ocr.py` | 输出层带偏置的模型类 | | `tokenizer.json`、`tokenizer_config.json`、`processor_config.json`、`chat_template.jinja` | 分词器(已含新增的 159,052 个记号;同训练时一样,不做 Unicode 规范化)和图像处理的设置 | 给 MLX 用的: | 文件 | 内容 | |---|---| | `base/` | Xiaomi-OCR-0 的六个文件,一字节未改 | | `openglyph-ocr-0.safetensors` | 训练出来的全部 148 个数组:视觉末四层与合并层 54 个,LoRA 92 个,新增记号的输出行与偏置 | | `tokens.json` | 新增的 159,052 个记号,按编号排列(从 248077 起) | | `openglyph_ocr.py` | 把以上三样拼成模型并读一行图 | | `requirements.txt` | 测量时所用的各库版本 | 另有 `release.json`,记着每个文件的 SHA-256 和来历。 MLX 版里训练出来的数组没有并进底模:底模是 16 位浮点,并进去要舍入,算出来的就不是测量时的那个模型了。PyTorch 版把 LoRA 并进去之后按 32 位存放,没有舍入。 ## 许可与出处 本仓库按 Apache-2.0 发布。`base/` 是 Xiaomi-OCR-0,其作者以 Apache-2.0 发布,它又建立在 Qwen3.5-0.8B-Base 之上;这里原样转发,未作改动。字形名和用来排字的字形数据来自 GlyphWiki。本模型不是小米或 Xiaomi-OCR-0 作者的产品,也未经他们认可。详见 `NOTICE`。 --- ## English summary **What it is.** Xiaomi-OCR-0 (0.8B, Qwen3.5 architecture, Apache-2.0), unchanged, plus: rank-8 LoRA on every linear map of the last six language layers; the last four vision blocks and the patch merger retrained; and 159,052 added output tokens, each with an output row and a bias of its own — 229 catalog forms, all 80,992 characters of CJK Extensions A to J, and 77,831 GlyphWiki glyphs that have no code point. A glyph without a code point is written ``, NAME being its GlyphWiki name. **What it reads.** One line of horizontal printed text cropped from a paper or a book (about 100 to 130 pixels high). It does not read whole pages, and it does not read photographs or rubbings of inscriptions: it reads what scholars print about them. Vertical text and handwriting were not measured. **How to run it.** With `transformers` (PyTorch): `AutoModelForImageTextToText.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32)` and `AutoProcessor.from_pretrained(repo, trust_remote_code=True)`, then `model.read_line(processor, "line.png")`; see the code above. `trust_remote_code` is needed because the output layer has a bias for the added tokens, which the stock Qwen3.5 class lacks; everything else is the stock implementation. Load it in float32: the weights that training changed are stored in float32, and loading in 16-bit floats would round them, which was not measured. Install torchvision: with it the pictures are prepared exactly as they were when the figures were measured; without it `transformers` falls back to PIL and the pixel values differ by up to two grey levels, which was not measured. With MLX on Apple silicon: `base/` holds the untouched base model and the trained arrays are kept apart (`openglyph_ocr.py` puts them together), because folding them into 16-bit base weights would round them; install the versions in `requirements.txt` (torch and torchvision are in it for the pictures, as above). The figures below were measured with the MLX form. The two forms hold the same weights but compute in different arithmetic (16-bit activations in MLX, float32 in PyTorch). Before release, 43 test lines were read again with the PyTorch form (CPU, float32): 39 came out with exactly the token ids of the MLX form and 4 did not. In those 4 neither reading is the answer; counted in characters away from the answer, the PyTorch reading is the nearer one in 1, the MLX reading in 1, and they are level in 2 (20 characters away in all for each). Of the 4 that differ, 3 are in the extension-character test (8 lines read there) and 1 is in the papers' lines with hard characters (9 lines read there). The figures themselves were not measured again with the PyTorch form. On the CPU of the same Mac a line took about 38 s (about 8 GB of memory at the most), with other work running on the machine and without `causal_conv1d` and `flash-linear-attention` (so with the slow reference kernels of `transformers`); no GPU was tried. **Measured** (November 2026, with the MLX form; "base" is Xiaomi-OCR-0 as released, with its own prompt, on the same pictures): | | base | this model | |---|---|---| | 398 whole lines from four papers that gave no training line: lines exactly right | 261 | **324** | | the same lines: character error rate | 3.71% | **0.90%** | | extension characters in test sentences, a font not used in training | 0 / 849 | 396 / 849 | | GlyphWiki glyphs in test sentences, strokes thickened beyond anything trained on | — | 607 / 1000 | | four scanned book pages: 37 glyphs without a code point | 0 | 9 | **Known defects.** Extension characters are sometimes read as a look-alike GlyphWiki glyph: the two extension-character tests stood at 498 and 241 right before the glyph tokens were added and at 396 and 201 after. Only glyphs that GlyphWiki already holds can be written; a character an author composed anew is read as something that looks like it. The glyph figures come mostly from sentences set by a program. Everyday characters are neither better nor worse than in the base model. **Training lines.** Mostly sentences set by a program in a dozen fonts and with GlyphWiki glyphs, and real lines cropped from the born-digital PDFs of 26 palaeography papers. The four test papers and the scanned book gave no training line. No training or test data is in this repository. **Licence.** Apache-2.0. `base/` is Xiaomi-OCR-0, redistributed unchanged under its Apache-2.0 licence. Not a product of Xiaomi or of the authors of Xiaomi-OCR-0, and not endorsed by them.