astromandarin's picture
OpenGlyph-OCR-IDS: first release
20fc05e verified
Raw History Blame Contribute Delete
2.62 kB
OpenGlyph-OCR-IDS
This repository contains, in base/, the model Xiaomi-OCR-0
(https://huggingface.co/xiaomi-research/Xiaomi-OCR-0), released by its authors under the
Apache License, Version 2.0, and itself built on Qwen3.5-0.8B-Base. The files in base/
are redistributed unchanged; their SHA-256 hashes are listed in release.json.
openglyph-ocr-0.safetensors and tokens.json are the files of the same names in
OpenGlyph/OpenGlyph-OCR-0 (https://huggingface.co/OpenGlyph/OpenGlyph-OCR-0), the work of
the OpenGlyph project: the arrays trained on top of Xiaomi-OCR-0 (LoRA arrays on every
linear map of the last six language layers, the last four vision blocks and the patch
merger retrained, an output row and a bias for each of 159,052 added tokens) and the
added tokens.
The other files -- opener.safetensors, part-reader.safetensors, speller.safetensors,
ids-tables.json, reading.json, token-counts.json, the package openglyph_ocr_ids/,
figures/, requirements.txt, release.json and README.md -- are the work of the OpenGlyph
project and are licensed under the Apache License, Version 2.0 (see LICENSE), with the
exception named below. They add to the model:
- 14 tokens (<ids>, </ids> and the twelve structure characters U+2FF0..U+2FFB) with
fixed input rows taken from rows of the base embedding, and output rows and biases;
- three further sets of arrays that are put into the model in turn while a line is
read: LoRA arrays of the language layers 18 to 23 (the opener; the part reader), and
LoRA arrays of the language layers 18 to 23, of every vision block and of the patch
merger, output rows of the structure tokens and output rows of the steps of parts
(the speller).
Nothing is merged into the base weights: openglyph_ocr_ids/build.py puts everything
together when it builds the model.
canon.json is a table from Ideographic Description Sequences to encoded characters and to
GlyphWiki glyphs. It is derived from
- CJKVI IDS (https://github.com/cjkvi/cjkvi-ids), which is derived from the CHISE IDS
database and follows its terms (GNU General Public License, version 2);
- BabelStone's IDS data (https://www.babelstone.co.uk/CJK/IDS.TXT), which its author
states anyone is free to use without permission or attribution;
- GlyphWiki (https://glyphwiki.org): its glyph names and the IDS names of its glyphs.
canon.json is used and redistributed under the terms of those sources.
Glyph names of the form <gw:NAME> are names of glyphs in GlyphWiki (https://glyphwiki.org).
This is not a product of Xiaomi or of the authors of Xiaomi-OCR-0, and is not endorsed
by them.