Add files using upload-large-folder tool

README.md

@@ -1,3 +1,219 @@
----
-license: mit
----
+---
+license: mit
+language:
+- zh
+- en
+- fr
+- es
+- ru
+- de
+- ja
+- ko
+pipeline_tag: image-to-text
+library_name: transformers
+---
+
+# GLM-OCR
+
+<div align="center">
+<img src=https://raw.githubusercontent.com/zai-org/GLM-OCR/refs/heads/main/resources/logo.svg width="40%"/>
+</div>
+<p align="center">
+    👋 Join our <a href="https://raw.githubusercontent.com/zai-org/GLM-OCR/refs/heads/main/resources/wechat.jpg" target="_blank">WeChat</a> and <a href="https://discord.gg/QR7SARHRxK" target="_blank">Discord</a> community
+    <br>
+    📍 Use GLM-OCR's <a href="https://docs.z.ai/guides/vlm/glm-ocr" target="_blank">API</a>
+</p>
+
+
+## Introduction
+
+GLM-OCR is a multimodal OCR model for complex document understanding, built on the GLM-V encoder–decoder architecture. It introduces Multi-Token Prediction (MTP) loss and stable full-task reinforcement learning to improve training efficiency, recognition accuracy, and generalization. The model integrates the CogViT visual encoder pre-trained on large-scale image–text data, a lightweight cross-modal connector with efficient token downsampling, and a GLM-0.5B language decoder. Combined with a two-stage pipeline of layout analysis and parallel recognition based on PP-DocLayout-V3, GLM-OCR delivers robust and high-quality OCR performance across diverse document layouts.
+
+**Key Features**
+
+- **State-of-the-Art Performance**: Achieves a score of 94.62 on OmniDocBench V1.5, ranking #1 overall, and delivers state-of-the-art results across major document understanding benchmarks, including formula recognition, table recognition, and information extraction.
+
+- **Optimized for Real-World Scenarios**: Designed and optimized for practical business use cases, maintaining robust performance on complex tables, code-heavy documents, seals, and other challenging real-world layouts.
+
+- **Efficient Inference**: With only 0.9B parameters, GLM-OCR supports deployment via vLLM, SGLang, and Ollama, significantly reducing inference latency and compute cost, making it ideal for high-concurrency services and edge deployments.
+
+- **Easy to Use**: Fully open-sourced and equipped with a comprehensive [SDK](https://github.com/zai-org/GLM-OCR) and inference toolchain, offering simple installation, one-line invocation, and smooth integration into existing production pipelines.
+
+## Performance
+
+- Document Parsing & Information Extraction
+
+![image](https://raw.githubusercontent.com/zai-org/GLM-OCR/refs/heads/main/resources/docparse.png)
+
+
+- Real-World Scenarios Performance
+
+![image](https://raw.githubusercontent.com/zai-org/GLM-OCR/refs/heads/main/resources/realworld.png)
+
+
+- Speed Test
+
+For speed, we compared different OCR methods under identical hardware and testing conditions (single replica, single concurrency), evaluating their performance in parsing and exporting Markdown files from both image and PDF inputs. Results show GLM-OCR achieves a throughput of 1.86 pages/second for PDF documents and 0.67 images/second for images, significantly outperforming comparable models.
+
+![image](https://raw.githubusercontent.com/zai-org/GLM-OCR/refs/heads/main/resources/speed.png)
+
+## Usage
+
+### vLLM
+
+1. run 
+
+```bash
+pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
+```
+
+or using docker with:
+```
+docker pull vllm/vllm-openai:nightly
+```
+
+2. run with:
+
+```bash
+pip install git+https://github.com/huggingface/transformers.git
+vllm serve zai-org/GLM-OCR  --allowed-local-media-path /  --port 8080
+```
+
+### SGLang
+
+
+1. using docker with:
+
+```bash
+docker pull lmsysorg/sglang:dev
+```
+
+or build it from source with:
+
+```bash
+pip install git+https://github.com/sgl-project/sglang.git#subdirectory=python
+```
+
+2. run with:
+
+```bash
+pip install git+https://github.com/huggingface/transformers.git
+python -m sglang.launch_server --model zai-org/GLM-OCR --port 8080
+```
+
+### Ollama 
+
+1. Download [Ollama](https://ollama.com/download).
+2. run with: 
+
+```bash
+ollama run glm-ocr
+```
+
+Ollama will automatically use image file path when an image is dragged into the terminal: 
+
+```bash
+ollama run glm-ocr Text Recognition: ./image.png
+```
+
+### Transformers
+
+```
+pip install git+https://github.com/huggingface/transformers.git
+```
+
+```python
+from transformers import AutoProcessor, AutoModelForImageTextToText
+import torch
+
+MODEL_PATH = "zai-org/GLM-OCR"
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "url": "test_image.png"
+            },
+            {
+                "type": "text",
+                "text": "Text Recognition:"
+            }
+        ],
+    }
+]
+processor = AutoProcessor.from_pretrained(MODEL_PATH)
+model = AutoModelForImageTextToText.from_pretrained(
+    pretrained_model_name_or_path=MODEL_PATH,
+    torch_dtype="auto",
+    device_map="auto",
+)
+inputs = processor.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_dict=True,
+    return_tensors="pt"
+).to(model.device)
+inputs.pop("token_type_ids", None)
+generated_ids = model.generate(**inputs, max_new_tokens=8192)
+output_text = processor.decode(generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)
+print(output_text)
+```
+
+### Prompt Limited
+
+GLM-OCR currently supports two types of prompt scenarios:
+
+1. **Document Parsing** – extract raw content from documents. Supported tasks include:
+
+```python
+{
+    "text": "Text Recognition:",
+    "formula": "Formula Recognition:",
+    "table": "Table Recognition:"
+}
+```
+
+2. **Information Extraction** – extract structured information from documents. Prompts must follow a strict JSON schema. For example, to extract personal ID information:
+
+```python
+请按下列JSON格式输出图中信息:
+{
+    "id_number": "",
+    "last_name": "",
+    "first_name": "",
+    "date_of_birth": "",
+    "address": {
+        "street": "",
+        "city": "",
+        "state": "",
+        "zip_code": ""
+    },
+    "dates": {
+        "issue_date": "",
+        "expiration_date": ""
+    },
+    "sex": ""
+}
+```
+
+⚠️ Note: When using information extraction, the output must strictly adhere to the defined JSON schema to ensure downstream processing compatibility.
+
+## GLM-OCR SDK
+
+We provide an easy-to-use SDK for using GLM-OCR more efficiently and conveniently. please check our [github](https://github.com/zai-org/GLM-OCR) to get more detail.
+
+## Acknowledgement
+
+This project is inspired by the excellent work of the following projects and communities:
+
+- [PP-DocLayout-V3](https://huggingface.co/PaddlePaddle/PP-DocLayoutV3)
+- [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)
+- [MinerU](https://github.com/opendatalab/MinerU)
+
+## License
+
+The GLM-OCR model is released under the MIT License.
+
+The complete OCR pipeline integrates [PP-DocLayoutV3](https://huggingface.co/PaddlePaddle/PP-DocLayoutV3) for document layout analysis, which is licensed under the Apache License 2.0. Users should comply with both licenses when using this project.

chat_template.jinja

@@ -0,0 +1,140 @@
+[gMASK]<sop>
+{%- if tools -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{{ tool | tojson(ensure_ascii=False) }}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}
+<arg_key>{arg-key-1}</arg_key>
+<arg_value>{arg-value-1}</arg_value>
+<arg_key>{arg-key-2}</arg_key>
+<arg_value>{arg-value-2}</arg_value>
+...
+</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is mapping and (item.type == 'image' or 'image' in item) -%}
+                <|begin_of_image|><|image|><|end_of_image|>
+            {%- elif item is mapping and (item.type == 'video' or 'video' in item) -%}
+                <|begin_of_video|><|video|><|end_of_video|>
+            {%- elif item is string -%}
+                {{- item }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) %}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {% set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{% for m in messages %}
+{%- if m.role == 'user' -%}<|user|>
+{% if m.content is string %}
+{{ m.content }}
+{%- else %}
+{%- for item in m.content %}
+{% if item.type == 'video' or 'video' in item %}
+<|begin_of_video|><|video|><|end_of_video|>{% elif item.type == 'image' or 'image' in item %}
+<|begin_of_image|><|image|><|end_of_image|>{% elif item.type == 'text' %}
+{{ item.text }}
+{%- endif %}
+{%- endfor %}
+{%- endif %}
+{{- '/nothink' if (enable_thinking is defined and not enable_thinking and not visible_text(m.content).endswith("/nothink")) else '' -}}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set reasoning_content = '' %}
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- else %}
+    {%- if '</think>' in content %}
+        {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+        {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+    {%- endif %}
+{%- endif %}
+{%- if loop.index0 > ns.last_user_index and reasoning_content -%}
+{{ '\n<think>' + reasoning_content.strip() +  '</think>'}}
+{%- else -%}
+{{ '\n<think></think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ '\n' + content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{ '\n<tool_call>' + tc.name }}
+{% set _args = tc.arguments %}
+{% for k, v in _args.items() %}
+<arg_key>{{ k }}</arg_key>
+<arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>
+{% endfor %}
+</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if m.content is string -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' }}
+{%- endif %}
+{{- '\n<tool_response>\n' }}
+{{- m.content }}
+{{- '\n</tool_response>' }}
+{% elif m.content is iterable and m.content is not mapping %}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+{{- '<|observation|>' }}
+{%- endif %}
+{{- '\n<tool_response>\n' }}
+{%- for tr in m.content -%}
+  {%- if tr is mapping and tr.type is defined -%}
+    {%- set t = tr.type | lower -%}
+    {%- if t == 'text' and tr.text is defined -%}
+{{ tr.text }}
+    {%- elif t in ['image', 'image_url'] -%}
+<|begin_of_image|><|image|><|end_of_image|>
+    {%- elif t in ['video', 'video_url'] -%}
+<|begin_of_video|><|video|><|end_of_video|>
+    {%- else -%}
+{{ tr | tojson(ensure_ascii=False) }}
+    {%- endif -%}
+  {%- else -%}
+{{ tr.output if tr.output is defined else tr }}
+  {%- endif -%}
+{%- endfor -%}
+{{- '\n</tool_response>' }}
+{%- else -%}
+<|observation|>{% for tr in m.content %}
+
+<tool_response>
+{{ tr.output if tr.output is defined else tr }}
+</tool_response>{% endfor -%}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>
+{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+<|assistant|>
+{{'<think></think>\n' if (enable_thinking is defined and not enable_thinking) else ''}}
+{%- endif -%}

config.json

@@ -0,0 +1,65 @@
+{
+  "architectures": [
+    "GlmOcrForConditionalGeneration"
+  ],
+  "model_type": "glm_ocr",
+  "text_config": {
+    "model_type": "glm_ocr_text",
+    "pad_token_id": 59246,
+    "vocab_size": 59392,
+    "eos_token_id": [
+      59246,
+      59253
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 1536,
+    "initializer_range": 0.02,
+    "intermediate_size": 4608,
+    "max_position_embeddings": 131072,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 16,
+    "num_nextn_predict_layers": 1,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-05,
+    "dtype": "bfloat16",
+    "rope_parameters": {
+      "rope_type": "default",
+      "mrope_section": [
+        16,
+        24,
+        24
+      ],
+       "partial_rotary_factor": 1.0,
+       "rope_theta": 10000
+    },
+    "tie_word_embeddings": false,
+    "use_cache": true
+  },
+  "vision_config": {
+    "model_type": "glm_ocr_vision",
+    "hidden_size": 1024,
+    "depth": 24,
+    "num_heads": 16,
+    "attention_bias": true,
+    "intermediate_size": 4096,
+    "hidden_act": "silu",
+    "hidden_dropout_prob": 0.0,
+    "initializer_range": 0.02,
+    "image_size": 336,
+    "patch_size": 14,
+    "out_hidden_size": 1536,
+    "rms_norm_eps": 1e-05,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "image_start_token_id": 59256,
+  "image_end_token_id": 59257,
+  "video_start_token_id": 59258,
+  "video_end_token_id": 59259,
+  "image_token_id": 59280,
+  "video_token_id": 59281,
+  "transformers_version": "5.0.1dev0"
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "do_sample": false,
+  "eos_token_id": [
+    59246,
+    59253
+  ],
+  "pad_token_id": 59246,
+  "transformers_version": "5.0.1dev0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a16eb0de98d199293371c560f95f83130d2a2c9612449df16839f08ff9498815
+size 2650579464

preprocessor_config.json

@@ -0,0 +1,11 @@
+{
+    "size": {"shortest_edge": 12544, "longest_edge": 9633792},
+    "do_rescale": true,
+    "patch_size": 14,
+    "temporal_patch_size": 2,
+    "merge_size": 2,
+    "image_mean": [0.48145466, 0.4578275, 0.40821073],
+    "image_std": [0.26862954, 0.26130258, 0.27577711],
+    "image_processor_type": "Glm46VImageProcessor",
+    "processor_class": "Glm46VProcessor"
+}

tokenizer_config.json

@@ -0,0 +1,49 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": [
+    "<|endoftext|>",
+    "[MASK]",
+    "[gMASK]",
+    "[sMASK]",
+    "<sop>",
+    "<eop>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|observation|>",
+    "<|begin_of_image|>",
+    "<|end_of_image|>",
+    "<|begin_of_video|>",
+    "<|end_of_video|>",
+    "<|begin_of_audio|>",
+    "<|end_of_audio|>",
+    "<|begin_of_transcription|>",
+    "<|end_of_transcription|>",
+    "<|code_prefix|>",
+    "<|code_middle|>",
+    "<|code_suffix|>",
+    "<think>",
+    "</think>",
+    "<tool_call>",
+    "</tool_call>",
+    "<tool_response>",
+    "</tool_response>",
+    "<arg_key>",
+    "</arg_key>",
+    "<arg_value>",
+    "</arg_value>",
+    "/nothink",
+    "<|begin_of_box|>",
+    "<|end_of_box|>",
+    "<|image|>",
+    "<|video|>"
+  ],
+  "is_local": true,
+  "model_max_length": 655380,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "processor_class": "Glm46VProcessor",
+  "tokenizer_class": "TokenizersBackend"
+}