Eroux commited on
Commit
b1d6115
·
verified ·
1 Parent(s): b4a19b5

Yigdzin 1 (v1): grow26-ep2 checkpoint + model card (private test upload)

Browse files
README.md ADDED
@@ -0,0 +1,177 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: PaddlePaddle/PaddleOCR-VL-1.6
4
+ language:
5
+ - bo
6
+ tags:
7
+ - ocr
8
+ - tibetan
9
+ - vision-language
10
+ - paddleocr-vl
11
+ - pecha
12
+ pipeline_tag: image-text-to-text
13
+ library_name: transformers
14
+ metrics:
15
+ - cer
16
+ ---
17
+
18
+ # Tibetan OCR — Yigdzin 1
19
+
20
+ <!-- DRAFT model card for BDRC/tibetan-ocr, release tag v1 ("Yigdzin 1").
21
+ Numbers are PRELIMINARY (v1). Confirm display name sign-off before publishing. -->
22
+
23
+ **A vision-first OCR model for Tibetan pecha** — modern publications, woodblock
24
+ prints, and manuscripts (uchen and u-med), including pecha layout and orthographic
25
+ shorthands. Built by the [Buddhist Digital Resource Center (BDRC)](https://bdrc.io)
26
+ and trained on ~220k curated real + synthetic pages.
27
+
28
+ - **Repository:** `BDRC/tibetan-ocr` · release **`v1`** (display name **Yigdzin 1**)
29
+ - **Base model:** [`PaddlePaddle/PaddleOCR-VL-1.6`](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6) (Apache-2.0)
30
+ - **Live demo:** [ocr.bdrc.io](https://ocr.bdrc.io) · **Desktop app:** [buda-base/tibetan-ocr-app](https://github.com/buda-base/tibetan-ocr-app)
31
+
32
+ ## TL;DR
33
+
34
+ On a 1,070-page hand-transcribed benchmark, typical-page character error rate is
35
+ **~1.8% (median CER 0.0179)**; corpus micro-CER is **0.0756** (greedy). It is a
36
+ *specialized* OCR model, not a general chat VLM.
37
+
38
+ ## Model details
39
+
40
+ | | |
41
+ |---|---|
42
+ | Architecture | PaddleOCR-VL-1.6 (vision encoder + depth-upscaled decoder) |
43
+ | Parameters | ~810M (~413M vision · ~360M 26-layer decoder · ~4M embeddings/head) |
44
+ | Vision share | ~51–59% of parameters ("big eyes, small mouth" — vision-first) |
45
+ | Tokenizer | Tibetan **unicode-stack** tokenizer, vocab **3,560** (pruned + BoCorpus-warmed) |
46
+ | Languages | Tibetan (`bo`); Tibetan script |
47
+ | License | Apache-2.0 (derivative of PaddleOCR-VL-1.6; upstream `NOTICE` retained) |
48
+ | Funder | Khyentse Foundation ("The BDRC Etext Corpus") |
49
+
50
+ ## Intended use
51
+
52
+ - OCR of Tibetan pecha pages: **uchen** and **u-med** (dbu med) scripts across
53
+ **modern print, woodblock, and manuscript** sources.
54
+ - Batch OCR at corpus scale (BDRC is applying it to ~25M scanned pages).
55
+
56
+ **Out of scope:** general vision-language chat / VQA; non-Tibetan scripts; layout
57
+ analysis of illustrations, tables, or diagrams; line/region detection (this model
58
+ transcribes page or line crops it is given).
59
+
60
+ ## How to use
61
+
62
+ Prompt (used at training and inference), rendered through the repo's
63
+ `chat_template.jinja`:
64
+
65
+ ```
66
+ Extract all Tibetan text. Preserve line breaks.
67
+ ```
68
+
69
+ This checkpoint uses the **native** PaddleOCR-VL architecture (`model_type:
70
+ paddleocr_vl`, `PaddleOCRVLForConditionalGeneration`) — **no `trust_remote_code`
71
+ needed**, but it requires **transformers ≥ 5.15** (native `paddleocr_vl` support)
72
+ or **vLLM ≥ 0.26**. Decode **greedy** (`temperature=0`).
73
+
74
+ > **⚠ Position regime — required for correct output.** This model was trained in the
75
+ > **sequential** image-token M-RoPE regime (1-D positions), *not* the default "grid"
76
+ > regime. If the processor emits a non-zero `mm_token_type_ids`, structured pages
77
+ > loop badly (some pages regress from ~0.10 to ~0.62 CER). Serve it sequential:
78
+ > - **HF:** zero the mask before the forward —
79
+ > `inputs["mm_token_type_ids"] = torch.zeros_like(inputs["input_ids"])`.
80
+ > - **vLLM:** install the `vllm_paddleocr_seqpos` plugin and set
81
+ > `OCR_VLLM_IMAGE_TOKEN_POSITIONS=sequential`.
82
+
83
+ ### HF transformers (reference / accuracy baseline)
84
+
85
+ ```python
86
+ import torch
87
+ from PIL import Image
88
+ from transformers import AutoProcessor, AutoModelForImageTextToText
89
+
90
+ model_id = "BDRC/tibetan-ocr"
91
+ processor = AutoProcessor.from_pretrained(model_id)
92
+ model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="bfloat16", device_map="cuda")
93
+
94
+ image = Image.open("page.jpg").convert("RGB")
95
+ messages = [{"role": "user", "content": [
96
+ {"type": "image"},
97
+ {"type": "text", "text": "Extract all Tibetan text. Preserve line breaks."}]}]
98
+ prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
99
+ inputs = processor(text=[prompt], images=[image], return_tensors="pt").to("cuda")
100
+ inputs["mm_token_type_ids"] = torch.zeros_like(inputs["input_ids"]) # sequential regime (required)
101
+ out = model.generate(**inputs, do_sample=False, max_new_tokens=4096)
102
+ print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
103
+ ```
104
+
105
+ ### vLLM (production)
106
+
107
+ Serve greedy (`temperature=0`) with **vLLM ≥ 0.26**. The **only** extra requirement is
108
+ the sequential image-token position regime (above): install the `vllm_paddleocr_seqpos`
109
+ plugin and set `OCR_VLLM_IMAGE_TOKEN_POSITIONS=sequential`. Without it, structured
110
+ pages loop and accuracy collapses.
111
+
112
+ <!-- The snippets above are the intended native path; run them once on a GPU box
113
+ against deploy/fast_inference/bench.py (the source-of-truth harness) before
114
+ publishing, and paste the exact class/kwargs it uses. -->
115
+
116
+ Legitimate repetitive scripture (mantras, litanies) is real Tibetan text — do not add
117
+ aggressive anti-repetition settings that would "correct" it away.
118
+
119
+ ## Training data
120
+
121
+ ~220k-page curated mix (`bec_mixed_elie_v6`), assembled from BDRC alignment
122
+ collections + BoCorpus-rendered synthetic pages, with an easy→hard curriculum,
123
+ u-med over-weighting, and repetition-aware filtering. Released component datasets:
124
+
125
+ - Real transcriptions: [ALL-BDRC (ACIP Sungbum)](https://huggingface.co/datasets/BDRC/all-bdrc),
126
+ [PalriParkhang](https://huggingface.co/datasets/BDRC/palri-parkhang),
127
+ [Berkeley](https://huggingface.co/datasets/BDRC/berkeley),
128
+ [MonlamAI-transcriptions](https://huggingface.co/datasets/BDRC/monlamai-transcriptions),
129
+ [MonlamAI-handwritten](https://huggingface.co/datasets/BDRC/monlamai-handwritten),
130
+ [Stok](https://huggingface.co/datasets/BDRC/stok), [TibSchol](https://huggingface.co/datasets/BDRC/tibschol) *(gated)*.
131
+ - Synthetic: [Tibetan OCR synthetic v5](https://huggingface.co/datasets/BDRC/tibetan-ocr-synthetic).
132
+
133
+ <!-- Fill final dataset repo links once created; see the release plan. -->
134
+
135
+ ## Evaluation *(preliminary, v1)*
136
+
137
+ Evaluated on the BDRC hand-transcribed benchmark `20260315` (1,070 pages), greedy
138
+ decoding, `max_new_tokens=4096`, prompt policy `none`.
139
+
140
+ | Setting | micro-CER | median CER | hard-loops |
141
+ |---|---:|---:|---:|
142
+ | **Yigdzin 1** (greedy) | **0.0756** | **0.0179** | **2** |
143
+ | PaddleOCR-VL 18-layer baseline (ep1) | 0.1198 | 0.0211 | 16 |
144
+
145
+ CER = Levenshtein(hyp, ref)/len(ref) after botok normalization, whitespace
146
+ stripping, tsheg folding, and placeholder removal. Full method + the multi-system
147
+ leaderboard: **[BDRC Tibetan OCR benchmark](https://huggingface.co/datasets/BDRC/tibetan-ocr-benchmark)** and the **[leaderboard](https://huggingface.co/spaces/BDRC/tibetan-ocr-leaderboard)**.
148
+
149
+ ## Limitations & recommendations
150
+
151
+ - **Residual repetition loops** on a small fraction of pages (~0.2% greedy). Keep any
152
+ anti-loop mitigation gentle: legitimate repetitive scripture (mantras) must not be
153
+ "corrected" away.
154
+ - **Line-break joins** on some dense (6-line) woodblock pecha (~1.9% of such pages).
155
+ - Struggles on illustrations, non-Tibetan text, and illegible/damaged pages.
156
+ - Metrics are **preliminary (v1)** and may change in later generations.
157
+
158
+ ## Citation
159
+
160
+ A comprehensive article is forthcoming (Springer *Language Resources and
161
+ Evaluation*). Until then, please cite the model and BDRC:
162
+
163
+ ```bibtex
164
+ @misc{bdrc_tibetan_ocr_2026,
165
+ title = {Tibetan OCR (Yigdzin 1)},
166
+ author = {Roux, Elie and Werner, Eric},
167
+ year = {2026},
168
+ howpublished = {Buddhist Digital Resource Center, Hugging Face},
169
+ note = {https://huggingface.co/BDRC/tibetan-ocr}
170
+ }
171
+ ```
172
+
173
+ ## Acknowledgements
174
+
175
+ Built on [PaddleOCR-VL](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6)
176
+ (Apache-2.0). Funded by the **Khyentse Foundation**. Thanks to the BDRC, Dharmaduta,
177
+ and collaborating transcription teams.
chat_template.jinja ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- if not add_generation_prompt is defined -%}
2
+ {%- set add_generation_prompt = true -%}
3
+ {%- endif -%}
4
+ {%- if not cls_token is defined -%}
5
+ {%- set cls_token = "<|begin_of_sentence|>" -%}
6
+ {%- endif -%}
7
+ {%- if not eos_token is defined -%}
8
+ {%- set eos_token = "</s>" -%}
9
+ {%- endif -%}
10
+ {{- cls_token -}}
11
+ {%- for message in messages -%}
12
+ {%- if message["role"] == "user" -%}
13
+ {{- "User: " -}}
14
+ {%- for content in message["content"] -%}
15
+ {%- if content["type"] == "image" -%}
16
+ {{ "<|IMAGE_START|><|IMAGE_PLACEHOLDER|><|IMAGE_END|>" }}
17
+ {%- endif -%}
18
+ {%- endfor -%}
19
+ {%- for content in message["content"] -%}
20
+ {%- if content["type"] == "text" -%}
21
+ {{ content["text"] }}
22
+ {%- endif -%}
23
+ {%- endfor -%}
24
+ {{ "\n" -}}
25
+ {%- elif message["role"] == "assistant" -%}
26
+ {{- "Assistant:\n" -}}
27
+ {%- for content in message["content"] -%}
28
+ {%- if content["type"] == "text" -%}
29
+ {{ content["text"] }}
30
+ {%- endif -%}
31
+ {%- endfor -%}
32
+ {{ eos_token -}}
33
+ {%- elif message["role"] == "system" -%}
34
+ {%- for content in message["content"] -%}
35
+ {%- if content["type"] == "text" -%}
36
+ {{ content["text"] + "\n" }}
37
+ {%- endif -%}
38
+ {%- endfor -%}
39
+ {%- endif -%}
40
+ {%- endfor -%}
41
+ {%- if add_generation_prompt -%}
42
+ {{- "Assistant:\n" -}}
43
+ {%- endif -%}
config.json ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "PaddleOCRVLForConditionalGeneration"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.0,
6
+ "compression_ratio": 1.0,
7
+ "dtype": "bfloat16",
8
+ "hidden_dropout_prob": 0.0,
9
+ "ignored_index": -100,
10
+ "image_token_id": 2141,
11
+ "max_sequence_length": null,
12
+ "model_type": "paddleocr_vl",
13
+ "rope_is_neox_style": true,
14
+ "sliding_window": null,
15
+ "text_config": {
16
+ "bos_token_id": 1,
17
+ "dtype": "bfloat16",
18
+ "eos_token_id": 2,
19
+ "head_dim": 128,
20
+ "hidden_act": "silu",
21
+ "hidden_size": 1024,
22
+ "initializer_range": 0.02,
23
+ "intermediate_size": 3072,
24
+ "max_position_embeddings": 131072,
25
+ "model_type": "paddleocr_vl_text",
26
+ "num_attention_heads": 16,
27
+ "num_hidden_layers": 26,
28
+ "num_key_value_heads": 2,
29
+ "pad_token_id": 0,
30
+ "rms_norm_eps": 1e-05,
31
+ "rope_parameters": {
32
+ "mrope_section": [
33
+ 16,
34
+ 24,
35
+ 24
36
+ ],
37
+ "rope_theta": 500000,
38
+ "rope_type": "default",
39
+ "type": "default"
40
+ },
41
+ "tie_word_embeddings": true,
42
+ "use_bias": false,
43
+ "use_cache": false,
44
+ "vocab_size": 3560
45
+ },
46
+ "tie_word_embeddings": false,
47
+ "transformers_version": "5.10.2",
48
+ "use_3d_rope": true,
49
+ "use_cache": false,
50
+ "use_flash_attention": false,
51
+ "video_token_id": 3151,
52
+ "vision_config": {
53
+ "architectures": [
54
+ "PaddleOCRVisionModel"
55
+ ],
56
+ "attention_dropout": 0.0,
57
+ "auto_map": {
58
+ "AutoConfig": "configuration_paddleocr_vl.PaddleOCRVLConfig",
59
+ "AutoModel": "modeling_paddleocr_vl.PaddleOCRVisionModel"
60
+ },
61
+ "dtype": "bfloat16",
62
+ "hidden_act": "gelu_pytorch_tanh",
63
+ "hidden_size": 1152,
64
+ "image_size": 384,
65
+ "intermediate_size": 4304,
66
+ "layer_norm_eps": 1e-06,
67
+ "model_type": "paddleocr_vl_vision",
68
+ "num_attention_heads": 16,
69
+ "num_channels": 3,
70
+ "num_hidden_layers": 27,
71
+ "pad_token_id": 0,
72
+ "patch_size": 14,
73
+ "spatial_merge_size": 2,
74
+ "temporal_patch_size": 2,
75
+ "tokens_per_second": 2
76
+ },
77
+ "vision_end_token_id": 2140,
78
+ "vision_start_token_id": 2143,
79
+ "weight_share_add_bias": true
80
+ }
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "eos_token_id": 2,
4
+ "pad_token_id": 0,
5
+ "transformers_version": "5.10.2",
6
+ "use_cache": false
7
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c54eb065005184a30fd17c25e1393e72a457b500045fe510a79221df0af9a34d
3
+ size 1628770560
processor_config.json ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor": {
3
+ "auto_map": {
4
+ "AutoImageProcessor": "image_processing_paddleocr_vl.PaddleOCRVLImageProcessor",
5
+ "AutoProcessor": "processing_paddleocr_vl.PaddleOCRVLProcessor"
6
+ },
7
+ "do_convert_rgb": true,
8
+ "do_normalize": true,
9
+ "do_rescale": true,
10
+ "do_resize": true,
11
+ "image_mean": [
12
+ 0.5,
13
+ 0.5,
14
+ 0.5
15
+ ],
16
+ "image_processor_type": "PaddleOCRVLImageProcessor",
17
+ "image_std": [
18
+ 0.5,
19
+ 0.5,
20
+ 0.5
21
+ ],
22
+ "merge_size": 2,
23
+ "patch_size": 14,
24
+ "resample": 3,
25
+ "rescale_factor": 0.00392156862745098,
26
+ "size": {
27
+ "shortest_edge": 1024,
28
+ "longest_edge": 1003520
29
+ },
30
+ "temporal_patch_size": 1
31
+ },
32
+ "processor_class": "PaddleOCRVLProcessor"
33
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3f9f0fb64c94b09e89e9664c154c94dab86b447d1118b8dc3da64d6e00506558
3
+ size 59509
tokenizer_config.json ADDED
@@ -0,0 +1,1053 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "bos_token": "<s>",
4
+ "cls_token": "<|begin_of_sentence|>",
5
+ "eos_token": "</s>",
6
+ "extra_special_tokens": [
7
+ "0",
8
+ "1",
9
+ "2",
10
+ "3",
11
+ "4",
12
+ "5",
13
+ "6",
14
+ "7",
15
+ "8",
16
+ "9",
17
+ "<ecel>",
18
+ "<fcel>",
19
+ "<lcel>",
20
+ "<mask:4>",
21
+ "<mask:5>",
22
+ "<mask:6>",
23
+ "<mask:7>",
24
+ "<nl>",
25
+ "<ucel>",
26
+ "<xcel>",
27
+ "<|AUDIO_PLACEHOLDER|>",
28
+ "<|CROP_COL_SEP|>",
29
+ "<|CROP_ROW_SEP|>",
30
+ "<|IMAGE_END|>",
31
+ "<|IMAGE_PLACEHOLDER|>",
32
+ "<|IMAGE_SEP|>",
33
+ "<|IMAGE_START|>",
34
+ "<|LOC_0|>",
35
+ "<|LOC_1000|>",
36
+ "<|LOC_100|>",
37
+ "<|LOC_101|>",
38
+ "<|LOC_102|>",
39
+ "<|LOC_103|>",
40
+ "<|LOC_104|>",
41
+ "<|LOC_105|>",
42
+ "<|LOC_106|>",
43
+ "<|LOC_107|>",
44
+ "<|LOC_108|>",
45
+ "<|LOC_109|>",
46
+ "<|LOC_10|>",
47
+ "<|LOC_110|>",
48
+ "<|LOC_111|>",
49
+ "<|LOC_112|>",
50
+ "<|LOC_113|>",
51
+ "<|LOC_114|>",
52
+ "<|LOC_115|>",
53
+ "<|LOC_116|>",
54
+ "<|LOC_117|>",
55
+ "<|LOC_118|>",
56
+ "<|LOC_119|>",
57
+ "<|LOC_11|>",
58
+ "<|LOC_120|>",
59
+ "<|LOC_121|>",
60
+ "<|LOC_122|>",
61
+ "<|LOC_123|>",
62
+ "<|LOC_124|>",
63
+ "<|LOC_125|>",
64
+ "<|LOC_126|>",
65
+ "<|LOC_127|>",
66
+ "<|LOC_128|>",
67
+ "<|LOC_129|>",
68
+ "<|LOC_12|>",
69
+ "<|LOC_130|>",
70
+ "<|LOC_131|>",
71
+ "<|LOC_132|>",
72
+ "<|LOC_133|>",
73
+ "<|LOC_134|>",
74
+ "<|LOC_135|>",
75
+ "<|LOC_136|>",
76
+ "<|LOC_137|>",
77
+ "<|LOC_138|>",
78
+ "<|LOC_139|>",
79
+ "<|LOC_13|>",
80
+ "<|LOC_140|>",
81
+ "<|LOC_141|>",
82
+ "<|LOC_142|>",
83
+ "<|LOC_143|>",
84
+ "<|LOC_144|>",
85
+ "<|LOC_145|>",
86
+ "<|LOC_146|>",
87
+ "<|LOC_147|>",
88
+ "<|LOC_148|>",
89
+ "<|LOC_149|>",
90
+ "<|LOC_14|>",
91
+ "<|LOC_150|>",
92
+ "<|LOC_151|>",
93
+ "<|LOC_152|>",
94
+ "<|LOC_153|>",
95
+ "<|LOC_154|>",
96
+ "<|LOC_155|>",
97
+ "<|LOC_156|>",
98
+ "<|LOC_157|>",
99
+ "<|LOC_158|>",
100
+ "<|LOC_159|>",
101
+ "<|LOC_15|>",
102
+ "<|LOC_160|>",
103
+ "<|LOC_161|>",
104
+ "<|LOC_162|>",
105
+ "<|LOC_163|>",
106
+ "<|LOC_164|>",
107
+ "<|LOC_165|>",
108
+ "<|LOC_166|>",
109
+ "<|LOC_167|>",
110
+ "<|LOC_168|>",
111
+ "<|LOC_169|>",
112
+ "<|LOC_16|>",
113
+ "<|LOC_170|>",
114
+ "<|LOC_171|>",
115
+ "<|LOC_172|>",
116
+ "<|LOC_173|>",
117
+ "<|LOC_174|>",
118
+ "<|LOC_175|>",
119
+ "<|LOC_176|>",
120
+ "<|LOC_177|>",
121
+ "<|LOC_178|>",
122
+ "<|LOC_179|>",
123
+ "<|LOC_17|>",
124
+ "<|LOC_180|>",
125
+ "<|LOC_181|>",
126
+ "<|LOC_182|>",
127
+ "<|LOC_183|>",
128
+ "<|LOC_184|>",
129
+ "<|LOC_185|>",
130
+ "<|LOC_186|>",
131
+ "<|LOC_187|>",
132
+ "<|LOC_188|>",
133
+ "<|LOC_189|>",
134
+ "<|LOC_18|>",
135
+ "<|LOC_190|>",
136
+ "<|LOC_191|>",
137
+ "<|LOC_192|>",
138
+ "<|LOC_193|>",
139
+ "<|LOC_194|>",
140
+ "<|LOC_195|>",
141
+ "<|LOC_196|>",
142
+ "<|LOC_197|>",
143
+ "<|LOC_198|>",
144
+ "<|LOC_199|>",
145
+ "<|LOC_19|>",
146
+ "<|LOC_1|>",
147
+ "<|LOC_200|>",
148
+ "<|LOC_201|>",
149
+ "<|LOC_202|>",
150
+ "<|LOC_203|>",
151
+ "<|LOC_204|>",
152
+ "<|LOC_205|>",
153
+ "<|LOC_206|>",
154
+ "<|LOC_207|>",
155
+ "<|LOC_208|>",
156
+ "<|LOC_209|>",
157
+ "<|LOC_20|>",
158
+ "<|LOC_210|>",
159
+ "<|LOC_211|>",
160
+ "<|LOC_212|>",
161
+ "<|LOC_213|>",
162
+ "<|LOC_214|>",
163
+ "<|LOC_215|>",
164
+ "<|LOC_216|>",
165
+ "<|LOC_217|>",
166
+ "<|LOC_218|>",
167
+ "<|LOC_219|>",
168
+ "<|LOC_21|>",
169
+ "<|LOC_220|>",
170
+ "<|LOC_221|>",
171
+ "<|LOC_222|>",
172
+ "<|LOC_223|>",
173
+ "<|LOC_224|>",
174
+ "<|LOC_225|>",
175
+ "<|LOC_226|>",
176
+ "<|LOC_227|>",
177
+ "<|LOC_228|>",
178
+ "<|LOC_229|>",
179
+ "<|LOC_22|>",
180
+ "<|LOC_230|>",
181
+ "<|LOC_231|>",
182
+ "<|LOC_232|>",
183
+ "<|LOC_233|>",
184
+ "<|LOC_234|>",
185
+ "<|LOC_235|>",
186
+ "<|LOC_236|>",
187
+ "<|LOC_237|>",
188
+ "<|LOC_238|>",
189
+ "<|LOC_239|>",
190
+ "<|LOC_23|>",
191
+ "<|LOC_240|>",
192
+ "<|LOC_241|>",
193
+ "<|LOC_242|>",
194
+ "<|LOC_243|>",
195
+ "<|LOC_244|>",
196
+ "<|LOC_245|>",
197
+ "<|LOC_246|>",
198
+ "<|LOC_247|>",
199
+ "<|LOC_248|>",
200
+ "<|LOC_249|>",
201
+ "<|LOC_24|>",
202
+ "<|LOC_250|>",
203
+ "<|LOC_251|>",
204
+ "<|LOC_252|>",
205
+ "<|LOC_253|>",
206
+ "<|LOC_254|>",
207
+ "<|LOC_255|>",
208
+ "<|LOC_256|>",
209
+ "<|LOC_257|>",
210
+ "<|LOC_258|>",
211
+ "<|LOC_259|>",
212
+ "<|LOC_25|>",
213
+ "<|LOC_260|>",
214
+ "<|LOC_261|>",
215
+ "<|LOC_262|>",
216
+ "<|LOC_263|>",
217
+ "<|LOC_264|>",
218
+ "<|LOC_265|>",
219
+ "<|LOC_266|>",
220
+ "<|LOC_267|>",
221
+ "<|LOC_268|>",
222
+ "<|LOC_269|>",
223
+ "<|LOC_26|>",
224
+ "<|LOC_270|>",
225
+ "<|LOC_271|>",
226
+ "<|LOC_272|>",
227
+ "<|LOC_273|>",
228
+ "<|LOC_274|>",
229
+ "<|LOC_275|>",
230
+ "<|LOC_276|>",
231
+ "<|LOC_277|>",
232
+ "<|LOC_278|>",
233
+ "<|LOC_279|>",
234
+ "<|LOC_27|>",
235
+ "<|LOC_280|>",
236
+ "<|LOC_281|>",
237
+ "<|LOC_282|>",
238
+ "<|LOC_283|>",
239
+ "<|LOC_284|>",
240
+ "<|LOC_285|>",
241
+ "<|LOC_286|>",
242
+ "<|LOC_287|>",
243
+ "<|LOC_288|>",
244
+ "<|LOC_289|>",
245
+ "<|LOC_28|>",
246
+ "<|LOC_290|>",
247
+ "<|LOC_291|>",
248
+ "<|LOC_292|>",
249
+ "<|LOC_293|>",
250
+ "<|LOC_294|>",
251
+ "<|LOC_295|>",
252
+ "<|LOC_296|>",
253
+ "<|LOC_297|>",
254
+ "<|LOC_298|>",
255
+ "<|LOC_299|>",
256
+ "<|LOC_29|>",
257
+ "<|LOC_2|>",
258
+ "<|LOC_300|>",
259
+ "<|LOC_301|>",
260
+ "<|LOC_302|>",
261
+ "<|LOC_303|>",
262
+ "<|LOC_304|>",
263
+ "<|LOC_305|>",
264
+ "<|LOC_306|>",
265
+ "<|LOC_307|>",
266
+ "<|LOC_308|>",
267
+ "<|LOC_309|>",
268
+ "<|LOC_30|>",
269
+ "<|LOC_310|>",
270
+ "<|LOC_311|>",
271
+ "<|LOC_312|>",
272
+ "<|LOC_313|>",
273
+ "<|LOC_314|>",
274
+ "<|LOC_315|>",
275
+ "<|LOC_316|>",
276
+ "<|LOC_317|>",
277
+ "<|LOC_318|>",
278
+ "<|LOC_319|>",
279
+ "<|LOC_31|>",
280
+ "<|LOC_320|>",
281
+ "<|LOC_321|>",
282
+ "<|LOC_322|>",
283
+ "<|LOC_323|>",
284
+ "<|LOC_324|>",
285
+ "<|LOC_325|>",
286
+ "<|LOC_326|>",
287
+ "<|LOC_327|>",
288
+ "<|LOC_328|>",
289
+ "<|LOC_329|>",
290
+ "<|LOC_32|>",
291
+ "<|LOC_330|>",
292
+ "<|LOC_331|>",
293
+ "<|LOC_332|>",
294
+ "<|LOC_333|>",
295
+ "<|LOC_334|>",
296
+ "<|LOC_335|>",
297
+ "<|LOC_336|>",
298
+ "<|LOC_337|>",
299
+ "<|LOC_338|>",
300
+ "<|LOC_339|>",
301
+ "<|LOC_33|>",
302
+ "<|LOC_340|>",
303
+ "<|LOC_341|>",
304
+ "<|LOC_342|>",
305
+ "<|LOC_343|>",
306
+ "<|LOC_344|>",
307
+ "<|LOC_345|>",
308
+ "<|LOC_346|>",
309
+ "<|LOC_347|>",
310
+ "<|LOC_348|>",
311
+ "<|LOC_349|>",
312
+ "<|LOC_34|>",
313
+ "<|LOC_350|>",
314
+ "<|LOC_351|>",
315
+ "<|LOC_352|>",
316
+ "<|LOC_353|>",
317
+ "<|LOC_354|>",
318
+ "<|LOC_355|>",
319
+ "<|LOC_356|>",
320
+ "<|LOC_357|>",
321
+ "<|LOC_358|>",
322
+ "<|LOC_359|>",
323
+ "<|LOC_35|>",
324
+ "<|LOC_360|>",
325
+ "<|LOC_361|>",
326
+ "<|LOC_362|>",
327
+ "<|LOC_363|>",
328
+ "<|LOC_364|>",
329
+ "<|LOC_365|>",
330
+ "<|LOC_366|>",
331
+ "<|LOC_367|>",
332
+ "<|LOC_368|>",
333
+ "<|LOC_369|>",
334
+ "<|LOC_36|>",
335
+ "<|LOC_370|>",
336
+ "<|LOC_371|>",
337
+ "<|LOC_372|>",
338
+ "<|LOC_373|>",
339
+ "<|LOC_374|>",
340
+ "<|LOC_375|>",
341
+ "<|LOC_376|>",
342
+ "<|LOC_377|>",
343
+ "<|LOC_378|>",
344
+ "<|LOC_379|>",
345
+ "<|LOC_37|>",
346
+ "<|LOC_380|>",
347
+ "<|LOC_381|>",
348
+ "<|LOC_382|>",
349
+ "<|LOC_383|>",
350
+ "<|LOC_384|>",
351
+ "<|LOC_385|>",
352
+ "<|LOC_386|>",
353
+ "<|LOC_387|>",
354
+ "<|LOC_388|>",
355
+ "<|LOC_389|>",
356
+ "<|LOC_38|>",
357
+ "<|LOC_390|>",
358
+ "<|LOC_391|>",
359
+ "<|LOC_392|>",
360
+ "<|LOC_393|>",
361
+ "<|LOC_394|>",
362
+ "<|LOC_395|>",
363
+ "<|LOC_396|>",
364
+ "<|LOC_397|>",
365
+ "<|LOC_398|>",
366
+ "<|LOC_399|>",
367
+ "<|LOC_39|>",
368
+ "<|LOC_3|>",
369
+ "<|LOC_400|>",
370
+ "<|LOC_401|>",
371
+ "<|LOC_402|>",
372
+ "<|LOC_403|>",
373
+ "<|LOC_404|>",
374
+ "<|LOC_405|>",
375
+ "<|LOC_406|>",
376
+ "<|LOC_407|>",
377
+ "<|LOC_408|>",
378
+ "<|LOC_409|>",
379
+ "<|LOC_40|>",
380
+ "<|LOC_410|>",
381
+ "<|LOC_411|>",
382
+ "<|LOC_412|>",
383
+ "<|LOC_413|>",
384
+ "<|LOC_414|>",
385
+ "<|LOC_415|>",
386
+ "<|LOC_416|>",
387
+ "<|LOC_417|>",
388
+ "<|LOC_418|>",
389
+ "<|LOC_419|>",
390
+ "<|LOC_41|>",
391
+ "<|LOC_420|>",
392
+ "<|LOC_421|>",
393
+ "<|LOC_422|>",
394
+ "<|LOC_423|>",
395
+ "<|LOC_424|>",
396
+ "<|LOC_425|>",
397
+ "<|LOC_426|>",
398
+ "<|LOC_427|>",
399
+ "<|LOC_428|>",
400
+ "<|LOC_429|>",
401
+ "<|LOC_42|>",
402
+ "<|LOC_430|>",
403
+ "<|LOC_431|>",
404
+ "<|LOC_432|>",
405
+ "<|LOC_433|>",
406
+ "<|LOC_434|>",
407
+ "<|LOC_435|>",
408
+ "<|LOC_436|>",
409
+ "<|LOC_437|>",
410
+ "<|LOC_438|>",
411
+ "<|LOC_439|>",
412
+ "<|LOC_43|>",
413
+ "<|LOC_440|>",
414
+ "<|LOC_441|>",
415
+ "<|LOC_442|>",
416
+ "<|LOC_443|>",
417
+ "<|LOC_444|>",
418
+ "<|LOC_445|>",
419
+ "<|LOC_446|>",
420
+ "<|LOC_447|>",
421
+ "<|LOC_448|>",
422
+ "<|LOC_449|>",
423
+ "<|LOC_44|>",
424
+ "<|LOC_450|>",
425
+ "<|LOC_451|>",
426
+ "<|LOC_452|>",
427
+ "<|LOC_453|>",
428
+ "<|LOC_454|>",
429
+ "<|LOC_455|>",
430
+ "<|LOC_456|>",
431
+ "<|LOC_457|>",
432
+ "<|LOC_458|>",
433
+ "<|LOC_459|>",
434
+ "<|LOC_45|>",
435
+ "<|LOC_460|>",
436
+ "<|LOC_461|>",
437
+ "<|LOC_462|>",
438
+ "<|LOC_463|>",
439
+ "<|LOC_464|>",
440
+ "<|LOC_465|>",
441
+ "<|LOC_466|>",
442
+ "<|LOC_467|>",
443
+ "<|LOC_468|>",
444
+ "<|LOC_469|>",
445
+ "<|LOC_46|>",
446
+ "<|LOC_470|>",
447
+ "<|LOC_471|>",
448
+ "<|LOC_472|>",
449
+ "<|LOC_473|>",
450
+ "<|LOC_474|>",
451
+ "<|LOC_475|>",
452
+ "<|LOC_476|>",
453
+ "<|LOC_477|>",
454
+ "<|LOC_478|>",
455
+ "<|LOC_479|>",
456
+ "<|LOC_47|>",
457
+ "<|LOC_480|>",
458
+ "<|LOC_481|>",
459
+ "<|LOC_482|>",
460
+ "<|LOC_483|>",
461
+ "<|LOC_484|>",
462
+ "<|LOC_485|>",
463
+ "<|LOC_486|>",
464
+ "<|LOC_487|>",
465
+ "<|LOC_488|>",
466
+ "<|LOC_489|>",
467
+ "<|LOC_48|>",
468
+ "<|LOC_490|>",
469
+ "<|LOC_491|>",
470
+ "<|LOC_492|>",
471
+ "<|LOC_493|>",
472
+ "<|LOC_494|>",
473
+ "<|LOC_495|>",
474
+ "<|LOC_496|>",
475
+ "<|LOC_497|>",
476
+ "<|LOC_498|>",
477
+ "<|LOC_499|>",
478
+ "<|LOC_49|>",
479
+ "<|LOC_4|>",
480
+ "<|LOC_500|>",
481
+ "<|LOC_501|>",
482
+ "<|LOC_502|>",
483
+ "<|LOC_503|>",
484
+ "<|LOC_504|>",
485
+ "<|LOC_505|>",
486
+ "<|LOC_506|>",
487
+ "<|LOC_507|>",
488
+ "<|LOC_508|>",
489
+ "<|LOC_509|>",
490
+ "<|LOC_50|>",
491
+ "<|LOC_510|>",
492
+ "<|LOC_511|>",
493
+ "<|LOC_512|>",
494
+ "<|LOC_513|>",
495
+ "<|LOC_514|>",
496
+ "<|LOC_515|>",
497
+ "<|LOC_516|>",
498
+ "<|LOC_517|>",
499
+ "<|LOC_518|>",
500
+ "<|LOC_519|>",
501
+ "<|LOC_51|>",
502
+ "<|LOC_520|>",
503
+ "<|LOC_521|>",
504
+ "<|LOC_522|>",
505
+ "<|LOC_523|>",
506
+ "<|LOC_524|>",
507
+ "<|LOC_525|>",
508
+ "<|LOC_526|>",
509
+ "<|LOC_527|>",
510
+ "<|LOC_528|>",
511
+ "<|LOC_529|>",
512
+ "<|LOC_52|>",
513
+ "<|LOC_530|>",
514
+ "<|LOC_531|>",
515
+ "<|LOC_532|>",
516
+ "<|LOC_533|>",
517
+ "<|LOC_534|>",
518
+ "<|LOC_535|>",
519
+ "<|LOC_536|>",
520
+ "<|LOC_537|>",
521
+ "<|LOC_538|>",
522
+ "<|LOC_539|>",
523
+ "<|LOC_53|>",
524
+ "<|LOC_540|>",
525
+ "<|LOC_541|>",
526
+ "<|LOC_542|>",
527
+ "<|LOC_543|>",
528
+ "<|LOC_544|>",
529
+ "<|LOC_545|>",
530
+ "<|LOC_546|>",
531
+ "<|LOC_547|>",
532
+ "<|LOC_548|>",
533
+ "<|LOC_549|>",
534
+ "<|LOC_54|>",
535
+ "<|LOC_550|>",
536
+ "<|LOC_551|>",
537
+ "<|LOC_552|>",
538
+ "<|LOC_553|>",
539
+ "<|LOC_554|>",
540
+ "<|LOC_555|>",
541
+ "<|LOC_556|>",
542
+ "<|LOC_557|>",
543
+ "<|LOC_558|>",
544
+ "<|LOC_559|>",
545
+ "<|LOC_55|>",
546
+ "<|LOC_560|>",
547
+ "<|LOC_561|>",
548
+ "<|LOC_562|>",
549
+ "<|LOC_563|>",
550
+ "<|LOC_564|>",
551
+ "<|LOC_565|>",
552
+ "<|LOC_566|>",
553
+ "<|LOC_567|>",
554
+ "<|LOC_568|>",
555
+ "<|LOC_569|>",
556
+ "<|LOC_56|>",
557
+ "<|LOC_570|>",
558
+ "<|LOC_571|>",
559
+ "<|LOC_572|>",
560
+ "<|LOC_573|>",
561
+ "<|LOC_574|>",
562
+ "<|LOC_575|>",
563
+ "<|LOC_576|>",
564
+ "<|LOC_577|>",
565
+ "<|LOC_578|>",
566
+ "<|LOC_579|>",
567
+ "<|LOC_57|>",
568
+ "<|LOC_580|>",
569
+ "<|LOC_581|>",
570
+ "<|LOC_582|>",
571
+ "<|LOC_583|>",
572
+ "<|LOC_584|>",
573
+ "<|LOC_585|>",
574
+ "<|LOC_586|>",
575
+ "<|LOC_587|>",
576
+ "<|LOC_588|>",
577
+ "<|LOC_589|>",
578
+ "<|LOC_58|>",
579
+ "<|LOC_590|>",
580
+ "<|LOC_591|>",
581
+ "<|LOC_592|>",
582
+ "<|LOC_593|>",
583
+ "<|LOC_594|>",
584
+ "<|LOC_595|>",
585
+ "<|LOC_596|>",
586
+ "<|LOC_597|>",
587
+ "<|LOC_598|>",
588
+ "<|LOC_599|>",
589
+ "<|LOC_59|>",
590
+ "<|LOC_5|>",
591
+ "<|LOC_600|>",
592
+ "<|LOC_601|>",
593
+ "<|LOC_602|>",
594
+ "<|LOC_603|>",
595
+ "<|LOC_604|>",
596
+ "<|LOC_605|>",
597
+ "<|LOC_606|>",
598
+ "<|LOC_607|>",
599
+ "<|LOC_608|>",
600
+ "<|LOC_609|>",
601
+ "<|LOC_60|>",
602
+ "<|LOC_610|>",
603
+ "<|LOC_611|>",
604
+ "<|LOC_612|>",
605
+ "<|LOC_613|>",
606
+ "<|LOC_614|>",
607
+ "<|LOC_615|>",
608
+ "<|LOC_616|>",
609
+ "<|LOC_617|>",
610
+ "<|LOC_618|>",
611
+ "<|LOC_619|>",
612
+ "<|LOC_61|>",
613
+ "<|LOC_620|>",
614
+ "<|LOC_621|>",
615
+ "<|LOC_622|>",
616
+ "<|LOC_623|>",
617
+ "<|LOC_624|>",
618
+ "<|LOC_625|>",
619
+ "<|LOC_626|>",
620
+ "<|LOC_627|>",
621
+ "<|LOC_628|>",
622
+ "<|LOC_629|>",
623
+ "<|LOC_62|>",
624
+ "<|LOC_630|>",
625
+ "<|LOC_631|>",
626
+ "<|LOC_632|>",
627
+ "<|LOC_633|>",
628
+ "<|LOC_634|>",
629
+ "<|LOC_635|>",
630
+ "<|LOC_636|>",
631
+ "<|LOC_637|>",
632
+ "<|LOC_638|>",
633
+ "<|LOC_639|>",
634
+ "<|LOC_63|>",
635
+ "<|LOC_640|>",
636
+ "<|LOC_641|>",
637
+ "<|LOC_642|>",
638
+ "<|LOC_643|>",
639
+ "<|LOC_644|>",
640
+ "<|LOC_645|>",
641
+ "<|LOC_646|>",
642
+ "<|LOC_647|>",
643
+ "<|LOC_648|>",
644
+ "<|LOC_649|>",
645
+ "<|LOC_64|>",
646
+ "<|LOC_650|>",
647
+ "<|LOC_651|>",
648
+ "<|LOC_652|>",
649
+ "<|LOC_653|>",
650
+ "<|LOC_654|>",
651
+ "<|LOC_655|>",
652
+ "<|LOC_656|>",
653
+ "<|LOC_657|>",
654
+ "<|LOC_658|>",
655
+ "<|LOC_659|>",
656
+ "<|LOC_65|>",
657
+ "<|LOC_660|>",
658
+ "<|LOC_661|>",
659
+ "<|LOC_662|>",
660
+ "<|LOC_663|>",
661
+ "<|LOC_664|>",
662
+ "<|LOC_665|>",
663
+ "<|LOC_666|>",
664
+ "<|LOC_667|>",
665
+ "<|LOC_668|>",
666
+ "<|LOC_669|>",
667
+ "<|LOC_66|>",
668
+ "<|LOC_670|>",
669
+ "<|LOC_671|>",
670
+ "<|LOC_672|>",
671
+ "<|LOC_673|>",
672
+ "<|LOC_674|>",
673
+ "<|LOC_675|>",
674
+ "<|LOC_676|>",
675
+ "<|LOC_677|>",
676
+ "<|LOC_678|>",
677
+ "<|LOC_679|>",
678
+ "<|LOC_67|>",
679
+ "<|LOC_680|>",
680
+ "<|LOC_681|>",
681
+ "<|LOC_682|>",
682
+ "<|LOC_683|>",
683
+ "<|LOC_684|>",
684
+ "<|LOC_685|>",
685
+ "<|LOC_686|>",
686
+ "<|LOC_687|>",
687
+ "<|LOC_688|>",
688
+ "<|LOC_689|>",
689
+ "<|LOC_68|>",
690
+ "<|LOC_690|>",
691
+ "<|LOC_691|>",
692
+ "<|LOC_692|>",
693
+ "<|LOC_693|>",
694
+ "<|LOC_694|>",
695
+ "<|LOC_695|>",
696
+ "<|LOC_696|>",
697
+ "<|LOC_697|>",
698
+ "<|LOC_698|>",
699
+ "<|LOC_699|>",
700
+ "<|LOC_69|>",
701
+ "<|LOC_6|>",
702
+ "<|LOC_700|>",
703
+ "<|LOC_701|>",
704
+ "<|LOC_702|>",
705
+ "<|LOC_703|>",
706
+ "<|LOC_704|>",
707
+ "<|LOC_705|>",
708
+ "<|LOC_706|>",
709
+ "<|LOC_707|>",
710
+ "<|LOC_708|>",
711
+ "<|LOC_709|>",
712
+ "<|LOC_70|>",
713
+ "<|LOC_710|>",
714
+ "<|LOC_711|>",
715
+ "<|LOC_712|>",
716
+ "<|LOC_713|>",
717
+ "<|LOC_714|>",
718
+ "<|LOC_715|>",
719
+ "<|LOC_716|>",
720
+ "<|LOC_717|>",
721
+ "<|LOC_718|>",
722
+ "<|LOC_719|>",
723
+ "<|LOC_71|>",
724
+ "<|LOC_720|>",
725
+ "<|LOC_721|>",
726
+ "<|LOC_722|>",
727
+ "<|LOC_723|>",
728
+ "<|LOC_724|>",
729
+ "<|LOC_725|>",
730
+ "<|LOC_726|>",
731
+ "<|LOC_727|>",
732
+ "<|LOC_728|>",
733
+ "<|LOC_729|>",
734
+ "<|LOC_72|>",
735
+ "<|LOC_730|>",
736
+ "<|LOC_731|>",
737
+ "<|LOC_732|>",
738
+ "<|LOC_733|>",
739
+ "<|LOC_734|>",
740
+ "<|LOC_735|>",
741
+ "<|LOC_736|>",
742
+ "<|LOC_737|>",
743
+ "<|LOC_738|>",
744
+ "<|LOC_739|>",
745
+ "<|LOC_73|>",
746
+ "<|LOC_740|>",
747
+ "<|LOC_741|>",
748
+ "<|LOC_742|>",
749
+ "<|LOC_743|>",
750
+ "<|LOC_744|>",
751
+ "<|LOC_745|>",
752
+ "<|LOC_746|>",
753
+ "<|LOC_747|>",
754
+ "<|LOC_748|>",
755
+ "<|LOC_749|>",
756
+ "<|LOC_74|>",
757
+ "<|LOC_750|>",
758
+ "<|LOC_751|>",
759
+ "<|LOC_752|>",
760
+ "<|LOC_753|>",
761
+ "<|LOC_754|>",
762
+ "<|LOC_755|>",
763
+ "<|LOC_756|>",
764
+ "<|LOC_757|>",
765
+ "<|LOC_758|>",
766
+ "<|LOC_759|>",
767
+ "<|LOC_75|>",
768
+ "<|LOC_760|>",
769
+ "<|LOC_761|>",
770
+ "<|LOC_762|>",
771
+ "<|LOC_763|>",
772
+ "<|LOC_764|>",
773
+ "<|LOC_765|>",
774
+ "<|LOC_766|>",
775
+ "<|LOC_767|>",
776
+ "<|LOC_768|>",
777
+ "<|LOC_769|>",
778
+ "<|LOC_76|>",
779
+ "<|LOC_770|>",
780
+ "<|LOC_771|>",
781
+ "<|LOC_772|>",
782
+ "<|LOC_773|>",
783
+ "<|LOC_774|>",
784
+ "<|LOC_775|>",
785
+ "<|LOC_776|>",
786
+ "<|LOC_777|>",
787
+ "<|LOC_778|>",
788
+ "<|LOC_779|>",
789
+ "<|LOC_77|>",
790
+ "<|LOC_780|>",
791
+ "<|LOC_781|>",
792
+ "<|LOC_782|>",
793
+ "<|LOC_783|>",
794
+ "<|LOC_784|>",
795
+ "<|LOC_785|>",
796
+ "<|LOC_786|>",
797
+ "<|LOC_787|>",
798
+ "<|LOC_788|>",
799
+ "<|LOC_789|>",
800
+ "<|LOC_78|>",
801
+ "<|LOC_790|>",
802
+ "<|LOC_791|>",
803
+ "<|LOC_792|>",
804
+ "<|LOC_793|>",
805
+ "<|LOC_794|>",
806
+ "<|LOC_795|>",
807
+ "<|LOC_796|>",
808
+ "<|LOC_797|>",
809
+ "<|LOC_798|>",
810
+ "<|LOC_799|>",
811
+ "<|LOC_79|>",
812
+ "<|LOC_7|>",
813
+ "<|LOC_800|>",
814
+ "<|LOC_801|>",
815
+ "<|LOC_802|>",
816
+ "<|LOC_803|>",
817
+ "<|LOC_804|>",
818
+ "<|LOC_805|>",
819
+ "<|LOC_806|>",
820
+ "<|LOC_807|>",
821
+ "<|LOC_808|>",
822
+ "<|LOC_809|>",
823
+ "<|LOC_80|>",
824
+ "<|LOC_810|>",
825
+ "<|LOC_811|>",
826
+ "<|LOC_812|>",
827
+ "<|LOC_813|>",
828
+ "<|LOC_814|>",
829
+ "<|LOC_815|>",
830
+ "<|LOC_816|>",
831
+ "<|LOC_817|>",
832
+ "<|LOC_818|>",
833
+ "<|LOC_819|>",
834
+ "<|LOC_81|>",
835
+ "<|LOC_820|>",
836
+ "<|LOC_821|>",
837
+ "<|LOC_822|>",
838
+ "<|LOC_823|>",
839
+ "<|LOC_824|>",
840
+ "<|LOC_825|>",
841
+ "<|LOC_826|>",
842
+ "<|LOC_827|>",
843
+ "<|LOC_828|>",
844
+ "<|LOC_829|>",
845
+ "<|LOC_82|>",
846
+ "<|LOC_830|>",
847
+ "<|LOC_831|>",
848
+ "<|LOC_832|>",
849
+ "<|LOC_833|>",
850
+ "<|LOC_834|>",
851
+ "<|LOC_835|>",
852
+ "<|LOC_836|>",
853
+ "<|LOC_837|>",
854
+ "<|LOC_838|>",
855
+ "<|LOC_839|>",
856
+ "<|LOC_83|>",
857
+ "<|LOC_840|>",
858
+ "<|LOC_841|>",
859
+ "<|LOC_842|>",
860
+ "<|LOC_843|>",
861
+ "<|LOC_844|>",
862
+ "<|LOC_845|>",
863
+ "<|LOC_846|>",
864
+ "<|LOC_847|>",
865
+ "<|LOC_848|>",
866
+ "<|LOC_849|>",
867
+ "<|LOC_84|>",
868
+ "<|LOC_850|>",
869
+ "<|LOC_851|>",
870
+ "<|LOC_852|>",
871
+ "<|LOC_853|>",
872
+ "<|LOC_854|>",
873
+ "<|LOC_855|>",
874
+ "<|LOC_856|>",
875
+ "<|LOC_857|>",
876
+ "<|LOC_858|>",
877
+ "<|LOC_859|>",
878
+ "<|LOC_85|>",
879
+ "<|LOC_860|>",
880
+ "<|LOC_861|>",
881
+ "<|LOC_862|>",
882
+ "<|LOC_863|>",
883
+ "<|LOC_864|>",
884
+ "<|LOC_865|>",
885
+ "<|LOC_866|>",
886
+ "<|LOC_867|>",
887
+ "<|LOC_868|>",
888
+ "<|LOC_869|>",
889
+ "<|LOC_86|>",
890
+ "<|LOC_870|>",
891
+ "<|LOC_871|>",
892
+ "<|LOC_872|>",
893
+ "<|LOC_873|>",
894
+ "<|LOC_874|>",
895
+ "<|LOC_875|>",
896
+ "<|LOC_876|>",
897
+ "<|LOC_877|>",
898
+ "<|LOC_878|>",
899
+ "<|LOC_879|>",
900
+ "<|LOC_87|>",
901
+ "<|LOC_880|>",
902
+ "<|LOC_881|>",
903
+ "<|LOC_882|>",
904
+ "<|LOC_883|>",
905
+ "<|LOC_884|>",
906
+ "<|LOC_885|>",
907
+ "<|LOC_886|>",
908
+ "<|LOC_887|>",
909
+ "<|LOC_888|>",
910
+ "<|LOC_889|>",
911
+ "<|LOC_88|>",
912
+ "<|LOC_890|>",
913
+ "<|LOC_891|>",
914
+ "<|LOC_892|>",
915
+ "<|LOC_893|>",
916
+ "<|LOC_894|>",
917
+ "<|LOC_895|>",
918
+ "<|LOC_896|>",
919
+ "<|LOC_897|>",
920
+ "<|LOC_898|>",
921
+ "<|LOC_899|>",
922
+ "<|LOC_89|>",
923
+ "<|LOC_8|>",
924
+ "<|LOC_900|>",
925
+ "<|LOC_901|>",
926
+ "<|LOC_902|>",
927
+ "<|LOC_903|>",
928
+ "<|LOC_904|>",
929
+ "<|LOC_905|>",
930
+ "<|LOC_906|>",
931
+ "<|LOC_907|>",
932
+ "<|LOC_908|>",
933
+ "<|LOC_909|>",
934
+ "<|LOC_90|>",
935
+ "<|LOC_910|>",
936
+ "<|LOC_911|>",
937
+ "<|LOC_912|>",
938
+ "<|LOC_913|>",
939
+ "<|LOC_914|>",
940
+ "<|LOC_915|>",
941
+ "<|LOC_916|>",
942
+ "<|LOC_917|>",
943
+ "<|LOC_918|>",
944
+ "<|LOC_919|>",
945
+ "<|LOC_91|>",
946
+ "<|LOC_920|>",
947
+ "<|LOC_921|>",
948
+ "<|LOC_922|>",
949
+ "<|LOC_923|>",
950
+ "<|LOC_924|>",
951
+ "<|LOC_925|>",
952
+ "<|LOC_926|>",
953
+ "<|LOC_927|>",
954
+ "<|LOC_928|>",
955
+ "<|LOC_929|>",
956
+ "<|LOC_92|>",
957
+ "<|LOC_930|>",
958
+ "<|LOC_931|>",
959
+ "<|LOC_932|>",
960
+ "<|LOC_933|>",
961
+ "<|LOC_934|>",
962
+ "<|LOC_935|>",
963
+ "<|LOC_936|>",
964
+ "<|LOC_937|>",
965
+ "<|LOC_938|>",
966
+ "<|LOC_939|>",
967
+ "<|LOC_93|>",
968
+ "<|LOC_940|>",
969
+ "<|LOC_941|>",
970
+ "<|LOC_942|>",
971
+ "<|LOC_943|>",
972
+ "<|LOC_944|>",
973
+ "<|LOC_945|>",
974
+ "<|LOC_946|>",
975
+ "<|LOC_947|>",
976
+ "<|LOC_948|>",
977
+ "<|LOC_949|>",
978
+ "<|LOC_94|>",
979
+ "<|LOC_950|>",
980
+ "<|LOC_951|>",
981
+ "<|LOC_952|>",
982
+ "<|LOC_953|>",
983
+ "<|LOC_954|>",
984
+ "<|LOC_955|>",
985
+ "<|LOC_956|>",
986
+ "<|LOC_957|>",
987
+ "<|LOC_958|>",
988
+ "<|LOC_959|>",
989
+ "<|LOC_95|>",
990
+ "<|LOC_960|>",
991
+ "<|LOC_961|>",
992
+ "<|LOC_962|>",
993
+ "<|LOC_963|>",
994
+ "<|LOC_964|>",
995
+ "<|LOC_965|>",
996
+ "<|LOC_966|>",
997
+ "<|LOC_967|>",
998
+ "<|LOC_968|>",
999
+ "<|LOC_969|>",
1000
+ "<|LOC_96|>",
1001
+ "<|LOC_970|>",
1002
+ "<|LOC_971|>",
1003
+ "<|LOC_972|>",
1004
+ "<|LOC_973|>",
1005
+ "<|LOC_974|>",
1006
+ "<|LOC_975|>",
1007
+ "<|LOC_976|>",
1008
+ "<|LOC_977|>",
1009
+ "<|LOC_978|>",
1010
+ "<|LOC_979|>",
1011
+ "<|LOC_97|>",
1012
+ "<|LOC_980|>",
1013
+ "<|LOC_981|>",
1014
+ "<|LOC_982|>",
1015
+ "<|LOC_983|>",
1016
+ "<|LOC_984|>",
1017
+ "<|LOC_985|>",
1018
+ "<|LOC_986|>",
1019
+ "<|LOC_987|>",
1020
+ "<|LOC_988|>",
1021
+ "<|LOC_989|>",
1022
+ "<|LOC_98|>",
1023
+ "<|LOC_990|>",
1024
+ "<|LOC_991|>",
1025
+ "<|LOC_992|>",
1026
+ "<|LOC_993|>",
1027
+ "<|LOC_994|>",
1028
+ "<|LOC_995|>",
1029
+ "<|LOC_996|>",
1030
+ "<|LOC_997|>",
1031
+ "<|LOC_998|>",
1032
+ "<|LOC_999|>",
1033
+ "<|LOC_99|>",
1034
+ "<|LOC_9|>",
1035
+ "<|LOC_BEGIN|>",
1036
+ "<|LOC_END|>",
1037
+ "<|LOC_SEP|>",
1038
+ "<|TEXT_END|>",
1039
+ "<|TEXT_START|>",
1040
+ "<|image_pad|>",
1041
+ "<|video_pad|>"
1042
+ ],
1043
+ "is_local": true,
1044
+ "local_files_only": false,
1045
+ "mask_token": "<mask:1>",
1046
+ "model_max_length": 1000000000000000019884624838656,
1047
+ "pad_token": "<unk>",
1048
+ "processor_class": "PaddleOCRVLProcessor",
1049
+ "sep_token": "<|end_of_sentence|>",
1050
+ "tokenizer_class": "TokenizersBackend",
1051
+ "unk_token": "<unk>",
1052
+ "image_token": "<|IMAGE_PLACEHOLDER|>"
1053
+ }