Instructions to use zenosai/MonkeyOCRv2-B-Parsing with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zenosai/MonkeyOCRv2-B-Parsing with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="zenosai/MonkeyOCRv2-B-Parsing", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("zenosai/MonkeyOCRv2-B-Parsing", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zenosai/MonkeyOCRv2-B-Parsing with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zenosai/MonkeyOCRv2-B-Parsing"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zenosai/MonkeyOCRv2-B-Parsing",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/zenosai/MonkeyOCRv2-B-Parsing

SGLang

How to use zenosai/MonkeyOCRv2-B-Parsing with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zenosai/MonkeyOCRv2-B-Parsing" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zenosai/MonkeyOCRv2-B-Parsing",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zenosai/MonkeyOCRv2-B-Parsing" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zenosai/MonkeyOCRv2-B-Parsing",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use zenosai/MonkeyOCRv2-B-Parsing with Docker Model Runner:
```
docker model run hf.co/zenosai/MonkeyOCRv2-B-Parsing
```

zenosai commited on 5 days ago

Commit

0af694f

verified ·

1 Parent(s): 9a047b8

Upload folder using huggingface_hub

Browse files

Files changed (19) hide show

.gitattributes +1 -0
added_tokens.json +28 -0
args.json +688 -0
chat_template.jinja +120 -0
config.json +88 -0
configuration_monkeyocrv2.py +78 -0
generation_config.json +13 -0
merges.txt +0 -0
model.safetensors +3 -0
modeling_monkeyocrv2.py +142 -0
modeling_monkeyocrv2_vision.py +531 -0
preprocessor1.pth +3 -0
preprocessor2.pth +3 -0
preprocessor_config.json +22 -0
processor_config.json +6 -0
special_tokens_map.json +31 -0
tokenizer.json +3 -0
tokenizer_config.json +243 -0
vocab.json +0 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

added_tokens.json ADDED Viewed

	@@ -0,0 +1,28 @@

+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

args.json ADDED Viewed

	@@ -0,0 +1,688 @@

+{
+  "output_dir": "/home/kas/kas_checkpoint/zhangziyang/monkeyocrv2/sft_s1_base_v2",
+  "overwrite_output_dir": false,
+  "do_train": false,
+  "do_eval": false,
+  "do_predict": false,
+  "eval_strategy": "no",
+  "prediction_loss_only": false,
+  "per_device_train_batch_size": 4,
+  "per_device_eval_batch_size": 2,
+  "per_gpu_train_batch_size": null,
+  "per_gpu_eval_batch_size": null,
+  "gradient_accumulation_steps": 1,
+  "eval_accumulation_steps": null,
+  "eval_delay": 0,
+  "torch_empty_cache_steps": null,
+  "learning_rate": 2e-05,
+  "weight_decay": 0.1,
+  "adam_beta1": 0.9,
+  "adam_beta2": 0.95,
+  "adam_epsilon": 1e-08,
+  "max_grad_norm": 1.0,
+  "num_train_epochs": 1.0,
+  "max_steps": -1,
+  "lr_scheduler_type": "cosine",
+  "lr_scheduler_kwargs": null,
+  "warmup_ratio": 0.05,
+  "warmup_steps": 0,
+  "log_level": "passive",
+  "log_level_replica": "warning",
+  "log_on_each_node": true,
+  "logging_dir": "/home/kas/kas_checkpoint/zhangziyang/monkeyocrv2/sft_s1_base_v2/runs",
+  "logging_strategy": "steps",
+  "logging_first_step": true,
+  "logging_steps": 5,
+  "logging_nan_inf_filter": true,
+  "save_strategy": "steps",
+  "save_steps": 5000.0,
+  "save_total_limit": 20,
+  "save_safetensors": true,
+  "save_on_each_node": false,
+  "save_only_model": false,
+  "restore_callback_states_from_checkpoint": false,
+  "no_cuda": false,
+  "use_cpu": false,
+  "use_mps_device": false,
+  "seed": 42,
+  "data_seed": 42,
+  "jit_mode_eval": false,
+  "bf16": true,
+  "fp16": false,
+  "fp16_opt_level": "O1",
+  "half_precision_backend": "auto",
+  "bf16_full_eval": false,
+  "fp16_full_eval": false,
+  "tf32": null,
+  "local_rank": 0,
+  "ddp_backend": null,
+  "tpu_num_cores": null,
+  "tpu_metrics_debug": false,
+  "debug": null,
+  "dataloader_drop_last": false,
+  "eval_steps": 400000.0,
+  "dataloader_num_workers": 16,
+  "dataloader_prefetch_factor": null,
+  "past_index": -1,
+  "run_name": "/home/kas/kas_checkpoint/zhangziyang/monkeyocrv2/sft_s1_base_v2",
+  "disable_tqdm": null,
+  "remove_unused_columns": true,
+  "label_names": null,
+  "load_best_model_at_end": false,
+  "metric_for_best_model": "loss",
+  "greater_is_better": false,
+  "ignore_data_skip": false,
+  "fsdp": null,
+  "fsdp_min_num_params": 0,
+  "fsdp_config": null,
+  "fsdp_transformer_layer_cls_to_wrap": null,
+  "accelerator_config": {
+    "dispatch_batches": false
+  },
+  "parallelism_config": null,
+  "deepspeed": {
+    "fp16": {
+      "enabled": "auto",
+      "loss_scale": 0,
+      "loss_scale_window": 1000,
+      "initial_scale_power": 16,
+      "hysteresis": 2,
+      "min_loss_scale": 1
+    },
+    "bf16": {
+      "enabled": "auto"
+    },
+    "zero_optimization": {
+      "stage": 1,
+      "offload_optimizer": {
+        "device": "none",
+        "pin_memory": true
+      },
+      "allgather_partitions": true,
+      "allgather_bucket_size": 200000000.0,
+      "overlap_comm": false,
+      "reduce_scatter": true,
+      "reduce_bucket_size": 200000000.0,
+      "contiguous_gradients": true
+    },
+    "gradient_accumulation_steps": "auto",
+    "gradient_clipping": "auto",
+    "steps_per_print": 2000,
+    "train_batch_size": "auto",
+    "train_micro_batch_size_per_gpu": "auto",
+    "wall_clock_breakdown": false
+  },
+  "label_smoothing_factor": 0.0,
+  "optim": "adamw_torch_fused",
+  "optim_args": null,
+  "adafactor": false,
+  "group_by_length": false,
+  "length_column_name": "length",
+  "report_to": [
+    "tensorboard",
+    "swanlab"
+  ],
+  "project": "huggingface",
+  "trackio_space_id": "trackio",
+  "ddp_find_unused_parameters": null,
+  "ddp_bucket_cap_mb": null,
+  "ddp_broadcast_buffers": null,
+  "dataloader_pin_memory": true,
+  "dataloader_persistent_workers": false,
+  "skip_memory_metrics": true,
+  "use_legacy_prediction_loop": false,
+  "push_to_hub": false,
+  "resume_from_checkpoint": "/home/kas/kas_checkpoint/zhangziyang/monkeyocrv2/sft_s1_base_v2/checkpoint-355000",
+  "hub_model_id": null,
+  "hub_strategy": "every_save",
+  "hub_token": null,
+  "hub_private_repo": null,
+  "hub_always_push": false,
+  "hub_revision": null,
+  "gradient_checkpointing": true,
+  "gradient_checkpointing_kwargs": null,
+  "include_inputs_for_metrics": false,
+  "include_for_metrics": [],
+  "eval_do_concat_batches": true,
+  "fp16_backend": "auto",
+  "push_to_hub_model_id": null,
+  "push_to_hub_organization": null,
+  "push_to_hub_token": null,
+  "mp_parameters": "",
+  "auto_find_batch_size": false,
+  "full_determinism": false,
+  "torchdynamo": null,
+  "ray_scope": "last",
+  "ddp_timeout": 18000000,
+  "torch_compile": false,
+  "torch_compile_backend": null,
+  "torch_compile_mode": null,
+  "include_tokens_per_second": false,
+  "include_num_input_tokens_seen": false,
+  "neftune_noise_alpha": null,
+  "optim_target_modules": null,
+  "batch_eval_metrics": false,
+  "eval_on_start": false,
+  "use_liger_kernel": false,
+  "liger_kernel_config": null,
+  "eval_use_gather_object": false,
+  "average_tokens_across_devices": true,
+  "sortish_sampler": false,
+  "predict_with_generate": false,
+  "generation_max_length": null,
+  "generation_num_beams": null,
+  "generation_config": null,
+  "tuner_backend": "peft",
+  "vit_gradient_checkpointing": null,
+  "router_aux_loss_coef": 0.0,
+  "enable_dft_loss": false,
+  "enable_channel_loss": false,
+  "check_model": true,
+  "acc_strategy": "token",
+  "train_dataloader_shuffle": true,
+  "max_epochs": null,
+  "aligner_lr": null,
+  "vit_lr": null,
+  "use_logits_to_keep": null,
+  "ds3_gather_for_generation": true,
+  "resume_only_model": false,
+  "optimizer": null,
+  "loss_type": null,
+  "metric": null,
+  "eval_use_evalscope": false,
+  "eval_dataset": [],
+  "eval_dataset_args": null,
+  "eval_limit": null,
+  "eval_generation_config": null,
+  "extra_eval_args": null,
+  "use_flash_ckpt": false,
+  "use_ray": false,
+  "ray_exp_name": null,
+  "device_groups": null,
+  "model": "/home/kas/kas_checkpoint/zhangziyang/monkeyocrv2/sft_s0_base_v2_12m/checkpoint-46641",
+  "model_type": "monkeyocrv2",
+  "model_revision": null,
+  "task_type": "causal_lm",
+  "torch_dtype": "bfloat16",
+  "attn_impl": "flash_attention_2",
+  "new_special_tokens": [],
+  "num_labels": null,
+  "problem_type": null,
+  "rope_scaling": null,
+  "device_map": null,
+  "max_memory": {},
+  "max_model_len": null,
+  "local_repo_path": null,
+  "init_strategy": null,
+  "template": "monkeyocrv2",
+  "system": null,
+  "max_length": 16384,
+  "truncation_strategy": "right",
+  "max_pixels": 1003520,
+  "agent_template": null,
+  "norm_bbox": null,
+  "use_chat_template": true,
+  "padding_free": true,
+  "padding_side": "right",
+  "loss_scale": "default",
+  "sequence_parallel_size": 1,
+  "response_prefix": null,
+  "template_backend": "swift",
+  "dataset": [
+    "/home/kas/kas_outside/zhangziyang/swift/data/cont_1771k_0701.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/text/master_labels_s10m_splits/1.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/text/master_labels_s10m_splits/10.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/text/master_labels_s10m_splits/2.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/text/master_labels_s10m_splits/3.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/text/master_labels_s10m_splits/4.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/text/master_labels_s10m_splits/5.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/text/master_labels_s10m_splits/6.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/text/master_labels_s10m_splits/7.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/text/master_labels_s10m_splits/8.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/text/master_labels_s10m_splits/9.json",
+    "/home/kas/kas_outside/songjiajun/tool/get_final_json_rec/final_json/merged_rec_use_2_s1m.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/aug_s6m_splits/1.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/aug_s6m_splits/2.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/aug_s6m_splits/3.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/aug_s6m_splits/4.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/aug_s6m_splits/5.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/aug_s6m_splits/6.json",
+    "/home/kas/kas_outside/zhangziyang/swift/data/for_311k_0701.json",
+    "/home/kas/kas_outside/zhangziyang/swift/data/unimer_0616.json",
+    "/home/kas/kas_outside/zhangziyang/latextract/images/latextract.json",
+    "/home/kas/kas_outside/zhangziyang/latextract/images/latextract_aug.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_html_labels_s7m_splits/1.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_html_labels_s7m_splits/2.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_html_labels_s7m_splits/3.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_html_labels_s7m_splits/4.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_html_labels_s7m_splits/5.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_html_labels_s7m_splits/6.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_html_labels_s7m_splits/7.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/20260111_1203/pubtabnet_train_filter.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/20260111_1203/html_tab_621k_0701.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/html/aug.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/html/ordinary.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_otsl_labels_cleaned_s7m_splits/1.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_otsl_labels_cleaned_s7m_splits/2.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_otsl_labels_cleaned_s7m_splits/3.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_otsl_labels_cleaned_s7m_splits/4.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_otsl_labels_cleaned_s7m_splits/5.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_otsl_labels_cleaned_s7m_splits/6.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/table/master_otsl_labels_cleaned_s7m_splits/7.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/20260111_1203/pubtabnet_train_filter_otsl.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/20260111_1203/html_tab_621k_0701_otsl.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/formula/aug.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/formula/ordinary.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/20260111_1203/aug_otsl.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/20260111_1203/ordinary_otsl.json",
+    "/home/kas/kas_outside/zhiyinma/html_render/output/inner_table_all_s0d5.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/inner_img_table/master_html_inner_img_labels_cleaned_s0d5.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/liushui_table/master_html_labels.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/liushui_table/master_otsl_labels.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/liushui_table/master_html_labels_original.json",
+    "/home/kas/kas_outside/zdguo/text_table_data_v2.1/liushui_table/master_otsl_labels_original.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout/aug__splits/1.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout/aug__splits/2.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout/aug__splits/3.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout/aug__splits/4.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout/ordinary__splits/1.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout/ordinary__splits/2.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout/ordinary__splits/3.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout/ordinary__splits/4.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout_rec/aug__s0d5_splits/1.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout_rec/aug__s0d5_splits/2.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout_rec/aug__s0d5_splits/3.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout_rec/aug__s0d5_splits/4.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout_rec/ordinary__s0d5_splits/1.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout_rec/ordinary__s0d5_splits/2.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout_rec/ordinary__s0d5_splits/3.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout_rec/ordinary__s0d5_splits/4.json",
+    "/home/kas/kas_outside/zdguo/20260211_rotate_data_fast/table/master_html_labels_splits/1.json",
+    "/home/kas/kas_outside/zdguo/20260211_rotate_data_fast/table/master_html_labels_splits/2.json",
+    "/home/kas/kas_outside/zdguo/20260211_rotate_data_fast/table/master_otsl_labels_splits/1.json",
+    "/home/kas/kas_outside/zdguo/20260211_rotate_data_fast/table/master_otsl_labels_splits/2.json",
+    "/home/kas/kas_outside/zdguo/20260211_rotate_data_fast/text/text_rotated_labels_splits/1.json",
+    "/home/kas/kas_outside/zdguo/20260211_rotate_data_fast/text/text_rotated_labels_splits/2.json",
+    "/home/kas/kas_outside/zdguo/20260211_rotate_data_fast/liushui_table/master_html_labels.json",
+    "/home/kas/kas_outside/zdguo/20260211_rotate_data_fast/liushui_table/master_otsl_labels.json",
+    "/home/kas/mzy/xhs/dots.ocr-master/data/vertical/layout_0212.json",
+    "/home/kas/mzy/xhs/dots.ocr-master/data/M5HisDoc/layout_0212.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout/rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout_rec/rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/block/html_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/block/html_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/block/otsl_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/block/otsl_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/block/text_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/block/text_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/page/layout_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/page/layout_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/block/html_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/block/html_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/block/otsl_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/block/otsl_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/block/text_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/block/text_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/page/layout_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/page/layout_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/block/html_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/block/html_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/block/otsl_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/block/otsl_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/block/text_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/block/text_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/page/layout_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/page/layout_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/block/html_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/block/html_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/block/otsl_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/block/otsl_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/block/text_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/block/text_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/page/layout_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/page/layout_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/block/html_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/block/html_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/block/otsl_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/block/otsl_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/block/text_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/block/text_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/page/layout_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/page/layout_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/LangSample/format_0305/page/layout_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/LangSample/format_0305/page/layout_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/block/html_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/block/html_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/block/otsl_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/block/otsl_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/block/text_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/block/text_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/page/layout_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/page/layout_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/block/html_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/block/html_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/block/otsl_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/block/otsl_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/block/text_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/block/text_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/page/layout_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/page/layout_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/block/html_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/block/html_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/block/otsl_rot_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/block/otsl_filter_reward_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/block/text_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/block/text_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/page/layout_rot_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/page/layout_ori.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/kingsoft/layout_0303_ori.json",
+    "/home/kas/mzy/layout/ominidoc_order/datas_infer_table_html-0309_filter_reward.json",
+    "/home/kas/mzy/layout/ominidoc_order/datas_infer_table_otsl_filter_reward.json",
+    "/home/kas/mzy/layout/ominidoc_order/datas_infer_text.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout/ordinary__resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout/rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/page/layout_rec/rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/block/html_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/block/html_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/block/otsl_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/block/otsl_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/block/text_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/block/text_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/page/layout_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/CDLA/format_0305/page/layout_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/block/html_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/block/html_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/block/otsl_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/block/otsl_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/block/text_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/block/text_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/page/layout_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/D4LA/format_0305/page/layout_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/block/html_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/block/html_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/block/otsl_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/block/otsl_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/block/text_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/block/text_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/page/layout_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocGenome/format_0305/page/layout_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/block/html_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/block/html_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/block/otsl_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/block/otsl_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/block/text_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/block/text_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/page/layout_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/DocLayNet/format_0305/page/layout_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/block/html_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/block/html_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/block/otsl_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/block/otsl_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/block/text_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/block/text_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/page/layout_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/k2k3k4/format_0305/page/layout_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/LangSample/format_0305/page/layout_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/LangSample/format_0305/page/layout_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/block/html_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/block/html_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/block/otsl_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/block/otsl_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/block/text_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/block/text_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/page/layout_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/M6Doc/format_0305/page/layout_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/block/html_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/block/html_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/block/otsl_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/block/otsl_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/block/text_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/block/text_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/page/layout_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/SVRD/format_0305/page/layout_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/block/html_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/block/html_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/block/otsl_rot_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/block/otsl_filter_reward_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/block/text_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/block/text_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/page/layout_rot_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/TabRecSet/format_0305/page/layout_ori_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/kingsoft/layout_0303_ori_resize.json",
+    "/home/kas/mzy/layout/ominidoc_order/datas_infer_table_html-0309_filter_reward_resize.json",
+    "/home/kas/mzy/layout/ominidoc_order/datas_infer_table_otsl_filter_reward_resize.json",
+    "/home/kas/mzy/layout/ominidoc_order/datas_infer_text_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/complex_table/format/block/html_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/complex_table/format/block/otsl_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/complex_table/format/block/text.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/complex_table/format/page/layout.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/handwritten/format/block/html_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/handwritten/format/block/otsl_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/handwritten/format/block/text.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/handwritten/format/page/layout.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/photo_table/format/block/html_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/photo_table/format/block/otsl_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/photo_table/format/block/text.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/photo_table/format/page/layout.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/ppt/format/block/html_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/ppt/format/block/otsl_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/ppt/format/block/text.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/ppt/format/page/layout.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/youdao/format/block/html_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/youdao/format/block/otsl_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/youdao/format/block/text.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/youdao/format/page/layout.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/c2/format/block/html_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/c2/format/block/otsl_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/c2/format/block/text_500k.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/c2/format/page/layout.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/jiaocai/format/block/html_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/jiaocai/format/block/otsl_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/jiaocai/format/block/text_500k.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/jiaocai/format/page/layout.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/note/format/block/html_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/note/format/block/otsl_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/note/format/block/text.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/note/format/page/layout.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/newspaper/english/all_format/block/html_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/newspaper/english/all_format/block/otsl_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/newspaper/english/all_format/block/text_500k.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/newspaper/english/all_format/page/layout.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/newspaper/chinese/all_format/block/html_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/newspaper/chinese/all_format/block/otsl_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/newspaper/chinese/all_format/block/text_500k.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/newspaper/chinese/all_format/page/layout.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/all_format/page/layout_rot.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/all_format/block/html_rot_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/all_format/block/otsl_rot_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/all_format/block/text_rot.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot/10_rot.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot/13_rot.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot/16_rot.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot/19_rot.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot/1_rot.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot/22_rot.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot/25_rot.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot/28_rot.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot/31_rot.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot/34_rot.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot/4_rot.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot/7_rot.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/all_format/block/html_resize_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/all_format/block/otsl_resize_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/all_format/block/html_rot_resize_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/all_format/block/otsl_rot_resize_filter_reward.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/all_format/block/text_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/all_format/block/text_rot_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/all_format/page/layout_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/all_format/page/layout_rot_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_resize/12_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_resize/15_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_resize/18_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_resize/21_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_resize/24_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_resize/27_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_resize/30_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_resize/33_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_resize/36_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_resize/3_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_resize/6_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_resize/9_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot_resize/10_rot_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot_resize/1_rot_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot_resize/4_rot_resize.json",
+    "/home/kas/kas_outside/zhangshuo/data/FinePDF/filter/0111/block/text/ordinary_splits_rot_resize/7_rot_resize.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/ppt/format/page/layout1.json",
+    "/home/kas/kas_outside/zhangshuo/PaddleOCR-VL-1.5/data/0304/ppt/format/block/text1.json"
+  ],
+  "val_dataset": [],
+  "cached_dataset": [],
+  "split_dataset_ratio": 0.0,
+  "dataset_num_proc": 32,
+  "load_from_cache_file": true,
+  "dataset_shuffle": true,
+  "val_dataset_shuffle": false,
+  "streaming": false,
+  "interleave_prob": null,
+  "stopping_strategy": "first_exhausted",
+  "shuffle_buffer_size": 1000,
+  "download_mode": "reuse_dataset_if_exists",
+  "columns": {},
+  "strict": false,
+  "model_name": null,
+  "model_author": null,
+  "custom_dataset_info": [],
+  "quant_method": null,
+  "quant_bits": null,
+  "hqq_axis": null,
+  "bnb_4bit_compute_dtype": "bfloat16",
+  "bnb_4bit_quant_type": "nf4",
+  "bnb_4bit_use_double_quant": true,
+  "bnb_4bit_quant_storage": null,
+  "max_new_tokens": 64,
+  "temperature": 0.0,
+  "top_k": null,
+  "top_p": null,
+  "repetition_penalty": null,
+  "num_beams": 1,
+  "stream": false,
+  "stop_words": [],
+  "logprobs": false,
+  "top_logprobs": null,
+  "ckpt_dir": "/home/kas/kas_checkpoint/zhangziyang/monkeyocrv2/sft_s0_base_v2_12m/checkpoint-46641",
+  "lora_modules": [],
+  "train_type": "full",
+  "adapters": [],
+  "external_plugins": [],
+  "model_kwargs": {},
+  "load_args": false,
+  "load_data_args": false,
+  "packing": false,
+  "packing_length": null,
+  "packing_num_proc": 1,
+  "lazy_tokenize": true,
+  "custom_register_path": [],
+  "use_hf": false,
+  "ignore_args_error": false,
+  "use_swift_lora": false,
+  "freeze_parameters": [
+    "vision_tower"
+  ],
+  "freeze_parameters_regex": null,
+  "freeze_parameters_ratio": 0.0,
+  "trainable_parameters": [
+    "vision_tower.merger"
+  ],
+  "trainable_parameters_regex": null,
+  "freeze_llm": false,
+  "freeze_vit": true,
+  "freeze_aligner": false,
+  "target_modules": [
+    "all-linear"
+  ],
+  "target_regex": null,
+  "target_parameters": null,
+  "modules_to_save": [],
+  "lora_rank": 8,
+  "lora_alpha": 32,
+  "lora_dropout": 0.05,
+  "lora_bias": "none",
+  "lora_dtype": null,
+  "lorap_lr_ratio": null,
+  "use_rslora": false,
+  "use_dora": false,
+  "lora_ga_batch_size": 2,
+  "lora_ga_iters": 2,
+  "lora_ga_max_length": 1024,
+  "lora_ga_direction": "ArB2r",
+  "lora_ga_scale": "stable",
+  "lora_ga_stable_gamma": 16,
+  "init_weights": true,
+  "fourier_n_frequency": 2000,
+  "fourier_scaling": 300.0,
+  "boft_block_size": 4,
+  "boft_block_num": 0,
+  "boft_n_butterfly_factor": 1,
+  "boft_dropout": 0.0,
+  "vera_rank": 256,
+  "vera_projection_prng_key": 0,
+  "vera_dropout": 0.0,
+  "vera_d_initial": 0.1,
+  "adapter_act": "gelu",
+  "adapter_length": 128,
+  "use_galore": false,
+  "galore_target_modules": null,
+  "galore_rank": 128,
+  "galore_update_proj_gap": 50,
+  "galore_scale": 1.0,
+  "galore_proj_type": "std",
+  "galore_optim_per_parameter": false,
+  "galore_with_embedding": false,
+  "galore_quantization": false,
+  "galore_proj_quant": false,
+  "galore_proj_bits": 4,
+  "galore_proj_group_size": 256,
+  "galore_cos_threshold": 0.4,
+  "galore_gamma_proj": 2,
+  "galore_queue_size": 5,
+  "adalora_target_r": 8,
+  "adalora_init_r": 12,
+  "adalora_tinit": 0,
+  "adalora_tfinal": 0,
+  "adalora_deltaT": 1,
+  "adalora_beta1": 0.85,
+  "adalora_beta2": 0.85,
+  "adalora_orth_reg_weight": 0.5,
+  "llamapro_num_new_blocks": 4,
+  "llamapro_num_groups": null,
+  "lisa_activated_layers": 0,
+  "lisa_step_interval": 20,
+  "reft_layer_key": null,
+  "reft_layers": null,
+  "reft_rank": 4,
+  "reft_intervention_type": "LoreftIntervention",
+  "reft_args": null,
+  "swanlab_token": null,
+  "swanlab_project": "MonkeyOCR2sft",
+  "swanlab_workspace": null,
+  "swanlab_exp_name": "sft_s1_base_v2",
+  "swanlab_lark_webhook_url": null,
+  "swanlab_lark_secret": null,
+  "swanlab_mode": "cloud",
+  "add_version": false,
+  "create_checkpoint_symlink": false,
+  "zero_hpz_partition_size": null,
+  "deepspeed_autotp_size": null,
+  "early_stop_interval": null,
+  "rank": 0,
+  "global_world_size": 64,
+  "local_world_size": 8,
+  "model_suffix": "checkpoint-46641",
+  "model_info": "ModelInfo(model_type='monkeyocrv2', model_dir='/home/kas/kas_checkpoint/zhangziyang/monkeyocrv2/sft_s0_base_v2_12m/checkpoint-46641', torch_dtype=torch.bfloat16, max_model_len=40960, quant_method=None, quant_bits=None, rope_scaling=None, is_moe_model=False, config=None, task_type='causal_lm', num_labels=None)",
+  "model_meta": "ModelMeta(model_type='monkeyocrv2', model_groups=[ModelGroup(models=[Model(ms_model_id='/home/kas/kas_outside/models/monkeyocrv2_pretrain', hf_model_id='/home/kas/kas_outside/models/monkeyocrv2_pretrain', model_path=None, ms_revision=None, hf_revision=None)], ignore_patterns=None, requires=None, tags=[])], template='monkeyocrv2', get_function=<function get_model_tokenizer_monkeyocrv2 at 0x7f24869feef0>, model_arch=MultiModelKeys(arch_name='monkeyocrv2', embedding=None, module_list=None, lm_head=None, q_proj=None, k_proj=None, v_proj=None, o_proj=None, attention=None, mlp=None, down_proj=None, qkv_proj=None, qk_proj=None, qa_proj=None, qb_proj=None, kv_proj=None, kva_proj=None, kvb_proj=None, language_model=['model', 'lm_head'], aligner=['vision_tower.merger'], vision_tower=['vision_tower'], generator=[]), architectures=['MonkeyOCRv2ForCausalLM'], additional_saved_files=[], torch_dtype=None, is_multimodal=True, is_reward=False, is_reranker=False, task_type=None, ignore_patterns=None, requires=['transformers>=4.57.1'], tags=[])",
+  "model_dir": "/home/kas/kas_checkpoint/zhangziyang/monkeyocrv2/sft_s0_base_v2_12m/checkpoint-46641",
+  "_val_dataset_exists": [],
+  "hub": "<class 'swift.hub.hub.MSHub'>",
+  "evaluation_strategy": "steps",
+  "training_args": "Seq2SeqTrainingArguments(output_dir='/home/kas/kas_checkpoint/zhangziyang/monkeyocrv2/sft_s1_base_v2', overwrite_output_dir=False, do_train=False, do_eval=False, do_predict=False, eval_strategy=<IntervalStrategy.NO: 'no'>, prediction_loss_only=False, per_device_train_batch_size=4, per_device_eval_batch_size=2, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, eval_delay=0, torch_empty_cache_steps=None, learning_rate=2e-05, weight_decay=0.1, adam_beta1=0.9, adam_beta2=0.95, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, lr_scheduler_type=<SchedulerType.COSINE: 'cosine'>, lr_scheduler_kwargs=None, warmup_ratio=0.05, warmup_steps=0, log_level='passive', log_level_replica='warning', log_on_each_node=True, logging_dir='/home/kas/kas_checkpoint/zhangziyang/monkeyocrv2/sft_s1_base_v2/runs', logging_strategy=<IntervalStrategy.STEPS: 'steps'>, logging_first_step=True, logging_steps=5, logging_nan_inf_filter=True, save_strategy=<SaveStrategy.STEPS: 'steps'>, save_steps=5000, save_total_limit=20, save_safetensors=True, save_on_each_node=False, save_only_model=False, restore_callback_states_from_checkpoint=False, no_cuda=False, use_cpu=False, use_mps_device=False, seed=42, data_seed=42, jit_mode_eval=False, bf16=True, fp16=False, fp16_opt_level='O1', half_precision_backend='auto', bf16_full_eval=False, fp16_full_eval=False, tf32=None, local_rank=0, ddp_backend=None, tpu_num_cores=None, tpu_metrics_debug=False, debug=[], dataloader_drop_last=False, eval_steps=400000.0, dataloader_num_workers=16, dataloader_prefetch_factor=10, past_index=-1, run_name='/home/kas/kas_checkpoint/zhangziyang/monkeyocrv2/sft_s1_base_v2', disable_tqdm=False, remove_unused_columns=False, label_names=None, load_best_model_at_end=False, metric_for_best_model='loss', greater_is_better=False, ignore_data_skip=False, fsdp=[], fsdp_min_num_params=0, fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, fsdp_transformer_layer_cls_to_wrap=None, accelerator_config=AcceleratorConfig(split_batches=False, dispatch_batches=False, even_batches=True, use_seedable_sampler=True, non_blocking=False, gradient_accumulation_kwargs=None, use_configured_state=False), parallelism_config=None, deepspeed={'fp16': {'enabled': 'auto', 'loss_scale': 0, 'loss_scale_window': 1000, 'initial_scale_power': 16, 'hysteresis': 2, 'min_loss_scale': 1}, 'bf16': {'enabled': 'auto'}, 'zero_optimization': {'stage': 1, 'offload_optimizer': {'device': 'none', 'pin_memory': True}, 'allgather_partitions': True, 'allgather_bucket_size': 200000000.0, 'overlap_comm': False, 'reduce_scatter': True, 'reduce_bucket_size': 200000000.0, 'contiguous_gradients': True}, 'gradient_accumulation_steps': 'auto', 'gradient_clipping': 'auto', 'steps_per_print': 2000, 'train_batch_size': 'auto', 'train_micro_batch_size_per_gpu': 'auto', 'wall_clock_breakdown': False}, label_smoothing_factor=0.0, optim=<OptimizerNames.ADAMW_TORCH_FUSED: 'adamw_torch_fused'>, optim_args=None, adafactor=False, group_by_length=False, length_column_name='length', report_to=['tensorboard', 'swanlab'], project='huggingface', trackio_space_id='trackio', ddp_find_unused_parameters=None, ddp_bucket_cap_mb=None, ddp_broadcast_buffers=None, dataloader_pin_memory=True, dataloader_persistent_workers=False, skip_memory_metrics=True, use_legacy_prediction_loop=False, push_to_hub=False, resume_from_checkpoint='/home/kas/kas_checkpoint/zhangziyang/monkeyocrv2/sft_s1_base_v2/checkpoint-355000', hub_model_id=None, hub_strategy=<HubStrategy.EVERY_SAVE: 'every_save'>, hub_token=None, hub_private_repo=None, hub_always_push=False, hub_revision=None, gradient_checkpointing=True, gradient_checkpointing_kwargs=None, include_inputs_for_metrics=False, include_for_metrics=[], eval_do_concat_batches=True, fp16_backend='auto', push_to_hub_model_id=None, push_to_hub_organization=None, push_to_hub_token=None, mp_parameters='', auto_find_batch_size=False, full_determinism=False, torchdynamo=None, ray_scope='last', ddp_timeout=18000000, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, include_tokens_per_second=None, include_num_input_tokens_seen=None, neftune_noise_alpha=None, optim_target_modules=None, batch_eval_metrics=False, eval_on_start=False, use_liger_kernel=False, liger_kernel_config=None, eval_use_gather_object=False, average_tokens_across_devices=None, sortish_sampler=False, predict_with_generate=False, generation_max_length=None, generation_num_beams=None, generation_config=None, tuner_backend='peft', vit_gradient_checkpointing=True, router_aux_loss_coef=0.0, enable_dft_loss=False, enable_channel_loss=False, check_model=True, acc_strategy='token', train_dataloader_shuffle=True, max_epochs=None, aligner_lr=None, vit_lr=None, use_logits_to_keep=None, ds3_gather_for_generation=True, resume_only_model=False, optimizer=None, loss_type=None, metric=None, eval_use_evalscope=False, eval_dataset=[], eval_dataset_args=None, eval_limit=None, eval_generation_config=None, extra_eval_args=None, use_flash_ckpt=False, sft_alpha=0, chord_sft_dataset=[], chord_sft_per_device_train_batch_size=None, chord_enable_phi_function=False, chord_mu_warmup_steps=None, chord_mu_decay_steps=None, chord_mu_peak=None, chord_mu_valley=None, train_type='full', local_repo_path=None, galore_config=None, padding_side='right', padding_free=True, task_type='causal_lm', problem_type=None)"
+}

chat_template.jinja ADDED Viewed

	@@ -0,0 +1,120 @@

+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' }}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- for message in messages %}
+    {%- if message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content in message.content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
+                    <|vision_start|><|image_pad|><|vision_end|>
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
+                    <|vision_start|><|video_pad|><|vision_end|>
+                {%- elif 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content_item in message.content %}
+                {%- if 'text' in content_item %}
+                    {{- content_item.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and message.content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content in message.content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
+                    <|vision_start|><|image_pad|><|vision_end|>
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
+                    <|vision_start|><|video_pad|><|vision_end|>
+                {%- elif 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

config.json ADDED Viewed

	@@ -0,0 +1,88 @@

+{
+  "architectures": [
+    "MonkeyOCRv2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_monkeyocrv2.MonkeyOCRv2Config",
+    "AutoModelForCausalLM": "modeling_monkeyocrv2.MonkeyOCRv2ForCausalLM"
+  },
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1024,
+  "image_token_id": 151655,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 40960,
+  "max_window_layers": 28,
+  "model_type": "monkeyocrv2",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "pad_token_id": 151643,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "transformers_version": "4.57.1",
+  "use_cache": false,
+  "use_sliding_window": false,
+  "video_token_id": 151656,
+  "vision_config": {
+    "attn_implementation": "flash_attention_2",
+    "dtype": "bfloat16",
+    "embed_dim": 768,
+    "gradient_checkpointing": false,
+    "hidden_size": 1024,
+    "init_merger_std": 0.02,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "is_causal": false,
+    "model_type": "monkeyocr_vit",
+    "num_attention_heads": 12,
+    "num_channels": 3,
+    "num_hidden_layers": 12,
+    "pad_token_id": 151643,
+    "patch_size": 14,
+    "post_norm": true,
+    "rms_norm_eps": 1e-05,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 1,
+    "use_bias": false
+  },
+  "vocab_size": 151936
+}

configuration_monkeyocrv2.py ADDED Viewed

	@@ -0,0 +1,78 @@

+from typing import Any, Optional
+from transformers.configuration_utils import PretrainedConfig
+from transformers.models.qwen3 import Qwen3Config
+from transformers import Qwen2_5_VLProcessor, AutoProcessor, AutoConfig
+class MonkeyOCRv2VisionConfig(PretrainedConfig):
+    model_type: str = "monkeyocr_vit"
+    def __init__(
+        self,
+        embed_dim: int = 1536,  # vision encoder embed size
+        hidden_size: int = 1536,  # after merger hidden size
+        intermediate_size: int = 4224,
+        num_hidden_layers: int = 42,
+        num_attention_heads: int = 12,
+        num_channels: int = 3,
+        patch_size: int = 14,
+        spatial_merge_size: int = 2,
+        temporal_patch_size: int = 1,
+        rms_norm_eps: float = 1e-5,
+        use_bias: bool = False,
+        attn_implementation="flash_attention_2",  # "eager","sdpa","flash_attention_2"
+        initializer_range=0.02,
+        init_merger_std=0.02,
+        is_causal=False,  # ve causal forward
+        post_norm=True,
+        gradient_checkpointing=False,
+        **kwargs: Any,
+    ):
+        super().__init__(**kwargs)
+        self.embed_dim = embed_dim
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_channels = num_channels
+        self.patch_size = patch_size
+        self.spatial_merge_size = spatial_merge_size
+        self.temporal_patch_size = temporal_patch_size
+        self.rms_norm_eps = rms_norm_eps
+        self.use_bias = use_bias
+        self.attn_implementation = attn_implementation
+        self.initializer_range = initializer_range
+        self.init_merger_std = init_merger_std
+        self.is_causal = is_causal
+        self.post_norm = post_norm
+        self.gradient_checkpointing = gradient_checkpointing
+class MonkeyOCRv2Config(Qwen3Config):
+    model_type = "monkeyocrv2"
+    def __init__(self,
+        image_token_id = 151655,
+        video_token_id = 151656,
+        vision_config: Optional[dict] = None, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.image_token_id = image_token_id
+        self.video_token_id = video_token_id
+        self.vision_config = MonkeyOCRv2VisionConfig(**(vision_config or {}))
+    def save_pretrained(self, save_directory, **kwargs):
+        self._auto_class = None
+        super().save_pretrained(save_directory, **kwargs)
+class MonkeyOCRv2Processor(Qwen2_5_VLProcessor):
+    attributes = ["image_processor", "tokenizer"]
+    def __init__(self, image_processor=None, tokenizer=None, chat_template=None, **kwargs):
+        super().__init__(image_processor, tokenizer, chat_template=chat_template)
+        self.image_token = "<|image_pad|>" if not hasattr(tokenizer, "image_token") else tokenizer.image_token
+        self.image_token_id = 151655 if not hasattr(tokenizer, "image_token_id") else tokenizer.image_token_id
+AutoProcessor.register("monkeyocrv2", MonkeyOCRv2Processor)
+AutoConfig.register("monkeyocrv2", MonkeyOCRv2Config)
+__all__ = ["MonkeyOCRv2Config", "MonkeyOCRv2VisionConfig", "MonkeyOCRv2Processor"]

generation_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "temperature": 0.6,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "4.57.1"
+}

merges.txt ADDED Viewed

The diff for this file is too large to render. See raw diff

model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0267fdc991c9be02cf1b60405f77fe0629d084970ad9d6d08163feda3d470284
+size 1755925032

modeling_monkeyocrv2.py ADDED Viewed

	@@ -0,0 +1,142 @@

+from typing import List, Optional, Tuple, Union
+import torch
+from transformers.modeling_outputs import CausalLMOutputWithPast
+from transformers.models.qwen3 import Qwen3ForCausalLM
+from .configuration_monkeyocrv2 import MonkeyOCRv2VisionConfig, MonkeyOCRv2Config
+from .modeling_monkeyocrv2_vision import MonkeyOCRv2VisionTransformer, VisionTransformerDecoder
+import torch.nn as nn
+from einops import rearrange
+IMAGENET_DEFAULT_MEAN = [ 0.48145466, 0.4578275, 0.40821073 ]
+IMAGENET_DEFAULT_STD = [ 0.26862954, 0.26130258, 0.27577711 ]
+VLM_MAX_IMAGES = 200
+class MonkeyOCRv2ForCausalLM(Qwen3ForCausalLM):
+    config_class = MonkeyOCRv2Config
+    def __init__(self, config: MonkeyOCRv2Config):
+        super().__init__(config)
+        if isinstance(self.config.vision_config, dict):
+            vision_config = MonkeyOCRv2VisionConfig(**self.config.vision_config)
+            self.config.vision_config = vision_config
+        else:
+            vision_config = self.config.vision_config
+        self.vision_tower = MonkeyOCRv2VisionTransformer(vision_config)
+    def prepare_inputs_embeds(
+        self,
+        input_ids: torch.LongTensor,
+        pixel_values: Optional[torch.FloatTensor] = None,
+        grid_thw: Optional[torch.FloatTensor] = None,
+        img_mask: Optional[torch.BoolTensor] = None,
+    ) -> torch.Tensor:
+        inputs_embeds = self.get_input_embeddings()(input_ids)
+        if pixel_values is not None:
+            assert img_mask is not None
+            if grid_thw.shape[0] > VLM_MAX_IMAGES:
+                print(
+                    f"Num image exceeded: {grid_thw.shape[0]} > {VLM_MAX_IMAGES}, which may cause FSDP hang"
+                )
+            vision_embeddings, vision_embeddings_nomerge = self.vision_tower(pixel_values, grid_thw)
+            true_indices = torch.nonzero(img_mask).squeeze()
+            if len(true_indices) > vision_embeddings.size(0):
+                print(
+                    f"img_mask sum > VE and will be truncated, mask.sum()={len(true_indices)} {vision_embeddings.size(0)=}"
+                )
+                true_indices = true_indices[: vision_embeddings.size(0)]
+                new_img_mask = torch.zeros_like(img_mask, device=img_mask.device)
+                new_img_mask[true_indices[:, 0], true_indices[:, 1]] = True
+            else:
+                new_img_mask = img_mask
+            assert (
+                vision_embeddings.size(0) == new_img_mask.sum()
+            ), f"{vision_embeddings.size(0)=}, {new_img_mask.sum()=}"
+            inputs_embeds = inputs_embeds.masked_scatter(
+                new_img_mask.to(inputs_embeds.device).unsqueeze(-1).expand_as(inputs_embeds),
+                vision_embeddings.to(inputs_embeds.device).type(inputs_embeds.dtype),
+            )
+            return inputs_embeds, vision_embeddings_nomerge
+        return inputs_embeds
+    def forward(
+        self,
+        input_ids: torch.LongTensor,
+        pixel_values: Optional[torch.FloatTensor] = None,
+        pixel_values_ori: Optional[torch.FloatTensor] = None,
+        image_grid_thw: Optional[torch.FloatTensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[torch.FloatTensor]] = None,
+        labels: Optional[torch.LongTensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        use_cache: Optional[bool] = None,
+        logits_to_keep: int = 0,
+        **loss_kwargs,
+    ) -> Union[Tuple, CausalLMOutputWithPast]:
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+        assert len(input_ids) >= 1, f"empty input_ids {input_ids.shape=} will cause gradnorm nan"
+        if inputs_embeds is None:
+            img_mask = input_ids == self.config.image_token_id
+            if pixel_values is not None:
+                inputs_embeds, vision_embeddings_nomerge = self.prepare_inputs_embeds(input_ids, pixel_values, image_grid_thw, img_mask)
+            else:
+                inputs_embeds = self.prepare_inputs_embeds(input_ids, pixel_values, image_grid_thw, img_mask)
+        outputs = super().forward(
+            inputs_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            labels=labels,
+            use_cache=use_cache if use_cache is not None else self.config.use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            # return_dict=return_dict,
+            logits_to_keep=logits_to_keep,
+            **loss_kwargs,
+        )
+        return outputs
+    def prepare_inputs_for_generation(
+        self,
+        input_ids,
+        past_key_values=None,
+        inputs_embeds=None,
+        pixel_values=None,
+        attention_mask=None,
+        cache_position=None,
+        num_logits_to_keep=None,
+        **kwargs,
+    ):
+        model_inputs = super().prepare_inputs_for_generation(
+            input_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            cache_position=cache_position,
+            num_logits_to_keep=num_logits_to_keep,
+            **kwargs,
+        )
+        if cache_position[0] == 0:
+            model_inputs["pixel_values"] = pixel_values
+        return model_inputs

modeling_monkeyocrv2_vision.py ADDED Viewed

	@@ -0,0 +1,531 @@

+import math
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torch.utils.checkpoint
+flash_attn_available = True
+npu_available = True
+try:
+    from flash_attn import flash_attn_varlen_func
+except ImportError:
+    flash_attn_available = False
+from torch.nn import LayerNorm
+from transformers.modeling_utils import PreTrainedModel
+from .configuration_monkeyocrv2 import MonkeyOCRv2VisionConfig
+try:
+    import torch_npu
+except ImportError:
+    npu_available = False
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2:]
+    return torch.cat((-x2, x1), dim=-1)
+def apply_rotary_pos_emb_vision(tensor: torch.Tensor, freqs: torch.Tensor) -> torch.Tensor:
+    orig_dtype = tensor.dtype
+    tensor = tensor.float()
+    cos = freqs.cos()
+    sin = freqs.sin()
+    cos = cos.unsqueeze(1).repeat(1, 1, 2).unsqueeze(0).float()
+    sin = sin.unsqueeze(1).repeat(1, 1, 2).unsqueeze(0).float()
+    output = (tensor * cos) + (rotate_half(tensor) * sin)
+    output = output.to(orig_dtype)
+    return output
+class VisionRotaryEmbedding(nn.Module):
+    def __init__(self, dim: int, theta: float = 10000.0) -> None:
+        super().__init__()
+        inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float) / dim))
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+    def forward(self, seqlen: int) -> torch.Tensor:
+        seq = torch.arange(seqlen, device=self.inv_freq.device, dtype=self.inv_freq.dtype)
+        freqs = torch.outer(seq, self.inv_freq)
+        return freqs
+class PatchMerger(nn.Module):
+    def __init__(
+            self,
+            dim: int,
+            context_dim: int,
+            spatial_merge_size: int = 2,
+            pre_norm="layernorm",
+            init_merger_std=None,
+    ) -> None:
+        super().__init__()
+        self.hidden_size = context_dim * (spatial_merge_size ** 2)
+        self.pre_norm = pre_norm
+        if self.pre_norm == "layernorm":
+            self.ln_q = LayerNorm(context_dim, eps=1e-6)
+        elif self.pre_norm == "rmsnorm":
+            self.ln_q = RMSNorm(context_dim, eps=1e-6)
+        else:
+            print("no norm in patch merger")
+        self.mlp = nn.Sequential(
+            nn.Linear(self.hidden_size, self.hidden_size),
+            nn.GELU(),
+            nn.Linear(self.hidden_size, dim),
+        )
+        if init_merger_std is not None:
+            nn.init.normal_(self.mlp[0].weight, mean=0.0, std=init_merger_std)
+            nn.init.zeros_(self.mlp[0].bias)
+            nn.init.normal_(self.mlp[2].weight, mean=0.0, std=init_merger_std)
+            nn.init.zeros_(self.mlp[2].bias)
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        if self.pre_norm:
+            x = self.mlp(self.ln_q(x).view(-1, self.hidden_size))
+        else:
+            x = self.mlp(x.view(-1, self.hidden_size))
+        return x
+class VisionAttention(nn.Module):
+    def __init__(self, config, dim: int, num_heads: int = 16, bias=True) -> None:
+        super().__init__()
+        self.num_heads = num_heads
+        self.head_dim = dim // num_heads
+        self.qkv = nn.Linear(dim, dim * 3, bias=bias)
+        self.proj = nn.Linear(dim, dim, bias=bias)
+    def forward(
+            self,
+            hidden_states: torch.Tensor,
+            cu_seqlens: torch.Tensor,
+            rotary_pos_emb: torch.Tensor = None,
+    ) -> torch.Tensor:
+        seq_length = hidden_states.shape[0]
+        q, k, v = self.qkv(hidden_states).reshape(seq_length, 3, self.num_heads, -1).permute(1, 0, 2, 3).unbind(0)
+        q = apply_rotary_pos_emb_vision(q.unsqueeze(0), rotary_pos_emb).squeeze(0)
+        k = apply_rotary_pos_emb_vision(k.unsqueeze(0), rotary_pos_emb).squeeze(0)
+        attention_mask = torch.full(
+            [1, seq_length, seq_length], torch.finfo(q.dtype).min, device=q.device, dtype=q.dtype
+        )
+        for i in range(1, len(cu_seqlens)):
+            attention_mask[..., cu_seqlens[i - 1]: cu_seqlens[i], cu_seqlens[i - 1]: cu_seqlens[i]] = 0
+        q = q.transpose(0, 1)
+        k = k.transpose(0, 1)
+        v = v.transpose(0, 1)
+        attn_weights = torch.matmul(q, k.transpose(1, 2)) / math.sqrt(self.head_dim)
+        attn_weights = attn_weights + attention_mask
+        attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(q.dtype)
+        attn_output = torch.matmul(attn_weights, v)
+        attn_output = attn_output.transpose(0, 1)
+        attn_output = attn_output.reshape(seq_length, -1)
+        attn_output = self.proj(attn_output)
+        return attn_output
+class VisionFlashAttention2(nn.Module):
+    def __init__(self, config, dim: int, num_heads: int = 16, bias=True) -> None:
+        super().__init__()
+        self.num_heads = num_heads
+        self.qkv = nn.Linear(dim, dim * 3, bias=bias)
+        self.proj = nn.Linear(dim, dim, bias=bias)
+        self.config = config
+        self.is_causal = config.is_causal
+    def forward(
+            self,
+            hidden_states: torch.Tensor,
+            cu_seqlens: torch.Tensor,
+            rotary_pos_emb: torch.Tensor = None,
+    ) -> torch.Tensor:
+        seq_length = hidden_states.shape[0]
+        q, k, v = (
+            self.qkv(hidden_states).reshape(seq_length, 3, self.num_heads, -1).permute(1, 0, 2, 3).unbind(0)
+        )  # 'shd'
+        q = apply_rotary_pos_emb_vision(q.unsqueeze(0), rotary_pos_emb).squeeze(0)
+        k = apply_rotary_pos_emb_vision(k.unsqueeze(0), rotary_pos_emb).squeeze(0)
+        max_seqlen = (cu_seqlens[1:] - cu_seqlens[:-1]).max().item()
+        attn_output = flash_attn_varlen_func(
+            q, k, v, cu_seqlens, cu_seqlens, max_seqlen, max_seqlen, causal=self.is_causal
+        ).reshape(seq_length, -1)
+        attn_output = self.proj(attn_output)
+        return attn_output
+class VisionAttentionV2(nn.Module):
+    def __init__(self, config, dim: int, num_heads: int = 16, bias=True) -> None:
+        super().__init__()
+        self.num_heads = num_heads
+        self.head_dim = dim // num_heads
+        self.qkv = nn.Linear(dim, dim * 3, bias=bias)
+        self.proj = nn.Linear(dim, dim, bias=bias)
+    def forward(
+            self,
+            hidden_states: torch.Tensor,
+            cu_seqlens: torch.Tensor,
+            rotary_pos_emb: torch.Tensor = None,
+    ) -> torch.Tensor:
+        seq_length = hidden_states.shape[0]
+        q, k, v = self.qkv(hidden_states).reshape(seq_length, 3, self.num_heads, -1).permute(1, 0, 2, 3).unbind(0)
+        q = apply_rotary_pos_emb_vision(q.unsqueeze(0), rotary_pos_emb).squeeze(0)
+        k = apply_rotary_pos_emb_vision(k.unsqueeze(0), rotary_pos_emb).squeeze(0)
+        seqlens = torch.diff(cu_seqlens).tolist()
+        q_list = torch.split(q, seqlens, 0)
+        k_list = torch.split(k, seqlens, 0)
+        v_list = torch.split(v, seqlens, 0)
+        outputs = []
+        for q_i, k_i, v_i in zip(q_list, k_list, v_list):
+            q_i = q_i.transpose(0, 1)
+            k_i = k_i.transpose(0, 1)
+            v_i = v_i.transpose(0, 1)
+            out = torch.matmul(q_i, k_i.transpose(1, 2)) / math.sqrt(self.head_dim)
+            out = nn.functional.softmax(out, dim=-1, dtype=torch.float32).to(q.dtype)
+            out = torch.matmul(out, v_i)
+            out = out.transpose(0, 1)
+            outputs.append(out)
+        attn_output = torch.concat(outputs, dim=0)
+        attn_output = attn_output.reshape(seq_length, -1)
+        attn_output = self.proj(attn_output)
+        return attn_output
+class VisionAscendAttention(nn.Module):
+    def __init__(self, config, dim: int, num_heads: int = 16, bias=True) -> None:
+        super().__init__()
+        self.num_heads = num_heads
+        self.head_dim = dim // num_heads
+        self.qkv = nn.Linear(dim, dim * 3, bias=bias)
+        self.proj = nn.Linear(dim, dim, bias=bias)
+        self.config = config
+    def forward(
+            self,
+            hidden_states: torch.Tensor,
+            cu_seqlens: torch.Tensor,
+            rotary_pos_emb: torch.Tensor = None,
+    ) -> torch.Tensor:
+        seq_length = hidden_states.shape[0]
+        q, k, v = self.qkv(hidden_states).reshape(seq_length, 3, self.num_heads, -1).permute(1, 0, 2, 3).unbind(0)
+        q = apply_rotary_pos_emb_vision(q.unsqueeze(0), rotary_pos_emb).squeeze(0)
+        k = apply_rotary_pos_emb_vision(k.unsqueeze(0), rotary_pos_emb).squeeze(0)
+        attention_mask = torch.ones([1, seq_length, seq_length], device=q.device, dtype=torch.bool)
+        for i in range(1, len(cu_seqlens)):
+            attention_mask[..., cu_seqlens[i - 1]: cu_seqlens[i], cu_seqlens[i - 1]: cu_seqlens[i]] = False
+        q = q.transpose(0, 1).unsqueeze(0)
+        k = k.transpose(0, 1).unsqueeze(0)
+        v = v.transpose(0, 1).unsqueeze(0)
+        attn_output = torch_npu.npu_prompt_flash_attention(q, k, v,
+                                                           atten_mask=attention_mask,
+                                                           num_heads=self.num_heads, input_layout="BNSD",
+                                                           scale_value=self.head_dim ** -0.5)
+        attn_output = attn_output.squeeze(0).transpose(0, 1)
+        attn_output = attn_output.reshape(seq_length, -1)
+        attn_output = self.proj(attn_output)
+        return attn_output
+class VisionSdpaAttention(nn.Module):
+    def __init__(self, config, dim: int, num_heads: int = 16, bias=True) -> None:
+        super().__init__()
+        self.num_heads = num_heads
+        self.qkv = nn.Linear(dim, dim * 3, bias=bias)
+        self.proj = nn.Linear(dim, dim, bias=bias)
+        self.config = config
+    def forward(
+            self,
+            hidden_states: torch.Tensor,
+            cu_seqlens: torch.Tensor,
+            rotary_pos_emb: torch.Tensor = None,
+    ) -> torch.Tensor:
+        seq_length = hidden_states.shape[0]
+        q, k, v = self.qkv(hidden_states).reshape(seq_length, 3, self.num_heads, -1).permute(1, 0, 2, 3).unbind(0)
+        q = apply_rotary_pos_emb_vision(q.unsqueeze(0), rotary_pos_emb).squeeze(0)
+        k = apply_rotary_pos_emb_vision(k.unsqueeze(0), rotary_pos_emb).squeeze(0)
+        attention_mask = torch.zeros([1, seq_length, seq_length], device=q.device, dtype=torch.bool)
+        for i in range(1, len(cu_seqlens)):
+            attention_mask[..., cu_seqlens[i - 1]: cu_seqlens[i], cu_seqlens[i - 1]: cu_seqlens[i]] = True
+        q = q.transpose(0, 1).unsqueeze(0)
+        k = k.transpose(0, 1).unsqueeze(0)
+        v = v.transpose(0, 1).unsqueeze(0)
+        if attention_mask.stride(-1) != 1:
+            attention_mask = torch.empty_like(attention_mask, memory_format=torch.contiguous_format).copy_(attention_mask)
+        from torch.nn.attention import SDPBackend, sdpa_kernel
+        with sdpa_kernel(SDPBackend.EFFICIENT_ATTENTION):
+            attn_output = F.scaled_dot_product_attention(q, k, v, attention_mask, dropout_p=0.0)
+        attn_output = attn_output.squeeze(0).transpose(0, 1)
+        attn_output = attn_output.reshape(seq_length, -1)
+        attn_output = self.proj(attn_output)
+        return attn_output
+VISION_ATTENTION_CLASSES = {
+    "eager": VisionAttention,
+    "eager_v2": VisionAttentionV2,
+    "flash_attention_2": VisionFlashAttention2,
+    "sdpa": VisionSdpaAttention,
+    "ascend_fa": VisionAscendAttention,
+}
+class RMSNorm(nn.Module):
+    def __init__(self, dim: int, eps: float = 1e-6):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(dim))
+        self.eps = eps
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        output = self._norm(x.float()).type_as(x)
+        return output * self.weight
+    def extra_repr(self) -> str:
+        return f"{tuple(self.weight.shape)}, eps={self.eps}"
+    def _norm(self, x: torch.Tensor) -> torch.Tensor:
+        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
+class SwiGLUFFN(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        hidden_features = config.intermediate_size
+        in_features = config.embed_dim
+        bias = config.use_bias
+        self.fc1 = nn.Linear(in_features, hidden_features, bias=bias)
+        self.fc2 = nn.Linear(hidden_features, in_features, bias=bias)
+        self.fc3 = nn.Linear(in_features, hidden_features, bias=bias)
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x = F.silu(self.fc1(x)) * self.fc3(x)
+        x = self.fc2(x)
+        return x
+class PatchEmbed(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.num_channels = config.num_channels
+        self.patch_size = config.patch_size
+        self.temporal_patch_size = config.temporal_patch_size
+        self.embed_dim = config.embed_dim
+        self.config = config
+        self.proj = nn.Conv2d(
+            config.num_channels,
+            config.embed_dim,
+            kernel_size=(config.patch_size, config.patch_size),
+            stride=(config.patch_size, config.patch_size),
+        )
+        self.norm = RMSNorm(config.embed_dim, eps=config.rms_norm_eps)
+    def forward(self, x: torch.Tensor, grid_thw=None) -> torch.Tensor:
+        x = x.view(-1, self.num_channels, self.temporal_patch_size, self.patch_size, self.patch_size)[:, :, 0]
+        x = self.proj(x).view(-1, self.embed_dim)
+        x = self.norm(x)
+        return x
+class ViTPreprocessor(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.patch_h = config.patch_size
+        self.patch_w = config.patch_size
+        self.embed_dim = config.embed_dim
+        self.config = config
+        self.patchifier = PatchEmbed(config)
+    def forward(self, x: torch.Tensor, grid_thw=None) -> torch.Tensor:
+        tokens = self.patchifier(x, grid_thw)
+        return tokens
+class VisionBlock(nn.Module):
+    def __init__(self, config, attn_implementation: str = "flash_attention_2"):
+        super().__init__()
+        if attn_implementation == "flash_attention_2" and not flash_attn_available:
+            if npu_available:
+                attn_implementation = "ascend_fa"
+                print("flash attention not available! fallback to ascend flash attention implementation ")
+            else:
+                # fallback to eager
+                attn_implementation = "sdpa"
+                print("flash attention not available! fallback to sdpa implementation ")
+        if attn_implementation == "ascend_fa" and not npu_available:
+            attn_implementation = "sdpa"
+            print("flash attention not available! fallback to sdpa implementation ")
+        self.attn = VISION_ATTENTION_CLASSES[attn_implementation](
+            config, config.embed_dim, num_heads=config.num_attention_heads, bias=config.use_bias
+        )
+        self.norm1 = RMSNorm(config.embed_dim, eps=config.rms_norm_eps)
+        self.mlp = SwiGLUFFN(config)
+        self.norm2 = RMSNorm(config.embed_dim, eps=config.rms_norm_eps)
+    def forward(self, hidden_states, cu_seqlens, rotary_pos_emb) -> torch.Tensor:
+        hidden_states = hidden_states + self.attn(
+            self.norm1(hidden_states), cu_seqlens=cu_seqlens, rotary_pos_emb=rotary_pos_emb
+        )
+        hidden_states = hidden_states + self.mlp(self.norm2(hidden_states))
+        return hidden_states
+class MonkeyOCRv2VisionTransformer(PreTrainedModel):
+    _supports_flash_attn = True
+    _supports_sdpa = True
+    _no_split_modules = ["VisionBlock","PatchMerger"]
+    def __init__(self, config: MonkeyOCRv2VisionConfig) -> None:
+        super().__init__(config)
+        self.config = config
+        self.spatial_merge_size = config.spatial_merge_size
+        self.patch_embed = ViTPreprocessor(config)
+        self._init_weights(self.patch_embed.patchifier.proj)
+        head_dim = config.embed_dim // config.num_attention_heads
+        self.rotary_pos_emb = VisionRotaryEmbedding(head_dim // 2)
+        _num_hidden_layers = config.num_hidden_layers
+        self.blocks = nn.ModuleList(
+            [VisionBlock(config, config.attn_implementation) for _ in range(_num_hidden_layers)]
+        )
+        if self.config.post_norm:
+            self.post_trunk_norm = RMSNorm(config.embed_dim, eps=config.rms_norm_eps)
+        self.merger = PatchMerger(
+            dim=config.hidden_size,
+            context_dim=config.embed_dim,
+            spatial_merge_size=config.spatial_merge_size,
+            init_merger_std=self.config.init_merger_std,
+        )
+        self.gradient_checkpointing = False
+        self._gradient_checkpointing_func = torch.utils.checkpoint.checkpoint
+    def _init_weights(self, module):
+        std = self.config.initializer_range
+        if isinstance(module, (nn.Linear, nn.Conv3d)):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+    @property
+    def dtype(self) -> torch.dtype:
+        return self.blocks[0].mlp.fc2.weight.dtype
+    @property
+    def device(self) -> torch.device:
+        return self.blocks[0].mlp.fc2.weight.device
+    def get_pos_ids_by_grid(self, grid_thw):
+        pos_ids = []
+        for t, h, w in grid_thw:
+            hpos_ids = torch.arange(h).unsqueeze(1).expand(-1, w)
+            hpos_ids = hpos_ids.reshape(
+                h // self.spatial_merge_size,
+                self.spatial_merge_size,
+                w // self.spatial_merge_size,
+                self.spatial_merge_size,
+            )
+            hpos_ids = hpos_ids.permute(0, 2, 1, 3)
+            hpos_ids = hpos_ids.flatten()
+            wpos_ids = torch.arange(w).unsqueeze(0).expand(h, -1)
+            wpos_ids = wpos_ids.reshape(
+                h // self.spatial_merge_size,
+                self.spatial_merge_size,
+                w // self.spatial_merge_size,
+                self.spatial_merge_size,
+            )
+            wpos_ids = wpos_ids.permute(0, 2, 1, 3)
+            wpos_ids = wpos_ids.flatten()
+            pos_ids.append(
+                torch.stack([hpos_ids, wpos_ids], dim=-1).repeat(t, 1)
+            )
+        return pos_ids
+    def rot_pos_emb(self, grid_thw):
+        pos_ids = self.get_pos_ids_by_grid(grid_thw)
+        pos_ids = torch.cat(pos_ids, dim=0)
+        max_grid_size = grid_thw[:, 1:].max()
+        rotary_pos_emb_full = self.rotary_pos_emb(max_grid_size)
+        emb = rotary_pos_emb_full[pos_ids]
+        rotary_pos_emb = torch.stack([emb[:,0], emb[:,1]], dim=2).reshape(emb.shape[0], -1)
+        return rotary_pos_emb
+    def forward(self, hidden_states: torch.Tensor, grid_thw: torch.Tensor, bf16=True) -> torch.Tensor:
+        if bf16:
+            hidden_states = hidden_states.bfloat16()
+        hidden_states = self.patch_embed(hidden_states, grid_thw)
+        rotary_pos_emb = self.rot_pos_emb(grid_thw)
+        cu_seqlens = torch.repeat_interleave(grid_thw[:, 1] * grid_thw[:, 2], grid_thw[:, 0]).cumsum(
+            dim=0,
+            dtype=grid_thw.dtype if torch.jit.is_tracing() else torch.int32,
+        )
+        cu_seqlens = F.pad(cu_seqlens, (1, 0), value=0)
+        for blk in self.blocks:
+            if self.gradient_checkpointing and self.training:
+                hidden_states = self._gradient_checkpointing_func(
+                    blk.__call__,
+                    hidden_states,
+                    cu_seqlens,
+                    rotary_pos_emb,
+                )
+            else:
+                hidden_states = blk(hidden_states, cu_seqlens=cu_seqlens, rotary_pos_emb=rotary_pos_emb)
+        if self.config.post_norm:
+            hidden_states = self.post_trunk_norm(hidden_states)
+        final_hidden_states = self.merger(hidden_states)
+        return final_hidden_states, hidden_states

preprocessor1.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d243b62c3a14bd854539a3b6eebbf69b7716f38f9c7bfe0270d74b2b3e702b55
+size 4823415

preprocessor2.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:27712ba877dfdf325d22ddb9f79467c7998da7e57b9d6fcc4b96b7722ac6b505
+size 288533650

preprocessor_config.json ADDED Viewed

	@@ -0,0 +1,22 @@

+{
+  "auto_map": {
+        "AutoProcessor": "configuration_monkeyocrv2.MonkeyOCRv2Processor"
+    },
+  "min_pixels": 3136,
+  "max_pixels": 11289600,
+  "patch_size": 14,
+  "temporal_patch_size": 1,
+  "merge_size": 2,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "image_processor_type": "Qwen2VLImageProcessor",
+  "processor_class": "MonkeyOCRv2Processor"
+}

processor_config.json ADDED Viewed

	@@ -0,0 +1,6 @@

+{
+  "auto_map": {
+    "AutoProcessor": "configuration_monkeyocrv2.MonkeyOCRv2Processor"
+  },
+  "processor_class": "MonkeyOCRv2Processor"
+}

special_tokens_map.json ADDED Viewed

	@@ -0,0 +1,31 @@

+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4
+size 11422654

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,243 @@

+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "auto_map": {
+    "AutoProcessor": "configuration_monkeyocrv2.MonkeyOCRv2Processor"
+  },
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "processor_class": "MonkeyOCRv2Processor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

vocab.json ADDED Viewed

The diff for this file is too large to render. See raw diff