/workspace/bina02-ppocr/.venv/lib/python3.12/site-packages/paddle/utils/cpp_extension/extension_utils.py:712: UserWarning: No ccache found. Please be aware that recompiling all source files may be required. You can download and install ccache from: https://github.com/ccache/ccache/blob/master/doc/INSTALL.md warnings.warn(warning_message) LAUNCH INFO 2026-07-25 18:12:21,479 ----------- Configuration ---------------------- LAUNCH INFO 2026-07-25 18:12:21,479 auto_cluster_config: 0 LAUNCH INFO 2026-07-25 18:12:21,479 auto_parallel_config: None LAUNCH INFO 2026-07-25 18:12:21,479 auto_tuner_json: None LAUNCH INFO 2026-07-25 18:12:21,479 devices: 0,1,2,3,4,5,6,7 LAUNCH INFO 2026-07-25 18:12:21,479 elastic_level: -1 LAUNCH INFO 2026-07-25 18:12:21,479 elastic_timeout: 30 LAUNCH INFO 2026-07-25 18:12:21,479 enable_gpu_log: True LAUNCH INFO 2026-07-25 18:12:21,479 gloo_port: 6767 LAUNCH INFO 2026-07-25 18:12:21,479 host: None LAUNCH INFO 2026-07-25 18:12:21,479 ips: None LAUNCH INFO 2026-07-25 18:12:21,479 job_id: default LAUNCH INFO 2026-07-25 18:12:21,479 legacy: False LAUNCH INFO 2026-07-25 18:12:21,479 log_dir: /workspace/bina02-ppocr/training/stage-005/distributed-logs LAUNCH INFO 2026-07-25 18:12:21,479 log_level: INFO LAUNCH INFO 2026-07-25 18:12:21,479 log_overwrite: False LAUNCH INFO 2026-07-25 18:12:21,479 master: None LAUNCH INFO 2026-07-25 18:12:21,479 max_restart: 3 LAUNCH INFO 2026-07-25 18:12:21,479 nnodes: 1 LAUNCH INFO 2026-07-25 18:12:21,479 nproc_per_node: None LAUNCH INFO 2026-07-25 18:12:21,479 rank: -1 LAUNCH INFO 2026-07-25 18:12:21,479 run_mode: collective LAUNCH INFO 2026-07-25 18:12:21,479 server_num: None LAUNCH INFO 2026-07-25 18:12:21,479 servers: LAUNCH INFO 2026-07-25 18:12:21,479 sort_ip: False LAUNCH INFO 2026-07-25 18:12:21,479 start_port: 6070 LAUNCH INFO 2026-07-25 18:12:21,479 trainer_num: None LAUNCH INFO 2026-07-25 18:12:21,479 trainers: LAUNCH INFO 2026-07-25 18:12:21,479 training_script: /workspace/bina02-ppocr/PaddleOCR/tools/train.py LAUNCH INFO 2026-07-25 18:12:21,479 training_script_args: ['-c', '/workspace/bina02-ppocr/training/stage-005/input_config.yml'] LAUNCH INFO 2026-07-25 18:12:21,479 with_gloo: 1 LAUNCH INFO 2026-07-25 18:12:21,479 -------------------------------------------------- LAUNCH INFO 2026-07-25 18:12:21,480 Job: default, mode collective, replicas 1[1:1], elastic False LAUNCH INFO 2026-07-25 18:12:21,482 Run Pod: qnbrrq, replicas 8, status ready LAUNCH INFO 2026-07-25 18:12:21,735 Watching Pod: qnbrrq, replicas 8, status running /workspace/bina02-ppocr/.venv/lib/python3.12/site-packages/paddle/utils/cpp_extension/extension_utils.py:712: UserWarning: No ccache found. Please be aware that recompiling all source files may be required. You can download and install ccache from: https://github.com/ccache/ccache/blob/master/doc/INSTALL.md warnings.warn(warning_message) Skipping import of the encryption module. [2026/07/25 18:12:29] ppocr INFO: Architecture : [2026/07/25 18:12:29] ppocr INFO: Backbone : [2026/07/25 18:12:29] ppocr INFO: force_pool_width : True [2026/07/25 18:12:29] ppocr INFO: model_size : medium [2026/07/25 18:12:29] ppocr INFO: name : PPLCNetV4 [2026/07/25 18:12:29] ppocr INFO: train_pool_width : 96 [2026/07/25 18:12:29] ppocr INFO: Head : [2026/07/25 18:12:29] ppocr INFO: head_list : [2026/07/25 18:12:29] ppocr INFO: CTCHead : [2026/07/25 18:12:29] ppocr INFO: Head : [2026/07/25 18:12:29] ppocr INFO: fc_decay : 1e-05 [2026/07/25 18:12:29] ppocr INFO: Neck : [2026/07/25 18:12:29] ppocr INFO: depth : 2 [2026/07/25 18:12:29] ppocr INFO: dims : 192 [2026/07/25 18:12:29] ppocr INFO: local_kernel : 7 [2026/07/25 18:12:29] ppocr INFO: mlp_ratio : 4.0 [2026/07/25 18:12:29] ppocr INFO: name : lightsvtr [2026/07/25 18:12:29] ppocr INFO: use_guide : False [2026/07/25 18:12:29] ppocr INFO: NRTRHead : [2026/07/25 18:12:29] ppocr INFO: max_text_length : 128 [2026/07/25 18:12:29] ppocr INFO: nrtr_dim : 512 [2026/07/25 18:12:29] ppocr INFO: name : MultiHead [2026/07/25 18:12:29] ppocr INFO: Transform : None [2026/07/25 18:12:29] ppocr INFO: algorithm : SVTR_LCNet [2026/07/25 18:12:29] ppocr INFO: model_type : rec [2026/07/25 18:12:29] ppocr INFO: Eval : [2026/07/25 18:12:29] ppocr INFO: dataset : [2026/07/25 18:12:29] ppocr INFO: data_dir : / [2026/07/25 18:12:29] ppocr INFO: label_file_list : ['/workspace/bina02-ppocr/stream/datasets/chunk-005/eval_ctc.txt'] [2026/07/25 18:12:29] ppocr INFO: name : SimpleDataSet [2026/07/25 18:12:29] ppocr INFO: transforms : [2026/07/25 18:12:29] ppocr INFO: DecodeImage : [2026/07/25 18:12:29] ppocr INFO: channel_first : False [2026/07/25 18:12:29] ppocr INFO: img_mode : BGR [2026/07/25 18:12:29] ppocr INFO: MultiLabelEncode : [2026/07/25 18:12:29] ppocr INFO: gtc_encode : NRTRLabelEncode [2026/07/25 18:12:29] ppocr INFO: RecResizeImg : [2026/07/25 18:12:29] ppocr INFO: image_shape : [3, 48, 768] [2026/07/25 18:12:29] ppocr INFO: padding : False [2026/07/25 18:12:29] ppocr INFO: KeepKeys : [2026/07/25 18:12:29] ppocr INFO: keep_keys : ['image', 'label_ctc', 'label_gtc', 'length', 'valid_ratio'] [2026/07/25 18:12:29] ppocr INFO: loader : [2026/07/25 18:12:29] ppocr INFO: batch_size_per_card : 32 [2026/07/25 18:12:29] ppocr INFO: drop_last : False [2026/07/25 18:12:29] ppocr INFO: num_workers : 4 [2026/07/25 18:12:29] ppocr INFO: shuffle : False [2026/07/25 18:12:29] ppocr INFO: Global : [2026/07/25 18:12:29] ppocr INFO: cal_metric_during_train : False [2026/07/25 18:12:29] ppocr INFO: character_dict_path : /workspace/bina02-ppocr/stream/datasets/chunk-005/persian_arabic_bina02_dict.txt [2026/07/25 18:12:29] ppocr INFO: checkpoints : None [2026/07/25 18:12:29] ppocr INFO: d2s_train_image_shape : [3, 48, 768] [2026/07/25 18:12:29] ppocr INFO: debug : False [2026/07/25 18:12:29] ppocr INFO: distributed : True [2026/07/25 18:12:29] ppocr INFO: epoch_num : 1 [2026/07/25 18:12:29] ppocr INFO: eval_batch_step : [0, 100] [2026/07/25 18:12:29] ppocr INFO: freeze_batch_norm : True [2026/07/25 18:12:29] ppocr INFO: infer_img : doc/imgs_words/ch/word_1.jpg [2026/07/25 18:12:29] ppocr INFO: infer_mode : False [2026/07/25 18:12:29] ppocr INFO: log_smooth_window : 20 [2026/07/25 18:12:29] ppocr INFO: max_text_length : 128 [2026/07/25 18:12:29] ppocr INFO: model_name : PP-OCRv6_medium_rec [2026/07/25 18:12:29] ppocr INFO: pretrained_model : /workspace/bina02-ppocr/training/stage-004/best_accuracy [2026/07/25 18:12:29] ppocr INFO: print_batch_step : 20 [2026/07/25 18:12:29] ppocr INFO: save_epoch_step : 1 [2026/07/25 18:12:29] ppocr INFO: save_inference_dir : None [2026/07/25 18:12:29] ppocr INFO: save_model_dir : /workspace/bina02-ppocr/training/stage-005 [2026/07/25 18:12:29] ppocr INFO: save_res_path : /workspace/bina02-ppocr/training/stage-005/predicts.txt [2026/07/25 18:12:29] ppocr INFO: trainable_parameter_prefixes : ['head.ctc_head.fc.'] [2026/07/25 18:12:29] ppocr INFO: use_gpu : True [2026/07/25 18:12:29] ppocr INFO: use_space_char : True [2026/07/25 18:12:29] ppocr INFO: use_visualdl : False [2026/07/25 18:12:29] ppocr INFO: Loss : [2026/07/25 18:12:29] ppocr INFO: loss_config_list : [2026/07/25 18:12:29] ppocr INFO: CTCLoss : None [2026/07/25 18:12:29] ppocr INFO: NRTRLoss : None [2026/07/25 18:12:29] ppocr INFO: name : MultiLoss [2026/07/25 18:12:29] ppocr INFO: weight_1 : 1.0 [2026/07/25 18:12:29] ppocr INFO: weight_2 : 0.0 [2026/07/25 18:12:29] ppocr INFO: Metric : [2026/07/25 18:12:29] ppocr INFO: ignore_space : False [2026/07/25 18:12:29] ppocr INFO: main_indicator : norm_edit_dis [2026/07/25 18:12:29] ppocr INFO: name : RecMetric [2026/07/25 18:12:29] ppocr INFO: Optimizer : [2026/07/25 18:12:29] ppocr INFO: beta1 : 0.9 [2026/07/25 18:12:29] ppocr INFO: beta2 : 0.999 [2026/07/25 18:12:29] ppocr INFO: lr : [2026/07/25 18:12:29] ppocr INFO: learning_rate : 1e-05 [2026/07/25 18:12:29] ppocr INFO: name : Cosine [2026/07/25 18:12:29] ppocr INFO: warmup_epoch : 0 [2026/07/25 18:12:29] ppocr INFO: name : Adam [2026/07/25 18:12:29] ppocr INFO: regularizer : [2026/07/25 18:12:29] ppocr INFO: factor : 0.0 [2026/07/25 18:12:29] ppocr INFO: name : L2 [2026/07/25 18:12:29] ppocr INFO: PostProcess : [2026/07/25 18:12:29] ppocr INFO: name : CTCLabelDecode [2026/07/25 18:12:29] ppocr INFO: Train : [2026/07/25 18:12:29] ppocr INFO: dataset : [2026/07/25 18:12:29] ppocr INFO: data_dir : / [2026/07/25 18:12:29] ppocr INFO: ds_width : False [2026/07/25 18:12:29] ppocr INFO: ext_op_transform_idx : 1 [2026/07/25 18:12:29] ppocr INFO: label_file_list : ['/workspace/bina02-ppocr/stream/datasets/chunk-005/train_ctc.txt'] [2026/07/25 18:12:29] ppocr INFO: name : MultiScaleDataSet [2026/07/25 18:12:29] ppocr INFO: padding : False [2026/07/25 18:12:29] ppocr INFO: ratio_list : [1.0] [2026/07/25 18:12:29] ppocr INFO: transforms : [2026/07/25 18:12:29] ppocr INFO: DecodeImage : [2026/07/25 18:12:29] ppocr INFO: channel_first : False [2026/07/25 18:12:29] ppocr INFO: img_mode : BGR [2026/07/25 18:12:29] ppocr INFO: RecConAug : [2026/07/25 18:12:29] ppocr INFO: ext_data_num : 0 [2026/07/25 18:12:29] ppocr INFO: image_shape : [48, 768, 3] [2026/07/25 18:12:29] ppocr INFO: max_text_length : 128 [2026/07/25 18:12:29] ppocr INFO: prob : 0.0 [2026/07/25 18:12:29] ppocr INFO: RecAug : None [2026/07/25 18:12:29] ppocr INFO: MultiLabelEncode : [2026/07/25 18:12:29] ppocr INFO: gtc_encode : NRTRLabelEncode [2026/07/25 18:12:29] ppocr INFO: KeepKeys : [2026/07/25 18:12:29] ppocr INFO: keep_keys : ['image', 'label_ctc', 'label_gtc', 'length', 'valid_ratio'] [2026/07/25 18:12:29] ppocr INFO: loader : [2026/07/25 18:12:29] ppocr INFO: batch_size_per_card : 32 [2026/07/25 18:12:29] ppocr INFO: drop_last : False [2026/07/25 18:12:29] ppocr INFO: num_workers : 4 [2026/07/25 18:12:29] ppocr INFO: shuffle : True [2026/07/25 18:12:29] ppocr INFO: sampler : [2026/07/25 18:12:29] ppocr INFO: divided_factor : [8, 16] [2026/07/25 18:12:29] ppocr INFO: first_bs : 32 [2026/07/25 18:12:29] ppocr INFO: fix_bs : True [2026/07/25 18:12:29] ppocr INFO: is_training : True [2026/07/25 18:12:29] ppocr INFO: name : MultiScaleSampler [2026/07/25 18:12:29] ppocr INFO: scales : [[768, 48]] [2026/07/25 18:12:29] ppocr INFO: profiler_options : None [2026/07/25 18:12:29] ppocr INFO: train with paddle 3.3.0 and device Place(gpu:0) ======================= Modified FLAGS detected ======================= FLAGS(name='FLAGS_curand_dir', current_value='/workspace/bina02-ppocr/.venv/lib/python3.12/site-packages/paddle/../nvidia/curand/lib', default_value='') FLAGS(name='FLAGS_cudnn_deterministic', current_value=True, default_value=False) FLAGS(name='FLAGS_selected_gpus', current_value='0', default_value='') FLAGS(name='FLAGS_cuda_cccl_dir', current_value='/workspace/bina02-ppocr/.venv/lib/python3.12/site-packages/paddle/../nvidia/cuda_cccl/include/', default_value='') FLAGS(name='FLAGS_nccl_dir', current_value='/workspace/bina02-ppocr/.venv/lib/python3.12/site-packages/paddle/../nvidia/nccl/lib', default_value='') FLAGS(name='FLAGS_cudnn_dir', current_value='/workspace/bina02-ppocr/.venv/lib/python3.12/site-packages/paddle/../nvidia/cudnn/lib', default_value='') FLAGS(name='FLAGS_cusparse_dir', current_value='/workspace/bina02-ppocr/.venv/lib/python3.12/site-packages/paddle/../nvidia/cusparse/lib', default_value='') FLAGS(name='FLAGS_cusolver_dir', current_value='/workspace/bina02-ppocr/.venv/lib/python3.12/site-packages/paddle/../nvidia/cusolver/lib', default_value='') FLAGS(name='FLAGS_enable_pir_in_executor', current_value=True, default_value=False) FLAGS(name='FLAGS_nvidia_package_dir', current_value='/workspace/bina02-ppocr/.venv/lib/python3.12/site-packages/paddle/../nvidia', default_value='') FLAGS(name='FLAGS_cupti_dir', current_value='/workspace/bina02-ppocr/.venv/lib/python3.12/site-packages/paddle/../nvidia/cuda_cupti/lib', default_value='') FLAGS(name='FLAGS_cublas_dir', current_value='/workspace/bina02-ppocr/.venv/lib/python3.12/site-packages/paddle/../nvidia/cublas/lib', default_value='') ======================================================================= I0725 18:12:29.193190 115569 tcp_store.cc:336] create libuv server at port: 53903 I0725 18:12:29.193780 115569 tcp_utils.cc:132] Successfully connected to 172.17.0.2:53903 I0725 18:12:30.652879 115569 process_group_nccl.cc:155] ProcessGroupNCCL pg_timeout_ 1800000 I0725 18:12:30.652935 115569 process_group_nccl.cc:156] ProcessGroupNCCL nccl_comm_init_option_ 0 [2026/07/25 18:12:30] ppocr INFO: Initialize indexs of datasets:['/workspace/bina02-ppocr/stream/datasets/chunk-005/train_ctc.txt'] [2026/07/25 18:12:30] ppocr INFO: Initialize indexs of datasets:['/workspace/bina02-ppocr/stream/datasets/chunk-005/eval_ctc.txt'] W0725 18:12:30.851824 115569 gpu_resources.cc:116] Please NOTE: device: 0, GPU Compute Capability: 12.0, Driver API Version: 13.0, Runtime API Version: 13.0 [2026/07/25 18:12:31] ppocr INFO: froze 61 BatchNorm layers with pretrained global statistics [2026/07/25 18:12:31] ppocr INFO: selective training enabled: 2 trainable tensors, 438 frozen tensors; head.ctc_head.fc.weight, head.ctc_head.fc.bias [2026-07-25 18:12:31,243] [ INFO] optimizer.py:254 - If regularizer of a Parameter has been set by 'paddle.ParamAttr' or 'static.WeightNormParamAttr' already. The weight_decay[0.0] in Optimizer will not take effect, and it will only be applied to other Parameters! [2026/07/25 18:12:31] ppocr INFO: train dataloader has 391 iters [2026/07/25 18:12:31] ppocr INFO: valid dataloader has 128 iters [2026/07/25 18:12:31] ppocr INFO: load pretrain successful from /workspace/bina02-ppocr/training/stage-004/best_accuracy [2026/07/25 18:12:32] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 100 iterations [2026/07/25 18:12:37] ppocr INFO: epoch: [1/1], global_step: 20, lr: 0.000010, CTCLoss: 57.700485, NRTRLoss: 0.000000, loss: 57.700485, avg_reader_cost: 0.09118 s, avg_batch_cost: 0.20650 s, avg_samples: 32.0, ips: 154.96476 samples/s, eta: 0:01:16, max_mem_reserved: 1045 MB, max_mem_allocated: 656 MB [2026/07/25 18:12:39] ppocr INFO: epoch: [1/1], global_step: 40, lr: 0.000010, CTCLoss: 58.909187, NRTRLoss: 0.000000, loss: 58.909187, avg_reader_cost: 0.00424 s, avg_batch_cost: 0.08007 s, avg_samples: 32.0, ips: 399.66937 samples/s, eta: 0:00:50, max_mem_reserved: 1045 MB, max_mem_allocated: 656 MB [2026/07/25 18:12:42] ppocr INFO: epoch: [1/1], global_step: 60, lr: 0.000010, CTCLoss: 60.158285, NRTRLoss: 0.000000, loss: 60.158285, avg_reader_cost: 0.00172 s, avg_batch_cost: 0.08121 s, avg_samples: 32.0, ips: 394.03815 samples/s, eta: 0:00:40, max_mem_reserved: 1045 MB, max_mem_allocated: 656 MB [2026/07/25 18:12:45] ppocr INFO: epoch: [1/1], global_step: 80, lr: 0.000009, CTCLoss: 61.277010, NRTRLoss: 0.000000, loss: 61.277010, avg_reader_cost: 0.00168 s, avg_batch_cost: 0.08314 s, avg_samples: 32.0, ips: 384.90298 samples/s, eta: 0:00:35, max_mem_reserved: 1045 MB, max_mem_allocated: 656 MB [2026/07/25 18:12:47] ppocr INFO: epoch: [1/1], global_step: 100, lr: 0.000009, CTCLoss: 54.436443, NRTRLoss: 0.000000, loss: 54.436443, avg_reader_cost: 0.00166 s, avg_batch_cost: 0.08089 s, avg_samples: 32.0, ips: 395.58923 samples/s, eta: 0:00:30, max_mem_reserved: 1045 MB, max_mem_allocated: 656 MB eval model:: 0%| | 0/128 [00:00