ocr-projects-archive / logs /smoke_train.log
prosperious's picture
Add 323 files (ocr-tech-mps-integration/tools/prepare_native_spotcheck_bundle.py ...)
0cfd653 verified
Raw History Blame Contribute Delete
2.84 kB
ctc cpu=24.265554 cuda=24.265553 |dloss|=1.91e-06 max|dgrad|=1.04e-06
Loading weights: 0%| | 0/1035 [00:00<?, ?it/s] Loading weights: 58%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š | 599/1035 [00:00<00:00, 5979.00it/s] Loading weights: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1035/1035 [00:00<00:00, 5674.35it/s]
grad-accum: max |full - 2x8| / max |full| = 1.90e-07
{
"real_kept": 4000,
"hw_kept": 2000,
"real_dev_kept": 300,
"hw_dev_kept": 300
}
reading image sizes...
Loading weights: 0%| | 0/1035 [00:00<?, ?it/s] Loading weights: 76%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ | 782/1035 [00:00<00:00, 7817.79it/s] Loading weights: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1035/1035 [00:00<00:00, 7015.63it/s]
device=cuda train_items=6000 steps/epoch=93 total=93
{"step": 0, "dev_mean_cp_cer": 0.3183555169212171, "target_mean_cp_cer": 0.6313131313131313, "eligible": true, "guard_regressions": {}, "real_dev_cer_codepoint": 0.0053979025293029, "real_dev_cer_grapheme": 0.008409785932721712, "real_dev_exact_line_acc": 0.9366666666666666, "hw_dev_cer_codepoint": 0.6313131313131313, "hw_dev_cer_grapheme": 0.8311425682507584, "hw_dev_exact_line_acc": 0.03333333333333333}
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s] Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 13.20it/s]
[W1001 19:07:31.645636384 CUDACachingAllocator.cpp:3934] memory allocation failed with OOM on device 0 while trying to allocate 65011712 bytes (free: 65011712, total: 15640231936).
epoch 0 step 50/93 loss 0.3706 lr 3.19e-05 50 img/s
{"step": 50, "dev_mean_cp_cer": 0.15958604137197596, "target_mean_cp_cer": 0.30976430976430974, "eligible": false, "guard_regressions": {"real_dev": 0.00401}, "real_dev_cer_codepoint": 0.009407772979642195, "real_dev_cer_grapheme": 0.015035677879714576, "real_dev_exact_line_acc": 0.8666666666666667, "hw_dev_cer_codepoint": 0.30976430976430974, "hw_dev_cer_grapheme": 0.4368048533872599, "hw_dev_exact_line_acc": 0.21666666666666667}
step 50: not eligible as best, guard sets regressed: {'real_dev': 0.00401}
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s] Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 12.78it/s]
{"step": 93, "dev_mean_cp_cer": 0.14468075089090923, "target_mean_cp_cer": 0.2822671156004489, "eligible": true, "guard_regressions": {}, "real_dev_cer_codepoint": 0.007094386181369525, "real_dev_cer_grapheme": 0.010703363914373088, "real_dev_exact_line_acc": 0.91, "hw_dev_cer_codepoint": 0.2822671156004489, "hw_dev_cer_grapheme": 0.40040444893832156, "hw_dev_exact_line_acc": 0.26666666666666666}
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s] Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 12.23it/s]
done: best target mean cp CER 0.2823 (targets ['hw_dev'], guards ['real_dev'])
SMOKE_DONE