13:4: not a valid test operator: (
13:4: not a valid test operator: 535.86.10
2026-04-28 04:33:01.866542: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /workspace/finetune/main_chars_lstm.py:36: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

[train] params: {"batch_size": 128, "buffer": 2000, "char_lstm_size": 25, "chars": "/workspace/finetune/data_kvkk_set2_v3/vocab.chars.txt", "dim": 50, "dim_chars": 100, "dropout": 0.5, "early_stop_max_steps": 600, "epochs": 20, "learning_rate": 0.001, "log_step_count_steps": 200, "lstm_size": 100, "min_steps": 600000, "num_oov_buckets": 1, "save_checkpoints_secs": 500, "save_summary_steps": 1000, "tags": "/workspace/finetune/data_kvkk_set2_v3/vocab.tags.txt", "trainable_embeddings": true, "vectors": "/workspace/finetune/data_kvkk_set2_v3/vectors.npz", "words": "/workspace/finetune/data_kvkk_set2_v3/vocab.words.txt"}
Using config: {'_model_dir': '/workspace/finetune/results/model', '_tf_random_seed': None, '_save_summary_steps': 1000, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 500, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 200, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fe07b876190>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
Not using Distribute Coordinator.
Running training and evaluation locally (non-distributed).
Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 500.
Calling model_fn.

The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable  TF_ALLOW_IOLIBS=1.
TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable  TF_ALLOW_IOLIBS=1.
TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable  TF_ALLOW_IOLIBS=1.
From /workspace/finetune/main_chars_lstm.py:104: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

From /workspace/finetune/main_chars_lstm.py:169: The name tf.metrics.accuracy is deprecated. Please use tf.compat.v1.metrics.accuracy instead.

From /py_packages/tf_metrics/__init__.py:152: The name tf.diag_part is deprecated. Please use tf.linalg.tensor_diag_part instead.

From /workspace/finetune/main_chars_lstm.py:175: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

From /workspace/finetune/main_chars_lstm.py:180: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

From /workspace/finetune/main_chars_lstm.py:181: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

Done calling model_fn.
Create CheckpointSaverHook.
Graph was finalized.
2026-04-28 04:33:05.296490: I tensorflow/core/platform/profile_utils/cpu_utils.cc:109] CPU Frequency: 2000000000 Hz
2026-04-28 04:33:05.328827: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x64d0070 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2026-04-28 04:33:05.328873: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2026-04-28 04:33:05.333954: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2026-04-28 04:33:05.523685: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x660a6c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2026-04-28 04:33:05.523735: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA H100, Compute Capability 9.0
2026-04-28 04:33:05.524596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1669] Found device 0 with properties: 
name: NVIDIA H100 major: 9 minor: 0 memoryClockRate(GHz): 1.98
pciBusID: 0000:ad:00.0
2026-04-28 04:33:05.524637: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-04-28 04:33:05.859509: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-04-28 04:33:05.891835: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-04-28 04:33:05.899990: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-04-28 04:33:05.908493: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2026-04-28 04:33:05.920724: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-04-28 04:33:05.922071: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-04-28 04:33:05.922505: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1797] Adding visible gpu devices: 0
2026-04-28 04:33:05.923841: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-04-28 04:33:05.929938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1209] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-04-28 04:33:05.929961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1215]      0 
2026-04-28 04:33:05.929969: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1228] 0:   N 
2026-04-28 04:33:05.930412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1354] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 60325 MB memory) -> physical GPU (device: 0, name: NVIDIA H100, pci bus id: 0000:ad:00.0, compute capability: 9.0)
Running local_init_op.
Done running local_init_op.
Saving checkpoints for 0 into /workspace/finetune/results/model/model.ckpt.
2026-04-28 04:33:09.576112: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
loss = 117.38817, step = 0
global_step/sec: 15.9262
loss = 4.773464, step = 200 (12.558 sec)
global_step/sec: 16.4469
loss = 2.8829772, step = 400 (12.160 sec)
global_step/sec: 16.4739
loss = 1.6619699, step = 600 (12.140 sec)
global_step/sec: 17.5305
loss = 1.4286156, step = 800 (11.409 sec)
global_step/sec: 16.591
loss = 1.1506453, step = 1000 (12.055 sec)
global_step/sec: 16.3338
loss = 1.6229303, step = 1200 (12.245 sec)
global_step/sec: 16.319
loss = 1.1030784, step = 1400 (12.256 sec)
global_step/sec: 16.0311
loss = 0.48492754, step = 1600 (12.476 sec)
global_step/sec: 15.9238
loss = 0.7973819, step = 1800 (12.560 sec)
global_step/sec: 15.9882
loss = 0.5444262, step = 2000 (12.510 sec)
global_step/sec: 16.0443
loss = 0.34822357, step = 2200 (12.465 sec)
global_step/sec: 16.0029
loss = 0.46935606, step = 2400 (12.498 sec)
global_step/sec: 15.9425
loss = 0.5678671, step = 2600 (12.548 sec)
global_step/sec: 16.3265
loss = 0.5309495, step = 2800 (12.247 sec)
global_step/sec: 16.3088
loss = 0.7498473, step = 3000 (12.264 sec)
global_step/sec: 15.5706
loss = 0.6110069, step = 3200 (12.844 sec)
global_step/sec: 16.7051
loss = 0.24717766, step = 3400 (11.973 sec)
global_step/sec: 16.8543
loss = 1.3274518, step = 3600 (11.866 sec)
global_step/sec: 16.4027
loss = 0.33537441, step = 3800 (12.193 sec)
global_step/sec: 16.2972
loss = 0.29465103, step = 4000 (12.272 sec)
global_step/sec: 16.6392
loss = 0.2784903, step = 4200 (12.020 sec)
global_step/sec: 16.6742
loss = 0.2184174, step = 4400 (11.994 sec)
global_step/sec: 16.755
loss = 0.44008517, step = 4600 (11.937 sec)
global_step/sec: 17.096
loss = 0.25767207, step = 4800 (11.698 sec)
global_step/sec: 16.899
loss = 0.17028493, step = 5000 (11.835 sec)
global_step/sec: 16.7529
loss = 0.17947096, step = 5200 (11.938 sec)
global_step/sec: 17.0098
loss = 0.28755808, step = 5400 (11.758 sec)
global_step/sec: 17.141
loss = 0.25743997, step = 5600 (11.668 sec)
global_step/sec: 17.1619
loss = 0.44591618, step = 5800 (11.654 sec)
global_step/sec: 17.2099
loss = 0.44130975, step = 6000 (11.621 sec)
global_step/sec: 16.7787
loss = 0.9595022, step = 6200 (11.920 sec)
global_step/sec: 16.5132
loss = 0.17380977, step = 6400 (12.112 sec)
global_step/sec: 17.1766
loss = 0.62790394, step = 6600 (11.644 sec)
global_step/sec: 16.6513
loss = 0.115854025, step = 6800 (12.011 sec)
global_step/sec: 16.8948
loss = 0.36853623, step = 7000 (11.839 sec)
global_step/sec: 16.8772
loss = 0.15148377, step = 7200 (11.850 sec)
global_step/sec: 17.1629
loss = 0.23943508, step = 7400 (11.653 sec)
global_step/sec: 16.7432
loss = 0.13623583, step = 7600 (11.945 sec)
global_step/sec: 17.1363
loss = 0.13282633, step = 7800 (11.671 sec)
global_step/sec: 17.5276
loss = 0.0774439, step = 8000 (11.413 sec)
global_step/sec: 17.2208
loss = 0.08795112, step = 8200 (11.612 sec)
Saving checkpoints for 8284 into /workspace/finetune/results/model/model.ckpt.
Calling model_fn.
Done calling model_fn.
Starting evaluation at 2026-04-28T04:41:29Z
Graph was finalized.
2026-04-28 04:41:29.885621: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1669] Found device 0 with properties: 
name: NVIDIA H100 major: 9 minor: 0 memoryClockRate(GHz): 1.98
pciBusID: 0000:ad:00.0
2026-04-28 04:41:29.885671: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-04-28 04:41:29.885708: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-04-28 04:41:29.885715: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-04-28 04:41:29.885723: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-04-28 04:41:29.885730: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2026-04-28 04:41:29.885736: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-04-28 04:41:29.885743: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-04-28 04:41:29.886053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1797] Adding visible gpu devices: 0
2026-04-28 04:41:29.886087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1209] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-04-28 04:41:29.886092: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1215]      0 
2026-04-28 04:41:29.886097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1228] 0:   N 
2026-04-28 04:41:29.886396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1354] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 60325 MB memory) -> physical GPU (device: 0, name: NVIDIA H100, pci bus id: 0000:ad:00.0, compute capability: 9.0)
Restoring parameters from /workspace/finetune/results/model/model.ckpt-8284
Running local_init_op.
Done running local_init_op.
Evaluation [10/100]
Evaluation [20/100]
Evaluation [30/100]
Evaluation [40/100]
Evaluation [50/100]
Evaluation [60/100]
Evaluation [70/100]
Evaluation [80/100]
Evaluation [90/100]
Evaluation [100/100]
Finished evaluation at 2026-04-28-04:41:34
Saving dict for global step 8284: acc = 0.99759144, f1 = 0.9933167, global_step = 8284, loss = 0.17885803, precision = 0.99258435, recall = 0.99405015
Saving 'checkpoint_path' summary for global step 8284: /workspace/finetune/results/model/model.ckpt-8284
global_step/sec: 11.4752
loss = 0.24231434, step = 8400 (17.429 sec)
global_step/sec: 17.3567
loss = 0.11804509, step = 8600 (11.523 sec)
global_step/sec: 17.0883
loss = 0.23675942, step = 8800 (11.704 sec)
global_step/sec: 17.4953
loss = 0.14836943, step = 9000 (11.432 sec)
global_step/sec: 17.495
loss = 0.1359005, step = 9200 (11.432 sec)
global_step/sec: 17.082
loss = 0.21677959, step = 9400 (11.709 sec)
global_step/sec: 17.2371
loss = 0.14764589, step = 9600 (11.603 sec)
global_step/sec: 17.2015
loss = 0.09986603, step = 9800 (11.627 sec)
global_step/sec: 17.0483
loss = 0.23195946, step = 10000 (11.732 sec)
global_step/sec: 17.4384
loss = 0.11996138, step = 10200 (11.469 sec)
global_step/sec: 17.4722
loss = 0.077587605, step = 10400 (11.447 sec)
global_step/sec: 17.2922
loss = 0.13140821, step = 10600 (11.566 sec)
global_step/sec: 17.3569
loss = 0.1550746, step = 10800 (11.523 sec)
global_step/sec: 17.3439
loss = 0.16950673, step = 11000 (11.531 sec)
global_step/sec: 17.4338
loss = 0.1085785, step = 11200 (11.472 sec)
global_step/sec: 17.5506
loss = 0.114634454, step = 11400 (11.396 sec)
global_step/sec: 17.7854
loss = 0.09087747, step = 11600 (11.245 sec)
global_step/sec: 17.3965
loss = 0.0742746, step = 11800 (11.497 sec)
global_step/sec: 17.3043
loss = 0.11870754, step = 12000 (11.558 sec)
global_step/sec: 17.6434
loss = 0.15823495, step = 12200 (11.335 sec)
global_step/sec: 17.4929
loss = 0.058683336, step = 12400 (11.433 sec)
global_step/sec: 17.6003
loss = 0.16609168, step = 12600 (11.364 sec)
global_step/sec: 17.6682
loss = 0.2326163, step = 12800 (11.320 sec)
global_step/sec: 17.6474
loss = 0.2841699, step = 13000 (11.333 sec)
global_step/sec: 17.306
loss = 0.17810643, step = 13200 (11.557 sec)
global_step/sec: 17.3409
loss = 0.049505234, step = 13400 (11.533 sec)
global_step/sec: 17.453
loss = 0.04317367, step = 13600 (11.459 sec)
global_step/sec: 17.4567
loss = 0.107658744, step = 13800 (11.457 sec)
global_step/sec: 17.7515
loss = 0.040947437, step = 14000 (11.267 sec)
global_step/sec: 17.5825
loss = 0.07045764, step = 14200 (11.375 sec)
global_step/sec: 17.4036
loss = 0.07488018, step = 14400 (11.492 sec)
global_step/sec: 17.6369
loss = 0.25159824, step = 14600 (11.340 sec)
global_step/sec: 17.7241
loss = 0.08559203, step = 14800 (11.284 sec)
global_step/sec: 17.4513
loss = 0.03275448, step = 15000 (11.461 sec)
global_step/sec: 17.5323
loss = 0.13171089, step = 15200 (11.407 sec)
global_step/sec: 17.6592
loss = 0.047302723, step = 15400 (11.326 sec)
global_step/sec: 17.6405
loss = 0.13909113, step = 15600 (11.338 sec)
global_step/sec: 17.6345
loss = 0.08797115, step = 15800 (11.342 sec)
global_step/sec: 17.8771
loss = 0.08592886, step = 16000 (11.187 sec)
global_step/sec: 17.5025
loss = 0.042692125, step = 16200 (11.427 sec)
global_step/sec: 17.5749
loss = 0.050394714, step = 16400 (11.380 sec)
global_step/sec: 17.6533
loss = 0.050355732, step = 16600 (11.329 sec)
global_step/sec: 17.6026
loss = 0.14212108, step = 16800 (11.362 sec)
Saving checkpoints for 16921 into /workspace/finetune/results/model/model.ckpt.
Calling model_fn.
Done calling model_fn.
Starting evaluation at 2026-04-28T04:49:49Z
Graph was finalized.
2026-04-28 04:49:49.668155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1669] Found device 0 with properties: 
name: NVIDIA H100 major: 9 minor: 0 memoryClockRate(GHz): 1.98
pciBusID: 0000:ad:00.0
2026-04-28 04:49:49.668199: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-04-28 04:49:49.668238: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-04-28 04:49:49.668246: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-04-28 04:49:49.668254: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-04-28 04:49:49.668262: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2026-04-28 04:49:49.668268: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-04-28 04:49:49.668274: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-04-28 04:49:49.668591: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1797] Adding visible gpu devices: 0
2026-04-28 04:49:49.668623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1209] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-04-28 04:49:49.668628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1215]      0 
2026-04-28 04:49:49.668633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1228] 0:   N 
2026-04-28 04:49:49.668917: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1354] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 60325 MB memory) -> physical GPU (device: 0, name: NVIDIA H100, pci bus id: 0000:ad:00.0, compute capability: 9.0)
Restoring parameters from /workspace/finetune/results/model/model.ckpt-16921
Running local_init_op.
Done running local_init_op.
Evaluation [10/100]
Evaluation [20/100]
Evaluation [30/100]
Evaluation [40/100]
Evaluation [50/100]
Evaluation [60/100]
Evaluation [70/100]
Evaluation [80/100]
Evaluation [90/100]
Evaluation [100/100]
Finished evaluation at 2026-04-28-04:49:54
Saving dict for global step 16921: acc = 0.99804306, f1 = 0.9946128, global_step = 16921, loss = 0.13094354, precision = 0.99445856, recall = 0.9947671
Saving 'checkpoint_path' summary for global step 16921: /workspace/finetune/results/model/model.ckpt-16921
global_step/sec: 12.0425
loss = 0.069523394, step = 17000 (16.608 sec)
global_step/sec: 16.8534
loss = 0.03588748, step = 17200 (11.867 sec)
global_step/sec: 17.389
loss = 0.060875297, step = 17400 (11.502 sec)
global_step/sec: 17.5332
loss = 0.093826056, step = 17600 (11.407 sec)
global_step/sec: 17.5162
loss = 0.04980874, step = 17800 (11.418 sec)
global_step/sec: 17.5958
loss = 0.08436108, step = 18000 (11.367 sec)
global_step/sec: 17.6499
loss = 0.07031548, step = 18200 (11.332 sec)
global_step/sec: 17.4897
loss = 0.08018917, step = 18400 (11.435 sec)
global_step/sec: 17.6163
loss = 0.028933227, step = 18600 (11.353 sec)
global_step/sec: 17.3619
loss = 0.06399143, step = 18800 (11.519 sec)
global_step/sec: 17.5151
loss = 0.104281485, step = 19000 (11.419 sec)
global_step/sec: 17.5888
loss = 0.10930133, step = 19200 (11.371 sec)
global_step/sec: 17.6307
loss = 0.056066155, step = 19400 (11.344 sec)
global_step/sec: 17.7029
loss = 0.19223732, step = 19600 (11.298 sec)
global_step/sec: 17.8645
loss = 0.13925654, step = 19800 (11.195 sec)
global_step/sec: 17.6568
loss = 0.05859512, step = 20000 (11.327 sec)
global_step/sec: 17.7123
loss = 0.06567615, step = 20200 (11.291 sec)
global_step/sec: 17.8937
loss = 0.106875956, step = 20400 (11.177 sec)
global_step/sec: 17.6091
loss = 0.03964001, step = 20600 (11.358 sec)
global_step/sec: 17.4414
loss = 0.09287375, step = 20800 (11.467 sec)
global_step/sec: 17.6155
loss = 0.03273791, step = 21000 (11.354 sec)
global_step/sec: 17.8087
loss = 0.0277282, step = 21200 (11.230 sec)
global_step/sec: 17.8569
loss = 0.07149267, step = 21400 (11.200 sec)
global_step/sec: 17.9393
loss = 0.05456364, step = 21600 (11.149 sec)
global_step/sec: 17.7899
loss = 0.011094809, step = 21800 (11.242 sec)
global_step/sec: 17.5774
loss = 0.04235983, step = 22000 (11.378 sec)
global_step/sec: 17.8274
loss = 0.056289375, step = 22200 (11.218 sec)
global_step/sec: 17.8135
loss = 0.06535977, step = 22400 (11.228 sec)
global_step/sec: 17.5839
loss = 0.11343026, step = 22600 (11.374 sec)
global_step/sec: 17.8426
loss = 0.035084844, step = 22800 (11.209 sec)
global_step/sec: 17.9744
loss = 0.02807188, step = 23000 (11.127 sec)
global_step/sec: 17.6131
loss = 0.12693703, step = 23200 (11.355 sec)
global_step/sec: 17.8415
loss = 0.064061224, step = 23400 (11.210 sec)
global_step/sec: 17.7546
loss = 0.090565205, step = 23600 (11.265 sec)
global_step/sec: 17.7205
loss = 0.0683707, step = 23800 (11.286 sec)
global_step/sec: 17.8176
loss = 0.031541705, step = 24000 (11.225 sec)
global_step/sec: 17.906
loss = 0.04308015, step = 24200 (11.169 sec)
global_step/sec: 17.643
loss = 0.12729162, step = 24400 (11.336 sec)
global_step/sec: 17.5792
loss = 0.05107689, step = 24600 (11.377 sec)
global_step/sec: 17.6869
loss = 0.037356377, step = 24800 (11.308 sec)
Saving checkpoints for 25000 into /workspace/finetune/results/model/model.ckpt.
Calling model_fn.
Done calling model_fn.
Starting evaluation at 2026-04-28T04:57:32Z
Graph was finalized.
2026-04-28 04:57:32.870073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1669] Found device 0 with properties: 
name: NVIDIA H100 major: 9 minor: 0 memoryClockRate(GHz): 1.98
pciBusID: 0000:ad:00.0
2026-04-28 04:57:32.870119: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-04-28 04:57:32.870154: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-04-28 04:57:32.870162: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-04-28 04:57:32.870170: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-04-28 04:57:32.870177: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2026-04-28 04:57:32.870184: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-04-28 04:57:32.870191: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-04-28 04:57:32.870457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1797] Adding visible gpu devices: 0
2026-04-28 04:57:32.870492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1209] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-04-28 04:57:32.870497: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1215]      0 
2026-04-28 04:57:32.870502: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1228] 0:   N 
2026-04-28 04:57:32.870790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1354] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 60325 MB memory) -> physical GPU (device: 0, name: NVIDIA H100, pci bus id: 0000:ad:00.0, compute capability: 9.0)
Restoring parameters from /workspace/finetune/results/model/model.ckpt-25000
Running local_init_op.
Done running local_init_op.
Evaluation [10/100]
Evaluation [20/100]
Evaluation [30/100]
Evaluation [40/100]
Evaluation [50/100]
Evaluation [60/100]
Evaluation [70/100]
Evaluation [80/100]
Evaluation [90/100]
Evaluation [100/100]
Finished evaluation at 2026-04-28-04:57:37
Saving dict for global step 25000: acc = 0.99808574, f1 = 0.9946638, global_step = 25000, loss = 0.12328317, precision = 0.9939835, recall = 0.995345
Saving 'checkpoint_path' summary for global step 25000: /workspace/finetune/results/model/model.ckpt-25000
Loss for final step: 0.035551548.
Calling model_fn.
Done calling model_fn.
Graph was finalized.
2026-04-28 04:57:37.865806: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1669] Found device 0 with properties: 
name: NVIDIA H100 major: 9 minor: 0 memoryClockRate(GHz): 1.98
pciBusID: 0000:ad:00.0
2026-04-28 04:57:37.865856: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-04-28 04:57:37.865881: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-04-28 04:57:37.865889: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-04-28 04:57:37.865897: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-04-28 04:57:37.865906: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2026-04-28 04:57:37.865912: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-04-28 04:57:37.865920: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-04-28 04:57:37.866178: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1797] Adding visible gpu devices: 0
2026-04-28 04:57:37.866207: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1209] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-04-28 04:57:37.866212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1215]      0 
2026-04-28 04:57:37.866217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1228] 0:   N 
2026-04-28 04:57:37.866511: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1354] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 60325 MB memory) -> physical GPU (device: 0, name: NVIDIA H100, pci bus id: 0000:ad:00.0, compute capability: 9.0)
Restoring parameters from /workspace/finetune/results/model/model.ckpt-25000
Running local_init_op.
Done running local_init_op.
[predict] wrote /workspace/finetune/results/score/train.preds.txt
Calling model_fn.
Done calling model_fn.
Graph was finalized.
2026-04-28 04:58:27.768805: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1669] Found device 0 with properties: 
name: NVIDIA H100 major: 9 minor: 0 memoryClockRate(GHz): 1.98
pciBusID: 0000:ad:00.0
2026-04-28 04:58:27.768847: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-04-28 04:58:27.768867: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-04-28 04:58:27.768874: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-04-28 04:58:27.768879: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-04-28 04:58:27.768884: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2026-04-28 04:58:27.768889: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-04-28 04:58:27.768895: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-04-28 04:58:27.769123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1797] Adding visible gpu devices: 0
2026-04-28 04:58:27.769147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1209] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-04-28 04:58:27.769151: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1215]      0 
2026-04-28 04:58:27.769155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1228] 0:   N 
2026-04-28 04:58:27.769420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1354] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 60325 MB memory) -> physical GPU (device: 0, name: NVIDIA H100, pci bus id: 0000:ad:00.0, compute capability: 9.0)
Restoring parameters from /workspace/finetune/results/model/model.ckpt-25000
Running local_init_op.
Done running local_init_op.
[predict] wrote /workspace/finetune/results/score/testa.preds.txt
Calling model_fn.
Done calling model_fn.
Graph was finalized.
2026-04-28 04:58:34.490921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1669] Found device 0 with properties: 
name: NVIDIA H100 major: 9 minor: 0 memoryClockRate(GHz): 1.98
pciBusID: 0000:ad:00.0
2026-04-28 04:58:34.490967: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2026-04-28 04:58:34.491002: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2026-04-28 04:58:34.491008: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2026-04-28 04:58:34.491014: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2026-04-28 04:58:34.491020: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2026-04-28 04:58:34.491025: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2026-04-28 04:58:34.491030: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2026-04-28 04:58:34.491264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1797] Adding visible gpu devices: 0
2026-04-28 04:58:34.491287: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1209] Device interconnect StreamExecutor with strength 1 edge matrix:
2026-04-28 04:58:34.491292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1215]      0 
2026-04-28 04:58:34.491296: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1228] 0:   N 
2026-04-28 04:58:34.491562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1354] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 60325 MB memory) -> physical GPU (device: 0, name: NVIDIA H100, pci bus id: 0000:ad:00.0, compute capability: 9.0)
Restoring parameters from /workspace/finetune/results/model/model.ckpt-25000
Running local_init_op.
Done running local_init_op.
[predict] wrote /workspace/finetune/results/score/testb.preds.txt