hyb1: results of w1

results/hyb1/w1/done.txt

@@ -0,0 +1 @@
+done

results/hyb1/w1/logs/auto.log

@@ -0,0 +1,9 @@
+15:58 START w1
+15:58 stockfish ok
+16:01 DATA_DONE
+16:09 PREP FASTPREP_DONE 392086372 540415377
+16:09 RUN hyb1 TOTAL=60000 H=1000 LR=1e-3 OLR=0.7 BS=4096 CK=5 WAIT=300 TAKEOVER=900
+16:35 CRASH hyb1 1 RuntimeError: commit hyb1: r40 w1 failed for good
+16:36 PREP_FIXED castling FASTPREP_DONE 392086372 570983690
+18:08 TRAINED hyb1
+18:26 MATCH hyb1_tm_sffull MATCH_DONE {"A": "gm2b16t", "B": "sffull", "spm": 1.0, "games": 32, "score": 4.0, "pct": 0.125, "elo_diff": -338, "minutes": 18.4, "gm2b16t_sims_per_move": 2482, "gm2b16t_ab_overrides": 0}

results/hyb1/w1/logs/diloco_hyb1.log

@@ -0,0 +1,80 @@
+16:09:41 w1 387,134,719 positions from 922 parts; GPU memory 16.6 GB
+16:10:33 w1 checkpoint at round 35
+16:10:38 w1 replayed up to round 37
+16:16:11 w1 joining at round 38 leader 0
+16:20:54 w1 round 39 step 39000/60000 inner 268s (268 ms/step) sync 12s used [0, 1, 2, 3] lr 2.88e-04 loss [1.2634, 0.5416]
+16:24:36 w1 round 40 step 40000/60000 inner 202s (202 ms/step) sync 18s used [0, 1, 2, 3] lr 2.65e-04 loss [1.2897, 0.5448]
+16:28:41 w1 commit hyb1: r40 w1 failed (BadRequestError: (Request ID: Root=1-6aef51b9-1edc366543acbc86248f3713;763e26dd-fdc2-473f-9421-a359b4210868)
+
+Bad request for commit endpoint:
+Private storage limit re), retry 1
+16:28:44 w1 commit hyb1: r40 w1 failed (BadRequestError: (Request ID: Root=1-6aef51bc-1ea00c40240d514f75d5431a;4ee67a99-0b8f-439f-b8c9-a37d0e146d1c)
+
+Bad request for commit endpoint:
+Private storage limit re), retry 2
+16:28:50 w1 commit hyb1: r40 w1 failed (BadRequestError: (Request ID: Root=1-6aef51c2-268b1b27650237de280b11aa;c2c91df5-11dc-4f4e-a089-626c2a3a89dc)
+
+Bad request for commit endpoint:
+Private storage limit re), retry 3
+16:29:03 w1 commit hyb1: r40 w1 failed (BadRequestError: (Request ID: Root=1-6aef51ce-54d8916739bf7cc5748079bc;8f747d80-6203-4c0f-9c3d-fa002ce0a597)
+
+Bad request for commit endpoint:
+Private storage limit re), retry 4
+16:29:27 w1 commit hyb1: r40 w1 failed (BadRequestError: (Request ID: Root=1-6aef51e7-322e446928fc07f56acceef7;5edd59a2-d492-4776-a1dd-2a135bb92b7b)
+
+Bad request for commit endpoint:
+Private storage limit re), retry 5
+16:30:15 w1 commit hyb1: r40 w1 failed (BadRequestError: (Request ID: Root=1-6aef5217-4cb333d26575658c279867ee;b3f150f2-d8dd-403e-9060-2ca4859a2bf7)
+
+Bad request for commit endpoint:
+Private storage limit re), retry 6
+16:31:15 w1 commit hyb1: r40 w1 failed (BadRequestError: (Request ID: Root=1-6aef5253-27bc44013cc5a5b7531bb2f5;803355df-8599-4e1e-a28a-ae60dc8292c2)
+
+Bad request for commit endpoint:
+Private storage limit re), retry 7
+16:32:16 w1 commit hyb1: r40 w1 failed (BadRequestError: (Request ID: Root=1-6aef528f-250de357447100681cf33d0d;b029367f-5c8a-464e-8dc5-03e6826aa2a3)
+
+Bad request for commit endpoint:
+Private storage limit re), retry 8
+16:33:16 w1 commit hyb1: r40 w1 failed (BadRequestError: (Request ID: Root=1-6aef52cc-338208514af9fc8a6e67478c;99d98822-fc06-4c25-8117-5731a5897dcf)
+
+Bad request for commit endpoint:
+Private storage limit re), retry 9
+16:34:17 w1 commit hyb1: r40 w1 failed (BadRequestError: (Request ID: Root=1-6aef5309-7f5de38f32e512a35e2a0b4c;9813c80d-6c7d-48c0-a7b0-22e41259b467)
+
+Bad request for commit endpoint:
+Private storage limit re), retry 10
+Traceback (most recent call last):
+  File "/marimo/chess/code/diloco2.py", line 293, in <module>
+    put([add(f"{ROOT}/r{r:05d}/w{WID}.pt", blob(own))], f"{run}: r{r} w{WID}")
+    ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/marimo/chess/code/diloco2.py", line 72, in put
+    retry(lambda: api.create_commit(REPO, operations=ops, commit_message=msg), "commit " + msg)
+    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/marimo/chess/code/diloco2.py", line 56, in retry
+    raise RuntimeError(what + " failed for good")
+RuntimeError: commit hyb1: r40 w1 failed for good
+16:36:09 w1 387,134,719 positions from 922 parts; GPU memory 16.6 GB
+16:36:15 w1 checkpoint at round 40
+16:36:15 w1 replayed up to round 40
+16:51:51 w1 joining at round 41 leader 0
+16:55:58 w1 round 42 step 42000/60000 inner 208s (208 ms/step) sync 36s used [0, 1, 2, 3] lr 2.22e-04 loss [1.2685, 0.5457]
+16:59:53 w1 round 43 step 43000/60000 inner 203s (203 ms/step) sync 31s used [0, 1, 2, 3] lr 2.02e-04 loss [1.2405, 0.5405]
+17:03:41 w1 round 44 step 44000/60000 inner 203s (203 ms/step) sync 23s used [0, 1, 2, 3] lr 1.82e-04 loss [1.2369, 0.5421]
+17:07:18 w1 round 45 step 45000/60000 inner 203s (203 ms/step) sync 13s used [0, 1, 2, 3] lr 1.64e-04 loss [1.293, 0.5354]
+17:12:26 w1 round 46 step 46000/60000 inner 203s (203 ms/step) sync 103s used [0, 1, 2, 3] lr 1.46e-04 loss [1.2628, 0.5486]
+17:16:14 w1 round 47 step 47000/60000 inner 203s (203 ms/step) sync 23s used [0, 1, 2, 3] lr 1.29e-04 loss [1.2477, 0.5374]
+17:20:03 w1 round 48 step 48000/60000 inner 203s (203 ms/step) sync 24s used [0, 1, 2, 3] lr 1.14e-04 loss [1.2325, 0.548]
+17:23:58 w1 round 49 step 49000/60000 inner 203s (203 ms/step) sync 31s used [0, 1, 2, 3] lr 9.91e-05 loss [1.2613, 0.5458]
+17:28:32 w1 round 50 step 50000/60000 inner 203s (203 ms/step) sync 69s used [0, 1, 2, 3] lr 8.57e-05 loss [1.2375, 0.5362]
+17:32:53 w1 round 51 step 51000/60000 inner 203s (203 ms/step) sync 56s used [0, 1, 2, 3] lr 7.34e-05 loss [1.2626, 0.5375]
+17:36:35 w1 round 52 step 52000/60000 inner 203s (203 ms/step) sync 18s used [0, 1, 2, 3] lr 6.24e-05 loss [1.2676, 0.5338]
+17:40:24 w1 round 53 step 53000/60000 inner 203s (203 ms/step) sync 24s used [0, 1, 2, 3] lr 5.26e-05 loss [1.2837, 0.5458]
+17:44:47 w1 round 54 step 54000/60000 inner 203s (203 ms/step) sync 58s used [0, 1, 2, 3] lr 4.40e-05 loss [1.2326, 0.5354]
+17:48:39 w1 round 55 step 55000/60000 inner 203s (203 ms/step) sync 27s used [0, 1, 2, 3] lr 3.67e-05 loss [1.2324, 0.5389]
+17:52:29 w1 round 56 step 56000/60000 inner 203s (203 ms/step) sync 26s used [0, 1, 2, 3] lr 3.07e-05 loss [1.2259, 0.5372]
+17:56:28 w1 round 57 step 57000/60000 inner 203s (203 ms/step) sync 34s used [0, 1, 2, 3] lr 2.60e-05 loss [1.277, 0.5396]
+18:00:37 w1 round 58 step 58000/60000 inner 203s (203 ms/step) sync 44s used [0, 1, 2, 3] lr 2.27e-05 loss [1.2907, 0.5458]
+18:04:52 w1 round 59 step 59000/60000 inner 203s (203 ms/step) sync 50s used [0, 1, 2, 3] lr 2.07e-05 loss [1.2606, 0.5362]
+18:08:25 w1 round 60 step 60000/60000 inner 203s (203 ms/step) sync 8s used [0, 1, 2, 3] lr 2.00e-05 loss [1.2591, 0.5357]
+18:08:25 w1 DILOCO_DONE

results/hyb1/w1/logs/fastprep.log

@@ -0,0 +1,20 @@
+1/922 row groups, 177,532 roots, 0 PV positions, 9s
+51/922 row groups, 13,747,406 roots, 16,640,410 PV positions, 25s
+101/922 row groups, 29,537,708 roots, 38,488,567 PV positions, 45s
+151/922 row groups, 46,445,308 roots, 62,056,978 PV positions, 63s
+201/922 row groups, 63,821,148 roots, 84,647,783 PV positions, 83s
+251/922 row groups, 81,922,064 roots, 109,653,900 PV positions, 103s
+301/922 row groups, 100,229,639 roots, 133,993,421 PV positions, 124s
+351/922 row groups, 118,971,326 roots, 158,433,500 PV positions, 145s
+401/922 row groups, 138,519,685 roots, 183,271,592 PV positions, 168s
+451/922 row groups, 169,233,100 roots, 229,349,568 PV positions, 213s
+501/922 row groups, 192,730,796 roots, 261,592,349 PV positions, 240s
+551/922 row groups, 215,576,641 roots, 292,416,127 PV positions, 264s
+601/922 row groups, 238,322,925 roots, 323,057,621 PV positions, 287s
+651/922 row groups, 260,430,465 roots, 355,855,776 PV positions, 311s
+701/922 row groups, 282,766,659 roots, 386,524,792 PV positions, 334s
+751/922 row groups, 305,763,381 roots, 420,364,462 PV positions, 360s
+801/922 row groups, 330,004,960 roots, 452,672,862 PV positions, 387s
+851/922 row groups, 354,498,498 roots, 487,813,082 PV positions, 413s
+901/922 row groups, 380,851,882 roots, 522,891,063 PV positions, 443s
+FASTPREP_DONE 392086372 540415377

results/hyb1/w1/logs/hyb1_tm_sffull.log

@@ -0,0 +1,33 @@
+game 1/32 gm2b16t - sffull 1/2-1/2 (81 plies) score gm2b16t 0.5/1
+game 2/32 sffull - gm2b16t 1-0 (107 plies) score gm2b16t 0.5/2
+game 3/32 sffull - gm2b16t 1-0 (131 plies) score gm2b16t 0.5/3
+game 4/32 gm2b16t - sffull 0-1 (154 plies) score gm2b16t 0.5/4
+game 5/32 gm2b16t - sffull 0-1 (146 plies) score gm2b16t 0.5/5
+game 6/32 gm2b16t - sffull 1/2-1/2 (105 plies) score gm2b16t 1.0/6
+game 7/32 sffull - gm2b16t 1-0 (139 plies) score gm2b16t 1.0/7
+game 8/32 sffull - gm2b16t 1-0 (193 plies) score gm2b16t 1.0/8
+game 9/32 gm2b16t - sffull 1/2-1/2 (96 plies) score gm2b16t 1.5/9
+game 10/32 sffull - gm2b16t 1-0 (127 plies) score gm2b16t 1.5/10
+game 11/32 gm2b16t - sffull 1/2-1/2 (127 plies) score gm2b16t 2.0/11
+game 12/32 sffull - gm2b16t 1-0 (135 plies) score gm2b16t 2.0/12
+game 13/32 gm2b16t - sffull 0-1 (148 plies) score gm2b16t 2.0/13
+game 14/32 sffull - gm2b16t 1-0 (167 plies) score gm2b16t 2.0/14
+game 15/32 gm2b16t - sffull 0-1 (130 plies) score gm2b16t 2.0/15
+game 16/32 sffull - gm2b16t 1-0 (143 plies) score gm2b16t 2.0/16
+game 17/32 gm2b16t - sffull 0-1 (162 plies) score gm2b16t 2.0/17
+game 18/32 gm2b16t - sffull 0-1 (140 plies) score gm2b16t 2.0/18
+game 19/32 sffull - gm2b16t 1-0 (173 plies) score gm2b16t 2.0/19
+game 20/32 sffull - gm2b16t 1-0 (145 plies) score gm2b16t 2.0/20
+game 21/32 gm2b16t - sffull 0-1 (130 plies) score gm2b16t 2.0/21
+game 22/32 sffull - gm2b16t 1-0 (133 plies) score gm2b16t 2.0/22
+game 23/32 gm2b16t - sffull 0-1 (150 plies) score gm2b16t 2.0/23
+game 24/32 gm2b16t - sffull 1/2-1/2 (95 plies) score gm2b16t 2.5/24
+game 25/32 sffull - gm2b16t 1/2-1/2 (170 plies) score gm2b16t 3.0/25
+game 26/32 gm2b16t - sffull 1/2-1/2 (99 plies) score gm2b16t 3.5/26
+game 27/32 sffull - gm2b16t 1-0 (129 plies) score gm2b16t 3.5/27
+game 28/32 gm2b16t - sffull 0-1 (158 plies) score gm2b16t 3.5/28
+game 29/32 gm2b16t - sffull 1/2-1/2 (93 plies) score gm2b16t 4.0/29
+game 30/32 sffull - gm2b16t 1-0 (221 plies) score gm2b16t 4.0/30
+game 31/32 sffull - gm2b16t 1-0 (165 plies) score gm2b16t 4.0/31
+game 32/32 sffull - gm2b16t 1-0 (149 plies) score gm2b16t 4.0/32
+MATCH_DONE {"A": "gm2b16t", "B": "sffull", "spm": 1.0, "games": 32, "score": 4.0, "pct": 0.125, "elo_diff": -338, "minutes": 18.4, "gm2b16t_sims_per_move": 2482, "gm2b16t_ab_overrides": 0}