| logging stdout/stderr to research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/train.log |
| seed=42 flash SDP TF32 |
| using measured preliminary seq=1024 layout batch=16 accum=8; CPT lock remains seq=2048 |
| train layout batch=16 accum=8 (128 seq/step, 131072 tok/step) checkpointing=off liger=True |
| loading tokenizer research/results/embedding_init/gemma_mapping_byteify |
| full token census over local corpus |
| using full token census from /workspace/ugrip-kd/research/scripts/embedding_initialization/results/corpus_token_counts.json (cached tokenizer /workspace/ugrip-kd/research/results/expansion/tokenizer; current research/results/embedding_init/gemma_mapping_byteify) |
| am census: 476898623 tokens (1959.7 tok/doc, 243349 docs) |
| ta census: 4369273220 tokens (924.0 tok/doc, 4728460 docs) |
| kk census: 4464108618 tokens (1632.8 tok/doc, 2733982 docs) |
| tl census: 306736374 tokens (880.3 tok/doc, 348453 docs) |
| en census: 1542145528 tokens (771.1 tok/doc, 2000000 docs) |
| splits |
| am: full 243349 eval 256 pool 243093 |
| ta: full 4728460 eval 256 pool 4728204 |
| kk: full 2733982 eval 256 pool 2733726 |
| tl: full 348453 eval 256 pool 348197 |
| en: full 2000000 eval 256 pool 1999744 |
| mix weights {'am': 0.0446, 'ta': 0.4089, 'kk': 0.4178, 'tl': 0.0287, 'en': 0.1} (corpus-proportional; no document repeat) |
| lang token budgets {'am': 49803264, 'ta': 456292352, 'kk': 466195456, 'tl': 32032768, 'en': 111591424} |
| packing via HuggingFace datasets.map (batched, num_proc; not cached) |
| am: datasets.map n=35332 batched=1024 num_proc=8 |
|
hf-map am (num_proc=8): 0%| | 0/35332 [00:00<?, ? examples/s]
hf-map am (num_proc=8): 3%|2 | 1024/35332 [00:07<04:09, 137.64 examples/s]
hf-map am (num_proc=8): 9%|8 | 3072/35332 [00:07<01:03, 511.12 examples/s]
hf-map am (num_proc=8): 14%|#4 | 5120/35332 [00:08<00:34, 879.97 examples/s]
hf-map am (num_proc=8): 20%|## | 7168/35332 [00:08<00:19, 1450.16 examples/s]
hf-map am (num_proc=8): 26%|##6 | 9216/35332 [00:11<00:26, 1000.86 examples/s]
hf-map am (num_proc=8): 32%|###1 | 11264/35332 [00:11<00:16, 1498.20 examples/s]
hf-map am (num_proc=8): 38%|###7 | 13312/35332 [00:12<00:11, 1955.47 examples/s]
hf-map am (num_proc=8): 41%|#### | 14336/35332 [00:12<00:09, 2104.40 examples/s]
hf-map am (num_proc=8): 43%|####3 | 15360/35332 [00:12<00:07, 2524.41 examples/s]
hf-map am (num_proc=8): 49%|####9 | 17408/35332 [00:15<00:13, 1303.45 examples/s]
hf-map am (num_proc=8): 52%|#####2 | 18432/35332 [00:15<00:10, 1585.01 examples/s]
hf-map am (num_proc=8): 55%|#####5 | 19456/35332 [00:15<00:08, 1947.46 examples/s]
hf-map am (num_proc=8): 61%|###### | 21504/35332 [00:15<00:04, 2909.92 examples/s]
hf-map am (num_proc=8): 64%|######3 | 22528/35332 [00:15<00:03, 3422.96 examples/s]
hf-map am (num_proc=8): 67%|######6 | 23552/35332 [00:16<00:03, 3747.85 examples/s]
hf-map am (num_proc=8): 70%|######9 | 24576/35332 [00:16<00:02, 4062.43 examples/s]
hf-map am (num_proc=8): 72%|#######2 | 25600/35332 [00:19<00:08, 1140.56 examples/s]
hf-map am (num_proc=8): 84%|########4 | 29696/35332 [00:19<00:02, 2679.47 examples/s]
hf-map am (num_proc=8): 87%|########6 | 30720/35332 [00:19<00:01, 2686.69 examples/s]
hf-map am (num_proc=8): 90%|########9 | 31744/35332 [00:19<00:01, 2871.15 examples/s]
hf-map am (num_proc=8): 93%|#########2| 32768/35332 [00:20<00:00, 2821.29 examples/s]
hf-map am (num_proc=8): 95%|#########4| 33409/35332 [00:20<00:00, 2900.15 examples/s]
hf-map am (num_proc=8): 96%|#########6| 34049/35332 [00:20<00:00, 3219.02 examples/s]
hf-map am (num_proc=8): 98%|#########8| 34691/35332 [00:20<00:00, 3011.10 examples/s]
hf-map am (num_proc=8): 100%|##########| 35332/35332 [00:21<00:00, 2249.48 examples/s]
hf-map am (num_proc=8): 100%|##########| 35332/35332 [00:21<00:00, 1640.80 examples/s] |
| am: datasets.map num_proc=8 done 22s |
| am: 25000 docs 48394736 tokens 27s |
| packed am: 49804449 tokens (1.000 of budget) rows=62355 docs=25725 pad-per-doc (HF map) |
| ta: datasets.map n=667659 batched=1024 num_proc=8 |
|
hf-map ta (num_proc=8): 0%| | 0/667659 [00:00<?, ? examples/s]
hf-map ta (num_proc=8): 0%| | 1024/667659 [00:07<1:23:31, 133.03 examples/s]
hf-map ta (num_proc=8): 0%| | 2048/667659 [00:07<35:16, 314.53 examples/s]
hf-map ta (num_proc=8): 1%| | 4096/667659 [00:08<14:04, 785.91 examples/s]
hf-map ta (num_proc=8): 1%| | 6144/667659 [00:08<07:48, 1412.93 examples/s]
hf-map ta (num_proc=8): 1%|1 | 8192/667659 [00:08<04:51, 2258.58 examples/s]
hf-map ta (num_proc=8): 2%|1 | 10240/667659 [00:12<11:46, 930.61 examples/s]
hf-map ta (num_proc=8): 2%|1 | 11264/667659 [00:13<09:43, 1124.24 examples/s]
hf-map ta (num_proc=8): 2%|1 | 13312/667659 [00:13<06:26, 1691.19 examples/s]
hf-map ta (num_proc=8): 2%|2 | 14336/667659 [00:13<05:30, 1973.79 examples/s]
hf-map ta (num_proc=8): 2%|2 | 16384/667659 [00:13<04:29, 2417.89 examples/s]
hf-map ta (num_proc=8): 3%|2 | 17408/667659 [00:17<10:44, 1009.57 examples/s]
hf-map ta (num_proc=8): 3%|3 | 20480/667659 [00:17<06:05, 1769.33 examples/s]
hf-map ta (num_proc=8): 3%|3 | 21504/667659 [00:17<05:23, 1996.45 examples/s]
hf-map ta (num_proc=8): 3%|3 | 22528/667659 [00:18<05:09, 2085.43 examples/s]
hf-map ta (num_proc=8): 4%|3 | 24576/667659 [00:18<04:20, 2469.19 examples/s]
hf-map ta (num_proc=8): 4%|3 | 25600/667659 [00:21<09:26, 1133.33 examples/s]
hf-map ta (num_proc=8): 4%|3 | 26624/667659 [00:21<07:32, 1417.30 examples/s]
hf-map ta (num_proc=8): 4%|4 | 28672/667659 [00:21<04:59, 2131.90 examples/s]
hf-map ta (num_proc=8): 4%|4 | 29696/667659 [00:21<04:06, 2584.08 examples/s]
hf-map ta (num_proc=8): 5%|4 | 30720/667659 [00:22<04:39, 2281.78 examples/s]
hf-map ta (num_proc=8): 5%|4 | 31744/667659 [00:22<03:43, 2843.01 examples/s]
hf-map ta (num_proc=8): 5%|4 | 32768/667659 [00:22<03:48, 2773.30 examples/s]
hf-map ta (num_proc=8): 5%|5 | 33792/667659 [00:25<09:00, 1171.93 examples/s]
hf-map ta (num_proc=8): 5%|5 | 34816/667659 [00:25<07:36, 1386.96 examples/s]
hf-map ta (num_proc=8): 6%|5 | 36864/667659 [00:25<04:37, 2275.25 examples/s]
hf-map ta (num_proc=8): 6%|5 | 37888/667659 [00:25<03:45, 2789.32 examples/s]
hf-map ta (num_proc=8): 6%|5 | 38912/667659 [00:26<04:38, 2261.40 examples/s]
hf-map ta (num_proc=8): 6%|5 | 39936/667659 [00:26<03:42, 2825.94 examples/s]
hf-map ta (num_proc=8): 6%|6 | 40960/667659 [00:27<04:02, 2585.99 examples/s]
hf-map ta (num_proc=8): 6%|6 | 41984/667659 [00:29<08:26, 1234.11 examples/s]
hf-map ta (num_proc=8): 6%|6 | 43008/667659 [00:29<07:01, 1481.08 examples/s]
hf-map ta (num_proc=8): 7%|6 | 45056/667659 [00:29<04:45, 2181.82 examples/s]
hf-map ta (num_proc=8): 7%|6 | 46080/667659 [00:29<04:04, 2539.72 examples/s]
hf-map ta (num_proc=8): 7%|7 | 47104/667659 [00:30<04:22, 2367.29 examples/s]
hf-map ta (num_proc=8): 7%|7 | 49152/667659 [00:31<04:03, 2540.93 examples/s]
hf-map ta (num_proc=8): 8%|7 | 50176/667659 [00:32<07:03, 1457.61 examples/s]
hf-map ta (num_proc=8): 8%|7 | 51200/667659 [00:33<05:45, 1785.28 examples/s]
hf-map ta (num_proc=8): 8%|7 | 52224/667659 [00:33<04:46, 2149.76 examples/s]
hf-map ta (num_proc=8): 8%|7 | 53248/667659 [00:33<04:08, 2468.38 examples/s]
hf-map ta (num_proc=8): 8%|8 | 54272/667659 [00:33<03:42, 2758.85 examples/s]
hf-map ta (num_proc=8): 8%|8 | 55296/667659 [00:34<03:53, 2622.30 examples/s]
hf-map ta (num_proc=8): 9%|8 | 57344/667659 [00:35<04:20, 2346.01 examples/s]
hf-map ta (num_proc=8): 9%|8 | 58368/667659 [00:36<07:07, 1425.81 examples/s]
hf-map ta (num_proc=8): 9%|8 | 59392/667659 [00:36<05:37, 1801.96 examples/s]
hf-map ta (num_proc=8): 9%|9 | 61440/667659 [00:37<03:41, 2737.97 examples/s]
hf-map ta (num_proc=8): 9%|9 | 62464/667659 [00:37<04:14, 2376.40 examples/s]
hf-map ta (num_proc=8): 10%|9 | 64512/667659 [00:37<02:53, 3467.85 examples/s]
hf-map ta (num_proc=8): 10%|9 | 65536/667659 [00:38<04:22, 2292.68 examples/s]
hf-map ta (num_proc=8): 10%|9 | 66560/667659 [00:40<06:45, 1481.69 examples/s]
hf-map ta (num_proc=8): 10%|# | 67584/667659 [00:40<05:23, 1855.89 examples/s]
hf-map ta (num_proc=8): 10%|# | 68608/667659 [00:40<04:26, 2249.87 examples/s]
hf-map ta (num_proc=8): 10%|# | 69632/667659 [00:40<03:45, 2651.25 examples/s]
hf-map ta (num_proc=8): 11%|# | 70656/667659 [00:41<04:31, 2198.99 examples/s]
hf-map ta (num_proc=8): 11%|# | 72704/667659 [00:41<02:53, 3428.70 examples/s]
hf-map ta (num_proc=8): 11%|#1 | 73728/667659 [00:42<04:51, 2039.28 examples/s]
hf-map ta (num_proc=8): 11%|#1 | 74752/667659 [00:44<07:10, 1376.43 examples/s]
hf-map ta (num_proc=8): 12%|#1 | 77824/667659 [00:44<04:12, 2337.10 examples/s]
hf-map ta (num_proc=8): 12%|#1 | 78848/667659 [00:45<04:04, 2407.01 examples/s]
hf-map ta (num_proc=8): 12%|#1 | 79872/667659 [00:45<03:40, 2667.60 examples/s]
hf-map ta (num_proc=8): 12%|#2 | 81920/667659 [00:46<04:29, 2175.72 examples/s]
hf-map ta (num_proc=8): 12%|#2 | 82944/667659 [00:48<06:19, 1539.10 examples/s]
hf-map ta (num_proc=8): 13%|#2 | 83968/667659 [00:48<05:10, 1878.62 examples/s]
hf-map ta (num_proc=8): 13%|#2 | 86016/667659 [00:48<04:03, 2384.41 examples/s]
hf-map ta (num_proc=8): 13%|#3 | 87040/667659 [00:49<04:04, 2377.90 examples/s]
hf-map ta (num_proc=8): 13%|#3 | 88064/667659 [00:49<03:39, 2642.93 examples/s]
hf-map ta (num_proc=8): 13%|#3 | 90112/667659 [00:50<04:37, 2080.09 examples/s]
hf-map ta (num_proc=8): 14%|#3 | 91136/667659 [00:51<05:50, 1643.44 examples/s]
hf-map ta (num_proc=8): 14%|#3 | 92160/667659 [00:52<04:56, 1942.04 examples/s]
hf-map ta (num_proc=8): 14%|#3 | 93184/667659 [00:52<03:58, 2406.08 examples/s]
hf-map ta (num_proc=8): 14%|#4 | 94208/667659 [00:52<03:18, 2883.69 examples/s]
hf-map ta (num_proc=8): 14%|#4 | 95232/667659 [00:52<03:38, 2624.45 examples/s]
hf-map ta (num_proc=8): 14%|#4 | 96256/667659 [00:53<03:07, 3040.40 examples/s]
hf-map ta (num_proc=8): 15%|#4 | 98304/667659 [00:54<04:47, 1980.29 examples/s]
hf-map ta (num_proc=8): 15%|#4 | 99328/667659 [00:56<07:33, 1253.13 examples/s]
hf-map ta (num_proc=8): 15%|#5 | 102400/667659 [00:56<04:02, 2327.00 examples/s]
hf-map ta (num_proc=8): 15%|#5 | 103424/667659 [00:57<04:22, 2145.81 examples/s]
hf-map ta (num_proc=8): 16%|#5 | 105472/667659 [00:57<03:06, 3009.65 examples/s]
hf-map ta (num_proc=8): 16%|#5 | 106496/667659 [00:59<05:50, 1601.49 examples/s]
hf-map ta (num_proc=8): 16%|#6 | 107520/667659 [00:59<05:28, 1707.23 examples/s]
hf-map ta (num_proc=8): 16%|#6 | 109568/667659 [00:59<03:31, 2638.49 examples/s]
hf-map ta (num_proc=8): 17%|#6 | 110592/667659 [01:00<03:28, 2676.06 examples/s]
hf-map ta (num_proc=8): 17%|#6 | 111616/667659 [01:00<03:57, 2339.34 examples/s]
hf-map ta (num_proc=8): 17%|#7 | 113664/667659 [01:00<02:38, 3489.23 examples/s]
hf-map ta (num_proc=8): 17%|#7 | 114688/667659 [01:02<05:52, 1567.47 examples/s]
hf-map ta (num_proc=8): 17%|#7 | 115712/667659 [01:02<04:52, 1888.70 examples/s]
hf-map ta (num_proc=8): 17%|#7 | 116736/667659 [01:03<04:09, 2205.94 examples/s]
hf-map ta (num_proc=8): 18%|#7 | 117760/667659 [01:03<03:30, 2618.40 examples/s]
hf-map ta (num_proc=8): 18%|#7 | 119808/667659 [01:04<03:39, 2495.38 examples/s]
hf-map ta (num_proc=8): 18%|#8 | 121856/667659 [01:04<02:33, 3560.45 examples/s]
hf-map ta (num_proc=8): 18%|#8 | 122880/667659 [01:05<05:01, 1809.79 examples/s]
hf-map ta (num_proc=8): 19%|#8 | 123904/667659 [01:06<04:19, 2099.28 examples/s]
hf-map ta (num_proc=8): 19%|#8 | 125952/667659 [01:06<03:01, 2989.85 examples/s]
hf-map ta (num_proc=8): 19%|#9 | 126976/667659 [01:06<02:49, 3180.95 examples/s]
hf-map ta (num_proc=8): 19%|#9 | 128000/667659 [01:07<03:07, 2878.74 examples/s]
hf-map ta (num_proc=8): 19%|#9 | 129024/667659 [01:07<03:15, 2759.14 examples/s]
hf-map ta (num_proc=8): 19%|#9 | 130048/667659 [01:07<02:57, 3026.56 examples/s]
hf-map ta (num_proc=8): 20%|#9 | 131072/667659 [01:09<05:37, 1590.13 examples/s]
hf-map ta (num_proc=8): 20%|#9 | 132096/667659 [01:09<04:21, 2044.34 examples/s]
hf-map ta (num_proc=8): 20%|#9 | 133120/667659 [01:09<03:43, 2392.14 examples/s]
hf-map ta (num_proc=8): 20%|## | 135168/667659 [01:09<02:19, 3817.89 examples/s]
hf-map ta (num_proc=8): 20%|## | 136192/667659 [01:10<02:40, 3314.63 examples/s]
hf-map ta (num_proc=8): 21%|## | 137216/667659 [01:10<03:19, 2652.62 examples/s]
hf-map ta (num_proc=8): 21%|## | 138240/667659 [01:11<02:52, 3071.68 examples/s]
hf-map ta (num_proc=8): 21%|## | 139264/667659 [01:12<05:18, 1659.16 examples/s]
hf-map ta (num_proc=8): 21%|##1 | 141312/667659 [01:12<03:11, 2745.73 examples/s]
hf-map ta (num_proc=8): 21%|##1 | 142336/667659 [01:12<02:45, 3172.99 examples/s]
hf-map ta (num_proc=8): 21%|##1 | 143360/667659 [01:13<02:39, 3278.73 examples/s]
hf-map ta (num_proc=8): 22%|##1 | 144384/667659 [01:13<03:34, 2442.12 examples/s]
hf-map ta (num_proc=8): 22%|##1 | 145408/667659 [01:14<03:32, 2453.26 examples/s]
hf-map ta (num_proc=8): 22%|##2 | 147456/667659 [01:15<04:33, 1903.32 examples/s]
hf-map ta (num_proc=8): 22%|##2 | 149504/667659 [01:15<03:01, 2849.42 examples/s]
hf-map ta (num_proc=8): 23%|##2 | 150528/667659 [01:15<02:40, 3225.96 examples/s]
hf-map ta (num_proc=8): 23%|##2 | 151552/667659 [01:16<02:45, 3124.81 examples/s]
hf-map ta (num_proc=8): 23%|##2 | 152576/667659 [01:16<03:24, 2514.61 examples/s]
hf-map ta (num_proc=8): 23%|##3 | 153600/667659 [01:17<03:17, 2602.29 examples/s]
hf-map ta (num_proc=8): 23%|##3 | 154624/667659 [01:17<03:28, 2457.21 examples/s]
hf-map ta (num_proc=8): 23%|##3 | 155648/667659 [01:18<04:11, 2032.03 examples/s]
hf-map ta (num_proc=8): 23%|##3 | 156672/667659 [01:18<03:18, 2568.75 examples/s]
hf-map ta (num_proc=8): 24%|##3 | 157696/667659 [01:18<02:51, 2978.58 examples/s]
hf-map ta (num_proc=8): 24%|##3 | 159744/667659 [01:19<02:54, 2912.51 examples/s]
hf-map ta (num_proc=8): 24%|##4 | 160768/667659 [01:19<02:47, 3018.60 examples/s]
hf-map ta (num_proc=8): 24%|##4 | 161792/667659 [01:19<02:34, 3272.88 examples/s]
hf-map ta (num_proc=8): 24%|##4 | 162816/667659 [01:20<03:12, 2625.37 examples/s]
hf-map ta (num_proc=8): 25%|##4 | 163840/667659 [01:21<03:38, 2303.23 examples/s]
hf-map ta (num_proc=8): 25%|##4 | 164864/667659 [01:21<03:02, 2748.32 examples/s]
hf-map ta (num_proc=8): 25%|##4 | 165888/667659 [01:21<03:24, 2448.54 examples/s]
hf-map ta (num_proc=8): 25%|##5 | 167936/667659 [01:22<02:44, 3029.95 examples/s]
hf-map ta (num_proc=8): 25%|##5 | 168960/667659 [01:22<02:45, 3012.84 examples/s]
hf-map ta (num_proc=8): 26%|##5 | 171008/667659 [01:23<02:50, 2920.26 examples/s]
hf-map ta (num_proc=8): 26%|##5 | 172032/667659 [01:24<03:19, 2485.02 examples/s]
hf-map ta (num_proc=8): 26%|##5 | 173056/667659 [01:24<02:54, 2839.62 examples/s]
hf-map ta (num_proc=8): 26%|##6 | 174080/667659 [01:24<02:51, 2870.06 examples/s]
hf-map ta (num_proc=8): 26%|##6 | 176128/667659 [01:25<02:41, 3051.75 examples/s]
hf-map ta (num_proc=8): 27%|##6 | 177152/667659 [01:25<02:56, 2776.24 examples/s]
hf-map ta (num_proc=8): 27%|##6 | 179200/667659 [01:26<02:37, 3099.76 examples/s]
hf-map ta (num_proc=8): 27%|##6 | 180224/667659 [01:26<03:11, 2551.51 examples/s]
hf-map ta (num_proc=8): 27%|##7 | 181248/667659 [01:27<02:45, 2937.18 examples/s]
hf-map ta (num_proc=8): 27%|##7 | 182272/667659 [01:27<02:57, 2740.51 examples/s]
hf-map ta (num_proc=8): 28%|##7 | 184320/667659 [01:28<03:02, 2654.55 examples/s]
hf-map ta (num_proc=8): 28%|##7 | 186368/667659 [01:28<02:19, 3461.52 examples/s]
hf-map ta (num_proc=8): 28%|##8 | 187392/667659 [01:29<02:45, 2899.98 examples/s]
hf-map ta (num_proc=8): 28%|##8 | 188416/667659 [01:29<03:00, 2654.55 examples/s]
hf-map ta (num_proc=8): 29%|##8 | 190464/667659 [01:30<02:50, 2797.61 examples/s]
hf-map ta (num_proc=8): 29%|##8 | 191488/667659 [01:30<02:35, 3068.71 examples/s]
hf-map ta (num_proc=8): 29%|##8 | 192512/667659 [01:31<03:06, 2544.37 examples/s]
hf-map ta (num_proc=8): 29%|##8 | 193536/667659 [01:31<02:34, 3072.09 examples/s]
hf-map ta (num_proc=8): 29%|##9 | 194560/667659 [01:31<02:14, 3510.55 examples/s]
hf-map ta (num_proc=8): 29%|##9 | 195584/667659 [01:32<03:06, 2536.62 examples/s]
hf-map ta (num_proc=8): 29%|##9 | 196608/667659 [01:32<02:56, 2672.47 examples/s]
hf-map ta (num_proc=8): 30%|##9 | 197632/667659 [01:32<02:23, 3264.73 examples/s]
hf-map ta (num_proc=8): 30%|##9 | 198656/667659 [01:33<02:43, 2864.29 examples/s]
hf-map ta (num_proc=8): 30%|##9 | 199680/667659 [01:33<02:34, 3024.48 examples/s]
hf-map ta (num_proc=8): 30%|### | 200704/667659 [01:34<03:49, 2038.27 examples/s]
hf-map ta (num_proc=8): 30%|### | 202752/667659 [01:34<02:25, 3187.23 examples/s]
hf-map ta (num_proc=8): 31%|### | 203776/667659 [01:34<02:33, 3017.34 examples/s]
hf-map ta (num_proc=8): 31%|### | 204800/667659 [01:35<03:07, 2467.61 examples/s]
hf-map ta (num_proc=8): 31%|### | 206848/667659 [01:36<02:39, 2884.28 examples/s]
hf-map ta (num_proc=8): 31%|###1 | 207872/667659 [01:36<02:41, 2841.52 examples/s]
hf-map ta (num_proc=8): 31%|###1 | 208896/667659 [01:37<03:26, 2223.05 examples/s]
hf-map ta (num_proc=8): 32%|###1 | 210944/667659 [01:37<02:12, 3454.86 examples/s]
hf-map ta (num_proc=8): 32%|###1 | 211968/667659 [01:38<03:13, 2350.62 examples/s]
hf-map ta (num_proc=8): 32%|###2 | 214016/667659 [01:38<02:05, 3611.45 examples/s]
hf-map ta (num_proc=8): 32%|###2 | 215040/667659 [01:39<02:41, 2803.54 examples/s]
hf-map ta (num_proc=8): 32%|###2 | 216064/667659 [01:39<02:46, 2706.24 examples/s]
hf-map ta (num_proc=8): 33%|###2 | 217088/667659 [01:40<03:47, 1983.85 examples/s]
hf-map ta (num_proc=8): 33%|###2 | 218112/667659 [01:40<03:02, 2467.88 examples/s]
hf-map ta (num_proc=8): 33%|###2 | 220160/667659 [01:41<02:57, 2520.04 examples/s]
hf-map ta (num_proc=8): 33%|###3 | 221184/667659 [01:41<02:29, 2986.89 examples/s]
hf-map ta (num_proc=8): 33%|###3 | 222208/667659 [01:41<02:33, 2897.21 examples/s]
hf-map ta (num_proc=8): 33%|###3 | 223232/667659 [01:41<02:07, 3495.86 examples/s]
hf-map ta (num_proc=8): 34%|###3 | 224256/667659 [01:42<02:18, 3201.34 examples/s]
hf-map ta (num_proc=8): 34%|###3 | 225280/667659 [01:43<03:18, 2223.94 examples/s]
hf-map ta (num_proc=8): 34%|###3 | 226304/667659 [01:43<02:44, 2687.58 examples/s]
hf-map ta (num_proc=8): 34%|###4 | 228352/667659 [01:44<02:50, 2579.39 examples/s]
hf-map ta (num_proc=8): 34%|###4 | 229376/667659 [01:44<02:43, 2674.62 examples/s]
hf-map ta (num_proc=8): 35%|###4 | 230400/667659 [01:44<02:23, 3057.35 examples/s]
hf-map ta (num_proc=8): 35%|###4 | 231424/667659 [01:44<02:02, 3565.12 examples/s]
hf-map ta (num_proc=8): 35%|###4 | 232448/667659 [01:45<02:04, 3490.43 examples/s]
hf-map ta (num_proc=8): 35%|###4 | 233472/667659 [01:46<03:18, 2183.02 examples/s]
hf-map ta (num_proc=8): 35%|###5 | 234496/667659 [01:46<02:57, 2443.25 examples/s]
hf-map ta (num_proc=8): 35%|###5 | 235520/667659 [01:46<02:18, 3126.50 examples/s]
hf-map ta (num_proc=8): 35%|###5 | 236544/667659 [01:47<02:50, 2533.59 examples/s]
hf-map ta (num_proc=8): 36%|###5 | 237568/667659 [01:47<02:29, 2869.09 examples/s]
hf-map ta (num_proc=8): 36%|###5 | 238592/667659 [01:47<02:30, 2848.97 examples/s]
hf-map ta (num_proc=8): 36%|###5 | 239616/667659 [01:47<02:10, 3291.27 examples/s]
hf-map ta (num_proc=8): 36%|###6 | 240640/667659 [01:48<01:51, 3825.29 examples/s]
hf-map ta (num_proc=8): 36%|###6 | 241664/667659 [01:48<02:52, 2475.74 examples/s]
hf-map ta (num_proc=8): 36%|###6 | 242688/667659 [01:49<02:57, 2389.84 examples/s]
hf-map ta (num_proc=8): 37%|###6 | 243712/667659 [01:49<02:26, 2892.72 examples/s]
hf-map ta (num_proc=8): 37%|###6 | 244736/667659 [01:50<03:25, 2053.73 examples/s]
hf-map ta (num_proc=8): 37%|###6 | 245760/667659 [01:50<02:47, 2514.64 examples/s]
hf-map ta (num_proc=8): 37%|###6 | 246784/667659 [01:50<02:44, 2562.32 examples/s]
hf-map ta (num_proc=8): 37%|###7 | 247808/667659 [01:51<02:23, 2935.70 examples/s]
hf-map ta (num_proc=8): 37%|###7 | 249856/667659 [01:52<03:08, 2211.06 examples/s]
hf-map ta (num_proc=8): 38%|###7 | 250880/667659 [01:52<02:45, 2521.21 examples/s]
hf-map ta (num_proc=8): 38%|###7 | 251904/667659 [01:53<03:13, 2144.89 examples/s]
hf-map ta (num_proc=8): 38%|###7 | 252928/667659 [01:53<03:28, 1992.68 examples/s]
hf-map ta (num_proc=8): 38%|###8 | 253952/667659 [01:54<02:55, 2352.87 examples/s]
hf-map ta (num_proc=8): 38%|###8 | 254976/667659 [01:54<02:43, 2518.02 examples/s]
hf-map ta (num_proc=8): 38%|###8 | 256000/667659 [01:54<02:24, 2848.09 examples/s]
hf-map ta (num_proc=8): 38%|###8 | 257024/667659 [01:54<01:55, 3557.97 examples/s]
hf-map ta (num_proc=8): 39%|###8 | 258048/667659 [01:56<03:47, 1796.82 examples/s]
hf-map ta (num_proc=8): 39%|###8 | 260096/667659 [01:56<02:34, 2636.21 examples/s]
hf-map ta (num_proc=8): 39%|###9 | 261120/667659 [01:57<03:21, 2019.77 examples/s]
hf-map ta (num_proc=8): 39%|###9 | 262144/667659 [01:57<02:48, 2410.08 examples/s]
hf-map ta (num_proc=8): 39%|###9 | 263168/667659 [01:57<03:02, 2221.52 examples/s]
hf-map ta (num_proc=8): 40%|###9 | 264192/667659 [01:58<02:43, 2467.93 examples/s]
hf-map ta (num_proc=8): 40%|###9 | 266240/667659 [01:58<02:33, 2619.03 examples/s]
hf-map ta (num_proc=8): 40%|#### | 267264/667659 [01:59<02:27, 2722.64 examples/s]
hf-map ta (num_proc=8): 40%|#### | 268288/667659 [01:59<01:59, 3341.03 examples/s]
hf-map ta (num_proc=8): 40%|#### | 269312/667659 [01:59<02:17, 2892.53 examples/s]
hf-map ta (num_proc=8): 40%|#### | 270336/667659 [02:00<02:33, 2592.23 examples/s]
hf-map ta (num_proc=8): 41%|#### | 271360/667659 [02:00<02:06, 3130.78 examples/s]
hf-map ta (num_proc=8): 41%|#### | 272384/667659 [02:00<02:07, 3103.42 examples/s]
hf-map ta (num_proc=8): 41%|#### | 273408/667659 [02:01<01:49, 3606.25 examples/s]
hf-map ta (num_proc=8): 41%|####1 | 274432/667659 [02:01<02:41, 2435.45 examples/s]
hf-map ta (num_proc=8): 41%|####1 | 275456/667659 [02:02<02:22, 2761.65 examples/s]
hf-map ta (num_proc=8): 42%|####1 | 277504/667659 [02:02<01:58, 3301.88 examples/s]
hf-map ta (num_proc=8): 42%|####1 | 278528/667659 [02:03<02:34, 2522.11 examples/s]
hf-map ta (num_proc=8): 42%|####2 | 280576/667659 [02:03<01:46, 3641.13 examples/s]
hf-map ta (num_proc=8): 42%|####2 | 281600/667659 [02:03<02:01, 3174.59 examples/s]
hf-map ta (num_proc=8): 42%|####2 | 282624/667659 [02:04<02:40, 2404.17 examples/s]
hf-map ta (num_proc=8): 42%|####2 | 283648/667659 [02:04<02:13, 2884.67 examples/s]
hf-map ta (num_proc=8): 43%|####2 | 285696/667659 [02:05<01:55, 3300.61 examples/s]
hf-map ta (num_proc=8): 43%|####2 | 286720/667659 [02:05<02:10, 2929.36 examples/s]
hf-map ta (num_proc=8): 43%|####3 | 287744/667659 [02:05<01:50, 3424.00 examples/s]
hf-map ta (num_proc=8): 43%|####3 | 288768/667659 [02:06<01:39, 3798.99 examples/s]
hf-map ta (num_proc=8): 43%|####3 | 289792/667659 [02:06<01:56, 3233.83 examples/s]
hf-map ta (num_proc=8): 44%|####3 | 290816/667659 [02:07<02:43, 2308.21 examples/s]
hf-map ta (num_proc=8): 44%|####3 | 291840/667659 [02:07<02:32, 2469.27 examples/s]
hf-map ta (num_proc=8): 44%|####4 | 293888/667659 [02:08<01:55, 3237.94 examples/s]
hf-map ta (num_proc=8): 44%|####4 | 294912/667659 [02:08<01:51, 3353.35 examples/s]
hf-map ta (num_proc=8): 44%|####4 | 295936/667659 [02:08<01:37, 3815.30 examples/s]
hf-map ta (num_proc=8): 44%|####4 | 296960/667659 [02:08<01:40, 3682.50 examples/s]
hf-map ta (num_proc=8): 45%|####4 | 297984/667659 [02:09<02:14, 2751.66 examples/s]
hf-map ta (num_proc=8): 45%|####4 | 299008/667659 [02:10<02:50, 2156.59 examples/s]
hf-map ta (num_proc=8): 45%|####4 | 300032/667659 [02:10<02:24, 2543.89 examples/s]
hf-map ta (num_proc=8): 45%|####5 | 302080/667659 [02:10<01:33, 3895.49 examples/s]
hf-map ta (num_proc=8): 45%|####5 | 303104/667659 [02:11<01:53, 3214.20 examples/s]
hf-map ta (num_proc=8): 46%|####5 | 305152/667659 [02:11<01:33, 3865.81 examples/s]
hf-map ta (num_proc=8): 46%|####5 | 306176/667659 [02:12<02:20, 2565.20 examples/s]
hf-map ta (num_proc=8): 46%|####6 | 307200/667659 [02:12<02:37, 2290.35 examples/s]
hf-map ta (num_proc=8): 46%|####6 | 308224/667659 [02:13<02:16, 2627.93 examples/s]
hf-map ta (num_proc=8): 46%|####6 | 310272/667659 [02:13<01:39, 3587.52 examples/s]
hf-map ta (num_proc=8): 47%|####6 | 312320/667659 [02:14<01:55, 3077.99 examples/s]
hf-map ta (num_proc=8): 47%|####7 | 314368/667659 [02:14<02:00, 2938.23 examples/s]
hf-map ta (num_proc=8): 47%|####7 | 315392/667659 [02:15<02:13, 2638.47 examples/s]
hf-map ta (num_proc=8): 47%|####7 | 316416/667659 [02:15<01:57, 2980.17 examples/s]
hf-map ta (num_proc=8): 48%|####7 | 317440/667659 [02:16<01:54, 3066.55 examples/s]
hf-map ta (num_proc=8): 48%|####7 | 319488/667659 [02:16<01:18, 4445.36 examples/s]
hf-map ta (num_proc=8): 48%|####8 | 320512/667659 [02:17<02:03, 2800.26 examples/s]
hf-map ta (num_proc=8): 48%|####8 | 322560/667659 [02:17<01:53, 3040.48 examples/s]
hf-map ta (num_proc=8): 48%|####8 | 323584/667659 [02:17<01:51, 3089.04 examples/s]
hf-map ta (num_proc=8): 49%|####8 | 324608/667659 [02:18<01:35, 3584.75 examples/s]
hf-map ta (num_proc=8): 49%|####8 | 325632/667659 [02:18<02:01, 2812.46 examples/s]
hf-map ta (num_proc=8): 49%|####8 | 326656/667659 [02:18<01:41, 3370.29 examples/s]
hf-map ta (num_proc=8): 49%|####9 | 327680/667659 [02:19<01:44, 3252.97 examples/s]
hf-map ta (num_proc=8): 49%|####9 | 328704/667659 [02:19<01:52, 3009.43 examples/s]
hf-map ta (num_proc=8): 49%|####9 | 329728/667659 [02:19<01:33, 3602.36 examples/s]
hf-map ta (num_proc=8): 50%|####9 | 330752/667659 [02:19<01:30, 3718.42 examples/s]
hf-map ta (num_proc=8): 50%|####9 | 331776/667659 [02:20<01:53, 2963.09 examples/s]
hf-map ta (num_proc=8): 50%|####9 | 333824/667659 [02:21<01:49, 3039.81 examples/s]
hf-map ta (num_proc=8): 50%|##### | 334848/667659 [02:21<01:42, 3243.22 examples/s]
hf-map ta (num_proc=8): 50%|##### | 335872/667659 [02:21<01:36, 3443.45 examples/s]
hf-map ta (num_proc=8): 50%|##### | 336896/667659 [02:22<01:47, 3068.21 examples/s]
hf-map ta (num_proc=8): 51%|##### | 337920/667659 [02:22<01:38, 3356.08 examples/s]
hf-map ta (num_proc=8): 51%|##### | 339968/667659 [02:23<01:51, 2929.13 examples/s]
hf-map ta (num_proc=8): 51%|#####1 | 340992/667659 [02:23<01:43, 3167.35 examples/s]
hf-map ta (num_proc=8): 51%|#####1 | 342016/667659 [02:23<01:31, 3558.10 examples/s]
hf-map ta (num_proc=8): 51%|#####1 | 343040/667659 [02:24<01:53, 2870.45 examples/s]
hf-map ta (num_proc=8): 52%|#####1 | 345088/667659 [02:24<01:44, 3077.71 examples/s]
hf-map ta (num_proc=8): 52%|#####1 | 346112/667659 [02:24<01:33, 3441.08 examples/s]
hf-map ta (num_proc=8): 52%|#####1 | 347136/667659 [02:24<01:18, 4106.76 examples/s]
hf-map ta (num_proc=8): 52%|#####2 | 348160/667659 [02:25<02:07, 2514.77 examples/s]
hf-map ta (num_proc=8): 52%|#####2 | 349184/667659 [02:26<02:00, 2646.25 examples/s]
hf-map ta (num_proc=8): 52%|#####2 | 350208/667659 [02:26<01:41, 3142.73 examples/s]
hf-map ta (num_proc=8): 53%|#####2 | 351232/667659 [02:26<01:22, 3826.59 examples/s]
hf-map ta (num_proc=8): 53%|#####2 | 352256/667659 [02:26<01:29, 3508.10 examples/s]
hf-map ta (num_proc=8): 53%|#####2 | 353280/667659 [02:27<01:54, 2751.41 examples/s]
hf-map ta (num_proc=8): 53%|#####3 | 354304/667659 [02:27<01:53, 2750.51 examples/s]
hf-map ta (num_proc=8): 53%|#####3 | 356352/667659 [02:28<02:06, 2458.58 examples/s]
hf-map ta (num_proc=8): 54%|#####3 | 357376/667659 [02:28<01:52, 2752.00 examples/s]
hf-map ta (num_proc=8): 54%|#####3 | 359424/667659 [02:29<01:34, 3255.16 examples/s]
hf-map ta (num_proc=8): 54%|#####3 | 360448/667659 [02:29<01:42, 3001.49 examples/s]
hf-map ta (num_proc=8): 54%|#####4 | 361472/667659 [02:29<01:28, 3445.48 examples/s]
hf-map ta (num_proc=8): 54%|#####4 | 362496/667659 [02:30<01:21, 3733.67 examples/s]
hf-map ta (num_proc=8): 54%|#####4 | 363520/667659 [02:30<01:15, 4033.06 examples/s]
hf-map ta (num_proc=8): 55%|#####4 | 364544/667659 [02:31<02:05, 2405.89 examples/s]
hf-map ta (num_proc=8): 55%|#####4 | 365568/667659 [02:31<01:55, 2614.95 examples/s]
hf-map ta (num_proc=8): 55%|#####4 | 366592/667659 [02:31<01:46, 2828.23 examples/s]
hf-map ta (num_proc=8): 55%|#####5 | 367616/667659 [02:31<01:31, 3295.96 examples/s]
hf-map ta (num_proc=8): 55%|#####5 | 368640/667659 [02:32<01:33, 3188.39 examples/s]
hf-map ta (num_proc=8): 55%|#####5 | 369664/667659 [02:32<01:26, 3453.42 examples/s]
hf-map ta (num_proc=8): 56%|#####5 | 370688/667659 [02:32<01:20, 3678.92 examples/s]
hf-map ta (num_proc=8): 56%|#####5 | 371712/667659 [02:32<01:12, 4080.74 examples/s]
hf-map ta (num_proc=8): 56%|#####5 | 372736/667659 [02:34<02:31, 1940.72 examples/s]
hf-map ta (num_proc=8): 56%|#####5 | 373760/667659 [02:34<02:00, 2443.30 examples/s]
hf-map ta (num_proc=8): 56%|#####6 | 374784/667659 [02:34<01:44, 2815.34 examples/s]
hf-map ta (num_proc=8): 56%|#####6 | 375808/667659 [02:35<02:10, 2237.90 examples/s]
hf-map ta (num_proc=8): 56%|#####6 | 376832/667659 [02:35<02:05, 2314.18 examples/s]
hf-map ta (num_proc=8): 57%|#####6 | 377856/667659 [02:36<02:05, 2308.55 examples/s]
hf-map ta (num_proc=8): 57%|#####6 | 378880/667659 [02:36<02:03, 2331.87 examples/s]
hf-map ta (num_proc=8): 57%|#####6 | 379904/667659 [02:36<01:38, 2921.07 examples/s]
hf-map ta (num_proc=8): 57%|#####7 | 380928/667659 [02:38<03:02, 1571.73 examples/s]
hf-map ta (num_proc=8): 57%|#####7 | 382976/667659 [02:38<01:51, 2558.76 examples/s]
hf-map ta (num_proc=8): 58%|#####7 | 384000/667659 [02:38<01:31, 3090.11 examples/s]
hf-map ta (num_proc=8): 58%|#####7 | 385024/667659 [02:38<01:44, 2707.57 examples/s]
hf-map ta (num_proc=8): 58%|#####7 | 386048/667659 [02:39<01:31, 3083.59 examples/s]
hf-map ta (num_proc=8): 58%|#####7 | 387072/667659 [02:39<01:46, 2639.45 examples/s]
hf-map ta (num_proc=8): 58%|#####8 | 389120/667659 [02:41<02:37, 1772.81 examples/s]
hf-map ta (num_proc=8): 59%|#####8 | 391168/667659 [02:41<01:49, 2534.32 examples/s]
hf-map ta (num_proc=8): 59%|#####8 | 392192/667659 [02:41<01:42, 2692.04 examples/s]
hf-map ta (num_proc=8): 59%|#####8 | 393216/667659 [02:42<01:39, 2753.57 examples/s]
hf-map ta (num_proc=8): 59%|#####9 | 394240/667659 [02:42<01:50, 2471.69 examples/s]
hf-map ta (num_proc=8): 59%|#####9 | 395264/667659 [02:42<01:32, 2938.62 examples/s]
hf-map ta (num_proc=8): 59%|#####9 | 396288/667659 [02:42<01:17, 3515.48 examples/s]
hf-map ta (num_proc=8): 60%|#####9 | 397312/667659 [02:44<02:19, 1941.91 examples/s]
hf-map ta (num_proc=8): 60%|#####9 | 399360/667659 [02:44<01:22, 3234.07 examples/s]
hf-map ta (num_proc=8): 60%|#####9 | 400384/667659 [02:44<01:32, 2885.55 examples/s]
hf-map ta (num_proc=8): 60%|###### | 401408/667659 [02:44<01:19, 3336.73 examples/s]
hf-map ta (num_proc=8): 60%|###### | 402432/667659 [02:45<01:26, 3063.39 examples/s]
hf-map ta (num_proc=8): 60%|###### | 403456/667659 [02:45<01:27, 3020.24 examples/s]
hf-map ta (num_proc=8): 61%|###### | 405504/667659 [02:46<01:53, 2301.76 examples/s]
hf-map ta (num_proc=8): 61%|######1 | 407552/667659 [02:46<01:18, 3325.50 examples/s]
hf-map ta (num_proc=8): 61%|######1 | 408576/667659 [02:47<01:32, 2811.59 examples/s]
hf-map ta (num_proc=8): 62%|######1 | 410624/667659 [02:48<01:33, 2756.90 examples/s]
hf-map ta (num_proc=8): 62%|######1 | 412672/667659 [02:48<01:17, 3279.48 examples/s]
hf-map ta (num_proc=8): 62%|######1 | 413696/667659 [02:49<01:18, 3242.54 examples/s]
hf-map ta (num_proc=8): 62%|######2 | 414720/667659 [02:49<01:31, 2756.02 examples/s]
hf-map ta (num_proc=8): 62%|######2 | 416768/667659 [02:49<01:13, 3397.01 examples/s]
hf-map ta (num_proc=8): 63%|######2 | 417792/667659 [02:50<01:07, 3677.07 examples/s]
hf-map ta (num_proc=8): 63%|######2 | 418816/667659 [02:51<02:18, 1793.25 examples/s]
hf-map ta (num_proc=8): 63%|######3 | 420864/667659 [02:52<01:38, 2517.09 examples/s]
hf-map ta (num_proc=8): 63%|######3 | 422912/667659 [02:52<01:11, 3404.94 examples/s]
hf-map ta (num_proc=8): 63%|######3 | 423936/667659 [02:52<01:10, 3459.17 examples/s]
hf-map ta (num_proc=8): 64%|######3 | 424960/667659 [02:52<01:02, 3853.99 examples/s]
hf-map ta (num_proc=8): 64%|######3 | 425984/667659 [02:52<01:01, 3926.44 examples/s]
hf-map ta (num_proc=8): 64%|######3 | 427008/667659 [02:54<02:12, 1809.50 examples/s]
hf-map ta (num_proc=8): 64%|######4 | 429056/667659 [02:54<01:36, 2468.66 examples/s]
hf-map ta (num_proc=8): 65%|######4 | 431104/667659 [02:55<01:11, 3300.71 examples/s]
hf-map ta (num_proc=8): 65%|######4 | 432128/667659 [02:55<01:17, 3025.01 examples/s]
hf-map ta (num_proc=8): 65%|######5 | 434176/667659 [02:55<00:56, 4111.60 examples/s]
hf-map ta (num_proc=8): 65%|######5 | 435200/667659 [02:56<01:36, 2397.87 examples/s]
hf-map ta (num_proc=8): 65%|######5 | 437248/667659 [02:57<01:28, 2599.41 examples/s]
hf-map ta (num_proc=8): 66%|######5 | 438272/667659 [02:57<01:21, 2815.28 examples/s]
hf-map ta (num_proc=8): 66%|######5 | 439296/667659 [02:58<01:17, 2945.32 examples/s]
hf-map ta (num_proc=8): 66%|######5 | 440320/667659 [02:58<01:10, 3227.15 examples/s]
hf-map ta (num_proc=8): 66%|######6 | 441344/667659 [02:58<00:58, 3859.12 examples/s]
hf-map ta (num_proc=8): 66%|######6 | 442368/667659 [02:58<01:12, 3098.73 examples/s]
hf-map ta (num_proc=8): 66%|######6 | 443392/667659 [02:59<01:31, 2443.38 examples/s]
hf-map ta (num_proc=8): 67%|######6 | 444416/667659 [02:59<01:21, 2727.26 examples/s]
hf-map ta (num_proc=8): 67%|######6 | 445440/667659 [03:00<01:53, 1955.07 examples/s]
hf-map ta (num_proc=8): 67%|######6 | 446464/667659 [03:01<01:40, 2211.71 examples/s]
hf-map ta (num_proc=8): 67%|######7 | 449536/667659 [03:01<00:50, 4318.61 examples/s]
hf-map ta (num_proc=8): 67%|######7 | 450560/667659 [03:02<01:31, 2383.38 examples/s]
hf-map ta (num_proc=8): 68%|######7 | 451584/667659 [03:02<01:33, 2315.15 examples/s]
hf-map ta (num_proc=8): 68%|######7 | 453632/667659 [03:03<01:37, 2186.54 examples/s]
hf-map ta (num_proc=8): 68%|######8 | 454656/667659 [03:04<01:31, 2323.73 examples/s]
hf-map ta (num_proc=8): 68%|######8 | 455680/667659 [03:04<01:44, 2021.51 examples/s]
hf-map ta (num_proc=8): 68%|######8 | 456704/667659 [03:04<01:23, 2540.07 examples/s]
hf-map ta (num_proc=8): 69%|######8 | 457728/667659 [03:05<01:13, 2860.37 examples/s]
hf-map ta (num_proc=8): 69%|######8 | 458752/667659 [03:06<01:44, 2004.68 examples/s]
hf-map ta (num_proc=8): 69%|######9 | 460800/667659 [03:06<01:15, 2724.38 examples/s]
hf-map ta (num_proc=8): 69%|######9 | 461824/667659 [03:06<01:15, 2723.90 examples/s]
hf-map ta (num_proc=8): 69%|######9 | 463872/667659 [03:07<01:16, 2662.07 examples/s]
hf-map ta (num_proc=8): 70%|######9 | 464896/667659 [03:08<01:13, 2767.31 examples/s]
hf-map ta (num_proc=8): 70%|######9 | 466944/667659 [03:08<01:05, 3064.61 examples/s]
hf-map ta (num_proc=8): 70%|####### | 467968/667659 [03:08<00:55, 3596.19 examples/s]
hf-map ta (num_proc=8): 70%|####### | 468992/667659 [03:09<01:06, 2977.10 examples/s]
hf-map ta (num_proc=8): 71%|####### | 471040/667659 [03:09<00:55, 3572.40 examples/s]
hf-map ta (num_proc=8): 71%|####### | 472064/667659 [03:10<01:04, 3040.51 examples/s]
hf-map ta (num_proc=8): 71%|####### | 473088/667659 [03:10<01:08, 2820.15 examples/s]
hf-map ta (num_proc=8): 71%|#######1 | 475136/667659 [03:11<00:59, 3261.33 examples/s]
hf-map ta (num_proc=8): 71%|#######1 | 476160/667659 [03:11<00:59, 3225.99 examples/s]
hf-map ta (num_proc=8): 71%|#######1 | 477184/667659 [03:11<00:55, 3423.08 examples/s]
hf-map ta (num_proc=8): 72%|#######1 | 478208/667659 [03:11<00:55, 3406.23 examples/s]
hf-map ta (num_proc=8): 72%|#######1 | 479232/667659 [03:12<00:48, 3914.25 examples/s]
hf-map ta (num_proc=8): 72%|#######1 | 480256/667659 [03:12<00:44, 4191.74 examples/s]
hf-map ta (num_proc=8): 72%|#######2 | 481280/667659 [03:12<01:03, 2938.00 examples/s]
hf-map ta (num_proc=8): 72%|#######2 | 482304/667659 [03:13<01:00, 3082.47 examples/s]
hf-map ta (num_proc=8): 72%|#######2 | 483328/667659 [03:13<00:56, 3285.05 examples/s]
hf-map ta (num_proc=8): 73%|#######2 | 484352/667659 [03:13<01:01, 2996.36 examples/s]
hf-map ta (num_proc=8): 73%|#######2 | 485376/667659 [03:14<00:54, 3317.10 examples/s]
hf-map ta (num_proc=8): 73%|#######2 | 486400/667659 [03:14<00:47, 3825.03 examples/s]
hf-map ta (num_proc=8): 73%|#######3 | 487424/667659 [03:14<00:41, 4345.52 examples/s]
hf-map ta (num_proc=8): 73%|#######3 | 488448/667659 [03:14<00:43, 4102.37 examples/s]
hf-map ta (num_proc=8): 73%|#######3 | 489472/667659 [03:15<01:06, 2699.65 examples/s]
hf-map ta (num_proc=8): 73%|#######3 | 490496/667659 [03:15<01:07, 2619.60 examples/s]
hf-map ta (num_proc=8): 74%|#######3 | 492544/667659 [03:16<01:03, 2762.26 examples/s]
hf-map ta (num_proc=8): 74%|#######4 | 494592/667659 [03:16<00:47, 3606.26 examples/s]
hf-map ta (num_proc=8): 74%|#######4 | 496640/667659 [03:17<00:44, 3808.42 examples/s]
hf-map ta (num_proc=8): 75%|#######4 | 497664/667659 [03:18<01:03, 2689.94 examples/s]
hf-map ta (num_proc=8): 75%|#######4 | 498688/667659 [03:18<01:08, 2468.44 examples/s]
hf-map ta (num_proc=8): 75%|#######4 | 500736/667659 [03:19<01:00, 2769.16 examples/s]
hf-map ta (num_proc=8): 75%|#######5 | 502784/667659 [03:19<00:45, 3645.51 examples/s]
hf-map ta (num_proc=8): 76%|#######5 | 504832/667659 [03:19<00:40, 4011.51 examples/s]
hf-map ta (num_proc=8): 76%|#######5 | 505856/667659 [03:21<01:09, 2320.56 examples/s]
hf-map ta (num_proc=8): 76%|#######5 | 506880/667659 [03:21<01:02, 2554.57 examples/s]
hf-map ta (num_proc=8): 76%|#######6 | 507904/667659 [03:21<00:54, 2940.82 examples/s]
hf-map ta (num_proc=8): 76%|#######6 | 508928/667659 [03:21<00:54, 2914.01 examples/s]
hf-map ta (num_proc=8): 76%|#######6 | 509952/667659 [03:22<00:47, 3316.66 examples/s]
hf-map ta (num_proc=8): 77%|#######6 | 510976/667659 [03:22<00:40, 3855.70 examples/s]
hf-map ta (num_proc=8): 77%|#######6 | 512000/667659 [03:22<00:33, 4583.74 examples/s]
hf-map ta (num_proc=8): 77%|#######6 | 513024/667659 [03:22<00:39, 3906.29 examples/s]
hf-map ta (num_proc=8): 77%|#######6 | 514048/667659 [03:23<00:59, 2560.93 examples/s]
hf-map ta (num_proc=8): 77%|#######7 | 515072/667659 [03:23<01:03, 2389.14 examples/s]
hf-map ta (num_proc=8): 77%|#######7 | 517120/667659 [03:24<00:51, 2912.22 examples/s]
hf-map ta (num_proc=8): 78%|#######7 | 518144/667659 [03:24<00:47, 3119.45 examples/s]
hf-map ta (num_proc=8): 78%|#######7 | 519168/667659 [03:24<00:40, 3682.43 examples/s]
hf-map ta (num_proc=8): 78%|#######7 | 520192/667659 [03:25<00:37, 3954.50 examples/s]
hf-map ta (num_proc=8): 78%|#######8 | 521216/667659 [03:25<00:42, 3484.37 examples/s]
hf-map ta (num_proc=8): 78%|#######8 | 522240/667659 [03:26<00:54, 2690.67 examples/s]
hf-map ta (num_proc=8): 78%|#######8 | 523264/667659 [03:26<00:45, 3183.90 examples/s]
hf-map ta (num_proc=8): 79%|#######8 | 524288/667659 [03:26<00:43, 3310.64 examples/s]
hf-map ta (num_proc=8): 79%|#######8 | 525312/667659 [03:26<00:35, 4053.04 examples/s]
hf-map ta (num_proc=8): 79%|#######8 | 526336/667659 [03:26<00:34, 4148.33 examples/s]
hf-map ta (num_proc=8): 79%|#######8 | 527360/667659 [03:27<00:30, 4564.33 examples/s]
hf-map ta (num_proc=8): 79%|#######9 | 528384/667659 [03:27<00:35, 3881.13 examples/s]
hf-map ta (num_proc=8): 79%|#######9 | 529408/667659 [03:27<00:47, 2893.49 examples/s]
hf-map ta (num_proc=8): 79%|#######9 | 530432/667659 [03:28<00:52, 2620.49 examples/s]
hf-map ta (num_proc=8): 80%|#######9 | 531456/667659 [03:28<00:55, 2460.53 examples/s]
hf-map ta (num_proc=8): 80%|#######9 | 533504/667659 [03:29<00:37, 3534.41 examples/s]
hf-map ta (num_proc=8): 80%|######## | 534528/667659 [03:29<00:38, 3423.34 examples/s]
hf-map ta (num_proc=8): 80%|######## | 535552/667659 [03:29<00:34, 3780.21 examples/s]
hf-map ta (num_proc=8): 80%|######## | 536576/667659 [03:29<00:28, 4530.96 examples/s]
hf-map ta (num_proc=8): 81%|######## | 537600/667659 [03:30<00:43, 3010.18 examples/s]
hf-map ta (num_proc=8): 81%|######## | 538624/667659 [03:30<00:38, 3313.35 examples/s]
hf-map ta (num_proc=8): 81%|######## | 539648/667659 [03:31<01:00, 2112.05 examples/s]
hf-map ta (num_proc=8): 81%|########1 | 541696/667659 [03:31<00:37, 3333.44 examples/s]
hf-map ta (num_proc=8): 81%|########1 | 542720/667659 [03:32<00:39, 3195.04 examples/s]
hf-map ta (num_proc=8): 82%|########1 | 544768/667659 [03:32<00:33, 3628.17 examples/s]
hf-map ta (num_proc=8): 82%|########1 | 545792/667659 [03:33<00:41, 2921.54 examples/s]
hf-map ta (num_proc=8): 82%|########2 | 547840/667659 [03:34<00:43, 2750.30 examples/s]
hf-map ta (num_proc=8): 82%|########2 | 548864/667659 [03:34<00:41, 2862.01 examples/s]
hf-map ta (num_proc=8): 82%|########2 | 549888/667659 [03:34<00:36, 3255.23 examples/s]
hf-map ta (num_proc=8): 83%|########2 | 550912/667659 [03:34<00:29, 3907.50 examples/s]
hf-map ta (num_proc=8): 83%|########2 | 551936/667659 [03:34<00:28, 4030.93 examples/s]
hf-map ta (num_proc=8): 83%|########2 | 552960/667659 [03:34<00:25, 4528.25 examples/s]
hf-map ta (num_proc=8): 83%|########2 | 553984/667659 [03:35<00:43, 2639.05 examples/s]
hf-map ta (num_proc=8): 83%|########3 | 555008/667659 [03:35<00:36, 3062.08 examples/s]
hf-map ta (num_proc=8): 83%|########3 | 556032/667659 [03:36<00:47, 2366.14 examples/s]
hf-map ta (num_proc=8): 84%|########3 | 558080/667659 [03:36<00:30, 3559.90 examples/s]
hf-map ta (num_proc=8): 84%|########3 | 560128/667659 [03:37<00:25, 4287.39 examples/s]
hf-map ta (num_proc=8): 84%|########4 | 561152/667659 [03:37<00:30, 3513.18 examples/s]
hf-map ta (num_proc=8): 84%|########4 | 562176/667659 [03:38<00:43, 2452.30 examples/s]
hf-map ta (num_proc=8): 84%|########4 | 563200/667659 [03:38<00:35, 2983.90 examples/s]
hf-map ta (num_proc=8): 85%|########4 | 564224/667659 [03:39<00:36, 2865.09 examples/s]
hf-map ta (num_proc=8): 85%|########4 | 565248/667659 [03:39<00:28, 3533.52 examples/s]
hf-map ta (num_proc=8): 85%|########4 | 566272/667659 [03:39<00:30, 3305.16 examples/s]
hf-map ta (num_proc=8): 85%|########5 | 568320/667659 [03:39<00:24, 4051.42 examples/s]
hf-map ta (num_proc=8): 85%|########5 | 569344/667659 [03:40<00:28, 3478.01 examples/s]
hf-map ta (num_proc=8): 85%|########5 | 570368/667659 [03:40<00:29, 3275.07 examples/s]
hf-map ta (num_proc=8): 86%|########5 | 571392/667659 [03:40<00:28, 3406.81 examples/s]
hf-map ta (num_proc=8): 86%|########5 | 572416/667659 [03:41<00:27, 3443.72 examples/s]
hf-map ta (num_proc=8): 86%|########5 | 573440/667659 [03:41<00:27, 3417.38 examples/s]
hf-map ta (num_proc=8): 86%|########6 | 574464/667659 [03:41<00:30, 3007.92 examples/s]
hf-map ta (num_proc=8): 86%|########6 | 576512/667659 [03:42<00:21, 4208.12 examples/s]
hf-map ta (num_proc=8): 87%|########6 | 577536/667659 [03:42<00:30, 2975.13 examples/s]
hf-map ta (num_proc=8): 87%|########6 | 578560/667659 [03:43<00:26, 3308.75 examples/s]
hf-map ta (num_proc=8): 87%|########6 | 579584/667659 [03:43<00:30, 2842.68 examples/s]
hf-map ta (num_proc=8): 87%|########6 | 580608/667659 [03:43<00:31, 2785.84 examples/s]
hf-map ta (num_proc=8): 87%|########7 | 581632/667659 [03:44<00:24, 3489.86 examples/s]
hf-map ta (num_proc=8): 87%|########7 | 582656/667659 [03:44<00:22, 3701.29 examples/s]
hf-map ta (num_proc=8): 87%|########7 | 583680/667659 [03:44<00:18, 4464.58 examples/s]
hf-map ta (num_proc=8): 88%|########7 | 584704/667659 [03:44<00:19, 4317.72 examples/s]
hf-map ta (num_proc=8): 88%|########7 | 585728/667659 [03:45<00:26, 3104.99 examples/s]
hf-map ta (num_proc=8): 88%|########7 | 586752/667659 [03:45<00:30, 2686.12 examples/s]
hf-map ta (num_proc=8): 88%|########8 | 587776/667659 [03:46<00:27, 2865.72 examples/s]
hf-map ta (num_proc=8): 88%|########8 | 588800/667659 [03:46<00:27, 2854.92 examples/s]
hf-map ta (num_proc=8): 88%|########8 | 589824/667659 [03:46<00:26, 2964.58 examples/s]
hf-map ta (num_proc=8): 88%|########8 | 590848/667659 [03:46<00:22, 3368.81 examples/s]
hf-map ta (num_proc=8): 89%|########8 | 591872/667659 [03:47<00:22, 3338.48 examples/s]
hf-map ta (num_proc=8): 89%|########8 | 592896/667659 [03:47<00:18, 4001.26 examples/s]
hf-map ta (num_proc=8): 89%|########8 | 593920/667659 [03:47<00:19, 3706.41 examples/s]
hf-map ta (num_proc=8): 89%|########9 | 594944/667659 [03:48<00:20, 3499.21 examples/s]
hf-map ta (num_proc=8): 89%|########9 | 595968/667659 [03:48<00:22, 3175.20 examples/s]
hf-map ta (num_proc=8): 89%|########9 | 596992/667659 [03:48<00:23, 2990.83 examples/s]
hf-map ta (num_proc=8): 90%|########9 | 598016/667659 [03:49<00:23, 2943.84 examples/s]
hf-map ta (num_proc=8): 90%|########9 | 599040/667659 [03:49<00:20, 3299.88 examples/s]
hf-map ta (num_proc=8): 90%|########9 | 600064/667659 [03:49<00:19, 3501.33 examples/s]
hf-map ta (num_proc=8): 90%|######### | 601088/667659 [03:49<00:16, 4059.22 examples/s]
hf-map ta (num_proc=8): 90%|######### | 602112/667659 [03:50<00:24, 2684.58 examples/s]
hf-map ta (num_proc=8): 90%|######### | 603136/667659 [03:50<00:20, 3168.13 examples/s]
hf-map ta (num_proc=8): 90%|######### | 604160/667659 [03:51<00:20, 3137.17 examples/s]
hf-map ta (num_proc=8): 91%|######### | 605184/667659 [03:51<00:16, 3763.88 examples/s]
hf-map ta (num_proc=8): 91%|######### | 606208/667659 [03:51<00:18, 3340.32 examples/s]
hf-map ta (num_proc=8): 91%|######### | 607232/667659 [03:51<00:18, 3225.19 examples/s]
hf-map ta (num_proc=8): 91%|#########1| 608256/667659 [03:52<00:15, 3785.78 examples/s]
hf-map ta (num_proc=8): 91%|#########1| 610304/667659 [03:52<00:17, 3337.96 examples/s]
hf-map ta (num_proc=8): 92%|#########1| 611328/667659 [03:53<00:21, 2592.50 examples/s]
hf-map ta (num_proc=8): 92%|#########1| 612352/667659 [03:53<00:20, 2752.08 examples/s]
hf-map ta (num_proc=8): 92%|#########2| 614400/667659 [03:54<00:14, 3586.11 examples/s]
hf-map ta (num_proc=8): 92%|#########2| 615424/667659 [03:54<00:15, 3275.03 examples/s]
hf-map ta (num_proc=8): 92%|#########2| 616448/667659 [03:54<00:13, 3887.22 examples/s]
hf-map ta (num_proc=8): 92%|#########2| 617472/667659 [03:54<00:11, 4543.88 examples/s]
hf-map ta (num_proc=8): 93%|#########2| 618496/667659 [03:55<00:17, 2790.61 examples/s]
hf-map ta (num_proc=8): 93%|#########2| 619520/667659 [03:55<00:19, 2526.48 examples/s]
hf-map ta (num_proc=8): 93%|#########2| 620544/667659 [03:56<00:15, 2954.23 examples/s]
hf-map ta (num_proc=8): 93%|#########3| 621568/667659 [03:56<00:12, 3630.44 examples/s]
hf-map ta (num_proc=8): 93%|#########3| 622592/667659 [03:56<00:13, 3271.90 examples/s]
hf-map ta (num_proc=8): 93%|#########3| 623616/667659 [03:56<00:11, 3803.55 examples/s]
hf-map ta (num_proc=8): 94%|#########3| 624640/667659 [03:57<00:11, 3690.66 examples/s]
hf-map ta (num_proc=8): 94%|#########3| 625664/667659 [03:57<00:09, 4385.57 examples/s]
hf-map ta (num_proc=8): 94%|#########3| 626688/667659 [03:57<00:11, 3713.66 examples/s]
hf-map ta (num_proc=8): 94%|#########4| 627712/667659 [03:58<00:15, 2542.38 examples/s]
hf-map ta (num_proc=8): 94%|#########4| 628736/667659 [03:58<00:12, 3036.90 examples/s]
hf-map ta (num_proc=8): 94%|#########4| 629760/667659 [03:58<00:10, 3679.91 examples/s]
hf-map ta (num_proc=8): 94%|#########4| 630784/667659 [03:59<00:12, 2946.47 examples/s]
hf-map ta (num_proc=8): 95%|#########4| 631808/667659 [03:59<00:12, 2764.68 examples/s]
hf-map ta (num_proc=8): 95%|#########4| 633856/667659 [03:59<00:08, 3778.87 examples/s]
hf-map ta (num_proc=8): 95%|#########5| 634880/667659 [04:00<00:07, 4143.95 examples/s]
hf-map ta (num_proc=8): 95%|#########5| 635904/667659 [04:00<00:10, 2985.78 examples/s]
hf-map ta (num_proc=8): 95%|#########5| 636928/667659 [04:01<00:10, 2893.30 examples/s]
hf-map ta (num_proc=8): 96%|#########5| 637952/667659 [04:01<00:10, 2809.77 examples/s]
hf-map ta (num_proc=8): 96%|#########5| 640000/667659 [04:01<00:06, 4022.89 examples/s]
hf-map ta (num_proc=8): 96%|#########6| 641024/667659 [04:01<00:06, 4200.82 examples/s]
hf-map ta (num_proc=8): 96%|#########6| 642048/667659 [04:02<00:06, 3796.41 examples/s]
hf-map ta (num_proc=8): 96%|#########6| 643072/667659 [04:02<00:05, 4283.49 examples/s]
hf-map ta (num_proc=8): 96%|#########6| 644096/667659 [04:03<00:09, 2409.66 examples/s]
hf-map ta (num_proc=8): 97%|#########6| 645120/667659 [04:03<00:08, 2788.73 examples/s]
hf-map ta (num_proc=8): 97%|#########6| 646144/667659 [04:03<00:07, 2743.61 examples/s]
hf-map ta (num_proc=8): 97%|#########6| 647168/667659 [04:04<00:06, 3244.58 examples/s]
hf-map ta (num_proc=8): 97%|#########7| 648192/667659 [04:04<00:05, 3491.78 examples/s]
hf-map ta (num_proc=8): 97%|#########7| 649216/667659 [04:04<00:05, 3569.10 examples/s]
hf-map ta (num_proc=8): 97%|#########7| 650240/667659 [04:04<00:04, 3591.43 examples/s]
hf-map ta (num_proc=8): 98%|#########7| 651264/667659 [04:05<00:04, 3735.37 examples/s]
hf-map ta (num_proc=8): 98%|#########7| 652288/667659 [04:05<00:05, 2930.53 examples/s]
hf-map ta (num_proc=8): 98%|#########7| 653312/667659 [04:05<00:04, 3200.61 examples/s]
hf-map ta (num_proc=8): 98%|#########8| 654336/667659 [04:06<00:04, 3032.67 examples/s]
hf-map ta (num_proc=8): 98%|#########8| 655873/667659 [04:06<00:03, 3908.12 examples/s]
hf-map ta (num_proc=8): 98%|#########8| 656897/667659 [04:06<00:02, 3955.46 examples/s]
hf-map ta (num_proc=8): 99%|#########8| 657921/667659 [04:06<00:02, 4435.01 examples/s]
hf-map ta (num_proc=8): 99%|#########8| 658434/667659 [04:07<00:03, 2958.97 examples/s]
hf-map ta (num_proc=8): 99%|#########8| 659458/667659 [04:07<00:02, 3269.23 examples/s]
hf-map ta (num_proc=8): 99%|#########8| 660482/667659 [04:08<00:03, 2259.55 examples/s]
hf-map ta (num_proc=8): 99%|#########9| 662530/667659 [04:08<00:01, 3731.75 examples/s]
hf-map ta (num_proc=8): 99%|#########9| 664068/667659 [04:09<00:01, 3430.39 examples/s]
hf-map ta (num_proc=8): 100%|#########9| 665092/667659 [04:09<00:00, 3726.88 examples/s]
hf-map ta (num_proc=8): 100%|#########9| 666119/667659 [04:09<00:00, 3109.26 examples/s]
hf-map ta (num_proc=8): 100%|#########9| 667146/667659 [04:10<00:00, 2599.27 examples/s]
hf-map ta (num_proc=8): 100%|##########| 667659/667659 [04:10<00:00, 2480.57 examples/s]
hf-map ta (num_proc=8): 100%|##########| 667659/667659 [04:11<00:00, 2659.26 examples/s] |
| ta: datasets.map num_proc=8 done 251s |
| ta: 25000 docs 23325458 tokens 256s |
| ta: 50000 docs 46092495 tokens 258s |
| ta: 75000 docs 69234775 tokens 260s |
| ta: 100000 docs 92178764 tokens 262s |
| ta: 125000 docs 114953253 tokens 264s |
| ta: 150000 docs 138281352 tokens 266s |
| ta: 175000 docs 161404187 tokens 268s |
| ta: 200000 docs 184541026 tokens 270s |
| ta: 225000 docs 207872624 tokens 273s |
| ta: 250000 docs 230922341 tokens 275s |
| ta: 275000 docs 254309793 tokens 277s |
| ta: 300000 docs 277730139 tokens 279s |
| ta: 325000 docs 301150397 tokens 281s |
| ta: 350000 docs 324210916 tokens 283s |
| ta: 375000 docs 346690917 tokens 286s |
| ta: 400000 docs 369202603 tokens 288s |
| ta: 425000 docs 392444860 tokens 290s |
| ta: 450000 docs 416252337 tokens 292s |
| ta: 475000 docs 439421148 tokens 294s |
| packed ta: 456292825 tokens (1.000 of budget) rows=728619 docs=493028 pad-per-doc (HF map) |
| kk: datasets.map n=386470 batched=1024 num_proc=8 |
|
hf-map kk (num_proc=8): 0%| | 0/386470 [00:00<?, ? examples/s]
hf-map kk (num_proc=8): 0%| | 1024/386470 [00:13<1:27:24, 73.50 examples/s]
hf-map kk (num_proc=8): 1%| | 2048/386470 [00:14<38:50, 164.97 examples/s]
hf-map kk (num_proc=8): 2%|1 | 6144/386470 [00:14<09:02, 700.65 examples/s]
hf-map kk (num_proc=8): 2%|2 | 8192/386470 [00:15<05:55, 1065.45 examples/s]
hf-map kk (num_proc=8): 3%|2 | 10240/386470 [00:20<09:34, 654.52 examples/s]
hf-map kk (num_proc=8): 3%|3 | 12288/386470 [00:20<06:48, 916.47 examples/s]
hf-map kk (num_proc=8): 4%|3 | 15360/386470 [00:21<04:08, 1493.18 examples/s]
hf-map kk (num_proc=8): 5%|4 | 17408/386470 [00:24<05:53, 1044.96 examples/s]
hf-map kk (num_proc=8): 5%|4 | 18432/386470 [00:25<05:38, 1085.97 examples/s]
hf-map kk (num_proc=8): 5%|5 | 19456/386470 [00:25<05:06, 1196.74 examples/s]
hf-map kk (num_proc=8): 6%|5 | 21504/386470 [00:25<03:23, 1793.12 examples/s]
hf-map kk (num_proc=8): 6%|6 | 23552/386470 [00:26<02:39, 2271.86 examples/s]
hf-map kk (num_proc=8): 6%|6 | 24576/386470 [00:26<02:15, 2670.27 examples/s]
hf-map kk (num_proc=8): 7%|6 | 25600/386470 [00:29<06:08, 980.30 examples/s]
hf-map kk (num_proc=8): 7%|6 | 26624/386470 [00:30<05:22, 1115.50 examples/s]
hf-map kk (num_proc=8): 7%|7 | 27648/386470 [00:30<04:35, 1301.00 examples/s]
hf-map kk (num_proc=8): 7%|7 | 28672/386470 [00:31<03:42, 1607.32 examples/s]
hf-map kk (num_proc=8): 8%|7 | 30720/386470 [00:31<02:14, 2635.22 examples/s]
hf-map kk (num_proc=8): 8%|8 | 32768/386470 [00:31<01:59, 2948.15 examples/s]
hf-map kk (num_proc=8): 9%|8 | 33792/386470 [00:34<05:11, 1132.08 examples/s]
hf-map kk (num_proc=8): 9%|9 | 34816/386470 [00:34<04:23, 1334.48 examples/s]
hf-map kk (num_proc=8): 9%|9 | 35840/386470 [00:35<04:18, 1356.87 examples/s]
hf-map kk (num_proc=8): 10%|9 | 36864/386470 [00:35<03:22, 1725.55 examples/s]
hf-map kk (num_proc=8): 10%|# | 38912/386470 [00:35<02:03, 2825.20 examples/s]
hf-map kk (num_proc=8): 10%|# | 39936/386470 [00:36<02:04, 2781.68 examples/s]
hf-map kk (num_proc=8): 11%|# | 40960/386470 [00:36<01:52, 3064.07 examples/s]
hf-map kk (num_proc=8): 11%|# | 41984/386470 [00:39<05:50, 983.56 examples/s]
hf-map kk (num_proc=8): 11%|#1 | 43008/386470 [00:39<04:25, 1293.32 examples/s]
hf-map kk (num_proc=8): 11%|#1 | 44032/386470 [00:40<03:56, 1449.78 examples/s]
hf-map kk (num_proc=8): 12%|#1 | 45056/386470 [00:40<03:05, 1844.89 examples/s]
hf-map kk (num_proc=8): 12%|#2 | 48128/386470 [00:40<01:43, 3270.51 examples/s]
hf-map kk (num_proc=8): 13%|#2 | 49152/386470 [00:41<01:41, 3319.47 examples/s]
hf-map kk (num_proc=8): 13%|#2 | 50176/386470 [00:44<05:23, 1038.95 examples/s]
hf-map kk (num_proc=8): 13%|#3 | 51200/386470 [00:44<04:21, 1282.43 examples/s]
hf-map kk (num_proc=8): 14%|#3 | 53248/386470 [00:44<02:39, 2092.97 examples/s]
hf-map kk (num_proc=8): 14%|#4 | 54272/386470 [00:45<02:33, 2164.98 examples/s]
hf-map kk (num_proc=8): 15%|#4 | 57344/386470 [00:45<01:44, 3141.49 examples/s]
hf-map kk (num_proc=8): 15%|#5 | 58368/386470 [00:48<04:53, 1119.55 examples/s]
hf-map kk (num_proc=8): 16%|#5 | 61440/386470 [00:49<02:48, 1932.21 examples/s]
hf-map kk (num_proc=8): 16%|#6 | 62464/386470 [00:49<02:38, 2038.72 examples/s]
hf-map kk (num_proc=8): 16%|#6 | 63488/386470 [00:49<02:17, 2346.18 examples/s]
hf-map kk (num_proc=8): 17%|#6 | 65536/386470 [00:50<01:52, 2850.02 examples/s]
hf-map kk (num_proc=8): 17%|#7 | 66560/386470 [00:53<04:48, 1110.13 examples/s]
hf-map kk (num_proc=8): 17%|#7 | 67584/386470 [00:53<04:07, 1289.70 examples/s]
hf-map kk (num_proc=8): 18%|#8 | 69632/386470 [00:53<02:35, 2034.12 examples/s]
hf-map kk (num_proc=8): 19%|#8 | 71680/386470 [00:54<02:00, 2611.87 examples/s]
hf-map kk (num_proc=8): 19%|#8 | 72704/386470 [00:54<01:56, 2695.09 examples/s]
hf-map kk (num_proc=8): 19%|#9 | 73728/386470 [00:54<01:39, 3131.34 examples/s]
hf-map kk (num_proc=8): 19%|#9 | 74752/386470 [00:57<04:56, 1052.86 examples/s]
hf-map kk (num_proc=8): 20%|#9 | 76800/386470 [00:57<03:00, 1719.23 examples/s]
hf-map kk (num_proc=8): 20%|## | 77824/386470 [00:57<02:33, 2012.41 examples/s]
hf-map kk (num_proc=8): 20%|## | 78848/386470 [00:57<02:04, 2475.90 examples/s]
hf-map kk (num_proc=8): 21%|## | 79872/386470 [00:58<01:50, 2778.64 examples/s]
hf-map kk (num_proc=8): 21%|## | 80896/386470 [00:58<01:59, 2554.15 examples/s]
hf-map kk (num_proc=8): 21%|##1 | 81920/386470 [00:58<01:43, 2944.49 examples/s]
hf-map kk (num_proc=8): 21%|##1 | 82944/386470 [01:01<04:32, 1113.93 examples/s]
hf-map kk (num_proc=8): 22%|##1 | 83968/386470 [01:01<03:29, 1441.38 examples/s]
hf-map kk (num_proc=8): 22%|##1 | 84992/386470 [01:01<02:49, 1777.49 examples/s]
hf-map kk (num_proc=8): 22%|##2 | 86016/386470 [01:01<02:17, 2183.23 examples/s]
hf-map kk (num_proc=8): 23%|##2 | 87040/386470 [01:02<01:46, 2799.24 examples/s]
hf-map kk (num_proc=8): 23%|##2 | 88064/386470 [01:02<01:53, 2618.30 examples/s]
hf-map kk (num_proc=8): 23%|##3 | 89088/386470 [01:02<01:44, 2846.58 examples/s]
hf-map kk (num_proc=8): 23%|##3 | 90112/386470 [01:03<01:35, 3091.38 examples/s]
hf-map kk (num_proc=8): 24%|##3 | 91136/386470 [01:05<04:05, 1203.56 examples/s]
hf-map kk (num_proc=8): 24%|##3 | 92160/386470 [01:05<03:42, 1325.25 examples/s]
hf-map kk (num_proc=8): 24%|##4 | 94208/386470 [01:05<02:14, 2175.49 examples/s]
hf-map kk (num_proc=8): 25%|##4 | 95232/386470 [01:06<01:56, 2505.50 examples/s]
hf-map kk (num_proc=8): 25%|##4 | 96256/386470 [01:06<02:07, 2275.47 examples/s]
hf-map kk (num_proc=8): 25%|##5 | 98304/386470 [01:06<01:20, 3580.45 examples/s]
hf-map kk (num_proc=8): 26%|##5 | 99328/386470 [01:09<03:37, 1318.81 examples/s]
hf-map kk (num_proc=8): 26%|##5 | 100352/386470 [01:09<02:59, 1596.51 examples/s]
hf-map kk (num_proc=8): 26%|##6 | 101376/386470 [01:09<02:23, 1989.13 examples/s]
hf-map kk (num_proc=8): 26%|##6 | 102400/386470 [01:09<02:09, 2194.40 examples/s]
hf-map kk (num_proc=8): 27%|##6 | 103424/386470 [01:10<02:03, 2290.34 examples/s]
hf-map kk (num_proc=8): 27%|##7 | 104448/386470 [01:10<01:38, 2851.88 examples/s]
hf-map kk (num_proc=8): 28%|##7 | 106496/386470 [01:10<01:05, 4250.97 examples/s]
hf-map kk (num_proc=8): 28%|##7 | 107520/386470 [01:13<03:33, 1304.65 examples/s]
hf-map kk (num_proc=8): 28%|##8 | 109568/386470 [01:13<02:21, 1960.78 examples/s]
hf-map kk (num_proc=8): 29%|##8 | 110592/386470 [01:13<02:19, 1976.68 examples/s]
hf-map kk (num_proc=8): 29%|##8 | 111616/386470 [01:14<02:21, 1944.74 examples/s]
hf-map kk (num_proc=8): 29%|##9 | 113664/386470 [01:14<01:37, 2784.24 examples/s]
hf-map kk (num_proc=8): 30%|##9 | 115712/386470 [01:16<02:35, 1740.61 examples/s]
hf-map kk (num_proc=8): 30%|### | 116736/386470 [01:16<02:12, 2039.44 examples/s]
hf-map kk (num_proc=8): 30%|### | 117760/386470 [01:17<02:23, 1866.54 examples/s]
hf-map kk (num_proc=8): 31%|### | 118784/386470 [01:17<02:03, 2161.10 examples/s]
hf-map kk (num_proc=8): 31%|###1 | 119808/386470 [01:18<01:52, 2367.72 examples/s]
hf-map kk (num_proc=8): 31%|###1 | 120832/386470 [01:18<01:33, 2852.06 examples/s]
hf-map kk (num_proc=8): 32%|###1 | 121856/386470 [01:18<01:42, 2576.70 examples/s]
hf-map kk (num_proc=8): 32%|###1 | 122880/386470 [01:19<01:33, 2820.25 examples/s]
hf-map kk (num_proc=8): 32%|###2 | 123904/386470 [01:20<02:38, 1660.01 examples/s]
hf-map kk (num_proc=8): 33%|###2 | 125952/386470 [01:21<02:03, 2104.10 examples/s]
hf-map kk (num_proc=8): 33%|###2 | 126976/386470 [01:21<02:08, 2021.72 examples/s]
hf-map kk (num_proc=8): 33%|###3 | 129024/386470 [01:22<01:36, 2665.31 examples/s]
hf-map kk (num_proc=8): 34%|###3 | 130048/386470 [01:22<01:45, 2436.04 examples/s]
hf-map kk (num_proc=8): 34%|###4 | 132096/386470 [01:23<02:09, 1970.71 examples/s]
hf-map kk (num_proc=8): 35%|###4 | 134144/386470 [01:24<02:02, 2057.95 examples/s]
hf-map kk (num_proc=8): 35%|###4 | 135168/386470 [01:24<01:43, 2422.96 examples/s]
hf-map kk (num_proc=8): 36%|###5 | 137216/386470 [01:25<01:19, 3143.38 examples/s]
hf-map kk (num_proc=8): 36%|###5 | 138240/386470 [01:26<01:43, 2403.10 examples/s]
hf-map kk (num_proc=8): 36%|###6 | 139264/386470 [01:26<01:28, 2797.15 examples/s]
hf-map kk (num_proc=8): 36%|###6 | 140288/386470 [01:27<01:52, 2179.11 examples/s]
hf-map kk (num_proc=8): 37%|###6 | 141312/386470 [01:27<01:36, 2543.62 examples/s]
hf-map kk (num_proc=8): 37%|###6 | 142336/386470 [01:28<02:24, 1690.77 examples/s]
hf-map kk (num_proc=8): 37%|###7 | 143360/386470 [01:28<01:50, 2196.51 examples/s]
hf-map kk (num_proc=8): 37%|###7 | 144384/386470 [01:28<01:30, 2677.78 examples/s]
hf-map kk (num_proc=8): 38%|###7 | 146432/386470 [01:29<01:25, 2818.89 examples/s]
hf-map kk (num_proc=8): 38%|###8 | 147456/386470 [01:29<01:29, 2683.43 examples/s]
hf-map kk (num_proc=8): 38%|###8 | 148480/386470 [01:30<01:39, 2388.00 examples/s]
hf-map kk (num_proc=8): 39%|###8 | 149504/386470 [01:30<01:44, 2266.38 examples/s]
hf-map kk (num_proc=8): 39%|###8 | 150528/386470 [01:31<02:08, 1842.23 examples/s]
hf-map kk (num_proc=8): 39%|###9 | 152576/386470 [01:32<01:27, 2662.83 examples/s]
hf-map kk (num_proc=8): 40%|#### | 154624/386470 [01:32<01:25, 2718.53 examples/s]
hf-map kk (num_proc=8): 40%|#### | 155648/386470 [01:33<01:27, 2629.08 examples/s]
hf-map kk (num_proc=8): 41%|#### | 156672/386470 [01:34<01:51, 2064.80 examples/s]
hf-map kk (num_proc=8): 41%|####1 | 158720/386470 [01:34<01:42, 2222.77 examples/s]
hf-map kk (num_proc=8): 41%|####1 | 159744/386470 [01:35<01:36, 2339.18 examples/s]
hf-map kk (num_proc=8): 42%|####1 | 160768/386470 [01:35<01:22, 2723.53 examples/s]
hf-map kk (num_proc=8): 42%|####2 | 162816/386470 [01:36<01:18, 2841.45 examples/s]
hf-map kk (num_proc=8): 42%|####2 | 163840/386470 [01:36<01:23, 2650.97 examples/s]
hf-map kk (num_proc=8): 43%|####2 | 164864/386470 [01:37<01:35, 2309.12 examples/s]
hf-map kk (num_proc=8): 43%|####2 | 165888/386470 [01:37<01:22, 2658.35 examples/s]
hf-map kk (num_proc=8): 43%|####3 | 166912/386470 [01:38<01:50, 1979.99 examples/s]
hf-map kk (num_proc=8): 43%|####3 | 167936/386470 [01:38<01:41, 2147.85 examples/s]
hf-map kk (num_proc=8): 44%|####3 | 168960/386470 [01:38<01:27, 2495.68 examples/s]
hf-map kk (num_proc=8): 44%|####4 | 171008/386470 [01:39<01:10, 3061.33 examples/s]
hf-map kk (num_proc=8): 45%|####4 | 172032/386470 [01:39<01:18, 2731.42 examples/s]
hf-map kk (num_proc=8): 45%|####4 | 173056/386470 [01:40<01:44, 2036.23 examples/s]
hf-map kk (num_proc=8): 45%|####5 | 174080/386470 [01:40<01:25, 2497.14 examples/s]
hf-map kk (num_proc=8): 45%|####5 | 175104/386470 [01:41<01:52, 1880.88 examples/s]
hf-map kk (num_proc=8): 46%|####5 | 177152/386470 [01:42<01:20, 2609.13 examples/s]
hf-map kk (num_proc=8): 46%|####6 | 178176/386470 [01:42<01:11, 2923.25 examples/s]
hf-map kk (num_proc=8): 46%|####6 | 179200/386470 [01:42<01:05, 3188.24 examples/s]
hf-map kk (num_proc=8): 47%|####6 | 180224/386470 [01:43<01:23, 2460.20 examples/s]
hf-map kk (num_proc=8): 47%|####6 | 181248/386470 [01:43<01:06, 3107.32 examples/s]
hf-map kk (num_proc=8): 47%|####7 | 182272/386470 [01:44<01:26, 2354.94 examples/s]
hf-map kk (num_proc=8): 47%|####7 | 183296/386470 [01:44<01:30, 2240.05 examples/s]
hf-map kk (num_proc=8): 48%|####7 | 184320/386470 [01:44<01:19, 2536.04 examples/s]
hf-map kk (num_proc=8): 48%|####7 | 185344/386470 [01:45<01:19, 2542.54 examples/s]
hf-map kk (num_proc=8): 48%|####8 | 186368/386470 [01:45<01:14, 2684.49 examples/s]
hf-map kk (num_proc=8): 48%|####8 | 187392/386470 [01:45<01:12, 2745.50 examples/s]
hf-map kk (num_proc=8): 49%|####8 | 188416/386470 [01:46<01:17, 2560.22 examples/s]
hf-map kk (num_proc=8): 49%|####9 | 189440/386470 [01:46<01:10, 2795.96 examples/s]
hf-map kk (num_proc=8): 49%|####9 | 190464/386470 [01:47<01:26, 2259.37 examples/s]
hf-map kk (num_proc=8): 50%|####9 | 191488/386470 [01:47<01:17, 2511.28 examples/s]
hf-map kk (num_proc=8): 50%|####9 | 192512/386470 [01:48<01:19, 2442.65 examples/s]
hf-map kk (num_proc=8): 50%|##### | 193536/386470 [01:48<01:05, 2946.38 examples/s]
hf-map kk (num_proc=8): 50%|##### | 194560/386470 [01:48<01:14, 2589.37 examples/s]
hf-map kk (num_proc=8): 51%|##### | 195584/386470 [01:49<01:13, 2580.28 examples/s]
hf-map kk (num_proc=8): 51%|##### | 196608/386470 [01:49<01:16, 2495.68 examples/s]
hf-map kk (num_proc=8): 51%|#####1 | 197632/386470 [01:49<01:08, 2741.18 examples/s]
hf-map kk (num_proc=8): 51%|#####1 | 198656/386470 [01:50<01:36, 1938.52 examples/s]
hf-map kk (num_proc=8): 52%|#####1 | 199680/386470 [01:51<01:20, 2318.39 examples/s]
hf-map kk (num_proc=8): 52%|#####1 | 200704/386470 [01:51<01:12, 2565.01 examples/s]
hf-map kk (num_proc=8): 52%|#####2 | 201728/386470 [01:51<01:00, 3053.11 examples/s]
hf-map kk (num_proc=8): 52%|#####2 | 202752/386470 [01:51<01:03, 2902.05 examples/s]
hf-map kk (num_proc=8): 53%|#####2 | 203776/386470 [01:52<01:05, 2782.61 examples/s]
hf-map kk (num_proc=8): 53%|#####2 | 204800/386470 [01:52<01:13, 2462.08 examples/s]
hf-map kk (num_proc=8): 53%|#####3 | 205824/386470 [01:53<01:19, 2261.91 examples/s]
hf-map kk (num_proc=8): 54%|#####3 | 206848/386470 [01:53<01:17, 2310.94 examples/s]
hf-map kk (num_proc=8): 54%|#####3 | 207872/386470 [01:54<01:09, 2588.10 examples/s]
hf-map kk (num_proc=8): 54%|#####4 | 208896/386470 [01:54<01:18, 2256.35 examples/s]
hf-map kk (num_proc=8): 54%|#####4 | 209920/386470 [01:54<01:02, 2843.29 examples/s]
hf-map kk (num_proc=8): 55%|#####4 | 210944/386470 [01:55<01:10, 2480.34 examples/s]
hf-map kk (num_proc=8): 55%|#####4 | 211968/386470 [01:55<01:18, 2230.62 examples/s]
hf-map kk (num_proc=8): 55%|#####5 | 212992/386470 [01:56<01:16, 2261.80 examples/s]
hf-map kk (num_proc=8): 55%|#####5 | 214016/386470 [01:56<01:21, 2107.79 examples/s]
hf-map kk (num_proc=8): 56%|#####5 | 215040/386470 [01:57<01:02, 2732.98 examples/s]
hf-map kk (num_proc=8): 56%|#####5 | 216064/386470 [01:57<01:11, 2385.76 examples/s]
hf-map kk (num_proc=8): 56%|#####6 | 217088/386470 [01:58<01:09, 2423.54 examples/s]
hf-map kk (num_proc=8): 56%|#####6 | 218112/386470 [01:58<01:00, 2799.65 examples/s]
hf-map kk (num_proc=8): 57%|#####6 | 220160/386470 [01:59<01:35, 1739.16 examples/s]
hf-map kk (num_proc=8): 57%|#####7 | 221184/386470 [02:00<01:21, 2022.93 examples/s]
hf-map kk (num_proc=8): 57%|#####7 | 222208/386470 [02:00<01:08, 2396.76 examples/s]
hf-map kk (num_proc=8): 58%|#####8 | 224256/386470 [02:01<01:03, 2573.51 examples/s]
hf-map kk (num_proc=8): 59%|#####8 | 226304/386470 [02:01<00:45, 3485.89 examples/s]
hf-map kk (num_proc=8): 59%|#####8 | 227328/386470 [02:01<00:47, 3372.18 examples/s]
hf-map kk (num_proc=8): 59%|#####9 | 228352/386470 [02:03<01:31, 1733.88 examples/s]
hf-map kk (num_proc=8): 60%|#####9 | 230400/386470 [02:03<01:12, 2148.15 examples/s]
hf-map kk (num_proc=8): 60%|###### | 232448/386470 [02:04<01:02, 2445.85 examples/s]
hf-map kk (num_proc=8): 60%|###### | 233472/386470 [02:04<00:56, 2699.96 examples/s]
hf-map kk (num_proc=8): 61%|###### | 234496/386470 [02:04<00:47, 3166.55 examples/s]
hf-map kk (num_proc=8): 61%|###### | 235520/386470 [02:05<00:54, 2794.61 examples/s]
hf-map kk (num_proc=8): 61%|######1 | 236544/386470 [02:06<01:17, 1930.31 examples/s]
hf-map kk (num_proc=8): 61%|######1 | 237568/386470 [02:06<01:04, 2293.26 examples/s]
hf-map kk (num_proc=8): 62%|######1 | 238592/386470 [02:07<01:11, 2059.62 examples/s]
hf-map kk (num_proc=8): 62%|######2 | 239616/386470 [02:07<01:07, 2186.21 examples/s]
hf-map kk (num_proc=8): 63%|######2 | 241664/386470 [02:08<00:49, 2954.11 examples/s]
hf-map kk (num_proc=8): 63%|######3 | 243712/386470 [02:08<00:42, 3358.76 examples/s]
hf-map kk (num_proc=8): 63%|######3 | 244736/386470 [02:09<00:58, 2430.56 examples/s]
hf-map kk (num_proc=8): 64%|######3 | 245760/386470 [02:09<00:52, 2684.90 examples/s]
hf-map kk (num_proc=8): 64%|######3 | 246784/386470 [02:10<01:09, 2015.62 examples/s]
hf-map kk (num_proc=8): 64%|######4 | 247808/386470 [02:10<01:02, 2235.62 examples/s]
hf-map kk (num_proc=8): 64%|######4 | 248832/386470 [02:10<00:49, 2802.29 examples/s]
hf-map kk (num_proc=8): 65%|######4 | 249856/386470 [02:11<00:47, 2874.18 examples/s]
hf-map kk (num_proc=8): 65%|######4 | 250880/386470 [02:11<00:40, 3371.23 examples/s]
hf-map kk (num_proc=8): 65%|######5 | 251904/386470 [02:11<00:40, 3307.89 examples/s]
hf-map kk (num_proc=8): 65%|######5 | 252928/386470 [02:12<00:57, 2337.37 examples/s]
hf-map kk (num_proc=8): 66%|######5 | 253952/386470 [02:13<00:59, 2224.09 examples/s]
hf-map kk (num_proc=8): 66%|######5 | 254976/386470 [02:13<01:06, 1969.98 examples/s]
hf-map kk (num_proc=8): 66%|######6 | 256000/386470 [02:14<01:00, 2164.73 examples/s]
hf-map kk (num_proc=8): 67%|######6 | 258048/386470 [02:14<00:48, 2665.29 examples/s]
hf-map kk (num_proc=8): 67%|######7 | 259072/386470 [02:14<00:41, 3048.57 examples/s]
hf-map kk (num_proc=8): 67%|######7 | 260096/386470 [02:14<00:35, 3556.39 examples/s]
hf-map kk (num_proc=8): 68%|######7 | 261120/386470 [02:15<00:42, 2950.48 examples/s]
hf-map kk (num_proc=8): 68%|######7 | 262144/386470 [02:15<00:46, 2688.99 examples/s]
hf-map kk (num_proc=8): 68%|######8 | 263168/386470 [02:16<00:58, 2111.68 examples/s]
hf-map kk (num_proc=8): 68%|######8 | 264192/386470 [02:17<01:04, 1888.06 examples/s]
hf-map kk (num_proc=8): 69%|######8 | 265216/386470 [02:17<00:50, 2396.85 examples/s]
hf-map kk (num_proc=8): 69%|######9 | 267264/386470 [02:18<00:46, 2549.89 examples/s]
hf-map kk (num_proc=8): 70%|######9 | 269312/386470 [02:18<00:39, 2989.00 examples/s]
hf-map kk (num_proc=8): 70%|######9 | 270336/386470 [02:19<00:42, 2748.76 examples/s]
hf-map kk (num_proc=8): 70%|####### | 271360/386470 [02:20<00:54, 2123.21 examples/s]
hf-map kk (num_proc=8): 70%|####### | 272384/386470 [02:20<00:55, 2058.14 examples/s]
hf-map kk (num_proc=8): 71%|####### | 273408/386470 [02:20<00:49, 2272.89 examples/s]
hf-map kk (num_proc=8): 71%|#######1 | 274432/386470 [02:21<00:41, 2715.98 examples/s]
hf-map kk (num_proc=8): 71%|#######1 | 275456/386470 [02:21<00:39, 2782.17 examples/s]
hf-map kk (num_proc=8): 72%|#######1 | 277504/386470 [02:22<00:35, 3113.04 examples/s]
hf-map kk (num_proc=8): 72%|#######2 | 278528/386470 [02:22<00:36, 2931.94 examples/s]
hf-map kk (num_proc=8): 72%|#######2 | 279552/386470 [02:23<00:54, 1951.74 examples/s]
hf-map kk (num_proc=8): 73%|#######2 | 280576/386470 [02:23<00:44, 2373.18 examples/s]
hf-map kk (num_proc=8): 73%|#######2 | 281600/386470 [02:24<00:45, 2312.16 examples/s]
hf-map kk (num_proc=8): 73%|#######3 | 282624/386470 [02:24<00:37, 2749.52 examples/s]
hf-map kk (num_proc=8): 73%|#######3 | 283648/386470 [02:24<00:33, 3046.97 examples/s]
hf-map kk (num_proc=8): 74%|#######3 | 284672/386470 [02:24<00:31, 3202.40 examples/s]
hf-map kk (num_proc=8): 74%|#######3 | 285696/386470 [02:24<00:26, 3843.27 examples/s]
hf-map kk (num_proc=8): 74%|#######4 | 286720/386470 [02:25<00:32, 3063.18 examples/s]
hf-map kk (num_proc=8): 74%|#######4 | 287744/386470 [02:26<00:53, 1837.16 examples/s]
hf-map kk (num_proc=8): 75%|#######4 | 289792/386470 [02:27<00:41, 2311.58 examples/s]
hf-map kk (num_proc=8): 75%|#######5 | 290816/386470 [02:27<00:40, 2341.67 examples/s]
hf-map kk (num_proc=8): 76%|#######5 | 292864/386470 [02:27<00:29, 3126.50 examples/s]
hf-map kk (num_proc=8): 76%|#######6 | 294912/386470 [02:28<00:30, 2996.70 examples/s]
hf-map kk (num_proc=8): 77%|#######6 | 295936/386470 [02:29<00:39, 2305.95 examples/s]
hf-map kk (num_proc=8): 77%|#######6 | 296960/386470 [02:29<00:33, 2668.50 examples/s]
hf-map kk (num_proc=8): 77%|#######7 | 297984/386470 [02:30<00:43, 2052.21 examples/s]
hf-map kk (num_proc=8): 78%|#######7 | 300032/386470 [02:30<00:27, 3141.02 examples/s]
hf-map kk (num_proc=8): 78%|#######8 | 302080/386470 [02:31<00:26, 3173.89 examples/s]
hf-map kk (num_proc=8): 79%|#######8 | 304128/386470 [02:32<00:36, 2264.10 examples/s]
hf-map kk (num_proc=8): 79%|#######8 | 305152/386470 [02:32<00:33, 2437.36 examples/s]
hf-map kk (num_proc=8): 79%|#######9 | 306176/386470 [02:33<00:40, 1958.95 examples/s]
hf-map kk (num_proc=8): 80%|######## | 309248/386470 [02:34<00:22, 3363.98 examples/s]
hf-map kk (num_proc=8): 80%|######## | 310272/386470 [02:34<00:22, 3454.48 examples/s]
hf-map kk (num_proc=8): 81%|######## | 311296/386470 [02:34<00:25, 2920.61 examples/s]
hf-map kk (num_proc=8): 81%|######## | 312320/386470 [02:36<00:40, 1832.97 examples/s]
hf-map kk (num_proc=8): 81%|########1 | 313344/386470 [02:36<00:36, 2023.79 examples/s]
hf-map kk (num_proc=8): 81%|########1 | 314368/386470 [02:36<00:34, 2108.83 examples/s]
hf-map kk (num_proc=8): 82%|########1 | 315392/386470 [02:37<00:27, 2608.92 examples/s]
hf-map kk (num_proc=8): 82%|########2 | 317440/386470 [02:37<00:17, 3855.09 examples/s]
hf-map kk (num_proc=8): 82%|########2 | 318464/386470 [02:38<00:26, 2613.62 examples/s]
hf-map kk (num_proc=8): 83%|########2 | 319488/386470 [02:38<00:26, 2550.80 examples/s]
hf-map kk (num_proc=8): 83%|########2 | 320512/386470 [02:39<00:34, 1927.63 examples/s]
hf-map kk (num_proc=8): 83%|########3 | 321536/386470 [02:39<00:29, 2227.43 examples/s]
hf-map kk (num_proc=8): 83%|########3 | 322560/386470 [02:40<00:28, 2265.67 examples/s]
hf-map kk (num_proc=8): 84%|########3 | 324608/386470 [02:40<00:19, 3190.75 examples/s]
hf-map kk (num_proc=8): 85%|########4 | 326656/386470 [02:41<00:20, 2879.39 examples/s]
hf-map kk (num_proc=8): 85%|########4 | 327680/386470 [02:42<00:27, 2163.88 examples/s]
hf-map kk (num_proc=8): 85%|########5 | 328704/386470 [02:42<00:29, 1940.02 examples/s]
hf-map kk (num_proc=8): 86%|########5 | 330752/386470 [02:43<00:21, 2649.87 examples/s]
hf-map kk (num_proc=8): 86%|########5 | 331776/386470 [02:43<00:17, 3106.14 examples/s]
hf-map kk (num_proc=8): 86%|########6 | 332800/386470 [02:43<00:15, 3504.84 examples/s]
hf-map kk (num_proc=8): 86%|########6 | 333824/386470 [02:43<00:13, 3969.77 examples/s]
hf-map kk (num_proc=8): 87%|########6 | 334848/386470 [02:45<00:30, 1674.19 examples/s]
hf-map kk (num_proc=8): 87%|########6 | 335872/386470 [02:45<00:23, 2156.26 examples/s]
hf-map kk (num_proc=8): 87%|########7 | 336896/386470 [02:46<00:27, 1823.73 examples/s]
hf-map kk (num_proc=8): 87%|########7 | 337920/386470 [02:46<00:26, 1810.99 examples/s]
hf-map kk (num_proc=8): 88%|########7 | 338944/386470 [02:46<00:22, 2134.70 examples/s]
hf-map kk (num_proc=8): 88%|########7 | 339968/386470 [02:47<00:21, 2197.47 examples/s]
hf-map kk (num_proc=8): 89%|########8 | 343040/386470 [02:48<00:17, 2443.29 examples/s]
hf-map kk (num_proc=8): 89%|########9 | 344064/386470 [02:48<00:14, 2886.87 examples/s]
hf-map kk (num_proc=8): 89%|########9 | 345088/386470 [02:49<00:21, 1953.38 examples/s]
hf-map kk (num_proc=8): 90%|########9 | 346112/386470 [02:50<00:19, 2044.12 examples/s]
hf-map kk (num_proc=8): 90%|########9 | 347136/386470 [02:50<00:17, 2281.40 examples/s]
hf-map kk (num_proc=8): 90%|######### | 348160/386470 [02:50<00:13, 2797.28 examples/s]
hf-map kk (num_proc=8): 90%|######### | 349184/386470 [02:50<00:12, 3076.47 examples/s]
hf-map kk (num_proc=8): 91%|######### | 350208/386470 [02:51<00:09, 3732.38 examples/s]
hf-map kk (num_proc=8): 91%|######### | 351232/386470 [02:51<00:13, 2690.88 examples/s]
hf-map kk (num_proc=8): 91%|#########1| 352256/386470 [02:51<00:11, 2890.45 examples/s]
hf-map kk (num_proc=8): 91%|#########1| 353280/386470 [02:53<00:21, 1529.03 examples/s]
hf-map kk (num_proc=8): 92%|#########1| 354304/386470 [02:53<00:19, 1677.19 examples/s]
hf-map kk (num_proc=8): 92%|#########2| 356352/386470 [02:54<00:12, 2487.70 examples/s]
hf-map kk (num_proc=8): 92%|#########2| 357376/386470 [02:54<00:09, 3019.36 examples/s]
hf-map kk (num_proc=8): 93%|#########2| 358400/386470 [02:54<00:09, 2920.16 examples/s]
hf-map kk (num_proc=8): 93%|#########3| 359424/386470 [02:54<00:08, 3332.41 examples/s]
hf-map kk (num_proc=8): 93%|#########3| 360448/386470 [02:55<00:08, 2905.96 examples/s]
hf-map kk (num_proc=8): 94%|#########3| 361472/386470 [02:56<00:16, 1495.15 examples/s]
hf-map kk (num_proc=8): 94%|#########3| 362496/386470 [02:57<00:13, 1742.61 examples/s]
hf-map kk (num_proc=8): 94%|#########4| 363520/386470 [02:57<00:10, 2116.65 examples/s]
hf-map kk (num_proc=8): 95%|#########4| 365568/386470 [02:57<00:06, 3296.48 examples/s]
hf-map kk (num_proc=8): 95%|#########4| 366592/386470 [02:57<00:05, 3578.60 examples/s]
hf-map kk (num_proc=8): 95%|#########5| 367616/386470 [02:58<00:04, 3826.08 examples/s]
hf-map kk (num_proc=8): 95%|#########5| 368640/386470 [02:58<00:05, 2981.63 examples/s]
hf-map kk (num_proc=8): 96%|#########5| 369664/386470 [03:00<00:11, 1520.69 examples/s]
hf-map kk (num_proc=8): 96%|#########5| 370688/386470 [03:00<00:09, 1679.72 examples/s]
hf-map kk (num_proc=8): 96%|#########6| 372736/386470 [03:00<00:05, 2735.75 examples/s]
hf-map kk (num_proc=8): 97%|#########6| 373760/386470 [03:01<00:04, 2867.58 examples/s]
hf-map kk (num_proc=8): 97%|#########6| 374784/386470 [03:01<00:05, 2172.29 examples/s]
hf-map kk (num_proc=8): 98%|#########7| 376832/386470 [03:02<00:03, 2965.05 examples/s]
hf-map kk (num_proc=8): 98%|#########7| 378037/386470 [03:03<00:04, 1710.54 examples/s]
hf-map kk (num_proc=8): 98%|#########8| 379061/386470 [03:03<00:03, 2138.73 examples/s]
hf-map kk (num_proc=8): 98%|#########8| 380085/386470 [03:03<00:02, 2649.09 examples/s]
hf-map kk (num_proc=8): 99%|#########8| 381109/386470 [03:04<00:01, 3190.75 examples/s]
hf-map kk (num_proc=8): 99%|#########8| 382133/386470 [03:04<00:01, 3334.09 examples/s]
hf-map kk (num_proc=8): 99%|#########9| 383700/386470 [03:04<00:00, 3813.59 examples/s]
hf-map kk (num_proc=8): 100%|#########9| 385085/386470 [03:05<00:00, 3200.81 examples/s]
hf-map kk (num_proc=8): 100%|#########9| 386290/386470 [03:06<00:00, 1927.83 examples/s]
hf-map kk (num_proc=8): 100%|##########| 386470/386470 [03:07<00:00, 2060.79 examples/s] |
| kk: datasets.map num_proc=8 done 188s |
| kk: 25000 docs 40985213 tokens 193s |
| kk: 50000 docs 81894926 tokens 197s |
| kk: 75000 docs 121953883 tokens 200s |
| kk: 100000 docs 163346527 tokens 204s |
| kk: 125000 docs 204439098 tokens 207s |
| kk: 150000 docs 245100472 tokens 210s |
| kk: 175000 docs 284907275 tokens 214s |
| kk: 200000 docs 326211845 tokens 217s |
| kk: 225000 docs 368265583 tokens 221s |
| kk: 250000 docs 408948851 tokens 224s |
| kk: 275000 docs 449889902 tokens 227s |
| packed kk: 466196065 tokens (1.000 of budget) rows=602321 docs=285249 pad-per-doc (HF map) |
| tl: datasets.map n=50150 batched=1024 num_proc=8 |
|
hf-map tl (num_proc=8): 0%| | 0/50150 [00:00<?, ? examples/s]
hf-map tl (num_proc=8): 2%|2 | 1024/50150 [00:06<04:47, 170.67 examples/s]
hf-map tl (num_proc=8): 8%|8 | 4096/50150 [00:06<00:52, 875.04 examples/s]
hf-map tl (num_proc=8): 14%|#4 | 7168/50150 [00:06<00:24, 1789.33 examples/s]
hf-map tl (num_proc=8): 18%|#8 | 9216/50150 [00:08<00:30, 1359.78 examples/s]
hf-map tl (num_proc=8): 25%|##4 | 12288/50150 [00:08<00:16, 2266.57 examples/s]
hf-map tl (num_proc=8): 29%|##8 | 14336/50150 [00:08<00:11, 3011.36 examples/s]
hf-map tl (num_proc=8): 33%|###2 | 16384/50150 [00:08<00:08, 3800.45 examples/s]
hf-map tl (num_proc=8): 37%|###6 | 18432/50150 [00:10<00:15, 2113.45 examples/s]
hf-map tl (num_proc=8): 45%|####4 | 22528/50150 [00:11<00:07, 3643.83 examples/s]
hf-map tl (num_proc=8): 49%|####9 | 24576/50150 [00:11<00:05, 4355.40 examples/s]
hf-map tl (num_proc=8): 53%|#####3 | 26624/50150 [00:12<00:08, 2616.28 examples/s]
hf-map tl (num_proc=8): 57%|#####7 | 28672/50150 [00:13<00:06, 3164.06 examples/s]
hf-map tl (num_proc=8): 63%|######3 | 31744/50150 [00:13<00:03, 4609.14 examples/s]
hf-map tl (num_proc=8): 67%|######7 | 33792/50150 [00:15<00:06, 2689.02 examples/s]
hf-map tl (num_proc=8): 69%|######9 | 34816/50150 [00:15<00:05, 2799.48 examples/s]
hf-map tl (num_proc=8): 76%|#######5 | 37888/50150 [00:15<00:02, 4384.19 examples/s]
hf-map tl (num_proc=8): 80%|#######9 | 39936/50150 [00:15<00:02, 4976.26 examples/s]
hf-map tl (num_proc=8): 84%|########3 | 41984/50150 [00:16<00:02, 3212.45 examples/s]
hf-map tl (num_proc=8): 86%|########6 | 43133/50150 [00:17<00:02, 3249.85 examples/s]
hf-map tl (num_proc=8): 90%|######### | 45181/50150 [00:17<00:01, 4234.28 examples/s]
hf-map tl (num_proc=8): 95%|#########4| 47478/50150 [00:17<00:00, 5464.28 examples/s]
hf-map tl (num_proc=8): 99%|#########9| 49775/50150 [00:17<00:00, 6862.21 examples/s]
hf-map tl (num_proc=8): 100%|##########| 50150/50150 [00:18<00:00, 2760.63 examples/s] |
| tl: datasets.map num_proc=8 done 18s |
| tl: 25000 docs 22165685 tokens 21s |
| packed tl: 32043235 tokens (1.000 of budget) rows=55838 docs=36428 pad-per-doc (HF map) |
| en: datasets.map n=196400 batched=1024 num_proc=8 |
|
hf-map en (num_proc=8): 0%| | 0/196400 [00:00<?, ? examples/s]
hf-map en (num_proc=8): 1%| | 1024/196400 [00:12<40:14, 80.91 examples/s]
hf-map en (num_proc=8): 1%|1 | 2048/196400 [00:12<16:43, 193.73 examples/s]
hf-map en (num_proc=8): 2%|2 | 4096/196400 [00:13<06:23, 500.83 examples/s]
hf-map en (num_proc=8): 3%|3 | 6144/196400 [00:13<03:32, 895.24 examples/s]
hf-map en (num_proc=8): 5%|4 | 9216/196400 [00:16<03:32, 879.41 examples/s]
hf-map en (num_proc=8): 6%|5 | 11264/196400 [00:17<02:26, 1260.46 examples/s]
hf-map en (num_proc=8): 6%|6 | 12288/196400 [00:17<02:09, 1421.44 examples/s]
hf-map en (num_proc=8): 7%|6 | 13312/196400 [00:17<01:47, 1705.39 examples/s]
hf-map en (num_proc=8): 8%|7 | 15360/196400 [00:17<01:10, 2574.29 examples/s]
hf-map en (num_proc=8): 9%|8 | 17408/196400 [00:21<02:42, 1102.69 examples/s]
hf-map en (num_proc=8): 10%|# | 20480/196400 [00:21<01:34, 1862.45 examples/s]
hf-map en (num_proc=8): 11%|#1 | 22528/196400 [00:21<01:09, 2495.28 examples/s]
hf-map en (num_proc=8): 13%|#2 | 24576/196400 [00:21<00:51, 3335.75 examples/s]
hf-map en (num_proc=8): 14%|#3 | 26624/196400 [00:25<01:59, 1417.96 examples/s]
hf-map en (num_proc=8): 15%|#5 | 29696/196400 [00:25<01:14, 2241.43 examples/s]
hf-map en (num_proc=8): 16%|#6 | 31744/196400 [00:25<01:00, 2700.21 examples/s]
hf-map en (num_proc=8): 17%|#7 | 33792/196400 [00:29<02:01, 1341.40 examples/s]
hf-map en (num_proc=8): 18%|#8 | 35840/196400 [00:29<01:28, 1812.85 examples/s]
hf-map en (num_proc=8): 19%|#9 | 37888/196400 [00:29<01:07, 2349.23 examples/s]
hf-map en (num_proc=8): 21%|##1 | 41984/196400 [00:32<01:31, 1683.67 examples/s]
hf-map en (num_proc=8): 22%|##1 | 43008/196400 [00:32<01:20, 1898.44 examples/s]
hf-map en (num_proc=8): 23%|##3 | 46080/196400 [00:33<00:51, 2917.22 examples/s]
hf-map en (num_proc=8): 25%|##4 | 48128/196400 [00:33<00:41, 3534.71 examples/s]
hf-map en (num_proc=8): 26%|##5 | 50176/196400 [00:36<01:26, 1682.03 examples/s]
hf-map en (num_proc=8): 27%|##7 | 53248/196400 [00:36<00:56, 2543.31 examples/s]
hf-map en (num_proc=8): 28%|##8 | 55296/196400 [00:36<00:44, 3146.69 examples/s]
hf-map en (num_proc=8): 30%|##9 | 58368/196400 [00:39<01:12, 1913.38 examples/s]
hf-map en (num_proc=8): 30%|### | 59392/196400 [00:39<01:04, 2111.71 examples/s]
hf-map en (num_proc=8): 31%|###1 | 61440/196400 [00:39<00:51, 2640.83 examples/s]
hf-map en (num_proc=8): 33%|###2 | 64512/196400 [00:40<00:36, 3587.43 examples/s]
hf-map en (num_proc=8): 34%|###3 | 66560/196400 [00:42<01:06, 1949.15 examples/s]
hf-map en (num_proc=8): 35%|###5 | 69632/196400 [00:42<00:43, 2928.25 examples/s]
hf-map en (num_proc=8): 36%|###6 | 71680/196400 [00:42<00:35, 3469.58 examples/s]
hf-map en (num_proc=8): 37%|###7 | 72704/196400 [00:43<00:35, 3481.27 examples/s]
hf-map en (num_proc=8): 38%|###7 | 73728/196400 [00:43<00:31, 3841.77 examples/s]
hf-map en (num_proc=8): 38%|###8 | 74752/196400 [00:45<01:09, 1750.94 examples/s]
hf-map en (num_proc=8): 39%|###8 | 75776/196400 [00:45<00:58, 2058.27 examples/s]
hf-map en (num_proc=8): 40%|###9 | 77824/196400 [00:45<00:38, 3059.25 examples/s]
hf-map en (num_proc=8): 41%|#### | 79872/196400 [00:45<00:27, 4289.44 examples/s]
hf-map en (num_proc=8): 41%|####1 | 80896/196400 [00:46<00:29, 3877.40 examples/s]
hf-map en (num_proc=8): 42%|####1 | 81920/196400 [00:46<00:28, 4067.37 examples/s]
hf-map en (num_proc=8): 42%|####2 | 82944/196400 [00:47<00:54, 2071.37 examples/s]
hf-map en (num_proc=8): 43%|####2 | 83968/196400 [00:47<00:49, 2263.37 examples/s]
hf-map en (num_proc=8): 43%|####3 | 84992/196400 [00:48<00:40, 2751.85 examples/s]
hf-map en (num_proc=8): 44%|####3 | 86016/196400 [00:48<00:31, 3451.05 examples/s]
hf-map en (num_proc=8): 44%|####4 | 87040/196400 [00:48<00:28, 3808.40 examples/s]
hf-map en (num_proc=8): 45%|####5 | 89088/196400 [00:48<00:21, 5033.38 examples/s]
hf-map en (num_proc=8): 46%|####5 | 90112/196400 [00:48<00:25, 4187.36 examples/s]
hf-map en (num_proc=8): 46%|####6 | 91136/196400 [00:50<00:49, 2130.17 examples/s]
hf-map en (num_proc=8): 47%|####6 | 92160/196400 [00:50<00:45, 2278.84 examples/s]
hf-map en (num_proc=8): 48%|####7 | 94208/196400 [00:50<00:33, 3082.60 examples/s]
hf-map en (num_proc=8): 48%|####8 | 95232/196400 [00:51<00:35, 2814.50 examples/s]
hf-map en (num_proc=8): 50%|####9 | 97280/196400 [00:51<00:25, 3925.77 examples/s]
hf-map en (num_proc=8): 50%|##### | 98304/196400 [00:51<00:23, 4089.60 examples/s]
hf-map en (num_proc=8): 51%|##### | 99328/196400 [00:52<00:41, 2352.61 examples/s]
hf-map en (num_proc=8): 51%|#####1 | 100352/196400 [00:52<00:34, 2795.14 examples/s]
hf-map en (num_proc=8): 52%|#####2 | 102400/196400 [00:53<00:28, 3264.54 examples/s]
hf-map en (num_proc=8): 53%|#####2 | 103424/196400 [00:53<00:30, 3097.70 examples/s]
hf-map en (num_proc=8): 53%|#####3 | 104448/196400 [00:54<00:28, 3209.95 examples/s]
hf-map en (num_proc=8): 54%|#####4 | 106496/196400 [00:54<00:23, 3829.30 examples/s]
hf-map en (num_proc=8): 55%|#####4 | 107520/196400 [00:55<00:29, 2996.18 examples/s]
hf-map en (num_proc=8): 55%|#####5 | 108544/196400 [00:55<00:29, 2974.30 examples/s]
hf-map en (num_proc=8): 56%|#####6 | 110592/196400 [00:56<00:28, 3028.31 examples/s]
hf-map en (num_proc=8): 57%|#####6 | 111616/196400 [00:56<00:24, 3502.07 examples/s]
hf-map en (num_proc=8): 57%|#####7 | 112640/196400 [00:56<00:25, 3292.08 examples/s]
hf-map en (num_proc=8): 58%|#####7 | 113664/196400 [00:56<00:22, 3626.02 examples/s]
hf-map en (num_proc=8): 58%|#####8 | 114688/196400 [00:57<00:24, 3274.06 examples/s]
hf-map en (num_proc=8): 59%|#####8 | 115712/196400 [00:57<00:25, 3107.08 examples/s]
hf-map en (num_proc=8): 59%|#####9 | 116736/196400 [00:57<00:28, 2806.75 examples/s]
hf-map en (num_proc=8): 60%|#####9 | 117760/196400 [00:58<00:23, 3301.06 examples/s]
hf-map en (num_proc=8): 60%|###### | 118784/196400 [00:58<00:25, 3053.52 examples/s]
hf-map en (num_proc=8): 62%|######1 | 120832/196400 [00:58<00:17, 4320.32 examples/s]
hf-map en (num_proc=8): 62%|######2 | 121856/196400 [00:59<00:21, 3523.73 examples/s]
hf-map en (num_proc=8): 63%|######2 | 122880/196400 [00:59<00:19, 3773.68 examples/s]
hf-map en (num_proc=8): 63%|######3 | 123904/196400 [01:00<00:29, 2477.06 examples/s]
hf-map en (num_proc=8): 64%|######3 | 124928/196400 [01:00<00:28, 2535.53 examples/s]
hf-map en (num_proc=8): 65%|######4 | 126976/196400 [01:00<00:18, 3783.20 examples/s]
hf-map en (num_proc=8): 65%|######5 | 128000/196400 [01:00<00:15, 4441.07 examples/s]
hf-map en (num_proc=8): 66%|######5 | 129024/196400 [01:01<00:15, 4338.10 examples/s]
hf-map en (num_proc=8): 66%|######6 | 130048/196400 [01:01<00:19, 3338.82 examples/s]
hf-map en (num_proc=8): 67%|######6 | 131072/196400 [01:01<00:17, 3805.35 examples/s]
hf-map en (num_proc=8): 67%|######7 | 132096/196400 [01:02<00:19, 3259.11 examples/s]
hf-map en (num_proc=8): 68%|######7 | 133120/196400 [01:03<00:32, 1918.76 examples/s]
hf-map en (num_proc=8): 69%|######8 | 135168/196400 [01:03<00:19, 3090.30 examples/s]
hf-map en (num_proc=8): 70%|######9 | 137216/196400 [01:03<00:14, 3997.83 examples/s]
hf-map en (num_proc=8): 70%|####### | 138240/196400 [01:04<00:16, 3558.11 examples/s]
hf-map en (num_proc=8): 71%|####### | 139264/196400 [01:04<00:14, 4060.22 examples/s]
hf-map en (num_proc=8): 71%|#######1 | 140288/196400 [01:04<00:13, 4280.15 examples/s]
hf-map en (num_proc=8): 72%|#######1 | 141312/196400 [01:05<00:28, 1948.07 examples/s]
hf-map en (num_proc=8): 72%|#######2 | 142336/196400 [01:05<00:21, 2498.19 examples/s]
hf-map en (num_proc=8): 74%|#######3 | 144384/196400 [01:06<00:13, 3757.51 examples/s]
hf-map en (num_proc=8): 74%|#######4 | 145408/196400 [01:06<00:16, 3155.35 examples/s]
hf-map en (num_proc=8): 75%|#######4 | 146432/196400 [01:06<00:13, 3696.41 examples/s]
hf-map en (num_proc=8): 75%|#######5 | 147456/196400 [01:07<00:12, 3880.24 examples/s]
hf-map en (num_proc=8): 76%|#######5 | 148480/196400 [01:07<00:12, 3989.36 examples/s]
hf-map en (num_proc=8): 76%|#######6 | 149504/196400 [01:08<00:21, 2198.52 examples/s]
hf-map en (num_proc=8): 77%|#######7 | 151552/196400 [01:08<00:15, 2963.51 examples/s]
hf-map en (num_proc=8): 78%|#######8 | 153600/196400 [01:09<00:12, 3305.29 examples/s]
hf-map en (num_proc=8): 79%|#######8 | 154624/196400 [01:09<00:11, 3547.79 examples/s]
hf-map en (num_proc=8): 80%|#######9 | 156672/196400 [01:10<00:13, 3001.74 examples/s]
hf-map en (num_proc=8): 80%|######## | 157696/196400 [01:10<00:13, 2856.76 examples/s]
hf-map en (num_proc=8): 81%|######## | 158720/196400 [01:10<00:12, 3096.08 examples/s]
hf-map en (num_proc=8): 81%|########1 | 159744/196400 [01:11<00:13, 2691.68 examples/s]
hf-map en (num_proc=8): 82%|########2 | 161792/196400 [01:11<00:09, 3474.05 examples/s]
hf-map en (num_proc=8): 83%|########2 | 162816/196400 [01:11<00:09, 3708.86 examples/s]
hf-map en (num_proc=8): 83%|########3 | 163840/196400 [01:12<00:07, 4306.74 examples/s]
hf-map en (num_proc=8): 84%|########3 | 164864/196400 [01:12<00:11, 2728.86 examples/s]
hf-map en (num_proc=8): 84%|########4 | 165888/196400 [01:13<00:09, 3120.21 examples/s]
hf-map en (num_proc=8): 85%|########4 | 166912/196400 [01:13<00:11, 2631.80 examples/s]
hf-map en (num_proc=8): 86%|########5 | 167936/196400 [01:13<00:09, 3147.10 examples/s]
hf-map en (num_proc=8): 86%|########6 | 168960/196400 [01:14<00:08, 3182.87 examples/s]
hf-map en (num_proc=8): 87%|########6 | 169984/196400 [01:14<00:07, 3691.70 examples/s]
hf-map en (num_proc=8): 87%|########7 | 171008/196400 [01:14<00:07, 3227.44 examples/s]
hf-map en (num_proc=8): 88%|########7 | 172032/196400 [01:14<00:06, 3920.18 examples/s]
hf-map en (num_proc=8): 88%|########8 | 173056/196400 [01:15<00:07, 3246.64 examples/s]
hf-map en (num_proc=8): 89%|########8 | 174080/196400 [01:15<00:07, 3180.97 examples/s]
hf-map en (num_proc=8): 89%|########9 | 175104/196400 [01:15<00:06, 3067.93 examples/s]
hf-map en (num_proc=8): 90%|########9 | 176128/196400 [01:16<00:07, 2698.93 examples/s]
hf-map en (num_proc=8): 90%|######### | 177152/196400 [01:16<00:06, 2923.65 examples/s]
hf-map en (num_proc=8): 91%|######### | 178176/196400 [01:16<00:05, 3447.93 examples/s]
hf-map en (num_proc=8): 91%|#########1| 179200/196400 [01:17<00:05, 3289.20 examples/s]
hf-map en (num_proc=8): 92%|#########2| 181248/196400 [01:18<00:05, 2847.28 examples/s]
hf-map en (num_proc=8): 93%|#########3| 183296/196400 [01:18<00:04, 3251.79 examples/s]
hf-map en (num_proc=8): 94%|#########3| 184320/196400 [01:19<00:04, 2573.89 examples/s]
hf-map en (num_proc=8): 95%|#########4| 186368/196400 [01:19<00:02, 3830.33 examples/s]
hf-map en (num_proc=8): 95%|#########5| 187392/196400 [01:19<00:02, 3409.61 examples/s]
hf-map en (num_proc=8): 96%|#########6| 189414/196400 [01:20<00:02, 2880.59 examples/s]
hf-map en (num_proc=8): 97%|#########6| 190412/196400 [01:20<00:01, 3168.23 examples/s]
hf-map en (num_proc=8): 97%|#########7| 191410/196400 [01:21<00:01, 2818.54 examples/s]
hf-map en (num_proc=8): 98%|#########7| 192408/196400 [01:21<00:01, 2895.92 examples/s]
hf-map en (num_proc=8): 99%|#########8| 194404/196400 [01:21<00:00, 4486.65 examples/s]
hf-map en (num_proc=8): 99%|#########9| 195402/196400 [01:22<00:00, 3688.30 examples/s]
hf-map en (num_proc=8): 100%|##########| 196400/196400 [01:22<00:00, 2380.94 examples/s] |
| en: datasets.map num_proc=8 done 83s |
| en: 25000 docs 19355384 tokens 85s |
| en: 50000 docs 38502040 tokens 87s |
| en: 75000 docs 57868565 tokens 89s |
| en: 100000 docs 77221904 tokens 91s |
| en: 125000 docs 96715846 tokens 93s |
| packed en: 111592053 tokens (1.000 of budget) rows=196029 docs=144428 pad-per-doc (HF map) |
| full mix budget 1115916236 (en=10%); packed 1645162 sequences, 984858 docs, 1115928627 tokens |
| eval every 100 steps; resume checkpoint-{step} every 500 steps, keep last 2; analysis model-{step} every 1000 steps (weights+tokenizer; upload SeragAmin/gemma_mapping_byteify_cpt_seq1024) |
| starting from step 0 (no resumable checkpoint-{step}) |
| loading research/results/embedding_init/gemma_mapping_byteify |
| liger OLMo2 kernels (fused linear CE, RMSNorm, SwiGLU, RoPE) |
|
Loading weights: 0%| | 0/179 [00:00<?, ?it/s]
Loading weights: 1%| | 1/179 [00:03<10:15, 3.46s/it]
Loading weights: 21%|##1 | 38/179 [00:03<00:10, 14.04it/s]
Loading weights: 26%|##6 | 47/179 [00:03<00:07, 16.96it/s]
Loading weights: 30%|### | 54/179 [00:04<00:06, 19.81it/s]
Loading weights: 34%|###3 | 60/179 [00:04<00:06, 17.96it/s]
Loading weights: 38%|###7 | 68/179 [00:04<00:05, 20.68it/s]
Loading weights: 40%|#### | 72/179 [00:05<00:05, 19.21it/s]
Loading weights: 44%|####4 | 79/179 [00:05<00:05, 19.57it/s]
Loading weights: 46%|####5 | 82/179 [00:05<00:05, 17.66it/s]
Loading weights: 50%|##### | 90/179 [00:05<00:04, 20.71it/s]
Loading weights: 52%|#####1 | 93/179 [00:06<00:04, 18.20it/s]
Loading weights: 56%|#####5 | 100/179 [00:06<00:03, 23.44it/s]
Loading weights: 58%|#####7 | 103/179 [00:06<00:03, 19.25it/s]
Loading weights: 61%|###### | 109/179 [00:06<00:02, 24.41it/s]
Loading weights: 63%|######3 | 113/179 [00:07<00:03, 18.09it/s]
Loading weights: 65%|######4 | 116/179 [00:07<00:03, 18.53it/s]
Loading weights: 69%|######8 | 123/179 [00:07<00:02, 26.02it/s]
Loading weights: 71%|####### | 127/179 [00:07<00:02, 19.96it/s]
Loading weights: 75%|#######5 | 135/179 [00:08<00:02, 17.76it/s]
Loading weights: 77%|#######7 | 138/179 [00:08<00:02, 18.14it/s]
Loading weights: 81%|########1 | 145/179 [00:08<00:01, 25.18it/s]
Loading weights: 83%|########3 | 149/179 [00:08<00:01, 16.71it/s]
Loading weights: 88%|########7 | 157/179 [00:09<00:01, 19.52it/s]
Loading weights: 89%|########9 | 160/179 [00:09<00:01, 17.27it/s]
Loading weights: 94%|#########3| 168/179 [00:09<00:00, 20.38it/s]
Loading weights: 96%|#########5| 171/179 [00:09<00:00, 20.80it/s]
Loading weights: 100%|##########| 179/179 [00:10<00:00, 17.87it/s] |
| gradient_checkpointing=off liger=on |
| seed=42 flash SDP TF32 |
| eval at step 0 |
| Backend impl 'liger_kernel.ops.backends._cutedsl.rope' for op 'rope' could not be imported: ImportError: cutedsl backend requires the NVIDIA CUTLASS Python DSL (CuTe DSL). Install it with `pip install nvidia-cutlass-dsl`, or when installing Liger-Kernel use `pip install 'liger-kernel[cutedsl]'`. |
| Backend impl 'liger_kernel.ops.backends._ascend.rope' for op 'rope' could not be imported: ImportError: cannot import name 'tanh' from 'triton.language.math' (/usr/local/lib/python3.12/dist-packages/triton/language/math.py) |
| am nll=11.4879 ppl=97524.63 bpb=4.0397 |
| ta nll=11.4029 ppl=89584.75 bpb=1.6562 |
| kk nll=8.5479 ppl=5155.85 bpb=2.8784 |
| tl nll=2.8077 ppl=16.57 bpb=1.3322 |
| en nll=2.8266 ppl=16.89 bpb=0.8801 |
| eval full_cpt step 0 01:27 (86.6s) |
| full_cpt: 1645162 sequences → 12852 steps batch=16 accum=8 real_tokens=1115858680 liger fused CE, labels=-100 on pad, no attn mask |
| full_cpt step 1/12852 ( 0.0%) nll=9.0161 ppl=8234.82 grad=44.750 lr=1.56e-07 7489 tok/s 11.42s/it [00:11<40:46:51] mem=28.42GB |
| full_cpt step 10/12852 ( 0.1%) nll=8.7396 ppl=6245.61 grad=43.500 lr=1.56e-06 7554 tok/s 11.55s/it [01:55<41:11:12] mem=34.07GB |
| full_cpt step 20/12852 ( 0.2%) nll=9.1986 ppl=9883.72 grad=43.250 lr=3.13e-06 7500 tok/s 11.57s/it [03:51<41:14:50] mem=34.07GB |
| full_cpt step 30/12852 ( 0.2%) nll=7.9113 ppl=2727.83 grad=27.875 lr=4.69e-06 7431 tok/s 11.58s/it [05:47<41:14:42] mem=34.07GB |
| full_cpt step 40/12852 ( 0.3%) nll=7.9068 ppl=2715.78 grad=19.500 lr=6.25e-06 7487 tok/s 11.59s/it [07:44<41:15:03] mem=34.07GB |
| full_cpt step 50/12852 ( 0.4%) nll=6.6994 ppl=811.90 grad=9.750 lr=7.81e-06 7488 tok/s 11.60s/it [09:40<41:14:13] mem=34.07GB |
| full_cpt step 60/12852 ( 0.5%) nll=6.7456 ppl=850.32 grad=6.844 lr=9.38e-06 7481 tok/s 11.60s/it [11:36<41:12:34] mem=34.07GB |
| full_cpt step 70/12852 ( 0.5%) nll=5.8483 ppl=346.64 grad=3.312 lr=1.09e-05 7468 tok/s 11.60s/it [13:32<41:10:42] mem=34.07GB |
| full_cpt step 80/12852 ( 0.6%) nll=5.6926 ppl=296.68 grad=2.453 lr=1.25e-05 7461 tok/s 11.60s/it [15:28<41:08:49] mem=34.07GB |
| full_cpt step 90/12852 ( 0.7%) nll=5.4719 ppl=237.91 grad=2.078 lr=1.41e-05 7458 tok/s 11.60s/it [17:24<41:07:00] mem=34.07GB |
| full_cpt step 100/12852 ( 0.8%) nll=5.0722 ppl=159.52 grad=1.828 lr=1.56e-05 7460 tok/s 11.60s/it [19:20<41:05:16] mem=34.07GB |
| am nll=5.8568 ppl=349.59 bpb=2.0595 |
| ta nll=5.9974 ppl=402.40 bpb=0.8711 |
| kk nll=4.8469 ppl=127.34 bpb=1.6322 |
| tl nll=2.8098 ppl=16.61 bpb=1.3332 |
| en nll=2.8291 ppl=16.93 bpb=0.8809 |
| eval full_cpt step 100 01:27 (86.5s) |
| full_cpt step 110/12852 ( 0.9%) nll=4.9608 ppl=142.71 grad=1.773 lr=1.72e-05 6998 tok/s 12.39s/it [22:43<43:50:44] mem=34.07GB |
| full_cpt step 120/12852 ( 0.9%) nll=4.8165 ppl=123.54 grad=1.359 lr=1.88e-05 7029 tok/s 12.32s/it [24:39<43:34:49] mem=34.07GB |
| full_cpt step 130/12852 ( 1.0%) nll=4.7707 ppl=118.01 grad=1.320 lr=2.00e-05 7062 tok/s 12.27s/it [26:35<43:21:10] mem=34.07GB |
| full_cpt step 140/12852 ( 1.1%) nll=4.6013 ppl=99.61 grad=1.422 lr=2.00e-05 7101 tok/s 12.22s/it [28:31<43:09:25] mem=34.07GB |
| full_cpt step 150/12852 ( 1.2%) nll=4.3607 ppl=78.31 grad=1.312 lr=2.00e-05 7132 tok/s 12.18s/it [30:27<42:58:54] mem=34.07GB |
| full_cpt step 160/12852 ( 1.2%) nll=4.1826 ppl=65.53 grad=0.961 lr=2.00e-05 7156 tok/s 12.15s/it [32:23<42:49:24] mem=34.07GB |
| full_cpt step 170/12852 ( 1.3%) nll=4.4758 ppl=87.87 grad=1.016 lr=2.00e-05 7160 tok/s 12.11s/it [34:19<42:40:31] mem=34.07GB |
| full_cpt step 180/12852 ( 1.4%) nll=4.4099 ppl=82.26 grad=1.117 lr=2.00e-05 7172 tok/s 12.09s/it [36:15<42:32:29] mem=34.07GB |
| full_cpt step 190/12852 ( 1.5%) nll=4.1856 ppl=65.73 grad=1.062 lr=2.00e-05 7182 tok/s 12.06s/it [38:11<42:25:04] mem=34.07GB |
| full_cpt step 200/12852 ( 1.6%) nll=4.2869 ppl=72.74 grad=1.062 lr=2.00e-05 7190 tok/s 12.04s/it [40:07<42:18:15] mem=34.07GB |
| am nll=4.7848 ppl=119.68 bpb=1.6826 |
| ta nll=5.0247 ppl=152.13 bpb=0.7298 |
| kk nll=3.7758 ppl=43.63 bpb=1.2715 |
| tl nll=2.7740 ppl=16.02 bpb=1.3162 |
| en nll=2.8229 ppl=16.82 bpb=0.8789 |
| eval full_cpt step 200 01:27 (86.6s) |
| full_cpt step 210/12852 ( 1.6%) nll=4.1040 ppl=60.58 grad=0.969 lr=2.00e-05 6963 tok/s 12.43s/it [43:30<43:38:47] mem=34.07GB |
| full_cpt step 220/12852 ( 1.7%) nll=4.2026 ppl=66.86 grad=1.156 lr=2.00e-05 6982 tok/s 12.39s/it [45:26<43:28:49] mem=34.07GB |
| full_cpt step 230/12852 ( 1.8%) nll=4.1434 ppl=63.02 grad=1.172 lr=2.00e-05 7002 tok/s 12.36s/it [47:22<43:19:33] mem=34.07GB |
| full_cpt step 240/12852 ( 1.9%) nll=4.1126 ppl=61.11 grad=0.996 lr=2.00e-05 7028 tok/s 12.33s/it [49:18<43:10:57] mem=34.07GB |
| full_cpt step 250/12852 ( 1.9%) nll=4.0005 ppl=54.62 grad=0.953 lr=2.00e-05 7042 tok/s 12.30s/it [51:14<43:02:48] mem=34.07GB |
| full_cpt step 260/12852 ( 2.0%) nll=4.1169 ppl=61.37 grad=0.961 lr=2.00e-05 7061 tok/s 12.27s/it [53:11<42:55:23] mem=34.07GB |
| full_cpt step 270/12852 ( 2.1%) nll=4.0314 ppl=56.34 grad=0.977 lr=2.00e-05 7074 tok/s 12.25s/it [55:07<42:48:14] mem=34.07GB |
| full_cpt step 280/12852 ( 2.2%) nll=4.0677 ppl=58.43 grad=0.953 lr=2.00e-05 7088 tok/s 12.22s/it [57:03<42:41:28] mem=34.07GB |
| full_cpt step 290/12852 ( 2.3%) nll=4.0905 ppl=59.77 grad=0.988 lr=2.00e-05 7097 tok/s 12.20s/it [58:59<42:35:00] mem=34.07GB |
| full_cpt step 300/12852 ( 2.3%) nll=3.9160 ppl=50.20 grad=0.926 lr=2.00e-05 7104 tok/s 12.18s/it [1:00:55<42:28:45] mem=34.07GB |
| am nll=4.6131 ppl=100.80 bpb=1.6222 |
| ta nll=4.7709 ppl=118.02 bpb=0.6929 |
| kk nll=3.4599 ppl=31.81 bpb=1.1651 |
| tl nll=2.7481 ppl=15.61 bpb=1.3039 |
| en nll=2.8202 ppl=16.78 bpb=0.8781 |
| eval full_cpt step 300 01:28 (88.3s) |
| full_cpt step 310/12852 ( 2.4%) nll=3.9002 ppl=49.41 grad=1.086 lr=2.00e-05 6953 tok/s 12.45s/it [1:04:19<43:22:26] mem=34.07GB |
| full_cpt step 320/12852 ( 2.5%) nll=3.9939 ppl=54.27 grad=1.133 lr=2.00e-05 6969 tok/s 12.42s/it [1:06:16<43:14:58] mem=34.07GB |
| full_cpt step 330/12852 ( 2.6%) nll=3.9894 ppl=54.02 grad=1.102 lr=2.00e-05 6987 tok/s 12.40s/it [1:08:12<43:07:51] mem=34.07GB |
| full_cpt step 340/12852 ( 2.6%) nll=3.8941 ppl=49.11 grad=0.871 lr=2.00e-05 7002 tok/s 12.38s/it [1:10:08<43:01:02] mem=34.07GB |
| full_cpt step 350/12852 ( 2.7%) nll=3.8811 ppl=48.48 grad=1.078 lr=2.00e-05 7011 tok/s 12.36s/it [1:12:04<42:54:23] mem=34.07GB |
| full_cpt step 360/12852 ( 2.8%) nll=3.8757 ppl=48.22 grad=0.902 lr=2.00e-05 7029 tok/s 12.34s/it [1:14:01<42:48:10] mem=34.07GB |
| full_cpt step 370/12852 ( 2.9%) nll=3.8915 ppl=48.98 grad=1.086 lr=2.00e-05 7043 tok/s 12.32s/it [1:15:57<42:42:09] mem=34.07GB |
| full_cpt step 380/12852 ( 3.0%) nll=3.8985 ppl=49.33 grad=1.039 lr=2.00e-05 7055 tok/s 12.30s/it [1:17:53<42:36:17] mem=34.07GB |
| full_cpt step 390/12852 ( 3.0%) nll=3.9137 ppl=50.08 grad=0.973 lr=2.00e-05 7060 tok/s 12.28s/it [1:19:49<42:30:32] mem=34.07GB |
| full_cpt step 400/12852 ( 3.1%) nll=3.8371 ppl=46.39 grad=0.957 lr=2.00e-05 7066 tok/s 12.26s/it [1:21:45<42:24:59] mem=34.07GB |
| am nll=4.5228 ppl=92.10 bpb=1.5905 |
| ta nll=4.6198 ppl=101.47 bpb=0.6710 |
| kk nll=3.3079 ppl=27.33 bpb=1.1139 |
| tl nll=2.7356 ppl=15.42 bpb=1.2980 |
| en nll=2.8197 ppl=16.77 bpb=0.8779 |
| eval full_cpt step 400 01:27 (87.2s) |
| full_cpt step 410/12852 ( 3.2%) nll=3.8910 ppl=48.96 grad=0.988 lr=2.00e-05 6951 tok/s 12.46s/it [1:25:08<43:03:39] mem=34.07GB |
| full_cpt step 420/12852 ( 3.3%) nll=3.6862 ppl=39.89 grad=0.965 lr=2.00e-05 6960 tok/s 12.44s/it [1:27:04<42:57:23] mem=34.07GB |
| full_cpt step 430/12852 ( 3.3%) nll=3.8859 ppl=48.71 grad=0.906 lr=2.00e-05 6972 tok/s 12.42s/it [1:29:01<42:51:20] mem=34.07GB |
| full_cpt step 440/12852 ( 3.4%) nll=3.8437 ppl=46.70 grad=0.961 lr=2.00e-05 6977 tok/s 12.40s/it [1:30:56<42:45:23] mem=34.07GB |
| full_cpt step 450/12852 ( 3.5%) nll=3.8862 ppl=48.72 grad=0.902 lr=2.00e-05 6988 tok/s 12.38s/it [1:32:53<42:39:42] mem=34.07GB |
| full_cpt step 460/12852 ( 3.6%) nll=3.7112 ppl=40.90 grad=1.008 lr=2.00e-05 6998 tok/s 12.37s/it [1:34:49<42:34:09] mem=34.07GB |
| full_cpt step 470/12852 ( 3.7%) nll=3.8068 ppl=45.01 grad=1.023 lr=2.00e-05 7010 tok/s 12.35s/it [1:36:45<42:28:49] mem=34.07GB |
| full_cpt step 480/12852 ( 3.7%) nll=3.6855 ppl=39.86 grad=0.891 lr=2.00e-05 7023 tok/s 12.34s/it [1:38:41<42:23:40] mem=34.07GB |
| full_cpt step 490/12852 ( 3.8%) nll=3.8315 ppl=46.13 grad=1.039 lr=2.00e-05 7029 tok/s 12.32s/it [1:40:37<42:18:32] mem=34.07GB |
| full_cpt step 500/12852 ( 3.9%) nll=3.6173 ppl=37.24 grad=0.988 lr=2.00e-05 7036 tok/s 12.31s/it [1:42:33<42:13:32] mem=34.07GB |
| am nll=4.4628 ppl=86.73 bpb=1.5693 |
| ta nll=4.5147 ppl=91.35 bpb=0.6557 |
| kk nll=3.2084 ppl=24.74 bpb=1.0804 |
| tl nll=2.7281 ppl=15.30 bpb=1.2944 |
| en nll=2.8188 ppl=16.76 bpb=0.8777 |
| eval full_cpt step 500 01:29 (88.6s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:03<00:00, 3.02s/it]
Writing model shards: 100%|##########| 1/1 [00:03<00:00, 3.02s/it] |
| kept last 1 resume checkpoints: checkpoint-500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-500 at step 500 next_row=64000 |
| full_cpt step 510/12852 ( 4.0%) nll=3.7847 ppl=44.02 grad=0.906 lr=2.00e-05 6900 tok/s 12.55s/it [1:46:40<43:01:16] mem=34.07GB |
| full_cpt step 520/12852 ( 4.0%) nll=3.6834 ppl=39.78 grad=0.926 lr=2.00e-05 6913 tok/s 12.53s/it [1:48:36<42:55:33] mem=34.07GB |
| full_cpt step 530/12852 ( 4.1%) nll=3.6554 ppl=38.68 grad=0.895 lr=2.00e-05 6922 tok/s 12.51s/it [1:50:32<42:49:57] mem=34.07GB |
| full_cpt step 540/12852 ( 4.2%) nll=3.6495 ppl=38.46 grad=0.914 lr=2.00e-05 6933 tok/s 12.50s/it [1:52:29<42:44:31] mem=34.07GB |
| full_cpt step 550/12852 ( 4.3%) nll=3.7495 ppl=42.50 grad=0.984 lr=2.00e-05 6942 tok/s 12.48s/it [1:54:25<42:39:10] mem=34.07GB |
| full_cpt step 560/12852 ( 4.4%) nll=3.6277 ppl=37.63 grad=0.910 lr=1.99e-05 6948 tok/s 12.47s/it [1:56:21<42:33:55] mem=34.07GB |
| full_cpt step 570/12852 ( 4.4%) nll=3.6966 ppl=40.31 grad=1.031 lr=1.99e-05 6954 tok/s 12.45s/it [1:58:17<42:28:45] mem=34.07GB |
| full_cpt step 580/12852 ( 4.5%) nll=3.7028 ppl=40.56 grad=1.062 lr=1.99e-05 6957 tok/s 12.44s/it [2:00:13<42:23:41] mem=34.07GB |
| full_cpt step 590/12852 ( 4.6%) nll=3.7709 ppl=43.42 grad=0.883 lr=1.99e-05 6965 tok/s 12.42s/it [2:02:09<42:18:45] mem=34.07GB |
| full_cpt step 600/12852 ( 4.7%) nll=3.7626 ppl=43.06 grad=0.918 lr=1.99e-05 6974 tok/s 12.41s/it [2:04:06<42:13:59] mem=34.07GB |
| am nll=4.4064 ppl=81.97 bpb=1.5495 |
| ta nll=4.4384 ppl=84.64 bpb=0.6446 |
| kk nll=3.1384 ppl=23.07 bpb=1.0568 |
| tl nll=2.7212 ppl=15.20 bpb=1.2912 |
| en nll=2.8186 ppl=16.75 bpb=0.8776 |
| eval full_cpt step 600 01:28 (87.9s) |
| full_cpt step 610/12852 ( 4.7%) nll=3.7003 ppl=40.46 grad=0.883 lr=1.99e-05 6904 tok/s 12.54s/it [2:07:30<42:38:42] mem=34.07GB |
| full_cpt step 620/12852 ( 4.8%) nll=3.5746 ppl=35.68 grad=1.008 lr=1.99e-05 6913 tok/s 12.53s/it [2:09:26<42:33:37] mem=34.07GB |
| full_cpt step 630/12852 ( 4.9%) nll=3.5998 ppl=36.59 grad=1.086 lr=1.99e-05 6922 tok/s 12.51s/it [2:11:22<42:28:38] mem=34.07GB |
| full_cpt step 640/12852 ( 5.0%) nll=3.7162 ppl=41.11 grad=0.992 lr=1.99e-05 6930 tok/s 12.50s/it [2:13:19<42:23:43] mem=34.07GB |
| full_cpt step 650/12852 ( 5.1%) nll=3.4840 ppl=32.59 grad=1.039 lr=1.99e-05 6937 tok/s 12.48s/it [2:15:15<42:18:52] mem=34.07GB |
| full_cpt step 660/12852 ( 5.1%) nll=3.7686 ppl=43.32 grad=0.902 lr=1.99e-05 6945 tok/s 12.47s/it [2:17:11<42:14:08] mem=34.07GB |
| full_cpt step 670/12852 ( 5.2%) nll=3.6015 ppl=36.65 grad=1.047 lr=1.99e-05 6951 tok/s 12.46s/it [2:19:07<42:09:28] mem=34.07GB |
| full_cpt step 680/12852 ( 5.3%) nll=3.5772 ppl=35.77 grad=0.969 lr=1.99e-05 6956 tok/s 12.45s/it [2:21:03<42:04:53] mem=34.07GB |
| full_cpt step 690/12852 ( 5.4%) nll=3.5768 ppl=35.76 grad=0.969 lr=1.99e-05 6961 tok/s 12.43s/it [2:22:59<42:00:23] mem=34.07GB |
| full_cpt step 700/12852 ( 5.4%) nll=3.6700 ppl=39.25 grad=0.984 lr=1.99e-05 6967 tok/s 12.42s/it [2:24:56<41:55:57] mem=34.07GB |
| am nll=4.3600 ppl=78.26 bpb=1.5332 |
| ta nll=4.3733 ppl=79.31 bpb=0.6352 |
| kk nll=3.0876 ppl=21.92 bpb=1.0397 |
| tl nll=2.7138 ppl=15.09 bpb=1.2877 |
| en nll=2.8185 ppl=16.75 bpb=0.8776 |
| eval full_cpt step 700 01:31 (90.9s) |
| full_cpt step 710/12852 ( 5.5%) nll=3.5730 ppl=35.62 grad=0.957 lr=1.99e-05 6903 tok/s 12.54s/it [2:28:23<42:17:28] mem=34.07GB |
| full_cpt step 720/12852 ( 5.6%) nll=3.6783 ppl=39.58 grad=1.031 lr=1.99e-05 6910 tok/s 12.53s/it [2:30:19<42:12:48] mem=34.07GB |
| full_cpt step 730/12852 ( 5.7%) nll=3.6362 ppl=37.95 grad=0.887 lr=1.99e-05 6917 tok/s 12.51s/it [2:32:15<42:08:14] mem=34.07GB |
| full_cpt step 740/12852 ( 5.8%) nll=3.6386 ppl=38.04 grad=0.871 lr=1.99e-05 6925 tok/s 12.50s/it [2:34:11<42:03:43] mem=34.07GB |
| full_cpt step 750/12852 ( 5.8%) nll=3.6336 ppl=37.85 grad=0.852 lr=1.99e-05 6930 tok/s 12.49s/it [2:36:08<41:59:15] mem=34.07GB |
| full_cpt step 760/12852 ( 5.9%) nll=3.6625 ppl=38.96 grad=0.887 lr=1.99e-05 6938 tok/s 12.48s/it [2:38:04<41:54:54] mem=34.07GB |
| full_cpt step 770/12852 ( 6.0%) nll=3.6674 ppl=39.15 grad=0.871 lr=1.99e-05 6944 tok/s 12.47s/it [2:40:00<41:50:35] mem=34.07GB |
| full_cpt step 780/12852 ( 6.1%) nll=3.5545 ppl=34.97 grad=0.953 lr=1.99e-05 6949 tok/s 12.46s/it [2:41:56<41:46:20] mem=34.07GB |
| full_cpt step 790/12852 ( 6.1%) nll=3.6614 ppl=38.92 grad=0.992 lr=1.99e-05 6954 tok/s 12.45s/it [2:43:53<41:42:08] mem=34.07GB |
| full_cpt step 800/12852 ( 6.2%) nll=3.5055 ppl=33.30 grad=0.938 lr=1.99e-05 6961 tok/s 12.44s/it [2:45:49<41:38:01] mem=34.07GB |
| am nll=4.3257 ppl=75.62 bpb=1.5211 |
| ta nll=4.3230 ppl=75.42 bpb=0.6279 |
| kk nll=3.0481 ppl=21.08 bpb=1.0264 |
| tl nll=2.7088 ppl=15.01 bpb=1.2853 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 800 01:29 (89.5s) |
| full_cpt step 810/12852 ( 6.3%) nll=3.5105 ppl=33.46 grad=0.898 lr=1.99e-05 6906 tok/s 12.54s/it [2:49:15<41:56:05] mem=34.07GB |
| full_cpt step 820/12852 ( 6.4%) nll=3.4255 ppl=30.74 grad=0.840 lr=1.99e-05 6912 tok/s 12.53s/it [2:51:11<41:51:46] mem=34.07GB |
| full_cpt step 830/12852 ( 6.5%) nll=3.5034 ppl=33.23 grad=0.898 lr=1.99e-05 6919 tok/s 12.51s/it [2:53:07<41:47:31] mem=34.07GB |
| full_cpt step 840/12852 ( 6.5%) nll=3.5037 ppl=33.24 grad=0.945 lr=1.99e-05 6925 tok/s 12.50s/it [2:55:03<41:43:19] mem=34.07GB |
| full_cpt step 850/12852 ( 6.6%) nll=3.6228 ppl=37.44 grad=0.898 lr=1.99e-05 6931 tok/s 12.49s/it [2:57:00<41:39:09] mem=34.07GB |
| full_cpt step 860/12852 ( 6.7%) nll=3.7638 ppl=43.11 grad=0.984 lr=1.99e-05 6938 tok/s 12.48s/it [2:58:56<41:35:05] mem=34.07GB |
| full_cpt step 870/12852 ( 6.8%) nll=3.6368 ppl=37.97 grad=0.883 lr=1.98e-05 6943 tok/s 12.47s/it [3:00:52<41:31:01] mem=34.07GB |
| full_cpt step 880/12852 ( 6.8%) nll=3.6277 ppl=37.63 grad=0.879 lr=1.98e-05 6947 tok/s 12.46s/it [3:02:48<41:27:00] mem=34.07GB |
| full_cpt step 890/12852 ( 6.9%) nll=3.7619 ppl=43.03 grad=0.883 lr=1.98e-05 6952 tok/s 12.45s/it [3:04:45<41:23:02] mem=34.07GB |
| full_cpt step 900/12852 ( 7.0%) nll=3.6840 ppl=39.80 grad=1.016 lr=1.98e-05 6957 tok/s 12.45s/it [3:06:41<41:19:08] mem=34.07GB |
| am nll=4.2913 ppl=73.06 bpb=1.5090 |
| ta nll=4.2802 ppl=72.25 bpb=0.6217 |
| kk nll=3.0145 ppl=20.38 bpb=1.0151 |
| tl nll=2.7040 ppl=14.94 bpb=1.2830 |
| en nll=2.8180 ppl=16.74 bpb=0.8774 |
| eval full_cpt step 900 01:30 (89.7s) |
| full_cpt step 910/12852 ( 7.1%) nll=3.5601 ppl=35.17 grad=0.980 lr=1.98e-05 6907 tok/s 12.53s/it [3:10:07<41:34:52] mem=34.07GB |
| full_cpt step 920/12852 ( 7.2%) nll=3.6371 ppl=37.98 grad=0.891 lr=1.98e-05 6910 tok/s 12.52s/it [3:12:03<41:30:47] mem=34.07GB |
| full_cpt step 930/12852 ( 7.2%) nll=3.5700 ppl=35.52 grad=0.867 lr=1.98e-05 6915 tok/s 12.51s/it [3:13:59<41:26:43] mem=34.07GB |
| full_cpt step 940/12852 ( 7.3%) nll=3.6872 ppl=39.93 grad=0.922 lr=1.98e-05 6921 tok/s 12.51s/it [3:15:55<41:22:44] mem=34.07GB |
| full_cpt step 950/12852 ( 7.4%) nll=3.4510 ppl=31.53 grad=0.836 lr=1.98e-05 6927 tok/s 12.50s/it [3:17:51<41:18:47] mem=34.07GB |
| full_cpt step 960/12852 ( 7.5%) nll=3.5752 ppl=35.70 grad=0.973 lr=1.98e-05 6934 tok/s 12.49s/it [3:19:47<41:14:52] mem=34.07GB |
| full_cpt step 970/12852 ( 7.5%) nll=3.5482 ppl=34.75 grad=0.996 lr=1.98e-05 6938 tok/s 12.48s/it [3:21:43<41:10:59] mem=34.07GB |
| full_cpt step 980/12852 ( 7.6%) nll=3.5146 ppl=33.60 grad=0.957 lr=1.98e-05 6943 tok/s 12.47s/it [3:23:39<41:07:08] mem=34.07GB |
| full_cpt step 990/12852 ( 7.7%) nll=3.3437 ppl=28.32 grad=0.914 lr=1.98e-05 6948 tok/s 12.46s/it [3:25:36<41:03:22] mem=34.07GB |
| full_cpt step 1000/12852 ( 7.8%) nll=3.6479 ppl=38.39 grad=0.871 lr=1.98e-05 6952 tok/s 12.45s/it [3:27:32<40:59:36] mem=34.07GB |
| am nll=4.2638 ppl=71.08 bpb=1.4994 |
| ta nll=4.2455 ppl=69.79 bpb=0.6166 |
| kk nll=2.9858 ppl=19.80 bpb=1.0055 |
| tl nll=2.7004 ppl=14.89 bpb=1.2813 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 1000 01:28 (88.3s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.35s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.35s/it] |
| kept last 2 resume checkpoints: checkpoint-500 checkpoint-1000 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-1000 at step 1000 next_row=128000 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:03<00:00, 3.86s/it]
Writing model shards: 100%|##########| 1/1 [00:03<00:00, 3.88s/it] |
| wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-1000 (weights+expanded tokenizer, no optimizer) |
| full_cpt step 1010/12852 ( 7.9%) nll=3.5644 ppl=35.32 grad=0.879 lr=1.98e-05 6845 tok/s 12.65s/it [3:32:53<41:36:05] mem=34.07GB |
| full_cpt step 1020/12852 ( 7.9%) nll=3.5107 ppl=33.47 grad=0.922 lr=1.98e-05 6850 tok/s 12.64s/it [3:34:50<41:32:00] mem=34.07GB |
| full_cpt step 1030/12852 ( 8.0%) nll=3.4797 ppl=32.45 grad=0.840 lr=1.98e-05 6856 tok/s 12.63s/it [3:36:46<41:27:58] mem=34.07GB |
| full_cpt step 1040/12852 ( 8.1%) nll=3.5795 ppl=35.86 grad=0.891 lr=1.98e-05 6864 tok/s 12.62s/it [3:38:42<41:24:00] mem=34.07GB |
| full_cpt step 1050/12852 ( 8.2%) nll=3.5643 ppl=35.31 grad=0.914 lr=1.98e-05 6870 tok/s 12.61s/it [3:40:39<41:20:03] mem=34.07GB |
| full_cpt step 1060/12852 ( 8.2%) nll=3.5734 ppl=35.64 grad=0.961 lr=1.98e-05 6875 tok/s 12.60s/it [3:42:35<41:16:06] mem=34.07GB |
| full_cpt step 1070/12852 ( 8.3%) nll=3.5230 ppl=33.89 grad=0.887 lr=1.98e-05 6880 tok/s 12.59s/it [3:44:31<41:12:12] mem=34.07GB |
| full_cpt step 1080/12852 ( 8.4%) nll=3.4551 ppl=31.66 grad=0.945 lr=1.98e-05 6885 tok/s 12.58s/it [3:46:27<41:08:19] mem=34.07GB |
| full_cpt step 1090/12852 ( 8.5%) nll=3.4399 ppl=31.18 grad=0.961 lr=1.97e-05 6890 tok/s 12.57s/it [3:48:23<41:04:28] mem=34.07GB |
| full_cpt step 1100/12852 ( 8.6%) nll=3.4967 ppl=33.01 grad=0.895 lr=1.97e-05 6895 tok/s 12.56s/it [3:50:19<41:00:40] mem=34.07GB |
| am nll=4.2356 ppl=69.11 bpb=1.4895 |
| ta nll=4.2159 ppl=67.76 bpb=0.6123 |
| kk nll=2.9628 ppl=19.35 bpb=0.9977 |
| tl nll=2.6950 ppl=14.81 bpb=1.2787 |
| en nll=2.8180 ppl=16.74 bpb=0.8774 |
| eval full_cpt step 1100 01:28 (88.3s) |
| full_cpt step 1110/12852 ( 8.6%) nll=3.5617 ppl=35.22 grad=0.918 lr=1.97e-05 6856 tok/s 12.63s/it [3:53:44<41:12:26] mem=34.07GB |
| full_cpt step 1120/12852 ( 8.7%) nll=3.4824 ppl=32.54 grad=0.879 lr=1.97e-05 6861 tok/s 12.62s/it [3:55:40<41:08:32] mem=34.07GB |
| full_cpt step 1130/12852 ( 8.8%) nll=3.4419 ppl=31.25 grad=1.047 lr=1.97e-05 6864 tok/s 12.62s/it [3:57:36<41:04:39] mem=34.07GB |
| full_cpt step 1140/12852 ( 8.9%) nll=3.5142 ppl=33.59 grad=0.879 lr=1.97e-05 6869 tok/s 12.61s/it [3:59:32<41:00:49] mem=34.07GB |
| full_cpt step 1150/12852 ( 8.9%) nll=3.5591 ppl=35.13 grad=0.898 lr=1.97e-05 6874 tok/s 12.60s/it [4:01:28<40:57:03] mem=34.07GB |
| full_cpt step 1160/12852 ( 9.0%) nll=3.5064 ppl=33.33 grad=0.859 lr=1.97e-05 6880 tok/s 12.59s/it [4:03:24<40:53:18] mem=34.07GB |
| full_cpt step 1170/12852 ( 9.1%) nll=3.6826 ppl=39.75 grad=0.887 lr=1.97e-05 6884 tok/s 12.58s/it [4:05:20<40:49:35] mem=34.07GB |
| full_cpt step 1180/12852 ( 9.2%) nll=3.4707 ppl=32.16 grad=0.871 lr=1.97e-05 6890 tok/s 12.57s/it [4:07:16<40:45:54] mem=34.07GB |
| full_cpt step 1190/12852 ( 9.3%) nll=3.5170 ppl=33.68 grad=0.852 lr=1.97e-05 6894 tok/s 12.56s/it [4:09:12<40:42:13] mem=34.07GB |
| full_cpt step 1200/12852 ( 9.3%) nll=3.5983 ppl=36.54 grad=0.879 lr=1.97e-05 6900 tok/s 12.56s/it [4:11:09<40:38:37] mem=34.07GB |
| am nll=4.2136 ppl=67.60 bpb=1.4817 |
| ta nll=4.1897 ppl=66.00 bpb=0.6085 |
| kk nll=2.9418 ppl=18.95 bpb=0.9906 |
| tl nll=2.6909 ppl=14.75 bpb=1.2768 |
| en nll=2.8173 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 1200 01:29 (88.9s) |
| full_cpt step 1210/12852 ( 9.4%) nll=3.4936 ppl=32.91 grad=0.863 lr=1.97e-05 6865 tok/s 12.62s/it [4:14:34<40:49:15] mem=34.07GB |
| full_cpt step 1220/12852 ( 9.5%) nll=3.5528 ppl=34.91 grad=0.953 lr=1.97e-05 6872 tok/s 12.61s/it [4:16:30<40:45:34] mem=34.07GB |
| full_cpt step 1230/12852 ( 9.6%) nll=3.5026 ppl=33.20 grad=0.875 lr=1.97e-05 6876 tok/s 12.61s/it [4:18:26<40:41:52] mem=34.07GB |
| full_cpt step 1240/12852 ( 9.6%) nll=3.5098 ppl=33.44 grad=0.883 lr=1.97e-05 6882 tok/s 12.60s/it [4:20:22<40:38:14] mem=34.07GB |
| full_cpt step 1250/12852 ( 9.7%) nll=3.5573 ppl=35.07 grad=0.906 lr=1.97e-05 6887 tok/s 12.59s/it [4:22:18<40:34:38] mem=34.07GB |
| full_cpt step 1260/12852 ( 9.8%) nll=3.3521 ppl=28.56 grad=0.887 lr=1.97e-05 6891 tok/s 12.58s/it [4:24:14<40:31:01] mem=34.07GB |
| full_cpt step 1270/12852 ( 9.9%) nll=3.3965 ppl=29.86 grad=0.902 lr=1.96e-05 6895 tok/s 12.58s/it [4:26:11<40:27:26] mem=34.07GB |
| full_cpt step 1280/12852 ( 10.0%) nll=3.4612 ppl=31.86 grad=0.898 lr=1.96e-05 6900 tok/s 12.57s/it [4:28:07<40:23:54] mem=34.07GB |
| full_cpt step 1290/12852 ( 10.0%) nll=3.4444 ppl=31.32 grad=0.848 lr=1.96e-05 6904 tok/s 12.56s/it [4:30:03<40:20:22] mem=34.07GB |
| full_cpt step 1300/12852 ( 10.1%) nll=3.5590 ppl=35.13 grad=0.969 lr=1.96e-05 6907 tok/s 12.55s/it [4:31:59<40:16:51] mem=34.07GB |
| am nll=4.1931 ppl=66.23 bpb=1.4745 |
| ta nll=4.1677 ppl=64.57 bpb=0.6053 |
| kk nll=2.9234 ppl=18.60 bpb=0.9844 |
| tl nll=2.6876 ppl=14.70 bpb=1.2752 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 1300 01:30 (90.3s) |
| full_cpt step 1310/12852 ( 10.2%) nll=3.4727 ppl=32.22 grad=0.891 lr=1.96e-05 6872 tok/s 12.61s/it [4:35:25<40:26:37] mem=34.07GB |
| full_cpt step 1320/12852 ( 10.3%) nll=3.5907 ppl=36.26 grad=0.926 lr=1.96e-05 6876 tok/s 12.61s/it [4:37:21<40:23:03] mem=34.07GB |
| full_cpt step 1330/12852 ( 10.3%) nll=3.4014 ppl=30.01 grad=0.895 lr=1.96e-05 6878 tok/s 12.60s/it [4:39:17<40:19:29] mem=34.07GB |
| full_cpt step 1340/12852 ( 10.4%) nll=3.5272 ppl=34.03 grad=0.895 lr=1.96e-05 6883 tok/s 12.59s/it [4:41:13<40:15:59] mem=34.07GB |
| full_cpt step 1350/12852 ( 10.5%) nll=3.5428 ppl=34.56 grad=0.914 lr=1.96e-05 6886 tok/s 12.58s/it [4:43:09<40:12:29] mem=34.07GB |
| full_cpt step 1360/12852 ( 10.6%) nll=3.5278 ppl=34.05 grad=0.930 lr=1.96e-05 6889 tok/s 12.58s/it [4:45:05<40:09:01] mem=34.07GB |
| full_cpt step 1370/12852 ( 10.7%) nll=3.4274 ppl=30.80 grad=0.871 lr=1.96e-05 6894 tok/s 12.57s/it [4:47:02<40:05:35] mem=34.07GB |
| full_cpt step 1380/12852 ( 10.7%) nll=3.4210 ppl=30.60 grad=0.910 lr=1.96e-05 6898 tok/s 12.56s/it [4:48:58<40:02:10] mem=34.07GB |
| full_cpt step 1390/12852 ( 10.8%) nll=3.3496 ppl=28.49 grad=0.898 lr=1.96e-05 6902 tok/s 12.56s/it [4:50:54<39:58:46] mem=34.07GB |
| full_cpt step 1400/12852 ( 10.9%) nll=3.5792 ppl=35.84 grad=0.902 lr=1.96e-05 6906 tok/s 12.55s/it [4:52:50<39:55:23] mem=34.07GB |
| am nll=4.1741 ppl=64.98 bpb=1.4678 |
| ta nll=4.1479 ppl=63.30 bpb=0.6024 |
| kk nll=2.9068 ppl=18.30 bpb=0.9788 |
| tl nll=2.6858 ppl=14.67 bpb=1.2744 |
| en nll=2.8172 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 1400 01:28 (87.8s) |
| full_cpt step 1410/12852 ( 11.0%) nll=3.3860 ppl=29.55 grad=0.844 lr=1.96e-05 6876 tok/s 12.61s/it [4:56:14<40:03:54] mem=34.07GB |
| full_cpt step 1420/12852 ( 11.0%) nll=3.4304 ppl=30.89 grad=0.812 lr=1.95e-05 6880 tok/s 12.60s/it [4:58:10<40:00:28] mem=34.07GB |
| full_cpt step 1430/12852 ( 11.1%) nll=3.4062 ppl=30.15 grad=0.984 lr=1.95e-05 6884 tok/s 12.59s/it [5:00:06<39:57:03] mem=34.07GB |
| full_cpt step 1440/12852 ( 11.2%) nll=3.4192 ppl=30.55 grad=0.863 lr=1.95e-05 6888 tok/s 12.58s/it [5:02:02<39:53:38] mem=34.07GB |
| full_cpt step 1450/12852 ( 11.3%) nll=3.4368 ppl=31.09 grad=0.926 lr=1.95e-05 6891 tok/s 12.58s/it [5:03:58<39:50:15] mem=34.07GB |
| full_cpt step 1460/12852 ( 11.4%) nll=3.3653 ppl=28.94 grad=0.953 lr=1.95e-05 6895 tok/s 12.57s/it [5:05:54<39:46:54] mem=34.07GB |
| full_cpt step 1470/12852 ( 11.4%) nll=3.4071 ppl=30.18 grad=0.914 lr=1.95e-05 6898 tok/s 12.56s/it [5:07:50<39:43:33] mem=34.07GB |
| full_cpt step 1480/12852 ( 11.5%) nll=3.3605 ppl=28.80 grad=0.953 lr=1.95e-05 6901 tok/s 12.56s/it [5:09:46<39:40:14] mem=34.07GB |
| full_cpt step 1490/12852 ( 11.6%) nll=3.3500 ppl=28.50 grad=0.848 lr=1.95e-05 6904 tok/s 12.55s/it [5:11:43<39:36:57] mem=34.07GB |
| full_cpt step 1500/12852 ( 11.7%) nll=3.4439 ppl=31.31 grad=0.863 lr=1.95e-05 6908 tok/s 12.55s/it [5:13:39<39:33:41] mem=34.07GB |
| am nll=4.1580 ppl=63.94 bpb=1.4622 |
| ta nll=4.1290 ppl=62.12 bpb=0.5997 |
| kk nll=2.8925 ppl=18.04 bpb=0.9740 |
| tl nll=2.6840 ppl=14.64 bpb=1.2735 |
| en nll=2.8175 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 1500 01:28 (88.2s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.43s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.43s/it] |
| removing old checkpoint checkpoint-500 |
| kept last 2 resume checkpoints: checkpoint-1000 checkpoint-1500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-1500 at step 1500 next_row=192000 |
| full_cpt step 1510/12852 ( 11.7%) nll=3.5295 ppl=34.11 grad=0.832 lr=1.95e-05 6848 tok/s 12.66s/it [5:18:33<39:52:39] mem=34.07GB |
| full_cpt step 1520/12852 ( 11.8%) nll=3.5562 ppl=35.03 grad=0.859 lr=1.95e-05 6852 tok/s 12.65s/it [5:20:29<39:49:16] mem=34.07GB |
| full_cpt step 1530/12852 ( 11.9%) nll=3.5207 ppl=33.81 grad=0.855 lr=1.95e-05 6856 tok/s 12.64s/it [5:22:25<39:45:54] mem=34.07GB |
| full_cpt step 1540/12852 ( 12.0%) nll=3.3791 ppl=29.34 grad=0.941 lr=1.95e-05 6859 tok/s 12.64s/it [5:24:21<39:42:33] mem=34.07GB |
| full_cpt step 1550/12852 ( 12.1%) nll=3.4615 ppl=31.87 grad=0.855 lr=1.95e-05 6863 tok/s 12.63s/it [5:26:18<39:39:13] mem=34.07GB |
| full_cpt step 1560/12852 ( 12.1%) nll=3.6005 ppl=36.62 grad=0.879 lr=1.94e-05 6868 tok/s 12.62s/it [5:28:14<39:35:54] mem=34.07GB |
| full_cpt step 1570/12852 ( 12.2%) nll=3.5305 ppl=34.14 grad=0.867 lr=1.94e-05 6872 tok/s 12.62s/it [5:30:10<39:32:36] mem=34.07GB |
| full_cpt step 1580/12852 ( 12.3%) nll=3.4097 ppl=30.26 grad=1.016 lr=1.94e-05 6874 tok/s 12.61s/it [5:32:06<39:29:19] mem=34.07GB |
| full_cpt step 1590/12852 ( 12.4%) nll=3.3741 ppl=29.20 grad=0.914 lr=1.94e-05 6878 tok/s 12.61s/it [5:34:03<39:26:02] mem=34.07GB |
| full_cpt step 1600/12852 ( 12.4%) nll=3.4263 ppl=30.76 grad=0.844 lr=1.94e-05 6882 tok/s 12.60s/it [5:35:59<39:22:48] mem=34.07GB |
| am nll=4.1428 ppl=62.98 bpb=1.4568 |
| ta nll=4.1132 ppl=61.14 bpb=0.5974 |
| kk nll=2.8815 ppl=17.84 bpb=0.9703 |
| tl nll=2.6822 ppl=14.62 bpb=1.2727 |
| en nll=2.8175 ppl=16.74 bpb=0.8772 |
| eval full_cpt step 1600 01:29 (89.2s) |
| full_cpt step 1610/12852 ( 12.5%) nll=3.5164 ppl=33.66 grad=0.992 lr=1.94e-05 6855 tok/s 12.65s/it [5:39:24<39:29:55] mem=34.07GB |
| full_cpt step 1620/12852 ( 12.6%) nll=3.4835 ppl=32.57 grad=0.969 lr=1.94e-05 6858 tok/s 12.64s/it [5:41:20<39:26:37] mem=34.07GB |
| full_cpt step 1630/12852 ( 12.7%) nll=3.3300 ppl=27.94 grad=0.902 lr=1.94e-05 6862 tok/s 12.64s/it [5:43:17<39:23:21] mem=34.07GB |
| full_cpt step 1640/12852 ( 12.8%) nll=3.3863 ppl=29.56 grad=0.980 lr=1.94e-05 6866 tok/s 12.63s/it [5:45:13<39:20:05] mem=34.07GB |
| full_cpt step 1650/12852 ( 12.8%) nll=3.4368 ppl=31.09 grad=0.895 lr=1.94e-05 6868 tok/s 12.62s/it [5:47:09<39:16:49] mem=34.07GB |
| full_cpt step 1660/12852 ( 12.9%) nll=3.4265 ppl=30.77 grad=0.984 lr=1.94e-05 6872 tok/s 12.62s/it [5:49:05<39:13:35] mem=34.07GB |
| full_cpt step 1670/12852 ( 13.0%) nll=3.5378 ppl=34.39 grad=0.879 lr=1.94e-05 6875 tok/s 12.61s/it [5:51:01<39:10:22] mem=34.07GB |
| full_cpt step 1680/12852 ( 13.1%) nll=3.3791 ppl=29.34 grad=1.023 lr=1.93e-05 6878 tok/s 12.61s/it [5:52:57<39:07:09] mem=34.07GB |
| full_cpt step 1690/12852 ( 13.1%) nll=3.3743 ppl=29.20 grad=0.906 lr=1.93e-05 6882 tok/s 12.60s/it [5:54:54<39:03:59] mem=34.07GB |
| full_cpt step 1700/12852 ( 13.2%) nll=3.3287 ppl=27.90 grad=0.859 lr=1.93e-05 6884 tok/s 12.59s/it [5:56:50<39:00:48] mem=34.07GB |
| am nll=4.1298 ppl=62.16 bpb=1.4522 |
| ta nll=4.0989 ppl=60.27 bpb=0.5953 |
| kk nll=2.8689 ppl=17.62 bpb=0.9661 |
| tl nll=2.6805 ppl=14.59 bpb=1.2718 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 1700 01:27 (87.4s) |
| full_cpt step 1710/12852 ( 13.3%) nll=3.4315 ppl=30.92 grad=0.887 lr=1.93e-05 6860 tok/s 12.64s/it [6:00:13<39:07:07] mem=34.07GB |
| full_cpt step 1720/12852 ( 13.4%) nll=3.4934 ppl=32.90 grad=0.957 lr=1.93e-05 6863 tok/s 12.63s/it [6:02:09<39:03:55] mem=34.07GB |
| full_cpt step 1730/12852 ( 13.5%) nll=3.1659 ppl=23.71 grad=0.910 lr=1.93e-05 6866 tok/s 12.63s/it [6:04:05<39:00:42] mem=34.07GB |
| full_cpt step 1740/12852 ( 13.5%) nll=3.4504 ppl=31.51 grad=0.887 lr=1.93e-05 6868 tok/s 12.62s/it [6:06:02<38:57:31] mem=34.07GB |
| full_cpt step 1750/12852 ( 13.6%) nll=3.4786 ppl=32.42 grad=0.977 lr=1.93e-05 6871 tok/s 12.62s/it [6:07:58<38:54:21] mem=34.07GB |
| full_cpt step 1760/12852 ( 13.7%) nll=3.3750 ppl=29.22 grad=0.840 lr=1.93e-05 6873 tok/s 12.61s/it [6:09:54<38:51:11] mem=34.07GB |
| full_cpt step 1770/12852 ( 13.8%) nll=3.4869 ppl=32.68 grad=0.855 lr=1.93e-05 6876 tok/s 12.60s/it [6:11:50<38:48:03] mem=34.07GB |
| full_cpt step 1780/12852 ( 13.8%) nll=3.5554 ppl=35.00 grad=0.914 lr=1.93e-05 6880 tok/s 12.60s/it [6:13:46<38:44:56] mem=34.07GB |
| full_cpt step 1790/12852 ( 13.9%) nll=3.5641 ppl=35.31 grad=0.934 lr=1.93e-05 6883 tok/s 12.59s/it [6:15:42<38:41:50] mem=34.07GB |
| full_cpt step 1800/12852 ( 14.0%) nll=3.3180 ppl=27.60 grad=0.875 lr=1.92e-05 6886 tok/s 12.59s/it [6:17:39<38:38:45] mem=34.07GB |
| am nll=4.1164 ppl=61.34 bpb=1.4476 |
| ta nll=4.0870 ppl=59.56 bpb=0.5936 |
| kk nll=2.8596 ppl=17.46 bpb=0.9630 |
| tl nll=2.6785 ppl=14.56 bpb=1.2709 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 1800 01:29 (88.6s) |
| full_cpt step 1810/12852 ( 14.1%) nll=3.3135 ppl=27.48 grad=0.812 lr=1.92e-05 6864 tok/s 12.63s/it [6:21:04<38:44:42] mem=34.07GB |
| full_cpt step 1820/12852 ( 14.2%) nll=3.4622 ppl=31.89 grad=0.891 lr=1.92e-05 6867 tok/s 12.63s/it [6:23:00<38:41:35] mem=34.07GB |
| full_cpt step 1830/12852 ( 14.2%) nll=3.3685 ppl=29.04 grad=0.871 lr=1.92e-05 6870 tok/s 12.62s/it [6:24:56<38:38:28] mem=34.07GB |
| full_cpt step 1840/12852 ( 14.3%) nll=3.5130 ppl=33.55 grad=0.898 lr=1.92e-05 6873 tok/s 12.62s/it [6:26:53<38:35:22] mem=34.07GB |
| full_cpt step 1850/12852 ( 14.4%) nll=3.3123 ppl=27.45 grad=0.844 lr=1.92e-05 6877 tok/s 12.61s/it [6:28:49<38:32:17] mem=34.07GB |
| full_cpt step 1860/12852 ( 14.5%) nll=3.4580 ppl=31.75 grad=0.871 lr=1.92e-05 6880 tok/s 12.60s/it [6:30:45<38:29:13] mem=34.07GB |
| full_cpt step 1870/12852 ( 14.6%) nll=3.2985 ppl=27.07 grad=0.941 lr=1.92e-05 6883 tok/s 12.60s/it [6:32:41<38:26:10] mem=34.07GB |
| full_cpt step 1880/12852 ( 14.6%) nll=3.4683 ppl=32.08 grad=0.863 lr=1.92e-05 6886 tok/s 12.59s/it [6:34:38<38:23:07] mem=34.07GB |
| full_cpt step 1890/12852 ( 14.7%) nll=3.4725 ppl=32.22 grad=0.969 lr=1.92e-05 6890 tok/s 12.59s/it [6:36:34<38:20:06] mem=34.07GB |
| full_cpt step 1900/12852 ( 14.8%) nll=3.3387 ppl=28.18 grad=0.902 lr=1.92e-05 6893 tok/s 12.58s/it [6:38:30<38:17:04] mem=34.07GB |
| am nll=4.1057 ppl=60.68 bpb=1.4438 |
| ta nll=4.0737 ppl=58.77 bpb=0.5917 |
| kk nll=2.8500 ppl=17.29 bpb=0.9597 |
| tl nll=2.6766 ppl=14.54 bpb=1.2700 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 1900 01:30 (89.7s) |
| full_cpt step 1910/12852 ( 14.9%) nll=3.4400 ppl=31.19 grad=0.938 lr=1.91e-05 6870 tok/s 12.63s/it [6:41:56<38:22:35] mem=34.07GB |
| full_cpt step 1920/12852 ( 14.9%) nll=3.4093 ppl=30.24 grad=0.871 lr=1.91e-05 6873 tok/s 12.62s/it [6:43:52<38:19:31] mem=34.07GB |
| full_cpt step 1930/12852 ( 15.0%) nll=3.3329 ppl=28.02 grad=0.852 lr=1.91e-05 6876 tok/s 12.62s/it [6:45:48<38:16:27] mem=34.07GB |
| full_cpt step 1940/12852 ( 15.1%) nll=3.5041 ppl=33.25 grad=0.914 lr=1.91e-05 6878 tok/s 12.61s/it [6:47:44<38:13:24] mem=34.07GB |
| full_cpt step 1950/12852 ( 15.2%) nll=3.3569 ppl=28.70 grad=0.902 lr=1.91e-05 6881 tok/s 12.61s/it [6:49:40<38:10:21] mem=34.07GB |
| full_cpt step 1960/12852 ( 15.3%) nll=3.4603 ppl=31.83 grad=0.922 lr=1.91e-05 6884 tok/s 12.60s/it [6:51:36<38:07:20] mem=34.07GB |
| full_cpt step 1970/12852 ( 15.3%) nll=3.4304 ppl=30.89 grad=0.867 lr=1.91e-05 6887 tok/s 12.59s/it [6:53:32<38:04:19] mem=34.07GB |
| full_cpt step 1980/12852 ( 15.4%) nll=3.3635 ppl=28.89 grad=0.828 lr=1.91e-05 6890 tok/s 12.59s/it [6:55:28<38:01:18] mem=34.07GB |
| full_cpt step 1990/12852 ( 15.5%) nll=3.3724 ppl=29.15 grad=0.887 lr=1.91e-05 6892 tok/s 12.59s/it [6:57:24<37:58:19] mem=34.07GB |
| full_cpt step 2000/12852 ( 15.6%) nll=3.4527 ppl=31.58 grad=0.902 lr=1.91e-05 6895 tok/s 12.58s/it [6:59:20<37:55:20] mem=34.07GB |
| am nll=4.0939 ppl=59.97 bpb=1.4396 |
| ta nll=4.0632 ppl=58.16 bpb=0.5901 |
| kk nll=2.8424 ppl=17.16 bpb=0.9572 |
| tl nll=2.6758 ppl=14.52 bpb=1.2696 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 2000 01:27 (87.3s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.27s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.27s/it] |
| removing old checkpoint checkpoint-1000 |
| kept last 2 resume checkpoints: checkpoint-1500 checkpoint-2000 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-2000 at step 2000 next_row=256000 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.74s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.74s/it] |
| wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-2000 (weights+expanded tokenizer, no optimizer) |
| full_cpt step 2010/12852 ( 15.6%) nll=3.3616 ppl=28.83 grad=0.922 lr=1.90e-05 6848 tok/s 12.67s/it [7:04:22<38:09:01] mem=34.07GB |
| full_cpt step 2020/12852 ( 15.7%) nll=3.4129 ppl=30.35 grad=0.891 lr=1.90e-05 6851 tok/s 12.66s/it [7:06:18<38:05:58] mem=34.07GB |
| full_cpt step 2030/12852 ( 15.8%) nll=3.4853 ppl=32.63 grad=0.891 lr=1.90e-05 6853 tok/s 12.66s/it [7:08:14<38:02:56] mem=34.07GB |
| full_cpt step 2040/12852 ( 15.9%) nll=3.4149 ppl=30.41 grad=0.859 lr=1.90e-05 6856 tok/s 12.65s/it [7:10:10<37:59:54] mem=34.07GB |
| full_cpt step 2050/12852 ( 16.0%) nll=3.4357 ppl=31.05 grad=0.926 lr=1.90e-05 6859 tok/s 12.65s/it [7:12:06<37:56:53] mem=34.07GB |
| full_cpt step 2060/12852 ( 16.0%) nll=3.4271 ppl=30.79 grad=0.965 lr=1.90e-05 6861 tok/s 12.64s/it [7:14:02<37:53:52] mem=34.07GB |
| full_cpt step 2070/12852 ( 16.1%) nll=3.3403 ppl=28.23 grad=0.863 lr=1.90e-05 6864 tok/s 12.64s/it [7:15:59<37:50:52] mem=34.07GB |
| full_cpt step 2080/12852 ( 16.2%) nll=3.2685 ppl=26.27 grad=0.898 lr=1.90e-05 6866 tok/s 12.63s/it [7:17:55<37:47:53] mem=34.07GB |
| full_cpt step 2090/12852 ( 16.3%) nll=3.4011 ppl=30.00 grad=0.895 lr=1.90e-05 6868 tok/s 12.63s/it [7:19:51<37:44:53] mem=34.07GB |
| full_cpt step 2100/12852 ( 16.3%) nll=3.3971 ppl=29.88 grad=0.930 lr=1.90e-05 6871 tok/s 12.62s/it [7:21:47<37:41:55] mem=34.07GB |
| am nll=4.0848 ppl=59.43 bpb=1.4364 |
| ta nll=4.0533 ppl=57.59 bpb=0.5887 |
| kk nll=2.8351 ppl=17.03 bpb=0.9547 |
| tl nll=2.6750 ppl=14.51 bpb=1.2692 |
| en nll=2.8172 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 2100 01:28 (87.7s) |
| full_cpt step 2110/12852 ( 16.4%) nll=3.3259 ppl=27.82 grad=1.047 lr=1.89e-05 6851 tok/s 12.66s/it [7:25:11<37:46:24] mem=34.07GB |
| full_cpt step 2120/12852 ( 16.5%) nll=3.6365 ppl=37.96 grad=0.922 lr=1.89e-05 6853 tok/s 12.65s/it [7:27:07<37:43:24] mem=34.07GB |
| full_cpt step 2130/12852 ( 16.6%) nll=3.3676 ppl=29.01 grad=0.941 lr=1.89e-05 6857 tok/s 12.65s/it [7:29:03<37:40:26] mem=34.07GB |
| full_cpt step 2140/12852 ( 16.7%) nll=3.4597 ppl=31.81 grad=0.883 lr=1.89e-05 6860 tok/s 12.64s/it [7:30:59<37:37:28] mem=34.07GB |
| full_cpt step 2150/12852 ( 16.7%) nll=3.3435 ppl=28.32 grad=0.883 lr=1.89e-05 6862 tok/s 12.64s/it [7:32:55<37:34:30] mem=34.07GB |
| full_cpt step 2160/12852 ( 16.8%) nll=3.4069 ppl=30.17 grad=0.848 lr=1.89e-05 6864 tok/s 12.63s/it [7:34:51<37:31:32] mem=34.07GB |
| full_cpt step 2170/12852 ( 16.9%) nll=3.3236 ppl=27.76 grad=0.984 lr=1.89e-05 6867 tok/s 12.63s/it [7:36:47<37:28:34] mem=34.07GB |
| full_cpt step 2180/12852 ( 17.0%) nll=3.3631 ppl=28.88 grad=0.891 lr=1.89e-05 6869 tok/s 12.63s/it [7:38:43<37:25:38] mem=34.07GB |
| full_cpt step 2190/12852 ( 17.0%) nll=3.4803 ppl=32.47 grad=0.863 lr=1.89e-05 6872 tok/s 12.62s/it [7:40:40<37:22:43] mem=34.07GB |
| full_cpt step 2200/12852 ( 17.1%) nll=3.4452 ppl=31.35 grad=0.926 lr=1.88e-05 6874 tok/s 12.62s/it [7:42:36<37:19:47] mem=34.07GB |
| am nll=4.0786 ppl=59.06 bpb=1.4342 |
| ta nll=4.0440 ppl=57.05 bpb=0.5874 |
| kk nll=2.8280 ppl=16.91 bpb=0.9523 |
| tl nll=2.6738 ppl=14.49 bpb=1.2686 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 2200 01:27 (86.9s) |
| full_cpt step 2210/12852 ( 17.2%) nll=3.4542 ppl=31.63 grad=1.203 lr=1.88e-05 6855 tok/s 12.65s/it [7:45:58<37:23:51] mem=34.07GB |
| full_cpt step 2220/12852 ( 17.3%) nll=3.3421 ppl=28.28 grad=0.918 lr=1.88e-05 6858 tok/s 12.65s/it [7:47:55<37:20:54] mem=34.07GB |
| full_cpt step 2230/12852 ( 17.4%) nll=3.3061 ppl=27.28 grad=0.918 lr=1.88e-05 6860 tok/s 12.64s/it [7:49:51<37:17:58] mem=34.07GB |
| full_cpt step 2240/12852 ( 17.4%) nll=3.3513 ppl=28.54 grad=0.891 lr=1.88e-05 6863 tok/s 12.64s/it [7:51:47<37:15:03] mem=34.07GB |
| full_cpt step 2250/12852 ( 17.5%) nll=3.3691 ppl=29.05 grad=0.855 lr=1.88e-05 6864 tok/s 12.63s/it [7:53:43<37:12:08] mem=34.07GB |
| full_cpt step 2260/12852 ( 17.6%) nll=3.3389 ppl=28.19 grad=0.926 lr=1.88e-05 6867 tok/s 12.63s/it [7:55:39<37:09:13] mem=34.07GB |
| full_cpt step 2270/12852 ( 17.7%) nll=3.2810 ppl=26.60 grad=0.867 lr=1.88e-05 6870 tok/s 12.62s/it [7:57:35<37:06:20] mem=34.07GB |
| full_cpt step 2280/12852 ( 17.7%) nll=3.4405 ppl=31.20 grad=0.910 lr=1.88e-05 6873 tok/s 12.62s/it [7:59:31<37:03:27] mem=34.07GB |
| full_cpt step 2290/12852 ( 17.8%) nll=3.4135 ppl=30.37 grad=0.906 lr=1.87e-05 6875 tok/s 12.61s/it [8:01:27<37:00:34] mem=34.07GB |
| full_cpt step 2300/12852 ( 17.9%) nll=3.3808 ppl=29.39 grad=0.859 lr=1.87e-05 6878 tok/s 12.61s/it [8:03:23<36:57:42] mem=34.07GB |
| am nll=4.0700 ppl=58.56 bpb=1.4312 |
| ta nll=4.0363 ppl=56.61 bpb=0.5862 |
| kk nll=2.8222 ppl=16.81 bpb=0.9504 |
| tl nll=2.6725 ppl=14.48 bpb=1.2680 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 2300 01:27 (86.7s) |
| full_cpt step 2310/12852 ( 18.0%) nll=3.3639 ppl=28.90 grad=0.891 lr=1.87e-05 6859 tok/s 12.64s/it [8:06:46<37:01:25] mem=34.07GB |
| full_cpt step 2320/12852 ( 18.1%) nll=3.3103 ppl=27.39 grad=0.863 lr=1.87e-05 6861 tok/s 12.64s/it [8:08:42<36:58:32] mem=34.07GB |
| full_cpt step 2330/12852 ( 18.1%) nll=3.3149 ppl=27.52 grad=0.887 lr=1.87e-05 6864 tok/s 12.63s/it [8:10:38<36:55:39] mem=34.07GB |
| full_cpt step 2340/12852 ( 18.2%) nll=3.1849 ppl=24.17 grad=0.930 lr=1.87e-05 6865 tok/s 12.63s/it [8:12:34<36:52:45] mem=34.07GB |
| full_cpt step 2350/12852 ( 18.3%) nll=3.4273 ppl=30.79 grad=0.855 lr=1.87e-05 6868 tok/s 12.63s/it [8:14:30<36:49:54] mem=34.07GB |
| full_cpt step 2360/12852 ( 18.4%) nll=3.3490 ppl=28.47 grad=1.055 lr=1.87e-05 6870 tok/s 12.62s/it [8:16:26<36:47:02] mem=34.07GB |
| full_cpt step 2370/12852 ( 18.4%) nll=3.5643 ppl=35.32 grad=0.879 lr=1.87e-05 6872 tok/s 12.62s/it [8:18:22<36:44:11] mem=34.07GB |
| full_cpt step 2380/12852 ( 18.5%) nll=3.4909 ppl=32.82 grad=0.859 lr=1.86e-05 6874 tok/s 12.61s/it [8:20:18<36:41:20] mem=34.07GB |
| full_cpt step 2390/12852 ( 18.6%) nll=3.3274 ppl=27.87 grad=0.934 lr=1.86e-05 6877 tok/s 12.61s/it [8:22:14<36:38:30] mem=34.07GB |
| full_cpt step 2400/12852 ( 18.7%) nll=3.3946 ppl=29.80 grad=0.930 lr=1.86e-05 6879 tok/s 12.60s/it [8:24:10<36:35:41] mem=34.07GB |
| am nll=4.0635 ppl=58.18 bpb=1.4289 |
| ta nll=4.0282 ppl=56.16 bpb=0.5851 |
| kk nll=2.8159 ppl=16.71 bpb=0.9482 |
| tl nll=2.6715 ppl=14.46 bpb=1.2676 |
| en nll=2.8173 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 2400 01:27 (86.8s) |
| full_cpt step 2410/12852 ( 18.8%) nll=3.3297 ppl=27.93 grad=0.941 lr=1.86e-05 6862 tok/s 12.64s/it [8:27:33<36:39:07] mem=34.07GB |
| full_cpt step 2420/12852 ( 18.8%) nll=3.3200 ppl=27.66 grad=0.922 lr=1.86e-05 6864 tok/s 12.63s/it [8:29:29<36:36:17] mem=34.07GB |
| full_cpt step 2430/12852 ( 18.9%) nll=3.3080 ppl=27.33 grad=0.906 lr=1.86e-05 6866 tok/s 12.63s/it [8:31:25<36:33:27] mem=34.07GB |
| full_cpt step 2440/12852 ( 19.0%) nll=3.4604 ppl=31.83 grad=1.008 lr=1.86e-05 6868 tok/s 12.62s/it [8:33:22<36:30:37] mem=34.07GB |
| full_cpt step 2450/12852 ( 19.1%) nll=3.3371 ppl=28.14 grad=0.848 lr=1.86e-05 6871 tok/s 12.62s/it [8:35:18<36:27:47] mem=34.07GB |
| full_cpt step 2460/12852 ( 19.1%) nll=3.4006 ppl=29.98 grad=0.840 lr=1.85e-05 6872 tok/s 12.62s/it [8:37:13<36:24:58] mem=34.07GB |
| full_cpt step 2470/12852 ( 19.2%) nll=3.3719 ppl=29.13 grad=0.941 lr=1.85e-05 6874 tok/s 12.61s/it [8:39:09<36:22:09] mem=34.07GB |
| full_cpt step 2480/12852 ( 19.3%) nll=3.2523 ppl=25.85 grad=0.934 lr=1.85e-05 6876 tok/s 12.61s/it [8:41:05<36:19:20] mem=34.07GB |
| full_cpt step 2490/12852 ( 19.4%) nll=3.1766 ppl=23.97 grad=0.844 lr=1.85e-05 6878 tok/s 12.60s/it [8:43:02<36:16:33] mem=34.07GB |
| full_cpt step 2500/12852 ( 19.5%) nll=3.3358 ppl=28.10 grad=1.016 lr=1.85e-05 6880 tok/s 12.60s/it [8:44:58<36:13:46] mem=34.07GB |
| am nll=4.0553 ppl=57.70 bpb=1.4260 |
| ta nll=4.0217 ppl=55.79 bpb=0.5841 |
| kk nll=2.8106 ppl=16.62 bpb=0.9464 |
| tl nll=2.6706 ppl=14.45 bpb=1.2671 |
| en nll=2.8175 ppl=16.74 bpb=0.8772 |
| eval full_cpt step 2500 01:27 (87.4s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.40s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.40s/it] |
| removing old checkpoint checkpoint-1500 |
| kept last 2 resume checkpoints: checkpoint-2000 checkpoint-2500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-2500 at step 2500 next_row=320000 |
| full_cpt step 2510/12852 ( 19.5%) nll=3.5412 ppl=34.51 grad=0.852 lr=1.85e-05 6849 tok/s 12.66s/it [8:49:30<36:21:43] mem=34.07GB |
| full_cpt step 2520/12852 ( 19.6%) nll=3.3689 ppl=29.05 grad=0.918 lr=1.85e-05 6852 tok/s 12.65s/it [8:51:26<36:18:54] mem=34.07GB |
| full_cpt step 2530/12852 ( 19.7%) nll=3.4242 ppl=30.70 grad=0.871 lr=1.85e-05 6854 tok/s 12.65s/it [8:53:22<36:16:05] mem=34.07GB |
| full_cpt step 2540/12852 ( 19.8%) nll=3.3884 ppl=29.62 grad=0.945 lr=1.85e-05 6856 tok/s 12.65s/it [8:55:18<36:13:16] mem=34.07GB |
| full_cpt step 2550/12852 ( 19.8%) nll=3.3614 ppl=28.83 grad=0.887 lr=1.84e-05 6858 tok/s 12.64s/it [8:57:14<36:10:27] mem=34.07GB |
| full_cpt step 2560/12852 ( 19.9%) nll=3.3251 ppl=27.80 grad=0.910 lr=1.84e-05 6860 tok/s 12.64s/it [8:59:10<36:07:39] mem=34.07GB |
| full_cpt step 2570/12852 ( 20.0%) nll=3.3147 ppl=27.51 grad=0.852 lr=1.84e-05 6862 tok/s 12.63s/it [9:01:07<36:04:52] mem=34.07GB |
| full_cpt step 2580/12852 ( 20.1%) nll=3.3842 ppl=29.49 grad=0.867 lr=1.84e-05 6865 tok/s 12.63s/it [9:03:03<36:02:05] mem=34.07GB |
| full_cpt step 2590/12852 ( 20.2%) nll=3.3318 ppl=27.99 grad=0.883 lr=1.84e-05 6867 tok/s 12.63s/it [9:04:59<35:59:18] mem=34.07GB |
| full_cpt step 2600/12852 ( 20.2%) nll=3.3510 ppl=28.53 grad=0.898 lr=1.84e-05 6870 tok/s 12.62s/it [9:06:55<35:56:32] mem=34.07GB |
| am nll=4.0492 ppl=57.35 bpb=1.4239 |
| ta nll=4.0158 ppl=55.47 bpb=0.5833 |
| kk nll=2.8058 ppl=16.54 bpb=0.9448 |
| tl nll=2.6697 ppl=14.44 bpb=1.2667 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 2600 01:27 (87.4s) |
| full_cpt step 2610/12852 ( 20.3%) nll=3.4266 ppl=30.77 grad=0.902 lr=1.84e-05 6854 tok/s 12.65s/it [9:10:18<35:59:29] mem=34.07GB |
| full_cpt step 2620/12852 ( 20.4%) nll=3.5226 ppl=33.87 grad=0.910 lr=1.83e-05 6856 tok/s 12.65s/it [9:12:14<35:56:41] mem=34.07GB |
| full_cpt step 2630/12852 ( 20.5%) nll=3.4147 ppl=30.41 grad=0.934 lr=1.83e-05 6858 tok/s 12.64s/it [9:14:11<35:53:54] mem=34.07GB |
| full_cpt step 2640/12852 ( 20.5%) nll=3.3771 ppl=29.29 grad=0.914 lr=1.83e-05 6860 tok/s 12.64s/it [9:16:06<35:51:07] mem=34.07GB |
| full_cpt step 2650/12852 ( 20.6%) nll=3.2391 ppl=25.51 grad=0.957 lr=1.83e-05 6862 tok/s 12.63s/it [9:18:02<35:48:21] mem=34.07GB |
| full_cpt step 2660/12852 ( 20.7%) nll=3.3965 ppl=29.86 grad=0.961 lr=1.83e-05 6864 tok/s 12.63s/it [9:19:59<35:45:36] mem=34.07GB |
| full_cpt step 2670/12852 ( 20.8%) nll=3.3516 ppl=28.55 grad=0.898 lr=1.83e-05 6867 tok/s 12.63s/it [9:21:55<35:42:50] mem=34.07GB |
| full_cpt step 2680/12852 ( 20.9%) nll=3.4038 ppl=30.08 grad=0.879 lr=1.83e-05 6869 tok/s 12.62s/it [9:23:51<35:40:05] mem=34.07GB |
| full_cpt step 2690/12852 ( 20.9%) nll=3.2725 ppl=26.38 grad=0.961 lr=1.83e-05 6870 tok/s 12.62s/it [9:25:47<35:37:20] mem=34.07GB |
| full_cpt step 2700/12852 ( 21.0%) nll=3.3709 ppl=29.11 grad=0.875 lr=1.82e-05 6872 tok/s 12.62s/it [9:27:43<35:34:36] mem=34.07GB |
| am nll=4.0434 ppl=57.02 bpb=1.4219 |
| ta nll=4.0096 ppl=55.12 bpb=0.5824 |
| kk nll=2.8017 ppl=16.47 bpb=0.9434 |
| tl nll=2.6688 ppl=14.42 bpb=1.2663 |
| en nll=2.8171 ppl=16.73 bpb=0.8771 |
| eval full_cpt step 2700 01:27 (87.2s) |
| full_cpt step 2710/12852 ( 21.1%) nll=3.2168 ppl=24.95 grad=0.910 lr=1.82e-05 6858 tok/s 12.64s/it [9:31:06<35:37:19] mem=34.07GB |
| full_cpt step 2720/12852 ( 21.2%) nll=3.3410 ppl=28.25 grad=0.836 lr=1.82e-05 6860 tok/s 12.64s/it [9:33:02<35:34:34] mem=34.07GB |
| full_cpt step 2730/12852 ( 21.2%) nll=3.3651 ppl=28.94 grad=0.887 lr=1.82e-05 6862 tok/s 12.64s/it [9:34:58<35:31:49] mem=34.07GB |
| full_cpt step 2740/12852 ( 21.3%) nll=3.4179 ppl=30.50 grad=0.926 lr=1.82e-05 6863 tok/s 12.63s/it [9:36:54<35:29:05] mem=34.07GB |
| full_cpt step 2750/12852 ( 21.4%) nll=3.3009 ppl=27.14 grad=0.871 lr=1.82e-05 6865 tok/s 12.63s/it [9:38:50<35:26:21] mem=34.07GB |
| full_cpt step 2760/12852 ( 21.5%) nll=3.4507 ppl=31.52 grad=0.879 lr=1.82e-05 6867 tok/s 12.63s/it [9:40:47<35:23:37] mem=34.07GB |
| full_cpt step 2770/12852 ( 21.6%) nll=3.3995 ppl=29.95 grad=0.879 lr=1.82e-05 6869 tok/s 12.62s/it [9:42:43<35:20:54] mem=34.07GB |
| full_cpt step 2780/12852 ( 21.6%) nll=3.3821 ppl=29.43 grad=0.859 lr=1.81e-05 6871 tok/s 12.62s/it [9:44:39<35:18:11] mem=34.07GB |
| full_cpt step 2790/12852 ( 21.7%) nll=3.4893 ppl=32.76 grad=0.953 lr=1.81e-05 6873 tok/s 12.61s/it [9:46:35<35:15:28] mem=34.07GB |
| full_cpt step 2800/12852 ( 21.8%) nll=3.3556 ppl=28.66 grad=0.855 lr=1.81e-05 6874 tok/s 12.61s/it [9:48:31<35:12:45] mem=34.07GB |
| am nll=4.0375 ppl=56.68 bpb=1.4198 |
| ta nll=4.0043 ppl=54.84 bpb=0.5816 |
| kk nll=2.7974 ppl=16.40 bpb=0.9420 |
| tl nll=2.6681 ppl=14.41 bpb=1.2659 |
| en nll=2.8179 ppl=16.74 bpb=0.8774 |
| eval full_cpt step 2800 01:29 (88.6s) |
| full_cpt step 2810/12852 ( 21.9%) nll=3.2925 ppl=26.91 grad=0.828 lr=1.81e-05 6859 tok/s 12.64s/it [9:51:55<35:15:20] mem=34.07GB |
| full_cpt step 2820/12852 ( 21.9%) nll=3.2135 ppl=24.86 grad=0.871 lr=1.81e-05 6862 tok/s 12.64s/it [9:53:51<35:12:37] mem=34.07GB |
| full_cpt step 2830/12852 ( 22.0%) nll=3.3102 ppl=27.39 grad=0.848 lr=1.81e-05 6864 tok/s 12.63s/it [9:55:47<35:09:54] mem=34.07GB |
| full_cpt step 2840/12852 ( 22.1%) nll=3.3786 ppl=29.33 grad=0.867 lr=1.81e-05 6866 tok/s 12.63s/it [9:57:43<35:07:11] mem=34.07GB |
| full_cpt step 2850/12852 ( 22.2%) nll=3.4983 ppl=33.06 grad=0.844 lr=1.80e-05 6867 tok/s 12.62s/it [9:59:39<35:04:29] mem=34.07GB |
| full_cpt step 2860/12852 ( 22.3%) nll=3.4347 ppl=31.02 grad=0.875 lr=1.80e-05 6869 tok/s 12.62s/it [10:01:35<35:01:47] mem=34.07GB |
| full_cpt step 2870/12852 ( 22.3%) nll=3.4857 ppl=32.65 grad=0.992 lr=1.80e-05 6870 tok/s 12.62s/it [10:03:31<34:59:05] mem=34.07GB |
| full_cpt step 2880/12852 ( 22.4%) nll=3.2174 ppl=24.96 grad=0.879 lr=1.80e-05 6872 tok/s 12.61s/it [10:05:28<34:56:24] mem=34.07GB |
| full_cpt step 2890/12852 ( 22.5%) nll=3.2848 ppl=26.70 grad=0.930 lr=1.80e-05 6874 tok/s 12.61s/it [10:07:24<34:53:44] mem=34.07GB |
| full_cpt step 2900/12852 ( 22.6%) nll=3.3325 ppl=28.01 grad=0.852 lr=1.80e-05 6876 tok/s 12.61s/it [10:09:20<34:51:04] mem=34.07GB |
| am nll=4.0324 ppl=56.40 bpb=1.4180 |
| ta nll=3.9994 ppl=54.56 bpb=0.5809 |
| kk nll=2.7936 ppl=16.34 bpb=0.9407 |
| tl nll=2.6673 ppl=14.40 bpb=1.2656 |
| en nll=2.8178 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 2900 01:27 (87.5s) |
| full_cpt step 2910/12852 ( 22.6%) nll=3.1911 ppl=24.31 grad=0.891 lr=1.80e-05 6862 tok/s 12.63s/it [10:12:43<34:53:22] mem=34.07GB |
| full_cpt step 2920/12852 ( 22.7%) nll=3.4941 ppl=32.92 grad=0.934 lr=1.79e-05 6863 tok/s 12.63s/it [10:14:39<34:50:40] mem=34.07GB |
| full_cpt step 2930/12852 ( 22.8%) nll=3.4455 ppl=31.36 grad=0.898 lr=1.79e-05 6865 tok/s 12.63s/it [10:16:36<34:48:00] mem=34.07GB |
| full_cpt step 2940/12852 ( 22.9%) nll=3.5284 ppl=34.07 grad=0.871 lr=1.79e-05 6867 tok/s 12.62s/it [10:18:32<34:45:19] mem=34.07GB |
| full_cpt step 2950/12852 ( 23.0%) nll=3.2054 ppl=24.66 grad=0.875 lr=1.79e-05 6869 tok/s 12.62s/it [10:20:28<34:42:39] mem=34.07GB |
| full_cpt step 2960/12852 ( 23.0%) nll=3.4266 ppl=30.77 grad=0.879 lr=1.79e-05 6871 tok/s 12.62s/it [10:22:24<34:39:59] mem=34.07GB |
| full_cpt step 2970/12852 ( 23.1%) nll=3.2708 ppl=26.33 grad=0.945 lr=1.79e-05 6873 tok/s 12.61s/it [10:24:20<34:37:19] mem=34.07GB |
| full_cpt step 2980/12852 ( 23.2%) nll=3.2657 ppl=26.20 grad=0.812 lr=1.79e-05 6874 tok/s 12.61s/it [10:26:16<34:34:39] mem=34.07GB |
| full_cpt step 2990/12852 ( 23.3%) nll=3.3059 ppl=27.27 grad=0.906 lr=1.78e-05 6876 tok/s 12.61s/it [10:28:12<34:32:01] mem=34.07GB |
| full_cpt step 3000/12852 ( 23.3%) nll=3.2788 ppl=26.54 grad=0.859 lr=1.78e-05 6878 tok/s 12.60s/it [10:30:08<34:29:22] mem=34.07GB |
| am nll=4.0275 ppl=56.12 bpb=1.4163 |
| ta nll=3.9949 ppl=54.32 bpb=0.5802 |
| kk nll=2.7903 ppl=16.29 bpb=0.9396 |
| tl nll=2.6668 ppl=14.39 bpb=1.2654 |
| en nll=2.8175 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 3000 01:27 (87.4s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.33s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.34s/it] |
| removing old checkpoint checkpoint-2000 |
| kept last 2 resume checkpoints: checkpoint-2500 checkpoint-3000 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-3000 at step 3000 next_row=384000 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.88s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.88s/it] |
| wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-3000 (weights+expanded tokenizer, no optimizer) |
| full_cpt step 3010/12852 ( 23.4%) nll=3.2212 ppl=25.06 grad=0.898 lr=1.78e-05 6847 tok/s 12.66s/it [10:35:07<34:36:40] mem=34.07GB |
| full_cpt step 3020/12852 ( 23.5%) nll=3.5225 ppl=33.87 grad=0.953 lr=1.78e-05 6849 tok/s 12.66s/it [10:37:03<34:33:59] mem=34.07GB |
| full_cpt step 3030/12852 ( 23.6%) nll=3.3544 ppl=28.63 grad=0.965 lr=1.78e-05 6851 tok/s 12.65s/it [10:38:59<34:31:18] mem=34.07GB |
| full_cpt step 3040/12852 ( 23.7%) nll=3.2071 ppl=24.71 grad=0.867 lr=1.78e-05 6853 tok/s 12.65s/it [10:40:55<34:28:38] mem=34.07GB |
| full_cpt step 3050/12852 ( 23.7%) nll=3.3298 ppl=27.93 grad=0.824 lr=1.78e-05 6855 tok/s 12.65s/it [10:42:51<34:25:58] mem=34.07GB |
| full_cpt step 3060/12852 ( 23.8%) nll=3.2161 ppl=24.93 grad=0.895 lr=1.77e-05 6857 tok/s 12.64s/it [10:44:47<34:23:19] mem=34.07GB |
| full_cpt step 3070/12852 ( 23.9%) nll=3.3118 ppl=27.43 grad=0.867 lr=1.77e-05 6858 tok/s 12.64s/it [10:46:43<34:20:40] mem=34.07GB |
| full_cpt step 3080/12852 ( 24.0%) nll=3.3263 ppl=27.83 grad=0.871 lr=1.77e-05 6860 tok/s 12.64s/it [10:48:39<34:18:00] mem=34.07GB |
| full_cpt step 3090/12852 ( 24.0%) nll=3.3137 ppl=27.49 grad=0.906 lr=1.77e-05 6861 tok/s 12.63s/it [10:50:35<34:15:21] mem=34.07GB |
| full_cpt step 3100/12852 ( 24.1%) nll=3.3402 ppl=28.22 grad=0.906 lr=1.77e-05 6863 tok/s 12.63s/it [10:52:31<34:12:43] mem=34.07GB |
| am nll=4.0230 ppl=55.87 bpb=1.4147 |
| ta nll=3.9910 ppl=54.11 bpb=0.5797 |
| kk nll=2.7872 ppl=16.23 bpb=0.9386 |
| tl nll=2.6665 ppl=14.39 bpb=1.2652 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 3100 01:29 (88.8s) |
| full_cpt step 3110/12852 ( 24.2%) nll=3.3203 ppl=27.67 grad=0.926 lr=1.77e-05 6849 tok/s 12.65s/it [10:55:56<34:14:42] mem=34.07GB |
| full_cpt step 3120/12852 ( 24.3%) nll=3.3614 ppl=28.83 grad=0.887 lr=1.77e-05 6851 tok/s 12.65s/it [10:57:52<34:12:03] mem=34.07GB |
| full_cpt step 3130/12852 ( 24.4%) nll=3.3152 ppl=27.53 grad=0.977 lr=1.76e-05 6853 tok/s 12.65s/it [10:59:48<34:09:24] mem=34.07GB |
| full_cpt step 3140/12852 ( 24.4%) nll=3.2643 ppl=26.16 grad=0.895 lr=1.76e-05 6855 tok/s 12.64s/it [11:01:45<34:06:46] mem=34.07GB |
| full_cpt step 3150/12852 ( 24.5%) nll=3.3826 ppl=29.45 grad=0.949 lr=1.76e-05 6857 tok/s 12.64s/it [11:03:41<34:04:08] mem=34.07GB |
| full_cpt step 3160/12852 ( 24.6%) nll=3.1855 ppl=24.18 grad=0.863 lr=1.76e-05 6859 tok/s 12.64s/it [11:05:37<34:01:30] mem=34.07GB |
| full_cpt step 3170/12852 ( 24.7%) nll=3.3632 ppl=28.88 grad=0.895 lr=1.76e-05 6860 tok/s 12.64s/it [11:07:33<33:58:52] mem=34.07GB |
| full_cpt step 3180/12852 ( 24.7%) nll=3.3905 ppl=29.68 grad=0.867 lr=1.76e-05 6863 tok/s 12.63s/it [11:09:29<33:56:15] mem=34.07GB |
| full_cpt step 3190/12852 ( 24.8%) nll=3.3715 ppl=29.12 grad=0.852 lr=1.75e-05 6864 tok/s 12.63s/it [11:11:25<33:53:37] mem=34.07GB |
| full_cpt step 3200/12852 ( 24.9%) nll=3.3026 ppl=27.18 grad=0.898 lr=1.75e-05 6866 tok/s 12.63s/it [11:13:21<33:51:00] mem=34.07GB |
| am nll=4.0185 ppl=55.62 bpb=1.4131 |
| ta nll=3.9863 ppl=53.86 bpb=0.5790 |
| kk nll=2.7841 ppl=16.18 bpb=0.9375 |
| tl nll=2.6662 ppl=14.39 bpb=1.2651 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 3200 01:26 (86.5s) |
| full_cpt step 3210/12852 ( 25.0%) nll=3.4051 ppl=30.12 grad=0.957 lr=1.75e-05 6852 tok/s 12.65s/it [11:16:44<33:52:43] mem=34.07GB |
| full_cpt step 3220/12852 ( 25.1%) nll=3.3297 ppl=27.93 grad=0.930 lr=1.75e-05 6854 tok/s 12.65s/it [11:18:40<33:50:05] mem=34.07GB |
| full_cpt step 3230/12852 ( 25.1%) nll=3.4816 ppl=32.51 grad=0.973 lr=1.75e-05 6857 tok/s 12.64s/it [11:20:36<33:47:28] mem=34.07GB |
| full_cpt step 3240/12852 ( 25.2%) nll=3.3376 ppl=28.15 grad=0.930 lr=1.75e-05 6858 tok/s 12.64s/it [11:22:32<33:44:51] mem=34.07GB |
| full_cpt step 3250/12852 ( 25.3%) nll=3.2956 ppl=26.99 grad=0.832 lr=1.75e-05 6860 tok/s 12.64s/it [11:24:28<33:42:15] mem=34.07GB |
| full_cpt step 3260/12852 ( 25.4%) nll=3.2708 ppl=26.33 grad=0.918 lr=1.74e-05 6862 tok/s 12.63s/it [11:26:24<33:39:38] mem=34.07GB |
| full_cpt step 3270/12852 ( 25.4%) nll=3.4030 ppl=30.05 grad=0.957 lr=1.74e-05 6863 tok/s 12.63s/it [11:28:20<33:37:02] mem=34.07GB |
| full_cpt step 3280/12852 ( 25.5%) nll=3.4093 ppl=30.24 grad=1.000 lr=1.74e-05 6865 tok/s 12.63s/it [11:30:17<33:34:26] mem=34.07GB |
| full_cpt step 3290/12852 ( 25.6%) nll=3.3485 ppl=28.46 grad=0.836 lr=1.74e-05 6867 tok/s 12.62s/it [11:32:13<33:31:50] mem=34.07GB |
| full_cpt step 3300/12852 ( 25.7%) nll=3.3036 ppl=27.21 grad=0.922 lr=1.74e-05 6870 tok/s 12.62s/it [11:34:09<33:29:15] mem=34.07GB |
| am nll=4.0155 ppl=55.45 bpb=1.4121 |
| ta nll=3.9827 ppl=53.66 bpb=0.5785 |
| kk nll=2.7813 ppl=16.14 bpb=0.9366 |
| tl nll=2.6652 ppl=14.37 bpb=1.2646 |
| en nll=2.8179 ppl=16.74 bpb=0.8774 |
| eval full_cpt step 3300 01:28 (87.6s) |
| full_cpt step 3310/12852 ( 25.8%) nll=3.1741 ppl=23.91 grad=0.867 lr=1.74e-05 6857 tok/s 12.64s/it [11:37:33<33:30:52] mem=34.07GB |
| full_cpt step 3320/12852 ( 25.8%) nll=3.3239 ppl=27.77 grad=0.840 lr=1.73e-05 6858 tok/s 12.64s/it [11:39:29<33:28:16] mem=34.07GB |
| full_cpt step 3330/12852 ( 25.9%) nll=3.3961 ppl=29.85 grad=0.871 lr=1.73e-05 6860 tok/s 12.64s/it [11:41:25<33:25:40] mem=34.07GB |
| full_cpt step 3340/12852 ( 26.0%) nll=3.2743 ppl=26.43 grad=0.902 lr=1.73e-05 6861 tok/s 12.63s/it [11:43:21<33:23:03] mem=34.07GB |
| full_cpt step 3350/12852 ( 26.1%) nll=3.3892 ppl=29.64 grad=0.895 lr=1.73e-05 6863 tok/s 12.63s/it [11:45:17<33:20:28] mem=34.07GB |
| full_cpt step 3360/12852 ( 26.1%) nll=3.2678 ppl=26.25 grad=0.945 lr=1.73e-05 6864 tok/s 12.63s/it [11:47:13<33:17:52] mem=34.07GB |
| full_cpt step 3370/12852 ( 26.2%) nll=3.3735 ppl=29.18 grad=0.906 lr=1.73e-05 6866 tok/s 12.63s/it [11:49:09<33:15:17] mem=34.07GB |
| full_cpt step 3380/12852 ( 26.3%) nll=3.3481 ppl=28.45 grad=0.906 lr=1.73e-05 6868 tok/s 12.62s/it [11:51:05<33:12:42] mem=34.07GB |
| full_cpt step 3390/12852 ( 26.4%) nll=3.4800 ppl=32.46 grad=0.855 lr=1.72e-05 6869 tok/s 12.62s/it [11:53:01<33:10:08] mem=34.07GB |
| full_cpt step 3400/12852 ( 26.5%) nll=3.4799 ppl=32.46 grad=0.914 lr=1.72e-05 6870 tok/s 12.62s/it [11:54:57<33:07:33] mem=34.07GB |
| am nll=4.0123 ppl=55.27 bpb=1.4109 |
| ta nll=3.9802 ppl=53.53 bpb=0.5781 |
| kk nll=2.7792 ppl=16.11 bpb=0.9359 |
| tl nll=2.6655 ppl=14.37 bpb=1.2647 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 3400 01:27 (87.2s) |
| full_cpt step 3410/12852 ( 26.5%) nll=3.2276 ppl=25.22 grad=0.891 lr=1.72e-05 6858 tok/s 12.64s/it [11:58:20<33:09:00] mem=34.07GB |
| full_cpt step 3420/12852 ( 26.6%) nll=3.3239 ppl=27.77 grad=0.879 lr=1.72e-05 6860 tok/s 12.64s/it [12:00:16<33:06:25] mem=34.07GB |
| full_cpt step 3430/12852 ( 26.7%) nll=3.3931 ppl=29.76 grad=0.941 lr=1.72e-05 6861 tok/s 12.63s/it [12:02:12<33:03:51] mem=34.07GB |
| full_cpt step 3440/12852 ( 26.8%) nll=3.3246 ppl=27.79 grad=0.863 lr=1.72e-05 6863 tok/s 12.63s/it [12:04:08<33:01:16] mem=34.07GB |
| full_cpt step 3450/12852 ( 26.8%) nll=3.2674 ppl=26.24 grad=0.871 lr=1.71e-05 6864 tok/s 12.63s/it [12:06:04<32:58:42] mem=34.07GB |
| full_cpt step 3460/12852 ( 26.9%) nll=3.3705 ppl=29.09 grad=0.820 lr=1.71e-05 6865 tok/s 12.62s/it [12:08:00<32:56:08] mem=34.07GB |
| full_cpt step 3470/12852 ( 27.0%) nll=3.3075 ppl=27.32 grad=0.844 lr=1.71e-05 6867 tok/s 12.62s/it [12:09:56<32:53:34] mem=34.07GB |
| full_cpt step 3480/12852 ( 27.1%) nll=3.3699 ppl=29.07 grad=0.855 lr=1.71e-05 6869 tok/s 12.62s/it [12:11:52<32:51:01] mem=34.07GB |
| full_cpt step 3490/12852 ( 27.2%) nll=3.4049 ppl=30.11 grad=0.910 lr=1.71e-05 6870 tok/s 12.62s/it [12:13:48<32:48:27] mem=34.07GB |
| full_cpt step 3500/12852 ( 27.2%) nll=3.2355 ppl=25.42 grad=0.844 lr=1.71e-05 6872 tok/s 12.61s/it [12:15:45<32:45:54] mem=34.07GB |
| am nll=4.0086 ppl=55.07 bpb=1.4096 |
| ta nll=3.9767 ppl=53.34 bpb=0.5776 |
| kk nll=2.7766 ppl=16.06 bpb=0.9350 |
| tl nll=2.6644 ppl=14.36 bpb=1.2642 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 3500 01:27 (86.7s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.38s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.38s/it] |
| removing old checkpoint checkpoint-2500 |
| kept last 2 resume checkpoints: checkpoint-3000 checkpoint-3500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-3500 at step 3500 next_row=448000 |
| full_cpt step 3510/12852 ( 27.3%) nll=3.1593 ppl=23.55 grad=0.871 lr=1.70e-05 6848 tok/s 12.66s/it [12:20:25<32:50:38] mem=34.07GB |
| full_cpt step 3520/12852 ( 27.4%) nll=3.2587 ppl=26.01 grad=0.906 lr=1.70e-05 6849 tok/s 12.65s/it [12:22:21<32:48:04] mem=34.07GB |
| full_cpt step 3530/12852 ( 27.5%) nll=3.3581 ppl=28.73 grad=0.941 lr=1.70e-05 6850 tok/s 12.65s/it [12:24:17<32:45:29] mem=34.07GB |
| full_cpt step 3540/12852 ( 27.5%) nll=3.1733 ppl=23.89 grad=0.930 lr=1.70e-05 6852 tok/s 12.65s/it [12:26:13<32:42:55] mem=34.07GB |
| full_cpt step 3550/12852 ( 27.6%) nll=3.3385 ppl=28.18 grad=0.867 lr=1.70e-05 6853 tok/s 12.64s/it [12:28:09<32:40:21] mem=34.07GB |
| full_cpt step 3560/12852 ( 27.7%) nll=3.2603 ppl=26.06 grad=0.852 lr=1.70e-05 6855 tok/s 12.64s/it [12:30:05<32:37:48] mem=34.07GB |
| full_cpt step 3570/12852 ( 27.8%) nll=3.3034 ppl=27.20 grad=0.910 lr=1.69e-05 6856 tok/s 12.64s/it [12:32:01<32:35:14] mem=34.07GB |
| full_cpt step 3580/12852 ( 27.9%) nll=3.3650 ppl=28.93 grad=0.871 lr=1.69e-05 6858 tok/s 12.64s/it [12:33:57<32:32:41] mem=34.07GB |
| full_cpt step 3590/12852 ( 27.9%) nll=3.1906 ppl=24.30 grad=0.855 lr=1.69e-05 6859 tok/s 12.63s/it [12:35:53<32:30:08] mem=34.07GB |
| full_cpt step 3600/12852 ( 28.0%) nll=3.1864 ppl=24.20 grad=0.871 lr=1.69e-05 6860 tok/s 12.63s/it [12:37:49<32:27:35] mem=34.07GB |
| am nll=4.0069 ppl=54.98 bpb=1.4090 |
| ta nll=3.9750 ppl=53.25 bpb=0.5773 |
| kk nll=2.7747 ppl=16.03 bpb=0.9344 |
| tl nll=2.6645 ppl=14.36 bpb=1.2642 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 3600 01:27 (87.3s) |
| full_cpt step 3610/12852 ( 28.1%) nll=3.4219 ppl=30.63 grad=0.918 lr=1.69e-05 6849 tok/s 12.65s/it [12:41:12<32:28:46] mem=34.07GB |
| full_cpt step 3620/12852 ( 28.2%) nll=3.3176 ppl=27.59 grad=0.879 lr=1.69e-05 6850 tok/s 12.65s/it [12:43:08<32:26:13] mem=34.07GB |
| full_cpt step 3630/12852 ( 28.2%) nll=3.2296 ppl=25.27 grad=0.879 lr=1.68e-05 6851 tok/s 12.65s/it [12:45:04<32:23:40] mem=34.07GB |
| full_cpt step 3640/12852 ( 28.3%) nll=3.4400 ppl=31.19 grad=0.855 lr=1.68e-05 6853 tok/s 12.64s/it [12:47:00<32:21:07] mem=34.07GB |
| full_cpt step 3650/12852 ( 28.4%) nll=3.3133 ppl=27.48 grad=0.875 lr=1.68e-05 6855 tok/s 12.64s/it [12:48:57<32:18:35] mem=34.07GB |
| full_cpt step 3660/12852 ( 28.5%) nll=3.3801 ppl=29.37 grad=0.859 lr=1.68e-05 6856 tok/s 12.64s/it [12:50:53<32:16:02] mem=34.07GB |
| full_cpt step 3670/12852 ( 28.6%) nll=3.4741 ppl=32.27 grad=0.922 lr=1.68e-05 6857 tok/s 12.63s/it [12:52:49<32:13:30] mem=34.07GB |
| full_cpt step 3680/12852 ( 28.6%) nll=3.3065 ppl=27.29 grad=0.863 lr=1.68e-05 6859 tok/s 12.63s/it [12:54:45<32:10:58] mem=34.07GB |
| full_cpt step 3690/12852 ( 28.7%) nll=3.3576 ppl=28.72 grad=0.809 lr=1.67e-05 6861 tok/s 12.63s/it [12:56:41<32:08:27] mem=34.07GB |
| full_cpt step 3700/12852 ( 28.8%) nll=3.3239 ppl=27.77 grad=0.855 lr=1.67e-05 6863 tok/s 12.63s/it [12:58:37<32:05:55] mem=34.07GB |
| am nll=4.0026 ppl=54.74 bpb=1.4075 |
| ta nll=3.9726 ppl=53.12 bpb=0.5770 |
| kk nll=2.7729 ppl=16.00 bpb=0.9337 |
| tl nll=2.6637 ppl=14.35 bpb=1.2639 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 3700 01:27 (87.1s) |
| full_cpt step 3710/12852 ( 28.9%) nll=3.2854 ppl=26.72 grad=0.879 lr=1.67e-05 6852 tok/s 12.65s/it [13:02:00<32:06:59] mem=34.07GB |
| full_cpt step 3720/12852 ( 28.9%) nll=3.3596 ppl=28.78 grad=0.922 lr=1.67e-05 6854 tok/s 12.64s/it [13:03:56<32:04:27] mem=34.07GB |
| full_cpt step 3730/12852 ( 29.0%) nll=3.3704 ppl=29.09 grad=0.879 lr=1.67e-05 6855 tok/s 12.64s/it [13:05:52<32:01:55] mem=34.07GB |
| full_cpt step 3740/12852 ( 29.1%) nll=3.3628 ppl=28.87 grad=0.824 lr=1.67e-05 6856 tok/s 12.64s/it [13:07:49<31:59:23] mem=34.07GB |
| full_cpt step 3750/12852 ( 29.2%) nll=3.3194 ppl=27.64 grad=0.867 lr=1.66e-05 6858 tok/s 12.64s/it [13:09:45<31:56:52] mem=34.07GB |
| full_cpt step 3760/12852 ( 29.3%) nll=3.3252 ppl=27.80 grad=0.891 lr=1.66e-05 6860 tok/s 12.63s/it [13:11:41<31:54:21] mem=34.07GB |
| full_cpt step 3770/12852 ( 29.3%) nll=3.2963 ppl=27.01 grad=0.867 lr=1.66e-05 6861 tok/s 12.63s/it [13:13:37<31:51:50] mem=34.07GB |
| full_cpt step 3780/12852 ( 29.4%) nll=3.4094 ppl=30.25 grad=0.930 lr=1.66e-05 6863 tok/s 12.63s/it [13:15:33<31:49:20] mem=34.07GB |
| full_cpt step 3790/12852 ( 29.5%) nll=3.3182 ppl=27.61 grad=0.875 lr=1.66e-05 6864 tok/s 12.63s/it [13:17:29<31:46:49] mem=34.07GB |
| full_cpt step 3800/12852 ( 29.6%) nll=3.3543 ppl=28.62 grad=0.898 lr=1.65e-05 6865 tok/s 12.62s/it [13:19:25<31:44:18] mem=34.07GB |
| am nll=4.0004 ppl=54.62 bpb=1.4067 |
| ta nll=3.9692 ppl=52.94 bpb=0.5765 |
| kk nll=2.7712 ppl=15.98 bpb=0.9332 |
| tl nll=2.6634 ppl=14.35 bpb=1.2637 |
| en nll=2.8179 ppl=16.74 bpb=0.8774 |
| eval full_cpt step 3800 01:27 (87.0s) |
| full_cpt step 3810/12852 ( 29.6%) nll=3.3173 ppl=27.58 grad=0.879 lr=1.65e-05 6854 tok/s 12.64s/it [13:22:48<31:45:14] mem=34.07GB |
| full_cpt step 3820/12852 ( 29.7%) nll=3.3041 ppl=27.22 grad=0.930 lr=1.65e-05 6855 tok/s 12.64s/it [13:24:44<31:42:43] mem=34.07GB |
| full_cpt step 3830/12852 ( 29.8%) nll=3.3351 ppl=28.08 grad=0.891 lr=1.65e-05 6857 tok/s 12.64s/it [13:26:40<31:40:13] mem=34.07GB |
| full_cpt step 3840/12852 ( 29.9%) nll=3.3208 ppl=27.68 grad=0.945 lr=1.65e-05 6858 tok/s 12.63s/it [13:28:37<31:37:42] mem=34.07GB |
| full_cpt step 3850/12852 ( 30.0%) nll=3.3384 ppl=28.18 grad=0.914 lr=1.65e-05 6859 tok/s 12.63s/it [13:30:33<31:35:12] mem=34.07GB |
| full_cpt step 3860/12852 ( 30.0%) nll=3.2677 ppl=26.25 grad=0.906 lr=1.64e-05 6861 tok/s 12.63s/it [13:32:29<31:32:42] mem=34.07GB |
| full_cpt step 3870/12852 ( 30.1%) nll=3.2883 ppl=26.80 grad=0.949 lr=1.64e-05 6863 tok/s 12.63s/it [13:34:25<31:30:12] mem=34.07GB |
| full_cpt step 3880/12852 ( 30.2%) nll=3.3466 ppl=28.41 grad=0.859 lr=1.64e-05 6864 tok/s 12.62s/it [13:36:21<31:27:42] mem=34.07GB |
| full_cpt step 3890/12852 ( 30.3%) nll=3.3232 ppl=27.75 grad=0.863 lr=1.64e-05 6866 tok/s 12.62s/it [13:38:17<31:25:13] mem=34.07GB |
| full_cpt step 3900/12852 ( 30.3%) nll=3.2203 ppl=25.04 grad=0.809 lr=1.64e-05 6867 tok/s 12.62s/it [13:40:13<31:22:43] mem=34.07GB |
| am nll=3.9965 ppl=54.41 bpb=1.4054 |
| ta nll=3.9671 ppl=52.83 bpb=0.5762 |
| kk nll=2.7691 ppl=15.94 bpb=0.9325 |
| tl nll=2.6629 ppl=14.34 bpb=1.2635 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 3900 01:26 (86.4s) |
| full_cpt step 3910/12852 ( 30.4%) nll=3.3329 ppl=28.02 grad=0.898 lr=1.64e-05 6856 tok/s 12.64s/it [13:43:36<31:23:31] mem=34.07GB |
| full_cpt step 3920/12852 ( 30.5%) nll=3.3226 ppl=27.73 grad=0.902 lr=1.63e-05 6858 tok/s 12.64s/it [13:45:32<31:21:02] mem=34.07GB |
| full_cpt step 3930/12852 ( 30.6%) nll=3.3971 ppl=29.88 grad=0.871 lr=1.63e-05 6860 tok/s 12.63s/it [13:47:28<31:18:32] mem=34.07GB |
| full_cpt step 3940/12852 ( 30.7%) nll=3.4773 ppl=32.37 grad=0.895 lr=1.63e-05 6861 tok/s 12.63s/it [13:49:24<31:16:03] mem=34.07GB |
| full_cpt step 3950/12852 ( 30.7%) nll=3.3994 ppl=29.95 grad=0.879 lr=1.63e-05 6863 tok/s 12.63s/it [13:51:20<31:13:33] mem=34.07GB |
| full_cpt step 3960/12852 ( 30.8%) nll=3.2809 ppl=26.60 grad=0.875 lr=1.63e-05 6864 tok/s 12.63s/it [13:53:16<31:11:04] mem=34.07GB |
| full_cpt step 3970/12852 ( 30.9%) nll=3.2901 ppl=26.85 grad=0.867 lr=1.62e-05 6865 tok/s 12.62s/it [13:55:12<31:08:35] mem=34.07GB |
| full_cpt step 3980/12852 ( 31.0%) nll=3.1977 ppl=24.48 grad=0.887 lr=1.62e-05 6867 tok/s 12.62s/it [13:57:08<31:06:07] mem=34.07GB |
| full_cpt step 3990/12852 ( 31.0%) nll=3.2513 ppl=25.82 grad=0.875 lr=1.62e-05 6868 tok/s 12.62s/it [13:59:04<31:03:38] mem=34.07GB |
| full_cpt step 4000/12852 ( 31.1%) nll=3.2619 ppl=26.10 grad=0.883 lr=1.62e-05 6870 tok/s 12.62s/it [14:01:01<31:01:09] mem=34.07GB |
| am nll=3.9948 ppl=54.32 bpb=1.4048 |
| ta nll=3.9647 ppl=52.70 bpb=0.5758 |
| kk nll=2.7678 ppl=15.92 bpb=0.9321 |
| tl nll=2.6626 ppl=14.33 bpb=1.2633 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 4000 01:27 (87.1s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.38s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.38s/it] |
| removing old checkpoint checkpoint-3000 |
| kept last 2 resume checkpoints: checkpoint-3500 checkpoint-4000 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-4000 at step 4000 next_row=512000 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.66s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.66s/it] |
| wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-4000 (weights+expanded tokenizer, no optimizer) |
| full_cpt step 4010/12852 ( 31.2%) nll=3.3249 ppl=27.79 grad=0.926 lr=1.62e-05 6848 tok/s 12.65s/it [14:05:46<31:04:54] mem=34.07GB |
| full_cpt step 4020/12852 ( 31.3%) nll=3.4192 ppl=30.55 grad=1.094 lr=1.62e-05 6849 tok/s 12.65s/it [14:07:42<31:02:24] mem=34.07GB |
| full_cpt step 4030/12852 ( 31.4%) nll=3.2768 ppl=26.49 grad=0.879 lr=1.61e-05 6851 tok/s 12.65s/it [14:09:38<30:59:55] mem=34.07GB |
| full_cpt step 4040/12852 ( 31.4%) nll=3.2381 ppl=25.49 grad=0.859 lr=1.61e-05 6852 tok/s 12.65s/it [14:11:34<30:57:25] mem=34.07GB |
| full_cpt step 4050/12852 ( 31.5%) nll=3.2846 ppl=26.70 grad=0.902 lr=1.61e-05 6853 tok/s 12.64s/it [14:13:30<30:54:56] mem=34.07GB |
| full_cpt step 4060/12852 ( 31.6%) nll=3.4232 ppl=30.67 grad=0.953 lr=1.61e-05 6855 tok/s 12.64s/it [14:15:26<30:52:27] mem=34.07GB |
| full_cpt step 4070/12852 ( 31.7%) nll=3.3701 ppl=29.08 grad=0.848 lr=1.61e-05 6856 tok/s 12.64s/it [14:17:22<30:49:59] mem=34.07GB |
| full_cpt step 4080/12852 ( 31.7%) nll=3.5206 ppl=33.80 grad=0.969 lr=1.60e-05 6858 tok/s 12.64s/it [14:19:18<30:47:30] mem=34.07GB |
| full_cpt step 4090/12852 ( 31.8%) nll=3.3677 ppl=29.01 grad=0.938 lr=1.60e-05 6859 tok/s 12.63s/it [14:21:14<30:45:02] mem=34.07GB |
| full_cpt step 4100/12852 ( 31.9%) nll=3.3089 ppl=27.36 grad=0.898 lr=1.60e-05 6861 tok/s 12.63s/it [14:23:10<30:42:34] mem=34.07GB |
| am nll=3.9919 ppl=54.16 bpb=1.4038 |
| ta nll=3.9636 ppl=52.64 bpb=0.5757 |
| kk nll=2.7659 ppl=15.89 bpb=0.9314 |
| tl nll=2.6623 ppl=14.33 bpb=1.2632 |
| en nll=2.8175 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 4100 01:27 (86.9s) |
| full_cpt step 4110/12852 ( 32.0%) nll=3.3334 ppl=28.03 grad=0.941 lr=1.60e-05 6851 tok/s 12.65s/it [14:26:33<30:43:10] mem=34.07GB |
| full_cpt step 4120/12852 ( 32.1%) nll=3.2013 ppl=24.56 grad=0.895 lr=1.60e-05 6852 tok/s 12.65s/it [14:28:30<30:40:42] mem=34.07GB |
| full_cpt step 4130/12852 ( 32.1%) nll=3.3952 ppl=29.82 grad=0.926 lr=1.60e-05 6854 tok/s 12.65s/it [14:30:26<30:38:14] mem=34.07GB |
| full_cpt step 4140/12852 ( 32.2%) nll=3.3306 ppl=27.96 grad=0.914 lr=1.59e-05 6855 tok/s 12.64s/it [14:32:22<30:35:45] mem=34.07GB |
| full_cpt step 4150/12852 ( 32.3%) nll=3.2908 ppl=26.86 grad=0.836 lr=1.59e-05 6856 tok/s 12.64s/it [14:34:18<30:33:17] mem=34.07GB |
| full_cpt step 4160/12852 ( 32.4%) nll=3.2927 ppl=26.91 grad=0.930 lr=1.59e-05 6857 tok/s 12.64s/it [14:36:14<30:30:49] mem=34.07GB |
| full_cpt step 4170/12852 ( 32.4%) nll=3.1692 ppl=23.79 grad=0.852 lr=1.59e-05 6859 tok/s 12.64s/it [14:38:10<30:28:21] mem=34.07GB |
| full_cpt step 4180/12852 ( 32.5%) nll=3.2696 ppl=26.30 grad=0.930 lr=1.59e-05 6860 tok/s 12.63s/it [14:40:06<30:25:53] mem=34.07GB |
| full_cpt step 4190/12852 ( 32.6%) nll=3.3850 ppl=29.52 grad=0.867 lr=1.58e-05 6861 tok/s 12.63s/it [14:42:02<30:23:26] mem=34.07GB |
| full_cpt step 4200/12852 ( 32.7%) nll=3.2205 ppl=25.04 grad=0.875 lr=1.58e-05 6862 tok/s 12.63s/it [14:43:58<30:20:58] mem=34.07GB |
| am nll=3.9900 ppl=54.05 bpb=1.4031 |
| ta nll=3.9612 ppl=52.52 bpb=0.5753 |
| kk nll=2.7642 ppl=15.87 bpb=0.9308 |
| tl nll=2.6623 ppl=14.33 bpb=1.2632 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 4200 01:27 (86.7s) |
| full_cpt step 4210/12852 ( 32.8%) nll=3.3693 ppl=29.06 grad=0.859 lr=1.58e-05 6852 tok/s 12.65s/it [14:47:21<30:21:29] mem=34.07GB |
| full_cpt step 4220/12852 ( 32.8%) nll=3.3485 ppl=28.46 grad=0.879 lr=1.58e-05 6853 tok/s 12.64s/it [14:49:17<30:19:01] mem=34.07GB |
| full_cpt step 4230/12852 ( 32.9%) nll=3.3073 ppl=27.31 grad=0.828 lr=1.58e-05 6855 tok/s 12.64s/it [14:51:13<30:16:34] mem=34.07GB |
| full_cpt step 4240/12852 ( 33.0%) nll=3.2101 ppl=24.78 grad=0.910 lr=1.57e-05 6856 tok/s 12.64s/it [14:53:09<30:14:07] mem=34.07GB |
| full_cpt step 4250/12852 ( 33.1%) nll=3.4040 ppl=30.08 grad=0.879 lr=1.57e-05 6858 tok/s 12.64s/it [14:55:05<30:11:40] mem=34.07GB |
| full_cpt step 4260/12852 ( 33.1%) nll=3.4427 ppl=31.27 grad=0.867 lr=1.57e-05 6859 tok/s 12.63s/it [14:57:01<30:09:12] mem=34.07GB |
| full_cpt step 4270/12852 ( 33.2%) nll=3.2428 ppl=25.61 grad=0.906 lr=1.57e-05 6860 tok/s 12.63s/it [14:58:57<30:06:45] mem=34.07GB |
| logging stdout/stderr to research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/train.log |
| seed=42 flash SDP TF32 |
| using measured preliminary seq=1024 layout batch=16 accum=8; CPT lock remains seq=2048 |
| train layout batch=16 accum=8 (128 seq/step, 131072 tok/step) checkpointing=off liger=True |
| loading tokenizer research/results/embedding_init/gemma_mapping_byteify |
| full token census over local corpus |
| using full token census from /workspace/ugrip-kd/research/scripts/embedding_initialization/results/corpus_token_counts.json (cached tokenizer /workspace/ugrip-kd/research/results/expansion/tokenizer; current research/results/embedding_init/gemma_mapping_byteify) |
| am census: 476898623 tokens (1959.7 tok/doc, 243349 docs) |
| ta census: 4369273220 tokens (924.0 tok/doc, 4728460 docs) |
| kk census: 4464108618 tokens (1632.8 tok/doc, 2733982 docs) |
| tl census: 306736374 tokens (880.3 tok/doc, 348453 docs) |
| en census: 1542145528 tokens (771.1 tok/doc, 2000000 docs) |
| splits |
| am: full 243349 eval 256 pool 243093 |
| ta: full 4728460 eval 256 pool 4728204 |
| kk: full 2733982 eval 256 pool 2733726 |
| tl: full 348453 eval 256 pool 348197 |
| en: full 2000000 eval 256 pool 1999744 |
| mix weights {'am': 0.0446, 'ta': 0.4089, 'kk': 0.4178, 'tl': 0.0287, 'en': 0.1} (corpus-proportional; no document repeat) |
| lang token budgets {'am': 49803264, 'ta': 456292352, 'kk': 466195456, 'tl': 32032768, 'en': 111591424} |
| packing via HuggingFace datasets.map (batched, num_proc; not cached) |
| am: datasets.map n=35332 batched=1024 num_proc=8 |
|
hf-map am (num_proc=8): 0%| | 0/35332 [00:00<?, ? examples/s]
hf-map am (num_proc=8): 3%|2 | 1024/35332 [00:06<03:32, 161.71 examples/s]
hf-map am (num_proc=8): 6%|5 | 2048/35332 [00:06<01:31, 364.95 examples/s]
hf-map am (num_proc=8): 14%|#4 | 5120/35332 [00:06<00:24, 1244.58 examples/s]
hf-map am (num_proc=8): 20%|## | 7168/35332 [00:06<00:14, 1930.95 examples/s]
hf-map am (num_proc=8): 26%|##6 | 9216/35332 [00:10<00:24, 1053.42 examples/s]
hf-map am (num_proc=8): 32%|###1 | 11264/35332 [00:10<00:16, 1464.56 examples/s]
hf-map am (num_proc=8): 43%|####3 | 15360/35332 [00:11<00:07, 2688.24 examples/s]
hf-map am (num_proc=8): 49%|####9 | 17408/35332 [00:14<00:12, 1474.63 examples/s]
hf-map am (num_proc=8): 55%|#####5 | 19456/35332 [00:14<00:08, 1956.44 examples/s]
hf-map am (num_proc=8): 61%|###### | 21504/35332 [00:14<00:05, 2451.33 examples/s]
hf-map am (num_proc=8): 70%|######9 | 24576/35332 [00:14<00:02, 3663.52 examples/s]
hf-map am (num_proc=8): 75%|#######5 | 26624/35332 [00:17<00:04, 1804.26 examples/s]
hf-map am (num_proc=8): 78%|#######8 | 27648/35332 [00:17<00:03, 2047.79 examples/s]
hf-map am (num_proc=8): 84%|########4 | 29696/35332 [00:17<00:02, 2657.47 examples/s]
hf-map am (num_proc=8): 90%|########9 | 31744/35332 [00:18<00:01, 3105.52 examples/s]
hf-map am (num_proc=8): 94%|#########3| 33088/35332 [00:18<00:00, 3121.10 examples/s]
hf-map am (num_proc=8): 95%|#########5| 33728/35332 [00:18<00:00, 3326.03 examples/s]
hf-map am (num_proc=8): 97%|#########7| 34370/35332 [00:19<00:00, 3293.26 examples/s]
hf-map am (num_proc=8): 99%|#########9| 35012/35332 [00:19<00:00, 2574.26 examples/s]
hf-map am (num_proc=8): 100%|##########| 35332/35332 [00:19<00:00, 1780.96 examples/s] |
| am: datasets.map num_proc=8 done 20s |
| am: 25000 docs 48394736 tokens 25s |
| packed am: 49804449 tokens (1.000 of budget) rows=62355 docs=25725 pad-per-doc (HF map) |
| ta: datasets.map n=667659 batched=1024 num_proc=8 |
|
hf-map ta (num_proc=8): 0%| | 0/667659 [00:00<?, ? examples/s]
hf-map ta (num_proc=8): 0%| | 1024/667659 [00:07<1:25:15, 130.31 examples/s]
hf-map ta (num_proc=8): 0%| | 3072/667659 [00:08<22:40, 488.46 examples/s]
hf-map ta (num_proc=8): 1%| | 4096/667659 [00:08<15:17, 723.38 examples/s]
hf-map ta (num_proc=8): 1%| | 5120/667659 [00:08<10:55, 1009.99 examples/s]
hf-map ta (num_proc=8): 1%|1 | 8192/667659 [00:08<04:57, 2219.47 examples/s]
hf-map ta (num_proc=8): 1%|1 | 9216/667659 [00:12<13:31, 811.72 examples/s]
hf-map ta (num_proc=8): 2%|1 | 10240/667659 [00:13<11:33, 948.13 examples/s]
hf-map ta (num_proc=8): 2%|1 | 12288/667659 [00:13<07:30, 1454.00 examples/s]
hf-map ta (num_proc=8): 2%|1 | 13312/667659 [00:13<06:28, 1686.45 examples/s]
hf-map ta (num_proc=8): 2%|2 | 14336/667659 [00:13<05:12, 2088.40 examples/s]
hf-map ta (num_proc=8): 2%|2 | 16384/667659 [00:14<03:19, 3256.84 examples/s]
hf-map ta (num_proc=8): 3%|2 | 17408/667659 [00:17<11:08, 973.30 examples/s]
hf-map ta (num_proc=8): 3%|2 | 18432/667659 [00:17<09:22, 1155.02 examples/s]
hf-map ta (num_proc=8): 3%|2 | 19456/667659 [00:18<08:18, 1300.04 examples/s]
hf-map ta (num_proc=8): 3%|3 | 20480/667659 [00:18<06:35, 1638.14 examples/s]
hf-map ta (num_proc=8): 3%|3 | 21504/667659 [00:19<05:52, 1834.15 examples/s]
hf-map ta (num_proc=8): 4%|3 | 24576/667659 [00:19<03:08, 3419.41 examples/s]
hf-map ta (num_proc=8): 4%|3 | 25600/667659 [00:22<09:43, 1101.08 examples/s]
hf-map ta (num_proc=8): 4%|3 | 26624/667659 [00:23<09:04, 1177.97 examples/s]
hf-map ta (num_proc=8): 4%|4 | 27648/667659 [00:23<07:42, 1382.66 examples/s]
hf-map ta (num_proc=8): 4%|4 | 28672/667659 [00:24<06:39, 1600.28 examples/s]
hf-map ta (num_proc=8): 4%|4 | 29696/667659 [00:24<05:29, 1935.33 examples/s]
hf-map ta (num_proc=8): 5%|4 | 31744/667659 [00:24<03:28, 3047.60 examples/s]
hf-map ta (num_proc=8): 5%|4 | 32768/667659 [00:24<02:57, 3584.55 examples/s]
hf-map ta (num_proc=8): 5%|5 | 33792/667659 [00:27<10:08, 1041.12 examples/s]
hf-map ta (num_proc=8): 5%|5 | 34816/667659 [00:28<10:02, 1050.70 examples/s]
hf-map ta (num_proc=8): 5%|5 | 35840/667659 [00:28<07:33, 1394.29 examples/s]
hf-map ta (num_proc=8): 6%|5 | 36864/667659 [00:28<06:08, 1713.30 examples/s]
hf-map ta (num_proc=8): 6%|5 | 37888/667659 [00:29<04:49, 2172.10 examples/s]
hf-map ta (num_proc=8): 6%|5 | 38912/667659 [00:29<05:05, 2054.81 examples/s]
hf-map ta (num_proc=8): 6%|6 | 40960/667659 [00:29<03:04, 3392.48 examples/s]
hf-map ta (num_proc=8): 6%|6 | 41984/667659 [00:32<08:52, 1174.66 examples/s]
hf-map ta (num_proc=8): 6%|6 | 43008/667659 [00:33<10:05, 1032.02 examples/s]
hf-map ta (num_proc=8): 7%|6 | 45056/667659 [00:33<06:02, 1718.59 examples/s]
hf-map ta (num_proc=8): 7%|6 | 46080/667659 [00:34<05:22, 1926.19 examples/s]
hf-map ta (num_proc=8): 7%|7 | 47104/667659 [00:34<05:19, 1943.58 examples/s]
hf-map ta (num_proc=8): 7%|7 | 48128/667659 [00:34<04:43, 2186.30 examples/s]
hf-map ta (num_proc=8): 8%|7 | 50176/667659 [00:37<07:21, 1399.58 examples/s]
hf-map ta (num_proc=8): 8%|7 | 51200/667659 [00:38<08:35, 1194.85 examples/s]
hf-map ta (num_proc=8): 8%|7 | 52224/667659 [00:38<06:47, 1509.42 examples/s]
hf-map ta (num_proc=8): 8%|7 | 53248/667659 [00:38<05:22, 1902.84 examples/s]
hf-map ta (num_proc=8): 8%|8 | 54272/667659 [00:38<04:37, 2206.48 examples/s]
hf-map ta (num_proc=8): 8%|8 | 55296/667659 [00:39<04:52, 2092.04 examples/s]
hf-map ta (num_proc=8): 8%|8 | 56320/667659 [00:39<04:04, 2497.15 examples/s]
hf-map ta (num_proc=8): 9%|8 | 58368/667659 [00:41<06:09, 1650.95 examples/s]
hf-map ta (num_proc=8): 9%|8 | 59392/667659 [00:42<07:57, 1274.16 examples/s]
hf-map ta (num_proc=8): 9%|9 | 61440/667659 [00:43<05:24, 1866.88 examples/s]
hf-map ta (num_proc=8): 9%|9 | 62464/667659 [00:43<05:22, 1878.76 examples/s]
hf-map ta (num_proc=8): 10%|9 | 63488/667659 [00:43<04:21, 2306.69 examples/s]
hf-map ta (num_proc=8): 10%|9 | 65536/667659 [00:44<03:10, 3159.93 examples/s]
hf-map ta (num_proc=8): 10%|9 | 66560/667659 [00:45<05:43, 1748.37 examples/s]
hf-map ta (num_proc=8): 10%|# | 67584/667659 [00:46<06:35, 1517.54 examples/s]
hf-map ta (num_proc=8): 10%|# | 68608/667659 [00:46<05:18, 1882.12 examples/s]
hf-map ta (num_proc=8): 10%|# | 69632/667659 [00:47<05:11, 1917.03 examples/s]
hf-map ta (num_proc=8): 11%|# | 70656/667659 [00:47<05:14, 1899.26 examples/s]
hf-map ta (num_proc=8): 11%|# | 71680/667659 [00:47<04:12, 2361.11 examples/s]
hf-map ta (num_proc=8): 11%|# | 72704/667659 [00:48<03:45, 2636.80 examples/s]
hf-map ta (num_proc=8): 11%|#1 | 73728/667659 [00:48<03:45, 2638.38 examples/s]
hf-map ta (num_proc=8): 11%|#1 | 74752/667659 [00:50<06:39, 1485.69 examples/s]
hf-map ta (num_proc=8): 11%|#1 | 75776/667659 [00:50<07:12, 1367.41 examples/s]
hf-map ta (num_proc=8): 12%|#1 | 76800/667659 [00:51<05:34, 1768.38 examples/s]
hf-map ta (num_proc=8): 12%|#1 | 77824/667659 [00:51<05:42, 1723.20 examples/s]
hf-map ta (num_proc=8): 12%|#1 | 78848/667659 [00:51<04:37, 2119.67 examples/s]
hf-map ta (num_proc=8): 12%|#1 | 79872/667659 [00:52<04:16, 2294.58 examples/s]
hf-map ta (num_proc=8): 12%|#2 | 80896/667659 [00:52<04:15, 2295.98 examples/s]
hf-map ta (num_proc=8): 12%|#2 | 81920/667659 [00:53<03:38, 2675.98 examples/s]
hf-map ta (num_proc=8): 12%|#2 | 82944/667659 [00:53<05:18, 1836.10 examples/s]
hf-map ta (num_proc=8): 13%|#2 | 83968/667659 [00:55<07:18, 1331.16 examples/s]
hf-map ta (num_proc=8): 13%|#2 | 86016/667659 [00:55<05:19, 1820.54 examples/s]
hf-map ta (num_proc=8): 13%|#3 | 87040/667659 [00:56<05:02, 1917.91 examples/s]
hf-map ta (num_proc=8): 13%|#3 | 88064/667659 [00:56<04:39, 2071.43 examples/s]
hf-map ta (num_proc=8): 13%|#3 | 89088/667659 [00:57<04:19, 2231.25 examples/s]
hf-map ta (num_proc=8): 13%|#3 | 90112/667659 [00:57<03:55, 2456.83 examples/s]
hf-map ta (num_proc=8): 14%|#3 | 91136/667659 [00:57<03:25, 2810.83 examples/s]
hf-map ta (num_proc=8): 14%|#3 | 92160/667659 [00:58<05:04, 1892.78 examples/s]
hf-map ta (num_proc=8): 14%|#3 | 93184/667659 [00:59<05:07, 1866.94 examples/s]
hf-map ta (num_proc=8): 14%|#4 | 94208/667659 [00:59<05:05, 1878.62 examples/s]
hf-map ta (num_proc=8): 14%|#4 | 95232/667659 [01:00<05:09, 1848.70 examples/s]
hf-map ta (num_proc=8): 14%|#4 | 96256/667659 [01:00<04:14, 2242.27 examples/s]
hf-map ta (num_proc=8): 15%|#4 | 97280/667659 [01:01<04:24, 2158.00 examples/s]
hf-map ta (num_proc=8): 15%|#4 | 98304/667659 [01:01<04:52, 1949.34 examples/s]
hf-map ta (num_proc=8): 15%|#4 | 99328/667659 [01:01<04:04, 2321.59 examples/s]
hf-map ta (num_proc=8): 15%|#5 | 100352/667659 [01:03<06:22, 1484.46 examples/s]
hf-map ta (num_proc=8): 15%|#5 | 101376/667659 [01:03<06:03, 1558.56 examples/s]
hf-map ta (num_proc=8): 15%|#5 | 102400/667659 [01:04<05:16, 1787.01 examples/s]
hf-map ta (num_proc=8): 15%|#5 | 103424/667659 [01:04<05:31, 1701.30 examples/s]
hf-map ta (num_proc=8): 16%|#5 | 104448/667659 [01:05<04:48, 1954.74 examples/s]
hf-map ta (num_proc=8): 16%|#5 | 105472/667659 [01:05<03:51, 2432.96 examples/s]
hf-map ta (num_proc=8): 16%|#5 | 106496/667659 [01:05<03:27, 2704.25 examples/s]
hf-map ta (num_proc=8): 16%|#6 | 107520/667659 [01:06<03:49, 2436.60 examples/s]
hf-map ta (num_proc=8): 16%|#6 | 108544/667659 [01:06<03:56, 2362.46 examples/s]
hf-map ta (num_proc=8): 16%|#6 | 109568/667659 [01:07<04:48, 1935.24 examples/s]
hf-map ta (num_proc=8): 17%|#6 | 110592/667659 [01:07<04:29, 2063.94 examples/s]
hf-map ta (num_proc=8): 17%|#6 | 111616/667659 [01:08<05:32, 1672.12 examples/s]
hf-map ta (num_proc=8): 17%|#6 | 112640/667659 [01:08<04:24, 2100.86 examples/s]
hf-map ta (num_proc=8): 17%|#7 | 114688/667659 [01:09<02:49, 3253.72 examples/s]
hf-map ta (num_proc=8): 17%|#7 | 115712/667659 [01:10<05:15, 1747.97 examples/s]
hf-map ta (num_proc=8): 17%|#7 | 116736/667659 [01:10<04:20, 2116.95 examples/s]
hf-map ta (num_proc=8): 18%|#7 | 117760/667659 [01:11<05:14, 1747.65 examples/s]
hf-map ta (num_proc=8): 18%|#7 | 118784/667659 [01:12<05:05, 1797.06 examples/s]
hf-map ta (num_proc=8): 18%|#7 | 119808/667659 [01:12<05:52, 1556.33 examples/s]
hf-map ta (num_proc=8): 19%|#8 | 123904/667659 [01:14<04:35, 1971.37 examples/s]
hf-map ta (num_proc=8): 19%|#8 | 124928/667659 [01:14<04:09, 2173.59 examples/s]
hf-map ta (num_proc=8): 19%|#8 | 125952/667659 [01:15<05:07, 1760.21 examples/s]
hf-map ta (num_proc=8): 19%|#9 | 126976/667659 [01:16<04:19, 2083.34 examples/s]
hf-map ta (num_proc=8): 19%|#9 | 128000/667659 [01:16<04:29, 2005.32 examples/s]
hf-map ta (num_proc=8): 19%|#9 | 129024/667659 [01:16<03:58, 2259.25 examples/s]
hf-map ta (num_proc=8): 19%|#9 | 130048/667659 [01:17<03:12, 2789.65 examples/s]
hf-map ta (num_proc=8): 20%|#9 | 131072/667659 [01:17<03:40, 2438.52 examples/s]
hf-map ta (num_proc=8): 20%|#9 | 132096/667659 [01:18<05:16, 1691.70 examples/s]
hf-map ta (num_proc=8): 20%|#9 | 133120/667659 [01:19<04:24, 2024.72 examples/s]
hf-map ta (num_proc=8): 20%|## | 134144/667659 [01:19<05:29, 1618.99 examples/s]
hf-map ta (num_proc=8): 20%|## | 135168/667659 [01:20<04:21, 2035.84 examples/s]
hf-map ta (num_proc=8): 20%|## | 136192/667659 [01:20<04:04, 2173.98 examples/s]
hf-map ta (num_proc=8): 21%|## | 137216/667659 [01:20<03:24, 2589.71 examples/s]
hf-map ta (num_proc=8): 21%|## | 139264/667659 [01:21<03:10, 2775.84 examples/s]
hf-map ta (num_proc=8): 21%|##1 | 140288/667659 [01:22<04:43, 1858.56 examples/s]
hf-map ta (num_proc=8): 21%|##1 | 141312/667659 [01:22<03:50, 2283.71 examples/s]
hf-map ta (num_proc=8): 21%|##1 | 142336/667659 [01:23<04:00, 2188.63 examples/s]
hf-map ta (num_proc=8): 21%|##1 | 143360/667659 [01:23<04:21, 2004.83 examples/s]
hf-map ta (num_proc=8): 22%|##1 | 144384/667659 [01:24<04:37, 1883.41 examples/s]
hf-map ta (num_proc=8): 22%|##1 | 145408/667659 [01:24<04:31, 1921.81 examples/s]
hf-map ta (num_proc=8): 22%|##2 | 147456/667659 [01:25<02:55, 2971.64 examples/s]
hf-map ta (num_proc=8): 22%|##2 | 148480/667659 [01:26<04:55, 1754.43 examples/s]
hf-map ta (num_proc=8): 22%|##2 | 149504/667659 [01:26<04:03, 2130.31 examples/s]
hf-map ta (num_proc=8): 23%|##2 | 150528/667659 [01:27<04:17, 2007.41 examples/s]
hf-map ta (num_proc=8): 23%|##2 | 151552/667659 [01:27<04:39, 1844.78 examples/s]
hf-map ta (num_proc=8): 23%|##2 | 152576/667659 [01:28<04:48, 1784.56 examples/s]
hf-map ta (num_proc=8): 23%|##3 | 153600/667659 [01:29<04:32, 1887.24 examples/s]
hf-map ta (num_proc=8): 23%|##3 | 155648/667659 [01:29<03:23, 2511.35 examples/s]
hf-map ta (num_proc=8): 23%|##3 | 156672/667659 [01:30<04:36, 1844.79 examples/s]
hf-map ta (num_proc=8): 24%|##3 | 157696/667659 [01:31<04:21, 1946.43 examples/s]
hf-map ta (num_proc=8): 24%|##3 | 159744/667659 [01:31<03:30, 2407.99 examples/s]
hf-map ta (num_proc=8): 24%|##4 | 160768/667659 [01:32<04:05, 2068.30 examples/s]
hf-map ta (num_proc=8): 24%|##4 | 161792/667659 [01:33<04:36, 1828.23 examples/s]
hf-map ta (num_proc=8): 24%|##4 | 162816/667659 [01:33<03:36, 2327.17 examples/s]
hf-map ta (num_proc=8): 25%|##4 | 163840/667659 [01:33<03:08, 2675.04 examples/s]
hf-map ta (num_proc=8): 25%|##4 | 164864/667659 [01:34<03:55, 2131.61 examples/s]
hf-map ta (num_proc=8): 25%|##4 | 165888/667659 [01:34<04:01, 2075.61 examples/s]
hf-map ta (num_proc=8): 25%|##5 | 167936/667659 [01:35<03:57, 2100.09 examples/s]
hf-map ta (num_proc=8): 25%|##5 | 168960/667659 [01:36<04:07, 2013.21 examples/s]
hf-map ta (num_proc=8): 25%|##5 | 169984/667659 [01:36<03:37, 2292.63 examples/s]
hf-map ta (num_proc=8): 26%|##5 | 171008/667659 [01:36<03:21, 2460.87 examples/s]
hf-map ta (num_proc=8): 26%|##5 | 172032/667659 [01:37<03:16, 2517.79 examples/s]
hf-map ta (num_proc=8): 26%|##5 | 173056/667659 [01:37<03:45, 2196.83 examples/s]
hf-map ta (num_proc=8): 26%|##6 | 174080/667659 [01:38<03:57, 2082.11 examples/s]
hf-map ta (num_proc=8): 26%|##6 | 175104/667659 [01:38<03:15, 2524.50 examples/s]
hf-map ta (num_proc=8): 26%|##6 | 176128/667659 [01:39<04:51, 1684.86 examples/s]
hf-map ta (num_proc=8): 27%|##6 | 177152/667659 [01:39<04:13, 1934.76 examples/s]
hf-map ta (num_proc=8): 27%|##6 | 179200/667659 [01:40<02:51, 2850.58 examples/s]
hf-map ta (num_proc=8): 27%|##6 | 180224/667659 [01:40<03:07, 2603.00 examples/s]
hf-map ta (num_proc=8): 27%|##7 | 181248/667659 [01:41<03:14, 2507.09 examples/s]
hf-map ta (num_proc=8): 27%|##7 | 182272/667659 [01:41<03:44, 2161.69 examples/s]
hf-map ta (num_proc=8): 27%|##7 | 183296/667659 [01:42<03:24, 2369.91 examples/s]
hf-map ta (num_proc=8): 28%|##7 | 184320/667659 [01:43<04:11, 1919.63 examples/s]
hf-map ta (num_proc=8): 28%|##7 | 185344/667659 [01:43<03:13, 2491.32 examples/s]
hf-map ta (num_proc=8): 28%|##7 | 186368/667659 [01:43<02:54, 2765.23 examples/s]
hf-map ta (num_proc=8): 28%|##8 | 187392/667659 [01:43<02:25, 3301.03 examples/s]
hf-map ta (num_proc=8): 28%|##8 | 188416/667659 [01:44<03:39, 2186.60 examples/s]
hf-map ta (num_proc=8): 28%|##8 | 189440/667659 [01:44<03:20, 2383.47 examples/s]
hf-map ta (num_proc=8): 29%|##8 | 190464/667659 [01:45<03:27, 2295.26 examples/s]
hf-map ta (num_proc=8): 29%|##8 | 191488/667659 [01:45<03:37, 2186.84 examples/s]
hf-map ta (num_proc=8): 29%|##8 | 192512/667659 [01:46<04:19, 1829.53 examples/s]
hf-map ta (num_proc=8): 29%|##8 | 193536/667659 [01:46<03:56, 2008.28 examples/s]
hf-map ta (num_proc=8): 29%|##9 | 195584/667659 [01:47<02:20, 3350.73 examples/s]
hf-map ta (num_proc=8): 29%|##9 | 196608/667659 [01:48<04:28, 1756.19 examples/s]
hf-map ta (num_proc=8): 30%|##9 | 198656/667659 [01:49<03:37, 2160.19 examples/s]
hf-map ta (num_proc=8): 30%|##9 | 199680/667659 [01:49<03:34, 2183.33 examples/s]
hf-map ta (num_proc=8): 30%|### | 200704/667659 [01:50<03:44, 2079.79 examples/s]
hf-map ta (num_proc=8): 30%|### | 201728/667659 [01:50<03:15, 2381.74 examples/s]
hf-map ta (num_proc=8): 30%|### | 202752/667659 [01:50<02:52, 2692.11 examples/s]
hf-map ta (num_proc=8): 31%|### | 203776/667659 [01:50<02:29, 3100.19 examples/s]
hf-map ta (num_proc=8): 31%|### | 204800/667659 [01:51<04:04, 1889.91 examples/s]
hf-map ta (num_proc=8): 31%|### | 205824/667659 [01:52<03:14, 2371.77 examples/s]
hf-map ta (num_proc=8): 31%|### | 206848/667659 [01:52<03:21, 2286.64 examples/s]
hf-map ta (num_proc=8): 31%|###1 | 207872/667659 [01:53<03:32, 2158.92 examples/s]
hf-map ta (num_proc=8): 31%|###1 | 208896/667659 [01:53<03:42, 2066.07 examples/s]
hf-map ta (num_proc=8): 31%|###1 | 209920/667659 [01:53<03:03, 2497.41 examples/s]
hf-map ta (num_proc=8): 32%|###1 | 210944/667659 [01:54<02:51, 2662.54 examples/s]
hf-map ta (num_proc=8): 32%|###1 | 211968/667659 [01:54<02:43, 2790.83 examples/s]
hf-map ta (num_proc=8): 32%|###1 | 212992/667659 [01:55<03:28, 2177.51 examples/s]
hf-map ta (num_proc=8): 32%|###2 | 214016/667659 [01:55<03:20, 2268.14 examples/s]
hf-map ta (num_proc=8): 32%|###2 | 215040/667659 [01:55<02:37, 2870.04 examples/s]
hf-map ta (num_proc=8): 32%|###2 | 216064/667659 [01:56<03:10, 2371.85 examples/s]
hf-map ta (num_proc=8): 33%|###2 | 217088/667659 [01:56<03:18, 2269.34 examples/s]
hf-map ta (num_proc=8): 33%|###2 | 219136/667659 [01:57<02:27, 3045.01 examples/s]
hf-map ta (num_proc=8): 33%|###2 | 220160/667659 [01:58<03:38, 2050.72 examples/s]
hf-map ta (num_proc=8): 33%|###3 | 221184/667659 [01:58<03:30, 2124.81 examples/s]
hf-map ta (num_proc=8): 33%|###3 | 222208/667659 [01:59<03:20, 2222.62 examples/s]
hf-map ta (num_proc=8): 33%|###3 | 223232/667659 [01:59<03:48, 1944.59 examples/s]
hf-map ta (num_proc=8): 34%|###3 | 224256/667659 [02:00<03:24, 2169.62 examples/s]
hf-map ta (num_proc=8): 34%|###3 | 225280/667659 [02:00<02:58, 2480.16 examples/s]
hf-map ta (num_proc=8): 34%|###3 | 226304/667659 [02:00<02:54, 2526.67 examples/s]
hf-map ta (num_proc=8): 34%|###4 | 228352/667659 [02:02<03:58, 1845.35 examples/s]
hf-map ta (num_proc=8): 35%|###4 | 230400/667659 [02:02<02:49, 2583.93 examples/s]
hf-map ta (num_proc=8): 35%|###4 | 231424/667659 [02:03<03:37, 2003.96 examples/s]
hf-map ta (num_proc=8): 35%|###4 | 232448/667659 [02:03<03:13, 2249.34 examples/s]
hf-map ta (num_proc=8): 35%|###4 | 233472/667659 [02:04<03:00, 2399.84 examples/s]
hf-map ta (num_proc=8): 35%|###5 | 234496/667659 [02:04<02:37, 2750.30 examples/s]
hf-map ta (num_proc=8): 35%|###5 | 235520/667659 [02:04<02:48, 2570.75 examples/s]
hf-map ta (num_proc=8): 35%|###5 | 236544/667659 [02:05<04:08, 1732.18 examples/s]
hf-map ta (num_proc=8): 36%|###5 | 237568/667659 [02:05<03:14, 2212.43 examples/s]
hf-map ta (num_proc=8): 36%|###5 | 238592/667659 [02:06<02:51, 2508.52 examples/s]
hf-map ta (num_proc=8): 36%|###5 | 239616/667659 [02:06<03:06, 2292.79 examples/s]
hf-map ta (num_proc=8): 36%|###6 | 240640/667659 [02:07<03:06, 2287.61 examples/s]
hf-map ta (num_proc=8): 36%|###6 | 241664/667659 [02:07<02:36, 2726.80 examples/s]
hf-map ta (num_proc=8): 36%|###6 | 242688/667659 [02:07<02:11, 3220.82 examples/s]
hf-map ta (num_proc=8): 37%|###6 | 243712/667659 [02:08<02:52, 2456.29 examples/s]
hf-map ta (num_proc=8): 37%|###6 | 244736/667659 [02:09<04:04, 1730.28 examples/s]
hf-map ta (num_proc=8): 37%|###6 | 245760/667659 [02:09<03:07, 2247.34 examples/s]
hf-map ta (num_proc=8): 37%|###6 | 246784/667659 [02:09<02:57, 2372.03 examples/s]
hf-map ta (num_proc=8): 37%|###7 | 247808/667659 [02:09<02:27, 2842.10 examples/s]
hf-map ta (num_proc=8): 37%|###7 | 248832/667659 [02:10<02:54, 2406.94 examples/s]
hf-map ta (num_proc=8): 38%|###7 | 251904/667659 [02:11<02:44, 2527.24 examples/s]
hf-map ta (num_proc=8): 38%|###7 | 252928/667659 [02:12<03:33, 1938.33 examples/s]
hf-map ta (num_proc=8): 38%|###8 | 253952/667659 [02:13<03:09, 2180.56 examples/s]
hf-map ta (num_proc=8): 38%|###8 | 254976/667659 [02:13<02:32, 2706.45 examples/s]
hf-map ta (num_proc=8): 38%|###8 | 257024/667659 [02:14<02:47, 2445.72 examples/s]
hf-map ta (num_proc=8): 39%|###8 | 259072/667659 [02:14<02:01, 3371.23 examples/s]
hf-map ta (num_proc=8): 39%|###8 | 260096/667659 [02:15<02:58, 2281.03 examples/s]
hf-map ta (num_proc=8): 39%|###9 | 261120/667659 [02:16<03:40, 1845.88 examples/s]
hf-map ta (num_proc=8): 39%|###9 | 263168/667659 [02:16<02:39, 2528.10 examples/s]
hf-map ta (num_proc=8): 40%|###9 | 264192/667659 [02:16<02:14, 2993.73 examples/s]
hf-map ta (num_proc=8): 40%|###9 | 265216/667659 [02:17<02:56, 2277.86 examples/s]
hf-map ta (num_proc=8): 40%|###9 | 266240/667659 [02:17<02:37, 2547.62 examples/s]
hf-map ta (num_proc=8): 40%|#### | 268288/667659 [02:18<02:41, 2468.43 examples/s]
hf-map ta (num_proc=8): 40%|#### | 269312/667659 [02:19<03:31, 1882.10 examples/s]
hf-map ta (num_proc=8): 40%|#### | 270336/667659 [02:19<03:00, 2201.86 examples/s]
hf-map ta (num_proc=8): 41%|#### | 272384/667659 [02:20<02:14, 2932.69 examples/s]
hf-map ta (num_proc=8): 41%|#### | 273408/667659 [02:20<02:36, 2526.42 examples/s]
hf-map ta (num_proc=8): 41%|####1 | 274432/667659 [02:21<02:36, 2506.41 examples/s]
hf-map ta (num_proc=8): 41%|####1 | 276480/667659 [02:22<03:00, 2172.74 examples/s]
hf-map ta (num_proc=8): 42%|####1 | 277504/667659 [02:22<03:07, 2080.29 examples/s]
hf-map ta (num_proc=8): 42%|####1 | 278528/667659 [02:23<02:39, 2446.77 examples/s]
hf-map ta (num_proc=8): 42%|####2 | 280576/667659 [02:23<01:49, 3521.53 examples/s]
hf-map ta (num_proc=8): 42%|####2 | 281600/667659 [02:23<01:57, 3276.49 examples/s]
hf-map ta (num_proc=8): 42%|####2 | 282624/667659 [02:24<02:17, 2801.81 examples/s]
hf-map ta (num_proc=8): 43%|####2 | 284672/667659 [02:25<02:59, 2131.15 examples/s]
hf-map ta (num_proc=8): 43%|####2 | 285696/667659 [02:25<02:52, 2208.49 examples/s]
hf-map ta (num_proc=8): 43%|####2 | 286720/667659 [02:26<02:29, 2543.31 examples/s]
hf-map ta (num_proc=8): 43%|####3 | 289792/667659 [02:26<01:24, 4495.72 examples/s]
hf-map ta (num_proc=8): 44%|####3 | 290816/667659 [02:26<01:41, 3708.07 examples/s]
hf-map ta (num_proc=8): 44%|####3 | 291840/667659 [02:27<01:43, 3614.03 examples/s]
hf-map ta (num_proc=8): 44%|####3 | 292864/667659 [02:28<03:43, 1680.51 examples/s]
hf-map ta (num_proc=8): 44%|####4 | 293888/667659 [02:29<03:24, 1830.62 examples/s]
hf-map ta (num_proc=8): 44%|####4 | 295936/667659 [02:29<02:12, 2809.77 examples/s]
hf-map ta (num_proc=8): 44%|####4 | 296960/667659 [02:29<02:03, 2997.70 examples/s]
hf-map ta (num_proc=8): 45%|####4 | 297984/667659 [02:30<02:18, 2673.66 examples/s]
hf-map ta (num_proc=8): 45%|####4 | 300032/667659 [02:30<01:41, 3624.87 examples/s]
hf-map ta (num_proc=8): 45%|####5 | 301056/667659 [02:32<03:34, 1712.11 examples/s]
hf-map ta (num_proc=8): 45%|####5 | 302080/667659 [02:32<02:56, 2075.60 examples/s]
hf-map ta (num_proc=8): 45%|####5 | 303104/667659 [02:32<02:35, 2345.07 examples/s]
hf-map ta (num_proc=8): 46%|####5 | 304128/667659 [02:32<02:14, 2706.80 examples/s]
hf-map ta (num_proc=8): 46%|####5 | 305152/667659 [02:32<02:03, 2946.73 examples/s]
hf-map ta (num_proc=8): 46%|####5 | 306176/667659 [02:33<01:41, 3566.46 examples/s]
hf-map ta (num_proc=8): 46%|####6 | 307200/667659 [02:33<01:50, 3268.57 examples/s]
hf-map ta (num_proc=8): 46%|####6 | 308224/667659 [02:33<01:29, 4033.78 examples/s]
hf-map ta (num_proc=8): 46%|####6 | 309248/667659 [02:34<03:04, 1945.15 examples/s]
hf-map ta (num_proc=8): 46%|####6 | 310272/667659 [02:35<02:37, 2268.42 examples/s]
hf-map ta (num_proc=8): 47%|####6 | 311296/667659 [02:35<02:06, 2814.65 examples/s]
hf-map ta (num_proc=8): 47%|####6 | 312320/667659 [02:35<01:55, 3085.54 examples/s]
hf-map ta (num_proc=8): 47%|####6 | 313344/667659 [02:35<01:51, 3167.09 examples/s]
hf-map ta (num_proc=8): 47%|####7 | 314368/667659 [02:35<01:41, 3465.39 examples/s]
hf-map ta (num_proc=8): 47%|####7 | 315392/667659 [02:36<01:33, 3786.29 examples/s]
hf-map ta (num_proc=8): 48%|####7 | 317440/667659 [02:37<02:46, 2105.41 examples/s]
hf-map ta (num_proc=8): 48%|####7 | 319488/667659 [02:38<02:02, 2850.14 examples/s]
hf-map ta (num_proc=8): 48%|####8 | 320512/667659 [02:38<01:44, 3335.50 examples/s]
hf-map ta (num_proc=8): 48%|####8 | 321536/667659 [02:38<01:29, 3849.60 examples/s]
hf-map ta (num_proc=8): 48%|####8 | 322560/667659 [02:38<01:49, 3152.81 examples/s]
hf-map ta (num_proc=8): 49%|####8 | 324608/667659 [02:39<01:22, 4148.53 examples/s]
hf-map ta (num_proc=8): 49%|####8 | 325632/667659 [02:40<02:35, 2200.70 examples/s]
hf-map ta (num_proc=8): 49%|####8 | 326656/667659 [02:40<02:22, 2397.80 examples/s]
hf-map ta (num_proc=8): 49%|####9 | 328704/667659 [02:40<01:30, 3747.31 examples/s]
hf-map ta (num_proc=8): 49%|####9 | 329728/667659 [02:40<01:22, 4117.37 examples/s]
hf-map ta (num_proc=8): 50%|####9 | 330752/667659 [02:41<01:50, 3049.90 examples/s]
hf-map ta (num_proc=8): 50%|####9 | 332800/667659 [02:42<01:59, 2810.69 examples/s]
hf-map ta (num_proc=8): 50%|####9 | 333824/667659 [02:42<02:11, 2540.05 examples/s]
hf-map ta (num_proc=8): 50%|##### | 334848/667659 [02:43<02:05, 2643.38 examples/s]
hf-map ta (num_proc=8): 50%|##### | 336896/667659 [02:43<01:26, 3824.75 examples/s]
hf-map ta (num_proc=8): 51%|##### | 337920/667659 [02:43<01:14, 4443.37 examples/s]
hf-map ta (num_proc=8): 51%|##### | 338944/667659 [02:43<01:40, 3263.89 examples/s]
hf-map ta (num_proc=8): 51%|##### | 339968/667659 [02:44<01:31, 3590.91 examples/s]
hf-map ta (num_proc=8): 51%|#####1 | 340992/667659 [02:44<02:16, 2387.72 examples/s]
hf-map ta (num_proc=8): 51%|#####1 | 342016/667659 [02:45<02:12, 2456.08 examples/s]
hf-map ta (num_proc=8): 51%|#####1 | 343040/667659 [02:45<02:10, 2487.44 examples/s]
hf-map ta (num_proc=8): 52%|#####1 | 344064/667659 [02:46<01:53, 2855.21 examples/s]
hf-map ta (num_proc=8): 52%|#####1 | 345088/667659 [02:46<01:30, 3565.34 examples/s]
hf-map ta (num_proc=8): 52%|#####1 | 346112/667659 [02:46<01:18, 4077.94 examples/s]
hf-map ta (num_proc=8): 52%|#####1 | 347136/667659 [02:46<01:42, 3116.75 examples/s]
hf-map ta (num_proc=8): 52%|#####2 | 348160/667659 [02:47<01:51, 2860.48 examples/s]
hf-map ta (num_proc=8): 52%|#####2 | 349184/667659 [02:47<01:59, 2670.01 examples/s]
hf-map ta (num_proc=8): 52%|#####2 | 350208/667659 [02:48<01:57, 2700.34 examples/s]
hf-map ta (num_proc=8): 53%|#####2 | 351232/667659 [02:48<02:01, 2595.77 examples/s]
hf-map ta (num_proc=8): 53%|#####2 | 353280/667659 [02:48<01:22, 3811.40 examples/s]
hf-map ta (num_proc=8): 53%|#####3 | 354304/667659 [02:49<01:25, 3648.10 examples/s]
hf-map ta (num_proc=8): 53%|#####3 | 355328/667659 [02:49<01:19, 3947.77 examples/s]
hf-map ta (num_proc=8): 53%|#####3 | 356352/667659 [02:49<01:29, 3486.85 examples/s]
hf-map ta (num_proc=8): 54%|#####3 | 357376/667659 [02:50<02:00, 2584.98 examples/s]
hf-map ta (num_proc=8): 54%|#####3 | 359424/667659 [02:51<01:58, 2604.83 examples/s]
hf-map ta (num_proc=8): 54%|#####4 | 361472/667659 [02:51<01:38, 3111.62 examples/s]
hf-map ta (num_proc=8): 54%|#####4 | 362496/667659 [02:51<01:26, 3522.82 examples/s]
hf-map ta (num_proc=8): 55%|#####4 | 364544/667659 [02:52<01:30, 3345.82 examples/s]
hf-map ta (num_proc=8): 55%|#####4 | 365568/667659 [02:52<01:49, 2754.61 examples/s]
hf-map ta (num_proc=8): 55%|#####4 | 366592/667659 [02:53<01:49, 2740.54 examples/s]
hf-map ta (num_proc=8): 55%|#####5 | 367616/667659 [02:53<01:50, 2706.74 examples/s]
hf-map ta (num_proc=8): 55%|#####5 | 368640/667659 [02:53<01:31, 3283.68 examples/s]
hf-map ta (num_proc=8): 55%|#####5 | 369664/667659 [02:54<01:32, 3236.52 examples/s]
hf-map ta (num_proc=8): 56%|#####5 | 370688/667659 [02:54<01:24, 3513.73 examples/s]
hf-map ta (num_proc=8): 56%|#####5 | 372736/667659 [02:55<01:27, 3356.90 examples/s]
hf-map ta (num_proc=8): 56%|#####5 | 373760/667659 [02:55<01:46, 2757.42 examples/s]
hf-map ta (num_proc=8): 56%|#####6 | 374784/667659 [02:56<02:05, 2330.34 examples/s]
hf-map ta (num_proc=8): 57%|#####6 | 377856/667659 [02:56<01:18, 3714.39 examples/s]
hf-map ta (num_proc=8): 57%|#####6 | 378880/667659 [02:57<01:31, 3150.75 examples/s]
hf-map ta (num_proc=8): 57%|#####7 | 380928/667659 [02:57<01:32, 3089.29 examples/s]
hf-map ta (num_proc=8): 57%|#####7 | 381952/667659 [02:58<02:04, 2300.51 examples/s]
hf-map ta (num_proc=8): 57%|#####7 | 382976/667659 [02:59<02:10, 2182.04 examples/s]
hf-map ta (num_proc=8): 58%|#####7 | 385024/667659 [02:59<01:31, 3078.57 examples/s]
hf-map ta (num_proc=8): 58%|#####7 | 386048/667659 [02:59<01:33, 3013.44 examples/s]
hf-map ta (num_proc=8): 58%|#####7 | 387072/667659 [03:00<01:40, 2786.29 examples/s]
hf-map ta (num_proc=8): 58%|#####8 | 388096/667659 [03:00<01:24, 3312.24 examples/s]
hf-map ta (num_proc=8): 58%|#####8 | 389120/667659 [03:00<01:35, 2911.84 examples/s]
hf-map ta (num_proc=8): 58%|#####8 | 390144/667659 [03:01<02:11, 2114.67 examples/s]
hf-map ta (num_proc=8): 59%|#####8 | 391168/667659 [03:02<02:19, 1982.19 examples/s]
hf-map ta (num_proc=8): 59%|#####8 | 393216/667659 [03:02<01:40, 2720.64 examples/s]
hf-map ta (num_proc=8): 59%|#####9 | 394240/667659 [03:03<01:34, 2884.05 examples/s]
hf-map ta (num_proc=8): 59%|#####9 | 395264/667659 [03:03<01:41, 2675.13 examples/s]
hf-map ta (num_proc=8): 59%|#####9 | 396288/667659 [03:04<01:53, 2391.87 examples/s]
hf-map ta (num_proc=8): 60%|#####9 | 397312/667659 [03:04<02:01, 2228.27 examples/s]
hf-map ta (num_proc=8): 60%|#####9 | 398336/667659 [03:04<01:43, 2606.83 examples/s]
hf-map ta (num_proc=8): 60%|#####9 | 399360/667659 [03:05<01:39, 2699.96 examples/s]
hf-map ta (num_proc=8): 60%|#####9 | 400384/667659 [03:05<01:25, 3133.80 examples/s]
hf-map ta (num_proc=8): 60%|###### | 401408/667659 [03:06<01:52, 2373.03 examples/s]
hf-map ta (num_proc=8): 60%|###### | 402432/667659 [03:06<01:41, 2620.33 examples/s]
hf-map ta (num_proc=8): 60%|###### | 403456/667659 [03:06<01:18, 3362.79 examples/s]
hf-map ta (num_proc=8): 61%|###### | 404480/667659 [03:07<02:11, 1997.25 examples/s]
hf-map ta (num_proc=8): 61%|###### | 405504/667659 [03:07<02:03, 2129.85 examples/s]
hf-map ta (num_proc=8): 61%|###### | 406528/667659 [03:08<01:42, 2540.10 examples/s]
hf-map ta (num_proc=8): 61%|######1 | 407552/667659 [03:08<01:39, 2625.98 examples/s]
hf-map ta (num_proc=8): 61%|######1 | 409600/667659 [03:09<01:31, 2806.16 examples/s]
hf-map ta (num_proc=8): 62%|######1 | 410624/667659 [03:09<01:26, 2964.56 examples/s]
hf-map ta (num_proc=8): 62%|######1 | 411648/667659 [03:09<01:19, 3221.54 examples/s]
hf-map ta (num_proc=8): 62%|######1 | 412672/667659 [03:10<01:58, 2144.18 examples/s]
hf-map ta (num_proc=8): 62%|######1 | 413696/667659 [03:10<01:36, 2630.31 examples/s]
hf-map ta (num_proc=8): 62%|######2 | 414720/667659 [03:11<01:40, 2509.10 examples/s]
hf-map ta (num_proc=8): 62%|######2 | 415744/667659 [03:11<01:30, 2787.77 examples/s]
hf-map ta (num_proc=8): 63%|######2 | 417792/667659 [03:12<01:44, 2391.61 examples/s]
hf-map ta (num_proc=8): 63%|######2 | 418816/667659 [03:12<01:36, 2581.27 examples/s]
hf-map ta (num_proc=8): 63%|######2 | 419840/667659 [03:12<01:25, 2881.68 examples/s]
hf-map ta (num_proc=8): 63%|######3 | 420864/667659 [03:13<01:29, 2762.93 examples/s]
hf-map ta (num_proc=8): 63%|######3 | 422912/667659 [03:14<01:28, 2759.79 examples/s]
hf-map ta (num_proc=8): 63%|######3 | 423936/667659 [03:14<01:13, 3322.06 examples/s]
hf-map ta (num_proc=8): 64%|######3 | 424960/667659 [03:14<01:06, 3636.02 examples/s]
hf-map ta (num_proc=8): 64%|######3 | 425984/667659 [03:15<01:34, 2548.29 examples/s]
hf-map ta (num_proc=8): 64%|######3 | 427008/667659 [03:15<01:20, 2998.49 examples/s]
hf-map ta (num_proc=8): 64%|######4 | 428032/667659 [03:15<01:06, 3619.38 examples/s]
hf-map ta (num_proc=8): 64%|######4 | 429056/667659 [03:16<01:33, 2539.28 examples/s]
hf-map ta (num_proc=8): 65%|######4 | 431104/667659 [03:17<01:40, 2358.64 examples/s]
hf-map ta (num_proc=8): 65%|######4 | 433152/667659 [03:17<01:17, 3010.73 examples/s]
hf-map ta (num_proc=8): 65%|######5 | 434176/667659 [03:18<01:29, 2617.25 examples/s]
hf-map ta (num_proc=8): 65%|######5 | 435200/667659 [03:18<01:35, 2424.81 examples/s]
hf-map ta (num_proc=8): 65%|######5 | 437248/667659 [03:19<01:20, 2867.62 examples/s]
hf-map ta (num_proc=8): 66%|######5 | 438272/667659 [03:19<01:26, 2651.31 examples/s]
hf-map ta (num_proc=8): 66%|######5 | 439296/667659 [03:20<01:31, 2483.15 examples/s]
hf-map ta (num_proc=8): 66%|######6 | 441344/667659 [03:20<01:17, 2919.60 examples/s]
hf-map ta (num_proc=8): 66%|######6 | 442368/667659 [03:20<01:05, 3432.16 examples/s]
hf-map ta (num_proc=8): 66%|######6 | 443392/667659 [03:21<01:29, 2512.33 examples/s]
hf-map ta (num_proc=8): 67%|######6 | 444416/667659 [03:22<01:58, 1889.32 examples/s]
hf-map ta (num_proc=8): 67%|######6 | 445440/667659 [03:22<01:45, 2098.82 examples/s]
hf-map ta (num_proc=8): 67%|######6 | 446464/667659 [03:22<01:27, 2527.15 examples/s]
hf-map ta (num_proc=8): 67%|######7 | 447488/667659 [03:23<01:49, 2011.32 examples/s]
hf-map ta (num_proc=8): 67%|######7 | 449536/667659 [03:23<01:06, 3294.93 examples/s]
hf-map ta (num_proc=8): 67%|######7 | 450560/667659 [03:24<01:18, 2757.77 examples/s]
hf-map ta (num_proc=8): 68%|######7 | 451584/667659 [03:24<01:15, 2847.13 examples/s]
hf-map ta (num_proc=8): 68%|######7 | 452608/667659 [03:25<01:42, 2104.53 examples/s]
hf-map ta (num_proc=8): 68%|######7 | 453632/667659 [03:25<01:26, 2488.53 examples/s]
hf-map ta (num_proc=8): 68%|######8 | 455680/667659 [03:26<01:21, 2613.35 examples/s]
hf-map ta (num_proc=8): 68%|######8 | 456704/667659 [03:26<01:17, 2723.05 examples/s]
hf-map ta (num_proc=8): 69%|######8 | 457728/667659 [03:27<01:12, 2888.15 examples/s]
hf-map ta (num_proc=8): 69%|######8 | 459776/667659 [03:28<01:42, 2032.91 examples/s]
hf-map ta (num_proc=8): 69%|######9 | 460800/667659 [03:28<01:24, 2460.77 examples/s]
hf-map ta (num_proc=8): 69%|######9 | 461824/667659 [03:28<01:09, 2982.19 examples/s]
hf-map ta (num_proc=8): 69%|######9 | 462848/667659 [03:29<01:45, 1946.50 examples/s]
hf-map ta (num_proc=8): 70%|######9 | 464896/667659 [03:30<01:20, 2516.31 examples/s]
hf-map ta (num_proc=8): 70%|######9 | 465920/667659 [03:30<01:15, 2676.01 examples/s]
hf-map ta (num_proc=8): 70%|######9 | 466944/667659 [03:30<01:04, 3131.20 examples/s]
hf-map ta (num_proc=8): 70%|####### | 467968/667659 [03:31<01:02, 3174.59 examples/s]
hf-map ta (num_proc=8): 70%|####### | 468992/667659 [03:31<00:59, 3363.88 examples/s]
hf-map ta (num_proc=8): 70%|####### | 470016/667659 [03:31<01:00, 3265.28 examples/s]
hf-map ta (num_proc=8): 71%|####### | 471040/667659 [03:32<01:26, 2284.40 examples/s]
hf-map ta (num_proc=8): 71%|####### | 472064/667659 [03:32<01:13, 2673.75 examples/s]
hf-map ta (num_proc=8): 71%|####### | 473088/667659 [03:33<01:18, 2481.18 examples/s]
hf-map ta (num_proc=8): 71%|#######1 | 475136/667659 [03:33<00:53, 3615.16 examples/s]
hf-map ta (num_proc=8): 71%|#######1 | 476160/667659 [03:33<00:45, 4193.61 examples/s]
hf-map ta (num_proc=8): 71%|#######1 | 477184/667659 [03:33<00:45, 4150.20 examples/s]
hf-map ta (num_proc=8): 72%|#######1 | 478208/667659 [03:34<00:51, 3653.82 examples/s]
hf-map ta (num_proc=8): 72%|#######1 | 479232/667659 [03:34<01:15, 2485.10 examples/s]
hf-map ta (num_proc=8): 72%|#######1 | 480256/667659 [03:35<01:24, 2216.67 examples/s]
hf-map ta (num_proc=8): 72%|#######2 | 481280/667659 [03:35<01:19, 2335.98 examples/s]
hf-map ta (num_proc=8): 72%|#######2 | 482304/667659 [03:36<01:01, 2995.82 examples/s]
hf-map ta (num_proc=8): 72%|#######2 | 483328/667659 [03:36<00:58, 3169.37 examples/s]
hf-map ta (num_proc=8): 73%|#######2 | 484352/667659 [03:36<00:47, 3826.42 examples/s]
hf-map ta (num_proc=8): 73%|#######2 | 485376/667659 [03:36<00:55, 3272.10 examples/s]
hf-map ta (num_proc=8): 73%|#######3 | 487424/667659 [03:38<01:14, 2417.46 examples/s]
hf-map ta (num_proc=8): 73%|#######3 | 488448/667659 [03:38<01:12, 2463.11 examples/s]
hf-map ta (num_proc=8): 73%|#######3 | 489472/667659 [03:38<01:05, 2737.80 examples/s]
hf-map ta (num_proc=8): 73%|#######3 | 490496/667659 [03:39<01:04, 2759.11 examples/s]
hf-map ta (num_proc=8): 74%|#######3 | 492544/667659 [03:39<00:39, 4414.79 examples/s]
hf-map ta (num_proc=8): 74%|#######3 | 493568/667659 [03:39<00:46, 3720.44 examples/s]
hf-map ta (num_proc=8): 74%|#######4 | 494592/667659 [03:39<00:48, 3592.68 examples/s]
hf-map ta (num_proc=8): 74%|#######4 | 495616/667659 [03:40<01:08, 2507.93 examples/s]
hf-map ta (num_proc=8): 74%|#######4 | 496640/667659 [03:41<01:15, 2252.84 examples/s]
hf-map ta (num_proc=8): 75%|#######4 | 498688/667659 [03:41<00:58, 2912.84 examples/s]
hf-map ta (num_proc=8): 75%|#######4 | 499712/667659 [03:41<00:47, 3508.40 examples/s]
hf-map ta (num_proc=8): 75%|#######4 | 500736/667659 [03:42<00:51, 3266.34 examples/s]
hf-map ta (num_proc=8): 75%|#######5 | 501760/667659 [03:42<00:44, 3697.84 examples/s]
hf-map ta (num_proc=8): 75%|#######5 | 502784/667659 [03:42<00:39, 4226.77 examples/s]
hf-map ta (num_proc=8): 75%|#######5 | 503808/667659 [03:43<01:08, 2402.26 examples/s]
hf-map ta (num_proc=8): 76%|#######5 | 504832/667659 [03:44<01:20, 2020.04 examples/s]
hf-map ta (num_proc=8): 76%|#######5 | 506880/667659 [03:44<00:54, 2961.47 examples/s]
hf-map ta (num_proc=8): 76%|#######6 | 507904/667659 [03:44<00:53, 2979.61 examples/s]
hf-map ta (num_proc=8): 76%|#######6 | 508928/667659 [03:44<00:47, 3321.67 examples/s]
hf-map ta (num_proc=8): 76%|#######6 | 509952/667659 [03:45<01:11, 2208.74 examples/s]
hf-map ta (num_proc=8): 77%|#######6 | 510976/667659 [03:46<01:04, 2412.95 examples/s]
hf-map ta (num_proc=8): 77%|#######6 | 513024/667659 [03:46<00:56, 2717.53 examples/s]
hf-map ta (num_proc=8): 77%|#######6 | 514048/667659 [03:47<00:53, 2866.12 examples/s]
hf-map ta (num_proc=8): 77%|#######7 | 516096/667659 [03:47<00:43, 3477.76 examples/s]
hf-map ta (num_proc=8): 78%|#######7 | 518144/667659 [03:48<00:54, 2763.59 examples/s]
hf-map ta (num_proc=8): 78%|#######7 | 520192/667659 [03:48<00:45, 3258.33 examples/s]
hf-map ta (num_proc=8): 78%|#######8 | 521216/667659 [03:49<00:55, 2641.36 examples/s]
hf-map ta (num_proc=8): 78%|#######8 | 523264/667659 [03:49<00:38, 3743.41 examples/s]
hf-map ta (num_proc=8): 79%|#######8 | 524288/667659 [03:49<00:39, 3633.54 examples/s]
hf-map ta (num_proc=8): 79%|#######8 | 525312/667659 [03:50<00:41, 3415.69 examples/s]
hf-map ta (num_proc=8): 79%|#######8 | 526336/667659 [03:50<00:41, 3386.65 examples/s]
hf-map ta (num_proc=8): 79%|#######8 | 527360/667659 [03:51<00:43, 3218.43 examples/s]
hf-map ta (num_proc=8): 79%|#######9 | 528384/667659 [03:51<00:54, 2547.95 examples/s]
hf-map ta (num_proc=8): 79%|#######9 | 529408/667659 [03:52<00:56, 2451.37 examples/s]
hf-map ta (num_proc=8): 79%|#######9 | 530432/667659 [03:52<00:45, 2988.90 examples/s]
hf-map ta (num_proc=8): 80%|#######9 | 531456/667659 [03:52<00:37, 3639.06 examples/s]
hf-map ta (num_proc=8): 80%|#######9 | 532480/667659 [03:52<00:40, 3336.09 examples/s]
hf-map ta (num_proc=8): 80%|######## | 534528/667659 [03:53<00:34, 3872.36 examples/s]
hf-map ta (num_proc=8): 80%|######## | 535552/667659 [03:53<00:31, 4202.15 examples/s]
hf-map ta (num_proc=8): 80%|######## | 536576/667659 [03:54<00:47, 2759.45 examples/s]
hf-map ta (num_proc=8): 81%|######## | 537600/667659 [03:54<00:57, 2256.36 examples/s]
hf-map ta (num_proc=8): 81%|######## | 540672/667659 [03:55<00:41, 3049.86 examples/s]
hf-map ta (num_proc=8): 81%|########1 | 541696/667659 [03:55<00:37, 3387.06 examples/s]
hf-map ta (num_proc=8): 81%|########1 | 543744/667659 [03:55<00:27, 4528.33 examples/s]
hf-map ta (num_proc=8): 82%|########1 | 544768/667659 [03:56<00:40, 3014.87 examples/s]
hf-map ta (num_proc=8): 82%|########1 | 545792/667659 [03:57<00:45, 2663.39 examples/s]
hf-map ta (num_proc=8): 82%|########1 | 546816/667659 [03:57<00:38, 3134.23 examples/s]
hf-map ta (num_proc=8): 82%|########2 | 547840/667659 [03:57<00:33, 3620.57 examples/s]
hf-map ta (num_proc=8): 82%|########2 | 548864/667659 [03:58<00:42, 2764.43 examples/s]
hf-map ta (num_proc=8): 83%|########2 | 550912/667659 [03:58<00:27, 4170.44 examples/s]
hf-map ta (num_proc=8): 83%|########2 | 551936/667659 [03:59<00:44, 2615.44 examples/s]
hf-map ta (num_proc=8): 83%|########2 | 553984/667659 [03:59<00:42, 2669.16 examples/s]
hf-map ta (num_proc=8): 83%|########3 | 556032/667659 [04:00<00:33, 3342.32 examples/s]
hf-map ta (num_proc=8): 83%|########3 | 557056/667659 [04:00<00:39, 2825.53 examples/s]
hf-map ta (num_proc=8): 84%|########3 | 558080/667659 [04:00<00:34, 3207.86 examples/s]
hf-map ta (num_proc=8): 84%|########3 | 559104/667659 [04:01<00:35, 3050.07 examples/s]
hf-map ta (num_proc=8): 84%|########3 | 560128/667659 [04:01<00:42, 2559.03 examples/s]
hf-map ta (num_proc=8): 84%|########4 | 562176/667659 [04:02<00:36, 2906.97 examples/s]
hf-map ta (num_proc=8): 85%|########4 | 564224/667659 [04:03<00:32, 3150.49 examples/s]
hf-map ta (num_proc=8): 85%|########4 | 566272/667659 [04:03<00:24, 4064.72 examples/s]
hf-map ta (num_proc=8): 85%|########4 | 567296/667659 [04:03<00:25, 3980.69 examples/s]
hf-map ta (num_proc=8): 85%|########5 | 568320/667659 [04:04<00:32, 3086.29 examples/s]
hf-map ta (num_proc=8): 85%|########5 | 570368/667659 [04:04<00:29, 3347.56 examples/s]
hf-map ta (num_proc=8): 86%|########5 | 571392/667659 [04:04<00:28, 3385.59 examples/s]
hf-map ta (num_proc=8): 86%|########5 | 572416/667659 [04:05<00:29, 3271.67 examples/s]
hf-map ta (num_proc=8): 86%|########5 | 573440/667659 [04:05<00:25, 3714.25 examples/s]
hf-map ta (num_proc=8): 86%|########6 | 574464/667659 [04:05<00:23, 3889.37 examples/s]
hf-map ta (num_proc=8): 86%|########6 | 575488/667659 [04:05<00:21, 4312.07 examples/s]
hf-map ta (num_proc=8): 86%|########6 | 576512/667659 [04:06<00:29, 3051.84 examples/s]
hf-map ta (num_proc=8): 87%|########6 | 578560/667659 [04:06<00:22, 3973.35 examples/s]
hf-map ta (num_proc=8): 87%|########6 | 579584/667659 [04:07<00:23, 3824.06 examples/s]
hf-map ta (num_proc=8): 87%|########6 | 580608/667659 [04:07<00:29, 2975.60 examples/s]
hf-map ta (num_proc=8): 87%|########7 | 582656/667659 [04:07<00:21, 3878.75 examples/s]
hf-map ta (num_proc=8): 87%|########7 | 583680/667659 [04:08<00:21, 3913.47 examples/s]
hf-map ta (num_proc=8): 88%|########7 | 584704/667659 [04:08<00:24, 3318.70 examples/s]
hf-map ta (num_proc=8): 88%|########7 | 585728/667659 [04:08<00:21, 3891.89 examples/s]
hf-map ta (num_proc=8): 88%|########7 | 586752/667659 [04:08<00:19, 4104.98 examples/s]
hf-map ta (num_proc=8): 88%|########8 | 587776/667659 [04:09<00:20, 3944.62 examples/s]
hf-map ta (num_proc=8): 88%|########8 | 588800/667659 [04:09<00:25, 3148.11 examples/s]
hf-map ta (num_proc=8): 88%|########8 | 590848/667659 [04:10<00:20, 3816.53 examples/s]
hf-map ta (num_proc=8): 89%|########8 | 591872/667659 [04:10<00:23, 3180.65 examples/s]
hf-map ta (num_proc=8): 89%|########8 | 592896/667659 [04:10<00:22, 3292.95 examples/s]
hf-map ta (num_proc=8): 89%|########9 | 594944/667659 [04:11<00:16, 4401.50 examples/s]
hf-map ta (num_proc=8): 89%|########9 | 595968/667659 [04:11<00:20, 3474.47 examples/s]
hf-map ta (num_proc=8): 89%|########9 | 596992/667659 [04:11<00:19, 3557.91 examples/s]
hf-map ta (num_proc=8): 90%|########9 | 598016/667659 [04:12<00:22, 3068.53 examples/s]
hf-map ta (num_proc=8): 90%|########9 | 599040/667659 [04:12<00:20, 3391.05 examples/s]
hf-map ta (num_proc=8): 90%|########9 | 600064/667659 [04:13<00:29, 2305.07 examples/s]
hf-map ta (num_proc=8): 90%|######### | 601088/667659 [04:13<00:24, 2713.78 examples/s]
hf-map ta (num_proc=8): 90%|######### | 604160/667659 [04:14<00:16, 3927.87 examples/s]
hf-map ta (num_proc=8): 91%|######### | 605184/667659 [04:14<00:16, 3802.29 examples/s]
hf-map ta (num_proc=8): 91%|######### | 606208/667659 [04:14<00:18, 3297.43 examples/s]
hf-map ta (num_proc=8): 91%|######### | 607232/667659 [04:15<00:16, 3616.12 examples/s]
hf-map ta (num_proc=8): 91%|#########1| 608256/667659 [04:15<00:21, 2794.31 examples/s]
hf-map ta (num_proc=8): 91%|#########1| 610304/667659 [04:15<00:14, 4052.22 examples/s]
hf-map ta (num_proc=8): 92%|#########1| 612352/667659 [04:16<00:13, 4010.24 examples/s]
hf-map ta (num_proc=8): 92%|#########2| 614400/667659 [04:17<00:15, 3372.52 examples/s]
hf-map ta (num_proc=8): 92%|#########2| 616448/667659 [04:18<00:17, 2967.15 examples/s]
hf-map ta (num_proc=8): 92%|#########2| 617472/667659 [04:18<00:15, 3193.91 examples/s]
hf-map ta (num_proc=8): 93%|#########2| 620544/667659 [04:18<00:09, 4883.89 examples/s]
hf-map ta (num_proc=8): 93%|#########3| 621568/667659 [04:19<00:13, 3525.04 examples/s]
hf-map ta (num_proc=8): 93%|#########3| 622592/667659 [04:19<00:11, 3772.58 examples/s]
hf-map ta (num_proc=8): 93%|#########3| 623616/667659 [04:19<00:12, 3499.71 examples/s]
hf-map ta (num_proc=8): 94%|#########3| 624640/667659 [04:20<00:16, 2642.26 examples/s]
hf-map ta (num_proc=8): 94%|#########3| 625664/667659 [04:20<00:13, 3147.48 examples/s]
hf-map ta (num_proc=8): 94%|#########4| 628736/667659 [04:20<00:06, 5725.47 examples/s]
hf-map ta (num_proc=8): 94%|#########4| 629760/667659 [04:21<00:11, 3359.33 examples/s]
hf-map ta (num_proc=8): 94%|#########4| 630784/667659 [04:22<00:12, 2882.51 examples/s]
hf-map ta (num_proc=8): 95%|#########4| 631808/667659 [04:22<00:11, 3184.68 examples/s]
hf-map ta (num_proc=8): 95%|#########4| 632832/667659 [04:22<00:14, 2461.94 examples/s]
hf-map ta (num_proc=8): 95%|#########4| 633856/667659 [04:23<00:12, 2662.90 examples/s]
hf-map ta (num_proc=8): 96%|#########5| 637952/667659 [04:24<00:08, 3365.61 examples/s]
hf-map ta (num_proc=8): 96%|#########5| 638976/667659 [04:24<00:08, 3303.83 examples/s]
hf-map ta (num_proc=8): 96%|#########5| 640000/667659 [04:24<00:07, 3801.32 examples/s]
hf-map ta (num_proc=8): 96%|#########6| 641024/667659 [04:25<00:10, 2580.22 examples/s]
hf-map ta (num_proc=8): 96%|#########6| 642048/667659 [04:25<00:08, 3002.66 examples/s]
hf-map ta (num_proc=8): 96%|#########6| 644096/667659 [04:25<00:05, 4426.72 examples/s]
hf-map ta (num_proc=8): 97%|#########6| 645120/667659 [04:26<00:05, 4231.28 examples/s]
hf-map ta (num_proc=8): 97%|#########6| 646144/667659 [04:26<00:07, 2923.57 examples/s]
hf-map ta (num_proc=8): 97%|#########7| 647681/667659 [04:27<00:05, 3543.78 examples/s]
hf-map ta (num_proc=8): 97%|#########7| 649218/667659 [04:27<00:03, 4729.36 examples/s]
hf-map ta (num_proc=8): 97%|#########7| 650242/667659 [04:27<00:05, 3173.39 examples/s]
hf-map ta (num_proc=8): 98%|#########7| 651780/667659 [04:28<00:04, 3484.06 examples/s]
hf-map ta (num_proc=8): 98%|#########7| 652804/667659 [04:28<00:04, 3131.52 examples/s]
hf-map ta (num_proc=8): 98%|#########7| 653828/667659 [04:29<00:05, 2471.18 examples/s]
hf-map ta (num_proc=8): 98%|#########8| 655876/667659 [04:30<00:05, 2042.68 examples/s]
hf-map ta (num_proc=8): 98%|#########8| 657413/667659 [04:30<00:03, 2594.04 examples/s]
hf-map ta (num_proc=8): 99%|#########8| 658437/667659 [04:31<00:03, 2557.36 examples/s]
hf-map ta (num_proc=8): 99%|#########8| 659461/667659 [04:31<00:03, 2285.86 examples/s]
hf-map ta (num_proc=8): 99%|#########8| 660485/667659 [04:32<00:04, 1702.28 examples/s]
hf-map ta (num_proc=8): 99%|#########9| 662533/667659 [04:33<00:02, 2161.76 examples/s]
hf-map ta (num_proc=8): 99%|#########9| 663047/667659 [04:34<00:02, 1755.95 examples/s]
hf-map ta (num_proc=8): 99%|#########9| 664071/667659 [04:34<00:02, 1738.68 examples/s]
hf-map ta (num_proc=8): 100%|#########9| 665095/667659 [04:35<00:01, 1818.25 examples/s]
hf-map ta (num_proc=8): 100%|#########9| 665609/667659 [04:36<00:01, 1305.53 examples/s]
hf-map ta (num_proc=8): 100%|#########9| 667146/667659 [04:36<00:00, 1881.48 examples/s]
hf-map ta (num_proc=8): 100%|##########| 667659/667659 [04:37<00:00, 1258.97 examples/s]
hf-map ta (num_proc=8): 100%|##########| 667659/667659 [04:38<00:00, 2399.81 examples/s] |
| ta: datasets.map num_proc=8 done 278s |
| ta: 25000 docs 23325458 tokens 283s |
| ta: 50000 docs 46092495 tokens 285s |
| ta: 75000 docs 69234775 tokens 287s |
| ta: 100000 docs 92178764 tokens 289s |
| ta: 125000 docs 114953253 tokens 291s |
| ta: 150000 docs 138281352 tokens 294s |
| ta: 175000 docs 161404187 tokens 296s |
| ta: 200000 docs 184541026 tokens 299s |
| ta: 225000 docs 207872624 tokens 301s |
| ta: 250000 docs 230922341 tokens 303s |
| ta: 275000 docs 254309793 tokens 305s |
| ta: 300000 docs 277730139 tokens 307s |
| ta: 325000 docs 301150397 tokens 309s |
| ta: 350000 docs 324210916 tokens 311s |
| ta: 375000 docs 346690917 tokens 313s |
| ta: 400000 docs 369202603 tokens 315s |
| ta: 425000 docs 392444860 tokens 317s |
| ta: 450000 docs 416252337 tokens 319s |
| ta: 475000 docs 439421148 tokens 321s |
| packed ta: 456292825 tokens (1.000 of budget) rows=728619 docs=493028 pad-per-doc (HF map) |
| kk: datasets.map n=386470 batched=1024 num_proc=8 |
|
hf-map kk (num_proc=8): 0%| | 0/386470 [00:00<?, ? examples/s]
hf-map kk (num_proc=8): 0%| | 1024/386470 [00:09<58:52, 109.11 examples/s]
hf-map kk (num_proc=8): 1%| | 2048/386470 [00:09<24:55, 257.07 examples/s]
hf-map kk (num_proc=8): 1%| | 3072/386470 [00:09<13:53, 460.23 examples/s]
hf-map kk (num_proc=8): 1%|1 | 4096/386470 [00:09<08:37, 738.35 examples/s]
hf-map kk (num_proc=8): 2%|1 | 6144/386470 [00:09<04:13, 1501.61 examples/s]
hf-map kk (num_proc=8): 2%|2 | 8192/386470 [00:10<02:39, 2372.76 examples/s]
hf-map kk (num_proc=8): 3%|2 | 10240/386470 [00:14<07:05, 884.35 examples/s]
hf-map kk (num_proc=8): 3%|2 | 11264/386470 [00:15<06:06, 1023.22 examples/s]
hf-map kk (num_proc=8): 3%|3 | 13312/386470 [00:15<03:55, 1586.09 examples/s]
hf-map kk (num_proc=8): 4%|3 | 15360/386470 [00:15<02:46, 2222.78 examples/s]
hf-map kk (num_proc=8): 4%|4 | 16384/386470 [00:15<02:21, 2617.52 examples/s]
hf-map kk (num_proc=8): 5%|4 | 17408/386470 [00:19<06:14, 986.31 examples/s]
hf-map kk (num_proc=8): 5%|4 | 18432/386470 [00:19<05:13, 1172.90 examples/s]
hf-map kk (num_proc=8): 5%|5 | 19456/386470 [00:19<04:50, 1262.59 examples/s]
hf-map kk (num_proc=8): 5%|5 | 20480/386470 [00:20<03:49, 1596.45 examples/s]
hf-map kk (num_proc=8): 6%|5 | 22528/386470 [00:20<02:19, 2617.19 examples/s]
hf-map kk (num_proc=8): 6%|6 | 23552/386470 [00:20<01:55, 3149.20 examples/s]
hf-map kk (num_proc=8): 6%|6 | 24576/386470 [00:20<02:16, 2654.03 examples/s]
hf-map kk (num_proc=8): 7%|6 | 25600/386470 [00:23<05:17, 1135.09 examples/s]
hf-map kk (num_proc=8): 7%|6 | 26624/386470 [00:23<04:47, 1252.95 examples/s]
hf-map kk (num_proc=8): 7%|7 | 27648/386470 [00:24<03:43, 1606.15 examples/s]
hf-map kk (num_proc=8): 7%|7 | 28672/386470 [00:24<03:38, 1636.84 examples/s]
hf-map kk (num_proc=8): 8%|8 | 31744/386470 [00:24<01:46, 3319.22 examples/s]
hf-map kk (num_proc=8): 8%|8 | 32768/386470 [00:25<02:14, 2623.91 examples/s]
hf-map kk (num_proc=8): 9%|8 | 33792/386470 [00:27<04:32, 1293.03 examples/s]
hf-map kk (num_proc=8): 9%|9 | 34816/386470 [00:28<04:40, 1252.73 examples/s]
hf-map kk (num_proc=8): 10%|9 | 36864/386470 [00:29<03:51, 1509.45 examples/s]
hf-map kk (num_proc=8): 10%|9 | 37888/386470 [00:29<03:14, 1789.20 examples/s]
hf-map kk (num_proc=8): 10%|# | 39936/386470 [00:29<02:05, 2755.68 examples/s]
hf-map kk (num_proc=8): 11%|# | 40960/386470 [00:30<02:29, 2317.85 examples/s]
hf-map kk (num_proc=8): 11%|# | 41984/386470 [00:32<04:24, 1301.03 examples/s]
hf-map kk (num_proc=8): 11%|#1 | 43008/386470 [00:33<04:38, 1235.10 examples/s]
hf-map kk (num_proc=8): 11%|#1 | 44032/386470 [00:33<04:00, 1422.90 examples/s]
hf-map kk (num_proc=8): 12%|#1 | 45056/386470 [00:34<04:02, 1407.00 examples/s]
hf-map kk (num_proc=8): 12%|#2 | 48128/386470 [00:34<02:03, 2741.80 examples/s]
hf-map kk (num_proc=8): 13%|#2 | 49152/386470 [00:35<02:03, 2739.93 examples/s]
hf-map kk (num_proc=8): 13%|#2 | 50176/386470 [00:37<04:15, 1315.02 examples/s]
hf-map kk (num_proc=8): 13%|#3 | 51200/386470 [00:37<03:53, 1434.38 examples/s]
hf-map kk (num_proc=8): 14%|#3 | 52224/386470 [00:38<03:54, 1422.81 examples/s]
hf-map kk (num_proc=8): 14%|#3 | 53248/386470 [00:38<03:06, 1781.98 examples/s]
hf-map kk (num_proc=8): 14%|#4 | 54272/386470 [00:38<02:30, 2202.27 examples/s]
hf-map kk (num_proc=8): 14%|#4 | 55296/386470 [00:39<01:58, 2801.37 examples/s]
hf-map kk (num_proc=8): 15%|#5 | 58368/386470 [00:41<03:18, 1650.80 examples/s]
hf-map kk (num_proc=8): 15%|#5 | 59392/386470 [00:41<03:03, 1784.32 examples/s]
hf-map kk (num_proc=8): 16%|#5 | 60416/386470 [00:42<02:58, 1827.34 examples/s]
hf-map kk (num_proc=8): 16%|#5 | 61440/386470 [00:42<02:38, 2048.37 examples/s]
hf-map kk (num_proc=8): 16%|#6 | 63488/386470 [00:42<01:45, 3068.16 examples/s]
hf-map kk (num_proc=8): 17%|#7 | 66560/386470 [00:45<03:10, 1676.78 examples/s]
hf-map kk (num_proc=8): 17%|#7 | 67584/386470 [00:45<02:48, 1896.82 examples/s]
hf-map kk (num_proc=8): 18%|#7 | 68608/386470 [00:46<02:42, 1952.81 examples/s]
hf-map kk (num_proc=8): 18%|#8 | 69632/386470 [00:47<02:57, 1786.81 examples/s]
hf-map kk (num_proc=8): 18%|#8 | 70656/386470 [00:47<02:24, 2191.52 examples/s]
hf-map kk (num_proc=8): 19%|#8 | 71680/386470 [00:47<01:54, 2740.61 examples/s]
hf-map kk (num_proc=8): 19%|#8 | 72704/386470 [00:47<01:33, 3361.10 examples/s]
hf-map kk (num_proc=8): 19%|#9 | 73728/386470 [00:48<01:56, 2674.92 examples/s]
hf-map kk (num_proc=8): 19%|#9 | 74752/386470 [00:50<04:07, 1258.36 examples/s]
hf-map kk (num_proc=8): 20%|#9 | 75776/386470 [00:50<03:29, 1480.42 examples/s]
hf-map kk (num_proc=8): 20%|#9 | 76800/386470 [00:51<04:07, 1251.26 examples/s]
hf-map kk (num_proc=8): 20%|## | 77824/386470 [00:51<03:09, 1628.33 examples/s]
hf-map kk (num_proc=8): 21%|## | 79872/386470 [00:52<02:03, 2479.80 examples/s]
hf-map kk (num_proc=8): 21%|## | 80896/386470 [00:52<01:40, 3035.51 examples/s]
hf-map kk (num_proc=8): 21%|##1 | 81920/386470 [00:52<02:13, 2273.38 examples/s]
hf-map kk (num_proc=8): 21%|##1 | 82944/386470 [00:54<03:41, 1369.74 examples/s]
hf-map kk (num_proc=8): 22%|##1 | 83968/386470 [00:54<03:04, 1642.25 examples/s]
hf-map kk (num_proc=8): 22%|##1 | 84992/386470 [00:56<04:18, 1167.97 examples/s]
hf-map kk (num_proc=8): 22%|##2 | 86016/386470 [00:56<03:11, 1565.17 examples/s]
hf-map kk (num_proc=8): 23%|##2 | 88064/386470 [00:56<02:07, 2340.19 examples/s]
hf-map kk (num_proc=8): 23%|##3 | 89088/386470 [00:56<01:49, 2710.42 examples/s]
hf-map kk (num_proc=8): 23%|##3 | 90112/386470 [00:57<01:57, 2531.87 examples/s]
hf-map kk (num_proc=8): 24%|##3 | 91136/386470 [00:58<03:23, 1452.17 examples/s]
hf-map kk (num_proc=8): 24%|##3 | 92160/386470 [00:59<03:16, 1500.29 examples/s]
hf-map kk (num_proc=8): 24%|##4 | 93184/386470 [01:00<03:45, 1299.89 examples/s]
hf-map kk (num_proc=8): 25%|##4 | 95232/386470 [01:00<02:10, 2233.47 examples/s]
hf-map kk (num_proc=8): 25%|##4 | 96256/386470 [01:00<01:49, 2643.89 examples/s]
hf-map kk (num_proc=8): 25%|##5 | 97280/386470 [01:01<02:06, 2279.20 examples/s]
hf-map kk (num_proc=8): 25%|##5 | 98304/386470 [01:01<01:54, 2519.03 examples/s]
hf-map kk (num_proc=8): 26%|##5 | 99328/386470 [01:03<03:20, 1429.44 examples/s]
hf-map kk (num_proc=8): 26%|##5 | 100352/386470 [01:03<02:41, 1768.57 examples/s]
hf-map kk (num_proc=8): 26%|##6 | 101376/386470 [01:04<03:28, 1366.41 examples/s]
hf-map kk (num_proc=8): 26%|##6 | 102400/386470 [01:04<02:42, 1749.83 examples/s]
hf-map kk (num_proc=8): 27%|##6 | 103424/386470 [01:05<02:05, 2252.20 examples/s]
hf-map kk (num_proc=8): 27%|##7 | 104448/386470 [01:05<02:01, 2323.63 examples/s]
hf-map kk (num_proc=8): 27%|##7 | 105472/386470 [01:05<01:50, 2545.62 examples/s]
hf-map kk (num_proc=8): 28%|##7 | 106496/386470 [01:06<02:54, 1605.13 examples/s]
hf-map kk (num_proc=8): 28%|##7 | 107520/386470 [01:07<02:53, 1605.30 examples/s]
hf-map kk (num_proc=8): 28%|##8 | 108544/386470 [01:08<02:40, 1734.81 examples/s]
hf-map kk (num_proc=8): 28%|##8 | 109568/386470 [01:09<03:38, 1264.76 examples/s]
hf-map kk (num_proc=8): 29%|##8 | 111616/386470 [01:09<02:16, 2007.83 examples/s]
hf-map kk (num_proc=8): 29%|##9 | 112640/386470 [01:10<02:28, 1849.51 examples/s]
hf-map kk (num_proc=8): 29%|##9 | 113664/386470 [01:10<01:57, 2318.74 examples/s]
hf-map kk (num_proc=8): 30%|##9 | 114688/386470 [01:11<02:05, 2172.54 examples/s]
hf-map kk (num_proc=8): 30%|##9 | 115712/386470 [01:11<01:59, 2256.66 examples/s]
hf-map kk (num_proc=8): 30%|### | 116736/386470 [01:11<01:47, 2505.84 examples/s]
hf-map kk (num_proc=8): 30%|### | 117760/386470 [01:12<02:41, 1659.22 examples/s]
hf-map kk (num_proc=8): 31%|### | 118784/386470 [01:13<02:10, 2044.96 examples/s]
hf-map kk (num_proc=8): 31%|###1 | 119808/386470 [01:13<01:44, 2556.93 examples/s]
hf-map kk (num_proc=8): 31%|###1 | 120832/386470 [01:13<01:45, 2528.11 examples/s]
hf-map kk (num_proc=8): 32%|###1 | 121856/386470 [01:13<01:33, 2821.64 examples/s]
hf-map kk (num_proc=8): 32%|###1 | 122880/386470 [01:14<01:31, 2879.98 examples/s]
hf-map kk (num_proc=8): 32%|###2 | 123904/386470 [01:14<01:22, 3165.58 examples/s]
hf-map kk (num_proc=8): 32%|###2 | 124928/386470 [01:15<01:39, 2641.60 examples/s]
hf-map kk (num_proc=8): 33%|###2 | 125952/386470 [01:16<02:34, 1681.92 examples/s]
hf-map kk (num_proc=8): 33%|###3 | 128000/386470 [01:17<02:09, 2000.00 examples/s]
hf-map kk (num_proc=8): 33%|###3 | 129024/386470 [01:17<02:00, 2138.02 examples/s]
hf-map kk (num_proc=8): 34%|###3 | 131072/386470 [01:18<01:47, 2382.19 examples/s]
hf-map kk (num_proc=8): 34%|###4 | 133120/386470 [01:18<01:42, 2482.40 examples/s]
hf-map kk (num_proc=8): 35%|###4 | 134144/386470 [01:19<02:14, 1872.54 examples/s]
hf-map kk (num_proc=8): 35%|###5 | 136192/386470 [01:21<02:37, 1584.37 examples/s]
hf-map kk (num_proc=8): 36%|###5 | 137216/386470 [01:21<02:10, 1905.17 examples/s]
hf-map kk (num_proc=8): 36%|###6 | 139264/386470 [01:22<01:43, 2398.21 examples/s]
hf-map kk (num_proc=8): 37%|###6 | 141312/386470 [01:22<01:23, 2933.49 examples/s]
hf-map kk (num_proc=8): 37%|###6 | 142336/386470 [01:24<02:30, 1623.72 examples/s]
hf-map kk (num_proc=8): 37%|###7 | 143360/386470 [01:24<02:13, 1824.64 examples/s]
hf-map kk (num_proc=8): 37%|###7 | 144384/386470 [01:25<02:28, 1635.26 examples/s]
hf-map kk (num_proc=8): 38%|###7 | 146432/386470 [01:25<01:33, 2560.74 examples/s]
hf-map kk (num_proc=8): 38%|###8 | 147456/386470 [01:26<01:39, 2408.24 examples/s]
hf-map kk (num_proc=8): 39%|###8 | 149504/386470 [01:26<01:14, 3193.36 examples/s]
hf-map kk (num_proc=8): 39%|###8 | 150528/386470 [01:28<02:46, 1419.27 examples/s]
hf-map kk (num_proc=8): 39%|###9 | 151552/386470 [01:28<02:16, 1717.29 examples/s]
hf-map kk (num_proc=8): 39%|###9 | 152576/386470 [01:29<02:11, 1776.69 examples/s]
hf-map kk (num_proc=8): 40%|###9 | 153600/386470 [01:29<01:50, 2109.57 examples/s]
hf-map kk (num_proc=8): 40%|#### | 155648/386470 [01:29<01:17, 2997.48 examples/s]
hf-map kk (num_proc=8): 41%|#### | 156672/386470 [01:30<01:08, 3347.04 examples/s]
hf-map kk (num_proc=8): 41%|#### | 157696/386470 [01:30<01:05, 3518.54 examples/s]
hf-map kk (num_proc=8): 41%|####1 | 158720/386470 [01:32<02:28, 1537.87 examples/s]
hf-map kk (num_proc=8): 41%|####1 | 159744/386470 [01:32<02:19, 1621.54 examples/s]
hf-map kk (num_proc=8): 42%|####1 | 160768/386470 [01:33<02:13, 1684.41 examples/s]
hf-map kk (num_proc=8): 42%|####1 | 161792/386470 [01:33<01:47, 2086.00 examples/s]
hf-map kk (num_proc=8): 42%|####2 | 162816/386470 [01:33<01:23, 2674.55 examples/s]
hf-map kk (num_proc=8): 43%|####2 | 164864/386470 [01:33<00:59, 3720.38 examples/s]
hf-map kk (num_proc=8): 43%|####2 | 165888/386470 [01:33<00:56, 3911.86 examples/s]
hf-map kk (num_proc=8): 43%|####3 | 166912/386470 [01:36<02:38, 1389.49 examples/s]
hf-map kk (num_proc=8): 43%|####3 | 167936/386470 [01:36<02:31, 1438.10 examples/s]
hf-map kk (num_proc=8): 44%|####3 | 168960/386470 [01:36<02:08, 1693.12 examples/s]
hf-map kk (num_proc=8): 44%|####3 | 169984/386470 [01:37<02:08, 1679.21 examples/s]
hf-map kk (num_proc=8): 45%|####4 | 172032/386470 [01:37<01:25, 2500.79 examples/s]
hf-map kk (num_proc=8): 45%|####5 | 174080/386470 [01:38<01:01, 3457.88 examples/s]
hf-map kk (num_proc=8): 45%|####5 | 175104/386470 [01:40<02:10, 1625.46 examples/s]
hf-map kk (num_proc=8): 46%|####5 | 176128/386470 [01:40<02:11, 1604.42 examples/s]
hf-map kk (num_proc=8): 46%|####5 | 177152/386470 [01:40<01:46, 1960.61 examples/s]
hf-map kk (num_proc=8): 46%|####6 | 178176/386470 [01:41<01:55, 1806.53 examples/s]
hf-map kk (num_proc=8): 46%|####6 | 179200/386470 [01:41<01:29, 2316.46 examples/s]
hf-map kk (num_proc=8): 47%|####6 | 180224/386470 [01:41<01:16, 2698.09 examples/s]
hf-map kk (num_proc=8): 47%|####6 | 181248/386470 [01:42<01:07, 3036.98 examples/s]
hf-map kk (num_proc=8): 47%|####7 | 183296/386470 [01:43<01:59, 1698.61 examples/s]
hf-map kk (num_proc=8): 48%|####7 | 184320/386470 [01:44<01:59, 1689.83 examples/s]
hf-map kk (num_proc=8): 48%|####7 | 185344/386470 [01:44<01:46, 1897.17 examples/s]
hf-map kk (num_proc=8): 48%|####8 | 186368/386470 [01:45<01:29, 2236.37 examples/s]
hf-map kk (num_proc=8): 48%|####8 | 187392/386470 [01:45<01:28, 2254.42 examples/s]
hf-map kk (num_proc=8): 49%|####8 | 188416/386470 [01:45<01:12, 2733.52 examples/s]
hf-map kk (num_proc=8): 49%|####9 | 189440/386470 [01:46<01:16, 2592.13 examples/s]
hf-map kk (num_proc=8): 50%|####9 | 191488/386470 [01:47<01:46, 1839.04 examples/s]
hf-map kk (num_proc=8): 50%|####9 | 192512/386470 [01:48<01:42, 1892.19 examples/s]
hf-map kk (num_proc=8): 50%|##### | 193536/386470 [01:48<01:50, 1744.98 examples/s]
hf-map kk (num_proc=8): 51%|##### | 195584/386470 [01:49<01:13, 2603.74 examples/s]
hf-map kk (num_proc=8): 51%|##### | 196608/386470 [01:49<01:02, 3013.80 examples/s]
hf-map kk (num_proc=8): 51%|#####1 | 197632/386470 [01:49<01:14, 2523.39 examples/s]
hf-map kk (num_proc=8): 51%|#####1 | 198656/386470 [01:50<01:22, 2287.22 examples/s]
hf-map kk (num_proc=8): 52%|#####1 | 199680/386470 [01:51<01:45, 1764.08 examples/s]
hf-map kk (num_proc=8): 52%|#####1 | 200704/386470 [01:52<01:51, 1672.10 examples/s]
hf-map kk (num_proc=8): 52%|#####2 | 201728/386470 [01:52<01:43, 1781.80 examples/s]
hf-map kk (num_proc=8): 52%|#####2 | 202752/386470 [01:52<01:26, 2129.64 examples/s]
hf-map kk (num_proc=8): 53%|#####2 | 204800/386470 [01:53<00:57, 3135.82 examples/s]
hf-map kk (num_proc=8): 53%|#####3 | 205824/386470 [01:53<01:12, 2487.21 examples/s]
hf-map kk (num_proc=8): 54%|#####3 | 206848/386470 [01:54<01:06, 2713.48 examples/s]
hf-map kk (num_proc=8): 54%|#####3 | 207872/386470 [01:54<01:25, 2084.91 examples/s]
hf-map kk (num_proc=8): 54%|#####4 | 208896/386470 [01:55<01:23, 2117.75 examples/s]
hf-map kk (num_proc=8): 54%|#####4 | 209920/386470 [01:55<01:16, 2316.70 examples/s]
hf-map kk (num_proc=8): 55%|#####4 | 210944/386470 [01:56<01:15, 2326.16 examples/s]
hf-map kk (num_proc=8): 55%|#####5 | 212992/386470 [01:56<00:55, 3121.21 examples/s]
hf-map kk (num_proc=8): 55%|#####5 | 214016/386470 [01:56<00:47, 3654.48 examples/s]
hf-map kk (num_proc=8): 56%|#####5 | 215040/386470 [01:57<00:52, 3292.80 examples/s]
hf-map kk (num_proc=8): 56%|#####5 | 216064/386470 [01:57<01:13, 2333.98 examples/s]
hf-map kk (num_proc=8): 56%|#####6 | 217088/386470 [01:58<01:24, 2004.03 examples/s]
hf-map kk (num_proc=8): 56%|#####6 | 218112/386470 [01:59<01:24, 1999.89 examples/s]
hf-map kk (num_proc=8): 57%|#####6 | 219136/386470 [01:59<01:04, 2575.85 examples/s]
hf-map kk (num_proc=8): 57%|#####6 | 220160/386470 [01:59<01:05, 2558.43 examples/s]
hf-map kk (num_proc=8): 57%|#####7 | 221184/386470 [02:00<01:18, 2095.46 examples/s]
hf-map kk (num_proc=8): 57%|#####7 | 222208/386470 [02:00<01:22, 1989.70 examples/s]
hf-map kk (num_proc=8): 58%|#####8 | 224256/386470 [02:01<00:58, 2782.42 examples/s]
hf-map kk (num_proc=8): 58%|#####8 | 225280/386470 [02:02<01:22, 1963.19 examples/s]
hf-map kk (num_proc=8): 59%|#####9 | 228352/386470 [02:02<00:47, 3334.71 examples/s]
hf-map kk (num_proc=8): 59%|#####9 | 229376/386470 [02:03<01:03, 2471.26 examples/s]
hf-map kk (num_proc=8): 60%|#####9 | 231424/386470 [02:03<00:51, 3023.97 examples/s]
hf-map kk (num_proc=8): 60%|###### | 232448/386470 [02:04<00:50, 3021.44 examples/s]
hf-map kk (num_proc=8): 60%|###### | 233472/386470 [02:05<01:17, 1962.17 examples/s]
hf-map kk (num_proc=8): 61%|###### | 235520/386470 [02:05<00:51, 2924.89 examples/s]
hf-map kk (num_proc=8): 61%|######1 | 236544/386470 [02:06<00:59, 2525.54 examples/s]
hf-map kk (num_proc=8): 61%|######1 | 237568/386470 [02:06<00:53, 2764.52 examples/s]
hf-map kk (num_proc=8): 62%|######1 | 238592/386470 [02:06<00:52, 2832.40 examples/s]
hf-map kk (num_proc=8): 62%|######2 | 239616/386470 [02:06<00:42, 3460.53 examples/s]
hf-map kk (num_proc=8): 62%|######2 | 240640/386470 [02:07<00:39, 3690.87 examples/s]
hf-map kk (num_proc=8): 63%|######2 | 241664/386470 [02:08<01:09, 2082.18 examples/s]
hf-map kk (num_proc=8): 63%|######3 | 243712/386470 [02:08<00:47, 2999.04 examples/s]
hf-map kk (num_proc=8): 63%|######3 | 244736/386470 [02:08<00:47, 2999.79 examples/s]
hf-map kk (num_proc=8): 64%|######3 | 245760/386470 [02:09<00:55, 2531.56 examples/s]
hf-map kk (num_proc=8): 64%|######3 | 246784/386470 [02:09<00:52, 2677.06 examples/s]
hf-map kk (num_proc=8): 64%|######4 | 248832/386470 [02:09<00:34, 3958.21 examples/s]
hf-map kk (num_proc=8): 65%|######4 | 249856/386470 [02:10<00:59, 2290.33 examples/s]
hf-map kk (num_proc=8): 65%|######4 | 250880/386470 [02:11<00:56, 2403.84 examples/s]
hf-map kk (num_proc=8): 65%|######5 | 252928/386470 [02:11<00:36, 3670.20 examples/s]
hf-map kk (num_proc=8): 66%|######5 | 253952/386470 [02:12<00:58, 2266.71 examples/s]
hf-map kk (num_proc=8): 66%|######5 | 254976/386470 [02:12<00:47, 2779.05 examples/s]
hf-map kk (num_proc=8): 66%|######6 | 256000/386470 [02:12<00:42, 3074.93 examples/s]
hf-map kk (num_proc=8): 67%|######6 | 257024/386470 [02:13<00:38, 3326.93 examples/s]
hf-map kk (num_proc=8): 67%|######6 | 258048/386470 [02:13<00:58, 2212.12 examples/s]
hf-map kk (num_proc=8): 67%|######7 | 259072/386470 [02:14<00:49, 2577.53 examples/s]
hf-map kk (num_proc=8): 67%|######7 | 260096/386470 [02:14<00:41, 3067.55 examples/s]
hf-map kk (num_proc=8): 68%|######7 | 261120/386470 [02:14<00:38, 3278.53 examples/s]
hf-map kk (num_proc=8): 68%|######7 | 262144/386470 [02:15<00:57, 2156.06 examples/s]
hf-map kk (num_proc=8): 68%|######8 | 263168/386470 [02:15<00:44, 2779.57 examples/s]
hf-map kk (num_proc=8): 69%|######8 | 265216/386470 [02:16<00:36, 3289.25 examples/s]
hf-map kk (num_proc=8): 69%|######8 | 266240/386470 [02:16<00:50, 2372.35 examples/s]
hf-map kk (num_proc=8): 69%|######9 | 267264/386470 [02:16<00:42, 2824.62 examples/s]
hf-map kk (num_proc=8): 70%|######9 | 269312/386470 [02:17<00:42, 2745.33 examples/s]
hf-map kk (num_proc=8): 70%|######9 | 270336/386470 [02:18<00:40, 2867.67 examples/s]
hf-map kk (num_proc=8): 70%|####### | 271360/386470 [02:18<00:36, 3151.80 examples/s]
hf-map kk (num_proc=8): 70%|####### | 272384/386470 [02:18<00:32, 3486.39 examples/s]
hf-map kk (num_proc=8): 71%|####### | 273408/386470 [02:19<00:39, 2844.66 examples/s]
hf-map kk (num_proc=8): 71%|#######1 | 274432/386470 [02:19<00:45, 2468.48 examples/s]
hf-map kk (num_proc=8): 72%|#######1 | 276480/386470 [02:19<00:31, 3529.64 examples/s]
hf-map kk (num_proc=8): 72%|#######1 | 277504/386470 [02:20<00:42, 2561.35 examples/s]
hf-map kk (num_proc=8): 72%|#######2 | 278528/386470 [02:21<00:46, 2319.69 examples/s]
hf-map kk (num_proc=8): 72%|#######2 | 279552/386470 [02:21<00:37, 2843.15 examples/s]
hf-map kk (num_proc=8): 73%|#######2 | 280576/386470 [02:21<00:35, 2950.65 examples/s]
hf-map kk (num_proc=8): 73%|#######2 | 281600/386470 [02:22<00:48, 2151.63 examples/s]
hf-map kk (num_proc=8): 73%|#######3 | 282624/386470 [02:22<00:47, 2194.61 examples/s]
hf-map kk (num_proc=8): 74%|#######3 | 284672/386470 [02:23<00:44, 2309.83 examples/s]
hf-map kk (num_proc=8): 74%|#######3 | 285696/386470 [02:23<00:38, 2599.00 examples/s]
hf-map kk (num_proc=8): 74%|#######4 | 286720/386470 [02:24<00:46, 2149.70 examples/s]
hf-map kk (num_proc=8): 75%|#######4 | 288768/386470 [02:24<00:31, 3119.71 examples/s]
hf-map kk (num_proc=8): 75%|#######4 | 289792/386470 [02:25<00:39, 2478.25 examples/s]
hf-map kk (num_proc=8): 75%|#######5 | 290816/386470 [02:26<00:41, 2320.99 examples/s]
hf-map kk (num_proc=8): 76%|#######5 | 291840/386470 [02:26<00:34, 2759.10 examples/s]
hf-map kk (num_proc=8): 76%|#######5 | 292864/386470 [02:27<00:49, 1877.04 examples/s]
hf-map kk (num_proc=8): 76%|#######6 | 293888/386470 [02:27<00:46, 2005.32 examples/s]
hf-map kk (num_proc=8): 76%|#######6 | 294912/386470 [02:27<00:38, 2354.69 examples/s]
hf-map kk (num_proc=8): 77%|#######6 | 296960/386470 [02:28<00:25, 3535.96 examples/s]
hf-map kk (num_proc=8): 77%|#######7 | 297984/386470 [02:29<00:45, 1961.59 examples/s]
hf-map kk (num_proc=8): 77%|#######7 | 299008/386470 [02:29<00:38, 2262.37 examples/s]
hf-map kk (num_proc=8): 78%|#######7 | 300032/386470 [02:29<00:31, 2740.83 examples/s]
hf-map kk (num_proc=8): 78%|#######7 | 301056/386470 [02:30<00:46, 1838.79 examples/s]
hf-map kk (num_proc=8): 78%|#######8 | 302080/386470 [02:31<00:43, 1943.96 examples/s]
hf-map kk (num_proc=8): 79%|#######8 | 304128/386470 [02:31<00:27, 2955.88 examples/s]
hf-map kk (num_proc=8): 79%|#######9 | 306176/386470 [02:32<00:35, 2278.73 examples/s]
hf-map kk (num_proc=8): 79%|#######9 | 307200/386470 [02:33<00:33, 2367.77 examples/s]
hf-map kk (num_proc=8): 80%|######## | 309248/386470 [02:34<00:36, 2097.21 examples/s]
hf-map kk (num_proc=8): 80%|######## | 310272/386470 [02:34<00:31, 2439.46 examples/s]
hf-map kk (num_proc=8): 81%|######## | 311296/386470 [02:34<00:30, 2425.59 examples/s]
hf-map kk (num_proc=8): 81%|######## | 312320/386470 [02:35<00:25, 2853.84 examples/s]
hf-map kk (num_proc=8): 81%|########1 | 313344/386470 [02:35<00:24, 3032.80 examples/s]
hf-map kk (num_proc=8): 81%|########1 | 314368/386470 [02:36<00:32, 2249.82 examples/s]
hf-map kk (num_proc=8): 82%|########1 | 315392/386470 [02:36<00:31, 2289.83 examples/s]
hf-map kk (num_proc=8): 82%|########1 | 316416/386470 [02:36<00:25, 2737.53 examples/s]
hf-map kk (num_proc=8): 82%|########2 | 317440/386470 [02:37<00:38, 1801.10 examples/s]
hf-map kk (num_proc=8): 82%|########2 | 318464/386470 [02:37<00:28, 2366.06 examples/s]
hf-map kk (num_proc=8): 83%|########2 | 319488/386470 [02:38<00:30, 2224.95 examples/s]
hf-map kk (num_proc=8): 83%|########2 | 320512/386470 [02:38<00:28, 2292.58 examples/s]
hf-map kk (num_proc=8): 83%|########3 | 321536/386470 [02:39<00:25, 2519.08 examples/s]
hf-map kk (num_proc=8): 83%|########3 | 322560/386470 [02:39<00:22, 2885.82 examples/s]
hf-map kk (num_proc=8): 84%|########3 | 323584/386470 [02:40<00:26, 2332.98 examples/s]
hf-map kk (num_proc=8): 84%|########3 | 324608/386470 [02:40<00:21, 2862.58 examples/s]
hf-map kk (num_proc=8): 84%|########4 | 325632/386470 [02:41<00:29, 2039.20 examples/s]
hf-map kk (num_proc=8): 85%|########4 | 326656/386470 [02:41<00:27, 2154.18 examples/s]
hf-map kk (num_proc=8): 85%|########4 | 327680/386470 [02:42<00:31, 1846.96 examples/s]
hf-map kk (num_proc=8): 85%|########5 | 329728/386470 [02:42<00:21, 2580.03 examples/s]
hf-map kk (num_proc=8): 86%|########5 | 331776/386470 [02:43<00:17, 3063.70 examples/s]
hf-map kk (num_proc=8): 86%|########6 | 332800/386470 [02:43<00:17, 2995.79 examples/s]
hf-map kk (num_proc=8): 86%|########6 | 333824/386470 [02:43<00:17, 3048.10 examples/s]
hf-map kk (num_proc=8): 87%|########6 | 334848/386470 [02:44<00:18, 2813.75 examples/s]
hf-map kk (num_proc=8): 87%|########6 | 335872/386470 [02:44<00:22, 2253.03 examples/s]
hf-map kk (num_proc=8): 87%|########7 | 336896/386470 [02:45<00:19, 2561.03 examples/s]
hf-map kk (num_proc=8): 87%|########7 | 337920/386470 [02:45<00:17, 2752.50 examples/s]
hf-map kk (num_proc=8): 88%|########7 | 338944/386470 [02:45<00:13, 3440.64 examples/s]
hf-map kk (num_proc=8): 88%|########7 | 339968/386470 [02:45<00:13, 3356.23 examples/s]
hf-map kk (num_proc=8): 88%|########8 | 340992/386470 [02:46<00:14, 3058.95 examples/s]
hf-map kk (num_proc=8): 88%|########8 | 342016/386470 [02:46<00:12, 3614.85 examples/s]
hf-map kk (num_proc=8): 89%|########8 | 343040/386470 [02:47<00:19, 2177.11 examples/s]
hf-map kk (num_proc=8): 89%|########9 | 344064/386470 [02:47<00:20, 2050.66 examples/s]
hf-map kk (num_proc=8): 89%|########9 | 345088/386470 [02:48<00:21, 1924.65 examples/s]
hf-map kk (num_proc=8): 90%|########9 | 346112/386470 [02:48<00:17, 2358.66 examples/s]
hf-map kk (num_proc=8): 90%|######### | 348160/386470 [02:49<00:17, 2180.46 examples/s]
hf-map kk (num_proc=8): 91%|######### | 350208/386470 [02:49<00:11, 3287.72 examples/s]
hf-map kk (num_proc=8): 91%|######### | 351232/386470 [02:50<00:15, 2340.50 examples/s]
hf-map kk (num_proc=8): 91%|#########1| 352256/386470 [02:51<00:18, 1872.38 examples/s]
hf-map kk (num_proc=8): 91%|#########1| 353280/386470 [02:52<00:16, 2057.12 examples/s]
hf-map kk (num_proc=8): 92%|#########1| 354304/386470 [02:52<00:13, 2328.13 examples/s]
hf-map kk (num_proc=8): 92%|#########1| 355328/386470 [02:52<00:12, 2546.34 examples/s]
hf-map kk (num_proc=8): 92%|#########2| 356352/386470 [02:53<00:12, 2367.92 examples/s]
hf-map kk (num_proc=8): 92%|#########2| 357376/386470 [02:53<00:10, 2834.44 examples/s]
hf-map kk (num_proc=8): 93%|#########2| 358400/386470 [02:53<00:08, 3166.16 examples/s]
hf-map kk (num_proc=8): 93%|#########3| 359424/386470 [02:54<00:15, 1731.75 examples/s]
hf-map kk (num_proc=8): 93%|#########3| 360448/386470 [02:55<00:15, 1630.11 examples/s]
hf-map kk (num_proc=8): 94%|#########3| 361472/386470 [02:55<00:12, 1930.69 examples/s]
hf-map kk (num_proc=8): 94%|#########4| 363520/386470 [02:56<00:07, 3163.99 examples/s]
hf-map kk (num_proc=8): 94%|#########4| 364544/386470 [02:56<00:10, 2116.37 examples/s]
hf-map kk (num_proc=8): 95%|#########4| 365568/386470 [02:57<00:08, 2375.14 examples/s]
hf-map kk (num_proc=8): 95%|#########4| 366592/386470 [02:57<00:06, 2843.03 examples/s]
hf-map kk (num_proc=8): 95%|#########5| 367616/386470 [02:58<00:09, 2027.47 examples/s]
hf-map kk (num_proc=8): 95%|#########5| 368640/386470 [02:58<00:08, 2101.95 examples/s]
hf-map kk (num_proc=8): 96%|#########5| 369664/386470 [02:59<00:09, 1705.45 examples/s]
hf-map kk (num_proc=8): 96%|#########6| 371712/386470 [03:00<00:06, 2271.98 examples/s]
hf-map kk (num_proc=8): 97%|#########6| 373760/386470 [03:00<00:04, 2579.25 examples/s]
hf-map kk (num_proc=8): 97%|#########7| 375808/386470 [03:01<00:04, 2357.42 examples/s]
hf-map kk (num_proc=8): 98%|#########7| 376832/386470 [03:02<00:03, 2485.30 examples/s]
hf-map kk (num_proc=8): 98%|#########7| 378037/386470 [03:03<00:04, 1837.39 examples/s]
hf-map kk (num_proc=8): 98%|#########8| 380085/386470 [03:03<00:02, 2688.16 examples/s]
hf-map kk (num_proc=8): 99%|#########8| 381109/386470 [03:03<00:01, 2971.22 examples/s]
hf-map kk (num_proc=8): 99%|#########8| 382314/386470 [03:04<00:01, 2874.08 examples/s]
hf-map kk (num_proc=8): 99%|#########9| 383519/386470 [03:04<00:00, 3124.14 examples/s]
hf-map kk (num_proc=8): 100%|#########9| 384724/386470 [03:05<00:00, 2752.42 examples/s]
hf-map kk (num_proc=8): 100%|#########9| 385085/386470 [03:05<00:00, 2517.09 examples/s]
hf-map kk (num_proc=8): 100%|#########9| 386290/386470 [03:06<00:00, 1690.01 examples/s]
hf-map kk (num_proc=8): 100%|##########| 386470/386470 [03:07<00:00, 2059.14 examples/s] |
| kk: datasets.map num_proc=8 done 188s |
| kk: 25000 docs 40985213 tokens 194s |
| kk: 50000 docs 81894926 tokens 198s |
| kk: 75000 docs 121953883 tokens 201s |
| kk: 100000 docs 163346527 tokens 204s |
| kk: 125000 docs 204439098 tokens 208s |
| kk: 150000 docs 245100472 tokens 211s |
| kk: 175000 docs 284907275 tokens 214s |
| kk: 200000 docs 326211845 tokens 218s |
| kk: 225000 docs 368265583 tokens 221s |
| kk: 250000 docs 408948851 tokens 224s |
| kk: 275000 docs 449889902 tokens 228s |
| packed kk: 466196065 tokens (1.000 of budget) rows=602321 docs=285249 pad-per-doc (HF map) |
| tl: datasets.map n=50150 batched=1024 num_proc=8 |
|
hf-map tl (num_proc=8): 0%| | 0/50150 [00:00<?, ? examples/s]
hf-map tl (num_proc=8): 2%|2 | 1024/50150 [00:05<04:36, 177.77 examples/s]
hf-map tl (num_proc=8): 6%|6 | 3072/50150 [00:05<01:12, 653.14 examples/s]
hf-map tl (num_proc=8): 12%|#2 | 6144/50150 [00:06<00:26, 1633.64 examples/s]
hf-map tl (num_proc=8): 16%|#6 | 8192/50150 [00:06<00:17, 2395.63 examples/s]
hf-map tl (num_proc=8): 20%|## | 10240/50150 [00:08<00:24, 1629.19 examples/s]
hf-map tl (num_proc=8): 25%|##4 | 12288/50150 [00:08<00:16, 2229.91 examples/s]
hf-map tl (num_proc=8): 31%|### | 15360/50150 [00:08<00:09, 3570.79 examples/s]
hf-map tl (num_proc=8): 35%|###4 | 17408/50150 [00:10<00:15, 2133.75 examples/s]
hf-map tl (num_proc=8): 39%|###8 | 19456/50150 [00:10<00:10, 2816.14 examples/s]
hf-map tl (num_proc=8): 43%|####2 | 21504/50150 [00:10<00:07, 3652.85 examples/s]
hf-map tl (num_proc=8): 47%|####6 | 23552/50150 [00:10<00:05, 4718.59 examples/s]
hf-map tl (num_proc=8): 51%|#####1 | 25600/50150 [00:12<00:09, 2671.26 examples/s]
hf-map tl (num_proc=8): 53%|#####3 | 26624/50150 [00:12<00:08, 2887.76 examples/s]
hf-map tl (num_proc=8): 57%|#####7 | 28672/50150 [00:12<00:05, 3900.87 examples/s]
hf-map tl (num_proc=8): 61%|######1 | 30720/50150 [00:13<00:03, 4927.04 examples/s]
hf-map tl (num_proc=8): 63%|######3 | 31744/50150 [00:13<00:03, 5170.02 examples/s]
hf-map tl (num_proc=8): 67%|######7 | 33792/50150 [00:14<00:06, 2675.14 examples/s]
hf-map tl (num_proc=8): 71%|#######1 | 35840/50150 [00:15<00:04, 3225.36 examples/s]
hf-map tl (num_proc=8): 78%|#######7 | 38912/50150 [00:15<00:02, 4577.65 examples/s]
hf-map tl (num_proc=8): 80%|#######9 | 39936/50150 [00:15<00:02, 4862.49 examples/s]
hf-map tl (num_proc=8): 84%|########3 | 41984/50150 [00:16<00:02, 3331.14 examples/s]
hf-map tl (num_proc=8): 86%|########5 | 43008/50150 [00:16<00:01, 3783.25 examples/s]
hf-map tl (num_proc=8): 88%|########8 | 44282/50150 [00:17<00:01, 3431.20 examples/s]
hf-map tl (num_proc=8): 94%|#########4| 47354/50150 [00:17<00:00, 5266.06 examples/s]
hf-map tl (num_proc=8): 96%|#########6| 48378/50150 [00:17<00:00, 5631.04 examples/s]
hf-map tl (num_proc=8): 99%|#########9| 49651/50150 [00:17<00:00, 6472.24 examples/s]
hf-map tl (num_proc=8): 100%|##########| 50150/50150 [00:18<00:00, 2766.55 examples/s] |
| tl: datasets.map num_proc=8 done 18s |
| tl: 25000 docs 22165685 tokens 21s |
| packed tl: 32043235 tokens (1.000 of budget) rows=55838 docs=36428 pad-per-doc (HF map) |
| en: datasets.map n=196400 batched=1024 num_proc=8 |
|
hf-map en (num_proc=8): 0%| | 0/196400 [00:00<?, ? examples/s]
hf-map en (num_proc=8): 1%| | 1024/196400 [00:11<34:59, 93.05 examples/s]
hf-map en (num_proc=8): 3%|3 | 6144/196400 [00:11<04:15, 744.78 examples/s]
hf-map en (num_proc=8): 5%|5 | 10240/196400 [00:15<03:40, 842.49 examples/s]
hf-map en (num_proc=8): 6%|6 | 12288/196400 [00:15<02:43, 1125.82 examples/s]
hf-map en (num_proc=8): 7%|7 | 14336/196400 [00:15<02:00, 1515.76 examples/s]
hf-map en (num_proc=8): 8%|8 | 16384/196400 [00:15<01:28, 2033.16 examples/s]
hf-map en (num_proc=8): 9%|9 | 18432/196400 [00:19<02:36, 1140.64 examples/s]
hf-map en (num_proc=8): 10%|# | 20480/196400 [00:19<01:52, 1558.15 examples/s]
hf-map en (num_proc=8): 13%|#2 | 24576/196400 [00:19<01:03, 2713.98 examples/s]
hf-map en (num_proc=8): 14%|#3 | 26624/196400 [00:23<01:56, 1452.73 examples/s]
hf-map en (num_proc=8): 15%|#5 | 29696/196400 [00:23<01:19, 2100.03 examples/s]
hf-map en (num_proc=8): 17%|#6 | 32768/196400 [00:23<00:55, 2938.07 examples/s]
hf-map en (num_proc=8): 18%|#7 | 34816/196400 [00:26<01:40, 1608.36 examples/s]
hf-map en (num_proc=8): 19%|#8 | 36864/196400 [00:26<01:16, 2075.71 examples/s]
hf-map en (num_proc=8): 20%|#9 | 38912/196400 [00:27<00:59, 2643.47 examples/s]
hf-map en (num_proc=8): 21%|## | 40960/196400 [00:27<00:45, 3447.32 examples/s]
hf-map en (num_proc=8): 22%|##1 | 43008/196400 [00:30<01:32, 1656.57 examples/s]
hf-map en (num_proc=8): 23%|##3 | 46080/196400 [00:30<00:58, 2566.47 examples/s]
hf-map en (num_proc=8): 25%|##4 | 48128/196400 [00:30<00:45, 3226.55 examples/s]
hf-map en (num_proc=8): 26%|##5 | 50176/196400 [00:32<01:14, 1967.04 examples/s]
hf-map en (num_proc=8): 26%|##6 | 51200/196400 [00:32<01:06, 2190.79 examples/s]
hf-map en (num_proc=8): 27%|##7 | 53248/196400 [00:32<00:47, 3044.22 examples/s]
hf-map en (num_proc=8): 28%|##8 | 55296/196400 [00:32<00:36, 3895.94 examples/s]
hf-map en (num_proc=8): 29%|##9 | 57344/196400 [00:33<00:28, 4934.86 examples/s]
hf-map en (num_proc=8): 30%|### | 59392/196400 [00:35<01:03, 2163.08 examples/s]
hf-map en (num_proc=8): 32%|###1 | 62464/196400 [00:35<00:41, 3193.31 examples/s]
hf-map en (num_proc=8): 33%|###2 | 64512/196400 [00:35<00:32, 4100.27 examples/s]
hf-map en (num_proc=8): 34%|###3 | 66560/196400 [00:37<00:58, 2224.58 examples/s]
hf-map en (num_proc=8): 35%|###4 | 68608/196400 [00:38<00:47, 2713.46 examples/s]
hf-map en (num_proc=8): 36%|###6 | 71680/196400 [00:38<00:31, 3928.59 examples/s]
hf-map en (num_proc=8): 38%|###7 | 73728/196400 [00:38<00:30, 4082.41 examples/s]
hf-map en (num_proc=8): 38%|###8 | 74752/196400 [00:39<00:45, 2670.07 examples/s]
hf-map en (num_proc=8): 39%|###8 | 75776/196400 [00:40<00:45, 2641.33 examples/s]
hf-map en (num_proc=8): 39%|###9 | 76800/196400 [00:40<00:45, 2642.75 examples/s]
hf-map en (num_proc=8): 40%|###9 | 77824/196400 [00:40<00:37, 3131.09 examples/s]
hf-map en (num_proc=8): 41%|####1 | 80896/196400 [00:40<00:20, 5630.18 examples/s]
hf-map en (num_proc=8): 42%|####2 | 82944/196400 [00:42<00:36, 3136.05 examples/s]
hf-map en (num_proc=8): 43%|####2 | 83968/196400 [00:42<00:38, 2934.84 examples/s]
hf-map en (num_proc=8): 43%|####3 | 84992/196400 [00:42<00:37, 2947.40 examples/s]
hf-map en (num_proc=8): 44%|####4 | 87040/196400 [00:42<00:25, 4246.33 examples/s]
hf-map en (num_proc=8): 45%|####5 | 89088/196400 [00:43<00:20, 5263.78 examples/s]
hf-map en (num_proc=8): 46%|####6 | 91136/196400 [00:44<00:34, 3053.43 examples/s]
hf-map en (num_proc=8): 47%|####6 | 92160/196400 [00:44<00:38, 2727.04 examples/s]
hf-map en (num_proc=8): 47%|####7 | 93184/196400 [00:45<00:37, 2786.27 examples/s]
hf-map en (num_proc=8): 48%|####7 | 94208/196400 [00:45<00:35, 2919.03 examples/s]
hf-map en (num_proc=8): 50%|####9 | 97280/196400 [00:45<00:19, 5159.00 examples/s]
hf-map en (num_proc=8): 50%|##### | 98304/196400 [00:46<00:21, 4646.30 examples/s]
hf-map en (num_proc=8): 51%|##### | 99328/196400 [00:46<00:34, 2793.09 examples/s]
hf-map en (num_proc=8): 51%|#####1 | 100352/196400 [00:47<00:39, 2433.96 examples/s]
hf-map en (num_proc=8): 52%|#####1 | 101376/196400 [00:47<00:36, 2602.98 examples/s]
hf-map en (num_proc=8): 52%|#####2 | 102400/196400 [00:47<00:29, 3236.70 examples/s]
hf-map en (num_proc=8): 53%|#####3 | 104448/196400 [00:48<00:19, 4690.60 examples/s]
hf-map en (num_proc=8): 54%|#####4 | 106496/196400 [00:48<00:18, 4905.87 examples/s]
hf-map en (num_proc=8): 55%|#####4 | 107520/196400 [00:49<00:28, 3081.05 examples/s]
hf-map en (num_proc=8): 55%|#####5 | 108544/196400 [00:50<00:40, 2155.37 examples/s]
hf-map en (num_proc=8): 56%|#####5 | 109568/196400 [00:50<00:37, 2314.00 examples/s]
hf-map en (num_proc=8): 57%|#####7 | 112640/196400 [00:50<00:21, 3960.95 examples/s]
hf-map en (num_proc=8): 58%|#####7 | 113664/196400 [00:50<00:18, 4360.06 examples/s]
hf-map en (num_proc=8): 58%|#####8 | 114688/196400 [00:51<00:17, 4607.10 examples/s]
hf-map en (num_proc=8): 59%|#####8 | 115712/196400 [00:52<00:30, 2675.47 examples/s]
hf-map en (num_proc=8): 59%|#####9 | 116736/196400 [00:53<00:44, 1803.34 examples/s]
hf-map en (num_proc=8): 60%|###### | 118784/196400 [00:53<00:26, 2902.19 examples/s]
hf-map en (num_proc=8): 62%|######1 | 120832/196400 [00:53<00:18, 4094.07 examples/s]
hf-map en (num_proc=8): 62%|######2 | 121856/196400 [00:53<00:17, 4231.49 examples/s]
hf-map en (num_proc=8): 63%|######3 | 123904/196400 [00:54<00:25, 2809.30 examples/s]
hf-map en (num_proc=8): 64%|######3 | 124928/196400 [00:55<00:34, 2070.84 examples/s]
hf-map en (num_proc=8): 65%|######4 | 126976/196400 [00:55<00:22, 3072.02 examples/s]
hf-map en (num_proc=8): 66%|######5 | 129024/196400 [00:56<00:15, 4349.35 examples/s]
hf-map en (num_proc=8): 67%|######6 | 131072/196400 [00:56<00:15, 4182.16 examples/s]
hf-map en (num_proc=8): 67%|######7 | 132096/196400 [00:57<00:22, 2852.84 examples/s]
hf-map en (num_proc=8): 68%|######7 | 133120/196400 [00:58<00:29, 2128.36 examples/s]
hf-map en (num_proc=8): 68%|######8 | 134144/196400 [00:58<00:25, 2395.43 examples/s]
hf-map en (num_proc=8): 70%|######9 | 137216/196400 [00:58<00:14, 4216.92 examples/s]
hf-map en (num_proc=8): 70%|####### | 138240/196400 [00:58<00:12, 4490.24 examples/s]
hf-map en (num_proc=8): 71%|####### | 139264/196400 [00:59<00:12, 4713.23 examples/s]
hf-map en (num_proc=8): 71%|#######1 | 140288/196400 [01:00<00:27, 2035.65 examples/s]
hf-map en (num_proc=8): 72%|#######1 | 141312/196400 [01:01<00:30, 1832.96 examples/s]
hf-map en (num_proc=8): 72%|#######2 | 142336/196400 [01:01<00:23, 2286.83 examples/s]
hf-map en (num_proc=8): 73%|#######2 | 143360/196400 [01:01<00:18, 2841.71 examples/s]
hf-map en (num_proc=8): 74%|#######4 | 145408/196400 [01:01<00:12, 4207.35 examples/s]
hf-map en (num_proc=8): 75%|#######4 | 146432/196400 [01:02<00:13, 3746.82 examples/s]
hf-map en (num_proc=8): 75%|#######5 | 147456/196400 [01:02<00:11, 4279.29 examples/s]
hf-map en (num_proc=8): 76%|#######5 | 148480/196400 [01:03<00:20, 2288.76 examples/s]
hf-map en (num_proc=8): 76%|#######6 | 149504/196400 [01:03<00:22, 2069.66 examples/s]
hf-map en (num_proc=8): 77%|#######6 | 150528/196400 [01:04<00:18, 2517.61 examples/s]
hf-map en (num_proc=8): 78%|#######8 | 153600/196400 [01:04<00:11, 3621.80 examples/s]
hf-map en (num_proc=8): 79%|#######8 | 154624/196400 [01:04<00:10, 3936.79 examples/s]
hf-map en (num_proc=8): 80%|#######9 | 156672/196400 [01:05<00:14, 2794.69 examples/s]
hf-map en (num_proc=8): 80%|######## | 157696/196400 [01:06<00:15, 2523.16 examples/s]
hf-map en (num_proc=8): 81%|########1 | 159744/196400 [01:06<00:10, 3451.17 examples/s]
hf-map en (num_proc=8): 82%|########1 | 160768/196400 [01:06<00:10, 3534.73 examples/s]
hf-map en (num_proc=8): 82%|########2 | 161792/196400 [01:07<00:08, 4005.08 examples/s]
hf-map en (num_proc=8): 83%|########2 | 162816/196400 [01:07<00:08, 3960.41 examples/s]
hf-map en (num_proc=8): 83%|########3 | 163840/196400 [01:07<00:08, 3848.30 examples/s]
hf-map en (num_proc=8): 84%|########3 | 164864/196400 [01:08<00:15, 2074.71 examples/s]
hf-map en (num_proc=8): 84%|########4 | 165888/196400 [01:09<00:13, 2236.31 examples/s]
hf-map en (num_proc=8): 85%|########4 | 166912/196400 [01:09<00:10, 2877.13 examples/s]
hf-map en (num_proc=8): 86%|########5 | 167936/196400 [01:09<00:09, 2938.64 examples/s]
hf-map en (num_proc=8): 86%|########6 | 168960/196400 [01:09<00:08, 3163.31 examples/s]
hf-map en (num_proc=8): 87%|########7 | 171008/196400 [01:10<00:06, 4063.07 examples/s]
hf-map en (num_proc=8): 88%|########7 | 172032/196400 [01:10<00:06, 3621.43 examples/s]
hf-map en (num_proc=8): 88%|########8 | 173056/196400 [01:11<00:13, 1789.92 examples/s]
hf-map en (num_proc=8): 89%|########8 | 174080/196400 [01:12<00:11, 1867.96 examples/s]
hf-map en (num_proc=8): 89%|########9 | 175104/196400 [01:12<00:09, 2344.26 examples/s]
hf-map en (num_proc=8): 90%|########9 | 176128/196400 [01:12<00:07, 2569.04 examples/s]
hf-map en (num_proc=8): 90%|######### | 177152/196400 [01:12<00:06, 3170.28 examples/s]
hf-map en (num_proc=8): 91%|######### | 178176/196400 [01:13<00:04, 3671.04 examples/s]
hf-map en (num_proc=8): 91%|#########1| 179200/196400 [01:13<00:03, 4395.70 examples/s]
hf-map en (num_proc=8): 92%|#########1| 180224/196400 [01:13<00:03, 4699.68 examples/s]
hf-map en (num_proc=8): 92%|#########2| 181248/196400 [01:14<00:06, 2405.14 examples/s]
hf-map en (num_proc=8): 93%|#########2| 182272/196400 [01:14<00:06, 2320.34 examples/s]
hf-map en (num_proc=8): 93%|#########3| 183296/196400 [01:15<00:05, 2574.40 examples/s]
hf-map en (num_proc=8): 94%|#########3| 184320/196400 [01:15<00:03, 3133.96 examples/s]
hf-map en (num_proc=8): 94%|#########4| 185344/196400 [01:15<00:03, 3526.08 examples/s]
hf-map en (num_proc=8): 95%|#########5| 187392/196400 [01:15<00:02, 3839.55 examples/s]
hf-map en (num_proc=8): 96%|#########5| 188416/196400 [01:16<00:01, 4063.02 examples/s]
hf-map en (num_proc=8): 96%|#########6| 189414/196400 [01:16<00:02, 2719.96 examples/s]
hf-map en (num_proc=8): 97%|#########6| 190412/196400 [01:17<00:02, 2279.52 examples/s]
hf-map en (num_proc=8): 97%|#########7| 191410/196400 [01:17<00:01, 2684.44 examples/s]
hf-map en (num_proc=8): 98%|#########8| 193406/196400 [01:17<00:00, 3711.16 examples/s]
hf-map en (num_proc=8): 99%|#########8| 194404/196400 [01:18<00:00, 4002.99 examples/s]
hf-map en (num_proc=8): 99%|#########9| 195402/196400 [01:18<00:00, 4692.26 examples/s]
hf-map en (num_proc=8): 100%|##########| 196400/196400 [01:18<00:00, 3777.14 examples/s]
hf-map en (num_proc=8): 100%|##########| 196400/196400 [01:18<00:00, 2487.50 examples/s] |
| en: datasets.map num_proc=8 done 79s |
| en: 25000 docs 19355384 tokens 82s |
| en: 50000 docs 38502040 tokens 84s |
| en: 75000 docs 57868565 tokens 86s |
| en: 100000 docs 77221904 tokens 88s |
| en: 125000 docs 96715846 tokens 90s |
| packed en: 111592053 tokens (1.000 of budget) rows=196029 docs=144428 pad-per-doc (HF map) |
| full mix budget 1115916236 (en=10%); packed 1645162 sequences, 984858 docs, 1115928627 tokens |
| eval every 100 steps; resume checkpoint-{step} every 500 steps, keep last 2; analysis model-{step} every 1000 steps (weights+tokenizer; upload SeragAmin/gemma_mapping_byteify_cpt_seq1024) |
| resume research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-4000 after step 4000/12852 |
| data next_row=512000 / 1645162 (mixed shuffle, perm_seed=42) |
| am: rows 19525/62355 tokens 15554785/49804449 |
| ta: rows 226844/728619 tokens 142032738/456292825 |
| kk: rows 187196/602321 tokens 144792658/466196065 |
| tl: rows 17436/55838 tokens 10007303/32043235 |
| en: rows 60999/196029 tokens 34790050/111592053 |
| loading research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-4000 |
| liger OLMo2 kernels (fused linear CE, RMSNorm, SwiGLU, RoPE) |
|
Loading weights: 0%| | 0/179 [00:00<?, ?it/s]
Loading weights: 23%|##3 | 42/179 [00:00<00:00, 419.03it/s]
Loading weights: 47%|####6 | 84/179 [00:00<00:00, 311.30it/s]
Loading weights: 75%|#######4 | 134/179 [00:00<00:00, 383.65it/s]
Loading weights: 100%|##########| 179/179 [00:00<00:00, 395.03it/s] |
| gradient_checkpointing=off liger=on |
| seed=42 flash SDP TF32 |
| full_cpt: 1645162 sequences → 12852 steps batch=16 accum=8 real_tokens=1115858680 liger fused CE, labels=-100 on pad, no attn mask |
| full_cpt: resume after step 4000/12852 next mixed packed row 512000/1645056 (global shuffle; not per-language iterators) |
| Backend impl 'liger_kernel.ops.backends._cutedsl.rope' for op 'rope' could not be imported: ImportError: cutedsl backend requires the NVIDIA CUTLASS Python DSL (CuTe DSL). Install it with `pip install nvidia-cutlass-dsl`, or when installing Liger-Kernel use `pip install 'liger-kernel[cutedsl]'`. |
| Backend impl 'liger_kernel.ops.backends._ascend.rope' for op 'rope' could not be imported: ImportError: cannot import name 'tanh' from 'triton.language.math' (/usr/local/lib/python3.12/dist-packages/triton/language/math.py) |
| full_cpt step 4010/12852 ( 31.2%) nll=3.3252 ppl=27.80 grad=0.930 lr=1.62e-05 7319 tok/s 11.52s/it [01:55<28:17:51] mem=34.07GB |
| full_cpt step 4020/12852 ( 31.3%) nll=3.4189 ppl=30.54 grad=0.953 lr=1.62e-05 7357 tok/s 11.56s/it [03:51<28:21:21] mem=34.07GB |
| full_cpt step 4030/12852 ( 31.4%) nll=3.2766 ppl=26.48 grad=0.879 lr=1.61e-05 7378 tok/s 11.57s/it [05:47<28:21:25] mem=34.07GB |
| full_cpt step 4040/12852 ( 31.4%) nll=3.2380 ppl=25.48 grad=0.859 lr=1.61e-05 7403 tok/s 11.58s/it [07:43<28:20:44] mem=34.07GB |
| full_cpt step 4050/12852 ( 31.5%) nll=3.2843 ppl=26.69 grad=0.902 lr=1.61e-05 7423 tok/s 11.59s/it [09:39<28:19:43] mem=34.07GB |
| full_cpt step 4060/12852 ( 31.6%) nll=3.4233 ppl=30.67 grad=0.949 lr=1.61e-05 7410 tok/s 11.59s/it [11:35<28:18:10] mem=34.07GB |
| full_cpt step 4070/12852 ( 31.7%) nll=3.3702 ppl=29.08 grad=0.848 lr=1.61e-05 7449 tok/s 11.59s/it [13:31<28:16:45] mem=34.07GB |
| full_cpt step 4080/12852 ( 31.7%) nll=3.5208 ppl=33.81 grad=0.965 lr=1.60e-05 7444 tok/s 11.59s/it [15:27<28:14:48] mem=34.07GB |
| full_cpt step 4090/12852 ( 31.8%) nll=3.3677 ppl=29.01 grad=0.934 lr=1.60e-05 7449 tok/s 11.59s/it [17:23<28:12:58] mem=34.07GB |
| full_cpt step 4100/12852 ( 31.9%) nll=3.3091 ppl=27.36 grad=0.898 lr=1.60e-05 7456 tok/s 11.59s/it [19:19<28:11:08] mem=34.07GB |
| am nll=3.9921 ppl=54.17 bpb=1.4038 |
| ta nll=3.9636 ppl=52.65 bpb=0.5757 |
| kk nll=2.7664 ppl=15.90 bpb=0.9316 |
| tl nll=2.6622 ppl=14.33 bpb=1.2632 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 4100 01:28 (87.7s) |
| full_cpt step 4110/12852 ( 32.0%) nll=3.3330 ppl=28.02 grad=0.941 lr=1.60e-05 6984 tok/s 12.39s/it [22:43<30:05:24] mem=34.07GB |
| full_cpt step 4120/12852 ( 32.1%) nll=3.2017 ppl=24.58 grad=0.895 lr=1.60e-05 7029 tok/s 12.33s/it [24:39<29:53:50] mem=34.07GB |
| full_cpt step 4130/12852 ( 32.1%) nll=3.3954 ppl=29.83 grad=0.926 lr=1.60e-05 7074 tok/s 12.27s/it [26:35<29:43:47] mem=34.07GB |
| full_cpt step 4140/12852 ( 32.2%) nll=3.3304 ppl=27.95 grad=0.910 lr=1.59e-05 7093 tok/s 12.22s/it [28:31<29:34:40] mem=34.07GB |
| full_cpt step 4150/12852 ( 32.3%) nll=3.2906 ppl=26.86 grad=0.832 lr=1.59e-05 7121 tok/s 12.18s/it [30:27<29:26:40] mem=34.07GB |
| full_cpt step 4160/12852 ( 32.4%) nll=3.2929 ppl=26.92 grad=0.926 lr=1.59e-05 7131 tok/s 12.14s/it [32:23<29:19:12] mem=34.07GB |
| full_cpt step 4170/12852 ( 32.4%) nll=3.1690 ppl=23.78 grad=0.852 lr=1.59e-05 7149 tok/s 12.11s/it [34:19<29:12:29] mem=34.07GB |
| full_cpt step 4180/12852 ( 32.5%) nll=3.2696 ppl=26.30 grad=0.938 lr=1.59e-05 7160 tok/s 12.08s/it [36:15<29:06:16] mem=34.07GB |
| full_cpt step 4190/12852 ( 32.6%) nll=3.3850 ppl=29.52 grad=0.867 lr=1.58e-05 7170 tok/s 12.06s/it [38:11<29:00:31] mem=34.07GB |
| full_cpt step 4200/12852 ( 32.7%) nll=3.2208 ppl=25.05 grad=0.875 lr=1.58e-05 7177 tok/s 12.03s/it [40:07<28:55:05] mem=34.07GB |
| am nll=3.9894 ppl=54.02 bpb=1.4029 |
| ta nll=3.9611 ppl=52.52 bpb=0.5753 |
| kk nll=2.7642 ppl=15.87 bpb=0.9308 |
| tl nll=2.6622 ppl=14.33 bpb=1.2631 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 4200 01:27 (87.4s) |
| full_cpt step 4210/12852 ( 32.8%) nll=3.3693 ppl=29.06 grad=0.859 lr=1.58e-05 6948 tok/s 12.43s/it [43:30<29:49:59] mem=34.07GB |
| full_cpt step 4220/12852 ( 32.8%) nll=3.3484 ppl=28.46 grad=0.879 lr=1.58e-05 6967 tok/s 12.39s/it [45:26<29:42:27] mem=34.07GB |
| full_cpt step 4230/12852 ( 32.9%) nll=3.3073 ppl=27.31 grad=0.828 lr=1.58e-05 6998 tok/s 12.36s/it [47:22<29:35:35] mem=34.07GB |
| full_cpt step 4240/12852 ( 33.0%) nll=3.2099 ppl=24.78 grad=0.906 lr=1.57e-05 7014 tok/s 12.32s/it [49:18<29:29:00] mem=34.07GB |
| full_cpt step 4250/12852 ( 33.1%) nll=3.4038 ppl=30.08 grad=0.879 lr=1.57e-05 7036 tok/s 12.30s/it [51:14<29:22:49] mem=34.07GB |
| full_cpt step 4260/12852 ( 33.1%) nll=3.4426 ppl=31.27 grad=0.867 lr=1.57e-05 7052 tok/s 12.27s/it [53:10<29:16:57] mem=34.07GB |
| full_cpt step 4270/12852 ( 33.2%) nll=3.2424 ppl=25.60 grad=0.910 lr=1.57e-05 7065 tok/s 12.24s/it [55:06<29:11:20] mem=34.07GB |
| full_cpt step 4280/12852 ( 33.3%) nll=3.3108 ppl=27.41 grad=0.863 lr=1.57e-05 7080 tok/s 12.22s/it [57:02<29:06:01] mem=34.07GB |
| full_cpt step 4290/12852 ( 33.4%) nll=3.2816 ppl=26.62 grad=0.875 lr=1.57e-05 7101 tok/s 12.20s/it [58:58<29:01:02] mem=34.07GB |
| full_cpt step 4300/12852 ( 33.5%) nll=3.3467 ppl=28.41 grad=0.910 lr=1.56e-05 7112 tok/s 12.18s/it [1:00:54<28:56:07] mem=34.07GB |
| am nll=3.9879 ppl=53.94 bpb=1.4024 |
| ta nll=3.9598 ppl=52.44 bpb=0.5751 |
| kk nll=2.7632 ppl=15.85 bpb=0.9305 |
| tl nll=2.6621 ppl=14.33 bpb=1.2631 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 4300 01:28 (87.6s) |
| full_cpt step 4310/12852 ( 33.5%) nll=3.3887 ppl=29.63 grad=0.891 lr=1.56e-05 6960 tok/s 12.44s/it [1:04:18<29:31:36] mem=34.07GB |
| full_cpt step 4320/12852 ( 33.6%) nll=3.2807 ppl=26.59 grad=0.871 lr=1.56e-05 6975 tok/s 12.42s/it [1:06:14<29:25:46] mem=34.07GB |
| full_cpt step 4330/12852 ( 33.7%) nll=3.4546 ppl=31.65 grad=0.883 lr=1.56e-05 6992 tok/s 12.39s/it [1:08:10<29:20:12] mem=34.07GB |
| full_cpt step 4340/12852 ( 33.8%) nll=3.3319 ppl=27.99 grad=0.957 lr=1.56e-05 7007 tok/s 12.37s/it [1:10:06<29:14:51] mem=34.07GB |
| full_cpt step 4350/12852 ( 33.8%) nll=3.3390 ppl=28.19 grad=0.855 lr=1.55e-05 7014 tok/s 12.35s/it [1:12:01<29:09:35] mem=34.07GB |
| full_cpt step 4360/12852 ( 33.9%) nll=3.3022 ppl=27.17 grad=0.922 lr=1.55e-05 7025 tok/s 12.33s/it [1:13:57<29:04:33] mem=34.07GB |
| full_cpt step 4370/12852 ( 34.0%) nll=3.2387 ppl=25.50 grad=0.855 lr=1.55e-05 7039 tok/s 12.31s/it [1:15:53<28:59:44] mem=34.07GB |
| full_cpt step 4380/12852 ( 34.1%) nll=3.2537 ppl=25.88 grad=0.863 lr=1.55e-05 7050 tok/s 12.29s/it [1:17:49<28:55:04] mem=34.07GB |
| full_cpt step 4390/12852 ( 34.2%) nll=3.3776 ppl=29.30 grad=0.879 lr=1.55e-05 7063 tok/s 12.27s/it [1:19:46<28:50:33] mem=34.07GB |
| full_cpt step 4400/12852 ( 34.2%) nll=3.2460 ppl=25.69 grad=0.902 lr=1.54e-05 7074 tok/s 12.25s/it [1:21:42<28:46:09] mem=34.07GB |
| am nll=3.9865 ppl=53.86 bpb=1.4018 |
| ta nll=3.9582 ppl=52.36 bpb=0.5749 |
| kk nll=2.7626 ppl=15.84 bpb=0.9303 |
| tl nll=2.6618 ppl=14.32 bpb=1.2630 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 4400 01:27 (87.3s) |
| full_cpt step 4410/12852 ( 34.3%) nll=3.4419 ppl=31.25 grad=0.945 lr=1.54e-05 6964 tok/s 12.45s/it [1:25:05<29:11:49] mem=34.07GB |
| full_cpt step 4420/12852 ( 34.4%) nll=3.3409 ppl=28.24 grad=0.840 lr=1.54e-05 6976 tok/s 12.43s/it [1:27:01<29:06:54] mem=34.07GB |
| full_cpt step 4430/12852 ( 34.5%) nll=3.4550 ppl=31.66 grad=0.891 lr=1.54e-05 6986 tok/s 12.41s/it [1:28:57<29:02:05] mem=34.07GB |
| full_cpt step 4440/12852 ( 34.5%) nll=3.2877 ppl=26.78 grad=0.965 lr=1.54e-05 6995 tok/s 12.39s/it [1:30:53<28:57:25] mem=34.07GB |
| full_cpt step 4450/12852 ( 34.6%) nll=3.2810 ppl=26.60 grad=0.883 lr=1.53e-05 7010 tok/s 12.38s/it [1:32:49<28:52:57] mem=34.07GB |
| full_cpt step 4460/12852 ( 34.7%) nll=3.1600 ppl=23.57 grad=0.891 lr=1.53e-05 7021 tok/s 12.36s/it [1:34:45<28:48:34] mem=34.07GB |
| full_cpt step 4470/12852 ( 34.8%) nll=3.3485 ppl=28.46 grad=0.879 lr=1.53e-05 7029 tok/s 12.34s/it [1:36:41<28:44:14] mem=34.07GB |
| full_cpt step 4480/12852 ( 34.9%) nll=3.3101 ppl=27.39 grad=0.891 lr=1.53e-05 7039 tok/s 12.33s/it [1:38:37<28:40:02] mem=34.07GB |
| full_cpt step 4490/12852 ( 34.9%) nll=3.2536 ppl=25.88 grad=0.883 lr=1.53e-05 7048 tok/s 12.31s/it [1:40:33<28:35:55] mem=34.07GB |
| full_cpt step 4500/12852 ( 35.0%) nll=3.2665 ppl=26.22 grad=0.848 lr=1.52e-05 7060 tok/s 12.30s/it [1:42:29<28:31:56] mem=34.07GB |
| am nll=3.9849 ppl=53.78 bpb=1.4013 |
| ta nll=3.9568 ppl=52.29 bpb=0.5747 |
| kk nll=2.7613 ppl=15.82 bpb=0.9298 |
| tl nll=2.6612 ppl=14.31 bpb=1.2627 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 4500 01:29 (88.7s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.67s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.67s/it] |
| removing old checkpoint checkpoint-3500 |
| kept last 2 resume checkpoints: checkpoint-4000 checkpoint-4500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-4500 at step 4500 next_row=576000 |
| full_cpt step 4510/12852 ( 35.1%) nll=3.2223 ppl=25.09 grad=0.840 lr=1.52e-05 6898 tok/s 12.59s/it [1:47:01<29:10:19] mem=34.07GB |
| full_cpt step 4520/12852 ( 35.2%) nll=3.3475 ppl=28.43 grad=0.879 lr=1.52e-05 6911 tok/s 12.57s/it [1:48:57<29:05:39] mem=34.07GB |
| full_cpt step 4530/12852 ( 35.2%) nll=3.2950 ppl=26.98 grad=0.922 lr=1.52e-05 6921 tok/s 12.55s/it [1:50:53<29:01:03] mem=34.07GB |
| full_cpt step 4540/12852 ( 35.3%) nll=3.3350 ppl=28.08 grad=0.840 lr=1.52e-05 6933 tok/s 12.54s/it [1:52:49<28:56:33] mem=34.07GB |
| full_cpt step 4550/12852 ( 35.4%) nll=3.4140 ppl=30.39 grad=0.855 lr=1.51e-05 6943 tok/s 12.52s/it [1:54:45<28:52:08] mem=34.07GB |
| full_cpt step 4560/12852 ( 35.5%) nll=3.3828 ppl=29.45 grad=0.871 lr=1.51e-05 6952 tok/s 12.50s/it [1:56:41<28:47:47] mem=34.07GB |
| full_cpt step 4570/12852 ( 35.6%) nll=3.3417 ppl=28.27 grad=0.910 lr=1.51e-05 6961 tok/s 12.49s/it [1:58:37<28:43:33] mem=34.07GB |
| full_cpt step 4580/12852 ( 35.6%) nll=3.3181 ppl=27.61 grad=0.820 lr=1.51e-05 6971 tok/s 12.47s/it [2:00:33<28:39:23] mem=34.07GB |
| full_cpt step 4590/12852 ( 35.7%) nll=3.2864 ppl=26.75 grad=0.895 lr=1.51e-05 6980 tok/s 12.46s/it [2:02:30<28:35:19] mem=34.07GB |
| full_cpt step 4600/12852 ( 35.8%) nll=3.3052 ppl=27.25 grad=0.863 lr=1.50e-05 6988 tok/s 12.44s/it [2:04:26<28:31:18] mem=34.07GB |
| am nll=3.9841 ppl=53.74 bpb=1.4010 |
| ta nll=3.9546 ppl=52.17 bpb=0.5744 |
| kk nll=2.7603 ppl=15.81 bpb=0.9295 |
| tl nll=2.6611 ppl=14.31 bpb=1.2627 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 4600 01:29 (88.6s) |
| full_cpt step 4610/12852 ( 35.9%) nll=3.3785 ppl=29.33 grad=0.895 lr=1.50e-05 6913 tok/s 12.57s/it [2:07:50<28:47:17] mem=34.07GB |
| full_cpt step 4620/12852 ( 35.9%) nll=3.2436 ppl=25.63 grad=0.855 lr=1.50e-05 6921 tok/s 12.56s/it [2:09:46<28:43:05] mem=34.07GB |
| full_cpt step 4630/12852 ( 36.0%) nll=3.3338 ppl=28.04 grad=0.840 lr=1.50e-05 6928 tok/s 12.54s/it [2:11:43<28:38:56] mem=34.07GB |
| full_cpt step 4640/12852 ( 36.1%) nll=3.3128 ppl=27.46 grad=0.832 lr=1.50e-05 6937 tok/s 12.53s/it [2:13:39<28:34:54] mem=34.07GB |
| full_cpt step 4650/12852 ( 36.2%) nll=3.1552 ppl=23.46 grad=0.906 lr=1.49e-05 6945 tok/s 12.52s/it [2:15:35<28:30:54] mem=34.07GB |
| full_cpt step 4660/12852 ( 36.3%) nll=3.2342 ppl=25.38 grad=0.867 lr=1.49e-05 6955 tok/s 12.50s/it [2:17:32<28:26:59] mem=34.07GB |
| full_cpt step 4670/12852 ( 36.3%) nll=3.4462 ppl=31.38 grad=0.855 lr=1.49e-05 6961 tok/s 12.49s/it [2:19:28<28:23:04] mem=34.07GB |
| full_cpt step 4680/12852 ( 36.4%) nll=3.3392 ppl=28.20 grad=0.871 lr=1.49e-05 6968 tok/s 12.48s/it [2:21:24<28:19:14] mem=34.07GB |
| full_cpt step 4690/12852 ( 36.5%) nll=3.2266 ppl=25.19 grad=1.125 lr=1.49e-05 6975 tok/s 12.46s/it [2:23:20<28:15:29] mem=34.07GB |
| full_cpt step 4700/12852 ( 36.6%) nll=3.2964 ppl=27.01 grad=0.887 lr=1.48e-05 6981 tok/s 12.45s/it [2:25:16<28:11:45] mem=34.07GB |
| am nll=3.9829 ppl=53.67 bpb=1.4006 |
| ta nll=3.9546 ppl=52.17 bpb=0.5744 |
| kk nll=2.7592 ppl=15.79 bpb=0.9291 |
| tl nll=2.6609 ppl=14.31 bpb=1.2626 |
| en nll=2.8179 ppl=16.74 bpb=0.8774 |
| eval full_cpt step 4700 01:30 (90.0s) |
| full_cpt step 4710/12852 ( 36.6%) nll=3.1793 ppl=24.03 grad=0.906 lr=1.48e-05 6917 tok/s 12.57s/it [2:28:42<28:25:17] mem=34.07GB |
| full_cpt step 4720/12852 ( 36.7%) nll=3.1946 ppl=24.40 grad=0.875 lr=1.48e-05 6923 tok/s 12.55s/it [2:30:38<28:21:22] mem=34.07GB |
| full_cpt step 4730/12852 ( 36.8%) nll=3.4535 ppl=31.61 grad=0.918 lr=1.48e-05 6932 tok/s 12.54s/it [2:32:34<28:17:32] mem=34.07GB |
| full_cpt step 4740/12852 ( 36.9%) nll=3.3056 ppl=27.27 grad=0.949 lr=1.48e-05 6939 tok/s 12.53s/it [2:34:31<28:13:45] mem=34.07GB |
| full_cpt step 4750/12852 ( 37.0%) nll=3.2337 ppl=25.37 grad=0.828 lr=1.47e-05 6946 tok/s 12.52s/it [2:36:27<28:10:02] mem=34.07GB |
| full_cpt step 4760/12852 ( 37.0%) nll=3.3127 ppl=27.46 grad=0.824 lr=1.47e-05 6952 tok/s 12.50s/it [2:38:23<28:06:22] mem=34.07GB |
| full_cpt step 4770/12852 ( 37.1%) nll=3.2047 ppl=24.65 grad=0.961 lr=1.47e-05 6956 tok/s 12.49s/it [2:40:19<28:02:44] mem=34.07GB |
| full_cpt step 4780/12852 ( 37.2%) nll=3.2856 ppl=26.72 grad=0.898 lr=1.47e-05 6963 tok/s 12.48s/it [2:42:15<27:59:10] mem=34.07GB |
| full_cpt step 4790/12852 ( 37.3%) nll=3.3269 ppl=27.85 grad=0.875 lr=1.47e-05 6967 tok/s 12.47s/it [2:44:12<27:55:36] mem=34.07GB |
| full_cpt step 4800/12852 ( 37.3%) nll=3.2832 ppl=26.66 grad=0.871 lr=1.46e-05 6974 tok/s 12.46s/it [2:46:08<27:52:05] mem=34.07GB |
| am nll=3.9824 ppl=53.64 bpb=1.4004 |
| ta nll=3.9526 ppl=52.07 bpb=0.5741 |
| kk nll=2.7588 ppl=15.78 bpb=0.9290 |
| tl nll=2.6609 ppl=14.31 bpb=1.2626 |
| en nll=2.8178 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 4800 01:29 (88.9s) |
| full_cpt step 4810/12852 ( 37.4%) nll=3.2861 ppl=26.74 grad=0.949 lr=1.46e-05 6919 tok/s 12.56s/it [2:49:33<28:03:18] mem=34.07GB |
| full_cpt step 4820/12852 ( 37.5%) nll=3.2658 ppl=26.20 grad=0.859 lr=1.46e-05 6925 tok/s 12.55s/it [2:51:29<27:59:39] mem=34.07GB |
| full_cpt step 4830/12852 ( 37.6%) nll=3.4103 ppl=30.27 grad=0.883 lr=1.46e-05 6932 tok/s 12.54s/it [2:53:25<27:56:05] mem=34.07GB |
| full_cpt step 4840/12852 ( 37.7%) nll=3.2648 ppl=26.18 grad=0.883 lr=1.46e-05 6936 tok/s 12.53s/it [2:55:21<27:52:31] mem=34.07GB |
| full_cpt step 4850/12852 ( 37.7%) nll=3.4691 ppl=32.11 grad=0.824 lr=1.45e-05 6943 tok/s 12.51s/it [2:57:17<27:49:02] mem=34.07GB |
| full_cpt step 4860/12852 ( 37.8%) nll=3.2791 ppl=26.55 grad=0.855 lr=1.45e-05 6949 tok/s 12.50s/it [2:59:13<27:45:32] mem=34.07GB |
| full_cpt step 4870/12852 ( 37.9%) nll=3.4440 ppl=31.31 grad=0.910 lr=1.45e-05 6955 tok/s 12.49s/it [3:01:10<27:42:06] mem=34.07GB |
| full_cpt step 4880/12852 ( 38.0%) nll=3.4100 ppl=30.26 grad=0.887 lr=1.45e-05 6960 tok/s 12.48s/it [3:03:06<27:38:40] mem=34.07GB |
| full_cpt step 4890/12852 ( 38.0%) nll=3.3922 ppl=29.73 grad=0.887 lr=1.45e-05 6964 tok/s 12.47s/it [3:05:02<27:35:16] mem=34.07GB |
| full_cpt step 4900/12852 ( 38.1%) nll=3.4386 ppl=31.14 grad=0.922 lr=1.44e-05 6968 tok/s 12.46s/it [3:06:58<27:31:54] mem=34.07GB |
| am nll=3.9806 ppl=53.55 bpb=1.3998 |
| ta nll=3.9517 ppl=52.02 bpb=0.5740 |
| kk nll=2.7578 ppl=15.77 bpb=0.9287 |
| tl nll=2.6609 ppl=14.31 bpb=1.2625 |
| en nll=2.8172 ppl=16.73 bpb=0.8771 |
| eval full_cpt step 4900 01:27 (87.4s) |
| full_cpt step 4910/12852 ( 38.2%) nll=3.2977 ppl=27.05 grad=0.875 lr=1.44e-05 6918 tok/s 12.55s/it [3:10:21<27:41:16] mem=34.07GB |
| full_cpt step 4920/12852 ( 38.3%) nll=3.3037 ppl=27.21 grad=0.852 lr=1.44e-05 6923 tok/s 12.54s/it [3:12:17<27:37:49] mem=34.07GB |
| full_cpt step 4930/12852 ( 38.4%) nll=3.3750 ppl=29.22 grad=0.871 lr=1.44e-05 6929 tok/s 12.53s/it [3:14:13<27:34:25] mem=34.07GB |
| full_cpt step 4940/12852 ( 38.4%) nll=3.2844 ppl=26.69 grad=0.836 lr=1.44e-05 6934 tok/s 12.52s/it [3:16:09<27:31:01] mem=34.07GB |
| full_cpt step 4950/12852 ( 38.5%) nll=3.3602 ppl=28.80 grad=0.914 lr=1.43e-05 6937 tok/s 12.51s/it [3:18:05<27:27:39] mem=34.07GB |
| full_cpt step 4960/12852 ( 38.6%) nll=3.2561 ppl=25.95 grad=0.859 lr=1.43e-05 6942 tok/s 12.50s/it [3:20:01<27:24:20] mem=34.07GB |
| full_cpt step 4970/12852 ( 38.7%) nll=3.3281 ppl=27.88 grad=0.855 lr=1.43e-05 6949 tok/s 12.49s/it [3:21:57<27:21:03] mem=34.07GB |
| full_cpt step 4980/12852 ( 38.7%) nll=3.2947 ppl=26.97 grad=0.922 lr=1.43e-05 6952 tok/s 12.48s/it [3:23:53<27:17:47] mem=34.07GB |
| full_cpt step 4990/12852 ( 38.8%) nll=3.3853 ppl=29.53 grad=0.938 lr=1.43e-05 6956 tok/s 12.47s/it [3:25:50<27:14:33] mem=34.07GB |
| full_cpt step 5000/12852 ( 38.9%) nll=3.4153 ppl=30.43 grad=0.852 lr=1.42e-05 6962 tok/s 12.47s/it [3:27:46<27:11:20] mem=34.07GB |
| am nll=3.9793 ppl=53.48 bpb=1.3993 |
| ta nll=3.9515 ppl=52.01 bpb=0.5739 |
| kk nll=2.7571 ppl=15.75 bpb=0.9284 |
| tl nll=2.6608 ppl=14.31 bpb=1.2625 |
| en nll=2.8178 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 5000 01:27 (87.2s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.29s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.29s/it] |
| removing old checkpoint checkpoint-4000 |
| kept last 2 resume checkpoints: checkpoint-4500 checkpoint-5000 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-5000 at step 5000 next_row=640000 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:04<00:00, 4.21s/it]
Writing model shards: 100%|##########| 1/1 [00:04<00:00, 4.22s/it] |
| wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-5000 (weights+expanded tokenizer, no optimizer) |
| full_cpt step 5010/12852 ( 39.0%) nll=3.3397 ppl=28.21 grad=0.883 lr=1.42e-05 6865 tok/s 12.64s/it [3:32:48<27:32:13] mem=34.07GB |
| full_cpt step 5020/12852 ( 39.1%) nll=3.3932 ppl=29.76 grad=0.934 lr=1.42e-05 6870 tok/s 12.63s/it [3:34:44<27:28:48] mem=34.07GB |
| full_cpt step 5030/12852 ( 39.1%) nll=3.2298 ppl=25.28 grad=0.875 lr=1.42e-05 6876 tok/s 12.62s/it [3:36:40<27:25:24] mem=34.07GB |
| full_cpt step 5040/12852 ( 39.2%) nll=3.4147 ppl=30.41 grad=0.883 lr=1.42e-05 6883 tok/s 12.61s/it [3:38:36<27:22:03] mem=34.07GB |
| full_cpt step 5050/12852 ( 39.3%) nll=3.1858 ppl=24.19 grad=0.836 lr=1.41e-05 6889 tok/s 12.60s/it [3:40:32<27:18:42] mem=34.07GB |
| full_cpt step 5060/12852 ( 39.4%) nll=3.2193 ppl=25.01 grad=0.840 lr=1.41e-05 6894 tok/s 12.59s/it [3:42:28<27:15:23] mem=34.07GB |
| full_cpt step 5070/12852 ( 39.4%) nll=3.2701 ppl=26.32 grad=0.902 lr=1.41e-05 6897 tok/s 12.58s/it [3:44:24<27:12:04] mem=34.07GB |
| full_cpt step 5080/12852 ( 39.5%) nll=3.3942 ppl=29.79 grad=0.941 lr=1.41e-05 6903 tok/s 12.57s/it [3:46:20<27:08:49] mem=34.07GB |
| full_cpt step 5090/12852 ( 39.6%) nll=3.2857 ppl=26.73 grad=0.863 lr=1.40e-05 6910 tok/s 12.57s/it [3:48:17<27:05:35] mem=34.07GB |
| full_cpt step 5100/12852 ( 39.7%) nll=3.3416 ppl=28.27 grad=1.359 lr=1.40e-05 6914 tok/s 12.56s/it [3:50:13<27:02:22] mem=34.07GB |
| am nll=3.9787 ppl=53.45 bpb=1.3991 |
| ta nll=3.9504 ppl=51.96 bpb=0.5738 |
| kk nll=2.7566 ppl=15.75 bpb=0.9283 |
| tl nll=2.6609 ppl=14.31 bpb=1.2625 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 5100 01:28 (87.5s) |
| full_cpt step 5110/12852 ( 39.8%) nll=3.4203 ppl=30.58 grad=0.895 lr=1.40e-05 6873 tok/s 12.63s/it [3:53:36<27:09:18] mem=34.07GB |
| full_cpt step 5120/12852 ( 39.8%) nll=3.3089 ppl=27.35 grad=0.824 lr=1.40e-05 6878 tok/s 12.62s/it [3:55:32<27:06:01] mem=34.07GB |
| full_cpt step 5130/12852 ( 39.9%) nll=3.3672 ppl=29.00 grad=0.898 lr=1.40e-05 6884 tok/s 12.61s/it [3:57:28<27:02:46] mem=34.07GB |
| full_cpt step 5140/12852 ( 40.0%) nll=3.2232 ppl=25.11 grad=0.852 lr=1.39e-05 6889 tok/s 12.60s/it [3:59:24<26:59:32] mem=34.07GB |
| full_cpt step 5150/12852 ( 40.1%) nll=3.3114 ppl=27.42 grad=0.852 lr=1.39e-05 6893 tok/s 12.59s/it [4:01:20<26:56:20] mem=34.07GB |
| full_cpt step 5160/12852 ( 40.1%) nll=3.3468 ppl=28.41 grad=0.863 lr=1.39e-05 6897 tok/s 12.58s/it [4:03:16<26:53:08] mem=34.07GB |
| full_cpt step 5170/12852 ( 40.2%) nll=3.3987 ppl=29.93 grad=0.852 lr=1.39e-05 6901 tok/s 12.57s/it [4:05:12<26:49:58] mem=34.07GB |
| full_cpt step 5180/12852 ( 40.3%) nll=3.3124 ppl=27.45 grad=0.859 lr=1.39e-05 6907 tok/s 12.57s/it [4:07:08<26:46:50] mem=34.07GB |
| full_cpt step 5190/12852 ( 40.4%) nll=3.3764 ppl=29.27 grad=0.875 lr=1.38e-05 6911 tok/s 12.56s/it [4:09:05<26:43:43] mem=34.07GB |
| full_cpt step 5200/12852 ( 40.5%) nll=3.3074 ppl=27.31 grad=0.887 lr=1.38e-05 6914 tok/s 12.55s/it [4:11:00<26:40:36] mem=34.07GB |
| am nll=3.9780 ppl=53.41 bpb=1.3989 |
| ta nll=3.9498 ppl=51.93 bpb=0.5737 |
| kk nll=2.7559 ppl=15.73 bpb=0.9280 |
| tl nll=2.6605 ppl=14.30 bpb=1.2624 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 5200 01:28 (87.8s) |
| full_cpt step 5210/12852 ( 40.5%) nll=3.2817 ppl=26.62 grad=0.855 lr=1.38e-05 6877 tok/s 12.61s/it [4:14:24<26:46:43] mem=34.07GB |
| full_cpt step 5220/12852 ( 40.6%) nll=3.1863 ppl=24.20 grad=0.844 lr=1.38e-05 6881 tok/s 12.61s/it [4:16:20<26:43:34] mem=34.07GB |
| full_cpt step 5230/12852 ( 40.7%) nll=3.4377 ppl=31.11 grad=0.867 lr=1.38e-05 6885 tok/s 12.60s/it [4:18:16<26:40:25] mem=34.07GB |
| full_cpt step 5240/12852 ( 40.8%) nll=3.3745 ppl=29.21 grad=0.852 lr=1.37e-05 6890 tok/s 12.59s/it [4:20:12<26:37:19] mem=34.07GB |
| full_cpt step 5250/12852 ( 40.8%) nll=3.3498 ppl=28.50 grad=0.887 lr=1.37e-05 6895 tok/s 12.58s/it [4:22:08<26:34:14] mem=34.07GB |
| full_cpt step 5260/12852 ( 40.9%) nll=3.2113 ppl=24.81 grad=0.879 lr=1.37e-05 6899 tok/s 12.58s/it [4:24:05<26:31:10] mem=34.07GB |
| full_cpt step 5270/12852 ( 41.0%) nll=3.4013 ppl=30.00 grad=0.863 lr=1.37e-05 6904 tok/s 12.57s/it [4:26:01<26:28:07] mem=34.07GB |
| full_cpt step 5280/12852 ( 41.1%) nll=3.3208 ppl=27.68 grad=0.863 lr=1.36e-05 6909 tok/s 12.56s/it [4:27:57<26:25:05] mem=34.07GB |
| full_cpt step 5290/12852 ( 41.2%) nll=3.3225 ppl=27.73 grad=0.910 lr=1.36e-05 6912 tok/s 12.55s/it [4:29:53<26:22:03] mem=34.07GB |
| full_cpt step 5300/12852 ( 41.2%) nll=3.3065 ppl=27.29 grad=0.879 lr=1.36e-05 6916 tok/s 12.55s/it [4:31:49<26:19:03] mem=34.07GB |
| am nll=3.9772 ppl=53.37 bpb=1.3986 |
| ta nll=3.9495 ppl=51.91 bpb=0.5736 |
| kk nll=2.7554 ppl=15.73 bpb=0.9279 |
| tl nll=2.6603 ppl=14.30 bpb=1.2623 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 5300 01:28 (87.9s) |
| full_cpt step 5310/12852 ( 41.3%) nll=3.3144 ppl=27.51 grad=0.875 lr=1.36e-05 6883 tok/s 12.61s/it [4:35:13<26:24:29] mem=34.07GB |
| full_cpt step 5320/12852 ( 41.4%) nll=3.3730 ppl=29.16 grad=0.867 lr=1.36e-05 6887 tok/s 12.60s/it [4:37:09<26:21:26] mem=34.07GB |
| full_cpt step 5330/12852 ( 41.5%) nll=3.3118 ppl=27.44 grad=0.883 lr=1.35e-05 6891 tok/s 12.59s/it [4:39:05<26:18:24] mem=34.07GB |
| full_cpt step 5340/12852 ( 41.5%) nll=3.2990 ppl=27.09 grad=0.828 lr=1.35e-05 6895 tok/s 12.58s/it [4:41:01<26:15:23] mem=34.07GB |
| full_cpt step 5350/12852 ( 41.6%) nll=3.3497 ppl=28.49 grad=0.848 lr=1.35e-05 6900 tok/s 12.58s/it [4:42:57<26:12:24] mem=34.07GB |
| full_cpt step 5360/12852 ( 41.7%) nll=3.3774 ppl=29.30 grad=0.906 lr=1.35e-05 6903 tok/s 12.57s/it [4:44:53<26:09:24] mem=34.07GB |
| full_cpt step 5370/12852 ( 41.8%) nll=3.3132 ppl=27.47 grad=0.910 lr=1.35e-05 6906 tok/s 12.56s/it [4:46:49<26:06:26] mem=34.07GB |
| full_cpt step 5380/12852 ( 41.9%) nll=3.2585 ppl=26.01 grad=0.906 lr=1.34e-05 6909 tok/s 12.55s/it [4:48:45<26:03:28] mem=34.07GB |
| full_cpt step 5390/12852 ( 41.9%) nll=3.2479 ppl=25.74 grad=0.863 lr=1.34e-05 6913 tok/s 12.55s/it [4:50:41<26:00:31] mem=34.07GB |
| full_cpt step 5400/12852 ( 42.0%) nll=3.2583 ppl=26.01 grad=0.852 lr=1.34e-05 6916 tok/s 12.54s/it [4:52:37<25:57:36] mem=34.07GB |
| am nll=3.9765 ppl=53.33 bpb=1.3984 |
| ta nll=3.9485 ppl=51.86 bpb=0.5735 |
| kk nll=2.7550 ppl=15.72 bpb=0.9277 |
| tl nll=2.6604 ppl=14.30 bpb=1.2623 |
| en nll=2.8173 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 5400 01:28 (88.0s) |
| full_cpt step 5410/12852 ( 42.1%) nll=3.2110 ppl=24.80 grad=0.902 lr=1.34e-05 6887 tok/s 12.60s/it [4:56:02<26:02:26] mem=34.07GB |
| full_cpt step 5420/12852 ( 42.2%) nll=3.2498 ppl=25.78 grad=0.816 lr=1.34e-05 6890 tok/s 12.59s/it [4:57:58<25:59:28] mem=34.07GB |
| full_cpt step 5430/12852 ( 42.3%) nll=3.2114 ppl=24.81 grad=0.844 lr=1.33e-05 6895 tok/s 12.58s/it [4:59:54<25:56:31] mem=34.07GB |
| full_cpt step 5440/12852 ( 42.3%) nll=3.2947 ppl=26.97 grad=0.918 lr=1.33e-05 6898 tok/s 12.58s/it [5:01:50<25:53:35] mem=34.07GB |
| full_cpt step 5450/12852 ( 42.4%) nll=3.3932 ppl=29.76 grad=0.902 lr=1.33e-05 6901 tok/s 12.57s/it [5:03:46<25:50:39] mem=34.07GB |
| full_cpt step 5460/12852 ( 42.5%) nll=3.1929 ppl=24.36 grad=0.918 lr=1.33e-05 6904 tok/s 12.56s/it [5:05:42<25:47:44] mem=34.07GB |
| full_cpt step 5470/12852 ( 42.6%) nll=3.1820 ppl=24.10 grad=0.828 lr=1.32e-05 6907 tok/s 12.56s/it [5:07:38<25:44:51] mem=34.07GB |
| full_cpt step 5480/12852 ( 42.6%) nll=3.2314 ppl=25.31 grad=0.840 lr=1.32e-05 6911 tok/s 12.55s/it [5:09:34<25:41:58] mem=34.07GB |
| full_cpt step 5490/12852 ( 42.7%) nll=3.3164 ppl=27.56 grad=0.895 lr=1.32e-05 6915 tok/s 12.54s/it [5:11:30<25:39:06] mem=34.07GB |
| full_cpt step 5500/12852 ( 42.8%) nll=3.2990 ppl=27.09 grad=0.926 lr=1.32e-05 6917 tok/s 12.54s/it [5:13:26<25:36:14] mem=34.07GB |
| am nll=3.9765 ppl=53.33 bpb=1.3983 |
| ta nll=3.9486 ppl=51.86 bpb=0.5735 |
| kk nll=2.7543 ppl=15.71 bpb=0.9275 |
| tl nll=2.6605 ppl=14.30 bpb=1.2623 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 5500 01:27 (87.5s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.30s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.30s/it] |
| removing old checkpoint checkpoint-4500 |
| kept last 2 resume checkpoints: checkpoint-5000 checkpoint-5500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-5500 at step 5500 next_row=704000 |
| full_cpt step 5510/12852 ( 42.9%) nll=3.1887 ppl=24.26 grad=0.879 lr=1.32e-05 6860 tok/s 12.64s/it [5:18:07<25:46:47] mem=34.07GB |
| full_cpt step 5520/12852 ( 43.0%) nll=3.3598 ppl=28.78 grad=0.910 lr=1.31e-05 6865 tok/s 12.63s/it [5:20:04<25:43:52] mem=34.07GB |
| full_cpt step 5530/12852 ( 43.0%) nll=3.2223 ppl=25.09 grad=0.852 lr=1.31e-05 6868 tok/s 12.63s/it [5:22:00<25:40:56] mem=34.07GB |
| full_cpt step 5540/12852 ( 43.1%) nll=3.2783 ppl=26.53 grad=0.828 lr=1.31e-05 6872 tok/s 12.62s/it [5:23:56<25:38:02] mem=34.07GB |
| full_cpt step 5550/12852 ( 43.2%) nll=3.4123 ppl=30.34 grad=0.914 lr=1.31e-05 6875 tok/s 12.61s/it [5:25:52<25:35:08] mem=34.07GB |
| full_cpt step 5560/12852 ( 43.3%) nll=3.4105 ppl=30.28 grad=0.910 lr=1.30e-05 6879 tok/s 12.61s/it [5:27:48<25:32:15] mem=34.07GB |
| full_cpt step 5570/12852 ( 43.3%) nll=3.3011 ppl=27.14 grad=0.855 lr=1.30e-05 6882 tok/s 12.60s/it [5:29:44<25:29:22] mem=34.07GB |
| full_cpt step 5580/12852 ( 43.4%) nll=3.3152 ppl=27.53 grad=0.953 lr=1.30e-05 6885 tok/s 12.59s/it [5:31:40<25:26:30] mem=34.07GB |
| full_cpt step 5590/12852 ( 43.5%) nll=3.3125 ppl=27.45 grad=0.863 lr=1.30e-05 6889 tok/s 12.59s/it [5:33:36<25:23:40] mem=34.07GB |
| full_cpt step 5600/12852 ( 43.6%) nll=3.2374 ppl=25.47 grad=0.852 lr=1.30e-05 6891 tok/s 12.58s/it [5:35:32<25:20:49] mem=34.07GB |
| am nll=3.9753 ppl=53.27 bpb=1.3979 |
| ta nll=3.9479 ppl=51.83 bpb=0.5734 |
| kk nll=2.7537 ppl=15.70 bpb=0.9273 |
| tl nll=2.6601 ppl=14.30 bpb=1.2622 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 5600 01:28 (88.3s) |
| full_cpt step 5610/12852 ( 43.7%) nll=3.2760 ppl=26.47 grad=0.855 lr=1.29e-05 6864 tok/s 12.63s/it [5:38:56<25:24:36] mem=34.07GB |
| full_cpt step 5620/12852 ( 43.7%) nll=3.2072 ppl=24.71 grad=0.879 lr=1.29e-05 6868 tok/s 12.62s/it [5:40:52<25:21:44] mem=34.07GB |
| full_cpt step 5630/12852 ( 43.8%) nll=3.4398 ppl=31.18 grad=0.918 lr=1.29e-05 6872 tok/s 12.62s/it [5:42:49<25:18:53] mem=34.07GB |
| full_cpt step 5640/12852 ( 43.9%) nll=3.3349 ppl=28.08 grad=0.902 lr=1.29e-05 6874 tok/s 12.61s/it [5:44:45<25:16:02] mem=34.07GB |
| full_cpt step 5650/12852 ( 44.0%) nll=3.3972 ppl=29.88 grad=0.910 lr=1.29e-05 6878 tok/s 12.61s/it [5:46:41<25:13:12] mem=34.07GB |
| full_cpt step 5660/12852 ( 44.0%) nll=3.2712 ppl=26.34 grad=0.926 lr=1.28e-05 6882 tok/s 12.60s/it [5:48:37<25:10:23] mem=34.07GB |
| full_cpt step 5670/12852 ( 44.1%) nll=3.3413 ppl=28.26 grad=0.816 lr=1.28e-05 6884 tok/s 12.59s/it [5:50:33<25:07:34] mem=34.07GB |
| full_cpt step 5680/12852 ( 44.2%) nll=3.3810 ppl=29.40 grad=0.875 lr=1.28e-05 6888 tok/s 12.59s/it [5:52:29<25:04:46] mem=34.07GB |
| full_cpt step 5690/12852 ( 44.3%) nll=3.4133 ppl=30.37 grad=0.875 lr=1.28e-05 6891 tok/s 12.58s/it [5:54:25<25:01:59] mem=34.07GB |
| full_cpt step 5700/12852 ( 44.4%) nll=3.3733 ppl=29.17 grad=0.844 lr=1.27e-05 6893 tok/s 12.58s/it [5:56:21<24:59:12] mem=34.07GB |
| am nll=3.9746 ppl=53.23 bpb=1.3977 |
| ta nll=3.9477 ppl=51.82 bpb=0.5734 |
| kk nll=2.7535 ppl=15.70 bpb=0.9272 |
| tl nll=2.6602 ppl=14.30 bpb=1.2622 |
| en nll=2.8173 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 5700 01:28 (87.8s) |
| full_cpt step 5710/12852 ( 44.4%) nll=3.3942 ppl=29.79 grad=0.895 lr=1.27e-05 6868 tok/s 12.62s/it [5:59:45<25:02:32] mem=34.07GB |
| full_cpt step 5720/12852 ( 44.5%) nll=3.3362 ppl=28.11 grad=0.863 lr=1.27e-05 6872 tok/s 12.62s/it [6:01:41<24:59:44] mem=34.07GB |
| full_cpt step 5730/12852 ( 44.6%) nll=3.2601 ppl=26.05 grad=0.918 lr=1.27e-05 6875 tok/s 12.61s/it [6:03:37<24:56:56] mem=34.07GB |
| full_cpt step 5740/12852 ( 44.7%) nll=3.3397 ppl=28.21 grad=0.926 lr=1.27e-05 6879 tok/s 12.61s/it [6:05:33<24:54:09] mem=34.07GB |
| full_cpt step 5750/12852 ( 44.7%) nll=3.3901 ppl=29.67 grad=0.883 lr=1.26e-05 6882 tok/s 12.60s/it [6:07:29<24:51:23] mem=34.07GB |
| full_cpt step 5760/12852 ( 44.8%) nll=3.3085 ppl=27.35 grad=0.879 lr=1.26e-05 6884 tok/s 12.59s/it [6:09:25<24:48:37] mem=34.07GB |
| full_cpt step 5770/12852 ( 44.9%) nll=3.3332 ppl=28.03 grad=0.879 lr=1.26e-05 6887 tok/s 12.59s/it [6:11:22<24:45:51] mem=34.07GB |
| full_cpt step 5780/12852 ( 45.0%) nll=3.2783 ppl=26.53 grad=0.891 lr=1.26e-05 6890 tok/s 12.58s/it [6:13:18<24:43:06] mem=34.07GB |
| full_cpt step 5790/12852 ( 45.1%) nll=3.4794 ppl=32.44 grad=0.906 lr=1.25e-05 6892 tok/s 12.58s/it [6:15:14<24:40:21] mem=34.07GB |
| full_cpt step 5800/12852 ( 45.1%) nll=3.4331 ppl=30.97 grad=0.906 lr=1.25e-05 6895 tok/s 12.57s/it [6:17:10<24:37:38] mem=34.07GB |
| am nll=3.9737 ppl=53.18 bpb=1.3974 |
| ta nll=3.9469 ppl=51.77 bpb=0.5733 |
| kk nll=2.7530 ppl=15.69 bpb=0.9270 |
| tl nll=2.6600 ppl=14.30 bpb=1.2621 |
| en nll=2.8173 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 5800 01:33 (93.2s) |
| full_cpt step 5810/12852 ( 45.2%) nll=3.3261 ppl=27.83 grad=0.867 lr=1.25e-05 6870 tok/s 12.62s/it [6:20:39<24:40:57] mem=34.07GB |
| full_cpt step 5820/12852 ( 45.3%) nll=3.2357 ppl=25.43 grad=0.855 lr=1.25e-05 6873 tok/s 12.61s/it [6:22:35<24:38:12] mem=34.07GB |
| full_cpt step 5830/12852 ( 45.4%) nll=3.3501 ppl=28.51 grad=0.867 lr=1.25e-05 6876 tok/s 12.61s/it [6:24:31<24:35:28] mem=34.07GB |
| full_cpt step 5840/12852 ( 45.4%) nll=3.2598 ppl=26.04 grad=0.832 lr=1.24e-05 6879 tok/s 12.60s/it [6:26:28<24:32:46] mem=34.07GB |
| full_cpt step 5850/12852 ( 45.5%) nll=3.4198 ppl=30.56 grad=0.891 lr=1.24e-05 6882 tok/s 12.60s/it [6:28:24<24:30:04] mem=34.07GB |
| full_cpt step 5860/12852 ( 45.6%) nll=3.4026 ppl=30.04 grad=0.883 lr=1.24e-05 6885 tok/s 12.59s/it [6:30:21<24:27:21] mem=34.07GB |
| full_cpt step 5870/12852 ( 45.7%) nll=3.4840 ppl=32.59 grad=0.898 lr=1.24e-05 6888 tok/s 12.59s/it [6:32:17<24:24:39] mem=34.07GB |
| full_cpt step 5880/12852 ( 45.8%) nll=3.3249 ppl=27.80 grad=0.895 lr=1.24e-05 6890 tok/s 12.58s/it [6:34:13<24:21:57] mem=34.07GB |
| full_cpt step 5890/12852 ( 45.8%) nll=3.3780 ppl=29.31 grad=0.906 lr=1.23e-05 6894 tok/s 12.58s/it [6:36:10<24:19:17] mem=34.07GB |
| full_cpt step 5900/12852 ( 45.9%) nll=3.2935 ppl=26.94 grad=0.906 lr=1.23e-05 6897 tok/s 12.57s/it [6:38:06<24:16:37] mem=34.07GB |
| am nll=3.9737 ppl=53.18 bpb=1.3973 |
| ta nll=3.9468 ppl=51.77 bpb=0.5732 |
| kk nll=2.7527 ppl=15.68 bpb=0.9270 |
| tl nll=2.6601 ppl=14.30 bpb=1.2622 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 5900 01:41 (100.7s) |
| full_cpt step 5910/12852 ( 46.0%) nll=3.3723 ppl=29.15 grad=0.898 lr=1.23e-05 6871 tok/s 12.62s/it [6:41:43<24:20:02] mem=34.07GB |
| full_cpt step 5920/12852 ( 46.1%) nll=3.3066 ppl=27.29 grad=0.836 lr=1.23e-05 6873 tok/s 12.61s/it [6:43:39<24:17:20] mem=34.07GB |
| full_cpt step 5930/12852 ( 46.1%) nll=3.3472 ppl=28.42 grad=0.902 lr=1.22e-05 6877 tok/s 12.61s/it [6:45:35<24:14:38] mem=34.07GB |
| full_cpt step 5940/12852 ( 46.2%) nll=3.3615 ppl=28.83 grad=0.871 lr=1.22e-05 6879 tok/s 12.60s/it [6:47:31<24:11:56] mem=34.07GB |
| full_cpt step 5950/12852 ( 46.3%) nll=3.2696 ppl=26.30 grad=0.926 lr=1.22e-05 6881 tok/s 12.60s/it [6:49:27<24:09:14] mem=34.07GB |
| full_cpt step 5960/12852 ( 46.4%) nll=3.2585 ppl=26.01 grad=0.938 lr=1.22e-05 6884 tok/s 12.59s/it [6:51:23<24:06:33] mem=34.07GB |
| full_cpt step 5970/12852 ( 46.5%) nll=3.2932 ppl=26.93 grad=0.922 lr=1.22e-05 6886 tok/s 12.59s/it [6:53:19<24:03:52] mem=34.07GB |
| full_cpt step 5980/12852 ( 46.5%) nll=3.2102 ppl=24.78 grad=0.871 lr=1.21e-05 6888 tok/s 12.58s/it [6:55:15<24:01:12] mem=34.07GB |
| full_cpt step 5990/12852 ( 46.6%) nll=3.3069 ppl=27.30 grad=0.891 lr=1.21e-05 6891 tok/s 12.58s/it [6:57:11<23:58:32] mem=34.07GB |
| full_cpt step 6000/12852 ( 46.7%) nll=3.3209 ppl=27.69 grad=0.859 lr=1.21e-05 6894 tok/s 12.57s/it [6:59:07<23:55:53] mem=34.07GB |
| am nll=3.9733 ppl=53.16 bpb=1.3972 |
| ta nll=3.9462 ppl=51.74 bpb=0.5732 |
| kk nll=2.7526 ppl=15.68 bpb=0.9269 |
| tl nll=2.6599 ppl=14.30 bpb=1.2621 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 6000 01:50 (109.9s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.26s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.26s/it] |
| removing old checkpoint checkpoint-5000 |
| kept last 2 resume checkpoints: checkpoint-5500 checkpoint-6000 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-6000 at step 6000 next_row=768000 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:03<00:00, 3.12s/it]
Writing model shards: 100%|##########| 1/1 [00:03<00:00, 3.12s/it] |
| wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-6000 (weights+expanded tokenizer, no optimizer) |
| full_cpt step 6010/12852 ( 46.8%) nll=3.2879 ppl=26.79 grad=0.863 lr=1.21e-05 6841 tok/s 12.67s/it [7:04:27<24:04:48] mem=34.07GB |
| full_cpt step 6020/12852 ( 46.8%) nll=3.2718 ppl=26.36 grad=0.934 lr=1.20e-05 6844 tok/s 12.66s/it [7:06:23<24:02:05] mem=34.07GB |
| full_cpt step 6030/12852 ( 46.9%) nll=3.3071 ppl=27.31 grad=0.891 lr=1.20e-05 6847 tok/s 12.66s/it [7:08:19<23:59:23] mem=34.07GB |
| full_cpt step 6040/12852 ( 47.0%) nll=3.3184 ppl=27.62 grad=0.871 lr=1.20e-05 6851 tok/s 12.65s/it [7:10:15<23:56:42] mem=34.07GB |
| full_cpt step 6050/12852 ( 47.1%) nll=3.3727 ppl=29.16 grad=0.879 lr=1.20e-05 6854 tok/s 12.65s/it [7:12:11<23:54:01] mem=34.07GB |
| full_cpt step 6060/12852 ( 47.2%) nll=3.2168 ppl=24.95 grad=0.914 lr=1.20e-05 6857 tok/s 12.64s/it [7:14:07<23:51:19] mem=34.07GB |
| full_cpt step 6070/12852 ( 47.2%) nll=3.2441 ppl=25.64 grad=0.902 lr=1.19e-05 6859 tok/s 12.64s/it [7:16:03<23:48:39] mem=34.07GB |
| full_cpt step 6080/12852 ( 47.3%) nll=3.2790 ppl=26.55 grad=0.859 lr=1.19e-05 6862 tok/s 12.63s/it [7:17:59<23:45:58] mem=34.07GB |
| full_cpt step 6090/12852 ( 47.4%) nll=3.1845 ppl=24.15 grad=0.832 lr=1.19e-05 6864 tok/s 12.63s/it [7:19:55<23:43:18] mem=34.07GB |
| full_cpt step 6100/12852 ( 47.5%) nll=3.2546 ppl=25.91 grad=0.883 lr=1.19e-05 6867 tok/s 12.62s/it [7:21:51<23:40:39] mem=34.07GB |
| am nll=3.9731 ppl=53.15 bpb=1.3972 |
| ta nll=3.9463 ppl=51.74 bpb=0.5732 |
| kk nll=2.7518 ppl=15.67 bpb=0.9266 |
| tl nll=2.6602 ppl=14.30 bpb=1.2622 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 6100 01:28 (87.5s) |
| full_cpt step 6110/12852 ( 47.5%) nll=3.3447 ppl=28.35 grad=0.805 lr=1.18e-05 6848 tok/s 12.66s/it [7:25:14<23:42:39] mem=34.07GB |
| full_cpt step 6120/12852 ( 47.6%) nll=3.2332 ppl=25.36 grad=0.883 lr=1.18e-05 6850 tok/s 12.66s/it [7:27:10<23:39:59] mem=34.07GB |
| full_cpt step 6130/12852 ( 47.7%) nll=3.4210 ppl=30.60 grad=0.898 lr=1.18e-05 6853 tok/s 12.65s/it [7:29:06<23:37:19] mem=34.07GB |
| full_cpt step 6140/12852 ( 47.8%) nll=3.3735 ppl=29.18 grad=0.863 lr=1.18e-05 6855 tok/s 12.65s/it [7:31:02<23:34:39] mem=34.07GB |
| full_cpt step 6150/12852 ( 47.9%) nll=3.3067 ppl=27.30 grad=0.848 lr=1.18e-05 6858 tok/s 12.64s/it [7:32:58<23:32:00] mem=34.07GB |
| full_cpt step 6160/12852 ( 47.9%) nll=3.2277 ppl=25.22 grad=0.902 lr=1.17e-05 6859 tok/s 12.64s/it [7:34:54<23:29:21] mem=34.07GB |
| full_cpt step 6170/12852 ( 48.0%) nll=3.2197 ppl=25.02 grad=0.906 lr=1.17e-05 6862 tok/s 12.63s/it [7:36:50<23:26:42] mem=34.07GB |
| full_cpt step 6180/12852 ( 48.1%) nll=3.2985 ppl=27.07 grad=0.953 lr=1.17e-05 6865 tok/s 12.63s/it [7:38:46<23:24:04] mem=34.07GB |
| full_cpt step 6190/12852 ( 48.2%) nll=3.3410 ppl=28.25 grad=0.879 lr=1.17e-05 6868 tok/s 12.62s/it [7:40:42<23:21:27] mem=34.07GB |
| full_cpt step 6200/12852 ( 48.2%) nll=3.2986 ppl=27.08 grad=0.879 lr=1.16e-05 6870 tok/s 12.62s/it [7:42:38<23:18:50] mem=34.07GB |
| am nll=3.9731 ppl=53.15 bpb=1.3971 |
| ta nll=3.9458 ppl=51.72 bpb=0.5731 |
| kk nll=2.7520 ppl=15.67 bpb=0.9267 |
| tl nll=2.6601 ppl=14.30 bpb=1.2621 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 6200 01:27 (87.3s) |
| full_cpt step 6210/12852 ( 48.3%) nll=3.2270 ppl=25.20 grad=0.820 lr=1.16e-05 6851 tok/s 12.65s/it [7:46:01<23:20:35] mem=34.07GB |
| full_cpt step 6220/12852 ( 48.4%) nll=3.2329 ppl=25.35 grad=0.859 lr=1.16e-05 6853 tok/s 12.65s/it [7:47:57<23:17:57] mem=34.07GB |
| full_cpt step 6230/12852 ( 48.5%) nll=3.3161 ppl=27.55 grad=0.832 lr=1.16e-05 6855 tok/s 12.64s/it [7:49:53<23:15:19] mem=34.07GB |
| full_cpt step 6240/12852 ( 48.6%) nll=3.3070 ppl=27.30 grad=0.840 lr=1.16e-05 6858 tok/s 12.64s/it [7:51:49<23:12:42] mem=34.07GB |
| full_cpt step 6250/12852 ( 48.6%) nll=3.4116 ppl=30.31 grad=0.820 lr=1.15e-05 6860 tok/s 12.63s/it [7:53:45<23:10:05] mem=34.07GB |
| full_cpt step 6260/12852 ( 48.7%) nll=3.3744 ppl=29.21 grad=0.871 lr=1.15e-05 6862 tok/s 12.63s/it [7:55:41<23:07:28] mem=34.07GB |
| full_cpt step 6270/12852 ( 48.8%) nll=3.3055 ppl=27.26 grad=0.863 lr=1.15e-05 6865 tok/s 12.62s/it [7:57:37<23:04:52] mem=34.07GB |
| full_cpt step 6280/12852 ( 48.9%) nll=3.3385 ppl=28.18 grad=0.871 lr=1.15e-05 6867 tok/s 12.62s/it [7:59:33<23:02:16] mem=34.07GB |
| full_cpt step 6290/12852 ( 48.9%) nll=3.3552 ppl=28.65 grad=0.934 lr=1.14e-05 6869 tok/s 12.62s/it [8:01:29<22:59:40] mem=34.07GB |
| full_cpt step 6300/12852 ( 49.0%) nll=3.2521 ppl=25.85 grad=0.891 lr=1.14e-05 6873 tok/s 12.61s/it [8:03:25<22:57:05] mem=34.07GB |
| am nll=3.9728 ppl=53.14 bpb=1.3971 |
| ta nll=3.9459 ppl=51.72 bpb=0.5731 |
| kk nll=2.7516 ppl=15.67 bpb=0.9266 |
| tl nll=2.6598 ppl=14.29 bpb=1.2620 |
| en nll=2.8178 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 6300 01:27 (86.8s) |
| full_cpt step 6310/12852 ( 49.1%) nll=3.2305 ppl=25.29 grad=0.816 lr=1.14e-05 6854 tok/s 12.64s/it [8:06:47<22:58:36] mem=34.07GB |
| full_cpt step 6320/12852 ( 49.2%) nll=3.2737 ppl=26.41 grad=0.887 lr=1.14e-05 6857 tok/s 12.64s/it [8:08:43<22:56:00] mem=34.07GB |
| full_cpt step 6330/12852 ( 49.3%) nll=3.3588 ppl=28.75 grad=0.910 lr=1.14e-05 6860 tok/s 12.63s/it [8:10:39<22:53:24] mem=34.07GB |
| full_cpt step 6340/12852 ( 49.3%) nll=3.3837 ppl=29.48 grad=0.902 lr=1.13e-05 6861 tok/s 12.63s/it [8:12:35<22:50:49] mem=34.07GB |
| full_cpt step 6350/12852 ( 49.4%) nll=3.2549 ppl=25.92 grad=0.863 lr=1.13e-05 6864 tok/s 12.63s/it [8:14:31<22:48:14] mem=34.07GB |
| full_cpt step 6360/12852 ( 49.5%) nll=3.4559 ppl=31.69 grad=0.812 lr=1.13e-05 6867 tok/s 12.62s/it [8:16:27<22:45:39] mem=34.07GB |
| full_cpt step 6370/12852 ( 49.6%) nll=3.2707 ppl=26.33 grad=0.855 lr=1.13e-05 6869 tok/s 12.62s/it [8:18:23<22:43:05] mem=34.07GB |
| full_cpt step 6380/12852 ( 49.6%) nll=3.4500 ppl=31.50 grad=0.922 lr=1.12e-05 6871 tok/s 12.61s/it [8:20:19<22:40:31] mem=34.07GB |
| full_cpt step 6390/12852 ( 49.7%) nll=3.3094 ppl=27.37 grad=0.859 lr=1.12e-05 6873 tok/s 12.61s/it [8:22:15<22:37:57] mem=34.07GB |
| full_cpt step 6400/12852 ( 49.8%) nll=3.3178 ppl=27.60 grad=0.934 lr=1.12e-05 6875 tok/s 12.60s/it [8:24:10<22:35:23] mem=34.07GB |
| am nll=3.9727 ppl=53.13 bpb=1.3970 |
| ta nll=3.9448 ppl=51.67 bpb=0.5730 |
| kk nll=2.7512 ppl=15.66 bpb=0.9264 |
| tl nll=2.6599 ppl=14.29 bpb=1.2621 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 6400 01:27 (87.5s) |
| full_cpt step 6410/12852 ( 49.9%) nll=3.2677 ppl=26.25 grad=0.879 lr=1.12e-05 6859 tok/s 12.64s/it [8:27:34<22:36:44] mem=34.07GB |
| full_cpt step 6420/12852 ( 50.0%) nll=3.3061 ppl=27.28 grad=0.910 lr=1.12e-05 6861 tok/s 12.63s/it [8:29:30<22:34:10] mem=34.07GB |
| full_cpt step 6430/12852 ( 50.0%) nll=3.3911 ppl=29.70 grad=0.914 lr=1.11e-05 6863 tok/s 12.63s/it [8:31:26<22:31:36] mem=34.07GB |
| full_cpt step 6440/12852 ( 50.1%) nll=3.3627 ppl=28.87 grad=0.832 lr=1.11e-05 6866 tok/s 12.62s/it [8:33:22<22:29:03] mem=34.07GB |
| full_cpt step 6450/12852 ( 50.2%) nll=3.2594 ppl=26.04 grad=0.828 lr=1.11e-05 6868 tok/s 12.62s/it [8:35:18<22:26:30] mem=34.07GB |
| full_cpt step 6460/12852 ( 50.3%) nll=3.2492 ppl=25.77 grad=0.961 lr=1.11e-05 6871 tok/s 12.62s/it [8:37:14<22:23:57] mem=34.07GB |
| full_cpt step 6470/12852 ( 50.3%) nll=3.2841 ppl=26.69 grad=0.875 lr=1.10e-05 6873 tok/s 12.61s/it [8:39:10<22:21:25] mem=34.07GB |
| full_cpt step 6480/12852 ( 50.4%) nll=3.3185 ppl=27.62 grad=0.871 lr=1.10e-05 6875 tok/s 12.61s/it [8:41:06<22:18:52] mem=34.07GB |
| full_cpt step 6490/12852 ( 50.5%) nll=3.2944 ppl=26.96 grad=0.855 lr=1.10e-05 6877 tok/s 12.60s/it [8:43:02<22:16:20] mem=34.07GB |
| full_cpt step 6500/12852 ( 50.6%) nll=3.4613 ppl=31.86 grad=0.824 lr=1.10e-05 6879 tok/s 12.60s/it [8:44:58<22:13:49] mem=34.07GB |
| am nll=3.9721 ppl=53.10 bpb=1.3968 |
| ta nll=3.9449 ppl=51.67 bpb=0.5730 |
| kk nll=2.7511 ppl=15.66 bpb=0.9264 |
| tl nll=2.6599 ppl=14.29 bpb=1.2621 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 6500 01:27 (86.9s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.64s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.65s/it] |
| removing old checkpoint checkpoint-5500 |
| kept last 2 resume checkpoints: checkpoint-6000 checkpoint-6500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-6500 at step 6500 next_row=832000 |
| full_cpt step 6510/12852 ( 50.7%) nll=3.3151 ppl=27.53 grad=0.906 lr=1.10e-05 6846 tok/s 12.66s/it [8:49:36<22:18:08] mem=34.07GB |
| full_cpt step 6520/12852 ( 50.7%) nll=3.3557 ppl=28.66 grad=0.859 lr=1.09e-05 6847 tok/s 12.66s/it [8:51:32<22:15:35] mem=34.07GB |
| full_cpt step 6530/12852 ( 50.8%) nll=3.4649 ppl=31.97 grad=0.887 lr=1.09e-05 6849 tok/s 12.65s/it [8:53:28<22:13:02] mem=34.07GB |
| full_cpt step 6540/12852 ( 50.9%) nll=3.3843 ppl=29.50 grad=0.914 lr=1.09e-05 6852 tok/s 12.65s/it [8:55:24<22:10:30] mem=34.07GB |
| full_cpt step 6550/12852 ( 51.0%) nll=3.0930 ppl=22.04 grad=0.922 lr=1.09e-05 6854 tok/s 12.64s/it [8:57:20<22:07:57] mem=34.07GB |
| full_cpt step 6560/12852 ( 51.0%) nll=3.3314 ppl=27.98 grad=0.824 lr=1.08e-05 6856 tok/s 12.64s/it [8:59:16<22:05:25] mem=34.07GB |
| full_cpt step 6570/12852 ( 51.1%) nll=3.3110 ppl=27.41 grad=0.891 lr=1.08e-05 6859 tok/s 12.64s/it [9:01:12<22:02:53] mem=34.07GB |
| full_cpt step 6580/12852 ( 51.2%) nll=3.3909 ppl=29.69 grad=0.871 lr=1.08e-05 6861 tok/s 12.63s/it [9:03:08<22:00:22] mem=34.07GB |
| full_cpt step 6590/12852 ( 51.3%) nll=3.3570 ppl=28.70 grad=0.934 lr=1.08e-05 6862 tok/s 12.63s/it [9:05:04<21:57:50] mem=34.07GB |
| full_cpt step 6600/12852 ( 51.4%) nll=3.3418 ppl=28.27 grad=0.887 lr=1.08e-05 6865 tok/s 12.62s/it [9:07:00<21:55:19] mem=34.07GB |
| am nll=3.9720 ppl=53.09 bpb=1.3968 |
| ta nll=3.9447 ppl=51.66 bpb=0.5729 |
| kk nll=2.7510 ppl=15.66 bpb=0.9264 |
| tl nll=2.6599 ppl=14.29 bpb=1.2621 |
| en nll=2.8175 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 6600 01:27 (86.9s) |
| full_cpt step 6610/12852 ( 51.4%) nll=3.4597 ppl=31.81 grad=0.906 lr=1.07e-05 6849 tok/s 12.65s/it [9:10:22<21:56:16] mem=34.07GB |
| full_cpt step 6620/12852 ( 51.5%) nll=3.2058 ppl=24.67 grad=0.844 lr=1.07e-05 6851 tok/s 12.65s/it [9:12:18<21:53:44] mem=34.07GB |
| full_cpt step 6630/12852 ( 51.6%) nll=3.2620 ppl=26.10 grad=0.844 lr=1.07e-05 6854 tok/s 12.64s/it [9:14:14<21:51:12] mem=34.07GB |
| full_cpt step 6640/12852 ( 51.7%) nll=3.3069 ppl=27.30 grad=0.895 lr=1.07e-05 6857 tok/s 12.64s/it [9:16:10<21:48:42] mem=34.07GB |
| full_cpt step 6650/12852 ( 51.7%) nll=3.2561 ppl=25.95 grad=0.816 lr=1.06e-05 6859 tok/s 12.64s/it [9:18:06<21:46:11] mem=34.07GB |
| full_cpt step 6660/12852 ( 51.8%) nll=3.1758 ppl=23.95 grad=0.840 lr=1.06e-05 6861 tok/s 12.63s/it [9:20:02<21:43:40] mem=34.07GB |
| full_cpt step 6670/12852 ( 51.9%) nll=3.3205 ppl=27.67 grad=0.863 lr=1.06e-05 6864 tok/s 12.63s/it [9:21:59<21:41:10] mem=34.07GB |
| full_cpt step 6680/12852 ( 52.0%) nll=3.3634 ppl=28.89 grad=0.883 lr=1.06e-05 6866 tok/s 12.62s/it [9:23:54<21:38:40] mem=34.07GB |
| full_cpt step 6690/12852 ( 52.1%) nll=3.3397 ppl=28.21 grad=0.871 lr=1.06e-05 6868 tok/s 12.62s/it [9:25:50<21:36:11] mem=34.07GB |
| full_cpt step 6700/12852 ( 52.1%) nll=3.3583 ppl=28.74 grad=0.930 lr=1.05e-05 6870 tok/s 12.62s/it [9:27:46<21:33:41] mem=34.07GB |
| am nll=3.9715 ppl=53.06 bpb=1.3966 |
| ta nll=3.9444 ppl=51.64 bpb=0.5729 |
| kk nll=2.7507 ppl=15.65 bpb=0.9263 |
| tl nll=2.6597 ppl=14.29 bpb=1.2620 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 6700 01:29 (89.0s) |
| full_cpt step 6710/12852 ( 52.2%) nll=3.2345 ppl=25.39 grad=0.934 lr=1.05e-05 6854 tok/s 12.65s/it [9:31:11<21:34:33] mem=34.07GB |
| full_cpt step 6720/12852 ( 52.3%) nll=3.2897 ppl=26.84 grad=0.883 lr=1.05e-05 6856 tok/s 12.64s/it [9:33:07<21:32:03] mem=34.07GB |
| full_cpt step 6730/12852 ( 52.4%) nll=3.4466 ppl=31.39 grad=0.859 lr=1.05e-05 6859 tok/s 12.64s/it [9:35:03<21:29:34] mem=34.07GB |
| full_cpt step 6740/12852 ( 52.4%) nll=3.2301 ppl=25.28 grad=0.863 lr=1.04e-05 6861 tok/s 12.63s/it [9:37:00<21:27:04] mem=34.07GB |
| full_cpt step 6750/12852 ( 52.5%) nll=3.3340 ppl=28.05 grad=0.906 lr=1.04e-05 6864 tok/s 12.63s/it [9:38:56<21:24:35] mem=34.07GB |
| full_cpt step 6760/12852 ( 52.6%) nll=3.4084 ppl=30.22 grad=0.824 lr=1.04e-05 6866 tok/s 12.63s/it [9:40:52<21:22:06] mem=34.07GB |
| full_cpt step 6770/12852 ( 52.7%) nll=3.3090 ppl=27.36 grad=0.875 lr=1.04e-05 6868 tok/s 12.62s/it [9:42:48<21:19:37] mem=34.07GB |
| full_cpt step 6780/12852 ( 52.8%) nll=3.2311 ppl=25.31 grad=0.883 lr=1.04e-05 6870 tok/s 12.62s/it [9:44:44<21:17:09] mem=34.07GB |
| full_cpt step 6790/12852 ( 52.8%) nll=3.3513 ppl=28.54 grad=0.891 lr=1.03e-05 6873 tok/s 12.62s/it [9:46:40<21:14:41] mem=34.07GB |
| full_cpt step 6800/12852 ( 52.9%) nll=3.3295 ppl=27.93 grad=0.891 lr=1.03e-05 6874 tok/s 12.61s/it [9:48:36<21:12:13] mem=34.07GB |
| am nll=3.9713 ppl=53.05 bpb=1.3965 |
| ta nll=3.9446 ppl=51.66 bpb=0.5729 |
| kk nll=2.7507 ppl=15.65 bpb=0.9263 |
| tl nll=2.6599 ppl=14.29 bpb=1.2620 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 6800 01:30 (89.6s) |
| full_cpt step 6810/12852 ( 53.0%) nll=3.3408 ppl=28.24 grad=0.859 lr=1.03e-05 6860 tok/s 12.64s/it [9:52:01<21:12:57] mem=34.07GB |
| full_cpt step 6820/12852 ( 53.1%) nll=3.1812 ppl=24.08 grad=0.871 lr=1.03e-05 6862 tok/s 12.64s/it [9:53:57<21:10:29] mem=34.07GB |
| full_cpt step 6830/12852 ( 53.1%) nll=3.3399 ppl=28.22 grad=0.871 lr=1.02e-05 6864 tok/s 12.63s/it [9:55:53<21:08:00] mem=34.07GB |
| full_cpt step 6840/12852 ( 53.2%) nll=3.3186 ppl=27.62 grad=0.859 lr=1.02e-05 6866 tok/s 12.63s/it [9:57:49<21:05:32] mem=34.07GB |
| full_cpt step 6850/12852 ( 53.3%) nll=3.4017 ppl=30.02 grad=0.906 lr=1.02e-05 6868 tok/s 12.63s/it [9:59:45<21:03:04] mem=34.07GB |
| full_cpt step 6860/12852 ( 53.4%) nll=3.2830 ppl=26.66 grad=0.910 lr=1.02e-05 6870 tok/s 12.62s/it [10:01:41<21:00:36] mem=34.07GB |
| full_cpt step 6870/12852 ( 53.5%) nll=3.3422 ppl=28.28 grad=0.840 lr=1.02e-05 6872 tok/s 12.62s/it [10:03:37<20:58:09] mem=34.07GB |
| full_cpt step 6880/12852 ( 53.5%) nll=3.3762 ppl=29.26 grad=0.898 lr=1.01e-05 6874 tok/s 12.62s/it [10:05:33<20:55:41] mem=34.07GB |
| full_cpt step 6890/12852 ( 53.6%) nll=3.3573 ppl=28.71 grad=0.863 lr=1.01e-05 6875 tok/s 12.61s/it [10:07:29<20:53:14] mem=34.07GB |
| full_cpt step 6900/12852 ( 53.7%) nll=3.2813 ppl=26.61 grad=0.938 lr=1.01e-05 6876 tok/s 12.61s/it [10:09:25<20:50:47] mem=34.07GB |
| am nll=3.9714 ppl=53.06 bpb=1.3965 |
| ta nll=3.9448 ppl=51.66 bpb=0.5730 |
| kk nll=2.7506 ppl=15.65 bpb=0.9262 |
| tl nll=2.6599 ppl=14.29 bpb=1.2620 |
| en nll=2.8173 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 6900 01:29 (89.1s) |
| full_cpt step 6910/12852 ( 53.8%) nll=3.2731 ppl=26.39 grad=0.863 lr=1.01e-05 6861 tok/s 12.64s/it [10:12:50<20:51:22] mem=34.07GB |
| full_cpt step 6920/12852 ( 53.8%) nll=3.2718 ppl=26.36 grad=0.926 lr=1.00e-05 6863 tok/s 12.63s/it [10:14:46<20:48:54] mem=34.07GB |
| full_cpt step 6930/12852 ( 53.9%) nll=3.3421 ppl=28.28 grad=0.863 lr=1.00e-05 6865 tok/s 12.63s/it [10:16:42<20:46:27] mem=34.07GB |
| full_cpt step 6940/12852 ( 54.0%) nll=3.3435 ppl=28.32 grad=0.996 lr=1.00e-05 6866 tok/s 12.63s/it [10:18:38<20:44:00] mem=34.07GB |
| full_cpt step 6950/12852 ( 54.1%) nll=3.3876 ppl=29.59 grad=0.879 lr=9.98e-06 6868 tok/s 12.62s/it [10:20:34<20:41:33] mem=34.07GB |
| full_cpt step 6960/12852 ( 54.2%) nll=3.3107 ppl=27.41 grad=0.887 lr=9.96e-06 6870 tok/s 12.62s/it [10:22:30<20:39:07] mem=34.07GB |
| full_cpt step 6970/12852 ( 54.2%) nll=3.3488 ppl=28.47 grad=0.871 lr=9.94e-06 6872 tok/s 12.61s/it [10:24:26<20:36:41] mem=34.07GB |
| full_cpt step 6980/12852 ( 54.3%) nll=3.2260 ppl=25.18 grad=0.906 lr=9.91e-06 6874 tok/s 12.61s/it [10:26:22<20:34:15] mem=34.07GB |
| full_cpt step 6990/12852 ( 54.4%) nll=3.3591 ppl=28.76 grad=0.852 lr=9.89e-06 6876 tok/s 12.61s/it [10:28:18<20:31:49] mem=34.07GB |
| full_cpt step 7000/12852 ( 54.5%) nll=3.2314 ppl=25.32 grad=0.867 lr=9.87e-06 6878 tok/s 12.60s/it [10:30:14<20:29:23] mem=34.07GB |
| am nll=3.9713 ppl=53.05 bpb=1.3965 |
| ta nll=3.9443 ppl=51.64 bpb=0.5729 |
| kk nll=2.7505 ppl=15.65 bpb=0.9262 |
| tl nll=2.6598 ppl=14.29 bpb=1.2620 |
| en nll=2.8175 ppl=16.74 bpb=0.8772 |
| eval full_cpt step 7000 01:29 (89.4s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.37s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.37s/it] |
| removing old checkpoint checkpoint-6000 |
| kept last 2 resume checkpoints: checkpoint-6500 checkpoint-7000 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-7000 at step 7000 next_row=896000 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:03<00:00, 3.39s/it]
Writing model shards: 100%|##########| 1/1 [00:03<00:00, 3.39s/it] |
| wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-7000 (weights+expanded tokenizer, no optimizer) |
| full_cpt step 7010/12852 ( 54.5%) nll=3.2987 ppl=27.08 grad=0.863 lr=9.85e-06 6845 tok/s 12.66s/it [10:35:20<20:33:06] mem=34.07GB |
| full_cpt step 7020/12852 ( 54.6%) nll=3.5208 ppl=33.81 grad=0.875 lr=9.83e-06 6847 tok/s 12.66s/it [10:37:16<20:30:39] mem=34.07GB |
| full_cpt step 7030/12852 ( 54.7%) nll=3.2201 ppl=25.03 grad=0.895 lr=9.80e-06 6849 tok/s 12.66s/it [10:39:12<20:28:12] mem=34.07GB |
| full_cpt step 7040/12852 ( 54.8%) nll=3.3088 ppl=27.35 grad=0.828 lr=9.78e-06 6850 tok/s 12.65s/it [10:41:08<20:25:45] mem=34.07GB |
| full_cpt step 7050/12852 ( 54.9%) nll=3.2482 ppl=25.74 grad=0.891 lr=9.76e-06 6852 tok/s 12.65s/it [10:43:04<20:23:19] mem=34.07GB |
| full_cpt step 7060/12852 ( 54.9%) nll=3.2973 ppl=27.04 grad=0.812 lr=9.74e-06 6854 tok/s 12.65s/it [10:45:00<20:20:53] mem=34.07GB |
| full_cpt step 7070/12852 ( 55.0%) nll=3.3779 ppl=29.31 grad=0.910 lr=9.72e-06 6856 tok/s 12.64s/it [10:46:57<20:18:27] mem=34.07GB |
| full_cpt step 7080/12852 ( 55.1%) nll=3.3133 ppl=27.48 grad=0.918 lr=9.69e-06 6858 tok/s 12.64s/it [10:48:53<20:16:01] mem=34.07GB |
| full_cpt step 7090/12852 ( 55.2%) nll=3.2208 ppl=25.05 grad=0.887 lr=9.67e-06 6859 tok/s 12.64s/it [10:50:48<20:13:34] mem=34.07GB |
| full_cpt step 7100/12852 ( 55.2%) nll=3.3418 ppl=28.27 grad=0.867 lr=9.65e-06 6862 tok/s 12.63s/it [10:52:44<20:11:09] mem=34.07GB |
| am nll=3.9711 ppl=53.04 bpb=1.3964 |
| ta nll=3.9446 ppl=51.66 bpb=0.5729 |
| kk nll=2.7507 ppl=15.65 bpb=0.9263 |
| tl nll=2.6596 ppl=14.29 bpb=1.2619 |
| en nll=2.8172 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 7100 01:28 (88.5s) |
| full_cpt step 7110/12852 ( 55.3%) nll=3.2652 ppl=26.19 grad=0.816 lr=9.63e-06 6849 tok/s 12.66s/it [10:56:09<20:11:27] mem=34.07GB |
| full_cpt step 7120/12852 ( 55.4%) nll=3.2273 ppl=25.21 grad=0.840 lr=9.61e-06 6851 tok/s 12.66s/it [10:58:05<20:09:01] mem=34.07GB |
| full_cpt step 7130/12852 ( 55.5%) nll=3.3603 ppl=28.80 grad=0.910 lr=9.58e-06 6852 tok/s 12.65s/it [11:00:01<20:06:35] mem=34.07GB |
| full_cpt step 7140/12852 ( 55.6%) nll=3.3376 ppl=28.15 grad=0.879 lr=9.56e-06 6854 tok/s 12.65s/it [11:01:57<20:04:10] mem=34.07GB |
| full_cpt step 7150/12852 ( 55.6%) nll=3.2913 ppl=26.88 grad=0.879 lr=9.54e-06 6856 tok/s 12.65s/it [11:03:53<20:01:45] mem=34.07GB |
| full_cpt step 7160/12852 ( 55.7%) nll=3.2475 ppl=25.73 grad=0.836 lr=9.52e-06 6859 tok/s 12.64s/it [11:05:49<19:59:19] mem=34.07GB |
| full_cpt step 7170/12852 ( 55.8%) nll=3.4687 ppl=32.10 grad=0.867 lr=9.50e-06 6860 tok/s 12.64s/it [11:07:45<19:56:54] mem=34.07GB |
| full_cpt step 7180/12852 ( 55.9%) nll=3.2642 ppl=26.16 grad=0.863 lr=9.47e-06 6862 tok/s 12.64s/it [11:09:41<19:54:29] mem=34.07GB |
| full_cpt step 7190/12852 ( 55.9%) nll=3.2787 ppl=26.54 grad=0.867 lr=9.45e-06 6863 tok/s 12.63s/it [11:11:37<19:52:04] mem=34.07GB |
| full_cpt step 7200/12852 ( 56.0%) nll=3.3572 ppl=28.71 grad=0.871 lr=9.43e-06 6865 tok/s 12.63s/it [11:13:33<19:49:39] mem=34.07GB |
| am nll=3.9711 ppl=53.04 bpb=1.3964 |
| ta nll=3.9439 ppl=51.62 bpb=0.5728 |
| kk nll=2.7501 ppl=15.64 bpb=0.9261 |
| tl nll=2.6598 ppl=14.29 bpb=1.2620 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 7200 01:27 (86.8s) |
| full_cpt step 7210/12852 ( 56.1%) nll=3.2478 ppl=25.73 grad=0.859 lr=9.41e-06 6852 tok/s 12.65s/it [11:16:55<19:49:47] mem=34.07GB |
| full_cpt step 7220/12852 ( 56.2%) nll=3.3286 ppl=27.90 grad=0.875 lr=9.39e-06 6854 tok/s 12.65s/it [11:18:51<19:47:22] mem=34.07GB |
| full_cpt step 7230/12852 ( 56.3%) nll=3.2479 ppl=25.74 grad=0.895 lr=9.36e-06 6855 tok/s 12.65s/it [11:20:47<19:44:57] mem=34.07GB |
| full_cpt step 7240/12852 ( 56.3%) nll=3.2291 ppl=25.26 grad=0.902 lr=9.34e-06 6857 tok/s 12.64s/it [11:22:43<19:42:32] mem=34.07GB |
| full_cpt step 7250/12852 ( 56.4%) nll=3.4218 ppl=30.62 grad=0.848 lr=9.32e-06 6859 tok/s 12.64s/it [11:24:39<19:40:08] mem=34.07GB |
| full_cpt step 7260/12852 ( 56.5%) nll=3.3661 ppl=28.97 grad=0.887 lr=9.30e-06 6860 tok/s 12.64s/it [11:26:35<19:37:43] mem=34.07GB |
| full_cpt step 7270/12852 ( 56.6%) nll=3.3162 ppl=27.56 grad=0.965 lr=9.28e-06 6862 tok/s 12.63s/it [11:28:31<19:35:19] mem=34.07GB |
| full_cpt step 7280/12852 ( 56.6%) nll=3.3324 ppl=28.01 grad=0.902 lr=9.26e-06 6864 tok/s 12.63s/it [11:30:27<19:32:55] mem=34.07GB |
| full_cpt step 7290/12852 ( 56.7%) nll=3.2756 ppl=26.46 grad=0.836 lr=9.23e-06 6866 tok/s 12.63s/it [11:32:23<19:30:31] mem=34.07GB |
| full_cpt step 7300/12852 ( 56.8%) nll=3.1915 ppl=24.33 grad=0.887 lr=9.21e-06 6867 tok/s 12.62s/it [11:34:19<19:28:08] mem=34.07GB |
| am nll=3.9706 ppl=53.01 bpb=1.3963 |
| ta nll=3.9442 ppl=51.63 bpb=0.5729 |
| kk nll=2.7502 ppl=15.65 bpb=0.9261 |
| tl nll=2.6598 ppl=14.29 bpb=1.2620 |
| en nll=2.8173 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 7300 01:26 (86.4s) |
| full_cpt step 7310/12852 ( 56.9%) nll=3.3443 ppl=28.34 grad=0.871 lr=9.19e-06 6855 tok/s 12.65s/it [11:37:41<19:28:09] mem=34.07GB |
| full_cpt step 7320/12852 ( 57.0%) nll=3.2857 ppl=26.73 grad=0.840 lr=9.17e-06 6857 tok/s 12.64s/it [11:39:37<19:25:45] mem=34.07GB |
| full_cpt step 7330/12852 ( 57.0%) nll=3.2615 ppl=26.09 grad=0.855 lr=9.15e-06 6858 tok/s 12.64s/it [11:41:33<19:23:21] mem=34.07GB |
| full_cpt step 7340/12852 ( 57.1%) nll=3.1866 ppl=24.21 grad=0.934 lr=9.13e-06 6860 tok/s 12.64s/it [11:43:29<19:20:57] mem=34.07GB |
| full_cpt step 7350/12852 ( 57.2%) nll=3.3548 ppl=28.64 grad=0.852 lr=9.10e-06 6861 tok/s 12.63s/it [11:45:25<19:18:34] mem=34.07GB |
| full_cpt step 7360/12852 ( 57.3%) nll=3.3198 ppl=27.65 grad=0.871 lr=9.08e-06 6863 tok/s 12.63s/it [11:47:21<19:16:10] mem=34.07GB |
| full_cpt step 7370/12852 ( 57.3%) nll=3.2278 ppl=25.22 grad=0.898 lr=9.06e-06 6865 tok/s 12.63s/it [11:49:17<19:13:47] mem=34.07GB |
| full_cpt step 7380/12852 ( 57.4%) nll=3.3613 ppl=28.83 grad=0.824 lr=9.04e-06 6867 tok/s 12.63s/it [11:51:13<19:11:24] mem=34.07GB |
| full_cpt step 7390/12852 ( 57.5%) nll=3.2545 ppl=25.91 grad=0.844 lr=9.02e-06 6869 tok/s 12.62s/it [11:53:09<19:09:01] mem=34.07GB |
| full_cpt step 7400/12852 ( 57.6%) nll=3.3893 ppl=29.64 grad=0.859 lr=8.99e-06 6870 tok/s 12.62s/it [11:55:05<19:06:39] mem=34.07GB |
| am nll=3.9706 ppl=53.01 bpb=1.3963 |
| ta nll=3.9437 ppl=51.61 bpb=0.5728 |
| kk nll=2.7503 ppl=15.65 bpb=0.9261 |
| tl nll=2.6597 ppl=14.29 bpb=1.2620 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 7400 01:27 (86.5s) |
| full_cpt step 7410/12852 ( 57.7%) nll=3.2257 ppl=25.17 grad=0.879 lr=8.97e-06 6858 tok/s 12.64s/it [11:58:27<19:06:34] mem=34.07GB |
| full_cpt step 7420/12852 ( 57.7%) nll=3.2082 ppl=24.73 grad=0.891 lr=8.95e-06 6859 tok/s 12.64s/it [12:00:23<19:04:11] mem=34.07GB |
| full_cpt step 7430/12852 ( 57.8%) nll=3.2458 ppl=25.68 grad=0.895 lr=8.93e-06 6861 tok/s 12.64s/it [12:02:19<19:01:48] mem=34.07GB |
| full_cpt step 7440/12852 ( 57.9%) nll=3.3389 ppl=28.19 grad=0.871 lr=8.91e-06 6863 tok/s 12.63s/it [12:04:15<18:59:26] mem=34.07GB |
| full_cpt step 7450/12852 ( 58.0%) nll=3.2336 ppl=25.37 grad=0.844 lr=8.89e-06 6864 tok/s 12.63s/it [12:06:11<18:57:03] mem=34.07GB |
| full_cpt step 7460/12852 ( 58.0%) nll=3.2342 ppl=25.39 grad=0.891 lr=8.87e-06 6866 tok/s 12.63s/it [12:08:07<18:54:41] mem=34.07GB |
| full_cpt step 7470/12852 ( 58.1%) nll=3.3145 ppl=27.51 grad=0.875 lr=8.84e-06 6867 tok/s 12.62s/it [12:10:03<18:52:19] mem=34.07GB |
| full_cpt step 7480/12852 ( 58.2%) nll=3.2484 ppl=25.75 grad=0.887 lr=8.82e-06 6868 tok/s 12.62s/it [12:11:59<18:49:56] mem=34.07GB |
| full_cpt step 7490/12852 ( 58.3%) nll=3.3281 ppl=27.89 grad=0.887 lr=8.80e-06 6870 tok/s 12.62s/it [12:13:55<18:47:34] mem=34.07GB |
| full_cpt step 7500/12852 ( 58.4%) nll=3.3426 ppl=28.29 grad=0.828 lr=8.78e-06 6872 tok/s 12.61s/it [12:15:51<18:45:13] mem=34.07GB |
| am nll=3.9705 ppl=53.01 bpb=1.3962 |
| ta nll=3.9442 ppl=51.63 bpb=0.5729 |
| kk nll=2.7500 ppl=15.64 bpb=0.9260 |
| tl nll=2.6594 ppl=14.29 bpb=1.2619 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 7500 01:29 (88.8s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.32s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.32s/it] |
| removing old checkpoint checkpoint-6500 |
| kept last 2 resume checkpoints: checkpoint-7000 checkpoint-7500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-7500 at step 7500 next_row=960000 |
| full_cpt step 7510/12852 ( 58.4%) nll=3.2909 ppl=26.87 grad=0.816 lr=8.76e-06 6847 tok/s 12.66s/it [12:20:35<18:47:08] mem=34.07GB |
| full_cpt step 7520/12852 ( 58.5%) nll=3.2871 ppl=26.76 grad=0.828 lr=8.74e-06 6849 tok/s 12.66s/it [12:22:32<18:44:46] mem=34.07GB |
| full_cpt step 7530/12852 ( 58.6%) nll=3.2609 ppl=26.07 grad=0.840 lr=8.71e-06 6851 tok/s 12.65s/it [12:24:28<18:42:23] mem=34.07GB |
| full_cpt step 7540/12852 ( 58.7%) nll=3.2925 ppl=26.91 grad=0.887 lr=8.69e-06 6852 tok/s 12.65s/it [12:26:23<18:40:01] mem=34.07GB |
| full_cpt step 7550/12852 ( 58.7%) nll=3.2524 ppl=25.85 grad=0.848 lr=8.67e-06 6853 tok/s 12.65s/it [12:28:20<18:37:38] mem=34.07GB |
| full_cpt step 7560/12852 ( 58.8%) nll=3.2874 ppl=26.77 grad=0.902 lr=8.65e-06 6855 tok/s 12.64s/it [12:30:16<18:35:16] mem=34.07GB |
| full_cpt step 7570/12852 ( 58.9%) nll=3.3021 ppl=27.17 grad=0.914 lr=8.63e-06 6856 tok/s 12.64s/it [12:32:12<18:32:55] mem=34.07GB |
| full_cpt step 7580/12852 ( 59.0%) nll=3.2742 ppl=26.42 grad=0.875 lr=8.61e-06 6857 tok/s 12.64s/it [12:34:08<18:30:33] mem=34.07GB |
| full_cpt step 7590/12852 ( 59.1%) nll=3.3652 ppl=28.94 grad=0.832 lr=8.59e-06 6859 tok/s 12.64s/it [12:36:04<18:28:11] mem=34.07GB |
| full_cpt step 7600/12852 ( 59.1%) nll=3.3812 ppl=29.41 grad=0.898 lr=8.56e-06 6860 tok/s 12.63s/it [12:37:59<18:25:49] mem=34.07GB |
| am nll=3.9704 ppl=53.00 bpb=1.3962 |
| ta nll=3.9439 ppl=51.62 bpb=0.5728 |
| kk nll=2.7499 ppl=15.64 bpb=0.9260 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 7600 01:29 (89.1s) |
| full_cpt step 7610/12852 ( 59.2%) nll=3.2511 ppl=25.82 grad=0.867 lr=8.54e-06 6847 tok/s 12.65s/it [12:41:24<18:25:37] mem=34.07GB |
| full_cpt step 7620/12852 ( 59.3%) nll=3.4024 ppl=30.04 grad=0.875 lr=8.52e-06 6849 tok/s 12.65s/it [12:43:20<18:23:16] mem=34.07GB |
| full_cpt step 7630/12852 ( 59.4%) nll=3.3702 ppl=29.08 grad=0.867 lr=8.50e-06 6851 tok/s 12.65s/it [12:45:17<18:20:54] mem=34.07GB |
| full_cpt step 7640/12852 ( 59.4%) nll=3.3735 ppl=29.18 grad=0.852 lr=8.48e-06 6852 tok/s 12.65s/it [12:47:13<18:18:33] mem=34.07GB |
| full_cpt step 7650/12852 ( 59.5%) nll=3.3221 ppl=27.72 grad=0.879 lr=8.46e-06 6854 tok/s 12.64s/it [12:49:09<18:16:11] mem=34.07GB |
| full_cpt step 7660/12852 ( 59.6%) nll=3.2162 ppl=24.93 grad=0.844 lr=8.44e-06 6855 tok/s 12.64s/it [12:51:05<18:13:50] mem=34.07GB |
| full_cpt step 7670/12852 ( 59.7%) nll=3.2502 ppl=25.80 grad=0.848 lr=8.41e-06 6857 tok/s 12.64s/it [12:53:01<18:11:29] mem=34.07GB |
| full_cpt step 7680/12852 ( 59.8%) nll=3.3408 ppl=28.24 grad=0.840 lr=8.39e-06 6859 tok/s 12.63s/it [12:54:57<18:09:08] mem=34.07GB |
| full_cpt step 7690/12852 ( 59.8%) nll=3.3039 ppl=27.22 grad=0.852 lr=8.37e-06 6860 tok/s 12.63s/it [12:56:53<18:06:47] mem=34.07GB |
| full_cpt step 7700/12852 ( 59.9%) nll=3.3040 ppl=27.22 grad=0.852 lr=8.35e-06 6862 tok/s 12.63s/it [12:58:49<18:04:27] mem=34.07GB |
| am nll=3.9704 ppl=53.01 bpb=1.3962 |
| ta nll=3.9439 ppl=51.62 bpb=0.5728 |
| kk nll=2.7499 ppl=15.64 bpb=0.9260 |
| tl nll=2.6595 ppl=14.29 bpb=1.2619 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 7700 01:26 (86.4s) |
| full_cpt step 7710/12852 ( 60.0%) nll=3.3561 ppl=28.68 grad=0.840 lr=8.33e-06 6851 tok/s 12.65s/it [13:02:11<18:04:05] mem=34.07GB |
| full_cpt step 7720/12852 ( 60.1%) nll=3.3603 ppl=28.80 grad=0.922 lr=8.31e-06 6851 tok/s 12.65s/it [13:04:06<18:01:44] mem=34.07GB |
| full_cpt step 7730/12852 ( 60.1%) nll=3.1528 ppl=23.40 grad=0.832 lr=8.29e-06 6853 tok/s 12.64s/it [13:06:02<17:59:23] mem=34.07GB |
| full_cpt step 7740/12852 ( 60.2%) nll=3.2494 ppl=25.77 grad=0.855 lr=8.27e-06 6855 tok/s 12.64s/it [13:07:58<17:57:02] mem=34.07GB |
| full_cpt step 7750/12852 ( 60.3%) nll=3.2215 ppl=25.07 grad=0.898 lr=8.25e-06 6856 tok/s 12.64s/it [13:09:54<17:54:42] mem=34.07GB |
| full_cpt step 7760/12852 ( 60.4%) nll=3.1838 ppl=24.14 grad=0.879 lr=8.22e-06 6858 tok/s 12.64s/it [13:11:50<17:52:21] mem=34.07GB |
| full_cpt step 7770/12852 ( 60.5%) nll=3.3091 ppl=27.36 grad=0.926 lr=8.20e-06 6859 tok/s 12.63s/it [13:13:46<17:50:01] mem=34.07GB |
| full_cpt step 7780/12852 ( 60.5%) nll=3.1512 ppl=23.36 grad=0.828 lr=8.18e-06 6861 tok/s 12.63s/it [13:15:42<17:47:41] mem=34.07GB |
| full_cpt step 7790/12852 ( 60.6%) nll=3.3210 ppl=27.69 grad=0.902 lr=8.16e-06 6862 tok/s 12.63s/it [13:17:38<17:45:20] mem=34.07GB |
| full_cpt step 7800/12852 ( 60.7%) nll=3.3420 ppl=28.27 grad=0.883 lr=8.14e-06 6863 tok/s 12.62s/it [13:19:34<17:43:00] mem=34.07GB |
| am nll=3.9705 ppl=53.01 bpb=1.3962 |
| ta nll=3.9435 ppl=51.60 bpb=0.5728 |
| kk nll=2.7499 ppl=15.64 bpb=0.9260 |
| tl nll=2.6597 ppl=14.29 bpb=1.2620 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 7800 01:26 (86.3s) |
| full_cpt step 7810/12852 ( 60.8%) nll=3.3332 ppl=28.03 grad=0.832 lr=8.12e-06 6853 tok/s 12.64s/it [13:22:56<17:42:35] mem=34.07GB |
| full_cpt step 7820/12852 ( 60.8%) nll=3.2543 ppl=25.90 grad=0.832 lr=8.10e-06 6854 tok/s 12.64s/it [13:24:52<17:40:14] mem=34.07GB |
| full_cpt step 7830/12852 ( 60.9%) nll=3.3115 ppl=27.43 grad=0.879 lr=8.08e-06 6856 tok/s 12.64s/it [13:26:48<17:37:54] mem=34.07GB |
| full_cpt step 7840/12852 ( 61.0%) nll=3.3011 ppl=27.14 grad=0.863 lr=8.06e-06 6857 tok/s 12.64s/it [13:28:44<17:35:34] mem=34.07GB |
| full_cpt step 7850/12852 ( 61.1%) nll=3.2757 ppl=26.46 grad=0.953 lr=8.03e-06 6858 tok/s 12.63s/it [13:30:40<17:33:14] mem=34.07GB |
| full_cpt step 7860/12852 ( 61.2%) nll=3.4445 ppl=31.33 grad=0.930 lr=8.01e-06 6859 tok/s 12.63s/it [13:32:36<17:30:54] mem=34.07GB |
| full_cpt step 7870/12852 ( 61.2%) nll=3.3976 ppl=29.89 grad=0.867 lr=7.99e-06 6861 tok/s 12.63s/it [13:34:32<17:28:34] mem=34.07GB |
| full_cpt step 7880/12852 ( 61.3%) nll=3.1665 ppl=23.72 grad=0.816 lr=7.97e-06 6863 tok/s 12.63s/it [13:36:28<17:26:15] mem=34.07GB |
| full_cpt step 7890/12852 ( 61.4%) nll=3.2262 ppl=25.18 grad=0.867 lr=7.95e-06 6864 tok/s 12.62s/it [13:38:24<17:23:56] mem=34.07GB |
| full_cpt step 7900/12852 ( 61.5%) nll=3.2984 ppl=27.07 grad=0.828 lr=7.93e-06 6866 tok/s 12.62s/it [13:40:20<17:21:36] mem=34.07GB |
| am nll=3.9700 ppl=52.99 bpb=1.3961 |
| ta nll=3.9439 ppl=51.62 bpb=0.5728 |
| kk nll=2.7494 ppl=15.63 bpb=0.9259 |
| tl nll=2.6595 ppl=14.29 bpb=1.2619 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 7900 01:26 (86.2s) |
| full_cpt step 7910/12852 ( 61.5%) nll=3.3325 ppl=28.01 grad=0.895 lr=7.91e-06 6855 tok/s 12.64s/it [13:43:42<17:21:06] mem=34.07GB |
| full_cpt step 7920/12852 ( 61.6%) nll=3.3036 ppl=27.21 grad=0.836 lr=7.89e-06 6857 tok/s 12.64s/it [13:45:38<17:18:47] mem=34.07GB |
| full_cpt step 7930/12852 ( 61.7%) nll=3.3334 ppl=28.03 grad=0.898 lr=7.87e-06 6859 tok/s 12.63s/it [13:47:34<17:16:27] mem=34.07GB |
| full_cpt step 7940/12852 ( 61.8%) nll=3.3658 ppl=28.96 grad=0.852 lr=7.85e-06 6860 tok/s 12.63s/it [13:49:30<17:14:08] mem=34.07GB |
| full_cpt step 7950/12852 ( 61.9%) nll=3.2885 ppl=26.80 grad=0.828 lr=7.83e-06 6862 tok/s 12.63s/it [13:51:26<17:11:49] mem=34.07GB |
| full_cpt step 7960/12852 ( 61.9%) nll=3.3966 ppl=29.86 grad=0.840 lr=7.80e-06 6863 tok/s 12.63s/it [13:53:21<17:09:29] mem=34.07GB |
| full_cpt step 7970/12852 ( 62.0%) nll=3.2600 ppl=26.05 grad=0.852 lr=7.78e-06 6864 tok/s 12.62s/it [13:55:17<17:07:11] mem=34.07GB |
| full_cpt step 7980/12852 ( 62.1%) nll=3.2507 ppl=25.81 grad=0.863 lr=7.76e-06 6866 tok/s 12.62s/it [13:57:14<17:04:52] mem=34.07GB |
| full_cpt step 7990/12852 ( 62.2%) nll=3.2973 ppl=27.04 grad=0.867 lr=7.74e-06 6868 tok/s 12.62s/it [13:59:10<17:02:33] mem=34.07GB |
| full_cpt step 8000/12852 ( 62.2%) nll=3.2094 ppl=24.76 grad=0.836 lr=7.72e-06 6869 tok/s 12.62s/it [14:01:05<17:00:15] mem=34.07GB |
| am nll=3.9697 ppl=52.97 bpb=1.3960 |
| ta nll=3.9438 ppl=51.61 bpb=0.5728 |
| kk nll=2.7494 ppl=15.63 bpb=0.9258 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 8000 01:27 (86.6s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.32s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.33s/it] |
| removing old checkpoint checkpoint-7000 |
| kept last 2 resume checkpoints: checkpoint-7500 checkpoint-8000 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-8000 at step 8000 next_row=1024000 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.63s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.64s/it] |
| wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-8000 (weights+expanded tokenizer, no optimizer) |
| full_cpt step 8010/12852 ( 62.3%) nll=3.1602 ppl=23.57 grad=0.855 lr=7.70e-06 6845 tok/s 12.66s/it [14:06:10<17:01:44] mem=34.07GB |
| full_cpt step 8020/12852 ( 62.4%) nll=3.2906 ppl=26.86 grad=0.863 lr=7.68e-06 6846 tok/s 12.66s/it [14:08:06<16:59:25] mem=34.07GB |
| full_cpt step 8030/12852 ( 62.5%) nll=3.3302 ppl=27.94 grad=0.840 lr=7.66e-06 6848 tok/s 12.66s/it [14:10:02<16:57:05] mem=34.07GB |
| full_cpt step 8040/12852 ( 62.6%) nll=3.2308 ppl=25.30 grad=0.875 lr=7.64e-06 6849 tok/s 12.65s/it [14:11:58<16:54:46] mem=34.07GB |
| full_cpt step 8050/12852 ( 62.6%) nll=3.1966 ppl=24.45 grad=0.895 lr=7.62e-06 6850 tok/s 12.65s/it [14:13:54<16:52:27] mem=34.07GB |
| full_cpt step 8060/12852 ( 62.7%) nll=3.2061 ppl=24.68 grad=0.891 lr=7.60e-06 6852 tok/s 12.65s/it [14:15:50<16:50:08] mem=34.07GB |
| full_cpt step 8070/12852 ( 62.8%) nll=3.2828 ppl=26.65 grad=0.898 lr=7.58e-06 6853 tok/s 12.65s/it [14:17:46<16:47:49] mem=34.07GB |
| full_cpt step 8080/12852 ( 62.9%) nll=3.3417 ppl=28.27 grad=0.941 lr=7.56e-06 6855 tok/s 12.64s/it [14:19:41<16:45:30] mem=34.07GB |
| full_cpt step 8090/12852 ( 62.9%) nll=3.3617 ppl=28.84 grad=0.898 lr=7.54e-06 6856 tok/s 12.64s/it [14:21:37<16:43:11] mem=34.07GB |
| full_cpt step 8100/12852 ( 63.0%) nll=3.2946 ppl=26.97 grad=0.902 lr=7.52e-06 6857 tok/s 12.64s/it [14:23:33<16:40:53] mem=34.07GB |
| am nll=3.9703 ppl=53.00 bpb=1.3962 |
| ta nll=3.9437 ppl=51.61 bpb=0.5728 |
| kk nll=2.7496 ppl=15.64 bpb=0.9259 |
| tl nll=2.6594 ppl=14.29 bpb=1.2619 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 8100 01:26 (86.3s) |
| full_cpt step 8110/12852 ( 63.1%) nll=3.2212 ppl=25.06 grad=0.816 lr=7.50e-06 6848 tok/s 12.66s/it [14:26:55<16:40:14] mem=34.07GB |
| full_cpt step 8120/12852 ( 63.2%) nll=3.2217 ppl=25.07 grad=0.867 lr=7.47e-06 6849 tok/s 12.65s/it [14:28:51<16:37:55] mem=34.07GB |
| full_cpt step 8130/12852 ( 63.3%) nll=3.3605 ppl=28.80 grad=0.918 lr=7.45e-06 6851 tok/s 12.65s/it [14:30:47<16:35:36] mem=34.07GB |
| full_cpt step 8140/12852 ( 63.3%) nll=3.2702 ppl=26.32 grad=0.895 lr=7.43e-06 6852 tok/s 12.65s/it [14:32:43<16:33:18] mem=34.07GB |
| full_cpt step 8150/12852 ( 63.4%) nll=3.3433 ppl=28.31 grad=0.863 lr=7.41e-06 6853 tok/s 12.65s/it [14:34:39<16:30:59] mem=34.07GB |
| full_cpt step 8160/12852 ( 63.5%) nll=3.3018 ppl=27.16 grad=0.910 lr=7.39e-06 6855 tok/s 12.64s/it [14:36:35<16:28:41] mem=34.07GB |
| full_cpt step 8170/12852 ( 63.6%) nll=3.2327 ppl=25.35 grad=0.812 lr=7.37e-06 6856 tok/s 12.64s/it [14:38:31<16:26:23] mem=34.07GB |
| full_cpt step 8180/12852 ( 63.6%) nll=3.2686 ppl=26.27 grad=0.875 lr=7.35e-06 6857 tok/s 12.64s/it [14:40:27<16:24:05] mem=34.07GB |
| full_cpt step 8190/12852 ( 63.7%) nll=3.3119 ppl=27.44 grad=0.887 lr=7.33e-06 6858 tok/s 12.64s/it [14:42:23<16:21:47] mem=34.07GB |
| full_cpt step 8200/12852 ( 63.8%) nll=3.2357 ppl=25.42 grad=0.859 lr=7.31e-06 6860 tok/s 12.63s/it [14:44:19<16:19:29] mem=34.07GB |
| am nll=3.9700 ppl=52.98 bpb=1.3960 |
| ta nll=3.9436 ppl=51.61 bpb=0.5728 |
| kk nll=2.7495 ppl=15.64 bpb=0.9259 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 8200 01:26 (86.2s) |
| full_cpt step 8210/12852 ( 63.9%) nll=3.3063 ppl=27.28 grad=0.910 lr=7.29e-06 6850 tok/s 12.65s/it [14:47:41<16:18:46] mem=34.07GB |
| full_cpt step 8220/12852 ( 64.0%) nll=3.1597 ppl=23.56 grad=0.859 lr=7.27e-06 6852 tok/s 12.65s/it [14:49:37<16:16:28] mem=34.07GB |
| full_cpt step 8230/12852 ( 64.0%) nll=3.2571 ppl=25.97 grad=0.891 lr=7.25e-06 6853 tok/s 12.65s/it [14:51:33<16:14:10] mem=34.07GB |
| full_cpt step 8240/12852 ( 64.1%) nll=3.4470 ppl=31.41 grad=0.871 lr=7.23e-06 6854 tok/s 12.64s/it [14:53:28<16:11:52] mem=34.07GB |
| full_cpt step 8250/12852 ( 64.2%) nll=3.2754 ppl=26.45 grad=0.852 lr=7.21e-06 6855 tok/s 12.64s/it [14:55:24<16:09:34] mem=34.07GB |
| full_cpt step 8260/12852 ( 64.3%) nll=3.3093 ppl=27.37 grad=0.883 lr=7.19e-06 6857 tok/s 12.64s/it [14:57:20<16:07:16] mem=34.07GB |
| full_cpt step 8270/12852 ( 64.3%) nll=3.2490 ppl=25.76 grad=0.898 lr=7.17e-06 6859 tok/s 12.64s/it [14:59:16<16:04:59] mem=34.07GB |
| full_cpt step 8280/12852 ( 64.4%) nll=3.1901 ppl=24.29 grad=0.840 lr=7.15e-06 6860 tok/s 12.63s/it [15:01:12<16:02:41] mem=34.07GB |
| full_cpt step 8290/12852 ( 64.5%) nll=3.3850 ppl=29.52 grad=0.906 lr=7.13e-06 6861 tok/s 12.63s/it [15:03:08<16:00:24] mem=34.07GB |
| full_cpt step 8300/12852 ( 64.6%) nll=3.2681 ppl=26.26 grad=0.898 lr=7.11e-06 6862 tok/s 12.63s/it [15:05:04<15:58:06] mem=34.07GB |
| am nll=3.9696 ppl=52.97 bpb=1.3959 |
| ta nll=3.9433 ppl=51.59 bpb=0.5727 |
| kk nll=2.7496 ppl=15.64 bpb=0.9259 |
| tl nll=2.6595 ppl=14.29 bpb=1.2619 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 8300 01:30 (89.5s) |
| full_cpt step 8310/12852 ( 64.7%) nll=3.3057 ppl=27.27 grad=0.879 lr=7.09e-06 6852 tok/s 12.65s/it [15:08:29<15:57:23] mem=34.07GB |
| full_cpt step 8320/12852 ( 64.7%) nll=3.3435 ppl=28.32 grad=0.848 lr=7.07e-06 6854 tok/s 12.64s/it [15:10:25<15:55:06] mem=34.07GB |
| full_cpt step 8330/12852 ( 64.8%) nll=3.2883 ppl=26.80 grad=0.871 lr=7.05e-06 6855 tok/s 12.64s/it [15:12:21<15:52:49] mem=34.07GB |
| full_cpt step 8340/12852 ( 64.9%) nll=3.3330 ppl=28.02 grad=0.836 lr=7.03e-06 6856 tok/s 12.64s/it [15:14:17<15:50:31] mem=34.07GB |
| full_cpt step 8350/12852 ( 65.0%) nll=3.2772 ppl=26.50 grad=0.828 lr=7.01e-06 6858 tok/s 12.64s/it [15:16:13<15:48:14] mem=34.07GB |
| full_cpt step 8360/12852 ( 65.0%) nll=3.4091 ppl=30.24 grad=0.887 lr=6.99e-06 6859 tok/s 12.64s/it [15:18:09<15:45:57] mem=34.07GB |
| full_cpt step 8370/12852 ( 65.1%) nll=3.2744 ppl=26.43 grad=0.852 lr=6.97e-06 6860 tok/s 12.63s/it [15:20:05<15:43:40] mem=34.07GB |
| full_cpt step 8380/12852 ( 65.2%) nll=3.3816 ppl=29.42 grad=0.938 lr=6.95e-06 6861 tok/s 12.63s/it [15:22:01<15:41:23] mem=34.07GB |
| full_cpt step 8390/12852 ( 65.3%) nll=3.3735 ppl=29.18 grad=0.883 lr=6.93e-06 6863 tok/s 12.63s/it [15:23:57<15:39:06] mem=34.07GB |
| full_cpt step 8400/12852 ( 65.4%) nll=3.2125 ppl=24.84 grad=0.863 lr=6.91e-06 6863 tok/s 12.63s/it [15:25:53<15:36:49] mem=34.07GB |
| am nll=3.9695 ppl=52.96 bpb=1.3959 |
| ta nll=3.9438 ppl=51.61 bpb=0.5728 |
| kk nll=2.7498 ppl=15.64 bpb=0.9260 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 8400 01:27 (87.2s) |
| full_cpt step 8410/12852 ( 65.4%) nll=3.2820 ppl=26.63 grad=0.840 lr=6.89e-06 6854 tok/s 12.64s/it [15:29:16<15:36:00] mem=34.07GB |
| full_cpt step 8420/12852 ( 65.5%) nll=3.3200 ppl=27.66 grad=0.875 lr=6.87e-06 6855 tok/s 12.64s/it [15:31:12<15:33:43] mem=34.07GB |
| full_cpt step 8430/12852 ( 65.6%) nll=3.2474 ppl=25.72 grad=0.832 lr=6.85e-06 6856 tok/s 12.64s/it [15:33:07<15:31:26] mem=34.07GB |
| full_cpt step 8440/12852 ( 65.7%) nll=3.3084 ppl=27.34 grad=0.891 lr=6.83e-06 6858 tok/s 12.64s/it [15:35:04<15:29:10] mem=34.07GB |
| full_cpt step 8450/12852 ( 65.7%) nll=3.1642 ppl=23.67 grad=0.816 lr=6.81e-06 6860 tok/s 12.63s/it [15:37:00<15:26:53] mem=34.07GB |
| full_cpt step 8460/12852 ( 65.8%) nll=3.3078 ppl=27.33 grad=0.922 lr=6.79e-06 6861 tok/s 12.63s/it [15:38:56<15:24:37] mem=34.07GB |
| full_cpt step 8470/12852 ( 65.9%) nll=3.2183 ppl=24.99 grad=0.863 lr=6.77e-06 6862 tok/s 12.63s/it [15:40:52<15:22:20] mem=34.07GB |
| full_cpt step 8480/12852 ( 66.0%) nll=3.1616 ppl=23.61 grad=0.859 lr=6.75e-06 6863 tok/s 12.63s/it [15:42:48<15:20:04] mem=34.07GB |
| full_cpt step 8490/12852 ( 66.1%) nll=3.1917 ppl=24.33 grad=0.812 lr=6.73e-06 6864 tok/s 12.62s/it [15:44:44<15:17:48] mem=34.07GB |
| full_cpt step 8500/12852 ( 66.1%) nll=3.3693 ppl=29.06 grad=0.871 lr=6.71e-06 6866 tok/s 12.62s/it [15:46:40<15:15:32] mem=34.07GB |
| am nll=3.9693 ppl=52.95 bpb=1.3958 |
| ta nll=3.9432 ppl=51.58 bpb=0.5727 |
| kk nll=2.7498 ppl=15.64 bpb=0.9260 |
| tl nll=2.6595 ppl=14.29 bpb=1.2619 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 8500 01:27 (86.7s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.62s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.63s/it] |
| removing old checkpoint checkpoint-7500 |
| kept last 2 resume checkpoints: checkpoint-8000 checkpoint-8500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-8500 at step 8500 next_row=1088000 |
| full_cpt step 8510/12852 ( 66.2%) nll=3.2425 ppl=25.60 grad=0.805 lr=6.70e-06 6847 tok/s 12.66s/it [15:51:26<15:15:59] mem=34.07GB |
| full_cpt step 8520/12852 ( 66.3%) nll=3.3693 ppl=29.06 grad=0.969 lr=6.68e-06 6848 tok/s 12.66s/it [15:53:22<15:13:42] mem=34.07GB |
| full_cpt step 8530/12852 ( 66.4%) nll=3.4402 ppl=31.19 grad=0.887 lr=6.66e-06 6849 tok/s 12.65s/it [15:55:18<15:11:26] mem=34.07GB |
| full_cpt step 8540/12852 ( 66.4%) nll=3.3694 ppl=29.06 grad=0.883 lr=6.64e-06 6850 tok/s 12.65s/it [15:57:14<15:09:09] mem=34.07GB |
| full_cpt step 8550/12852 ( 66.5%) nll=3.2935 ppl=26.94 grad=0.930 lr=6.62e-06 6852 tok/s 12.65s/it [15:59:10<15:06:53] mem=34.07GB |
| full_cpt step 8560/12852 ( 66.6%) nll=3.3468 ppl=28.41 grad=0.836 lr=6.60e-06 6853 tok/s 12.65s/it [16:01:06<15:04:37] mem=34.07GB |
| full_cpt step 8570/12852 ( 66.7%) nll=3.3175 ppl=27.59 grad=0.855 lr=6.58e-06 6854 tok/s 12.64s/it [16:03:02<15:02:20] mem=34.07GB |
| full_cpt step 8580/12852 ( 66.8%) nll=3.3342 ppl=28.06 grad=0.945 lr=6.56e-06 6855 tok/s 12.64s/it [16:04:58<15:00:04] mem=34.07GB |
| full_cpt step 8590/12852 ( 66.8%) nll=3.3623 ppl=28.86 grad=0.852 lr=6.54e-06 6856 tok/s 12.64s/it [16:06:54<14:57:48] mem=34.07GB |
| full_cpt step 8600/12852 ( 66.9%) nll=3.2980 ppl=27.06 grad=0.840 lr=6.52e-06 6858 tok/s 12.64s/it [16:08:50<14:55:32] mem=34.07GB |
| am nll=3.9693 ppl=52.95 bpb=1.3958 |
| ta nll=3.9433 ppl=51.59 bpb=0.5727 |
| kk nll=2.7494 ppl=15.63 bpb=0.9258 |
| tl nll=2.6593 ppl=14.29 bpb=1.2618 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 8600 01:28 (87.5s) |
| full_cpt step 8610/12852 ( 67.0%) nll=3.2773 ppl=26.50 grad=0.805 lr=6.50e-06 6849 tok/s 12.65s/it [16:12:13<14:54:37] mem=34.07GB |
| full_cpt step 8620/12852 ( 67.1%) nll=3.2131 ppl=24.85 grad=0.875 lr=6.48e-06 6850 tok/s 12.65s/it [16:14:09<14:52:20] mem=34.07GB |
| full_cpt step 8630/12852 ( 67.1%) nll=3.3496 ppl=28.49 grad=0.883 lr=6.46e-06 6851 tok/s 12.65s/it [16:16:05<14:50:04] mem=34.07GB |
| full_cpt step 8640/12852 ( 67.2%) nll=3.3939 ppl=29.78 grad=0.859 lr=6.44e-06 6853 tok/s 12.65s/it [16:18:01<14:47:48] mem=34.07GB |
| full_cpt step 8650/12852 ( 67.3%) nll=3.1312 ppl=22.90 grad=0.855 lr=6.42e-06 6854 tok/s 12.64s/it [16:19:57<14:45:32] mem=34.07GB |
| full_cpt step 8660/12852 ( 67.4%) nll=3.2593 ppl=26.03 grad=0.910 lr=6.41e-06 6856 tok/s 12.64s/it [16:21:53<14:43:16] mem=34.07GB |
| full_cpt step 8670/12852 ( 67.5%) nll=3.2149 ppl=24.90 grad=0.875 lr=6.39e-06 6857 tok/s 12.64s/it [16:23:49<14:41:01] mem=34.07GB |
| full_cpt step 8680/12852 ( 67.5%) nll=3.1961 ppl=24.44 grad=0.906 lr=6.37e-06 6858 tok/s 12.64s/it [16:25:45<14:38:45] mem=34.07GB |
| full_cpt step 8690/12852 ( 67.6%) nll=3.2869 ppl=26.76 grad=0.871 lr=6.35e-06 6860 tok/s 12.64s/it [16:27:41<14:36:30] mem=34.07GB |
| full_cpt step 8700/12852 ( 67.7%) nll=3.2899 ppl=26.84 grad=0.820 lr=6.33e-06 6861 tok/s 12.63s/it [16:29:37<14:34:14] mem=34.07GB |
| am nll=3.9697 ppl=52.97 bpb=1.3959 |
| ta nll=3.9432 ppl=51.58 bpb=0.5727 |
| kk nll=2.7496 ppl=15.64 bpb=0.9259 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 8700 01:27 (87.0s) |
| full_cpt step 8710/12852 ( 67.8%) nll=3.3029 ppl=27.19 grad=0.883 lr=6.31e-06 6852 tok/s 12.65s/it [16:33:00<14:33:15] mem=34.07GB |
| full_cpt step 8720/12852 ( 67.8%) nll=3.2654 ppl=26.19 grad=0.875 lr=6.29e-06 6853 tok/s 12.65s/it [16:34:56<14:30:59] mem=34.07GB |
| full_cpt step 8730/12852 ( 67.9%) nll=3.3671 ppl=28.99 grad=0.867 lr=6.27e-06 6854 tok/s 12.65s/it [16:36:52<14:28:44] mem=34.07GB |
| full_cpt step 8740/12852 ( 68.0%) nll=3.2224 ppl=25.09 grad=0.875 lr=6.25e-06 6855 tok/s 12.64s/it [16:38:48<14:26:28] mem=34.07GB |
| full_cpt step 8750/12852 ( 68.1%) nll=3.2239 ppl=25.13 grad=0.828 lr=6.23e-06 6857 tok/s 12.64s/it [16:40:44<14:24:13] mem=34.07GB |
| full_cpt step 8760/12852 ( 68.2%) nll=3.2695 ppl=26.30 grad=0.832 lr=6.22e-06 6858 tok/s 12.64s/it [16:42:40<14:21:57] mem=34.07GB |
| full_cpt step 8770/12852 ( 68.2%) nll=3.1655 ppl=23.70 grad=0.824 lr=6.20e-06 6859 tok/s 12.64s/it [16:44:36<14:19:42] mem=34.07GB |
| full_cpt step 8780/12852 ( 68.3%) nll=3.2058 ppl=24.67 grad=0.824 lr=6.18e-06 6860 tok/s 12.63s/it [16:46:32<14:17:27] mem=34.07GB |
| full_cpt step 8790/12852 ( 68.4%) nll=3.3325 ppl=28.01 grad=0.828 lr=6.16e-06 6862 tok/s 12.63s/it [16:48:28<14:15:11] mem=34.07GB |
| full_cpt step 8800/12852 ( 68.5%) nll=3.3215 ppl=27.70 grad=0.887 lr=6.14e-06 6863 tok/s 12.63s/it [16:50:24<14:12:56] mem=34.07GB |
| am nll=3.9698 ppl=52.97 bpb=1.3960 |
| ta nll=3.9434 ppl=51.59 bpb=0.5728 |
| kk nll=2.7493 ppl=15.63 bpb=0.9258 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 8800 01:26 (86.4s) |
| full_cpt step 8810/12852 ( 68.5%) nll=3.4132 ppl=30.36 grad=0.809 lr=6.12e-06 6854 tok/s 12.65s/it [16:53:46<14:11:54] mem=34.07GB |
| full_cpt step 8820/12852 ( 68.6%) nll=3.3718 ppl=29.13 grad=0.855 lr=6.10e-06 6856 tok/s 12.64s/it [16:55:42<14:09:39] mem=34.07GB |
| full_cpt step 8830/12852 ( 68.7%) nll=3.2657 ppl=26.20 grad=0.895 lr=6.08e-06 6857 tok/s 12.64s/it [16:57:38<14:07:24] mem=34.07GB |
| full_cpt step 8840/12852 ( 68.8%) nll=3.2375 ppl=25.47 grad=0.914 lr=6.07e-06 6858 tok/s 12.64s/it [16:59:34<14:05:09] mem=34.07GB |
| full_cpt step 8850/12852 ( 68.9%) nll=3.2847 ppl=26.70 grad=0.875 lr=6.05e-06 6860 tok/s 12.64s/it [17:01:30<14:02:54] mem=34.07GB |
| full_cpt step 8860/12852 ( 68.9%) nll=3.3063 ppl=27.28 grad=0.926 lr=6.03e-06 6861 tok/s 12.63s/it [17:03:26<14:00:39] mem=34.07GB |
| full_cpt step 8870/12852 ( 69.0%) nll=3.2991 ppl=27.09 grad=0.820 lr=6.01e-06 6862 tok/s 12.63s/it [17:05:22<13:58:24] mem=34.07GB |
| full_cpt step 8880/12852 ( 69.1%) nll=3.3511 ppl=28.53 grad=0.836 lr=5.99e-06 6863 tok/s 12.63s/it [17:07:17<13:56:09] mem=34.07GB |
| full_cpt step 8890/12852 ( 69.2%) nll=3.3804 ppl=29.38 grad=0.883 lr=5.97e-06 6864 tok/s 12.63s/it [17:09:13<13:53:54] mem=34.07GB |
| full_cpt step 8900/12852 ( 69.2%) nll=3.2070 ppl=24.70 grad=0.910 lr=5.96e-06 6865 tok/s 12.63s/it [17:11:09<13:51:40] mem=34.07GB |
| am nll=3.9696 ppl=52.96 bpb=1.3959 |
| ta nll=3.9432 ppl=51.58 bpb=0.5727 |
| kk nll=2.7494 ppl=15.63 bpb=0.9258 |
| tl nll=2.6591 ppl=14.28 bpb=1.2617 |
| en nll=2.8173 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 8900 01:26 (86.2s) |
| full_cpt step 8910/12852 ( 69.3%) nll=3.3377 ppl=28.15 grad=0.820 lr=5.94e-06 6857 tok/s 12.64s/it [17:14:32<13:50:34] mem=34.07GB |
| full_cpt step 8920/12852 ( 69.4%) nll=3.3744 ppl=29.21 grad=0.895 lr=5.92e-06 6858 tok/s 12.64s/it [17:16:27<13:48:19] mem=34.07GB |
| full_cpt step 8930/12852 ( 69.5%) nll=3.3003 ppl=27.12 grad=0.836 lr=5.90e-06 6859 tok/s 12.64s/it [17:18:23<13:46:05] mem=34.07GB |
| full_cpt step 8940/12852 ( 69.6%) nll=3.2992 ppl=27.09 grad=0.891 lr=5.88e-06 6860 tok/s 12.64s/it [17:20:19<13:43:50] mem=34.07GB |
| full_cpt step 8950/12852 ( 69.6%) nll=3.2359 ppl=25.43 grad=0.828 lr=5.86e-06 6862 tok/s 12.63s/it [17:22:15<13:41:35] mem=34.07GB |
| full_cpt step 8960/12852 ( 69.7%) nll=3.2321 ppl=25.33 grad=0.863 lr=5.85e-06 6863 tok/s 12.63s/it [17:24:11<13:39:21] mem=34.07GB |
| full_cpt step 8970/12852 ( 69.8%) nll=3.2445 ppl=25.65 grad=0.898 lr=5.83e-06 6864 tok/s 12.63s/it [17:26:07<13:37:06] mem=34.07GB |
| full_cpt step 8980/12852 ( 69.9%) nll=3.2224 ppl=25.09 grad=0.855 lr=5.81e-06 6865 tok/s 12.63s/it [17:28:03<13:34:52] mem=34.07GB |
| full_cpt step 8990/12852 ( 70.0%) nll=3.3423 ppl=28.28 grad=0.879 lr=5.79e-06 6866 tok/s 12.63s/it [17:29:59<13:32:38] mem=34.07GB |
| full_cpt step 9000/12852 ( 70.0%) nll=3.4348 ppl=31.03 grad=0.844 lr=5.77e-06 6867 tok/s 12.62s/it [17:31:55<13:30:24] mem=34.07GB |
| am nll=3.9695 ppl=52.96 bpb=1.3959 |
| ta nll=3.9431 ppl=51.58 bpb=0.5727 |
| kk nll=2.7493 ppl=15.63 bpb=0.9258 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 9000 01:26 (86.1s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.46s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.46s/it] |
| removing old checkpoint checkpoint-8000 |
| kept last 2 resume checkpoints: checkpoint-8500 checkpoint-9000 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-9000 at step 9000 next_row=1152000 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.68s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.68s/it] |
| wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-9000 (weights+expanded tokenizer, no optimizer) |
| full_cpt step 9010/12852 ( 70.1%) nll=3.3050 ppl=27.25 grad=0.812 lr=5.75e-06 6853 tok/s 12.65s/it [17:36:18<13:30:02] mem=34.07GB |
| full_cpt step 9020/12852 ( 70.2%) nll=3.3260 ppl=27.83 grad=0.848 lr=5.74e-06 6854 tok/s 12.65s/it [17:38:14<13:27:48] mem=34.07GB |
| full_cpt step 9030/12852 ( 70.3%) nll=3.3661 ppl=28.97 grad=0.855 lr=5.72e-06 6855 tok/s 12.65s/it [17:40:10<13:25:33] mem=34.07GB |
| full_cpt step 9040/12852 ( 70.3%) nll=3.2030 ppl=24.61 grad=0.918 lr=5.70e-06 6856 tok/s 12.64s/it [17:42:06<13:23:19] mem=34.07GB |
| full_cpt step 9050/12852 ( 70.4%) nll=3.3193 ppl=27.64 grad=0.902 lr=5.68e-06 6857 tok/s 12.64s/it [17:44:01<13:21:04] mem=34.07GB |
| full_cpt step 9060/12852 ( 70.5%) nll=3.2559 ppl=25.94 grad=0.828 lr=5.66e-06 6858 tok/s 12.64s/it [17:45:57<13:18:50] mem=34.07GB |
| full_cpt step 9070/12852 ( 70.6%) nll=3.3017 ppl=27.16 grad=0.875 lr=5.65e-06 6859 tok/s 12.64s/it [17:47:53<13:16:36] mem=34.07GB |
| full_cpt step 9080/12852 ( 70.7%) nll=3.3319 ppl=27.99 grad=0.852 lr=5.63e-06 6860 tok/s 12.64s/it [17:49:49<13:14:22] mem=34.07GB |
| full_cpt step 9090/12852 ( 70.7%) nll=3.3994 ppl=29.95 grad=0.855 lr=5.61e-06 6862 tok/s 12.63s/it [17:51:45<13:12:08] mem=34.07GB |
| full_cpt step 9100/12852 ( 70.8%) nll=3.3619 ppl=28.84 grad=0.867 lr=5.59e-06 6863 tok/s 12.63s/it [17:53:41<13:09:54] mem=34.07GB |
| am nll=3.9695 ppl=52.96 bpb=1.3959 |
| ta nll=3.9436 ppl=51.60 bpb=0.5728 |
| kk nll=2.7493 ppl=15.63 bpb=0.9258 |
| tl nll=2.6593 ppl=14.29 bpb=1.2618 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 9100 01:26 (86.3s) |
| full_cpt step 9110/12852 ( 70.9%) nll=3.2238 ppl=25.12 grad=0.875 lr=5.58e-06 6854 tok/s 12.65s/it [17:57:03<13:08:43] mem=34.07GB |
| full_cpt step 9120/12852 ( 71.0%) nll=3.1622 ppl=23.62 grad=0.875 lr=5.56e-06 6855 tok/s 12.64s/it [17:58:59<13:06:29] mem=34.07GB |
| full_cpt step 9130/12852 ( 71.0%) nll=3.3299 ppl=27.94 grad=0.852 lr=5.54e-06 6857 tok/s 12.64s/it [18:00:55<13:04:15] mem=34.07GB |
| full_cpt step 9140/12852 ( 71.1%) nll=3.2462 ppl=25.69 grad=0.887 lr=5.52e-06 6858 tok/s 12.64s/it [18:02:51<13:02:01] mem=34.07GB |
| full_cpt step 9150/12852 ( 71.2%) nll=3.4121 ppl=30.33 grad=0.898 lr=5.51e-06 6859 tok/s 12.64s/it [18:04:47<12:59:47] mem=34.07GB |
| full_cpt step 9160/12852 ( 71.3%) nll=3.1818 ppl=24.09 grad=0.859 lr=5.49e-06 6860 tok/s 12.64s/it [18:06:43<12:57:33] mem=34.07GB |
| full_cpt step 9170/12852 ( 71.4%) nll=3.3793 ppl=29.35 grad=0.879 lr=5.47e-06 6861 tok/s 12.63s/it [18:08:39<12:55:19] mem=34.07GB |
| full_cpt step 9180/12852 ( 71.4%) nll=3.2624 ppl=26.11 grad=0.859 lr=5.45e-06 6863 tok/s 12.63s/it [18:10:35<12:53:05] mem=34.07GB |
| full_cpt step 9190/12852 ( 71.5%) nll=3.2933 ppl=26.93 grad=0.867 lr=5.43e-06 6863 tok/s 12.63s/it [18:12:31<12:50:52] mem=34.07GB |
| full_cpt step 9200/12852 ( 71.6%) nll=3.2531 ppl=25.87 grad=0.832 lr=5.42e-06 6864 tok/s 12.63s/it [18:14:27<12:48:38] mem=34.07GB |
| am nll=3.9695 ppl=52.96 bpb=1.3959 |
| ta nll=3.9430 ppl=51.57 bpb=0.5727 |
| kk nll=2.7496 ppl=15.64 bpb=0.9259 |
| tl nll=2.6593 ppl=14.29 bpb=1.2618 |
| en nll=2.8178 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 9200 01:26 (86.4s) |
| full_cpt step 9210/12852 ( 71.7%) nll=3.2385 ppl=25.50 grad=0.871 lr=5.40e-06 6856 tok/s 12.64s/it [18:17:49<12:47:25] mem=34.07GB |
| full_cpt step 9220/12852 ( 71.7%) nll=3.2284 ppl=25.24 grad=0.875 lr=5.38e-06 6857 tok/s 12.64s/it [18:19:45<12:45:11] mem=34.07GB |
| full_cpt step 9230/12852 ( 71.8%) nll=3.1938 ppl=24.38 grad=0.840 lr=5.37e-06 6859 tok/s 12.64s/it [18:21:41<12:42:58] mem=34.07GB |
| full_cpt step 9240/12852 ( 71.9%) nll=3.2554 ppl=25.93 grad=0.848 lr=5.35e-06 6860 tok/s 12.64s/it [18:23:37<12:40:44] mem=34.07GB |
| full_cpt step 9250/12852 ( 72.0%) nll=3.2754 ppl=26.45 grad=0.828 lr=5.33e-06 6861 tok/s 12.63s/it [18:25:33<12:38:30] mem=34.07GB |
| full_cpt step 9260/12852 ( 72.1%) nll=3.3610 ppl=28.82 grad=0.871 lr=5.31e-06 6862 tok/s 12.63s/it [18:27:28<12:36:17] mem=34.07GB |
| full_cpt step 9270/12852 ( 72.1%) nll=3.3201 ppl=27.66 grad=0.832 lr=5.30e-06 6863 tok/s 12.63s/it [18:29:24<12:34:04] mem=34.07GB |
| full_cpt step 9280/12852 ( 72.2%) nll=3.2430 ppl=25.61 grad=0.871 lr=5.28e-06 6864 tok/s 12.63s/it [18:31:21<12:31:50] mem=34.07GB |
| full_cpt step 9290/12852 ( 72.3%) nll=3.4122 ppl=30.33 grad=0.844 lr=5.26e-06 6865 tok/s 12.63s/it [18:33:16<12:29:37] mem=34.07GB |
| full_cpt step 9300/12852 ( 72.4%) nll=3.3032 ppl=27.20 grad=0.867 lr=5.24e-06 6867 tok/s 12.63s/it [18:35:13<12:27:24] mem=34.07GB |
| am nll=3.9695 ppl=52.96 bpb=1.3959 |
| ta nll=3.9432 ppl=51.59 bpb=0.5727 |
| kk nll=2.7491 ppl=15.63 bpb=0.9257 |
| tl nll=2.6593 ppl=14.29 bpb=1.2618 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 9300 01:26 (86.3s) |
| full_cpt step 9310/12852 ( 72.4%) nll=3.3678 ppl=29.01 grad=0.844 lr=5.23e-06 6859 tok/s 12.64s/it [18:38:35<12:26:09] mem=34.07GB |
| full_cpt step 9320/12852 ( 72.5%) nll=3.2698 ppl=26.31 grad=0.844 lr=5.21e-06 6860 tok/s 12.64s/it [18:40:31<12:23:55] mem=34.07GB |
| full_cpt step 9330/12852 ( 72.6%) nll=3.3573 ppl=28.71 grad=0.824 lr=5.19e-06 6861 tok/s 12.64s/it [18:42:27<12:21:42] mem=34.07GB |
| full_cpt step 9340/12852 ( 72.7%) nll=3.3734 ppl=29.18 grad=0.840 lr=5.18e-06 6862 tok/s 12.63s/it [18:44:23<12:19:29] mem=34.07GB |
| full_cpt step 9350/12852 ( 72.8%) nll=3.2741 ppl=26.42 grad=0.887 lr=5.16e-06 6863 tok/s 12.63s/it [18:46:18<12:17:15] mem=34.07GB |
| full_cpt step 9360/12852 ( 72.8%) nll=3.3592 ppl=28.77 grad=0.883 lr=5.14e-06 6864 tok/s 12.63s/it [18:48:14<12:15:02] mem=34.07GB |
| full_cpt step 9370/12852 ( 72.9%) nll=3.2532 ppl=25.87 grad=0.902 lr=5.13e-06 6865 tok/s 12.63s/it [18:50:10<12:12:49] mem=34.07GB |
| full_cpt step 9380/12852 ( 73.0%) nll=3.3081 ppl=27.33 grad=0.859 lr=5.11e-06 6866 tok/s 12.63s/it [18:52:06<12:10:36] mem=34.07GB |
| full_cpt step 9390/12852 ( 73.1%) nll=3.2580 ppl=26.00 grad=0.852 lr=5.09e-06 6867 tok/s 12.62s/it [18:54:02<12:08:23] mem=34.07GB |
| full_cpt step 9400/12852 ( 73.1%) nll=3.1491 ppl=23.32 grad=0.863 lr=5.08e-06 6868 tok/s 12.62s/it [18:55:58<12:06:10] mem=34.07GB |
| am nll=3.9692 ppl=52.94 bpb=1.3958 |
| ta nll=3.9430 ppl=51.57 bpb=0.5727 |
| kk nll=2.7494 ppl=15.63 bpb=0.9258 |
| tl nll=2.6592 ppl=14.28 bpb=1.2617 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 9400 01:26 (86.5s) |
| full_cpt step 9410/12852 ( 73.2%) nll=3.2305 ppl=25.29 grad=0.875 lr=5.06e-06 6860 tok/s 12.64s/it [18:59:20<12:04:53] mem=34.07GB |
| full_cpt step 9420/12852 ( 73.3%) nll=3.3492 ppl=28.48 grad=0.863 lr=5.04e-06 6861 tok/s 12.63s/it [19:01:16<12:02:40] mem=34.07GB |
| full_cpt step 9430/12852 ( 73.4%) nll=3.2282 ppl=25.23 grad=0.840 lr=5.03e-06 6862 tok/s 12.63s/it [19:03:12<12:00:27] mem=34.07GB |
| full_cpt step 9440/12852 ( 73.5%) nll=3.4221 ppl=30.63 grad=0.891 lr=5.01e-06 6864 tok/s 12.63s/it [19:05:08<11:58:14] mem=34.07GB |
| full_cpt step 9450/12852 ( 73.5%) nll=3.2776 ppl=26.51 grad=0.863 lr=4.99e-06 6864 tok/s 12.63s/it [19:07:04<11:56:01] mem=34.07GB |
| full_cpt step 9460/12852 ( 73.6%) nll=3.3667 ppl=28.98 grad=0.895 lr=4.98e-06 6865 tok/s 12.63s/it [19:09:00<11:53:49] mem=34.07GB |
| full_cpt step 9470/12852 ( 73.7%) nll=3.3592 ppl=28.77 grad=0.848 lr=4.96e-06 6866 tok/s 12.62s/it [19:10:56<11:51:36] mem=34.07GB |
| full_cpt step 9480/12852 ( 73.8%) nll=3.1893 ppl=24.27 grad=0.844 lr=4.94e-06 6867 tok/s 12.62s/it [19:12:52<11:49:23] mem=34.07GB |
| full_cpt step 9490/12852 ( 73.8%) nll=3.3568 ppl=28.70 grad=0.867 lr=4.93e-06 6869 tok/s 12.62s/it [19:14:48<11:47:11] mem=34.07GB |
| full_cpt step 9500/12852 ( 73.9%) nll=3.4310 ppl=30.91 grad=0.875 lr=4.91e-06 6870 tok/s 12.62s/it [19:16:44<11:44:58] mem=34.07GB |
| am nll=3.9689 ppl=52.93 bpb=1.3957 |
| ta nll=3.9430 ppl=51.57 bpb=0.5727 |
| kk nll=2.7491 ppl=15.63 bpb=0.9257 |
| tl nll=2.6590 ppl=14.28 bpb=1.2617 |
| en nll=2.8172 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 9500 01:26 (86.5s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.16s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.16s/it] |
| removing old checkpoint checkpoint-8500 |
| kept last 2 resume checkpoints: checkpoint-9000 checkpoint-9500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-9500 at step 9500 next_row=1216000 |
| full_cpt step 9510/12852 ( 74.0%) nll=3.3138 ppl=27.49 grad=0.875 lr=4.89e-06 6854 tok/s 12.65s/it [19:21:34<11:44:32] mem=34.07GB |
| full_cpt step 9520/12852 ( 74.1%) nll=3.3470 ppl=28.42 grad=0.832 lr=4.88e-06 6855 tok/s 12.65s/it [19:23:30<11:42:19] mem=34.07GB |
| full_cpt step 9530/12852 ( 74.2%) nll=3.3686 ppl=29.04 grad=0.852 lr=4.86e-06 6856 tok/s 12.64s/it [19:25:26<11:40:06] mem=34.07GB |
| full_cpt step 9540/12852 ( 74.2%) nll=3.1781 ppl=24.00 grad=0.918 lr=4.85e-06 6857 tok/s 12.64s/it [19:27:22<11:37:54] mem=34.07GB |
| full_cpt step 9550/12852 ( 74.3%) nll=3.3168 ppl=27.57 grad=0.891 lr=4.83e-06 6858 tok/s 12.64s/it [19:29:18<11:35:41] mem=34.07GB |
| full_cpt step 9560/12852 ( 74.4%) nll=3.2975 ppl=27.05 grad=0.859 lr=4.81e-06 6859 tok/s 12.64s/it [19:31:14<11:33:29] mem=34.07GB |
| full_cpt step 9570/12852 ( 74.5%) nll=3.3533 ppl=28.60 grad=0.867 lr=4.80e-06 6860 tok/s 12.64s/it [19:33:10<11:31:16] mem=34.07GB |
| full_cpt step 9580/12852 ( 74.5%) nll=3.1986 ppl=24.50 grad=0.895 lr=4.78e-06 6861 tok/s 12.64s/it [19:35:06<11:29:03] mem=34.07GB |
| full_cpt step 9590/12852 ( 74.6%) nll=3.4405 ppl=31.20 grad=0.895 lr=4.76e-06 6862 tok/s 12.63s/it [19:37:02<11:26:51] mem=34.07GB |
| full_cpt step 9600/12852 ( 74.7%) nll=3.2519 ppl=25.84 grad=0.809 lr=4.75e-06 6863 tok/s 12.63s/it [19:38:58<11:24:39] mem=34.07GB |
| am nll=3.9692 ppl=52.94 bpb=1.3958 |
| ta nll=3.9431 ppl=51.58 bpb=0.5727 |
| kk nll=2.7493 ppl=15.63 bpb=0.9258 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 9600 01:27 (86.7s) |
| full_cpt step 9610/12852 ( 74.8%) nll=3.2055 ppl=24.67 grad=0.887 lr=4.73e-06 6856 tok/s 12.65s/it [19:42:21<11:23:16] mem=34.07GB |
| full_cpt step 9620/12852 ( 74.9%) nll=3.3598 ppl=28.78 grad=0.832 lr=4.72e-06 6857 tok/s 12.64s/it [19:44:17<11:21:04] mem=34.07GB |
| full_cpt step 9630/12852 ( 74.9%) nll=3.2981 ppl=27.06 grad=0.891 lr=4.70e-06 6858 tok/s 12.64s/it [19:46:13<11:18:52] mem=34.07GB |
| full_cpt step 9640/12852 ( 75.0%) nll=3.3724 ppl=29.15 grad=0.828 lr=4.68e-06 6859 tok/s 12.64s/it [19:48:09<11:16:39] mem=34.07GB |
| full_cpt step 9650/12852 ( 75.1%) nll=3.3147 ppl=27.51 grad=0.875 lr=4.67e-06 6860 tok/s 12.64s/it [19:50:05<11:14:27] mem=34.07GB |
| full_cpt step 9660/12852 ( 75.2%) nll=3.2752 ppl=26.45 grad=0.859 lr=4.65e-06 6861 tok/s 12.64s/it [19:52:01<11:12:15] mem=34.07GB |
| full_cpt step 9670/12852 ( 75.2%) nll=3.3240 ppl=27.77 grad=0.875 lr=4.64e-06 6862 tok/s 12.63s/it [19:53:57<11:10:02] mem=34.07GB |
| full_cpt step 9680/12852 ( 75.3%) nll=3.2927 ppl=26.91 grad=0.867 lr=4.62e-06 6863 tok/s 12.63s/it [19:55:53<11:07:50] mem=34.07GB |
| full_cpt step 9690/12852 ( 75.4%) nll=3.3300 ppl=27.94 grad=0.883 lr=4.61e-06 6864 tok/s 12.63s/it [19:57:48<11:05:38] mem=34.07GB |
| full_cpt step 9700/12852 ( 75.5%) nll=3.2749 ppl=26.44 grad=0.871 lr=4.59e-06 6865 tok/s 12.63s/it [19:59:44<11:03:26] mem=34.07GB |
| am nll=3.9692 ppl=52.94 bpb=1.3958 |
| ta nll=3.9434 ppl=51.59 bpb=0.5727 |
| kk nll=2.7492 ppl=15.63 bpb=0.9258 |
| tl nll=2.6592 ppl=14.28 bpb=1.2617 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 9700 01:26 (86.2s) |
| full_cpt step 9710/12852 ( 75.6%) nll=3.2895 ppl=26.83 grad=0.809 lr=4.58e-06 6858 tok/s 12.64s/it [20:03:06<11:02:02] mem=34.07GB |
| full_cpt step 9720/12852 ( 75.6%) nll=3.2979 ppl=27.06 grad=0.875 lr=4.56e-06 6859 tok/s 12.64s/it [20:05:02<10:59:49] mem=34.07GB |
| full_cpt step 9730/12852 ( 75.7%) nll=3.3490 ppl=28.47 grad=0.887 lr=4.54e-06 6860 tok/s 12.64s/it [20:06:58<10:57:37] mem=34.07GB |
| full_cpt step 9740/12852 ( 75.8%) nll=3.3658 ppl=28.96 grad=0.914 lr=4.53e-06 6861 tok/s 12.64s/it [20:08:54<10:55:25] mem=34.07GB |
| full_cpt step 9750/12852 ( 75.9%) nll=3.4192 ppl=30.55 grad=0.859 lr=4.51e-06 6862 tok/s 12.63s/it [20:10:50<10:53:13] mem=34.07GB |
| full_cpt step 9760/12852 ( 75.9%) nll=3.3840 ppl=29.49 grad=0.953 lr=4.50e-06 6863 tok/s 12.63s/it [20:12:46<10:51:01] mem=34.07GB |
| full_cpt step 9770/12852 ( 76.0%) nll=3.2199 ppl=25.03 grad=0.871 lr=4.48e-06 6864 tok/s 12.63s/it [20:14:42<10:48:49] mem=34.07GB |
| full_cpt step 9780/12852 ( 76.1%) nll=3.3488 ppl=28.47 grad=0.879 lr=4.47e-06 6865 tok/s 12.63s/it [20:16:38<10:46:38] mem=34.07GB |
| full_cpt step 9790/12852 ( 76.2%) nll=3.2712 ppl=26.34 grad=0.898 lr=4.45e-06 6865 tok/s 12.63s/it [20:18:34<10:44:26] mem=34.07GB |
| full_cpt step 9800/12852 ( 76.3%) nll=3.2652 ppl=26.19 grad=0.902 lr=4.44e-06 6866 tok/s 12.63s/it [20:20:30<10:42:14] mem=34.07GB |
| am nll=3.9691 ppl=52.94 bpb=1.3957 |
| ta nll=3.9430 ppl=51.58 bpb=0.5727 |
| kk nll=2.7494 ppl=15.63 bpb=0.9258 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8178 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 9800 01:27 (86.7s) |
| full_cpt step 9810/12852 ( 76.3%) nll=3.2995 ppl=27.10 grad=0.914 lr=4.42e-06 6859 tok/s 12.64s/it [20:23:52<10:40:48] mem=34.07GB |
| full_cpt step 9820/12852 ( 76.4%) nll=3.3823 ppl=29.44 grad=0.883 lr=4.41e-06 6860 tok/s 12.64s/it [20:25:48<10:38:36] mem=34.07GB |
| full_cpt step 9830/12852 ( 76.5%) nll=3.3130 ppl=27.47 grad=0.859 lr=4.39e-06 6861 tok/s 12.64s/it [20:27:44<10:36:24] mem=34.07GB |
| full_cpt step 9840/12852 ( 76.6%) nll=3.2611 ppl=26.08 grad=0.883 lr=4.38e-06 6862 tok/s 12.63s/it [20:29:40<10:34:12] mem=34.07GB |
| full_cpt step 9850/12852 ( 76.6%) nll=3.3671 ppl=28.99 grad=0.855 lr=4.36e-06 6863 tok/s 12.63s/it [20:31:36<10:32:01] mem=34.07GB |
| full_cpt step 9860/12852 ( 76.7%) nll=3.3158 ppl=27.54 grad=0.840 lr=4.35e-06 6864 tok/s 12.63s/it [20:33:32<10:29:49] mem=34.07GB |
| full_cpt step 9870/12852 ( 76.8%) nll=3.3485 ppl=28.46 grad=0.879 lr=4.33e-06 6865 tok/s 12.63s/it [20:35:28<10:27:38] mem=34.07GB |
| full_cpt step 9880/12852 ( 76.9%) nll=3.3031 ppl=27.20 grad=0.824 lr=4.32e-06 6866 tok/s 12.63s/it [20:37:24<10:25:26] mem=34.07GB |
| full_cpt step 9890/12852 ( 77.0%) nll=3.4453 ppl=31.35 grad=0.863 lr=4.30e-06 6867 tok/s 12.62s/it [20:39:20<10:23:15] mem=34.07GB |
| full_cpt step 9900/12852 ( 77.0%) nll=3.3408 ppl=28.24 grad=0.871 lr=4.29e-06 6868 tok/s 12.62s/it [20:41:16<10:21:03] mem=34.07GB |
| am nll=3.9693 ppl=52.95 bpb=1.3958 |
| ta nll=3.9431 ppl=51.58 bpb=0.5727 |
| kk nll=2.7493 ppl=15.63 bpb=0.9258 |
| tl nll=2.6593 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 9900 01:27 (86.6s) |
| full_cpt step 9910/12852 ( 77.1%) nll=3.1985 ppl=24.50 grad=0.855 lr=4.27e-06 6861 tok/s 12.64s/it [20:44:38<10:19:35] mem=34.07GB |
| full_cpt step 9920/12852 ( 77.2%) nll=3.2777 ppl=26.52 grad=0.898 lr=4.26e-06 6862 tok/s 12.63s/it [20:46:34<10:17:23] mem=34.07GB |
| full_cpt step 9930/12852 ( 77.3%) nll=3.3016 ppl=27.16 grad=0.828 lr=4.24e-06 6863 tok/s 12.63s/it [20:48:30<10:15:12] mem=34.07GB |
| full_cpt step 9940/12852 ( 77.3%) nll=3.3257 ppl=27.82 grad=0.941 lr=4.23e-06 6863 tok/s 12.63s/it [20:50:26<10:13:00] mem=34.07GB |
| full_cpt step 9950/12852 ( 77.4%) nll=3.3342 ppl=28.06 grad=0.945 lr=4.21e-06 6864 tok/s 12.63s/it [20:52:22<10:10:49] mem=34.07GB |
| full_cpt step 9960/12852 ( 77.5%) nll=3.1883 ppl=24.25 grad=0.875 lr=4.20e-06 6865 tok/s 12.63s/it [20:54:18<10:08:38] mem=34.07GB |
| full_cpt step 9970/12852 ( 77.6%) nll=3.3295 ppl=27.92 grad=0.902 lr=4.18e-06 6866 tok/s 12.63s/it [20:56:14<10:06:27] mem=34.07GB |
| full_cpt step 9980/12852 ( 77.7%) nll=3.2195 ppl=25.02 grad=0.840 lr=4.17e-06 6867 tok/s 12.62s/it [20:58:10<10:04:15] mem=34.07GB |
| full_cpt step 9990/12852 ( 77.7%) nll=3.2803 ppl=26.58 grad=0.871 lr=4.16e-06 6868 tok/s 12.62s/it [21:00:06<10:02:04] mem=34.07GB |
| full_cpt step 10000/12852 ( 77.8%) nll=3.3804 ppl=29.38 grad=0.848 lr=4.14e-06 6869 tok/s 12.62s/it [21:02:02<9:59:53] mem=34.07GB |
| am nll=3.9692 ppl=52.94 bpb=1.3958 |
| ta nll=3.9430 ppl=51.58 bpb=0.5727 |
| kk nll=2.7492 ppl=15.63 bpb=0.9258 |
| tl nll=2.6595 ppl=14.29 bpb=1.2619 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 10000 01:27 (86.5s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.26s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.26s/it] |
| removing old checkpoint checkpoint-9000 |
| kept last 2 resume checkpoints: checkpoint-9500 checkpoint-10000 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-10000 at step 10000 next_row=1280000 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.53s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.53s/it] |
| wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-10000 (weights+expanded tokenizer, no optimizer) |
| full_cpt step 10010/12852 ( 77.9%) nll=3.2738 ppl=26.41 grad=0.910 lr=4.13e-06 6855 tok/s 12.65s/it [21:06:44<9:59:01] mem=34.07GB |
| full_cpt step 10020/12852 ( 78.0%) nll=3.2301 ppl=25.28 grad=0.961 lr=4.11e-06 6856 tok/s 12.64s/it [21:08:40<9:56:49] mem=34.07GB |
| full_cpt step 10030/12852 ( 78.0%) nll=3.2540 ppl=25.89 grad=0.918 lr=4.10e-06 6857 tok/s 12.64s/it [21:10:36<9:54:38] mem=34.07GB |
| full_cpt step 10040/12852 ( 78.1%) nll=3.3511 ppl=28.54 grad=0.812 lr=4.08e-06 6858 tok/s 12.64s/it [21:12:32<9:52:27] mem=34.07GB |
| full_cpt step 10050/12852 ( 78.2%) nll=3.2198 ppl=25.02 grad=0.867 lr=4.07e-06 6858 tok/s 12.64s/it [21:14:28<9:50:15] mem=34.07GB |
| full_cpt step 10060/12852 ( 78.3%) nll=3.2543 ppl=25.90 grad=0.891 lr=4.06e-06 6859 tok/s 12.64s/it [21:16:24<9:48:04] mem=34.07GB |
| full_cpt step 10070/12852 ( 78.4%) nll=3.2844 ppl=26.69 grad=0.828 lr=4.04e-06 6860 tok/s 12.64s/it [21:18:20<9:45:53] mem=34.07GB |
| full_cpt step 10080/12852 ( 78.4%) nll=3.2946 ppl=26.97 grad=0.840 lr=4.03e-06 6861 tok/s 12.63s/it [21:20:16<9:43:42] mem=34.07GB |
| full_cpt step 10090/12852 ( 78.5%) nll=3.2286 ppl=25.25 grad=0.938 lr=4.01e-06 6861 tok/s 12.63s/it [21:22:11<9:41:31] mem=34.07GB |
| full_cpt step 10100/12852 ( 78.6%) nll=3.3637 ppl=28.89 grad=0.891 lr=4.00e-06 6862 tok/s 12.63s/it [21:24:07<9:39:20] mem=34.07GB |
| am nll=3.9690 ppl=52.93 bpb=1.3957 |
| ta nll=3.9429 ppl=51.57 bpb=0.5727 |
| kk nll=2.7493 ppl=15.63 bpb=0.9258 |
| tl nll=2.6592 ppl=14.28 bpb=1.2617 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 10100 01:26 (86.2s) |
| full_cpt step 10110/12852 ( 78.7%) nll=3.3598 ppl=28.78 grad=0.855 lr=3.98e-06 6855 tok/s 12.64s/it [21:27:30<9:37:47] mem=34.07GB |
| full_cpt step 10120/12852 ( 78.7%) nll=3.2096 ppl=24.77 grad=0.855 lr=3.97e-06 6856 tok/s 12.64s/it [21:29:25<9:35:36] mem=34.07GB |
| full_cpt step 10130/12852 ( 78.8%) nll=3.3657 ppl=28.95 grad=0.867 lr=3.96e-06 6857 tok/s 12.64s/it [21:31:21<9:33:25] mem=34.07GB |
| full_cpt step 10140/12852 ( 78.9%) nll=3.2137 ppl=24.87 grad=0.898 lr=3.94e-06 6858 tok/s 12.64s/it [21:33:17<9:31:14] mem=34.07GB |
| full_cpt step 10150/12852 ( 79.0%) nll=3.3429 ppl=28.30 grad=0.863 lr=3.93e-06 6859 tok/s 12.64s/it [21:35:13<9:29:03] mem=34.07GB |
| full_cpt step 10160/12852 ( 79.1%) nll=3.2481 ppl=25.74 grad=0.879 lr=3.92e-06 6860 tok/s 12.63s/it [21:37:09<9:26:52] mem=34.07GB |
| full_cpt step 10170/12852 ( 79.1%) nll=3.3288 ppl=27.90 grad=0.895 lr=3.90e-06 6861 tok/s 12.63s/it [21:39:05<9:24:42] mem=34.07GB |
| full_cpt step 10180/12852 ( 79.2%) nll=3.4284 ppl=30.83 grad=0.879 lr=3.89e-06 6862 tok/s 12.63s/it [21:41:01<9:22:31] mem=34.07GB |
| full_cpt step 10190/12852 ( 79.3%) nll=3.2280 ppl=25.23 grad=0.891 lr=3.87e-06 6863 tok/s 12.63s/it [21:42:57<9:20:20] mem=34.07GB |
| full_cpt step 10200/12852 ( 79.4%) nll=3.2800 ppl=26.58 grad=0.848 lr=3.86e-06 6863 tok/s 12.63s/it [21:44:53<9:18:09] mem=34.07GB |
| am nll=3.9692 ppl=52.94 bpb=1.3958 |
| ta nll=3.9428 ppl=51.56 bpb=0.5727 |
| kk nll=2.7492 ppl=15.63 bpb=0.9258 |
| tl nll=2.6593 ppl=14.29 bpb=1.2618 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 10200 01:27 (86.9s) |
| full_cpt step 10210/12852 ( 79.4%) nll=3.2334 ppl=25.37 grad=0.906 lr=3.85e-06 6857 tok/s 12.64s/it [21:48:16<9:16:35] mem=34.07GB |
| full_cpt step 10220/12852 ( 79.5%) nll=3.2003 ppl=24.54 grad=0.875 lr=3.83e-06 6857 tok/s 12.64s/it [21:50:11<9:14:25] mem=34.07GB |
| full_cpt step 10230/12852 ( 79.6%) nll=3.3162 ppl=27.56 grad=0.895 lr=3.82e-06 6858 tok/s 12.64s/it [21:52:07<9:12:14] mem=34.07GB |
| full_cpt step 10240/12852 ( 79.7%) nll=3.3679 ppl=29.02 grad=0.840 lr=3.81e-06 6859 tok/s 12.64s/it [21:54:03<9:10:03] mem=34.07GB |
| full_cpt step 10250/12852 ( 79.8%) nll=3.3755 ppl=29.24 grad=0.883 lr=3.79e-06 6860 tok/s 12.63s/it [21:55:59<9:07:52] mem=34.07GB |
| full_cpt step 10260/12852 ( 79.8%) nll=3.4084 ppl=30.22 grad=0.879 lr=3.78e-06 6861 tok/s 12.63s/it [21:57:55<9:05:42] mem=34.07GB |
| full_cpt step 10270/12852 ( 79.9%) nll=3.2672 ppl=26.24 grad=0.875 lr=3.77e-06 6862 tok/s 12.63s/it [21:59:51<9:03:31] mem=34.07GB |
| full_cpt step 10280/12852 ( 80.0%) nll=3.1289 ppl=22.85 grad=0.879 lr=3.75e-06 6863 tok/s 12.63s/it [22:01:47<9:01:21] mem=34.07GB |
| full_cpt step 10290/12852 ( 80.1%) nll=3.2088 ppl=24.75 grad=0.863 lr=3.74e-06 6864 tok/s 12.63s/it [22:03:43<8:59:10] mem=34.07GB |
| full_cpt step 10300/12852 ( 80.1%) nll=3.3692 ppl=29.06 grad=0.867 lr=3.73e-06 6865 tok/s 12.63s/it [22:05:39<8:57:00] mem=34.07GB |
| am nll=3.9691 ppl=52.93 bpb=1.3957 |
| ta nll=3.9427 ppl=51.56 bpb=0.5726 |
| kk nll=2.7497 ppl=15.64 bpb=0.9259 |
| tl nll=2.6592 ppl=14.28 bpb=1.2617 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 10300 01:27 (87.1s) |
| full_cpt step 10310/12852 ( 80.2%) nll=3.3257 ppl=27.82 grad=0.879 lr=3.72e-06 6858 tok/s 12.64s/it [22:09:02<8:55:24] mem=34.07GB |
| full_cpt step 10320/12852 ( 80.3%) nll=3.2483 ppl=25.75 grad=0.840 lr=3.70e-06 6859 tok/s 12.64s/it [22:10:58<8:53:14] mem=34.07GB |
| full_cpt step 10330/12852 ( 80.4%) nll=3.2974 ppl=27.04 grad=0.867 lr=3.69e-06 6860 tok/s 12.63s/it [22:12:54<8:51:03] mem=34.07GB |
| full_cpt step 10340/12852 ( 80.5%) nll=3.2313 ppl=25.31 grad=0.824 lr=3.68e-06 6861 tok/s 12.63s/it [22:14:50<8:48:53] mem=34.07GB |
| full_cpt step 10350/12852 ( 80.5%) nll=3.4022 ppl=30.03 grad=0.863 lr=3.66e-06 6861 tok/s 12.63s/it [22:16:46<8:46:42] mem=34.07GB |
| full_cpt step 10360/12852 ( 80.6%) nll=3.3053 ppl=27.26 grad=0.859 lr=3.65e-06 6862 tok/s 12.63s/it [22:18:42<8:44:32] mem=34.07GB |
| full_cpt step 10370/12852 ( 80.7%) nll=3.3111 ppl=27.41 grad=0.883 lr=3.64e-06 6863 tok/s 12.63s/it [22:20:38<8:42:22] mem=34.07GB |
| full_cpt step 10380/12852 ( 80.8%) nll=3.2821 ppl=26.63 grad=0.855 lr=3.62e-06 6864 tok/s 12.63s/it [22:22:34<8:40:12] mem=34.07GB |
| full_cpt step 10390/12852 ( 80.8%) nll=3.3771 ppl=29.29 grad=0.898 lr=3.61e-06 6865 tok/s 12.62s/it [22:24:30<8:38:01] mem=34.07GB |
| full_cpt step 10400/12852 ( 80.9%) nll=3.2285 ppl=25.24 grad=0.832 lr=3.60e-06 6866 tok/s 12.62s/it [22:26:26<8:35:51] mem=34.07GB |
| am nll=3.9692 ppl=52.94 bpb=1.3958 |
| ta nll=3.9424 ppl=51.54 bpb=0.5726 |
| kk nll=2.7492 ppl=15.63 bpb=0.9258 |
| tl nll=2.6594 ppl=14.29 bpb=1.2619 |
| en nll=2.8178 ppl=16.74 bpb=0.8774 |
| eval full_cpt step 10400 01:27 (87.0s) |
| full_cpt step 10410/12852 ( 81.0%) nll=3.2760 ppl=26.47 grad=0.816 lr=3.59e-06 6860 tok/s 12.63s/it [22:29:49<8:34:14] mem=34.07GB |
| full_cpt step 10420/12852 ( 81.1%) nll=3.4020 ppl=30.02 grad=0.840 lr=3.57e-06 6860 tok/s 12.63s/it [22:31:45<8:32:04] mem=34.07GB |
| full_cpt step 10430/12852 ( 81.2%) nll=3.2908 ppl=26.86 grad=0.930 lr=3.56e-06 6861 tok/s 12.63s/it [22:33:41<8:29:54] mem=34.07GB |
| full_cpt step 10440/12852 ( 81.2%) nll=3.2738 ppl=26.41 grad=0.875 lr=3.55e-06 6862 tok/s 12.63s/it [22:35:37<8:27:43] mem=34.07GB |
| full_cpt step 10450/12852 ( 81.3%) nll=3.4129 ppl=30.35 grad=0.875 lr=3.54e-06 6863 tok/s 12.63s/it [22:37:33<8:25:33] mem=34.07GB |
| full_cpt step 10460/12852 ( 81.4%) nll=3.2423 ppl=25.59 grad=0.848 lr=3.52e-06 6864 tok/s 12.63s/it [22:39:29<8:23:23] mem=34.07GB |
| full_cpt step 10470/12852 ( 81.5%) nll=3.2920 ppl=26.90 grad=0.922 lr=3.51e-06 6865 tok/s 12.63s/it [22:41:25<8:21:13] mem=34.07GB |
| full_cpt step 10480/12852 ( 81.5%) nll=3.3300 ppl=27.94 grad=0.898 lr=3.50e-06 6866 tok/s 12.62s/it [22:43:21<8:19:03] mem=34.07GB |
| full_cpt step 10490/12852 ( 81.6%) nll=3.2706 ppl=26.33 grad=0.898 lr=3.49e-06 6867 tok/s 12.62s/it [22:45:16<8:16:53] mem=34.07GB |
| full_cpt step 10500/12852 ( 81.7%) nll=3.2467 ppl=25.70 grad=0.855 lr=3.48e-06 6867 tok/s 12.62s/it [22:47:12<8:14:43] mem=34.07GB |
| am nll=3.9693 ppl=52.95 bpb=1.3958 |
| ta nll=3.9426 ppl=51.55 bpb=0.5726 |
| kk nll=2.7495 ppl=15.64 bpb=0.9259 |
| tl nll=2.6593 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 10500 01:27 (86.9s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.29s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.30s/it] |
| removing old checkpoint checkpoint-9500 |
| kept last 2 resume checkpoints: checkpoint-10000 checkpoint-10500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-10500 at step 10500 next_row=1344000 |
| full_cpt step 10510/12852 ( 81.8%) nll=3.2198 ppl=25.02 grad=0.902 lr=3.46e-06 6858 tok/s 12.64s/it [22:51:12<8:13:18] mem=34.07GB |
| full_cpt step 10520/12852 ( 81.9%) nll=3.3195 ppl=27.65 grad=0.914 lr=3.45e-06 6859 tok/s 12.64s/it [22:53:08<8:11:08] mem=34.07GB |
| full_cpt step 10530/12852 ( 81.9%) nll=3.3433 ppl=28.31 grad=0.895 lr=3.44e-06 6860 tok/s 12.63s/it [22:55:04<8:08:58] mem=34.07GB |
| full_cpt step 10540/12852 ( 82.0%) nll=3.3988 ppl=29.93 grad=0.863 lr=3.43e-06 6861 tok/s 12.63s/it [22:57:00<8:06:48] mem=34.07GB |
| full_cpt step 10550/12852 ( 82.1%) nll=3.3007 ppl=27.13 grad=0.863 lr=3.41e-06 6862 tok/s 12.63s/it [22:58:56<8:04:38] mem=34.07GB |
| full_cpt step 10560/12852 ( 82.2%) nll=3.1874 ppl=24.23 grad=0.855 lr=3.40e-06 6862 tok/s 12.63s/it [23:00:52<8:02:28] mem=34.07GB |
| full_cpt step 10570/12852 ( 82.2%) nll=3.4589 ppl=31.78 grad=0.871 lr=3.39e-06 6863 tok/s 12.63s/it [23:02:48<8:00:18] mem=34.07GB |
| full_cpt step 10580/12852 ( 82.3%) nll=3.3617 ppl=28.84 grad=0.828 lr=3.38e-06 6864 tok/s 12.63s/it [23:04:44<7:58:08] mem=34.07GB |
| full_cpt step 10590/12852 ( 82.4%) nll=3.3878 ppl=29.60 grad=0.938 lr=3.37e-06 6865 tok/s 12.63s/it [23:06:40<7:55:58] mem=34.07GB |
| full_cpt step 10600/12852 ( 82.5%) nll=3.1713 ppl=23.84 grad=0.844 lr=3.36e-06 6866 tok/s 12.62s/it [23:08:36<7:53:48] mem=34.07GB |
| am nll=3.9690 ppl=52.93 bpb=1.3957 |
| ta nll=3.9428 ppl=51.56 bpb=0.5727 |
| kk nll=2.7491 ppl=15.63 bpb=0.9258 |
| tl nll=2.6593 ppl=14.29 bpb=1.2618 |
| en nll=2.8179 ppl=16.74 bpb=0.8774 |
| eval full_cpt step 10600 01:26 (86.3s) |
| full_cpt step 10610/12852 ( 82.6%) nll=3.2706 ppl=26.33 grad=0.934 lr=3.34e-06 6860 tok/s 12.64s/it [23:11:58<7:52:08] mem=34.07GB |
| full_cpt step 10620/12852 ( 82.6%) nll=3.3724 ppl=29.15 grad=0.871 lr=3.33e-06 6861 tok/s 12.63s/it [23:13:54<7:49:58] mem=34.07GB |
| full_cpt step 10630/12852 ( 82.7%) nll=3.3626 ppl=28.86 grad=0.875 lr=3.32e-06 6862 tok/s 12.63s/it [23:15:50<7:47:48] mem=34.07GB |
| full_cpt step 10640/12852 ( 82.8%) nll=3.3617 ppl=28.84 grad=0.891 lr=3.31e-06 6863 tok/s 12.63s/it [23:17:46<7:45:39] mem=34.07GB |
| full_cpt step 10650/12852 ( 82.9%) nll=3.3686 ppl=29.04 grad=0.836 lr=3.30e-06 6864 tok/s 12.63s/it [23:19:42<7:43:29] mem=34.07GB |
| full_cpt step 10660/12852 ( 82.9%) nll=3.3623 ppl=28.86 grad=0.820 lr=3.29e-06 6865 tok/s 12.63s/it [23:21:38<7:41:19] mem=34.07GB |
| full_cpt step 10670/12852 ( 83.0%) nll=3.3187 ppl=27.63 grad=0.945 lr=3.27e-06 6865 tok/s 12.63s/it [23:23:34<7:39:10] mem=34.07GB |
| full_cpt step 10680/12852 ( 83.1%) nll=3.3234 ppl=27.75 grad=0.879 lr=3.26e-06 6866 tok/s 12.62s/it [23:25:30<7:37:00] mem=34.07GB |
| full_cpt step 10690/12852 ( 83.2%) nll=3.4001 ppl=29.97 grad=0.883 lr=3.25e-06 6867 tok/s 12.62s/it [23:27:26<7:34:50] mem=34.07GB |
| full_cpt step 10700/12852 ( 83.3%) nll=3.4171 ppl=30.48 grad=0.828 lr=3.24e-06 6868 tok/s 12.62s/it [23:29:22<7:32:41] mem=34.07GB |
| am nll=3.9692 ppl=52.94 bpb=1.3958 |
| ta nll=3.9432 ppl=51.58 bpb=0.5727 |
| kk nll=2.7491 ppl=15.63 bpb=0.9258 |
| tl nll=2.6592 ppl=14.28 bpb=1.2617 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 10700 01:26 (86.3s) |
| full_cpt step 10710/12852 ( 83.3%) nll=3.2121 ppl=24.83 grad=0.812 lr=3.23e-06 6862 tok/s 12.63s/it [23:32:44<7:30:59] mem=34.07GB |
| full_cpt step 10720/12852 ( 83.4%) nll=3.3478 ppl=28.44 grad=0.855 lr=3.22e-06 6862 tok/s 12.63s/it [23:34:40<7:28:49] mem=34.07GB |
| full_cpt step 10730/12852 ( 83.5%) nll=3.3152 ppl=27.53 grad=0.902 lr=3.21e-06 6863 tok/s 12.63s/it [23:36:36<7:26:40] mem=34.07GB |
| full_cpt step 10740/12852 ( 83.6%) nll=3.3570 ppl=28.70 grad=0.895 lr=3.20e-06 6864 tok/s 12.63s/it [23:38:32<7:24:30] mem=34.07GB |
| full_cpt step 10750/12852 ( 83.6%) nll=3.3583 ppl=28.74 grad=0.922 lr=3.19e-06 6865 tok/s 12.63s/it [23:40:29<7:22:21] mem=34.07GB |
| full_cpt step 10760/12852 ( 83.7%) nll=3.1867 ppl=24.21 grad=0.867 lr=3.17e-06 6865 tok/s 12.62s/it [23:42:25<7:20:11] mem=34.07GB |
| full_cpt step 10770/12852 ( 83.8%) nll=3.3477 ppl=28.44 grad=0.875 lr=3.16e-06 6866 tok/s 12.62s/it [23:44:21<7:18:02] mem=34.07GB |
| full_cpt step 10780/12852 ( 83.9%) nll=3.2901 ppl=26.85 grad=0.852 lr=3.15e-06 6867 tok/s 12.62s/it [23:46:16<7:15:53] mem=34.07GB |
| full_cpt step 10790/12852 ( 84.0%) nll=3.3143 ppl=27.50 grad=0.848 lr=3.14e-06 6867 tok/s 12.62s/it [23:48:12<7:13:43] mem=34.07GB |
| full_cpt step 10800/12852 ( 84.0%) nll=3.3161 ppl=27.55 grad=0.879 lr=3.13e-06 6868 tok/s 12.62s/it [23:50:08<7:11:34] mem=34.07GB |
| am nll=3.9691 ppl=52.94 bpb=1.3957 |
| ta nll=3.9430 ppl=51.57 bpb=0.5727 |
| kk nll=2.7495 ppl=15.63 bpb=0.9259 |
| tl nll=2.6593 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 10800 01:27 (86.6s) |
| full_cpt step 10810/12852 ( 84.1%) nll=3.3150 ppl=27.52 grad=0.820 lr=3.12e-06 6862 tok/s 12.63s/it [23:53:31<7:09:51] mem=34.07GB |
| full_cpt step 10820/12852 ( 84.2%) nll=3.2914 ppl=26.88 grad=0.891 lr=3.11e-06 6863 tok/s 12.63s/it [23:55:27<7:07:41] mem=34.07GB |
| full_cpt step 10830/12852 ( 84.3%) nll=3.3773 ppl=29.29 grad=0.879 lr=3.10e-06 6864 tok/s 12.63s/it [23:57:23<7:05:32] mem=34.07GB |
| full_cpt step 10840/12852 ( 84.3%) nll=3.2342 ppl=25.39 grad=0.844 lr=3.09e-06 6865 tok/s 12.63s/it [23:59:19<7:03:23] mem=34.07GB |
| full_cpt step 10850/12852 ( 84.4%) nll=3.2465 ppl=25.70 grad=0.910 lr=3.08e-06 6866 tok/s 12.62s/it [24:01:15<7:01:13] mem=34.07GB |
| full_cpt step 10860/12852 ( 84.5%) nll=3.3472 ppl=28.42 grad=0.875 lr=3.07e-06 6866 tok/s 12.62s/it [24:03:11<6:59:04] mem=34.07GB |
| full_cpt step 10870/12852 ( 84.6%) nll=3.2407 ppl=25.55 grad=0.832 lr=3.06e-06 6867 tok/s 12.62s/it [24:05:07<6:56:55] mem=34.07GB |
| full_cpt step 10880/12852 ( 84.7%) nll=3.2950 ppl=26.98 grad=0.891 lr=3.05e-06 6868 tok/s 12.62s/it [24:07:03<6:54:46] mem=34.07GB |
| full_cpt step 10890/12852 ( 84.7%) nll=3.2224 ppl=25.09 grad=0.922 lr=3.04e-06 6869 tok/s 12.62s/it [24:08:59<6:52:37] mem=34.07GB |
| full_cpt step 10900/12852 ( 84.8%) nll=3.2434 ppl=25.62 grad=0.836 lr=3.03e-06 6870 tok/s 12.62s/it [24:10:55<6:50:28] mem=34.07GB |
| am nll=3.9691 ppl=52.94 bpb=1.3957 |
| ta nll=3.9434 ppl=51.60 bpb=0.5728 |
| kk nll=2.7494 ppl=15.63 bpb=0.9258 |
| tl nll=2.6592 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 10900 01:27 (86.9s) |
| full_cpt step 10910/12852 ( 84.9%) nll=3.3836 ppl=29.48 grad=0.832 lr=3.01e-06 6864 tok/s 12.63s/it [24:14:17<6:48:43] mem=34.07GB |
| full_cpt step 10920/12852 ( 85.0%) nll=3.2734 ppl=26.40 grad=0.871 lr=3.00e-06 6865 tok/s 12.63s/it [24:16:13<6:46:34] mem=34.07GB |
| full_cpt step 10930/12852 ( 85.0%) nll=3.3069 ppl=27.30 grad=0.852 lr=2.99e-06 6865 tok/s 12.62s/it [24:18:09<6:44:25] mem=34.07GB |
| full_cpt step 10940/12852 ( 85.1%) nll=3.4619 ppl=31.88 grad=0.891 lr=2.98e-06 6866 tok/s 12.62s/it [24:20:05<6:42:16] mem=34.07GB |
| full_cpt step 10950/12852 ( 85.2%) nll=3.3695 ppl=29.06 grad=0.887 lr=2.97e-06 6867 tok/s 12.62s/it [24:22:01<6:40:07] mem=34.07GB |
| full_cpt step 10960/12852 ( 85.3%) nll=3.2327 ppl=25.35 grad=0.828 lr=2.96e-06 6868 tok/s 12.62s/it [24:23:57<6:37:58] mem=34.07GB |
| full_cpt step 10970/12852 ( 85.4%) nll=3.2898 ppl=26.84 grad=0.855 lr=2.95e-06 6869 tok/s 12.62s/it [24:25:53<6:35:49] mem=34.07GB |
| full_cpt step 10980/12852 ( 85.4%) nll=3.2185 ppl=24.99 grad=0.871 lr=2.94e-06 6869 tok/s 12.62s/it [24:27:49<6:33:40] mem=34.07GB |
| full_cpt step 10990/12852 ( 85.5%) nll=3.1923 ppl=24.34 grad=0.918 lr=2.93e-06 6870 tok/s 12.62s/it [24:29:45<6:31:31] mem=34.07GB |
| full_cpt step 11000/12852 ( 85.6%) nll=3.3846 ppl=29.51 grad=0.852 lr=2.92e-06 6871 tok/s 12.61s/it [24:31:41<6:29:22] mem=34.07GB |
| am nll=3.9688 ppl=52.92 bpb=1.3956 |
| ta nll=3.9428 ppl=51.56 bpb=0.5727 |
| kk nll=2.7493 ppl=15.63 bpb=0.9258 |
| tl nll=2.6596 ppl=14.29 bpb=1.2619 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 11000 01:26 (86.5s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.20s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.20s/it] |
| removing old checkpoint checkpoint-10000 |
| kept last 2 resume checkpoints: checkpoint-10500 checkpoint-11000 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-11000 at step 11000 next_row=1408000 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.44s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.44s/it] |
| wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-11000 (weights+expanded tokenizer, no optimizer) |
| full_cpt step 11010/12852 ( 85.7%) nll=3.3893 ppl=29.64 grad=0.910 lr=2.91e-06 6861 tok/s 12.63s/it [24:35:57<6:27:50] mem=34.07GB |
| full_cpt step 11020/12852 ( 85.7%) nll=3.3378 ppl=28.16 grad=0.855 lr=2.91e-06 6862 tok/s 12.63s/it [24:37:53<6:25:41] mem=34.07GB |
| full_cpt step 11030/12852 ( 85.8%) nll=3.3418 ppl=28.27 grad=0.875 lr=2.90e-06 6863 tok/s 12.63s/it [24:39:49<6:23:32] mem=34.07GB |
| full_cpt step 11040/12852 ( 85.9%) nll=3.2500 ppl=25.79 grad=0.895 lr=2.89e-06 6863 tok/s 12.63s/it [24:41:45<6:21:23] mem=34.07GB |
| full_cpt step 11050/12852 ( 86.0%) nll=3.3929 ppl=29.75 grad=0.859 lr=2.88e-06 6864 tok/s 12.63s/it [24:43:41<6:19:14] mem=34.07GB |
| full_cpt step 11060/12852 ( 86.1%) nll=3.2196 ppl=25.02 grad=0.871 lr=2.87e-06 6865 tok/s 12.63s/it [24:45:37<6:17:05] mem=34.07GB |
| full_cpt step 11070/12852 ( 86.1%) nll=3.3425 ppl=28.29 grad=0.914 lr=2.86e-06 6865 tok/s 12.62s/it [24:47:33<6:14:56] mem=34.07GB |
| full_cpt step 11080/12852 ( 86.2%) nll=3.3355 ppl=28.09 grad=0.855 lr=2.85e-06 6866 tok/s 12.62s/it [24:49:29<6:12:47] mem=34.07GB |
| full_cpt step 11090/12852 ( 86.3%) nll=3.2289 ppl=25.25 grad=0.852 lr=2.84e-06 6867 tok/s 12.62s/it [24:51:24<6:10:39] mem=34.07GB |
| full_cpt step 11100/12852 ( 86.4%) nll=3.2911 ppl=26.87 grad=0.820 lr=2.83e-06 6867 tok/s 12.62s/it [24:53:20<6:08:30] mem=34.07GB |
| am nll=3.9689 ppl=52.93 bpb=1.3957 |
| ta nll=3.9433 ppl=51.59 bpb=0.5727 |
| kk nll=2.7496 ppl=15.64 bpb=0.9259 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 11100 01:26 (86.2s) |
| full_cpt step 11110/12852 ( 86.4%) nll=3.4101 ppl=30.27 grad=0.906 lr=2.82e-06 6862 tok/s 12.63s/it [24:56:42<6:06:42] mem=34.07GB |
| full_cpt step 11120/12852 ( 86.5%) nll=3.3549 ppl=28.64 grad=0.855 lr=2.81e-06 6862 tok/s 12.63s/it [24:58:38<6:04:33] mem=34.07GB |
| full_cpt step 11130/12852 ( 86.6%) nll=3.3015 ppl=27.15 grad=0.898 lr=2.80e-06 6863 tok/s 12.63s/it [25:00:34<6:02:25] mem=34.07GB |
| full_cpt step 11140/12852 ( 86.7%) nll=3.3322 ppl=28.00 grad=0.848 lr=2.79e-06 6864 tok/s 12.63s/it [25:02:30<6:00:16] mem=34.07GB |
| full_cpt step 11150/12852 ( 86.8%) nll=3.1508 ppl=23.35 grad=0.844 lr=2.78e-06 6865 tok/s 12.62s/it [25:04:26<5:58:07] mem=34.07GB |
| full_cpt step 11160/12852 ( 86.8%) nll=3.3436 ppl=28.32 grad=0.852 lr=2.77e-06 6865 tok/s 12.62s/it [25:06:22<5:55:58] mem=34.07GB |
| full_cpt step 11170/12852 ( 86.9%) nll=3.2663 ppl=26.21 grad=0.887 lr=2.77e-06 6866 tok/s 12.62s/it [25:08:18<5:53:50] mem=34.07GB |
| full_cpt step 11180/12852 ( 87.0%) nll=3.3318 ppl=27.99 grad=0.875 lr=2.76e-06 6867 tok/s 12.62s/it [25:10:14<5:51:41] mem=34.07GB |
| full_cpt step 11190/12852 ( 87.1%) nll=3.3659 ppl=28.96 grad=0.863 lr=2.75e-06 6868 tok/s 12.62s/it [25:12:10<5:49:33] mem=34.07GB |
| full_cpt step 11200/12852 ( 87.1%) nll=3.4417 ppl=31.24 grad=0.934 lr=2.74e-06 6869 tok/s 12.62s/it [25:14:06<5:47:24] mem=34.07GB |
| am nll=3.9694 ppl=52.95 bpb=1.3958 |
| ta nll=3.9432 ppl=51.59 bpb=0.5727 |
| kk nll=2.7494 ppl=15.63 bpb=0.9258 |
| tl nll=2.6592 ppl=14.28 bpb=1.2617 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 11200 01:26 (86.1s) |
| full_cpt step 11210/12852 ( 87.2%) nll=3.2610 ppl=26.08 grad=0.875 lr=2.73e-06 6863 tok/s 12.63s/it [25:17:28<5:45:35] mem=34.07GB |
| full_cpt step 11220/12852 ( 87.3%) nll=3.2138 ppl=24.87 grad=0.883 lr=2.72e-06 6864 tok/s 12.63s/it [25:19:24<5:43:27] mem=34.07GB |
| full_cpt step 11230/12852 ( 87.4%) nll=3.2278 ppl=25.22 grad=0.918 lr=2.71e-06 6865 tok/s 12.63s/it [25:21:20<5:41:18] mem=34.07GB |
| full_cpt step 11240/12852 ( 87.5%) nll=3.3890 ppl=29.64 grad=0.898 lr=2.70e-06 6866 tok/s 12.62s/it [25:23:16<5:39:10] mem=34.07GB |
| full_cpt step 11250/12852 ( 87.5%) nll=3.2271 ppl=25.21 grad=0.855 lr=2.69e-06 6867 tok/s 12.62s/it [25:25:12<5:37:01] mem=34.07GB |
| full_cpt step 11260/12852 ( 87.6%) nll=3.2460 ppl=25.69 grad=0.852 lr=2.69e-06 6867 tok/s 12.62s/it [25:27:08<5:34:53] mem=34.07GB |
| full_cpt step 11270/12852 ( 87.7%) nll=3.2695 ppl=26.30 grad=0.844 lr=2.68e-06 6868 tok/s 12.62s/it [25:29:04<5:32:44] mem=34.07GB |
| full_cpt step 11280/12852 ( 87.8%) nll=3.3620 ppl=28.85 grad=0.836 lr=2.67e-06 6869 tok/s 12.62s/it [25:31:00<5:30:36] mem=34.07GB |
| full_cpt step 11290/12852 ( 87.8%) nll=3.3034 ppl=27.21 grad=0.855 lr=2.66e-06 6870 tok/s 12.62s/it [25:32:56<5:28:27] mem=34.07GB |
| full_cpt step 11300/12852 ( 87.9%) nll=3.2827 ppl=26.65 grad=0.828 lr=2.65e-06 6871 tok/s 12.62s/it [25:34:52<5:26:19] mem=34.07GB |
| am nll=3.9693 ppl=52.95 bpb=1.3958 |
| ta nll=3.9428 ppl=51.56 bpb=0.5727 |
| kk nll=2.7491 ppl=15.63 bpb=0.9258 |
| tl nll=2.6592 ppl=14.28 bpb=1.2617 |
| en nll=2.8175 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 11300 01:26 (86.1s) |
| full_cpt step 11310/12852 ( 88.0%) nll=3.3193 ppl=27.64 grad=0.852 lr=2.64e-06 6865 tok/s 12.63s/it [25:38:14<5:24:29] mem=34.07GB |
| full_cpt step 11320/12852 ( 88.1%) nll=3.3934 ppl=29.77 grad=0.879 lr=2.64e-06 6866 tok/s 12.62s/it [25:40:10<5:22:21] mem=34.07GB |
| full_cpt step 11330/12852 ( 88.2%) nll=3.3670 ppl=28.99 grad=0.855 lr=2.63e-06 6867 tok/s 12.62s/it [25:42:06<5:20:12] mem=34.07GB |
| full_cpt step 11340/12852 ( 88.2%) nll=3.2577 ppl=25.99 grad=0.832 lr=2.62e-06 6868 tok/s 12.62s/it [25:44:02<5:18:04] mem=34.07GB |
| full_cpt step 11350/12852 ( 88.3%) nll=3.2386 ppl=25.50 grad=0.879 lr=2.61e-06 6868 tok/s 12.62s/it [25:45:58<5:15:55] mem=34.07GB |
| full_cpt step 11360/12852 ( 88.4%) nll=3.1962 ppl=24.44 grad=0.891 lr=2.60e-06 6869 tok/s 12.62s/it [25:47:54<5:13:47] mem=34.07GB |
| full_cpt step 11370/12852 ( 88.5%) nll=3.3961 ppl=29.85 grad=0.824 lr=2.60e-06 6870 tok/s 12.62s/it [25:49:50<5:11:39] mem=34.07GB |
| full_cpt step 11380/12852 ( 88.5%) nll=3.2220 ppl=25.08 grad=0.906 lr=2.59e-06 6871 tok/s 12.62s/it [25:51:46<5:09:31] mem=34.07GB |
| full_cpt step 11390/12852 ( 88.6%) nll=3.2972 ppl=27.04 grad=0.898 lr=2.58e-06 6871 tok/s 12.61s/it [25:53:42<5:07:23] mem=34.07GB |
| full_cpt step 11400/12852 ( 88.7%) nll=3.3606 ppl=28.81 grad=0.879 lr=2.57e-06 6872 tok/s 12.61s/it [25:55:38<5:05:14] mem=34.07GB |
| am nll=3.9691 ppl=52.94 bpb=1.3957 |
| ta nll=3.9429 ppl=51.57 bpb=0.5727 |
| kk nll=2.7493 ppl=15.63 bpb=0.9258 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8177 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 11400 01:26 (86.3s) |
| full_cpt step 11410/12852 ( 88.8%) nll=3.4128 ppl=30.35 grad=0.875 lr=2.56e-06 6867 tok/s 12.62s/it [25:59:00<5:03:23] mem=34.07GB |
| full_cpt step 11420/12852 ( 88.9%) nll=3.3287 ppl=27.90 grad=0.840 lr=2.56e-06 6867 tok/s 12.62s/it [26:00:56<5:01:15] mem=34.07GB |
| full_cpt step 11430/12852 ( 88.9%) nll=3.3365 ppl=28.12 grad=0.875 lr=2.55e-06 6868 tok/s 12.62s/it [26:02:52<4:59:07] mem=34.07GB |
| full_cpt step 11440/12852 ( 89.0%) nll=3.3314 ppl=27.98 grad=0.887 lr=2.54e-06 6868 tok/s 12.62s/it [26:04:48<4:56:59] mem=34.07GB |
| full_cpt step 11450/12852 ( 89.1%) nll=3.2164 ppl=24.94 grad=0.797 lr=2.53e-06 6869 tok/s 12.62s/it [26:06:44<4:54:50] mem=34.07GB |
| full_cpt step 11460/12852 ( 89.2%) nll=3.4636 ppl=31.93 grad=0.910 lr=2.53e-06 6870 tok/s 12.62s/it [26:08:40<4:52:42] mem=34.07GB |
| full_cpt step 11470/12852 ( 89.2%) nll=3.1815 ppl=24.08 grad=0.883 lr=2.52e-06 6870 tok/s 12.62s/it [26:10:36<4:50:34] mem=34.07GB |
| full_cpt step 11480/12852 ( 89.3%) nll=3.3939 ppl=29.78 grad=1.070 lr=2.51e-06 6871 tok/s 12.61s/it [26:12:32<4:48:26] mem=34.07GB |
| full_cpt step 11490/12852 ( 89.4%) nll=3.3764 ppl=29.27 grad=0.848 lr=2.50e-06 6872 tok/s 12.61s/it [26:14:28<4:46:18] mem=34.07GB |
| full_cpt step 11500/12852 ( 89.5%) nll=3.4465 ppl=31.39 grad=0.848 lr=2.50e-06 6872 tok/s 12.61s/it [26:16:24<4:44:10] mem=34.07GB |
| am nll=3.9691 ppl=52.94 bpb=1.3957 |
| ta nll=3.9430 ppl=51.58 bpb=0.5727 |
| kk nll=2.7491 ppl=15.63 bpb=0.9257 |
| tl nll=2.6595 ppl=14.29 bpb=1.2619 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 11500 01:26 (86.2s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.17s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.17s/it] |
| removing old checkpoint checkpoint-10500 |
| kept last 2 resume checkpoints: checkpoint-11000 checkpoint-11500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-11500 at step 11500 next_row=1472000 |
| full_cpt step 11510/12852 ( 89.6%) nll=3.2031 ppl=24.61 grad=0.867 lr=2.49e-06 6863 tok/s 12.63s/it [26:20:45<4:42:28] mem=34.07GB |
| full_cpt step 11520/12852 ( 89.6%) nll=3.4097 ppl=30.26 grad=0.859 lr=2.48e-06 6863 tok/s 12.63s/it [26:22:41<4:40:20] mem=34.07GB |
| full_cpt step 11530/12852 ( 89.7%) nll=3.3344 ppl=28.06 grad=0.852 lr=2.48e-06 6864 tok/s 12.63s/it [26:24:37<4:38:12] mem=34.07GB |
| full_cpt step 11540/12852 ( 89.8%) nll=3.3690 ppl=29.05 grad=0.906 lr=2.47e-06 6865 tok/s 12.63s/it [26:26:33<4:36:04] mem=34.07GB |
| full_cpt step 11550/12852 ( 89.9%) nll=3.2948 ppl=26.97 grad=0.895 lr=2.46e-06 6865 tok/s 12.62s/it [26:28:29<4:33:56] mem=34.07GB |
| full_cpt step 11560/12852 ( 89.9%) nll=3.3474 ppl=28.43 grad=0.883 lr=2.45e-06 6866 tok/s 12.62s/it [26:30:25<4:31:48] mem=34.07GB |
| full_cpt step 11570/12852 ( 90.0%) nll=3.2816 ppl=26.62 grad=0.938 lr=2.45e-06 6867 tok/s 12.62s/it [26:32:21<4:29:40] mem=34.07GB |
| full_cpt step 11580/12852 ( 90.1%) nll=3.2605 ppl=26.06 grad=0.832 lr=2.44e-06 6868 tok/s 12.62s/it [26:34:17<4:27:32] mem=34.07GB |
| full_cpt step 11590/12852 ( 90.2%) nll=3.2791 ppl=26.55 grad=0.840 lr=2.43e-06 6869 tok/s 12.62s/it [26:36:13<4:25:24] mem=34.07GB |
| full_cpt step 11600/12852 ( 90.3%) nll=3.3715 ppl=29.12 grad=0.887 lr=2.43e-06 6870 tok/s 12.62s/it [26:38:10<4:23:17] mem=34.07GB |
| am nll=3.9693 ppl=52.95 bpb=1.3958 |
| ta nll=3.9428 ppl=51.56 bpb=0.5727 |
| kk nll=2.7495 ppl=15.63 bpb=0.9259 |
| tl nll=2.6593 ppl=14.29 bpb=1.2618 |
| en nll=2.8178 ppl=16.74 bpb=0.8774 |
| eval full_cpt step 11600 01:26 (86.5s) |
| full_cpt step 11610/12852 ( 90.3%) nll=3.3559 ppl=28.67 grad=0.844 lr=2.42e-06 6864 tok/s 12.63s/it [26:41:32<4:21:23] mem=34.07GB |
| full_cpt step 11620/12852 ( 90.4%) nll=3.2079 ppl=24.73 grad=0.887 lr=2.41e-06 6865 tok/s 12.63s/it [26:43:28<4:19:15] mem=34.07GB |
| full_cpt step 11630/12852 ( 90.5%) nll=3.3224 ppl=27.73 grad=0.836 lr=2.41e-06 6866 tok/s 12.62s/it [26:45:24<4:17:07] mem=34.07GB |
| full_cpt step 11640/12852 ( 90.6%) nll=3.3761 ppl=29.26 grad=0.863 lr=2.40e-06 6866 tok/s 12.62s/it [26:47:20<4:14:59] mem=34.07GB |
| full_cpt step 11650/12852 ( 90.6%) nll=3.2260 ppl=25.18 grad=0.887 lr=2.39e-06 6867 tok/s 12.62s/it [26:49:16<4:12:51] mem=34.07GB |
| full_cpt step 11660/12852 ( 90.7%) nll=3.3849 ppl=29.52 grad=0.836 lr=2.39e-06 6868 tok/s 12.62s/it [26:51:12<4:10:43] mem=34.07GB |
| full_cpt step 11670/12852 ( 90.8%) nll=3.3109 ppl=27.41 grad=0.855 lr=2.38e-06 6868 tok/s 12.62s/it [26:53:08<4:08:36] mem=34.07GB |
| full_cpt step 11680/12852 ( 90.9%) nll=3.2210 ppl=25.05 grad=0.895 lr=2.37e-06 6869 tok/s 12.62s/it [26:55:04<4:06:28] mem=34.07GB |
| full_cpt step 11690/12852 ( 91.0%) nll=3.2757 ppl=26.46 grad=0.844 lr=2.37e-06 6870 tok/s 12.62s/it [26:57:00<4:04:20] mem=34.07GB |
| full_cpt step 11700/12852 ( 91.0%) nll=3.1475 ppl=23.28 grad=0.863 lr=2.36e-06 6871 tok/s 12.62s/it [26:58:56<4:02:13] mem=34.07GB |
| am nll=3.9691 ppl=52.94 bpb=1.3957 |
| ta nll=3.9434 ppl=51.59 bpb=0.5727 |
| kk nll=2.7491 ppl=15.63 bpb=0.9257 |
| tl nll=2.6593 ppl=14.29 bpb=1.2618 |
| en nll=2.8173 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 11700 01:27 (86.5s) |
| full_cpt step 11710/12852 ( 91.1%) nll=3.3130 ppl=27.47 grad=0.859 lr=2.36e-06 6865 tok/s 12.62s/it [27:02:18<4:00:18] mem=34.07GB |
| full_cpt step 11720/12852 ( 91.2%) nll=3.3500 ppl=28.50 grad=0.883 lr=2.35e-06 6866 tok/s 12.62s/it [27:04:14<3:58:10] mem=34.07GB |
| full_cpt step 11730/12852 ( 91.3%) nll=3.2161 ppl=24.93 grad=0.844 lr=2.34e-06 6867 tok/s 12.62s/it [27:06:10<3:56:02] mem=34.07GB |
| full_cpt step 11740/12852 ( 91.3%) nll=3.4248 ppl=30.72 grad=0.875 lr=2.34e-06 6868 tok/s 12.62s/it [27:08:06<3:53:55] mem=34.07GB |
| full_cpt step 11750/12852 ( 91.4%) nll=3.4631 ppl=31.92 grad=0.852 lr=2.33e-06 6869 tok/s 12.62s/it [27:10:03<3:51:47] mem=34.07GB |
| full_cpt step 11760/12852 ( 91.5%) nll=3.4155 ppl=30.43 grad=0.895 lr=2.33e-06 6869 tok/s 12.62s/it [27:11:59<3:49:39] mem=34.07GB |
| full_cpt step 11770/12852 ( 91.6%) nll=3.4101 ppl=30.27 grad=0.852 lr=2.32e-06 6870 tok/s 12.62s/it [27:13:55<3:47:32] mem=34.07GB |
| full_cpt step 11780/12852 ( 91.7%) nll=3.2303 ppl=25.29 grad=0.863 lr=2.31e-06 6871 tok/s 12.62s/it [27:15:51<3:45:24] mem=34.07GB |
| full_cpt step 11790/12852 ( 91.7%) nll=3.2281 ppl=25.23 grad=0.863 lr=2.31e-06 6872 tok/s 12.61s/it [27:17:47<3:43:17] mem=34.07GB |
| full_cpt step 11800/12852 ( 91.8%) nll=3.3236 ppl=27.76 grad=0.875 lr=2.30e-06 6873 tok/s 12.61s/it [27:19:43<3:41:09] mem=34.07GB |
| am nll=3.9694 ppl=52.95 bpb=1.3958 |
| ta nll=3.9432 ppl=51.59 bpb=0.5727 |
| kk nll=2.7493 ppl=15.63 bpb=0.9258 |
| tl nll=2.6593 ppl=14.29 bpb=1.2618 |
| en nll=2.8173 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 11800 01:27 (86.6s) |
| full_cpt step 11810/12852 ( 91.9%) nll=3.2661 ppl=26.21 grad=0.887 lr=2.30e-06 6868 tok/s 12.62s/it [27:23:06<3:39:13] mem=34.07GB |
| full_cpt step 11820/12852 ( 92.0%) nll=3.3232 ppl=27.75 grad=0.871 lr=2.29e-06 6868 tok/s 12.62s/it [27:25:02<3:37:06] mem=34.07GB |
| full_cpt step 11830/12852 ( 92.0%) nll=3.3590 ppl=28.76 grad=0.918 lr=2.29e-06 6869 tok/s 12.62s/it [27:26:58<3:34:58] mem=34.07GB |
| full_cpt step 11840/12852 ( 92.1%) nll=3.3541 ppl=28.62 grad=0.855 lr=2.28e-06 6870 tok/s 12.62s/it [27:28:54<3:32:51] mem=34.07GB |
| full_cpt step 11850/12852 ( 92.2%) nll=3.5019 ppl=33.18 grad=0.930 lr=2.27e-06 6870 tok/s 12.62s/it [27:30:50<3:30:43] mem=34.07GB |
| full_cpt step 11860/12852 ( 92.3%) nll=3.2952 ppl=26.98 grad=0.922 lr=2.27e-06 6871 tok/s 12.62s/it [27:32:46<3:28:36] mem=34.07GB |
| full_cpt step 11870/12852 ( 92.4%) nll=3.3790 ppl=29.34 grad=0.836 lr=2.26e-06 6872 tok/s 12.62s/it [27:34:42<3:26:28] mem=34.07GB |
| full_cpt step 11880/12852 ( 92.4%) nll=3.2660 ppl=26.21 grad=0.867 lr=2.26e-06 6872 tok/s 12.61s/it [27:36:38<3:24:21] mem=34.07GB |
| full_cpt step 11890/12852 ( 92.5%) nll=3.2964 ppl=27.02 grad=0.852 lr=2.25e-06 6873 tok/s 12.61s/it [27:38:34<3:22:13] mem=34.07GB |
| full_cpt step 11900/12852 ( 92.6%) nll=3.2287 ppl=25.25 grad=0.871 lr=2.25e-06 6874 tok/s 12.61s/it [27:40:30<3:20:06] mem=34.07GB |
| am nll=3.9691 ppl=52.94 bpb=1.3957 |
| ta nll=3.9429 ppl=51.57 bpb=0.5727 |
| kk nll=2.7491 ppl=15.63 bpb=0.9257 |
| tl nll=2.6595 ppl=14.29 bpb=1.2619 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 11900 01:27 (86.5s) |
| full_cpt step 11910/12852 ( 92.7%) nll=3.3141 ppl=27.50 grad=0.852 lr=2.24e-06 6868 tok/s 12.62s/it [27:43:52<3:18:09] mem=34.07GB |
| full_cpt step 11920/12852 ( 92.7%) nll=3.3723 ppl=29.14 grad=0.867 lr=2.24e-06 6869 tok/s 12.62s/it [27:45:48<3:16:02] mem=34.07GB |
| full_cpt step 11930/12852 ( 92.8%) nll=3.3296 ppl=27.93 grad=0.930 lr=2.23e-06 6870 tok/s 12.62s/it [27:47:44<3:13:54] mem=34.07GB |
| full_cpt step 11940/12852 ( 92.9%) nll=3.2488 ppl=25.76 grad=0.875 lr=2.23e-06 6870 tok/s 12.62s/it [27:49:40<3:11:47] mem=34.07GB |
| full_cpt step 11950/12852 ( 93.0%) nll=3.3212 ppl=27.69 grad=0.879 lr=2.22e-06 6871 tok/s 12.62s/it [27:51:36<3:09:40] mem=34.07GB |
| full_cpt step 11960/12852 ( 93.1%) nll=3.3053 ppl=27.26 grad=0.867 lr=2.22e-06 6872 tok/s 12.61s/it [27:53:33<3:07:32] mem=34.07GB |
| full_cpt step 11970/12852 ( 93.1%) nll=3.1974 ppl=24.47 grad=0.836 lr=2.21e-06 6872 tok/s 12.61s/it [27:55:29<3:05:25] mem=34.07GB |
| full_cpt step 11980/12852 ( 93.2%) nll=3.3048 ppl=27.24 grad=0.926 lr=2.21e-06 6873 tok/s 12.61s/it [27:57:25<3:03:18] mem=34.07GB |
| full_cpt step 11990/12852 ( 93.3%) nll=3.2790 ppl=26.55 grad=0.848 lr=2.20e-06 6874 tok/s 12.61s/it [27:59:21<3:01:11] mem=34.07GB |
| full_cpt step 12000/12852 ( 93.4%) nll=3.2217 ppl=25.07 grad=0.879 lr=2.20e-06 6875 tok/s 12.61s/it [28:01:17<2:59:03] mem=34.07GB |
| am nll=3.9697 ppl=52.97 bpb=1.3959 |
| ta nll=3.9435 ppl=51.60 bpb=0.5728 |
| kk nll=2.7491 ppl=15.63 bpb=0.9257 |
| tl nll=2.6592 ppl=14.29 bpb=1.2618 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 12000 01:27 (86.9s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.29s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.29s/it] |
| removing old checkpoint checkpoint-11000 |
| kept last 2 resume checkpoints: checkpoint-11500 checkpoint-12000 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-12000 at step 12000 next_row=1536000 |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.49s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.49s/it] |
| wrote HF-loadable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/model-12000 (weights+expanded tokenizer, no optimizer) |
| full_cpt step 12010/12852 ( 93.4%) nll=3.4169 ppl=30.47 grad=0.871 lr=2.19e-06 6867 tok/s 12.63s/it [28:05:28<2:57:10] mem=34.07GB |
| full_cpt step 12020/12852 ( 93.5%) nll=3.3119 ppl=27.44 grad=0.855 lr=2.19e-06 6868 tok/s 12.62s/it [28:07:24<2:55:03] mem=34.07GB |
| full_cpt step 12030/12852 ( 93.6%) nll=3.2800 ppl=26.58 grad=0.906 lr=2.18e-06 6868 tok/s 12.62s/it [28:09:20<2:52:56] mem=34.07GB |
| full_cpt step 12040/12852 ( 93.7%) nll=3.3163 ppl=27.56 grad=0.828 lr=2.18e-06 6869 tok/s 12.62s/it [28:11:16<2:50:49] mem=34.07GB |
| full_cpt step 12050/12852 ( 93.8%) nll=3.3509 ppl=28.53 grad=0.875 lr=2.18e-06 6870 tok/s 12.62s/it [28:13:12<2:48:41] mem=34.07GB |
| full_cpt step 12060/12852 ( 93.8%) nll=3.1908 ppl=24.31 grad=0.863 lr=2.17e-06 6871 tok/s 12.62s/it [28:15:08<2:46:34] mem=34.07GB |
| full_cpt step 12070/12852 ( 93.9%) nll=3.3532 ppl=28.59 grad=0.871 lr=2.17e-06 6871 tok/s 12.62s/it [28:17:04<2:44:27] mem=34.07GB |
| full_cpt step 12080/12852 ( 94.0%) nll=3.2121 ppl=24.83 grad=0.863 lr=2.16e-06 6872 tok/s 12.62s/it [28:19:00<2:42:20] mem=34.07GB |
| full_cpt step 12090/12852 ( 94.1%) nll=3.2520 ppl=25.84 grad=0.859 lr=2.16e-06 6873 tok/s 12.62s/it [28:20:56<2:40:13] mem=34.07GB |
| full_cpt step 12100/12852 ( 94.1%) nll=3.2555 ppl=25.93 grad=0.855 lr=2.15e-06 6873 tok/s 12.61s/it [28:22:52<2:38:06] mem=34.07GB |
| am nll=3.9691 ppl=52.93 bpb=1.3957 |
| ta nll=3.9430 ppl=51.57 bpb=0.5727 |
| kk nll=2.7493 ppl=15.63 bpb=0.9258 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 12100 01:27 (87.2s) |
| full_cpt step 12110/12852 ( 94.2%) nll=3.2516 ppl=25.83 grad=0.820 lr=2.15e-06 6868 tok/s 12.62s/it [28:26:15<2:36:07] mem=34.07GB |
| full_cpt step 12120/12852 ( 94.3%) nll=3.1082 ppl=22.38 grad=0.840 lr=2.15e-06 6869 tok/s 12.62s/it [28:28:11<2:33:59] mem=34.07GB |
| full_cpt step 12130/12852 ( 94.4%) nll=3.3477 ppl=28.44 grad=0.871 lr=2.14e-06 6870 tok/s 12.62s/it [28:30:07<2:31:52] mem=34.07GB |
| full_cpt step 12140/12852 ( 94.5%) nll=3.2855 ppl=26.72 grad=0.836 lr=2.14e-06 6870 tok/s 12.62s/it [28:32:04<2:29:45] mem=34.07GB |
| full_cpt step 12150/12852 ( 94.5%) nll=3.4098 ppl=30.26 grad=0.867 lr=2.13e-06 6871 tok/s 12.62s/it [28:34:00<2:27:38] mem=34.07GB |
| full_cpt step 12160/12852 ( 94.6%) nll=3.3428 ppl=28.30 grad=0.879 lr=2.13e-06 6872 tok/s 12.62s/it [28:35:56<2:25:31] mem=34.07GB |
| full_cpt step 12170/12852 ( 94.7%) nll=3.2810 ppl=26.60 grad=0.887 lr=2.13e-06 6872 tok/s 12.62s/it [28:37:52<2:23:24] mem=34.07GB |
| full_cpt step 12180/12852 ( 94.8%) nll=3.3206 ppl=27.68 grad=0.820 lr=2.12e-06 6873 tok/s 12.61s/it [28:39:48<2:21:17] mem=34.07GB |
| full_cpt step 12190/12852 ( 94.8%) nll=3.2238 ppl=25.12 grad=0.883 lr=2.12e-06 6874 tok/s 12.61s/it [28:41:44<2:19:10] mem=34.07GB |
| full_cpt step 12200/12852 ( 94.9%) nll=3.3122 ppl=27.45 grad=0.879 lr=2.12e-06 6875 tok/s 12.61s/it [28:43:40<2:17:03] mem=34.07GB |
| am nll=3.9692 ppl=52.94 bpb=1.3958 |
| ta nll=3.9433 ppl=51.59 bpb=0.5727 |
| kk nll=2.7490 ppl=15.63 bpb=0.9257 |
| tl nll=2.6592 ppl=14.28 bpb=1.2617 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 12200 01:26 (86.3s) |
| full_cpt step 12210/12852 ( 95.0%) nll=3.2278 ppl=25.22 grad=0.855 lr=2.11e-06 6870 tok/s 12.62s/it [28:47:02<2:15:03] mem=34.07GB |
| full_cpt step 12220/12852 ( 95.1%) nll=3.3338 ppl=28.04 grad=0.848 lr=2.11e-06 6870 tok/s 12.62s/it [28:48:58<2:12:56] mem=34.07GB |
| full_cpt step 12230/12852 ( 95.2%) nll=3.3224 ppl=27.73 grad=0.848 lr=2.11e-06 6871 tok/s 12.62s/it [28:50:54<2:10:49] mem=34.07GB |
| full_cpt step 12240/12852 ( 95.2%) nll=3.4459 ppl=31.37 grad=0.879 lr=2.10e-06 6872 tok/s 12.62s/it [28:52:50<2:08:42] mem=34.07GB |
| full_cpt step 12250/12852 ( 95.3%) nll=3.2774 ppl=26.51 grad=0.895 lr=2.10e-06 6872 tok/s 12.62s/it [28:54:46<2:06:35] mem=34.07GB |
| full_cpt step 12260/12852 ( 95.4%) nll=3.2019 ppl=24.58 grad=0.844 lr=2.10e-06 6873 tok/s 12.62s/it [28:56:42<2:04:28] mem=34.07GB |
| full_cpt step 12270/12852 ( 95.5%) nll=3.2508 ppl=25.81 grad=0.918 lr=2.09e-06 6873 tok/s 12.61s/it [28:58:38<2:02:21] mem=34.07GB |
| full_cpt step 12280/12852 ( 95.5%) nll=3.2031 ppl=24.61 grad=0.820 lr=2.09e-06 6874 tok/s 12.61s/it [29:00:34<2:00:15] mem=34.07GB |
| full_cpt step 12290/12852 ( 95.6%) nll=3.2755 ppl=26.46 grad=0.863 lr=2.09e-06 6875 tok/s 12.61s/it [29:02:30<1:58:08] mem=34.07GB |
| full_cpt step 12300/12852 ( 95.7%) nll=3.2840 ppl=26.68 grad=0.828 lr=2.08e-06 6876 tok/s 12.61s/it [29:04:26<1:56:01] mem=34.07GB |
| am nll=3.9689 ppl=52.93 bpb=1.3957 |
| ta nll=3.9429 ppl=51.57 bpb=0.5727 |
| kk nll=2.7496 ppl=15.64 bpb=0.9259 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 12300 01:26 (86.2s) |
| full_cpt step 12310/12852 ( 95.8%) nll=3.2591 ppl=26.03 grad=0.895 lr=2.08e-06 6871 tok/s 12.62s/it [29:07:48<1:54:00] mem=34.07GB |
| full_cpt step 12320/12852 ( 95.9%) nll=3.2440 ppl=25.63 grad=0.875 lr=2.08e-06 6871 tok/s 12.62s/it [29:09:44<1:51:53] mem=34.07GB |
| full_cpt step 12330/12852 ( 95.9%) nll=3.3189 ppl=27.63 grad=0.828 lr=2.07e-06 6872 tok/s 12.62s/it [29:11:40<1:49:46] mem=34.07GB |
| full_cpt step 12340/12852 ( 96.0%) nll=3.3313 ppl=27.97 grad=0.883 lr=2.07e-06 6873 tok/s 12.62s/it [29:13:36<1:47:39] mem=34.07GB |
| full_cpt step 12350/12852 ( 96.1%) nll=3.3223 ppl=27.72 grad=0.816 lr=2.07e-06 6873 tok/s 12.61s/it [29:15:32<1:45:33] mem=34.07GB |
| full_cpt step 12360/12852 ( 96.2%) nll=3.2524 ppl=25.85 grad=0.875 lr=2.07e-06 6874 tok/s 12.61s/it [29:17:28<1:43:26] mem=34.07GB |
| full_cpt step 12370/12852 ( 96.2%) nll=3.3972 ppl=29.88 grad=0.824 lr=2.06e-06 6875 tok/s 12.61s/it [29:19:25<1:41:19] mem=34.07GB |
| full_cpt step 12380/12852 ( 96.3%) nll=3.3230 ppl=27.74 grad=0.836 lr=2.06e-06 6876 tok/s 12.61s/it [29:21:21<1:39:12] mem=34.07GB |
| full_cpt step 12390/12852 ( 96.4%) nll=3.3778 ppl=29.31 grad=0.859 lr=2.06e-06 6876 tok/s 12.61s/it [29:23:16<1:37:06] mem=34.07GB |
| full_cpt step 12400/12852 ( 96.5%) nll=3.2428 ppl=25.61 grad=0.871 lr=2.06e-06 6877 tok/s 12.61s/it [29:25:12<1:34:59] mem=34.07GB |
| am nll=3.9692 ppl=52.94 bpb=1.3958 |
| ta nll=3.9424 ppl=51.54 bpb=0.5726 |
| kk nll=2.7493 ppl=15.63 bpb=0.9258 |
| tl nll=2.6594 ppl=14.29 bpb=1.2619 |
| en nll=2.8174 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 12400 01:26 (86.4s) |
| full_cpt step 12410/12852 ( 96.6%) nll=3.2718 ppl=26.36 grad=0.867 lr=2.05e-06 6872 tok/s 12.62s/it [29:28:35<1:32:57] mem=34.07GB |
| full_cpt step 12420/12852 ( 96.6%) nll=3.3622 ppl=28.85 grad=0.859 lr=2.05e-06 6873 tok/s 12.62s/it [29:30:31<1:30:50] mem=34.07GB |
| full_cpt step 12430/12852 ( 96.7%) nll=3.4398 ppl=31.18 grad=0.867 lr=2.05e-06 6873 tok/s 12.62s/it [29:32:27<1:28:44] mem=34.07GB |
| full_cpt step 12440/12852 ( 96.8%) nll=3.2690 ppl=26.29 grad=0.848 lr=2.05e-06 6874 tok/s 12.61s/it [29:34:23<1:26:37] mem=34.07GB |
| full_cpt step 12450/12852 ( 96.9%) nll=3.3431 ppl=28.31 grad=0.836 lr=2.04e-06 6875 tok/s 12.61s/it [29:36:19<1:24:30] mem=34.07GB |
| full_cpt step 12460/12852 ( 96.9%) nll=3.2528 ppl=25.86 grad=0.906 lr=2.04e-06 6876 tok/s 12.61s/it [29:38:15<1:22:24] mem=34.07GB |
| full_cpt step 12470/12852 ( 97.0%) nll=3.2068 ppl=24.70 grad=0.867 lr=2.04e-06 6876 tok/s 12.61s/it [29:40:11<1:20:17] mem=34.07GB |
| full_cpt step 12480/12852 ( 97.1%) nll=3.3284 ppl=27.89 grad=0.867 lr=2.04e-06 6877 tok/s 12.61s/it [29:42:07<1:18:11] mem=34.07GB |
| full_cpt step 12490/12852 ( 97.2%) nll=3.3612 ppl=28.82 grad=0.852 lr=2.04e-06 6877 tok/s 12.61s/it [29:44:03<1:16:04] mem=34.07GB |
| full_cpt step 12500/12852 ( 97.3%) nll=3.2447 ppl=25.65 grad=0.820 lr=2.03e-06 6878 tok/s 12.61s/it [29:46:00<1:13:58] mem=34.07GB |
| am nll=3.9690 ppl=52.93 bpb=1.3957 |
| ta nll=3.9425 ppl=51.55 bpb=0.5726 |
| kk nll=2.7495 ppl=15.63 bpb=0.9259 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 12500 01:26 (86.4s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.64s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.64s/it] |
| removing old checkpoint checkpoint-11500 |
| kept last 2 resume checkpoints: checkpoint-12000 checkpoint-12500 |
| wrote resumable research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-12500 at step 12500 next_row=1600000 |
| full_cpt step 12510/12852 ( 97.3%) nll=3.1386 ppl=23.07 grad=0.871 lr=2.03e-06 6869 tok/s 12.62s/it [29:50:32<1:11:57] mem=34.07GB |
| full_cpt step 12520/12852 ( 97.4%) nll=3.4173 ppl=30.49 grad=0.871 lr=2.03e-06 6869 tok/s 12.62s/it [29:52:28<1:09:51] mem=34.07GB |
| full_cpt step 12530/12852 ( 97.5%) nll=3.2343 ppl=25.39 grad=0.898 lr=2.03e-06 6870 tok/s 12.62s/it [29:54:25<1:07:44] mem=34.07GB |
| full_cpt step 12540/12852 ( 97.6%) nll=3.4352 ppl=31.04 grad=0.879 lr=2.03e-06 6870 tok/s 12.62s/it [29:56:20<1:05:38] mem=34.07GB |
| full_cpt step 12550/12852 ( 97.7%) nll=3.2317 ppl=25.32 grad=0.844 lr=2.03e-06 6871 tok/s 12.62s/it [29:58:17<1:03:31] mem=34.07GB |
| full_cpt step 12560/12852 ( 97.7%) nll=3.2916 ppl=26.89 grad=0.867 lr=2.02e-06 6872 tok/s 12.62s/it [30:00:13<1:01:25] mem=34.07GB |
| full_cpt step 12570/12852 ( 97.8%) nll=3.3406 ppl=28.24 grad=0.871 lr=2.02e-06 6872 tok/s 12.62s/it [30:02:09<59:18] mem=34.07GB |
| full_cpt step 12580/12852 ( 97.9%) nll=3.3618 ppl=28.84 grad=0.910 lr=2.02e-06 6873 tok/s 12.62s/it [30:04:05<57:12] mem=34.07GB |
| full_cpt step 12590/12852 ( 98.0%) nll=3.2779 ppl=26.52 grad=0.832 lr=2.02e-06 6874 tok/s 12.61s/it [30:06:01<55:05] mem=34.07GB |
| full_cpt step 12600/12852 ( 98.0%) nll=3.2749 ppl=26.44 grad=0.902 lr=2.02e-06 6874 tok/s 12.61s/it [30:07:57<52:59] mem=34.07GB |
| am nll=3.9693 ppl=52.95 bpb=1.3958 |
| ta nll=3.9426 ppl=51.55 bpb=0.5726 |
| kk nll=2.7492 ppl=15.63 bpb=0.9258 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8175 ppl=16.73 bpb=0.8772 |
| eval full_cpt step 12600 01:27 (86.9s) |
| full_cpt step 12610/12852 ( 98.1%) nll=3.2084 ppl=24.74 grad=0.867 lr=2.02e-06 6869 tok/s 12.62s/it [30:11:20<50:55] mem=34.07GB |
| full_cpt step 12620/12852 ( 98.2%) nll=3.3880 ppl=29.61 grad=0.883 lr=2.01e-06 6870 tok/s 12.62s/it [30:13:16<48:48] mem=34.07GB |
| full_cpt step 12630/12852 ( 98.3%) nll=3.4006 ppl=29.98 grad=0.875 lr=2.01e-06 6871 tok/s 12.62s/it [30:15:12<46:42] mem=34.07GB |
| full_cpt step 12640/12852 ( 98.4%) nll=3.2647 ppl=26.17 grad=0.875 lr=2.01e-06 6871 tok/s 12.62s/it [30:17:08<44:35] mem=34.07GB |
| full_cpt step 12650/12852 ( 98.4%) nll=3.2486 ppl=25.76 grad=0.879 lr=2.01e-06 6872 tok/s 12.62s/it [30:19:04<42:29] mem=34.07GB |
| full_cpt step 12660/12852 ( 98.5%) nll=3.2810 ppl=26.60 grad=0.824 lr=2.01e-06 6873 tok/s 12.62s/it [30:21:00<40:22] mem=34.07GB |
| full_cpt step 12670/12852 ( 98.6%) nll=3.2426 ppl=25.60 grad=0.832 lr=2.01e-06 6873 tok/s 12.62s/it [30:22:56<38:16] mem=34.07GB |
| full_cpt step 12680/12852 ( 98.7%) nll=3.2754 ppl=26.46 grad=0.848 lr=2.01e-06 6874 tok/s 12.61s/it [30:24:52<36:10] mem=34.07GB |
| full_cpt step 12690/12852 ( 98.7%) nll=3.2832 ppl=26.66 grad=0.836 lr=2.01e-06 6874 tok/s 12.61s/it [30:26:48<34:03] mem=34.07GB |
| full_cpt step 12700/12852 ( 98.8%) nll=3.2605 ppl=26.06 grad=0.871 lr=2.01e-06 6875 tok/s 12.61s/it [30:28:44<31:57] mem=34.07GB |
| am nll=3.9690 ppl=52.93 bpb=1.3957 |
| ta nll=3.9429 ppl=51.57 bpb=0.5727 |
| kk nll=2.7490 ppl=15.63 bpb=0.9257 |
| tl nll=2.6591 ppl=14.28 bpb=1.2617 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 12700 01:27 (87.3s) |
| full_cpt step 12710/12852 ( 98.9%) nll=3.3232 ppl=27.75 grad=0.848 lr=2.01e-06 6870 tok/s 12.62s/it [30:32:07<29:52] mem=34.07GB |
| full_cpt step 12720/12852 ( 99.0%) nll=3.3074 ppl=27.31 grad=0.852 lr=2.00e-06 6871 tok/s 12.62s/it [30:34:03<27:46] mem=34.07GB |
| full_cpt step 12730/12852 ( 99.1%) nll=3.2786 ppl=26.54 grad=0.875 lr=2.00e-06 6872 tok/s 12.62s/it [30:35:59<25:39] mem=34.07GB |
| full_cpt step 12740/12852 ( 99.1%) nll=3.3467 ppl=28.41 grad=0.871 lr=2.00e-06 6872 tok/s 12.62s/it [30:37:55<23:33] mem=34.07GB |
| full_cpt step 12750/12852 ( 99.2%) nll=3.2962 ppl=27.01 grad=0.840 lr=2.00e-06 6873 tok/s 12.62s/it [30:39:51<21:27] mem=34.07GB |
| full_cpt step 12760/12852 ( 99.3%) nll=3.2643 ppl=26.16 grad=0.891 lr=2.00e-06 6873 tok/s 12.62s/it [30:41:48<19:21] mem=34.07GB |
| full_cpt step 12770/12852 ( 99.4%) nll=3.2927 ppl=26.91 grad=0.895 lr=2.00e-06 6874 tok/s 12.61s/it [30:43:44<17:14] mem=34.07GB |
| full_cpt step 12780/12852 ( 99.4%) nll=3.2312 ppl=25.31 grad=0.883 lr=2.00e-06 6874 tok/s 12.61s/it [30:45:40<15:08] mem=34.07GB |
| full_cpt step 12790/12852 ( 99.5%) nll=3.3785 ppl=29.33 grad=0.902 lr=2.00e-06 6875 tok/s 12.61s/it [30:47:36<13:02] mem=34.07GB |
| full_cpt step 12800/12852 ( 99.6%) nll=3.2819 ppl=26.63 grad=0.855 lr=2.00e-06 6876 tok/s 12.61s/it [30:49:32<10:56] mem=34.07GB |
| am nll=3.9691 ppl=52.93 bpb=1.3957 |
| ta nll=3.9432 ppl=51.58 bpb=0.5727 |
| kk nll=2.7491 ppl=15.63 bpb=0.9257 |
| tl nll=2.6594 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 12800 01:28 (88.2s) |
| full_cpt step 12810/12852 ( 99.7%) nll=3.2556 ppl=25.94 grad=1.375 lr=2.00e-06 6871 tok/s 12.62s/it [30:52:56<08:50] mem=34.07GB |
| full_cpt step 12820/12852 ( 99.8%) nll=3.2670 ppl=26.23 grad=0.867 lr=2.00e-06 6872 tok/s 12.62s/it [30:54:53<06:44] mem=34.07GB |
| full_cpt step 12830/12852 ( 99.8%) nll=3.3131 ppl=27.47 grad=0.867 lr=2.00e-06 6872 tok/s 12.62s/it [30:56:49<04:38] mem=34.07GB |
| full_cpt step 12840/12852 ( 99.9%) nll=3.1935 ppl=24.37 grad=0.824 lr=2.00e-06 6873 tok/s 12.62s/it [30:58:45<02:31] mem=34.07GB |
| full_cpt step 12850/12852 (100.0%) nll=3.3764 ppl=29.26 grad=0.883 lr=2.00e-06 6874 tok/s 12.61s/it [31:00:41<00:25] mem=34.07GB |
| full_cpt step 12852/12852 (100.0%) nll=3.3621 ppl=28.85 grad=0.914 lr=2.00e-06 6874 tok/s 12.61s/it [31:01:04<00:00] mem=34.07GB |
| am nll=3.9691 ppl=52.94 bpb=1.3957 |
| ta nll=3.9430 ppl=51.57 bpb=0.5727 |
| kk nll=2.7493 ppl=15.63 bpb=0.9258 |
| tl nll=2.6592 ppl=14.29 bpb=1.2618 |
| en nll=2.8176 ppl=16.74 bpb=0.8773 |
| eval full_cpt step 12852 01:28 (88.2s) |
|
Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.30s/it]
Writing model shards: 100%|##########| 1/1 [00:02<00:00, 2.30s/it] |
| done research/results/embedding_init/gemma_mapping_byteify_cpt_seq1024/checkpoint-final |
|
|
