| 2026-07-20T08:37:27-07:00 stage 1: collecting seed-2 calibration stats |
|
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 33%|███▎ | 1/3 [00:01<00:03, 1.60s/it]
Loading checkpoint shards: 67%|██████▋ | 2/3 [00:03<00:01, 1.66s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:04<00:00, 1.48s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:04<00:00, 1.52s/it] |
|
calibration [c4]: 0%| | 0/61 [00:00<?, ?it/s]
calibration [c4]: 2%|▏ | 1/61 [00:08<08:51, 8.85s/it]
calibration [c4]: 3%|▎ | 2/61 [00:09<04:12, 4.28s/it]
calibration [c4]: 5%|▍ | 3/61 [00:11<02:43, 2.82s/it]
calibration [c4]: 7%|▋ | 4/61 [00:12<02:01, 2.13s/it]
calibration [c4]: 8%|▊ | 5/61 [00:13<01:37, 1.75s/it]
calibration [c4]: 10%|▉ | 6/61 [00:14<01:23, 1.52s/it]
calibration [c4]: 11%|█▏ | 7/61 [00:15<01:14, 1.38s/it]
calibration [c4]: 13%|█▎ | 8/61 [00:16<01:07, 1.28s/it]
calibration [c4]: 15%|█▍ | 9/61 [00:17<01:03, 1.21s/it]
calibration [c4]: 16%|█▋ | 10/61 [00:18<00:59, 1.17s/it]
calibration [c4]: 18%|█▊ | 11/61 [00:19<00:56, 1.14s/it]
calibration [c4]: 20%|█▉ | 12/61 [00:20<00:54, 1.12s/it]
calibration [c4]: 21%|██▏ | 13/61 [00:21<00:52, 1.10s/it]
calibration [c4]: 23%|██▎ | 14/61 [00:22<00:51, 1.09s/it]
calibration [c4]: 25%|██▍ | 15/61 [00:23<00:50, 1.09s/it]
calibration [c4]: 26%|██▌ | 16/61 [00:24<00:48, 1.08s/it]
calibration [c4]: 28%|██▊ | 17/61 [00:26<00:47, 1.08s/it]
calibration [c4]: 30%|██▉ | 18/61 [00:27<00:46, 1.08s/it]
calibration [c4]: 31%|███ | 19/61 [00:28<00:45, 1.08s/it]
calibration [c4]: 33%|███▎ | 20/61 [00:29<00:44, 1.07s/it]
calibration [c4]: 34%|███▍ | 21/61 [00:30<00:42, 1.07s/it]
calibration [c4]: 36%|███▌ | 22/61 [00:31<00:41, 1.07s/it]
calibration [c4]: 38%|███▊ | 23/61 [00:32<00:40, 1.07s/it]
calibration [c4]: 39%|███▉ | 24/61 [00:33<00:39, 1.07s/it]
calibration [c4]: 41%|████ | 25/61 [00:34<00:38, 1.07s/it]
calibration [c4]: 43%|████▎ | 26/61 [00:35<00:37, 1.07s/it]
calibration [c4]: 44%|████▍ | 27/61 [00:36<00:36, 1.07s/it]
calibration [c4]: 46%|████▌ | 28/61 [00:37<00:35, 1.07s/it]
calibration [c4]: 48%|████▊ | 29/61 [00:38<00:34, 1.07s/it]
calibration [c4]: 49%|████▉ | 30/61 [00:39<00:33, 1.07s/it]
calibration [c4]: 51%|█████ | 31/61 [00:41<00:32, 1.07s/it]
calibration [c4]: 52%|█████▏ | 32/61 [00:42<00:30, 1.07s/it]
calibration [c4]: 54%|█████▍ | 33/61 [00:43<00:29, 1.07s/it]
calibration [c4]: 56%|█████▌ | 34/61 [00:44<00:28, 1.07s/it]
calibration [c4]: 57%|█████▋ | 35/61 [00:45<00:27, 1.07s/it]
calibration [c4]: 59%|█████▉ | 36/61 [00:46<00:26, 1.07s/it]
calibration [c4]: 61%|██████ | 37/61 [00:47<00:25, 1.07s/it]
calibration [c4]: 62%|██████▏ | 38/61 [00:48<00:24, 1.07s/it]
calibration [c4]: 64%|██████▍ | 39/61 [00:49<00:23, 1.07s/it]
calibration [c4]: 66%|██████▌ | 40/61 [00:50<00:22, 1.07s/it]
calibration [c4]: 67%|██████▋ | 41/61 [00:51<00:21, 1.07s/it]
calibration [c4]: 69%|██████▉ | 42/61 [00:52<00:20, 1.07s/it]
calibration [c4]: 70%|███████ | 43/61 [00:53<00:19, 1.07s/it]
calibration [c4]: 72%|███████▏ | 44/61 [00:54<00:18, 1.07s/it]
calibration [c4]: 74%|███████▍ | 45/61 [00:55<00:17, 1.07s/it]
calibration [c4]: 75%|███████▌ | 46/61 [00:57<00:16, 1.07s/it]
calibration [c4]: 77%|███████▋ | 47/61 [00:58<00:14, 1.07s/it]
calibration [c4]: 79%|███████▊ | 48/61 [00:59<00:13, 1.07s/it]
calibration [c4]: 80%|████████ | 49/61 [01:00<00:12, 1.07s/it]
calibration [c4]: 82%|████████▏ | 50/61 [01:01<00:11, 1.07s/it]
calibration [c4]: 84%|████████▎ | 51/61 [01:02<00:10, 1.07s/it]
calibration [c4]: 85%|████████▌ | 52/61 [01:03<00:09, 1.07s/it]
calibration [c4]: 87%|████████▋ | 53/61 [01:04<00:08, 1.07s/it]
calibration [c4]: 89%|████████▊ | 54/61 [01:05<00:07, 1.07s/it]
calibration [c4]: 90%|█████████ | 55/61 [01:06<00:06, 1.08s/it]
calibration [c4]: 92%|█████████▏| 56/61 [01:07<00:05, 1.07s/it]
calibration [c4]: 93%|█████████▎| 57/61 [01:08<00:04, 1.07s/it]
calibration [c4]: 95%|█████████▌| 58/61 [01:09<00:03, 1.07s/it]
calibration [c4]: 97%|█████████▋| 59/61 [01:10<00:02, 1.07s/it]
calibration [c4]: 98%|█████████▊| 60/61 [01:12<00:01, 1.07s/it]
calibration [c4]: 100%|██████████| 61/61 [01:13<00:00, 1.07s/it]
calibration [c4]: 100%|██████████| 61/61 [01:13<00:00, 1.20s/it] |
| |
| tokens seen: 499,712 (244 seqs x 2048, dataset=c4) |
| token_count [L=16, E=64] — routed-token counts (top-k membership): |
| per-layer row sums (== n_tokens * top_k): min 3997696, max 3997696 |
| per-expert counts: min 10984, median 61647, max 270930, mean 62464.0 |
| zero-routed (layer, expert) pairs: 0 / 1024 |
| saved -> outputs/stats_0125inst_c4_0.5M_seed2.pt |
| 2026-07-20T08:39:14-07:00 stage 2: policy-pair Protocol-C evals |
| 2026-07-20T08:39:14-07:00 EVAL off_forward_seed1224 on GPU 0 (port 8430) |
| 2026-07-20T08:41:44-07:00 EVAL off_forward_seed1225 on GPU 1 (port 8431) |
| 2026-07-20T08:44:14-07:00 EVAL off_forward_seed1226 on GPU 2 (port 8432) |
| 2026-07-20T08:54:10-07:00 EVAL on_reverse_seed1224 on GPU 0 (port 8430) |
| 2026-07-20T08:57:11-07:00 EVAL on_reverse_seed1225 on GPU 1 (port 8431) |
| 2026-07-20T08:59:04-07:00 EVAL on_reverse_seed1226 on GPU 2 (port 8432) |
| 2026-07-20T09:10:10-07:00 EVAL combo_on25_seed1224 on GPU 0 (port 8430) |
| 2026-07-20T09:14:00-07:00 EVAL combo_on25_seed1225 on GPU 1 (port 8431) |
| 2026-07-20T09:15:51-07:00 EVAL combo_on25_seed1226 on GPU 2 (port 8432) |
| 2026-07-20T09:25:56-07:00 lane GPU0 complete |
| 2026-07-20T09:30:21-07:00 lane GPU1 complete |
| 2026-07-20T09:31:33-07:00 lane GPU2 complete |
| 2026-07-20T09:31:33-07:00 stage 3: allocation overlap report |
| Traceback (most recent call last): |
| File "/home/henry/Documents/PythonProjects/variable-reap/scripts/43_alloc_overlap.py", line 88, in <module> |
| main() |
| File "/home/henry/Documents/PythonProjects/variable-reap/scripts/43_alloc_overlap.py", line 61, in main |
| sa = layer_scores(stats_a, args.criterion) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/home/henry/Documents/PythonProjects/variable-reap/scripts/43_alloc_overlap.py", line 31, in layer_scores |
| return channel_scores(stats, b=b, alpha=alpha, beta=beta) # [L, E, C] |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| TypeError: channel_scores() missing 1 required positional argument: 'colnorms' |
| 2026-07-20T09:31:34-07:00 ###### PROTOCOL-C BACKLOG COMPLETE ###### |
| |
