| wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /root/.netrc. |
| wandb: Currently logged in as: dalletest123 to https://api.wandb.ai. Use `wandb login --relogin` to force relogin |
| wandb: Tracking run with wandb version 0.26.1 |
| wandb: Run data is saved locally in /tmp/sparse-attn-git/wandb/run-20260509_091956-hcukr8rw |
| wandb: Run `wandb offline` to turn off syncing. |
| wandb: Syncing run all32-d128-block-causal-reserve-0-1-2-35 |
| wandb: ⭐️ View project at https://wandb.ai/dalletest123/ann-sparse |
| wandb: 🚀 View run at https://wandb.ai/dalletest123/ann-sparse/runs/hcukr8rw |
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
| [config] preset=all32_d128_block run=all32-d128-block-causal-reserve-0-1-2-35 steps=1000 layers=32 d_search=128 ckpt_dir=/tmp/checkpoints_all32_d128_block_reserve_0_1_2_35 |
| Training search projections for layers: [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34] |
| Reserved as full attention: [0, 1, 2, 35] |
| [perf] flash_attention_3 unavailable (ImportError: FlashAttention3 has been toggled on, but it cannot be used due to the following error: the package for FlashAttention3 doesn't seem to be installed.); trying next. |
| [perf] flash_attention_2 unavailable (ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package for FlashAttention2 doesn't seem to be installed.); trying next. |
|
Loading weights: 0%| | 0/398 [00:00<?, ?it/s]
Loading weights: 0%| | 1/398 [00:00<01:30, 4.37it/s]
Loading weights: 4%|▍ | 16/398 [00:00<00:06, 59.46it/s]
Loading weights: 9%|▉ | 36/398 [00:00<00:03, 108.68it/s]
Loading weights: 14%|█▍ | 55/398 [00:00<00:02, 133.88it/s]
Loading weights: 18%|█▊ | 71/398 [00:00<00:02, 139.15it/s]
Loading weights: 23%|██▎ | 92/398 [00:00<00:01, 157.57it/s]
Loading weights: 28%|██▊ | 113/398 [00:00<00:01, 169.89it/s]
Loading weights: 33%|███▎ | 131/398 [00:00<00:01, 172.45it/s]
Loading weights: 37%|███▋ | 149/398 [00:01<00:01, 169.89it/s]
Loading weights: 42%|████▏ | 169/398 [00:01<00:01, 175.68it/s]
Loading weights: 47%|████▋ | 188/398 [00:01<00:01, 179.38it/s]
Loading weights: 52%|█████▏ | 207/398 [00:01<00:01, 174.09it/s]
Loading weights: 57%|█████▋ | 225/398 [00:01<00:01, 166.50it/s]
Loading weights: 62%|██████▏ | 246/398 [00:01<00:00, 170.46it/s]
Loading weights: 67%|██████▋ | 268/398 [00:01<00:00, 179.23it/s]
Loading weights: 73%|███████▎ | 289/398 [00:01<00:00, 185.21it/s]
Loading weights: 77%|███████▋ | 308/398 [00:01<00:00, 185.02it/s]
Loading weights: 82%|████████▏ | 327/398 [00:02<00:00, 183.69it/s]
Loading weights: 87%|████████▋ | 346/398 [00:02<00:00, 175.75it/s]
Loading weights: 92%|█████████▏| 367/398 [00:02<00:00, 183.35it/s]
Loading weights: 97%|█████████▋| 388/398 [00:02<00:00, 186.89it/s]
Loading weights: 100%|██████████| 398/398 [00:02<00:00, 163.77it/s] |
| [perf] attention implementation: sdpa |
| [perf] Liger kernels applied via apply_liger_kernel_to_qwen3. |
| Trainable parameters: 20,971,520 (20.97M) |
| [ckpt] step 25 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_25.pt |
| [ckpt] step 50 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_50.pt |
| [ckpt] step 75 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_75.pt |
| [ckpt] step 100 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_100.pt |
| [ckpt] step 125 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_125.pt |
| [ckpt] step 150 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_150.pt |
| [ckpt] step 175 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_175.pt |
| [ckpt] step 200 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_200.pt |
| [ckpt] step 225 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_225.pt |
| |
| [step 250] === ACTIONABLE DIAGNOSTIC === |
| Recall@K_eval: 0.812 |
| PPL gap (relative): 2.283% |
| >> WORKING: High recall and quality preserved. Both set and ranking aligned with the teacher. |
| |
| [ckpt] step 250 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_250.pt |
| [ckpt] step 275 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_275.pt |
| [ckpt] step 300 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_300.pt |
| [ckpt] step 325 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_325.pt |
| [ckpt] step 350 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_350.pt |
| [ckpt] step 375 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_375.pt |
| [ckpt] step 400 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_400.pt |
| [ckpt] step 425 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_425.pt |
| [ckpt] step 450 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_450.pt |
| [ckpt] step 475 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_475.pt |
| |
| [step 500] === ACTIONABLE DIAGNOSTIC === |
| Recall@K_eval: 0.823 |
| PPL gap (relative): 1.753% |
| >> WORKING: High recall and quality preserved. Both set and ranking aligned with the teacher. |
| |
| [ckpt] step 500 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_500.pt |
| [ckpt] step 525 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_525.pt |
| [ckpt] step 550 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_550.pt |
| [ckpt] step 575 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_575.pt |
| [ckpt] step 600 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_600.pt |
| [ckpt] step 625 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_625.pt |
| [ckpt] step 650 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_650.pt |
| [ckpt] step 675 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_675.pt |
| [ckpt] step 700 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_700.pt |
| [ckpt] step 725 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_725.pt |
| |
| [step 750] === ACTIONABLE DIAGNOSTIC === |
| Recall@K_eval: 0.825 |
| PPL gap (relative): 1.943% |
| >> WORKING: High recall and quality preserved. Both set and ranking aligned with the teacher. |
| |
| [ckpt] step 750 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_750.pt |
| [ckpt] step 775 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_775.pt |
| [ckpt] step 800 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_800.pt |
| [ckpt] step 825 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_825.pt |
| [ckpt] step 850 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_850.pt |
| [ckpt] step 875 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_875.pt |
| [ckpt] step 900 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_900.pt |
| [ckpt] step 925 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_925.pt |
| [ckpt] step 950 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_950.pt |
| [ckpt] step 975 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_975.pt |
| |
| [step 1000] === ACTIONABLE DIAGNOSTIC === |
| Recall@K_eval: 0.825 |
| PPL gap (relative): 1.746% |
| >> WORKING: High recall and quality preserved. Both set and ranking aligned with the teacher. |
| |
| [ckpt] step 1000 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_1000.pt |
| [ckpt] step 1000 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_1000.pt |
| [1;34mwandb[0m: |
| [1;34mwandb[0m: 🚀 View run [33mall32-d128-block-causal-reserve-0-1-2-35[0m at: [34mhttps://wandb.ai/dalletest123/ann-sparse/runs/hcukr8rw[0m |
| [1;34mwandb[0m: Find logs at: [1;35mwandb/run-20260509_091956-hcukr8rw/logs[0m |
| |
