wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /root/.netrc. wandb: Currently logged in as: dalletest123 to https://api.wandb.ai. Use `wandb login --relogin` to force relogin wandb: Tracking run with wandb version 0.26.1 wandb: Run data is saved locally in /tmp/sparse-attn-git/wandb/run-20260509_091956-hcukr8rw wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run all32-d128-block-causal-reserve-0-1-2-35 wandb: ⭐️ View project at https://wandb.ai/dalletest123/ann-sparse wandb: 🚀 View run at https://wandb.ai/dalletest123/ann-sparse/runs/hcukr8rw Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [config] preset=all32_d128_block run=all32-d128-block-causal-reserve-0-1-2-35 steps=1000 layers=32 d_search=128 ckpt_dir=/tmp/checkpoints_all32_d128_block_reserve_0_1_2_35 Training search projections for layers: [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34] Reserved as full attention: [0, 1, 2, 35] [perf] flash_attention_3 unavailable (ImportError: FlashAttention3 has been toggled on, but it cannot be used due to the following error: the package for FlashAttention3 doesn't seem to be installed.); trying next. [perf] flash_attention_2 unavailable (ImportError: FlashAttention2 has been toggled on, but it cannot be used due to the following error: the package for FlashAttention2 doesn't seem to be installed.); trying next. Loading weights: 0%| | 0/398 [00:00 /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_25.pt [ckpt] step 50 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_50.pt [ckpt] step 75 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_75.pt [ckpt] step 100 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_100.pt [ckpt] step 125 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_125.pt [ckpt] step 150 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_150.pt [ckpt] step 175 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_175.pt [ckpt] step 200 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_200.pt [ckpt] step 225 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_225.pt [step 250] === ACTIONABLE DIAGNOSTIC === Recall@K_eval: 0.812 PPL gap (relative): 2.283% >> WORKING: High recall and quality preserved. Both set and ranking aligned with the teacher. [ckpt] step 250 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_250.pt [ckpt] step 275 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_275.pt [ckpt] step 300 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_300.pt [ckpt] step 325 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_325.pt [ckpt] step 350 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_350.pt [ckpt] step 375 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_375.pt [ckpt] step 400 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_400.pt [ckpt] step 425 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_425.pt [ckpt] step 450 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_450.pt [ckpt] step 475 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_475.pt [step 500] === ACTIONABLE DIAGNOSTIC === Recall@K_eval: 0.823 PPL gap (relative): 1.753% >> WORKING: High recall and quality preserved. Both set and ranking aligned with the teacher. [ckpt] step 500 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_500.pt [ckpt] step 525 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_525.pt [ckpt] step 550 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_550.pt [ckpt] step 575 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_575.pt [ckpt] step 600 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_600.pt [ckpt] step 625 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_625.pt [ckpt] step 650 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_650.pt [ckpt] step 675 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_675.pt [ckpt] step 700 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_700.pt [ckpt] step 725 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_725.pt [step 750] === ACTIONABLE DIAGNOSTIC === Recall@K_eval: 0.825 PPL gap (relative): 1.943% >> WORKING: High recall and quality preserved. Both set and ranking aligned with the teacher. [ckpt] step 750 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_750.pt [ckpt] step 775 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_775.pt [ckpt] step 800 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_800.pt [ckpt] step 825 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_825.pt [ckpt] step 850 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_850.pt [ckpt] step 875 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_875.pt [ckpt] step 900 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_900.pt [ckpt] step 925 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_925.pt [ckpt] step 950 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_950.pt [ckpt] step 975 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_975.pt [step 1000] === ACTIONABLE DIAGNOSTIC === Recall@K_eval: 0.825 PPL gap (relative): 1.746% >> WORKING: High recall and quality preserved. Both set and ranking aligned with the teacher. [ckpt] step 1000 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_1000.pt [ckpt] step 1000 -> /tmp/checkpoints_all32_d128_block_reserve_0_1_2_35/search_step_1000.pt wandb: wandb: 🚀 View run all32-d128-block-causal-reserve-0-1-2-35 at: https://wandb.ai/dalletest123/ann-sparse/runs/hcukr8rw wandb: Find logs at: wandb/run-20260509_091956-hcukr8rw/logs