/egr/research-optml/wangc168/anaconda3/envs/py310/lib/python3.10/site-packages/transformers/utils/hub.py:127: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(

Downloading shards:   0%|          | 0/8 [00:00<?, ?it/s]
Downloading shards: 100%|██████████| 8/8 [00:00<00:00, 2715.42it/s]

Loading checkpoint shards:   0%|          | 0/8 [00:00<?, ?it/s]
Loading checkpoint shards:  12%|█▎        | 1/8 [00:00<00:02,  2.59it/s]
Loading checkpoint shards:  25%|██▌       | 2/8 [00:00<00:02,  2.66it/s]
Loading checkpoint shards:  38%|███▊      | 3/8 [00:01<00:01,  2.55it/s]
Loading checkpoint shards:  50%|█████     | 4/8 [00:01<00:01,  2.57it/s]
Loading checkpoint shards:  62%|██████▎   | 5/8 [00:01<00:01,  2.51it/s]
Loading checkpoint shards:  75%|███████▌  | 6/8 [00:02<00:00,  2.50it/s]
Loading checkpoint shards:  88%|████████▊ | 7/8 [00:02<00:00,  2.48it/s]
Loading checkpoint shards: 100%|██████████| 8/8 [00:03<00:00,  2.82it/s]
Loading checkpoint shards: 100%|██████████| 8/8 [00:03<00:00,  2.64it/s]

Downloading shards:   0%|          | 0/8 [00:00<?, ?it/s]
Downloading shards: 100%|██████████| 8/8 [00:00<00:00, 3622.02it/s]

Loading checkpoint shards:   0%|          | 0/8 [00:00<?, ?it/s]
Loading checkpoint shards:  12%|█▎        | 1/8 [00:00<00:02,  2.70it/s]
Loading checkpoint shards:  25%|██▌       | 2/8 [00:00<00:02,  2.74it/s]
Loading checkpoint shards:  38%|███▊      | 3/8 [00:01<00:01,  2.71it/s]
Loading checkpoint shards:  50%|█████     | 4/8 [00:01<00:01,  2.66it/s]
Loading checkpoint shards:  62%|██████▎   | 5/8 [00:01<00:01,  2.55it/s]
Loading checkpoint shards:  75%|███████▌  | 6/8 [00:02<00:00,  2.53it/s]
Loading checkpoint shards:  88%|████████▊ | 7/8 [00:02<00:00,  2.50it/s]
Loading checkpoint shards: 100%|██████████| 8/8 [00:02<00:00,  2.92it/s]
Loading checkpoint shards: 100%|██████████| 8/8 [00:02<00:00,  2.72it/s]
wandb: Currently logged in as: wangc168 (wcs23187) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: setting up run kstsc84q
wandb: Tracking run with wandb version 0.23.1
wandb: Run data is saved locally in /egr/research-optml/wangc168/Muon_wmdp/rmu_wmdp/wandb/run-20260403_161129-kstsc84q
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run muon-v1-rank-models/muon_v1_test_V1_K64_batches150_bs4_alpha1200-1200_steer15-15_muonlr2.6e-3_muonmom0.95_muonwd0.1_adamlr5e-5_seed42_layer7_layers5-6-7_params6
wandb: ⭐️ View project at https://wandb.ai/wcs23187/Muon-constraint
wandb: 🚀 View run at https://wandb.ai/wcs23187/Muon-constraint/runs/kstsc84q
====rmu Config====
model_name_or_path=HuggingFaceH4/zephyr-7b-beta
module_str={model_name}.model.layers[{layer_id}]
output_dir=models/muon_v1_test_V1_K64_batches150_bs4_alpha1200-1200_steer15-15_muonlr2.6e-3_muonmom0.95_muonwd0.1_adamlr5e-5_seed42_layer7_layers5-6-7_params6
retain_corpora=['wikitext', 'wikitext']
forget_corpora=['bio_remove_dataset', 'cyber-forget-corpus']
alpha=[1200.0, 1200.0]
steering_coeffs=15,15
muon_lr=0.0026
muon_momentum=0.95
muon_weight_decay=0.1
muon_ns_steps=5
muon_nesterov=True
adam_emb_lr=5e-05
adam_scalar_lr=5e-05
enable_muon_svd_logging=False
M_stats_path=
M_check_T=0
V1_K=64
min_len=0
max_len=2000
batch_size=4
max_num_batches=150
layer_id=7
layer_ids=[5, 6, 7]
param_ids=[6]
seed=42
verbose=False
steering_coeff_list=[15.0, 15.0]
=====
MUON model.layers.5.mlp.down_proj.weight (4096, 14336)
MUON model.layers.6.mlp.down_proj.weight (4096, 14336)
MUON model.layers.7.mlp.down_proj.weight (4096, 14336)
======= Epoch 0 =======

  0%|          | 0/150 [00:00<?, ?it/s]We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)
/egr/research-optml/wangc168/Muon_wmdp/rmu_wmdp/rmu_wb/unlearn_muon_v1.py:201: UserWarning: Using a target size (torch.Size([1, 1, 4096])) that is different to the input size (torch.Size([4, 512, 4096])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  unlearn_loss = torch.nn.functional.mse_loss(

  1%|          | 1/150 [00:04<12:20,  4.97s/it]/egr/research-optml/wangc168/Muon_wmdp/rmu_wmdp/rmu_wb/unlearn_muon_v1.py:201: UserWarning: Using a target size (torch.Size([1, 1, 4096])) that is different to the input size (torch.Size([4, 768, 4096])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  unlearn_loss = torch.nn.functional.mse_loss(

  1%|▏         | 2/150 [00:13<16:51,  6.84s/it]
  2%|▏         | 3/150 [00:21<18:04,  7.38s/it]
  3%|▎         | 4/150 [00:29<18:34,  7.63s/it]
  3%|▎         | 5/150 [00:37<18:45,  7.76s/it]
  4%|▍         | 6/150 [00:45<18:48,  7.84s/it]
  5%|▍         | 7/150 [00:53<18:48,  7.89s/it]
  5%|▌         | 8/150 [01:01<18:43,  7.91s/it]
  6%|▌         | 9/150 [01:09<18:40,  7.94s/it]
  7%|▋         | 10/150 [01:17<18:32,  7.95s/it]
  7%|▋         | 11/150 [01:25<18:28,  7.97s/it]
  8%|▊         | 12/150 [01:33<18:21,  7.98s/it]
  9%|▊         | 13/150 [01:41<18:15,  8.00s/it]
  9%|▉         | 14/150 [01:49<18:10,  8.02s/it]
 10%|█         | 15/150 [01:57<17:57,  7.98s/it]
 11%|█         | 16/150 [02:04<17:46,  7.96s/it]
 11%|█▏        | 17/150 [02:12<17:36,  7.95s/it]
 12%|█▏        | 18/150 [02:20<17:29,  7.95s/it]
 13%|█▎        | 19/150 [02:28<17:19,  7.94s/it]
 13%|█▎        | 20/150 [02:36<17:10,  7.93s/it]
 14%|█▍        | 21/150 [02:44<17:02,  7.92s/it]
 15%|█▍        | 22/150 [02:52<16:55,  7.94s/it]
 15%|█▌        | 23/150 [03:00<16:47,  7.93s/it]
 16%|█▌        | 24/150 [03:08<16:37,  7.92s/it]
 17%|█▋        | 25/150 [03:16<16:33,  7.94s/it]
 17%|█▋        | 26/150 [03:24<16:24,  7.94s/it]
 18%|█▊        | 27/150 [03:32<16:17,  7.94s/it]
 19%|█▊        | 28/150 [03:40<16:09,  7.95s/it]
 19%|█▉        | 29/150 [03:48<15:59,  7.93s/it]
 20%|██        | 30/150 [03:56<15:50,  7.92s/it]
 21%|██        | 31/150 [04:04<15:45,  7.95s/it]
 21%|██▏       | 32/150 [04:11<15:37,  7.94s/it]
 22%|██▏       | 33/150 [04:19<15:29,  7.94s/it]
 23%|██▎       | 34/150 [04:27<15:21,  7.95s/it]
 23%|██▎       | 35/150 [04:35<15:17,  7.98s/it]
 24%|██▍       | 36/150 [04:43<15:07,  7.96s/it]
 25%|██▍       | 37/150 [04:51<15:00,  7.97s/it]
 25%|██▌       | 38/150 [04:59<14:53,  7.98s/it]
 26%|██▌       | 39/150 [05:07<14:41,  7.94s/it]
 27%|██▋       | 40/150 [05:15<14:33,  7.94s/it]
 27%|██▋       | 41/150 [05:23<14:26,  7.95s/it]
 28%|██▊       | 42/150 [05:31<14:19,  7.96s/it]
 29%|██▊       | 43/150 [05:39<14:10,  7.95s/it]
 29%|██▉       | 44/150 [05:47<14:02,  7.95s/it]
 30%|███       | 45/150 [05:55<13:54,  7.95s/it]
 31%|███       | 46/150 [06:03<13:46,  7.95s/it]
 31%|███▏      | 47/150 [06:11<13:46,  8.02s/it]
 32%|███▏      | 48/150 [06:19<13:33,  7.97s/it]
 33%|███▎      | 49/150 [06:27<13:22,  7.95s/it]
 33%|███▎      | 50/150 [06:35<13:15,  7.96s/it]
 34%|███▍      | 51/150 [06:43<13:05,  7.94s/it]
 35%|███▍      | 52/150 [06:51<12:56,  7.93s/it]
 35%|███▌      | 53/150 [06:58<12:49,  7.93s/it]
 36%|███▌      | 54/150 [07:06<12:41,  7.93s/it]
 37%|███▋      | 55/150 [07:14<12:33,  7.94s/it]
 37%|███▋      | 56/150 [07:22<12:26,  7.94s/it]
 38%|███▊      | 57/150 [07:30<12:18,  7.95s/it]
 39%|███▊      | 58/150 [07:38<12:12,  7.96s/it]
 39%|███▉      | 59/150 [07:46<12:05,  7.97s/it]
 40%|████      | 60/150 [07:54<11:59,  7.99s/it]
 41%|████      | 61/150 [08:02<11:52,  8.00s/it]now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1377 | unlearn_loss: 0.1377 | retain_loss: 0 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.0791 | unlearn_loss: 0.0791 | retain_loss: 4.816e-05 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1426 | unlearn_loss: 0.1416 | retain_loss: 0.001022 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07959 | unlearn_loss: 0.0791 | retain_loss: 0.0006828 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1387 | unlearn_loss: 0.1377 | retain_loss: 0.0008354 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07959 | unlearn_loss: 0.07861 | retain_loss: 0.0009842 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1357 | unlearn_loss: 0.1357 | retain_loss: 0.0004139 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07959 | unlearn_loss: 0.07861 | retain_loss: 0.0009918 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1416 | unlearn_loss: 0.1406 | retain_loss: 0.0009003 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.0791 | unlearn_loss: 0.07861 | retain_loss: 0.0004692 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1377 | unlearn_loss: 0.1367 | retain_loss: 0.001373 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.0791 | unlearn_loss: 0.07861 | retain_loss: 0.0004501 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1377 | unlearn_loss: 0.1367 | retain_loss: 0.0005684 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.0791 | unlearn_loss: 0.07861 | retain_loss: 0.0007057 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1436 | unlearn_loss: 0.1396 | retain_loss: 0.003998 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.08008 | unlearn_loss: 0.07861 | retain_loss: 0.001457 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1377 | unlearn_loss: 0.1367 | retain_loss: 0.001244 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.08008 | unlearn_loss: 0.07861 | retain_loss: 0.001556 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1396 | unlearn_loss: 0.1387 | retain_loss: 0.001022 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.0791 | unlearn_loss: 0.07812 | retain_loss: 0.0009079 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1406 | unlearn_loss: 0.1377 | retain_loss: 0.003036 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07959 | unlearn_loss: 0.07812 | retain_loss: 0.001228 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1406 | unlearn_loss: 0.1387 | retain_loss: 0.001526 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.0791 | unlearn_loss: 0.07812 | retain_loss: 0.0009193 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1406 | unlearn_loss: 0.1396 | retain_loss: 0.001152 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.0791 | unlearn_loss: 0.07812 | retain_loss: 0.001007 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1387 | unlearn_loss: 0.1357 | retain_loss: 0.003036 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.08008 | unlearn_loss: 0.07812 | retain_loss: 0.001801 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1396 | unlearn_loss: 0.1367 | retain_loss: 0.002579 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.0791 | unlearn_loss: 0.07764 | retain_loss: 0.001518 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1377 | unlearn_loss: 0.1367 | retain_loss: 0.001404 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07861 | unlearn_loss: 0.07764 | retain_loss: 0.001099 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1387 | unlearn_loss: 0.1357 | retain_loss: 0.002899 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.08008 | unlearn_loss: 0.07812 | retain_loss: 0.002121 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1406 | unlearn_loss: 0.1387 | retain_loss: 0.001907 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07959 | unlearn_loss: 0.07812 | retain_loss: 0.001358 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1416 | unlearn_loss: 0.1367 | retain_loss: 0.004425 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.08057 | unlearn_loss: 0.07812 | retain_loss: 0.002625 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1426 | unlearn_loss: 0.1396 | retain_loss: 0.002457 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07959 | unlearn_loss: 0.07764 | retain_loss: 0.001823 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1416 | unlearn_loss: 0.1367 | retain_loss: 0.004456 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.08008 | unlearn_loss: 0.07715 | retain_loss: 0.002975 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1455 | unlearn_loss: 0.1416 | retain_loss: 0.003616 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07959 | unlearn_loss: 0.07715 | retain_loss: 0.002594 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1416 | unlearn_loss: 0.1387 | retain_loss: 0.002579 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.0791 | unlearn_loss: 0.07715 | retain_loss: 0.002182 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1445 | unlearn_loss: 0.1396 | retain_loss: 0.004578 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.08057 | unlearn_loss: 0.07764 | retain_loss: 0.002914 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1416 | unlearn_loss: 0.1377 | retain_loss: 0.00351 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07959 | unlearn_loss: 0.07715 | retain_loss: 0.002396 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1396 | unlearn_loss: 0.1367 | retain_loss: 0.003021 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07861 | unlearn_loss: 0.07666 | retain_loss: 0.001945 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1377 | unlearn_loss: 0.1348 | retain_loss: 0.002563 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.0791 | unlearn_loss: 0.07715 | retain_loss: 0.001762 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1416 | unlearn_loss: 0.1338 | retain_loss: 0.00824 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.08252 | unlearn_loss: 0.07617 | retain_loss: 0.006256 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1279 | unlearn_loss: 0.1226 | retain_loss: 0.005066 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.08057 | unlearn_loss: 0.07617 | retain_loss: 0.004272 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1406 | unlearn_loss: 0.1309 | retain_loss: 0.009583 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.08447 | unlearn_loss: 0.07617 | retain_loss: 0.008179 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1396 | unlearn_loss: 0.1309 | retain_loss: 0.008667 | param_change: 0
now we use  the mode ..  v1
 41%|████▏     | 62/150 [08:10<11:45,  8.02s/it]
 42%|████▏     | 63/150 [08:18<11:38,  8.03s/it]
 43%|████▎     | 64/150 [08:26<11:31,  8.05s/it]
 43%|████▎     | 65/150 [08:34<11:21,  8.01s/it]
 44%|████▍     | 66/150 [08:42<11:12,  8.01s/it]
 45%|████▍     | 67/150 [08:50<11:03,  7.99s/it]
 45%|████▌     | 68/150 [08:58<10:54,  7.98s/it]
 46%|████▌     | 69/150 [09:06<10:46,  7.98s/it]
 47%|████▋     | 70/150 [09:14<10:37,  7.97s/it]
 47%|████▋     | 71/150 [09:22<10:29,  7.96s/it]
 48%|████▊     | 72/150 [09:30<10:21,  7.96s/it]
 49%|████▊     | 73/150 [09:38<10:14,  7.98s/it]
 49%|████▉     | 74/150 [09:46<10:04,  7.95s/it]
 50%|█████     | 75/150 [09:54<09:59,  7.99s/it]
 51%|█████     | 76/150 [10:02<09:51,  7.99s/it]
 51%|█████▏    | 77/150 [10:10<09:41,  7.97s/it]
 52%|█████▏    | 78/150 [10:18<09:32,  7.95s/it]
 53%|█████▎    | 79/150 [10:26<09:24,  7.95s/it]/egr/research-optml/wangc168/Muon_wmdp/rmu_wmdp/rmu_wb/unlearn_muon_v1.py:201: UserWarning: Using a target size (torch.Size([1, 1, 4096])) that is different to the input size (torch.Size([4, 679, 4096])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  unlearn_loss = torch.nn.functional.mse_loss(

 53%|█████▎    | 80/150 [10:34<09:15,  7.94s/it]
 54%|█████▍    | 81/150 [10:42<09:09,  7.97s/it]
 55%|█████▍    | 82/150 [10:50<09:01,  7.97s/it]
 55%|█████▌    | 83/150 [10:58<08:53,  7.96s/it]
 56%|█████▌    | 84/150 [11:06<08:44,  7.95s/it]
 57%|█████▋    | 85/150 [11:14<08:37,  7.96s/it]
 57%|█████▋    | 86/150 [11:22<08:29,  7.96s/it]
 58%|█████▊    | 87/150 [11:30<08:20,  7.95s/it]
 59%|█████▊    | 88/150 [11:38<08:12,  7.94s/it]
 59%|█████▉    | 89/150 [11:45<08:04,  7.94s/it]
 60%|██████    | 90/150 [11:53<07:56,  7.94s/it]
 61%|██████    | 91/150 [12:01<07:48,  7.94s/it]
 61%|██████▏   | 92/150 [12:09<07:40,  7.95s/it]
 62%|██████▏   | 93/150 [12:17<07:32,  7.93s/it]
 63%|██████▎   | 94/150 [12:25<07:23,  7.92s/it]
 63%|██████▎   | 95/150 [12:33<07:17,  7.95s/it]
 64%|██████▍   | 96/150 [12:41<07:09,  7.95s/it]
 65%|██████▍   | 97/150 [12:49<07:02,  7.97s/it]
 65%|██████▌   | 98/150 [12:57<06:54,  7.98s/it]
 66%|██████▌   | 99/150 [13:05<06:45,  7.96s/it]
 67%|██████▋   | 100/150 [13:13<06:38,  7.98s/it]
 67%|██████▋   | 101/150 [13:21<06:30,  7.96s/it]
 68%|██████▊   | 102/150 [13:29<06:22,  7.97s/it]
 69%|██████▊   | 103/150 [13:37<06:14,  7.97s/it]
 69%|██████▉   | 104/150 [13:45<06:06,  7.96s/it]
 70%|███████   | 105/150 [13:53<05:58,  7.97s/it]
 71%|███████   | 106/150 [14:01<05:51,  7.99s/it]
 71%|███████▏  | 107/150 [14:09<05:43,  7.98s/it]
 72%|███████▏  | 108/150 [14:17<05:35,  8.00s/it]
 73%|███████▎  | 109/150 [14:25<05:26,  7.97s/it]
 73%|███████▎  | 110/150 [14:33<05:23,  8.08s/it]
 74%|███████▍  | 111/150 [14:41<05:12,  8.02s/it]
 75%|███████▍  | 112/150 [14:49<05:04,  8.00s/it]
 75%|███████▌  | 113/150 [14:57<04:55,  7.99s/it]
 76%|███████▌  | 114/150 [15:05<04:47,  7.98s/it]
 77%|███████▋  | 115/150 [15:13<04:38,  7.97s/it]
 77%|███████▋  | 116/150 [15:21<04:31,  7.98s/it]
 78%|███████▊  | 117/150 [15:29<04:23,  7.99s/it]
 79%|███████▊  | 118/150 [15:37<04:16,  8.00s/it]
 79%|███████▉  | 119/150 [15:45<04:07,  7.99s/it]
 80%|████████  | 120/150 [15:53<03:59,  7.98s/it]
 81%|████████  | 121/150 [16:01<03:51,  7.98s/it]
 81%|████████▏ | 122/150 [16:09<03:43,  7.98s/it]
k_eff 64
k_eff 64
k_eff 64
loss: 0.08398 | unlearn_loss: 0.07666 | retain_loss: 0.007538 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1416 | unlearn_loss: 0.1328 | retain_loss: 0.008362 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.08301 | unlearn_loss: 0.07568 | retain_loss: 0.007507 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1426 | unlearn_loss: 0.1377 | retain_loss: 0.004974 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07959 | unlearn_loss: 0.0752 | retain_loss: 0.004547 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1367 | unlearn_loss: 0.1318 | retain_loss: 0.004517 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07812 | unlearn_loss: 0.07422 | retain_loss: 0.00386 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1318 | unlearn_loss: 0.127 | retain_loss: 0.004547 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07861 | unlearn_loss: 0.07471 | retain_loss: 0.003769 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1309 | unlearn_loss: 0.127 | retain_loss: 0.00415 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07764 | unlearn_loss: 0.07373 | retain_loss: 0.003708 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1289 | unlearn_loss: 0.125 | retain_loss: 0.004395 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07666 | unlearn_loss: 0.07275 | retain_loss: 0.003891 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1357 | unlearn_loss: 0.1279 | retain_loss: 0.007477 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07715 | unlearn_loss: 0.07178 | retain_loss: 0.005463 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1318 | unlearn_loss: 0.126 | retain_loss: 0.005585 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07568 | unlearn_loss: 0.07129 | retain_loss: 0.004608 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1289 | unlearn_loss: 0.1245 | retain_loss: 0.004852 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07764 | unlearn_loss: 0.07324 | retain_loss: 0.004211 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.124 | unlearn_loss: 0.1196 | retain_loss: 0.004608 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07471 | unlearn_loss: 0.07031 | retain_loss: 0.004242 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.126 | unlearn_loss: 0.1196 | retain_loss: 0.006012 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07422 | unlearn_loss: 0.06934 | retain_loss: 0.004913 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1309 | unlearn_loss: 0.127 | retain_loss: 0.004333 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07275 | unlearn_loss: 0.06885 | retain_loss: 0.003906 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1216 | unlearn_loss: 0.1177 | retain_loss: 0.003967 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07129 | unlearn_loss: 0.06787 | retain_loss: 0.003448 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1245 | unlearn_loss: 0.1211 | retain_loss: 0.003632 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07129 | unlearn_loss: 0.06787 | retain_loss: 0.003189 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1162 | unlearn_loss: 0.1128 | retain_loss: 0.003662 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07178 | unlearn_loss: 0.06885 | retain_loss: 0.003113 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.125 | unlearn_loss: 0.1191 | retain_loss: 0.006256 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.0708 | unlearn_loss: 0.06592 | retain_loss: 0.004822 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1172 | unlearn_loss: 0.1113 | retain_loss: 0.005798 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07324 | unlearn_loss: 0.06836 | retain_loss: 0.004913 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.126 | unlearn_loss: 0.1206 | retain_loss: 0.005646 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.07178 | unlearn_loss: 0.06689 | retain_loss: 0.004944 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.124 | unlearn_loss: 0.1196 | retain_loss: 0.004517 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.06787 | unlearn_loss: 0.06396 | retain_loss: 0.00386 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1191 | unlearn_loss: 0.1128 | retain_loss: 0.006439 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.06982 | unlearn_loss: 0.06445 | retain_loss: 0.005249 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1167 | unlearn_loss: 0.1123 | retain_loss: 0.004242 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.06592 | unlearn_loss: 0.0625 | retain_loss: 0.003296 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1104 | unlearn_loss: 0.1055 | retain_loss: 0.0047 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.06689 | unlearn_loss: 0.06299 | retain_loss: 0.003937 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.106 | unlearn_loss: 0.1021 | retain_loss: 0.004059 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.06177 | unlearn_loss: 0.05811 | retain_loss: 0.003555 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1157 | unlearn_loss: 0.1113 | retain_loss: 0.004517 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.06079 | unlearn_loss: 0.05688 | retain_loss: 0.00383 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.105 | unlearn_loss: 0.1011 | retain_loss: 0.00412 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.06055 | unlearn_loss: 0.05688 | retain_loss: 0.00354 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1094 | unlearn_loss: 0.105 | retain_loss: 0.004242 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.06006 | unlearn_loss: 0.05615 | retain_loss: 0.003815 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.104 | unlearn_loss: 0.1001 | retain_loss: 0.003906 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.05859 | unlearn_loss: 0.05518 | retain_loss: 0.003357 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1128 | unlearn_loss: 0.1084 | retain_loss: 0.004456 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.05664 | unlearn_loss: 0.05298 | retain_loss: 0.003769 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1123 | unlearn_loss: 0.1084 | retain_loss: 0.003708 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.05615 | unlearn_loss: 0.05298 | retain_loss: 0.003235 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.124 | unlearn_loss: 0.1191 | retain_loss: 0.004852 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.05591 | unlearn_loss: 0.05176 | retain_loss: 0.004211 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64

 82%|████████▏ | 123/150 [16:17<03:35,  7.97s/it]
 83%|████████▎ | 124/150 [16:25<03:27,  7.99s/it]
 83%|████████▎ | 125/150 [16:33<03:19,  7.99s/it]
 84%|████████▍ | 126/150 [16:41<03:11,  7.98s/it]
 85%|████████▍ | 127/150 [16:49<03:03,  7.96s/it]
 85%|████████▌ | 128/150 [16:57<02:55,  7.98s/it]
 86%|████████▌ | 129/150 [17:04<02:47,  7.96s/it]
 87%|████████▋ | 130/150 [17:12<02:39,  7.96s/it]
 87%|████████▋ | 131/150 [17:20<02:31,  7.95s/it]
 88%|████████▊ | 132/150 [17:28<02:23,  7.94s/it]
 89%|████████▊ | 133/150 [17:36<02:14,  7.93s/it]
 89%|████████▉ | 134/150 [17:44<02:07,  7.95s/it]
 90%|█████████ | 135/150 [17:52<01:59,  7.95s/it]
 91%|█████████ | 136/150 [18:00<01:51,  8.00s/it]
 91%|█████████▏| 137/150 [18:08<01:44,  8.01s/it]
 92%|█████████▏| 138/150 [18:17<01:37,  8.15s/it]
 93%|█████████▎| 139/150 [18:25<01:29,  8.11s/it]
 93%|█████████▎| 140/150 [18:33<01:20,  8.07s/it]
 94%|█████████▍| 141/150 [18:41<01:12,  8.03s/it]
 95%|█████████▍| 142/150 [18:49<01:04,  8.01s/it]
 95%|█████████▌| 143/150 [18:57<00:55,  7.99s/it]
 96%|█████████▌| 144/150 [19:05<00:47,  7.97s/it]
 97%|█████████▋| 145/150 [19:12<00:39,  7.95s/it]
 97%|█████████▋| 146/150 [19:20<00:31,  7.95s/it]
 98%|█████████▊| 147/150 [19:28<00:23,  7.96s/it]
 99%|█████████▊| 148/150 [19:36<00:15,  7.96s/it]
 99%|█████████▉| 149/150 [19:44<00:07,  7.94s/it]
100%|██████████| 150/150 [19:52<00:00,  7.94s/it]
100%|██████████| 150/150 [19:52<00:00,  7.95s/it]
wandb: updating run metadata
wandb: uploading config.yaml
wandb: 
wandb: Run history:
wandb:         loss ▄▄▄▇▄██▄▄▄▄▃▄██▄█▄▄█▇▃▇▃▆▃▆▇▆▃▂▅▆▂▂▁▂▅▁▁
wandb: param_change ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:  retain_loss ▁▁▁▁▁▂▁▂▂▂▂▂▂▁▂▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂█
wandb:         step ▁▁▁▁▁▁▂▂▂▂▂▂▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▆▆▆▆▇▇▇▇▇▇███
wandb: unlearn_loss ▃█▃█▃▃██▃█▃▃▃█▃██▃▃▇▇▆▃▃▃▆▃▅▅▂▂▅▅▆▆▄▁▅▄▄
wandb: 
wandb: Run summary:
wandb:         loss 0.06348
wandb: param_change 0
wandb:  retain_loss 0.0249
wandb:         step 149
wandb: unlearn_loss 0.03882
wandb: 
wandb: 🚀 View run muon-v1-rank-models/muon_v1_test_V1_K64_batches150_bs4_alpha1200-1200_steer15-15_muonlr2.6e-3_muonmom0.95_muonwd0.1_adamlr5e-5_seed42_layer7_layers5-6-7_params6 at: https://wandb.ai/wcs23187/Muon-constraint/runs/kstsc84q
wandb: ⭐️ View project at: https://wandb.ai/wcs23187/Muon-constraint
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20260403_161129-kstsc84q/logs
loss: 0.103 | unlearn_loss: 0.09912 | retain_loss: 0.003784 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.05322 | unlearn_loss: 0.05005 | retain_loss: 0.003296 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1099 | unlearn_loss: 0.1055 | retain_loss: 0.004242 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.05298 | unlearn_loss: 0.04956 | retain_loss: 0.003494 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1064 | unlearn_loss: 0.1025 | retain_loss: 0.003967 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.05298 | unlearn_loss: 0.04956 | retain_loss: 0.003494 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.09619 | unlearn_loss: 0.0918 | retain_loss: 0.004456 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.05127 | unlearn_loss: 0.04736 | retain_loss: 0.003891 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1006 | unlearn_loss: 0.09619 | retain_loss: 0.004608 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.05029 | unlearn_loss: 0.04663 | retain_loss: 0.003723 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.09863 | unlearn_loss: 0.09473 | retain_loss: 0.003815 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.05273 | unlearn_loss: 0.04956 | retain_loss: 0.00325 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.09961 | unlearn_loss: 0.09619 | retain_loss: 0.003571 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.04468 | unlearn_loss: 0.04175 | retain_loss: 0.002975 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.09668 | unlearn_loss: 0.09229 | retain_loss: 0.004425 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.04419 | unlearn_loss: 0.04053 | retain_loss: 0.003708 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1025 | unlearn_loss: 0.09863 | retain_loss: 0.003754 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.04395 | unlearn_loss: 0.04077 | retain_loss: 0.003204 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.08496 | unlearn_loss: 0.08105 | retain_loss: 0.00386 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.04517 | unlearn_loss: 0.04199 | retain_loss: 0.003098 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.09717 | unlearn_loss: 0.09326 | retain_loss: 0.004089 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.04492 | unlearn_loss: 0.0415 | retain_loss: 0.003464 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1064 | unlearn_loss: 0.1011 | retain_loss: 0.005493 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.04321 | unlearn_loss: 0.03882 | retain_loss: 0.004303 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.1064 | unlearn_loss: 0.1006 | retain_loss: 0.005829 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.04443 | unlearn_loss: 0.03955 | retain_loss: 0.004913 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.125 | unlearn_loss: 0.09082 | retain_loss: 0.03394 | param_change: 0
now we use  the mode ..  v1
k_eff 64
k_eff 64
k_eff 64
loss: 0.06348 | unlearn_loss: 0.03882 | retain_loss: 0.0249 | param_change: 0
Saved model to models/muon_v1_test_V1_K64_batches150_bs4_alpha1200-1200_steer15-15_muonlr2.6e-3_muonmom0.95_muonwd0.1_adamlr5e-5_seed42_layer7_layers5-6-7_params6
Saved model path to /egr/research-optml/wangc168/Muon_wmdp/rmu_wmdp/model_dir.txt with index 298