challenge/expA__software_development__arc_challenge_seed42/train.log

[2026-06-10 21:41:13,787][__main__][INFO] - Config:
model:
  model_id: allenai/OLMo-3-1025-7B
  torch_dtype: bfloat16
  attn_implementation: sdpa
  lora:
    r: 8
    target_modules:
    - q_proj
    - k_proj
    - v_proj
    - o_proj
    lora_alpha: 16
    lora_dropout: 0.05
    bias: none
    task_type: CAUSAL_LM
trainer:
  max_steps: 5000
  per_device_train_batch_size: 4
  gradient_accumulation_steps: 4
  learning_rate: 1.0e-05
  lr_scheduler_type: constant
  warmup_steps: 0
  weight_decay: 0.0
  adam_beta1: 0.9
  adam_beta2: 0.999
  adam_epsilon: 1.0e-08
  max_grad_norm: 1.0
  auto_lr: true
  lr_delta: 1.0e-05
  max_walltime_minutes: 340
  logging_steps: 50
  save_steps: 200
  eval_steps: 1000
  report_to:
  - wandb
  bf16: true
  fp16: false
  gradient_checkpointing: true
  length_normalized_loss: true
  ppl_shuffle_mode: segments
  ppl_n_factor: 10
  cooldown_enabled: false
  cooldown_every_k: 200
  cooldown_steps: 50
data:
  max_length: 2048
  batch_size: 4
  num_workers: 0
  max_forget_docs: 2000
  max_retain_docs: 9000
  docs_per_retain_bin: null
  min_tokens: 0
  resample_interval: 0
  seed: 42
  forget_manifest_path: null
topic_bin: software_development
topic_bins: null
output_dir: /storage/ice-shared/cs7634/staff/TDA/arc_rescore/binlevel_redo/challenge/expA__software_development__arc_challenge_seed42
retain_topics: null
seed: 42
wandb_project: data-attribution-unlearn-multiseed
wandb_entity: hcai-lab
forget_texts_file: /storage/ice-shared/cs7634/staff/TDA/arc_rescore/forget_texts_binlevel_challenge/expA__software_development__arc_challenge.parquet

[2026-06-10 21:41:13,796][__main__][INFO] - Target topic(s): software_development
[2026-06-10 21:41:14,041][unlearning.data.dolma_pool][INFO] - Loading 6T-filtered Arrow cache from /storage/ice-shared/cs7634/staff/TDA/arc_rescore/dolma_cache/dolma3_6t_filtered ...
[2026-06-10 21:41:14,250][unlearning.data.dolma_pool][INFO] - Faithful forget set from /storage/ice-shared/cs7634/staff/TDA/arc_rescore/forget_texts_binlevel_challenge/expA__software_development__arc_challenge.parquet: 200 texts
[2026-06-10 21:41:28,981][unlearning.data.dolma_pool][INFO] - Sampled (text-file): 200 forget, 9000 retain.
[2026-06-10 21:41:28,990][unlearning.data.sampling][INFO] - Sampling manifest saved to /storage/ice-shared/cs7634/staff/TDA/arc_rescore/binlevel_redo/challenge/expA__software_development__arc_challenge_seed42/sampling_manifest.json
[2026-06-10 21:41:28,990][unlearning.data.dolma_pool][INFO] - Tokenizing 200 forget + 9000 retain docs ...
[2026-06-10 21:42:12,667][__main__][INFO] - Starting fresh training in: /storage/ice-shared/cs7634/staff/TDA/arc_rescore/binlevel_redo/challenge/expA__software_development__arc_challenge_seed42
[2026-06-10 21:42:12,667][__main__][INFO] - Loading model allenai/OLMo-3-1025-7B ...
[2026-06-10 21:42:12,855][accelerate.utils.modeling][INFO] - We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
[2026-06-10 21:42:19,603][__main__][INFO] - Loading MMLU subset (100 questions) for safety guard ...
[2026-06-10 21:42:19,617][datasets.load][WARNING] - Using the latest cached version of the dataset since cais/mmlu couldn't be found on the Hugging Face Hub (offline mode is enabled).
[2026-06-10 21:42:19,627][datasets.packaged_modules.cache.cache][WARNING] - Found the latest cached dataset configuration 'all' at /home/hice1/gmatlin3/scratch/hf_cache/datasets/cais___mmlu/all/0.0.0/c30699e8356da336a370243923dbaf21066bb9fe (last modified on Mon May 25 20:02:16 2026).
[2026-06-10 21:42:24,466][__main__][INFO] - Original model MMLU accuracy (n=100): 0.5600
[2026-06-10 21:42:26,170][__main__][INFO] - Base model shuffled-token PPL (forget set, mode=segments): 33.8214
[2026-06-10 21:42:29,541][__main__][INFO] - Starting NGDiff unlearning for topic(s) 'software_development'...
[2026-06-10 21:42:29,541][__main__][INFO] - MMLU safety baseline: 0.5600  (threshold: 0.5040)
[2026-06-10 21:44:31,308][unlearning.trainer.ngdiff][INFO] - step=50  opt_step=12  f_loss=1.9342  r_loss=1.8192  mmlu=0.5600  lr=1.00e-05
[2026-06-10 21:46:31,685][unlearning.trainer.ngdiff][INFO] - step=100  opt_step=25  f_loss=1.9950  r_loss=2.1148  mmlu=0.5600  lr=1.00e-05
[2026-06-10 21:48:31,927][unlearning.trainer.ngdiff][INFO] - step=150  opt_step=37  f_loss=1.5786  r_loss=2.0658  mmlu=0.5600  lr=1.00e-05
[2026-06-10 21:50:31,664][unlearning.trainer.ngdiff][INFO] - step=200  opt_step=50  f_loss=2.5302  r_loss=2.8601  mmlu=0.5600  lr=3.69e-05
[2026-06-10 21:50:39,563][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=50: forget=12.5495  retain=10.2414
[2026-06-10 21:52:37,000][unlearning.trainer.ngdiff][INFO] - step=250  opt_step=62  f_loss=2.3535  r_loss=2.6546  mmlu=0.5600  lr=1.00e-05
[2026-06-10 21:54:32,525][unlearning.trainer.ngdiff][INFO] - step=300  opt_step=75  f_loss=2.2861  r_loss=2.4078  mmlu=0.5600  lr=1.00e-05
[2026-06-10 21:56:31,513][unlearning.trainer.ngdiff][INFO] - step=350  opt_step=87  f_loss=2.3618  r_loss=2.4846  mmlu=0.5600  lr=1.00e-05
[2026-06-10 21:58:32,289][unlearning.trainer.ngdiff][INFO] - step=400  opt_step=100  f_loss=2.5975  r_loss=2.5423  mmlu=0.5600  lr=1.00e-07
[2026-06-10 21:58:37,856][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=100: forget=12.5576  retain=10.2352
[2026-06-10 22:00:38,997][unlearning.trainer.ngdiff][INFO] - step=450  opt_step=112  f_loss=2.2136  r_loss=2.4375  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:02:37,674][unlearning.trainer.ngdiff][INFO] - step=500  opt_step=125  f_loss=1.9281  r_loss=1.4285  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:04:35,945][unlearning.trainer.ngdiff][INFO] - step=550  opt_step=137  f_loss=2.0134  r_loss=2.3413  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:06:33,817][unlearning.trainer.ngdiff][INFO] - step=600  opt_step=150  f_loss=2.1317  r_loss=1.9231  mmlu=0.5600  lr=4.59e-06
[2026-06-10 22:06:39,386][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=150: forget=12.5758  retain=10.2213
[2026-06-10 22:08:36,527][unlearning.trainer.ngdiff][INFO] - step=650  opt_step=162  f_loss=1.7897  r_loss=2.1232  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:10:36,620][unlearning.trainer.ngdiff][INFO] - step=700  opt_step=175  f_loss=2.2838  r_loss=2.1795  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:12:33,676][unlearning.trainer.ngdiff][INFO] - step=750  opt_step=187  f_loss=2.3875  r_loss=2.4258  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:14:32,654][unlearning.trainer.ngdiff][INFO] - step=800  opt_step=200  f_loss=2.1594  r_loss=2.2368  mmlu=0.5700  lr=1.00e-07
[2026-06-10 22:14:38,220][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=200: forget=12.5968  retain=10.1913
[2026-06-10 22:16:37,552][unlearning.trainer.ngdiff][INFO] - step=850  opt_step=212  f_loss=2.5226  r_loss=2.3141  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:18:33,235][unlearning.trainer.ngdiff][INFO] - step=900  opt_step=225  f_loss=2.5851  r_loss=2.3600  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:20:30,797][unlearning.trainer.ngdiff][INFO] - step=950  opt_step=237  f_loss=2.5773  r_loss=2.2950  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:22:29,593][unlearning.trainer.ngdiff][INFO] - step=1000  opt_step=250  f_loss=2.0661  r_loss=1.9653  mmlu=0.5600  lr=3.40e-06
[2026-06-10 22:22:35,163][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=250: forget=12.6247  retain=10.1635
[2026-06-10 22:24:34,275][unlearning.trainer.ngdiff][INFO] - step=1050  opt_step=262  f_loss=2.4972  r_loss=2.4188  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:26:36,272][unlearning.trainer.ngdiff][INFO] - step=1100  opt_step=275  f_loss=2.3315  r_loss=2.1740  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:28:33,510][unlearning.trainer.ngdiff][INFO] - step=1150  opt_step=287  f_loss=2.2186  r_loss=2.7399  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:30:28,649][unlearning.trainer.ngdiff][INFO] - step=1200  opt_step=300  f_loss=2.3440  r_loss=1.9980  mmlu=0.5600  lr=1.00e-07
[2026-06-10 22:30:34,222][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=300: forget=12.6740  retain=10.1280
[2026-06-10 22:32:31,870][unlearning.trainer.ngdiff][INFO] - step=1250  opt_step=312  f_loss=2.4116  r_loss=2.7144  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:34:29,122][unlearning.trainer.ngdiff][INFO] - step=1300  opt_step=325  f_loss=1.9328  r_loss=2.1666  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:36:25,722][unlearning.trainer.ngdiff][INFO] - step=1350  opt_step=337  f_loss=2.0941  r_loss=1.3656  mmlu=0.5600  lr=1.00e-05
[2026-06-10 22:38:24,488][unlearning.trainer.ngdiff][INFO] - step=1400  opt_step=350  f_loss=2.9752  r_loss=2.1972  mmlu=0.5600  lr=1.00e-07
[2026-06-10 22:38:30,059][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=350: forget=12.7396  retain=10.0846
[2026-06-10 22:40:27,543][unlearning.trainer.ngdiff][INFO] - step=1450  opt_step=362  f_loss=1.5483  r_loss=1.6715  mmlu=0.5500  lr=1.00e-05
[2026-06-10 22:42:23,747][unlearning.trainer.ngdiff][INFO] - step=1500  opt_step=375  f_loss=1.5960  r_loss=2.6208  mmlu=0.5500  lr=1.00e-05
[2026-06-10 22:44:21,561][unlearning.trainer.ngdiff][INFO] - step=1550  opt_step=387  f_loss=2.2563  r_loss=2.3927  mmlu=0.5500  lr=1.00e-05
[2026-06-10 22:46:18,675][unlearning.trainer.ngdiff][INFO] - step=1600  opt_step=400  f_loss=1.8324  r_loss=2.0282  mmlu=0.5500  lr=1.00e-07
[2026-06-10 22:46:24,251][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=400: forget=12.8317  retain=10.0423
[2026-06-10 22:48:24,235][unlearning.trainer.ngdiff][INFO] - step=1650  opt_step=412  f_loss=1.9814  r_loss=2.1594  mmlu=0.5500  lr=1.00e-05
[2026-06-10 22:50:22,252][unlearning.trainer.ngdiff][INFO] - step=1700  opt_step=425  f_loss=1.9772  r_loss=2.1477  mmlu=0.5500  lr=1.00e-05
[2026-06-10 22:52:19,022][unlearning.trainer.ngdiff][INFO] - step=1750  opt_step=437  f_loss=1.8802  r_loss=1.5406  mmlu=0.5500  lr=1.00e-05
[2026-06-10 22:54:16,649][unlearning.trainer.ngdiff][INFO] - step=1800  opt_step=450  f_loss=2.6681  r_loss=1.8974  mmlu=0.5500  lr=7.46e-06
[2026-06-10 22:54:22,219][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=450: forget=12.9267  retain=10.0159
[2026-06-10 22:56:21,320][unlearning.trainer.ngdiff][INFO] - step=1850  opt_step=462  f_loss=2.2032  r_loss=2.6290  mmlu=0.5500  lr=1.00e-05
[2026-06-10 22:58:19,977][unlearning.trainer.ngdiff][INFO] - step=1900  opt_step=475  f_loss=1.5752  r_loss=2.4905  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:00:15,574][unlearning.trainer.ngdiff][INFO] - step=1950  opt_step=487  f_loss=2.4883  r_loss=2.0024  mmlu=0.5600  lr=1.00e-05
[2026-06-10 23:02:16,031][unlearning.trainer.ngdiff][INFO] - step=2000  opt_step=500  f_loss=1.8361  r_loss=2.8894  mmlu=0.5500  lr=1.00e-07
[2026-06-10 23:02:21,608][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=500: forget=13.0664  retain=9.9591
[2026-06-10 23:04:20,069][unlearning.trainer.ngdiff][INFO] - step=2050  opt_step=512  f_loss=1.7969  r_loss=1.9311  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:06:20,591][unlearning.trainer.ngdiff][INFO] - step=2100  opt_step=525  f_loss=2.1193  r_loss=2.9789  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:08:22,417][unlearning.trainer.ngdiff][INFO] - step=2150  opt_step=537  f_loss=1.7888  r_loss=2.3706  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:10:18,088][unlearning.trainer.ngdiff][INFO] - step=2200  opt_step=550  f_loss=2.2581  r_loss=1.5369  mmlu=0.5500  lr=1.39e-05
[2026-06-10 23:10:23,679][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=550: forget=13.2448  retain=9.9323
[2026-06-10 23:12:25,801][unlearning.trainer.ngdiff][INFO] - step=2250  opt_step=562  f_loss=1.9289  r_loss=2.0559  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:14:23,861][unlearning.trainer.ngdiff][INFO] - step=2300  opt_step=575  f_loss=2.4632  r_loss=2.3129  mmlu=0.5400  lr=1.00e-05
[2026-06-10 23:16:22,096][unlearning.trainer.ngdiff][INFO] - step=2350  opt_step=587  f_loss=2.4468  r_loss=2.2309  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:18:17,348][unlearning.trainer.ngdiff][INFO] - step=2400  opt_step=600  f_loss=2.3821  r_loss=2.0774  mmlu=0.5400  lr=1.61e-06
[2026-06-10 23:18:22,917][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=600: forget=13.5177  retain=9.9043
[2026-06-10 23:20:20,406][unlearning.trainer.ngdiff][INFO] - step=2450  opt_step=612  f_loss=2.2753  r_loss=2.2893  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:22:19,866][unlearning.trainer.ngdiff][INFO] - step=2500  opt_step=625  f_loss=2.6275  r_loss=2.4085  mmlu=0.5400  lr=1.00e-05
[2026-06-10 23:24:19,882][unlearning.trainer.ngdiff][INFO] - step=2550  opt_step=637  f_loss=2.5972  r_loss=2.4917  mmlu=0.5400  lr=1.00e-05
[2026-06-10 23:26:18,114][unlearning.trainer.ngdiff][INFO] - step=2600  opt_step=650  f_loss=2.5199  r_loss=2.3844  mmlu=0.5500  lr=1.15e-05
[2026-06-10 23:26:23,678][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=650: forget=13.8895  retain=9.8691
[2026-06-10 23:28:23,399][unlearning.trainer.ngdiff][INFO] - step=2650  opt_step=662  f_loss=2.3198  r_loss=2.2728  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:30:20,294][unlearning.trainer.ngdiff][INFO] - step=2700  opt_step=675  f_loss=2.4707  r_loss=2.4743  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:32:19,473][unlearning.trainer.ngdiff][INFO] - step=2750  opt_step=687  f_loss=2.4171  r_loss=2.1865  mmlu=0.5600  lr=1.00e-05
[2026-06-10 23:34:17,334][unlearning.trainer.ngdiff][INFO] - step=2800  opt_step=700  f_loss=2.0202  r_loss=2.4900  mmlu=0.5400  lr=1.00e-07
[2026-06-10 23:34:22,896][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=700: forget=14.5543  retain=9.8130
[2026-06-10 23:36:24,637][unlearning.trainer.ngdiff][INFO] - step=2850  opt_step=712  f_loss=2.4672  r_loss=1.6831  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:38:23,465][unlearning.trainer.ngdiff][INFO] - step=2900  opt_step=725  f_loss=2.1588  r_loss=2.3225  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:40:20,236][unlearning.trainer.ngdiff][INFO] - step=2950  opt_step=737  f_loss=2.5177  r_loss=2.3316  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:42:17,853][unlearning.trainer.ngdiff][INFO] - step=3000  opt_step=750  f_loss=2.3686  r_loss=2.5297  mmlu=0.5400  lr=1.00e-07
[2026-06-10 23:42:23,427][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=750: forget=15.9445  retain=9.8333
[2026-06-10 23:44:21,762][unlearning.trainer.ngdiff][INFO] - step=3050  opt_step=762  f_loss=2.5947  r_loss=2.2352  mmlu=0.5500  lr=1.00e-05
[2026-06-10 23:46:18,887][unlearning.trainer.ngdiff][INFO] - step=3100  opt_step=775  f_loss=2.2645  r_loss=1.6914  mmlu=0.5300  lr=1.00e-05
[2026-06-10 23:48:15,205][unlearning.trainer.ngdiff][INFO] - step=3150  opt_step=787  f_loss=2.8270  r_loss=1.3421  mmlu=0.5300  lr=1.00e-05
[2026-06-10 23:50:11,423][unlearning.trainer.ngdiff][INFO] - step=3200  opt_step=800  f_loss=2.4175  r_loss=2.0665  mmlu=0.5200  lr=1.00e-07
[2026-06-10 23:50:16,953][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=800: forget=18.2140  retain=9.9117
[2026-06-10 23:52:12,890][unlearning.trainer.ngdiff][INFO] - step=3250  opt_step=812  f_loss=2.8985  r_loss=2.2323  mmlu=0.5200  lr=1.00e-05
[2026-06-10 23:54:09,285][unlearning.trainer.ngdiff][INFO] - step=3300  opt_step=825  f_loss=2.2495  r_loss=2.3690  mmlu=0.5200  lr=1.00e-05
[2026-06-10 23:56:06,547][unlearning.trainer.ngdiff][INFO] - step=3350  opt_step=837  f_loss=1.9016  r_loss=1.6075  mmlu=0.5200  lr=1.00e-05
[2026-06-10 23:58:07,622][unlearning.trainer.ngdiff][INFO] - step=3400  opt_step=850  f_loss=2.7846  r_loss=1.5755  mmlu=0.5200  lr=1.00e-07
[2026-06-10 23:58:13,131][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=850: forget=25.0166  retain=10.1404
[2026-06-11 00:00:10,825][unlearning.trainer.ngdiff][INFO] - step=3450  opt_step=862  f_loss=1.3504  r_loss=2.0656  mmlu=0.5300  lr=1.00e-05
[2026-06-11 00:02:05,687][unlearning.trainer.ngdiff][INFO] - step=3500  opt_step=875  f_loss=2.5110  r_loss=2.3606  mmlu=0.5300  lr=1.00e-05
[2026-06-11 00:04:05,472][unlearning.trainer.ngdiff][INFO] - step=3550  opt_step=887  f_loss=3.4372  r_loss=2.9506  mmlu=0.5300  lr=1.00e-05
[2026-06-11 00:06:05,581][unlearning.trainer.ngdiff][INFO] - step=3600  opt_step=900  f_loss=2.4486  r_loss=2.3315  mmlu=0.5300  lr=1.00e-07
[2026-06-11 00:06:11,080][unlearning.trainer.ngdiff][INFO] - PPL at optimizer_step=900: forget=46.3498  retain=10.6072
[2026-06-11 00:06:11,085][unlearning.trainer.ngdiff][INFO] - PPL stopping triggered at optimizer_step=900: PPL=46.3498 >= threshold=33.8214
[2026-06-11 00:06:11,516][__main__][INFO] - LoRA adapter saved to /storage/ice-shared/cs7634/staff/TDA/arc_rescore/binlevel_redo/challenge/expA__software_development__arc_challenge_seed42/adapter (step 936)
[2026-06-11 00:06:11,516][__main__][INFO] - Done.