[2026-05-12 22:50:52,003][model.qat][INFO] - QAT: wrapped 112 nn.Linear modules (n_bits=8, group_size=128, symmetric=False, train_quant_params=True, skip=['lm_head', 'embed_tokens'])
[2026-05-12 22:50:55,104][model][INFO] - Setting pad_token as eos token: <|eot_id|>
[2026-05-12 22:51:01,427][evaluator][INFO] - Evaluations stored in the experiment directory: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT
[2026-05-12 22:51:01,631][trainer][INFO] - GradDiff Trainer loaded, output_dir: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT
[2026-05-13 10:13:07,004][model.qat][INFO] - QAT: wrapped 112 nn.Linear modules (n_bits=8, group_size=128, symmetric=False, train_quant_params=True, skip=['lm_head', 'embed_tokens'])
[2026-05-13 10:13:07,423][model][INFO] - Setting pad_token as eos token: <|eot_id|>
[2026-05-13 10:13:15,051][evaluator][INFO] - Evaluations stored in the experiment directory: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT
[2026-05-13 10:13:15,536][trainer][INFO] - GradDiff Trainer loaded, output_dir: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT
[2026-05-13 10:13:15,822][accelerate.accelerator][WARNING] - [RANK 0] Gradient accumulation steps mismatch: GradientAccumulationPlugin has 1, DeepSpeed config has 4. Using DeepSpeed's value.
[2026-05-13 10:13:30,458][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 10:13:30,458][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-0/evals/TOFU_EVAL.json
[2026-05-13 10:13:30,458][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-0/evals/TOFU_SUMMARY.json
[2026-05-13 10:13:37,044][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 10:13:57,346][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 10:14:31,216][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 10:14:31,217][evaluator][INFO] - Result for metric forget_truth_ratio:	0.47705465705382816
[2026-05-13 10:14:31,222][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 10:14:31,223][metrics][INFO] - Evaluating forget_quality
[2026-05-13 10:14:31,223][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 10:14:31,223][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 10:14:36,634][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 10:14:41,245][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.87974609375
[2026-05-13 10:14:42,632][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 10:15:56,822][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.8168176212351753
[2026-05-13 10:15:59,174][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 10:16:10,419][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 10:16:50,609][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 10:16:57,321][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 10:17:18,375][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 10:17:24,689][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 10:17:27,498][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 10:17:30,320][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 10:17:31,547][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 10:17:43,182][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 10:17:43,182][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:17:43,182][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 10:17:45,868][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 10:17:48,045][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 10:17:52,090][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 10:17:53,373][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 10:18:11,022][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 10:18:11,023][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:18:11,023][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 10:18:11,023][metrics][INFO] - Evaluating model_utility
[2026-05-13 10:18:11,024][evaluator][INFO] - Result for metric model_utility:	0.5985466594660509
[2026-05-13 10:18:18,939][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 10:18:22,220][metrics][INFO] - Evaluating privleak
[2026-05-13 10:18:22,220][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 10:18:22,220][evaluator][INFO] - Result for metric privleak:	-99.34374998013125
[2026-05-13 10:18:24,136][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 10:18:25,031][evaluator][INFO] - Result for metric extraction_strength:	0.7109042565278512
[2026-05-13 10:19:30,766][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 10:19:30,766][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-7/evals/TOFU_EVAL.json
[2026-05-13 10:19:30,767][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-7/evals/TOFU_SUMMARY.json
[2026-05-13 10:19:36,580][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 10:19:40,871][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 10:19:55,486][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 10:19:55,486][evaluator][INFO] - Result for metric forget_truth_ratio:	0.5006871924207377
[2026-05-13 10:19:55,492][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 10:19:55,492][metrics][INFO] - Evaluating forget_quality
[2026-05-13 10:19:55,492][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 10:19:55,493][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 10:20:01,275][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 10:20:04,247][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.1836419677734375
[2026-05-13 10:20:05,492][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 10:22:20,846][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.41414601567222376
[2026-05-13 10:22:26,500][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 10:22:30,734][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 10:23:47,981][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 10:23:52,262][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 10:24:06,789][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 10:24:08,428][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 10:24:10,392][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 10:24:12,483][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 10:24:13,837][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 10:24:29,227][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 10:24:29,227][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:24:29,227][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 10:24:30,942][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 10:24:32,791][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 10:24:34,665][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 10:24:35,969][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 10:24:49,152][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 10:24:49,152][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:24:49,152][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 10:24:49,153][metrics][INFO] - Evaluating model_utility
[2026-05-13 10:24:49,153][evaluator][INFO] - Result for metric model_utility:	0.407200292926334
[2026-05-13 10:24:56,899][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 10:24:58,486][metrics][INFO] - Evaluating privleak
[2026-05-13 10:24:58,486][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 10:24:58,487][evaluator][INFO] - Result for metric privleak:	-89.00812498219837
[2026-05-13 10:24:59,859][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 10:25:00,953][evaluator][INFO] - Result for metric extraction_strength:	0.05924189741665229
[2026-05-13 10:25:21,980][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 10:25:21,980][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-14/evals/TOFU_EVAL.json
[2026-05-13 10:25:21,980][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-14/evals/TOFU_SUMMARY.json
[2026-05-13 10:25:23,647][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 10:25:28,116][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 10:25:42,793][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 10:25:42,794][evaluator][INFO] - Result for metric forget_truth_ratio:	0.4788654053437078
[2026-05-13 10:25:42,799][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 10:25:42,800][metrics][INFO] - Evaluating forget_quality
[2026-05-13 10:25:42,800][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 10:25:42,800][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 10:25:48,723][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 10:25:51,676][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.08483051776885986
[2026-05-13 10:25:52,943][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 10:26:25,467][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.34416566722932734
[2026-05-13 10:26:27,159][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 10:26:31,442][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 10:27:03,889][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 10:27:08,094][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 10:27:22,681][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 10:27:24,399][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 10:27:26,397][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 10:27:28,475][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 10:27:29,704][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 10:27:34,238][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 10:27:34,238][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:27:34,238][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 10:27:35,483][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 10:27:37,296][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 10:27:39,154][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 10:27:40,458][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 10:27:46,127][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 10:27:46,127][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:27:46,127][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 10:27:46,127][metrics][INFO] - Evaluating model_utility
[2026-05-13 10:27:46,128][evaluator][INFO] - Result for metric model_utility:	0.4838133878904574
[2026-05-13 10:27:53,142][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 10:27:54,705][metrics][INFO] - Evaluating privleak
[2026-05-13 10:27:54,705][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 10:27:54,705][evaluator][INFO] - Result for metric privleak:	-33.45624999330876
[2026-05-13 10:27:55,989][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 10:27:57,060][evaluator][INFO] - Result for metric extraction_strength:	0.09940587875409879
[2026-05-13 10:28:14,350][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 10:28:14,350][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-21/evals/TOFU_EVAL.json
[2026-05-13 10:28:14,350][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-21/evals/TOFU_SUMMARY.json
[2026-05-13 10:28:19,733][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 10:28:24,066][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 10:28:38,756][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 10:28:38,757][evaluator][INFO] - Result for metric forget_truth_ratio:	0.4670378053691596
[2026-05-13 10:28:38,763][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 10:28:38,763][metrics][INFO] - Evaluating forget_quality
[2026-05-13 10:28:38,763][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 10:28:38,763][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 10:28:40,414][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 10:28:43,378][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.00890925220965073
[2026-05-13 10:28:44,628][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 10:30:18,635][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.29235083818535434
[2026-05-13 10:30:20,343][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 10:30:24,809][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 10:31:14,851][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 10:31:19,205][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 10:31:33,756][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 10:31:39,404][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 10:31:41,692][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 10:31:43,771][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 10:31:45,025][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 10:32:00,677][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 10:32:00,678][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:32:00,678][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 10:32:02,337][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 10:32:04,177][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 10:32:06,036][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 10:32:07,266][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 10:32:16,242][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 10:32:16,243][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:32:16,243][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 10:32:16,243][metrics][INFO] - Evaluating model_utility
[2026-05-13 10:32:16,244][evaluator][INFO] - Result for metric model_utility:	0.48983935329236644
[2026-05-13 10:32:19,291][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 10:32:20,858][metrics][INFO] - Evaluating privleak
[2026-05-13 10:32:20,859][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 10:32:20,859][evaluator][INFO] - Result for metric privleak:	73.62562498527488
[2026-05-13 10:32:22,127][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 10:32:23,215][evaluator][INFO] - Result for metric extraction_strength:	0.047374115424220306
[2026-05-13 10:32:39,514][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 10:32:39,514][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-28/evals/TOFU_EVAL.json
[2026-05-13 10:32:39,514][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-28/evals/TOFU_SUMMARY.json
[2026-05-13 10:32:41,234][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 10:32:45,447][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 10:33:00,085][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 10:33:00,086][evaluator][INFO] - Result for metric forget_truth_ratio:	0.002696933255154829
[2026-05-13 10:33:00,092][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 10:33:00,092][metrics][INFO] - Evaluating forget_quality
[2026-05-13 10:33:00,093][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 10:33:00,093][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 10:33:01,847][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 10:33:04,814][evaluator][INFO] - Result for metric forget_Q_A_Prob:	1.948253397720595e-05
[2026-05-13 10:33:06,079][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 10:34:59,138][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.01650086453754834
[2026-05-13 10:35:04,817][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 10:35:08,994][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 10:37:02,133][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 10:37:06,353][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 10:37:20,937][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 10:37:22,747][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 10:37:24,932][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 10:37:27,021][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 10:37:28,722][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 10:38:36,750][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 10:38:36,750][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:38:36,750][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 10:38:38,403][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 10:38:40,231][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 10:38:42,107][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 10:38:43,350][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 10:39:45,411][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 10:39:45,412][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:39:45,412][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 10:39:45,412][metrics][INFO] - Evaluating model_utility
[2026-05-13 10:39:45,412][evaluator][INFO] - Result for metric model_utility:	0.40486175048229456
[2026-05-13 10:39:48,519][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 10:39:50,086][metrics][INFO] - Evaluating privleak
[2026-05-13 10:39:50,086][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 10:39:50,087][evaluator][INFO] - Result for metric privleak:	98.494999980301
[2026-05-13 10:39:51,352][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 10:39:52,445][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 10:40:05,127][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 10:40:05,127][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-35/evals/TOFU_EVAL.json
[2026-05-13 10:40:05,127][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-35/evals/TOFU_SUMMARY.json
[2026-05-13 10:40:10,779][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 10:40:15,114][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 10:40:29,806][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 10:40:29,806][evaluator][INFO] - Result for metric forget_truth_ratio:	1.5960496595360732e-05
[2026-05-13 10:40:29,811][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 10:40:29,812][metrics][INFO] - Evaluating forget_quality
[2026-05-13 10:40:29,812][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 10:40:29,812][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 10:40:35,616][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 10:40:38,568][evaluator][INFO] - Result for metric forget_Q_A_Prob:	3.709925050576768e-07
[2026-05-13 10:40:39,847][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 10:42:27,734][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.02630269027733792
[2026-05-13 10:42:32,766][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 10:42:36,850][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 10:43:07,211][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 10:43:11,540][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 10:43:26,132][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 10:43:31,780][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 10:43:33,723][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 10:43:35,818][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 10:43:37,166][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 10:43:56,615][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 10:43:56,615][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:43:56,615][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 10:44:02,211][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 10:44:04,065][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 10:44:05,938][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 10:44:07,270][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 10:44:40,322][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 10:44:40,323][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:44:40,323][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 10:44:40,323][metrics][INFO] - Evaluating model_utility
[2026-05-13 10:44:40,323][evaluator][INFO] - Result for metric model_utility:	0.48362329969675943
[2026-05-13 10:44:47,405][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 10:44:48,990][metrics][INFO] - Evaluating privleak
[2026-05-13 10:44:48,990][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 10:44:48,990][evaluator][INFO] - Result for metric privleak:	99.32437498013512
[2026-05-13 10:44:50,386][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 10:44:51,486][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 10:45:05,129][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 10:45:05,129][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-42/evals/TOFU_EVAL.json
[2026-05-13 10:45:05,130][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-42/evals/TOFU_SUMMARY.json
[2026-05-13 10:45:10,796][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 10:45:14,977][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 10:45:29,587][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 10:45:29,587][evaluator][INFO] - Result for metric forget_truth_ratio:	4.081972845723411e-05
[2026-05-13 10:45:29,592][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 10:45:29,593][metrics][INFO] - Evaluating forget_quality
[2026-05-13 10:45:29,593][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 10:45:29,593][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 10:45:35,300][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 10:45:38,272][evaluator][INFO] - Result for metric forget_Q_A_Prob:	1.142715101423615e-06
[2026-05-13 10:45:39,673][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 10:47:26,135][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.027156757898764088
[2026-05-13 10:47:31,777][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 10:47:36,149][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 10:48:05,188][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 10:48:09,405][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 10:48:23,997][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 10:48:25,702][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 10:48:27,834][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 10:48:29,922][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 10:48:31,172][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 10:48:43,235][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 10:48:43,235][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:48:43,235][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 10:48:48,804][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 10:48:50,747][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 10:48:52,604][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 10:48:53,855][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 10:49:12,845][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 10:49:12,845][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:49:12,845][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 10:49:12,845][metrics][INFO] - Evaluating model_utility
[2026-05-13 10:49:12,846][evaluator][INFO] - Result for metric model_utility:	0.4808609112702393
[2026-05-13 10:49:19,782][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 10:49:21,347][metrics][INFO] - Evaluating privleak
[2026-05-13 10:49:21,347][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 10:49:21,347][evaluator][INFO] - Result for metric privleak:	99.27874998014424
[2026-05-13 10:49:22,608][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 10:49:23,698][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 10:49:39,827][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 10:49:39,828][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-49/evals/TOFU_EVAL.json
[2026-05-13 10:49:39,828][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-49/evals/TOFU_SUMMARY.json
[2026-05-13 10:49:41,750][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 10:49:45,960][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 10:50:00,614][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 10:50:00,614][evaluator][INFO] - Result for metric forget_truth_ratio:	5.51433307131689e-05
[2026-05-13 10:50:00,620][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 10:50:00,620][metrics][INFO] - Evaluating forget_quality
[2026-05-13 10:50:00,620][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 10:50:00,620][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 10:50:04,990][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 10:50:07,975][evaluator][INFO] - Result for metric forget_Q_A_Prob:	9.273054357651032e-07
[2026-05-13 10:50:09,314][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 10:51:56,421][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.03125428696853153
[2026-05-13 10:51:58,148][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 10:52:02,292][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 10:52:30,691][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 10:52:34,907][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 10:52:49,609][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 10:52:54,852][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 10:52:56,812][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 10:52:58,920][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 10:53:00,204][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 10:53:06,005][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 10:53:06,005][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:53:06,005][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 10:53:07,643][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 10:53:09,454][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 10:53:11,331][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 10:53:12,587][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 10:53:45,864][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 10:53:45,864][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:53:45,864][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 10:53:45,864][metrics][INFO] - Evaluating model_utility
[2026-05-13 10:53:45,865][evaluator][INFO] - Result for metric model_utility:	0.4892190473756089
[2026-05-13 10:53:49,198][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 10:53:50,766][metrics][INFO] - Evaluating privleak
[2026-05-13 10:53:50,767][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 10:53:50,767][evaluator][INFO] - Result for metric privleak:	99.36124998012775
[2026-05-13 10:53:52,143][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 10:53:53,231][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 10:54:10,898][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 10:54:10,899][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-56/evals/TOFU_EVAL.json
[2026-05-13 10:54:10,899][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-56/evals/TOFU_SUMMARY.json
[2026-05-13 10:54:12,614][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 10:54:16,853][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 10:54:31,581][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 10:54:31,582][evaluator][INFO] - Result for metric forget_truth_ratio:	1.0176140978785472e-05
[2026-05-13 10:54:31,587][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 10:54:31,587][metrics][INFO] - Evaluating forget_quality
[2026-05-13 10:54:31,587][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 10:54:31,587][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 10:54:37,187][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 10:54:40,192][evaluator][INFO] - Result for metric forget_Q_A_Prob:	7.110820942058316e-07
[2026-05-13 10:54:41,459][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 10:56:28,499][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.031914009190753746
[2026-05-13 10:56:30,113][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 10:56:34,189][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 10:57:13,823][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 10:57:18,144][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 10:57:32,770][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 10:57:38,437][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 10:57:40,424][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 10:57:42,520][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 10:57:43,845][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 10:58:02,828][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 10:58:02,829][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:58:02,829][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 10:58:04,478][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 10:58:06,349][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 10:58:08,207][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 10:58:09,576][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 10:58:42,115][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 10:58:42,116][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 10:58:42,116][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 10:58:42,116][metrics][INFO] - Evaluating model_utility
[2026-05-13 10:58:42,116][evaluator][INFO] - Result for metric model_utility:	0.4994966541320685
[2026-05-13 10:58:49,065][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 10:58:50,634][metrics][INFO] - Evaluating privleak
[2026-05-13 10:58:50,635][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 10:58:50,635][evaluator][INFO] - Result for metric privleak:	99.15874998016825
[2026-05-13 10:58:51,903][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 10:58:53,002][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 10:59:00,101][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 10:59:00,101][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_EVAL.json
[2026-05-13 10:59:00,101][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_SUMMARY.json
[2026-05-13 10:59:05,048][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 10:59:09,256][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 10:59:23,897][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 10:59:23,898][evaluator][INFO] - Result for metric forget_truth_ratio:	9.252015280586869e-06
[2026-05-13 10:59:23,903][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 10:59:23,904][metrics][INFO] - Evaluating forget_quality
[2026-05-13 10:59:23,904][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 10:59:23,904][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 10:59:29,620][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 10:59:32,599][evaluator][INFO] - Result for metric forget_Q_A_Prob:	6.573859578862728e-07
[2026-05-13 10:59:34,000][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 11:01:21,484][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.03141659317008192
[2026-05-13 11:01:23,119][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 11:01:27,542][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 11:02:02,516][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 11:02:06,771][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 11:02:21,401][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 11:02:23,065][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 11:02:25,022][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 11:02:27,122][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 11:02:28,410][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 11:02:47,353][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 11:02:47,354][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 11:02:47,354][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 11:02:53,099][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 11:02:54,918][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 11:02:56,780][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 11:02:58,009][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 11:03:30,579][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 11:03:30,579][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 11:03:30,579][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 11:03:30,579][metrics][INFO] - Evaluating model_utility
[2026-05-13 11:03:30,580][evaluator][INFO] - Result for metric model_utility:	0.5018110010150688
[2026-05-13 11:03:33,606][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 11:03:35,173][metrics][INFO] - Evaluating privleak
[2026-05-13 11:03:35,173][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 11:03:35,173][evaluator][INFO] - Result for metric privleak:	99.11437498017712
[2026-05-13 11:03:36,409][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 11:03:37,496][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 11:03:37,554][model.qat][INFO] - QAT: baked 112 QATLinear modules back into nn.Linear
[2026-05-13 11:03:38,928][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 11:03:38,928][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_EVAL.json
[2026-05-13 11:03:38,928][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_SUMMARY.json
[2026-05-13 11:03:44,477][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 12:59:45,702][model.qat][INFO] - QAT: wrapped 112 nn.Linear modules (n_bits=8, group_size=128, symmetric=False, train_quant_params=True, skip=['lm_head', 'embed_tokens'])
[2026-05-13 12:59:46,102][model][INFO] - Setting pad_token as eos token: <|eot_id|>
[2026-05-13 12:59:49,937][evaluator][INFO] - Evaluations stored in the experiment directory: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT
[2026-05-13 12:59:50,470][trainer][INFO] - GradDiff Trainer loaded, output_dir: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT
[2026-05-13 12:59:50,883][accelerate.accelerator][WARNING] - [RANK 0] Gradient accumulation steps mismatch: GradientAccumulationPlugin has 1, DeepSpeed config has 4. Using DeepSpeed's value.
[2026-05-13 13:00:05,976][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 13:00:05,976][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-0/evals/TOFU_EVAL.json
[2026-05-13 13:00:05,976][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-0/evals/TOFU_SUMMARY.json
[2026-05-13 13:00:08,837][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 13:00:25,240][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 13:00:59,349][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 13:00:59,350][evaluator][INFO] - Result for metric forget_truth_ratio:	0.47705465705382816
[2026-05-13 13:00:59,356][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 13:00:59,356][metrics][INFO] - Evaluating forget_quality
[2026-05-13 13:00:59,357][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 13:00:59,357][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 13:01:01,443][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 13:01:06,082][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.87974609375
[2026-05-13 13:01:07,741][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 13:02:22,589][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.8168176212351753
[2026-05-13 13:02:24,231][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 13:02:34,154][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 13:03:10,746][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 13:03:20,314][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 13:03:41,319][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 13:03:42,988][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 13:03:45,817][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 13:03:48,615][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 13:03:49,906][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 13:04:01,488][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 13:04:01,488][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:04:01,488][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 13:04:07,107][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 13:04:09,420][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 13:04:13,406][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 13:04:14,803][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 13:04:32,579][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 13:04:32,580][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:04:32,580][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 13:04:32,580][metrics][INFO] - Evaluating model_utility
[2026-05-13 13:04:32,581][evaluator][INFO] - Result for metric model_utility:	0.5985466594660509
[2026-05-13 13:04:35,614][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 13:04:38,916][metrics][INFO] - Evaluating privleak
[2026-05-13 13:04:38,917][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 13:04:38,917][evaluator][INFO] - Result for metric privleak:	-99.34374998013125
[2026-05-13 13:04:40,309][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 13:04:41,224][evaluator][INFO] - Result for metric extraction_strength:	0.7109042565278512
[2026-05-13 13:05:47,504][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 13:05:47,504][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-7/evals/TOFU_EVAL.json
[2026-05-13 13:05:47,505][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-7/evals/TOFU_SUMMARY.json
[2026-05-13 13:05:50,147][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 13:05:54,566][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 13:06:09,246][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 13:06:09,247][evaluator][INFO] - Result for metric forget_truth_ratio:	0.5006871924207377
[2026-05-13 13:06:09,252][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 13:06:09,253][metrics][INFO] - Evaluating forget_quality
[2026-05-13 13:06:09,253][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 13:06:09,253][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 13:06:10,912][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 13:06:13,892][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.1836419677734375
[2026-05-13 13:06:15,296][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 13:08:33,302][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.41414601567222376
[2026-05-13 13:08:37,005][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 13:08:41,107][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 13:10:04,203][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 13:10:08,610][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 13:10:23,237][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 13:10:24,883][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 13:10:26,815][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 13:10:28,909][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 13:10:30,309][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 13:10:45,782][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 13:10:45,782][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:10:45,782][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 13:10:47,586][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 13:10:49,589][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 13:10:51,458][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 13:10:52,682][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 13:11:05,894][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 13:11:05,895][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:11:05,895][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 13:11:05,895][metrics][INFO] - Evaluating model_utility
[2026-05-13 13:11:05,896][evaluator][INFO] - Result for metric model_utility:	0.407200292926334
[2026-05-13 13:11:08,981][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 13:11:10,570][metrics][INFO] - Evaluating privleak
[2026-05-13 13:11:10,570][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 13:11:10,570][evaluator][INFO] - Result for metric privleak:	-89.00812498219837
[2026-05-13 13:11:11,839][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 13:11:12,983][evaluator][INFO] - Result for metric extraction_strength:	0.05924189741665229
[2026-05-13 13:11:34,464][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 13:11:34,464][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-14/evals/TOFU_EVAL.json
[2026-05-13 13:11:34,464][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-14/evals/TOFU_SUMMARY.json
[2026-05-13 13:11:36,236][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 13:11:40,522][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 13:11:55,293][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 13:11:55,293][evaluator][INFO] - Result for metric forget_truth_ratio:	0.4788654053437078
[2026-05-13 13:11:55,299][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 13:11:55,300][metrics][INFO] - Evaluating forget_quality
[2026-05-13 13:11:55,300][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 13:11:55,300][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 13:11:56,951][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 13:11:59,946][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.08483051776885986
[2026-05-13 13:12:01,244][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 13:12:34,483][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.34416566722932734
[2026-05-13 13:12:36,157][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 13:12:40,473][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 13:13:10,384][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 13:13:14,612][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 13:13:29,217][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 13:13:30,891][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 13:13:32,868][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 13:13:34,973][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 13:13:36,212][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 13:13:40,938][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 13:13:40,938][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:13:40,938][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 13:13:42,244][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 13:13:44,495][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 13:13:46,380][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 13:13:47,618][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 13:13:53,426][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 13:13:53,426][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:13:53,426][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 13:13:53,426][metrics][INFO] - Evaluating model_utility
[2026-05-13 13:13:53,427][evaluator][INFO] - Result for metric model_utility:	0.4838133878904574
[2026-05-13 13:13:56,342][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 13:13:57,975][metrics][INFO] - Evaluating privleak
[2026-05-13 13:13:57,976][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 13:13:57,976][evaluator][INFO] - Result for metric privleak:	-33.45624999330876
[2026-05-13 13:13:59,401][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 13:14:00,547][evaluator][INFO] - Result for metric extraction_strength:	0.09940587875409879
[2026-05-13 13:14:18,471][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 13:14:18,472][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-21/evals/TOFU_EVAL.json
[2026-05-13 13:14:18,472][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-21/evals/TOFU_SUMMARY.json
[2026-05-13 13:14:20,180][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 13:14:24,386][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 13:14:39,140][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 13:14:39,140][evaluator][INFO] - Result for metric forget_truth_ratio:	0.4670378053691596
[2026-05-13 13:14:39,147][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 13:14:39,147][metrics][INFO] - Evaluating forget_quality
[2026-05-13 13:14:39,147][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 13:14:39,147][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 13:14:40,812][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 13:14:43,787][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.00890925220965073
[2026-05-13 13:14:45,328][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 13:16:21,698][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.29235083818535434
[2026-05-13 13:16:23,306][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 13:16:27,471][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 13:17:18,567][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 13:17:23,290][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 13:17:37,919][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 13:17:39,692][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 13:17:41,652][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 13:17:43,764][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 13:17:45,020][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 13:18:01,150][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 13:18:01,151][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:18:01,151][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 13:18:02,822][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 13:18:06,054][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 13:18:07,945][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 13:18:09,230][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 13:18:18,415][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 13:18:18,415][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:18:18,415][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 13:18:18,415][metrics][INFO] - Evaluating model_utility
[2026-05-13 13:18:18,416][evaluator][INFO] - Result for metric model_utility:	0.48983935329236644
[2026-05-13 13:18:21,669][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 13:18:23,274][metrics][INFO] - Evaluating privleak
[2026-05-13 13:18:23,274][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 13:18:23,274][evaluator][INFO] - Result for metric privleak:	73.62562498527488
[2026-05-13 13:18:24,732][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 13:18:25,847][evaluator][INFO] - Result for metric extraction_strength:	0.047374115424220306
[2026-05-13 13:18:42,727][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 13:18:42,727][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-28/evals/TOFU_EVAL.json
[2026-05-13 13:18:42,727][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-28/evals/TOFU_SUMMARY.json
[2026-05-13 13:18:44,392][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 13:18:48,788][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 13:19:03,506][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 13:19:03,507][evaluator][INFO] - Result for metric forget_truth_ratio:	0.002696933255154829
[2026-05-13 13:19:03,513][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 13:19:03,514][metrics][INFO] - Evaluating forget_quality
[2026-05-13 13:19:03,514][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 13:19:03,514][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 13:19:06,020][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 13:19:08,985][evaluator][INFO] - Result for metric forget_Q_A_Prob:	1.948253397720595e-05
[2026-05-13 13:19:10,263][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 13:21:04,974][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.01650086453754834
[2026-05-13 13:21:06,639][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 13:21:10,755][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 13:23:01,251][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 13:23:05,841][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 13:23:20,408][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 13:23:22,605][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 13:23:24,698][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 13:23:26,795][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 13:23:28,052][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 13:24:37,770][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 13:24:37,770][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:24:37,770][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 13:24:40,676][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 13:24:42,769][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 13:24:44,691][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 13:24:45,931][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 13:25:49,152][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 13:25:49,153][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:25:49,153][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 13:25:49,153][metrics][INFO] - Evaluating model_utility
[2026-05-13 13:25:49,154][evaluator][INFO] - Result for metric model_utility:	0.40486175048229456
[2026-05-13 13:25:52,149][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 13:25:53,802][metrics][INFO] - Evaluating privleak
[2026-05-13 13:25:53,803][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 13:25:53,803][evaluator][INFO] - Result for metric privleak:	98.494999980301
[2026-05-13 13:25:55,074][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 13:25:56,189][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 13:26:09,555][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 13:26:09,555][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-35/evals/TOFU_EVAL.json
[2026-05-13 13:26:09,555][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-35/evals/TOFU_SUMMARY.json
[2026-05-13 13:26:11,229][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 13:26:15,475][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 13:26:30,196][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 13:26:30,197][evaluator][INFO] - Result for metric forget_truth_ratio:	1.5960496595360732e-05
[2026-05-13 13:26:30,203][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 13:26:30,203][metrics][INFO] - Evaluating forget_quality
[2026-05-13 13:26:30,203][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 13:26:30,203][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 13:26:31,846][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 13:26:34,859][evaluator][INFO] - Result for metric forget_Q_A_Prob:	3.709925050576768e-07
[2026-05-13 13:26:36,126][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 13:28:26,229][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.02630269027733792
[2026-05-13 13:28:28,070][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 13:28:32,214][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 13:29:00,022][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 13:29:04,296][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 13:29:18,910][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 13:29:20,556][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 13:29:22,506][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 13:29:24,600][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 13:29:26,198][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 13:29:46,112][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 13:29:46,112][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:29:46,113][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 13:29:47,816][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 13:29:49,713][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 13:29:51,618][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 13:29:52,977][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 13:30:27,588][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 13:30:27,588][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:30:27,588][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 13:30:27,589][metrics][INFO] - Evaluating model_utility
[2026-05-13 13:30:27,589][evaluator][INFO] - Result for metric model_utility:	0.48362329969675943
[2026-05-13 13:30:32,291][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 13:30:33,945][metrics][INFO] - Evaluating privleak
[2026-05-13 13:30:33,945][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 13:30:33,945][evaluator][INFO] - Result for metric privleak:	99.32437498013512
[2026-05-13 13:30:35,222][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 13:30:36,374][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 13:30:50,511][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 13:30:50,511][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-42/evals/TOFU_EVAL.json
[2026-05-13 13:30:50,512][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-42/evals/TOFU_SUMMARY.json
[2026-05-13 13:30:52,192][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 13:30:56,491][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 13:31:11,305][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 13:31:11,306][evaluator][INFO] - Result for metric forget_truth_ratio:	4.081972845723411e-05
[2026-05-13 13:31:11,311][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 13:31:11,312][metrics][INFO] - Evaluating forget_quality
[2026-05-13 13:31:11,312][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 13:31:11,312][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 13:31:12,976][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 13:31:15,953][evaluator][INFO] - Result for metric forget_Q_A_Prob:	1.142715101423615e-06
[2026-05-13 13:31:17,188][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 13:33:08,074][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.027156757898764088
[2026-05-13 13:33:09,721][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 13:33:13,850][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 13:33:39,922][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 13:33:44,153][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 13:33:58,789][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 13:34:03,850][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 13:34:05,940][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 13:34:08,039][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 13:34:09,304][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 13:34:21,696][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 13:34:21,697][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:34:21,697][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 13:34:23,338][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 13:34:25,174][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 13:34:27,051][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 13:34:28,291][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 13:34:48,030][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 13:34:48,030][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:34:48,030][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 13:34:48,031][metrics][INFO] - Evaluating model_utility
[2026-05-13 13:34:48,031][evaluator][INFO] - Result for metric model_utility:	0.4808609112702393
[2026-05-13 13:34:51,515][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 13:34:53,120][metrics][INFO] - Evaluating privleak
[2026-05-13 13:34:53,120][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 13:34:53,120][evaluator][INFO] - Result for metric privleak:	99.27874998014424
[2026-05-13 13:34:54,384][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 13:34:55,514][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 13:35:12,042][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 13:35:12,042][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-49/evals/TOFU_EVAL.json
[2026-05-13 13:35:12,042][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-49/evals/TOFU_SUMMARY.json
[2026-05-13 13:35:16,769][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 13:35:21,009][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 13:35:35,827][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 13:35:35,828][evaluator][INFO] - Result for metric forget_truth_ratio:	5.51433307131689e-05
[2026-05-13 13:35:35,833][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 13:35:35,834][metrics][INFO] - Evaluating forget_quality
[2026-05-13 13:35:35,834][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 13:35:35,834][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 13:35:41,702][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 13:35:44,676][evaluator][INFO] - Result for metric forget_Q_A_Prob:	9.273054357651032e-07
[2026-05-13 13:35:46,261][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 13:37:37,277][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.03125428696853153
[2026-05-13 13:37:42,226][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 13:37:46,303][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 13:38:19,412][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 13:38:23,650][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 13:38:38,335][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 13:38:40,112][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 13:38:42,044][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 13:38:44,164][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 13:38:45,452][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 13:38:51,436][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 13:38:51,437][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:38:51,437][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 13:38:53,127][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 13:38:55,134][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 13:38:57,026][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 13:38:58,342][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 13:39:32,261][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 13:39:32,261][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:39:32,261][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 13:39:32,261][metrics][INFO] - Evaluating model_utility
[2026-05-13 13:39:32,262][evaluator][INFO] - Result for metric model_utility:	0.4892190473756089
[2026-05-13 13:39:36,660][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 13:39:38,323][metrics][INFO] - Evaluating privleak
[2026-05-13 13:39:38,324][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 13:39:38,324][evaluator][INFO] - Result for metric privleak:	99.36124998012775
[2026-05-13 13:39:39,651][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 13:39:40,807][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 13:39:58,979][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 13:39:58,979][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-56/evals/TOFU_EVAL.json
[2026-05-13 13:39:58,979][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-56/evals/TOFU_SUMMARY.json
[2026-05-13 13:40:04,907][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 13:40:09,482][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 13:40:24,239][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 13:40:24,239][evaluator][INFO] - Result for metric forget_truth_ratio:	1.0176140978785472e-05
[2026-05-13 13:40:24,245][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 13:40:24,246][metrics][INFO] - Evaluating forget_quality
[2026-05-13 13:40:24,246][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 13:40:24,246][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 13:40:26,040][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 13:40:29,010][evaluator][INFO] - Result for metric forget_Q_A_Prob:	7.110820942058316e-07
[2026-05-13 13:40:30,573][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 13:42:21,029][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.031914009190753746
[2026-05-13 13:42:26,686][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 13:42:31,266][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 13:43:08,245][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 13:43:12,463][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 13:43:27,132][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 13:43:28,777][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 13:43:30,791][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 13:43:32,894][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 13:43:34,223][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 13:43:53,698][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 13:43:53,698][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:43:53,698][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 13:43:55,469][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 13:43:57,312][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 13:43:59,199][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 13:44:00,458][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 13:44:34,673][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 13:44:34,674][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:44:34,674][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 13:44:34,674][metrics][INFO] - Evaluating model_utility
[2026-05-13 13:44:34,674][evaluator][INFO] - Result for metric model_utility:	0.4994966541320685
[2026-05-13 13:44:37,591][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 13:44:39,242][metrics][INFO] - Evaluating privleak
[2026-05-13 13:44:39,242][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 13:44:39,242][evaluator][INFO] - Result for metric privleak:	99.15874998016825
[2026-05-13 13:44:40,707][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 13:44:41,848][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 13:44:49,140][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 13:44:49,141][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_EVAL.json
[2026-05-13 13:44:49,141][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_SUMMARY.json
[2026-05-13 13:44:50,765][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 13:44:54,990][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 13:45:09,710][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 13:45:09,711][evaluator][INFO] - Result for metric forget_truth_ratio:	9.252015280586869e-06
[2026-05-13 13:45:09,716][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 13:45:09,717][metrics][INFO] - Evaluating forget_quality
[2026-05-13 13:45:09,717][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 13:45:09,717][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 13:45:15,580][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 13:45:18,578][evaluator][INFO] - Result for metric forget_Q_A_Prob:	6.573859578862728e-07
[2026-05-13 13:45:19,935][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 13:47:12,082][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.03141659317008192
[2026-05-13 13:47:17,157][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 13:47:21,305][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 13:48:02,156][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 13:48:06,533][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 13:48:21,256][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 13:48:26,268][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 13:48:28,322][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 13:48:30,433][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 13:48:31,680][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 13:48:51,141][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 13:48:51,142][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:48:51,142][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 13:48:56,842][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 13:48:58,680][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 13:49:00,564][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 13:49:01,787][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 13:49:35,930][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 13:49:35,930][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 13:49:35,930][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 13:49:35,930][metrics][INFO] - Evaluating model_utility
[2026-05-13 13:49:35,931][evaluator][INFO] - Result for metric model_utility:	0.5018110010150688
[2026-05-13 13:49:38,855][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 13:49:40,484][metrics][INFO] - Evaluating privleak
[2026-05-13 13:49:40,484][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 13:49:40,484][evaluator][INFO] - Result for metric privleak:	99.11437498017712
[2026-05-13 13:49:41,860][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 13:49:43,019][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 13:49:43,080][model.qat][INFO] - QAT: baked 112 QATLinear modules back into nn.Linear
[2026-05-13 13:50:32,658][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 13:50:32,658][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_EVAL.json
[2026-05-13 13:50:32,658][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_SUMMARY.json
[2026-05-13 13:50:42,536][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 20:22:02,711][model.qat][INFO] - QAT: wrapped 112 nn.Linear modules (n_bits=8, group_size=128, symmetric=False, train_quant_params=True, skip=['lm_head', 'embed_tokens'])
[2026-05-13 20:22:03,136][model][INFO] - Setting pad_token as eos token: <|eot_id|>
[2026-05-13 20:22:06,897][evaluator][INFO] - Evaluations stored in the experiment directory: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT
[2026-05-13 20:22:07,407][trainer][INFO] - GradDiff Trainer loaded, output_dir: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT
[2026-05-13 20:22:07,871][accelerate.accelerator][WARNING] - [RANK 0] Gradient accumulation steps mismatch: GradientAccumulationPlugin has 1, DeepSpeed config has 4. Using DeepSpeed's value.
[2026-05-13 20:22:23,975][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 20:22:23,975][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-0/evals/TOFU_EVAL.json
[2026-05-13 20:22:23,975][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-0/evals/TOFU_SUMMARY.json
[2026-05-13 20:22:25,824][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 20:22:43,340][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 20:23:24,929][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 20:23:24,930][evaluator][INFO] - Result for metric forget_truth_ratio:	0.47705465705382816
[2026-05-13 20:23:24,936][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 20:23:24,936][metrics][INFO] - Evaluating forget_quality
[2026-05-13 20:23:24,936][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 20:23:24,937][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 20:23:26,975][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 20:23:31,804][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.87974609375
[2026-05-13 20:23:32,994][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 20:24:49,022][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.8168176212351753
[2026-05-13 20:24:50,687][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 20:24:59,789][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 20:25:37,592][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 20:25:45,943][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 20:26:14,130][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 20:26:15,775][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 20:26:19,579][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 20:26:22,481][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 20:26:24,112][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 20:26:35,922][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 20:26:35,922][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 20:26:35,922][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 20:26:37,590][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 20:26:40,048][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 20:26:44,161][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 20:26:45,507][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 20:27:03,656][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 20:27:03,656][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 20:27:03,657][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 20:27:03,657][metrics][INFO] - Evaluating model_utility
[2026-05-13 20:27:03,658][evaluator][INFO] - Result for metric model_utility:	0.5985466594660509
[2026-05-13 20:27:06,662][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 20:27:10,023][metrics][INFO] - Evaluating privleak
[2026-05-13 20:27:10,024][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 20:27:10,024][evaluator][INFO] - Result for metric privleak:	-99.34374998013125
[2026-05-13 20:27:11,278][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 20:27:12,232][evaluator][INFO] - Result for metric extraction_strength:	0.7109042565278512
[2026-05-13 20:28:18,904][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 20:28:18,904][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-7/evals/TOFU_EVAL.json
[2026-05-13 20:28:18,904][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-7/evals/TOFU_SUMMARY.json
[2026-05-13 20:28:20,746][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 20:28:28,063][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 20:28:57,200][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 20:28:57,201][evaluator][INFO] - Result for metric forget_truth_ratio:	0.5006871924207377
[2026-05-13 20:28:57,206][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 20:28:57,207][metrics][INFO] - Evaluating forget_quality
[2026-05-13 20:28:57,207][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 20:28:57,207][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 20:28:58,857][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 20:29:04,506][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.1836419677734375
[2026-05-13 20:29:06,207][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 20:31:32,618][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.41414601567222376
[2026-05-13 20:31:34,273][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 20:31:40,287][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 20:33:07,081][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 20:33:13,046][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 20:33:37,415][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 20:33:39,087][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 20:33:41,576][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 20:33:44,607][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 20:33:45,844][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 20:34:01,413][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 20:34:01,413][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 20:34:01,413][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 20:34:03,856][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 20:34:05,709][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 20:34:07,781][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 20:34:09,047][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 20:34:22,633][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 20:34:22,633][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 20:34:22,633][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 20:34:22,634][metrics][INFO] - Evaluating model_utility
[2026-05-13 20:34:22,635][evaluator][INFO] - Result for metric model_utility:	0.407200292926334
[2026-05-13 20:34:25,656][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 20:34:28,401][metrics][INFO] - Evaluating privleak
[2026-05-13 20:34:28,401][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 20:34:28,401][evaluator][INFO] - Result for metric privleak:	-89.00812498219837
[2026-05-13 20:34:29,713][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 20:34:35,381][evaluator][INFO] - Result for metric extraction_strength:	0.05924189741665229
[2026-05-13 20:34:57,388][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 20:34:57,388][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-14/evals/TOFU_EVAL.json
[2026-05-13 20:34:57,388][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-14/evals/TOFU_SUMMARY.json
[2026-05-13 20:34:59,055][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 20:35:06,410][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 20:35:31,793][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 20:35:31,794][evaluator][INFO] - Result for metric forget_truth_ratio:	0.4788654053437078
[2026-05-13 20:35:31,800][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 20:35:31,800][metrics][INFO] - Evaluating forget_quality
[2026-05-13 20:35:31,800][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 20:35:31,800][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 20:35:33,465][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 20:35:37,519][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.08483051776885986
[2026-05-13 20:35:38,782][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 20:36:14,661][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.34416566722932734
[2026-05-13 20:36:16,348][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 20:36:22,381][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 20:36:56,934][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 20:37:04,303][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 20:37:27,603][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 20:37:29,266][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 20:37:31,871][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 20:37:35,367][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 20:37:36,611][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 20:37:41,480][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 20:37:41,481][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 20:37:41,481][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 20:37:42,611][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 20:37:45,131][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 20:37:47,936][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 20:37:49,195][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 20:37:55,301][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 20:37:55,301][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 20:37:55,301][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 20:37:55,301][metrics][INFO] - Evaluating model_utility
[2026-05-13 20:37:55,302][evaluator][INFO] - Result for metric model_utility:	0.4838133878904574
[2026-05-13 20:37:58,187][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 20:38:01,029][metrics][INFO] - Evaluating privleak
[2026-05-13 20:38:01,029][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 20:38:01,029][evaluator][INFO] - Result for metric privleak:	-33.45624999330876
[2026-05-13 20:38:02,287][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 20:38:07,708][evaluator][INFO] - Result for metric extraction_strength:	0.09940587875409879
[2026-05-13 20:38:26,043][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 20:38:26,044][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-21/evals/TOFU_EVAL.json
[2026-05-13 20:38:26,044][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-21/evals/TOFU_SUMMARY.json
[2026-05-13 20:38:27,699][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 20:38:33,711][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 20:38:57,307][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 20:38:57,308][evaluator][INFO] - Result for metric forget_truth_ratio:	0.4670378053691596
[2026-05-13 20:38:57,315][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 20:38:57,315][metrics][INFO] - Evaluating forget_quality
[2026-05-13 20:38:57,315][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 20:38:57,315][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 20:38:58,935][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 20:39:03,867][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.00890925220965073
[2026-05-13 20:39:04,956][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 20:40:50,432][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.29235083818535434
[2026-05-13 20:40:52,127][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 20:40:59,019][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 20:41:58,363][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 20:42:04,100][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 20:42:28,488][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 20:42:30,157][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 20:42:33,828][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 20:42:37,258][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 20:42:38,519][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 20:42:54,842][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 20:42:54,842][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 20:42:54,843][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 20:42:56,460][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 20:42:58,756][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 20:43:01,702][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 20:43:03,044][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 20:43:12,651][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 20:43:12,652][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 20:43:12,652][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 20:43:12,652][metrics][INFO] - Evaluating model_utility
[2026-05-13 20:43:12,652][evaluator][INFO] - Result for metric model_utility:	0.48983935329236644
[2026-05-13 20:43:15,865][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 20:43:20,084][metrics][INFO] - Evaluating privleak
[2026-05-13 20:43:20,084][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 20:43:20,084][evaluator][INFO] - Result for metric privleak:	73.62562498527488
[2026-05-13 20:43:21,400][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 20:43:26,143][evaluator][INFO] - Result for metric extraction_strength:	0.047374115424220306
[2026-05-13 20:43:43,409][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 20:43:43,410][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-28/evals/TOFU_EVAL.json
[2026-05-13 20:43:43,410][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-28/evals/TOFU_SUMMARY.json
[2026-05-13 20:43:45,084][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 20:43:51,096][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 20:44:13,584][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 20:44:13,584][evaluator][INFO] - Result for metric forget_truth_ratio:	0.002696933255154829
[2026-05-13 20:44:13,591][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 20:44:13,591][metrics][INFO] - Evaluating forget_quality
[2026-05-13 20:44:13,591][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 20:44:13,591][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 20:44:15,324][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 20:44:21,022][evaluator][INFO] - Result for metric forget_Q_A_Prob:	1.948253397720595e-05
[2026-05-13 20:44:22,743][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 20:46:31,097][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.01650086453754834
[2026-05-13 20:46:32,805][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 20:46:38,754][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 20:48:38,219][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 20:48:44,368][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 20:49:06,109][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 20:49:07,819][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 20:49:10,259][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 20:49:13,629][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 20:49:14,956][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 20:50:26,776][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 20:50:26,776][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 20:50:26,776][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 20:50:28,441][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 20:50:30,491][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 20:50:33,401][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 20:50:34,648][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 20:51:39,358][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 20:51:39,358][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 20:51:39,358][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 20:51:39,359][metrics][INFO] - Evaluating model_utility
[2026-05-13 20:51:39,359][evaluator][INFO] - Result for metric model_utility:	0.40486175048229456
[2026-05-13 20:51:42,256][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 20:51:45,039][metrics][INFO] - Evaluating privleak
[2026-05-13 20:51:45,039][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 20:51:45,039][evaluator][INFO] - Result for metric privleak:	98.494999980301
[2026-05-13 20:51:46,289][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 20:51:51,522][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 20:52:05,419][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 20:52:05,420][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-35/evals/TOFU_EVAL.json
[2026-05-13 20:52:05,420][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-35/evals/TOFU_SUMMARY.json
[2026-05-13 20:52:07,084][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 20:52:13,071][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 20:52:36,516][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 20:52:36,517][evaluator][INFO] - Result for metric forget_truth_ratio:	1.5960496595360732e-05
[2026-05-13 20:52:36,522][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 20:52:36,522][metrics][INFO] - Evaluating forget_quality
[2026-05-13 20:52:36,522][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 20:52:36,523][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 20:52:38,128][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 20:52:43,881][evaluator][INFO] - Result for metric forget_Q_A_Prob:	3.709925050576768e-07
[2026-05-13 20:52:45,504][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 20:54:47,576][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.02630269027733792
[2026-05-13 20:54:49,253][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 20:54:55,163][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 20:55:24,042][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 20:55:31,105][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 20:55:55,554][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 20:55:57,215][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 20:55:59,727][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 20:56:03,214][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 20:56:04,809][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 20:56:25,962][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 20:56:25,962][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 20:56:25,962][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 20:56:27,628][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 20:56:29,863][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 20:56:32,812][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 20:56:34,078][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 20:57:10,719][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 20:57:10,719][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 20:57:10,719][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 20:57:10,719][metrics][INFO] - Evaluating model_utility
[2026-05-13 20:57:10,720][evaluator][INFO] - Result for metric model_utility:	0.48362329969675943
[2026-05-13 20:57:13,655][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 20:57:20,116][metrics][INFO] - Evaluating privleak
[2026-05-13 20:57:20,116][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 20:57:20,117][evaluator][INFO] - Result for metric privleak:	99.32437498013512
[2026-05-13 20:57:21,959][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 20:57:23,793][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 20:57:38,726][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 20:57:38,727][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-42/evals/TOFU_EVAL.json
[2026-05-13 20:57:38,727][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-42/evals/TOFU_SUMMARY.json
[2026-05-13 20:57:40,372][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 20:57:46,055][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 20:58:04,548][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 20:58:04,549][evaluator][INFO] - Result for metric forget_truth_ratio:	4.081972845723411e-05
[2026-05-13 20:58:04,555][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 20:58:04,555][metrics][INFO] - Evaluating forget_quality
[2026-05-13 20:58:04,555][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 20:58:04,555][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 20:58:06,296][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 20:58:11,810][evaluator][INFO] - Result for metric forget_Q_A_Prob:	1.142715101423615e-06
[2026-05-13 20:58:13,646][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 21:00:13,956][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.027156757898764088
[2026-05-13 21:00:15,770][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 21:00:21,851][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 21:00:49,554][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 21:00:55,935][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 21:01:19,807][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 21:01:21,708][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 21:01:24,285][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 21:01:27,023][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 21:01:28,270][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 21:01:43,216][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 21:01:43,216][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 21:01:43,216][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 21:01:44,940][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 21:01:47,320][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 21:01:49,345][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 21:01:50,575][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 21:02:11,517][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 21:02:11,517][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 21:02:11,517][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 21:02:11,518][metrics][INFO] - Evaluating model_utility
[2026-05-13 21:02:11,518][evaluator][INFO] - Result for metric model_utility:	0.4808609112702393
[2026-05-13 21:02:14,604][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 21:02:17,406][metrics][INFO] - Evaluating privleak
[2026-05-13 21:02:17,406][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 21:02:17,406][evaluator][INFO] - Result for metric privleak:	99.27874998014424
[2026-05-13 21:02:18,711][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 21:02:21,361][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 21:02:38,992][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 21:02:38,992][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-49/evals/TOFU_EVAL.json
[2026-05-13 21:02:38,992][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-49/evals/TOFU_SUMMARY.json
[2026-05-13 21:02:40,658][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 21:02:48,032][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 21:03:13,436][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 21:03:13,437][evaluator][INFO] - Result for metric forget_truth_ratio:	5.51433307131689e-05
[2026-05-13 21:03:13,443][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 21:03:13,443][metrics][INFO] - Evaluating forget_quality
[2026-05-13 21:03:13,443][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 21:03:13,443][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 21:03:15,157][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 21:03:20,872][evaluator][INFO] - Result for metric forget_Q_A_Prob:	9.273054357651032e-07
[2026-05-13 21:03:22,578][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 21:05:17,577][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.03125428696853153
[2026-05-13 21:05:19,597][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 21:05:25,358][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 21:06:01,088][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 21:06:05,600][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 21:06:27,074][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 21:06:29,085][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 21:06:31,786][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 21:06:35,226][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 21:06:36,567][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 21:06:42,486][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 21:06:42,486][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 21:06:42,486][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 21:06:44,500][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 21:06:46,623][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 21:06:48,499][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 21:06:49,842][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 21:07:27,795][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 21:07:27,795][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 21:07:27,795][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 21:07:27,796][metrics][INFO] - Evaluating model_utility
[2026-05-13 21:07:27,796][evaluator][INFO] - Result for metric model_utility:	0.4892190473756089
[2026-05-13 21:07:30,750][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 21:07:34,261][metrics][INFO] - Evaluating privleak
[2026-05-13 21:07:34,261][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 21:07:34,261][evaluator][INFO] - Result for metric privleak:	99.36124998012775
[2026-05-13 21:07:35,546][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 21:07:40,283][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 21:07:58,628][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 21:07:58,628][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-56/evals/TOFU_EVAL.json
[2026-05-13 21:07:58,628][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-56/evals/TOFU_SUMMARY.json
[2026-05-13 21:08:00,340][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 21:08:05,175][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 21:08:25,085][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 21:08:25,086][evaluator][INFO] - Result for metric forget_truth_ratio:	1.0176140978785472e-05
[2026-05-13 21:08:25,091][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 21:08:25,092][metrics][INFO] - Evaluating forget_quality
[2026-05-13 21:08:25,092][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 21:08:25,092][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 21:08:26,762][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 21:08:30,908][evaluator][INFO] - Result for metric forget_Q_A_Prob:	7.110820942058316e-07
[2026-05-13 21:08:32,161][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 21:10:27,171][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.031914009190753746
[2026-05-13 21:10:29,103][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 21:10:33,319][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 21:11:09,378][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 21:11:13,620][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 21:11:28,291][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 21:11:29,958][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 21:11:31,998][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 21:11:34,117][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 21:11:35,378][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 21:11:54,498][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 21:11:54,498][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 21:11:54,498][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 21:11:56,105][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 21:11:57,903][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 21:11:59,767][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 21:12:01,097][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 21:12:34,710][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 21:12:34,710][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 21:12:34,710][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 21:12:34,711][metrics][INFO] - Evaluating model_utility
[2026-05-13 21:12:34,711][evaluator][INFO] - Result for metric model_utility:	0.4994966541320685
[2026-05-13 21:12:37,710][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 21:12:39,322][metrics][INFO] - Evaluating privleak
[2026-05-13 21:12:39,322][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 21:12:39,323][evaluator][INFO] - Result for metric privleak:	99.15874998016825
[2026-05-13 21:12:40,591][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 21:12:41,720][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 21:12:49,065][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 21:12:49,065][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_EVAL.json
[2026-05-13 21:12:49,065][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_SUMMARY.json
[2026-05-13 21:12:50,743][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 21:12:55,000][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 21:13:09,712][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 21:13:09,713][evaluator][INFO] - Result for metric forget_truth_ratio:	9.252015280586869e-06
[2026-05-13 21:13:09,718][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 21:13:09,719][metrics][INFO] - Evaluating forget_quality
[2026-05-13 21:13:09,719][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 21:13:09,719][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 21:13:11,471][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 21:13:14,445][evaluator][INFO] - Result for metric forget_Q_A_Prob:	6.573859578862728e-07
[2026-05-13 21:13:15,822][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 21:15:04,662][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.03141659317008192
[2026-05-13 21:15:06,341][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 21:15:10,415][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 21:15:46,228][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 21:15:50,490][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 21:16:05,094][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 21:16:06,896][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 21:16:08,923][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 21:16:11,012][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 21:16:12,273][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 21:16:31,387][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 21:16:31,387][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 21:16:31,387][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 21:16:33,011][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 21:16:34,858][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 21:16:36,751][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 21:16:39,097][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 21:17:12,590][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 21:17:12,590][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 21:17:12,590][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 21:17:12,590][metrics][INFO] - Evaluating model_utility
[2026-05-13 21:17:12,591][evaluator][INFO] - Result for metric model_utility:	0.5018110010150688
[2026-05-13 21:17:15,479][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 21:17:17,107][metrics][INFO] - Evaluating privleak
[2026-05-13 21:17:17,107][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 21:17:17,107][evaluator][INFO] - Result for metric privleak:	99.11437498017712
[2026-05-13 21:17:18,575][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 21:17:19,685][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-13 21:17:19,743][model.qat][INFO] - QAT: baked 112 QATLinear modules back into nn.Linear
[2026-05-13 21:17:29,333][huggingface_hub.hf_api][WARNING] - No files have been modified since last commit. Skipping to prevent empty commit.
[2026-05-13 21:17:31,482][huggingface_hub.hf_api][WARNING] - No files have been modified since last commit. Skipping to prevent empty commit.
[2026-05-13 21:17:31,693][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-13 21:17:31,693][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_EVAL.json
[2026-05-13 21:17:31,693][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_SUMMARY.json
[2026-05-13 21:17:33,591][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-13 21:17:37,524][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-13 21:17:50,767][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-13 21:17:50,768][evaluator][INFO] - Result for metric forget_truth_ratio:	9.252015280586869e-06
[2026-05-13 21:17:50,773][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-13 21:17:50,774][metrics][INFO] - Evaluating forget_quality
[2026-05-13 21:17:50,774][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-13 21:17:50,774][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-13 21:17:52,451][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-13 21:17:55,104][evaluator][INFO] - Result for metric forget_Q_A_Prob:	6.573859578862728e-07
[2026-05-13 21:17:56,351][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-13 21:18:31,739][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.03141659317008192
[2026-05-13 21:18:33,448][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-13 21:18:37,249][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-13 21:18:50,032][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-13 21:18:53,959][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-13 21:19:06,967][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-13 21:19:08,741][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-13 21:19:10,595][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-13 21:19:12,343][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-13 21:19:13,611][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-13 21:19:19,744][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-13 21:19:19,744][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-13 21:19:19,745][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-13 21:19:21,414][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-13 21:19:23,119][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-13 21:19:24,650][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-13 21:19:25,896][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-13 21:19:36,744][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-13 21:19:36,744][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-13 21:19:36,744][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-13 21:19:36,745][metrics][INFO] - Evaluating model_utility
[2026-05-13 21:19:36,745][evaluator][INFO] - Result for metric model_utility:	0.5018110010150688
[2026-05-13 21:19:39,706][metrics][INFO] - Evaluating mia_min_k
[2026-05-13 21:19:40,679][metrics][INFO] - Evaluating privleak
[2026-05-13 21:19:40,679][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-13 21:19:40,679][evaluator][INFO] - Result for metric privleak:	99.11437498017712
[2026-05-13 21:19:41,965][metrics][INFO] - Evaluating extraction_strength
[2026-05-13 21:19:42,782][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522