[2026-05-16 08:49:13,821][evaluator][INFO] - Evaluations stored in the experiment directory: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10
[2026-05-16 08:49:15,395][trainer][INFO] - GradDiff Trainer loaded, output_dir: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10
[2026-05-16 08:49:15,882][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 08:49:15,882][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-0/evals/TOFU_EVAL.json
[2026-05-16 08:49:15,882][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-0/evals/TOFU_SUMMARY.json
[2026-05-16 08:49:17,542][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 08:49:34,542][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 08:50:10,117][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 08:50:10,118][evaluator][INFO] - Result for metric forget_truth_ratio:	0.47428738953801663
[2026-05-16 08:50:10,124][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 08:50:10,125][metrics][INFO] - Evaluating forget_quality
[2026-05-16 08:50:10,125][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 08:50:10,125][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 08:50:11,770][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 08:50:16,852][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.9914018808305264
[2026-05-16 08:50:18,521][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 08:51:16,501][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.9908990311431487
[2026-05-16 08:51:18,171][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 08:51:25,245][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 08:51:41,090][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 08:51:48,144][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 08:52:10,692][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 08:52:12,304][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 08:52:14,955][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 08:52:17,644][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 08:52:18,892][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 08:52:29,001][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 08:52:29,001][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 08:52:29,001][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 08:52:30,766][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 08:52:32,892][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 08:52:36,307][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 08:52:37,545][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 08:52:54,178][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 08:52:54,178][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 08:52:54,178][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 08:52:54,178][metrics][INFO] - Evaluating model_utility
[2026-05-16 08:52:54,179][evaluator][INFO] - Result for metric model_utility:	0.627659700025085
[2026-05-16 08:52:57,085][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 08:53:01,452][metrics][INFO] - Evaluating privleak
[2026-05-16 08:53:01,452][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 08:53:01,452][evaluator][INFO] - Result for metric privleak:	-99.92374998001526
[2026-05-16 08:53:02,728][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 08:53:03,989][evaluator][INFO] - Result for metric extraction_strength:	0.9789440160652491
[2026-05-16 08:54:54,070][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 08:54:54,070][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-7/evals/TOFU_EVAL.json
[2026-05-16 08:54:54,070][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-7/evals/TOFU_SUMMARY.json
[2026-05-16 08:54:56,063][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 08:55:00,920][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 08:55:18,719][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 08:55:18,720][evaluator][INFO] - Result for metric forget_truth_ratio:	0.5170824331418401
[2026-05-16 08:55:18,726][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 08:55:18,727][metrics][INFO] - Evaluating forget_quality
[2026-05-16 08:55:18,727][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 08:55:18,727][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 08:55:20,520][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 08:55:24,059][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.32237392665709197
[2026-05-16 08:55:25,305][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 08:56:58,055][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.377814897046466
[2026-05-16 08:56:59,748][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 08:57:04,762][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 08:57:41,022][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 08:57:45,915][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 08:58:03,049][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 08:58:04,723][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 08:58:06,912][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 08:58:09,072][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 08:58:10,318][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 08:59:16,189][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 08:59:16,190][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 08:59:16,190][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 08:59:17,962][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 08:59:19,975][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 08:59:21,887][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 08:59:23,297][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:00:08,368][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:00:08,368][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:00:08,368][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:00:08,369][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:00:08,369][evaluator][INFO] - Result for metric model_utility:	0.511630521588525
[2026-05-16 09:00:11,612][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:00:14,062][metrics][INFO] - Evaluating privleak
[2026-05-16 09:00:14,062][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:00:14,063][evaluator][INFO] - Result for metric privleak:	-77.66624998446677
[2026-05-16 09:00:16,256][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:00:17,741][evaluator][INFO] - Result for metric extraction_strength:	0.1922307880599734
[2026-05-16 09:00:55,742][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:00:55,743][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-14/evals/TOFU_EVAL.json
[2026-05-16 09:00:55,743][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-14/evals/TOFU_SUMMARY.json
[2026-05-16 09:00:57,397][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:01:02,339][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:01:20,091][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:01:20,091][evaluator][INFO] - Result for metric forget_truth_ratio:	0.0002358502369670376
[2026-05-16 09:01:20,098][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:01:20,099][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:01:20,099][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:01:20,099][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:01:22,112][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:01:25,637][evaluator][INFO] - Result for metric forget_Q_A_Prob:	2.473320499738617e-05
[2026-05-16 09:01:27,195][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:02:04,349][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.00587613070485939
[2026-05-16 09:02:07,552][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:02:12,352][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:02:50,786][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:02:55,500][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:03:12,598][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:03:14,269][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:03:16,266][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:03:18,430][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:03:19,829][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:03:30,465][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:03:30,466][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:03:30,466][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:03:32,185][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:03:34,036][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:03:35,948][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:03:37,184][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:03:48,270][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:03:48,271][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:03:48,271][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:03:48,271][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:03:48,272][evaluator][INFO] - Result for metric model_utility:	0.042665927099493704
[2026-05-16 09:03:51,165][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:03:53,602][metrics][INFO] - Evaluating privleak
[2026-05-16 09:03:53,602][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:03:53,602][evaluator][INFO] - Result for metric privleak:	80.69124998386177
[2026-05-16 09:03:54,840][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:03:56,366][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:04:25,537][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:04:25,537][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-21/evals/TOFU_EVAL.json
[2026-05-16 09:04:25,537][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-21/evals/TOFU_SUMMARY.json
[2026-05-16 09:04:27,242][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:04:32,089][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:04:49,826][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:04:49,827][evaluator][INFO] - Result for metric forget_truth_ratio:	1.4008392141079827e-05
[2026-05-16 09:04:49,832][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:04:49,833][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:04:49,833][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:04:49,833][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:04:51,522][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:04:55,045][evaluator][INFO] - Result for metric forget_Q_A_Prob:	2.012360473731981e-08
[2026-05-16 09:04:56,495][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:05:33,601][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.01103525077561752
[2026-05-16 09:05:35,270][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:05:40,515][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:05:57,575][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:06:02,393][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:06:19,490][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:06:21,105][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:06:23,753][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:06:25,909][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:06:27,144][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:06:37,657][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:06:37,658][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:06:37,658][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:06:39,772][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:06:41,845][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:06:43,763][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:06:45,003][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:06:56,006][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:06:56,006][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:06:56,006][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:06:56,006][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:06:56,007][evaluator][INFO] - Result for metric model_utility:	0.4805926435144167
[2026-05-16 09:07:00,350][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:07:02,793][metrics][INFO] - Evaluating privleak
[2026-05-16 09:07:02,793][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:07:02,793][evaluator][INFO] - Result for metric privleak:	93.8274999812345
[2026-05-16 09:07:04,528][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:07:06,050][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:07:33,902][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:07:33,902][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-28/evals/TOFU_EVAL.json
[2026-05-16 09:07:33,902][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-28/evals/TOFU_SUMMARY.json
[2026-05-16 09:07:36,538][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:07:41,658][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:07:59,392][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:07:59,392][evaluator][INFO] - Result for metric forget_truth_ratio:	0.0003886007688211728
[2026-05-16 09:07:59,398][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:07:59,399][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:07:59,399][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:07:59,399][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:08:01,054][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:08:04,574][evaluator][INFO] - Result for metric forget_Q_A_Prob:	1.1190059233390226e-10
[2026-05-16 09:08:05,850][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:08:42,965][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.004834296233083551
[2026-05-16 09:08:44,639][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:08:49,242][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:09:22,843][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:09:27,557][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:09:44,666][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:09:46,539][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:09:48,547][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:09:50,712][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:09:52,207][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:10:02,760][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:10:02,761][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:10:02,761][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:10:04,988][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:10:06,822][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:10:08,734][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:10:10,121][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:10:21,186][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:10:21,186][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:10:21,186][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:10:21,186][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:10:21,187][evaluator][INFO] - Result for metric model_utility:	0.3981741551265505
[2026-05-16 09:10:24,155][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:10:26,599][metrics][INFO] - Evaluating privleak
[2026-05-16 09:10:26,600][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:10:26,600][evaluator][INFO] - Result for metric privleak:	97.239999980552
[2026-05-16 09:10:28,232][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:10:29,753][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:10:57,310][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:10:57,310][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-35/evals/TOFU_EVAL.json
[2026-05-16 09:10:57,310][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-35/evals/TOFU_SUMMARY.json
[2026-05-16 09:10:58,974][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:11:03,974][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:11:21,710][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:11:21,711][evaluator][INFO] - Result for metric forget_truth_ratio:	1.0733886071473592e-08
[2026-05-16 09:11:21,717][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:11:21,718][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:11:21,718][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:11:21,718][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:11:24,058][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:11:27,574][evaluator][INFO] - Result for metric forget_Q_A_Prob:	5.910864665759595e-10
[2026-05-16 09:11:28,823][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:12:05,954][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.006585730293039643
[2026-05-16 09:12:07,978][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:12:12,762][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:12:37,273][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:12:42,117][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:12:59,209][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:13:00,880][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:13:03,078][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:13:05,234][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:13:06,475][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:13:16,947][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:13:16,947][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:13:16,947][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:13:18,722][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:13:20,654][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:13:22,571][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:13:24,102][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:13:35,079][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:13:35,080][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:13:35,080][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:13:35,080][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:13:35,081][evaluator][INFO] - Result for metric model_utility:	0.47354087991618066
[2026-05-16 09:13:38,014][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:13:40,444][metrics][INFO] - Evaluating privleak
[2026-05-16 09:13:40,444][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:13:40,444][evaluator][INFO] - Result for metric privleak:	99.53749998009249
[2026-05-16 09:13:41,712][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:13:43,229][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:14:07,882][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:14:07,882][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-42/evals/TOFU_EVAL.json
[2026-05-16 09:14:07,882][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-42/evals/TOFU_SUMMARY.json
[2026-05-16 09:14:09,535][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:14:14,515][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:14:32,229][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:14:32,230][evaluator][INFO] - Result for metric forget_truth_ratio:	7.651479199384577e-05
[2026-05-16 09:14:32,235][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:14:32,236][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:14:32,236][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:14:32,236][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:14:34,344][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:14:37,871][evaluator][INFO] - Result for metric forget_Q_A_Prob:	5.251549263657368e-09
[2026-05-16 09:14:39,307][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:15:16,492][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.00980175907174189
[2026-05-16 09:15:18,130][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:15:23,252][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:15:38,541][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:15:43,774][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:16:00,870][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:16:02,879][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:16:04,886][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:16:07,047][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:16:08,287][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:16:18,760][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:16:18,760][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:16:18,760][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:16:20,431][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:16:22,391][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:16:24,307][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:16:25,572][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:16:36,556][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:16:36,556][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:16:36,556][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:16:36,556][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:16:36,557][evaluator][INFO] - Result for metric model_utility:	0.5379545653612532
[2026-05-16 09:16:39,668][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:16:42,109][metrics][INFO] - Evaluating privleak
[2026-05-16 09:16:42,109][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:16:42,109][evaluator][INFO] - Result for metric privleak:	98.49124998030176
[2026-05-16 09:16:43,381][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:16:44,896][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:17:13,810][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:17:13,810][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-49/evals/TOFU_EVAL.json
[2026-05-16 09:17:13,810][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-49/evals/TOFU_SUMMARY.json
[2026-05-16 09:17:15,603][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:17:20,443][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:17:38,161][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:17:38,162][evaluator][INFO] - Result for metric forget_truth_ratio:	1.8510762166179053e-08
[2026-05-16 09:17:38,168][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:17:38,168][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:17:38,168][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:17:38,168][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:17:40,279][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:17:43,805][evaluator][INFO] - Result for metric forget_Q_A_Prob:	1.5114575750711751e-10
[2026-05-16 09:17:45,072][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:18:22,298][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.004150876564726612
[2026-05-16 09:18:24,091][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:18:28,921][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:18:53,022][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:18:57,760][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:19:14,864][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:19:16,576][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:19:18,653][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:19:20,808][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:19:22,038][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:19:32,521][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:19:32,522][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:19:32,522][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:19:34,162][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:19:36,078][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:19:37,995][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:19:39,254][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:19:50,304][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:19:50,305][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:19:50,305][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:19:50,305][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:19:50,306][evaluator][INFO] - Result for metric model_utility:	0.5059279004776867
[2026-05-16 09:19:53,362][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:19:55,800][metrics][INFO] - Evaluating privleak
[2026-05-16 09:19:55,800][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:19:55,801][evaluator][INFO] - Result for metric privleak:	98.444999980311
[2026-05-16 09:19:57,078][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:19:58,590][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:20:22,982][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:20:22,982][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-56/evals/TOFU_EVAL.json
[2026-05-16 09:20:22,982][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-56/evals/TOFU_SUMMARY.json
[2026-05-16 09:20:24,679][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:20:29,722][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:20:47,459][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:20:47,460][evaluator][INFO] - Result for metric forget_truth_ratio:	5.9647935388930286e-05
[2026-05-16 09:20:47,466][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:20:47,467][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:20:47,467][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:20:47,467][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:20:49,482][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:20:53,013][evaluator][INFO] - Result for metric forget_Q_A_Prob:	9.453696584674903e-11
[2026-05-16 09:20:54,672][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:21:31,862][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.003807050915331808
[2026-05-16 09:21:33,947][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:21:38,680][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:22:12,621][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:22:17,333][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:22:34,438][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:22:36,051][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:22:38,114][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:22:40,277][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:22:41,524][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:22:52,044][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:22:52,045][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:22:52,045][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:22:53,736][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:22:55,922][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:22:57,837][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:22:59,156][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:23:10,212][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:23:10,212][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:23:10,212][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:23:10,212][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:23:10,213][evaluator][INFO] - Result for metric model_utility:	0.468231519047291
[2026-05-16 09:23:13,156][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:23:15,583][metrics][INFO] - Evaluating privleak
[2026-05-16 09:23:15,584][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:23:15,584][evaluator][INFO] - Result for metric privleak:	98.70437498025912
[2026-05-16 09:23:17,088][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:23:18,603][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:23:45,213][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:23:45,213][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-63/evals/TOFU_EVAL.json
[2026-05-16 09:23:45,213][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-63/evals/TOFU_SUMMARY.json
[2026-05-16 09:23:46,824][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:23:51,736][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:24:09,472][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:24:09,473][evaluator][INFO] - Result for metric forget_truth_ratio:	0.0004952549536712157
[2026-05-16 09:24:09,479][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:24:09,479][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:24:09,479][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:24:09,479][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:24:11,177][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:24:14,708][evaluator][INFO] - Result for metric forget_Q_A_Prob:	4.1167143494971477e-17
[2026-05-16 09:24:16,019][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:24:53,184][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.002763191266209001
[2026-05-16 09:24:55,123][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:24:59,972][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:25:21,775][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:25:26,485][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:25:43,602][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:25:45,323][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:25:47,310][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:25:49,473][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:25:50,739][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:25:59,538][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:25:59,538][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:25:59,538][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:26:01,191][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:26:03,177][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:26:05,088][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:26:06,486][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:26:17,509][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:26:17,509][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:26:17,509][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:26:17,509][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:26:17,510][evaluator][INFO] - Result for metric model_utility:	0.5341074368641497
[2026-05-16 09:26:20,528][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:26:22,964][metrics][INFO] - Evaluating privleak
[2026-05-16 09:26:22,964][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:26:22,964][evaluator][INFO] - Result for metric privleak:	98.37562498032486
[2026-05-16 09:26:24,334][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:26:25,861][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:26:49,896][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:26:49,896][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-70/evals/TOFU_EVAL.json
[2026-05-16 09:26:49,896][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-70/evals/TOFU_SUMMARY.json
[2026-05-16 09:26:51,787][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:26:56,857][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:27:14,622][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:27:14,622][evaluator][INFO] - Result for metric forget_truth_ratio:	7.54664843496783e-05
[2026-05-16 09:27:14,628][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:27:14,629][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:27:14,629][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:27:14,629][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:27:16,380][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:27:19,904][evaluator][INFO] - Result for metric forget_Q_A_Prob:	2.600986081085082e-34
[2026-05-16 09:27:21,168][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:27:58,326][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.0022631912662090005
[2026-05-16 09:27:59,991][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:28:04,654][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:28:28,919][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:28:33,628][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:28:50,721][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:28:52,409][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:28:54,397][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:28:56,564][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:28:58,002][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:29:06,637][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:29:06,637][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:29:06,637][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:29:08,295][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:29:10,143][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:29:12,062][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:29:13,338][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:29:24,362][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:29:24,362][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:29:24,362][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:29:24,362][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:29:24,363][evaluator][INFO] - Result for metric model_utility:	0.5224804904210487
[2026-05-16 09:29:27,531][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:29:29,976][metrics][INFO] - Evaluating privleak
[2026-05-16 09:29:29,976][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:29:29,976][evaluator][INFO] - Result for metric privleak:	97.2824999805435
[2026-05-16 09:29:31,279][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:29:32,802][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:29:57,190][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:29:57,190][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-77/evals/TOFU_EVAL.json
[2026-05-16 09:29:57,190][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-77/evals/TOFU_SUMMARY.json
[2026-05-16 09:29:58,830][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:30:03,751][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:30:21,493][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:30:21,494][evaluator][INFO] - Result for metric forget_truth_ratio:	4.216673056287236e-05
[2026-05-16 09:30:21,500][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:30:21,500][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:30:21,500][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:30:21,501][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:30:23,248][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:30:26,771][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.0
[2026-05-16 09:30:28,023][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:31:05,172][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.002596524599542334
[2026-05-16 09:31:06,810][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:31:11,430][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:31:33,576][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:31:38,314][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:31:55,421][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:31:57,064][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:31:59,029][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:32:01,190][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:32:02,459][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:32:11,097][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:32:11,097][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:32:11,097][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:32:12,937][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:32:14,777][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:32:16,691][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:32:17,960][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:32:28,969][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:32:28,969][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:32:28,969][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:32:28,969][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:32:28,970][evaluator][INFO] - Result for metric model_utility:	0.5559064963530199
[2026-05-16 09:32:32,000][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:32:34,442][metrics][INFO] - Evaluating privleak
[2026-05-16 09:32:34,443][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:32:34,443][evaluator][INFO] - Result for metric privleak:	97.609999980478
[2026-05-16 09:32:35,832][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:32:37,350][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:33:01,373][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:33:01,374][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-84/evals/TOFU_EVAL.json
[2026-05-16 09:33:01,374][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-84/evals/TOFU_SUMMARY.json
[2026-05-16 09:33:02,999][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:33:07,939][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:33:25,685][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:33:25,685][evaluator][INFO] - Result for metric forget_truth_ratio:	0.0008704624723907797
[2026-05-16 09:33:25,691][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:33:25,692][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:33:25,692][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:33:25,692][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:33:27,481][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:33:31,005][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.0
[2026-05-16 09:33:32,267][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:34:09,472][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.0029298579328756675
[2026-05-16 09:34:11,108][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:34:15,958][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:34:34,225][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:34:39,193][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:34:56,305][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:34:57,955][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:35:01,401][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:35:03,558][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:35:04,813][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:35:11,235][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:35:11,236][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:35:11,236][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:35:12,920][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:35:14,771][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:35:16,687][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:35:17,957][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:35:28,942][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:35:28,943][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:35:28,943][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:35:28,943][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:35:28,944][evaluator][INFO] - Result for metric model_utility:	0.5762713042381238
[2026-05-16 09:35:31,997][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:35:34,727][metrics][INFO] - Evaluating privleak
[2026-05-16 09:35:34,728][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:35:34,728][evaluator][INFO] - Result for metric privleak:	97.75499998044899
[2026-05-16 09:35:35,975][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:35:37,501][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:36:01,712][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:36:01,712][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-91/evals/TOFU_EVAL.json
[2026-05-16 09:36:01,712][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-91/evals/TOFU_SUMMARY.json
[2026-05-16 09:36:03,398][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:36:08,246][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:36:25,994][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:36:25,995][evaluator][INFO] - Result for metric forget_truth_ratio:	0.004187546235345516
[2026-05-16 09:36:26,001][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:36:26,001][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:36:26,001][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:36:26,002][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:36:27,761][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:36:31,287][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.0
[2026-05-16 09:36:32,676][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:37:09,858][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.0029298579328756675
[2026-05-16 09:37:11,516][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:37:16,152][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:37:29,901][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:37:34,601][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:37:51,699][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:37:53,337][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:37:55,297][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:37:57,456][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:37:58,697][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:38:02,866][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:38:02,866][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:38:02,866][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:38:04,308][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:38:06,112][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:38:08,027][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:38:09,359][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:38:20,332][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:38:20,332][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:38:20,332][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:38:20,333][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:38:20,333][evaluator][INFO] - Result for metric model_utility:	0.5830553180509839
[2026-05-16 09:38:23,275][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:38:25,718][metrics][INFO] - Evaluating privleak
[2026-05-16 09:38:25,718][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:38:25,718][evaluator][INFO] - Result for metric privleak:	97.55624998048876
[2026-05-16 09:38:27,071][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:38:28,589][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:38:52,852][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:38:52,852][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-98/evals/TOFU_EVAL.json
[2026-05-16 09:38:52,852][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-98/evals/TOFU_SUMMARY.json
[2026-05-16 09:38:54,499][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:38:59,453][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:39:17,212][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:39:17,212][evaluator][INFO] - Result for metric forget_truth_ratio:	0.0023928304845221094
[2026-05-16 09:39:17,218][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:39:17,218][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:39:17,219][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:39:17,219][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:39:18,863][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:39:22,392][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.0
[2026-05-16 09:39:23,652][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:40:00,755][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.0029298579328756675
[2026-05-16 09:40:02,422][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:40:07,062][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:40:18,946][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:40:23,768][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:40:40,886][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:40:42,573][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:40:44,555][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:40:46,714][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:40:48,550][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:40:52,737][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:40:52,737][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:40:52,737][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:40:54,631][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:40:56,476][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:40:58,393][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:40:59,648][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:41:10,613][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:41:10,614][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:41:10,614][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:41:10,614][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:41:10,615][evaluator][INFO] - Result for metric model_utility:	0.5804643010659025
[2026-05-16 09:41:13,585][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:41:16,026][metrics][INFO] - Evaluating privleak
[2026-05-16 09:41:16,027][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:41:16,027][evaluator][INFO] - Result for metric privleak:	97.43937498051211
[2026-05-16 09:41:17,470][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:41:18,989][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:41:43,698][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:41:43,698][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-105/evals/TOFU_EVAL.json
[2026-05-16 09:41:43,698][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-105/evals/TOFU_SUMMARY.json
[2026-05-16 09:41:45,319][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:41:50,135][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:42:07,869][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:42:07,870][evaluator][INFO] - Result for metric forget_truth_ratio:	0.0018223785794616683
[2026-05-16 09:42:07,875][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:42:07,876][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:42:07,876][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:42:07,876][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:42:11,031][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:42:14,566][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.0
[2026-05-16 09:42:16,260][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:42:53,466][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.0029298579328756675
[2026-05-16 09:42:55,273][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:42:59,927][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:43:12,328][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:43:17,086][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:43:34,202][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:43:35,859][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:43:37,866][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:43:40,030][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:43:41,295][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:43:45,497][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:43:45,497][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:43:45,497][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:43:46,777][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:43:48,625][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:43:50,539][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:43:51,820][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:44:02,790][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:44:02,791][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:44:02,791][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:44:02,791][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:44:02,792][evaluator][INFO] - Result for metric model_utility:	0.5820021030858749
[2026-05-16 09:44:05,749][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:44:08,198][metrics][INFO] - Evaluating privleak
[2026-05-16 09:44:08,198][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:44:08,198][evaluator][INFO] - Result for metric privleak:	97.44312498051139
[2026-05-16 09:44:09,680][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:44:11,212][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:44:35,630][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:44:35,630][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-112/evals/TOFU_EVAL.json
[2026-05-16 09:44:35,630][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-112/evals/TOFU_SUMMARY.json
[2026-05-16 09:44:37,262][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:44:42,180][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:44:59,925][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:44:59,926][evaluator][INFO] - Result for metric forget_truth_ratio:	0.002150945459819556
[2026-05-16 09:44:59,932][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:44:59,932][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:44:59,933][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:44:59,933][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:45:01,582][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:45:05,118][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.0
[2026-05-16 09:45:06,384][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:45:43,585][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.0029298579328756675
[2026-05-16 09:45:45,219][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:45:49,849][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:46:01,341][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:46:06,055][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:46:23,157][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:46:24,795][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:46:26,744][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:46:28,903][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:46:30,165][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:46:34,383][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:46:34,383][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:46:34,383][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:46:35,608][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:46:37,484][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:46:39,401][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:46:40,628][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:46:51,597][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:46:51,597][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:46:51,597][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:46:51,597][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:46:51,598][evaluator][INFO] - Result for metric model_utility:	0.5841038983728979
[2026-05-16 09:46:54,537][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:46:56,978][metrics][INFO] - Evaluating privleak
[2026-05-16 09:46:56,979][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:46:56,979][evaluator][INFO] - Result for metric privleak:	97.37624998052475
[2026-05-16 09:46:58,266][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:46:59,784][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:47:24,079][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:47:24,079][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-119/evals/TOFU_EVAL.json
[2026-05-16 09:47:24,079][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-119/evals/TOFU_SUMMARY.json
[2026-05-16 09:47:25,726][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:47:30,666][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:47:48,408][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:47:48,409][evaluator][INFO] - Result for metric forget_truth_ratio:	0.0018856268147293422
[2026-05-16 09:47:48,415][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:47:48,415][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:47:48,415][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:47:48,415][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:47:50,138][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:47:53,674][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.0
[2026-05-16 09:47:54,988][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:48:32,179][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.0029298579328756675
[2026-05-16 09:48:33,893][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:48:38,511][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:48:50,171][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:48:54,877][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:49:11,979][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:49:13,606][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:49:15,582][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:49:17,737][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:49:18,999][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:49:23,179][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:49:23,179][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:49:23,179][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:49:24,431][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:49:26,338][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:49:28,252][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:49:29,620][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:49:40,578][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:49:40,579][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:49:40,579][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:49:40,579][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:49:40,580][evaluator][INFO] - Result for metric model_utility:	0.5830384010967027
[2026-05-16 09:49:43,506][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:49:45,940][metrics][INFO] - Evaluating privleak
[2026-05-16 09:49:45,940][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:49:45,940][evaluator][INFO] - Result for metric privleak:	97.40124998051975
[2026-05-16 09:49:47,293][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:49:48,812][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522
[2026-05-16 09:49:52,686][evaluator][INFO] - ***** Running TOFU evaluation suite *****
[2026-05-16 09:49:52,686][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-120/evals/TOFU_EVAL.json
[2026-05-16 09:49:52,686][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/C2_C1_GradDiff_NoQAT_tofu_Llama-3.1-8B-Instruct_forget10/checkpoint-120/evals/TOFU_SUMMARY.json
[2026-05-16 09:49:54,439][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
[2026-05-16 09:49:59,331][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
[2026-05-16 09:50:17,083][metrics][INFO] - Evaluating forget_truth_ratio
[2026-05-16 09:50:17,084][evaluator][INFO] - Result for metric forget_truth_ratio:	0.001991613276074895
[2026-05-16 09:50:17,089][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
[2026-05-16 09:50:17,090][metrics][INFO] - Evaluating forget_quality
[2026-05-16 09:50:17,090][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
[2026-05-16 09:50:17,090][evaluator][INFO] - Result for metric forget_quality:	None
[2026-05-16 09:50:18,778][metrics][INFO] - Evaluating forget_Q_A_Prob
[2026-05-16 09:50:22,310][evaluator][INFO] - Result for metric forget_Q_A_Prob:	0.0
[2026-05-16 09:50:23,552][metrics][INFO] - Evaluating forget_Q_A_ROUGE
[2026-05-16 09:51:00,810][evaluator][INFO] - Result for metric forget_Q_A_ROUGE:	0.0029298579328756675
[2026-05-16 09:51:02,477][metrics][INFO] - Evaluating retain_Q_A_Prob
[2026-05-16 09:51:07,082][metrics][INFO] - Evaluating retain_Q_A_ROUGE
[2026-05-16 09:51:19,039][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
[2026-05-16 09:51:23,773][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
[2026-05-16 09:51:40,866][metrics][INFO] - Evaluating retain_Truth_Ratio
[2026-05-16 09:51:42,567][metrics][INFO] - Evaluating ra_Q_A_Prob
[2026-05-16 09:51:44,819][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
[2026-05-16 09:51:46,978][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
[2026-05-16 09:51:48,187][metrics][INFO] - Evaluating ra_Q_A_ROUGE
[2026-05-16 09:51:52,465][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
[2026-05-16 09:51:52,465][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:51:52,465][metrics][INFO] - Evaluating ra_Truth_Ratio
[2026-05-16 09:51:53,713][metrics][INFO] - Evaluating wf_Q_A_Prob
[2026-05-16 09:51:55,543][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
[2026-05-16 09:51:57,455][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
[2026-05-16 09:51:58,680][metrics][INFO] - Evaluating wf_Q_A_ROUGE
[2026-05-16 09:52:09,628][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
[2026-05-16 09:52:09,628][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
[2026-05-16 09:52:09,628][metrics][INFO] - Evaluating wf_Truth_Ratio
[2026-05-16 09:52:09,629][metrics][INFO] - Evaluating model_utility
[2026-05-16 09:52:09,629][evaluator][INFO] - Result for metric model_utility:	0.5829634017386985
[2026-05-16 09:52:12,718][metrics][INFO] - Evaluating mia_min_k
[2026-05-16 09:52:15,165][metrics][INFO] - Evaluating privleak
[2026-05-16 09:52:15,166][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
[2026-05-16 09:52:15,166][evaluator][INFO] - Result for metric privleak:	97.4074999805185
[2026-05-16 09:52:16,437][metrics][INFO] - Evaluating extraction_strength
[2026-05-16 09:52:17,956][evaluator][INFO] - Result for metric extraction_strength:	0.03250892997513522