| - QAT: wrapped 112 nn.Linear modules (n_bits=8, group_size=128, symmetric=False, train_quant_params=True, skip=) |
| - Setting pad_token as eos token: <|eot_id|> |
| - Evaluations stored in the experiment directory: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT |
| - GradDiff Trainer loaded, output_dir: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT |
| - QAT: wrapped 112 nn.Linear modules (n_bits=8, group_size=128, symmetric=False, train_quant_params=True, skip=) |
| - Setting pad_token as eos token: <|eot_id|> |
| - Evaluations stored in the experiment directory: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT |
| - GradDiff Trainer loaded, output_dir: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT |
| - Gradient accumulation steps mismatch: GradientAccumulationPlugin has 1, DeepSpeed config has 4. Using DeepSpeed's value. |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-0/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-0/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.47705465705382816 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 0.87974609375 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.8168176212351753 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.5985466594660509 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: -99.34374998013125 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.7109042565278512 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-7/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-7/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.5006871924207377 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 0.1836419677734375 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.41414601567222376 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.407200292926334 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: -89.00812498219837 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.05924189741665229 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-14/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-14/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.4788654053437078 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 0.08483051776885986 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.34416566722932734 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.4838133878904574 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: -33.45624999330876 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.09940587875409879 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-21/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-21/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.4670378053691596 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 0.00890925220965073 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.29235083818535434 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.48983935329236644 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 73.62562498527488 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.047374115424220306 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-28/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-28/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.002696933255154829 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 1.948253397720595e-05 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.01650086453754834 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.40486175048229456 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 98.494999980301 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-35/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-35/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 1.5960496595360732e-05 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 3.709925050576768e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.02630269027733792 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.48362329969675943 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.32437498013512 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-42/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-42/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 4.081972845723411e-05 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 1.142715101423615e-06 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.027156757898764088 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.4808609112702393 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.27874998014424 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-49/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-49/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 5.51433307131689e-05 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 9.273054357651032e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.03125428696853153 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.4892190473756089 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.36124998012775 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-56/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-56/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 1.0176140978785472e-05 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 7.110820942058316e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.031914009190753746 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.4994966541320685 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.15874998016825 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 9.252015280586869e-06 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 6.573859578862728e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.03141659317008192 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.5018110010150688 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.11437498017712 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - QAT: baked 112 QATLinear modules back into nn.Linear |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - QAT: wrapped 112 nn.Linear modules (n_bits=8, group_size=128, symmetric=False, train_quant_params=True, skip=) |
| - Setting pad_token as eos token: <|eot_id|> |
| - Evaluations stored in the experiment directory: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT |
| - GradDiff Trainer loaded, output_dir: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT |
| - Gradient accumulation steps mismatch: GradientAccumulationPlugin has 1, DeepSpeed config has 4. Using DeepSpeed's value. |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-0/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-0/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.47705465705382816 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 0.87974609375 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.8168176212351753 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.5985466594660509 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: -99.34374998013125 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.7109042565278512 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-7/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-7/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.5006871924207377 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 0.1836419677734375 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.41414601567222376 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.407200292926334 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: -89.00812498219837 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.05924189741665229 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-14/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-14/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.4788654053437078 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 0.08483051776885986 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.34416566722932734 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.4838133878904574 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: -33.45624999330876 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.09940587875409879 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-21/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-21/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.4670378053691596 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 0.00890925220965073 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.29235083818535434 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.48983935329236644 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 73.62562498527488 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.047374115424220306 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-28/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-28/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.002696933255154829 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 1.948253397720595e-05 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.01650086453754834 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.40486175048229456 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 98.494999980301 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-35/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-35/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 1.5960496595360732e-05 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 3.709925050576768e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.02630269027733792 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.48362329969675943 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.32437498013512 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-42/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-42/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 4.081972845723411e-05 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 1.142715101423615e-06 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.027156757898764088 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.4808609112702393 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.27874998014424 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-49/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-49/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 5.51433307131689e-05 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 9.273054357651032e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.03125428696853153 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.4892190473756089 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.36124998012775 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-56/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-56/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 1.0176140978785472e-05 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 7.110820942058316e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.031914009190753746 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.4994966541320685 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.15874998016825 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 9.252015280586869e-06 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 6.573859578862728e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.03141659317008192 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.5018110010150688 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.11437498017712 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - QAT: baked 112 QATLinear modules back into nn.Linear |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - QAT: wrapped 112 nn.Linear modules (n_bits=8, group_size=128, symmetric=False, train_quant_params=True, skip=) |
| - Setting pad_token as eos token: <|eot_id|> |
| - Evaluations stored in the experiment directory: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT |
| - GradDiff Trainer loaded, output_dir: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT |
| - Gradient accumulation steps mismatch: GradientAccumulationPlugin has 1, DeepSpeed config has 4. Using DeepSpeed's value. |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-0/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-0/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.47705465705382816 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 0.87974609375 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.8168176212351753 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.5985466594660509 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: -99.34374998013125 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.7109042565278512 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-7/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-7/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.5006871924207377 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 0.1836419677734375 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.41414601567222376 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.407200292926334 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: -89.00812498219837 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.05924189741665229 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-14/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-14/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.4788654053437078 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 0.08483051776885986 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.34416566722932734 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.4838133878904574 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: -33.45624999330876 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.09940587875409879 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-21/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-21/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.4670378053691596 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 0.00890925220965073 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.29235083818535434 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.48983935329236644 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 73.62562498527488 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.047374115424220306 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-28/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-28/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 0.002696933255154829 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 1.948253397720595e-05 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.01650086453754834 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.40486175048229456 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 98.494999980301 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-35/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-35/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 1.5960496595360732e-05 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 3.709925050576768e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.02630269027733792 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.48362329969675943 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.32437498013512 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-42/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-42/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 4.081972845723411e-05 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 1.142715101423615e-06 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.027156757898764088 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.4808609112702393 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.27874998014424 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-49/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-49/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 5.51433307131689e-05 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 9.273054357651032e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.03125428696853153 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.4892190473756089 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.36124998012775 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-56/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-56/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 1.0176140978785472e-05 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 7.110820942058316e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.031914009190753746 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.4994966541320685 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.15874998016825 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 9.252015280586869e-06 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 6.573859578862728e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.03141659317008192 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.5018110010150688 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.11437498017712 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
| - QAT: baked 112 QATLinear modules back into nn.Linear |
| - No files have been modified since last commit. Skipping to prevent empty commit. |
| - No files have been modified since last commit. Skipping to prevent empty commit. |
| - ***** Running TOFU evaluation suite ***** |
| - Fine-grained evaluations will be saved to: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_EVAL.json |
| - Aggregated evaluations will be summarised in: ./saves/unlearn/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_QAT/checkpoint-60/evals/TOFU_SUMMARY.json |
| - Evaluating forget_Q_A_PARA_Prob |
| - Evaluating forget_Q_A_PERT_Prob |
| - Evaluating forget_truth_ratio |
| - Result for metric forget_truth_ratio: 9.252015280586869e-06 |
| - Skipping forget_quality's precompute forget_truth_ratio, already evaluated. |
| - Evaluating forget_quality |
| - retain_model_logs not provided in reference_logs, setting forget_quality to None |
| - Result for metric forget_quality: None |
| - Evaluating forget_Q_A_Prob |
| - Result for metric forget_Q_A_Prob: 6.573859578862728e-07 |
| - Evaluating forget_Q_A_ROUGE |
| - Result for metric forget_Q_A_ROUGE: 0.03141659317008192 |
| - Evaluating retain_Q_A_Prob |
| - Evaluating retain_Q_A_ROUGE |
| - Evaluating retain_Q_A_PARA_Prob |
| - Evaluating retain_Q_A_PERT_Prob |
| - Evaluating retain_Truth_Ratio |
| - Evaluating ra_Q_A_Prob |
| - Evaluating ra_Q_A_PERT_Prob |
| - Evaluating ra_Q_A_Prob_normalised |
| - Evaluating ra_Q_A_ROUGE |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated. |
| - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated. |
| - Evaluating ra_Truth_Ratio |
| - Evaluating wf_Q_A_Prob |
| - Evaluating wf_Q_A_PERT_Prob |
| - Evaluating wf_Q_A_Prob_normalised |
| - Evaluating wf_Q_A_ROUGE |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated. |
| - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated. |
| - Evaluating wf_Truth_Ratio |
| - Evaluating model_utility |
| - Result for metric model_utility: 0.5018110010150688 |
| - Evaluating mia_min_k |
| - Evaluating privleak |
| - retain_model_logs evals not provided for privleak, using default retain auc of 0.5 |
| - Result for metric privleak: 99.11437498017712 |
| - Evaluating extraction_strength |
| - Result for metric extraction_strength: 0.03250892997513522 |
|
|
