Instructions to use Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT")
model = AutoModelForCausalLM.from_pretrained("Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT

SGLang

How to use Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT with Docker Model Runner:
```
docker model run hf.co/Jeesup/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT
```

A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT / GradDiff.log

Jeesup

End of training

8ebdd7c verified 3 months ago

Raw

History Blame Contribute Delete

38.1 kB

	[2026-05-15 09:21:44,316][model][WARNING] - Model open-unlearning/tofu_Llama-3.2-1B-Instruct_full requested with {'attn_implementation': 'flash_attention_2'}
	[2026-05-15 09:25:37,977][model][INFO] - Setting pad_token as eos token: <\|eot_id\|>
	[2026-05-15 09:25:42,566][evaluator][INFO] - Evaluations stored in the experiment directory: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT
	[2026-05-15 09:25:43,093][trainer][INFO] - GradDiff Trainer loaded, output_dir: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT
	[2026-05-15 09:25:43,586][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-15 09:25:43,586][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-0/evals/TOFU_EVAL.json
	[2026-05-15 09:25:43,586][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-0/evals/TOFU_SUMMARY.json
	[2026-05-15 09:25:45,118][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-15 09:26:02,505][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-15 09:26:35,962][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-15 09:26:35,963][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4753744961107318
	[2026-05-15 09:26:35,969][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-15 09:26:35,970][metrics][INFO] - Evaluating forget_quality
	[2026-05-15 09:26:35,970][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-15 09:26:35,970][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-15 09:26:38,375][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-15 09:26:42,913][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.8805173517763615
	[2026-05-15 09:26:44,037][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-15 09:27:31,206][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.8174012352836957
	[2026-05-15 09:27:33,197][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-15 09:27:40,127][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-15 09:27:53,982][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-15 09:28:00,469][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-15 09:28:20,990][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-15 09:28:22,765][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-15 09:28:25,875][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-15 09:28:28,497][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-15 09:28:30,048][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-15 09:28:37,279][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-15 09:28:37,279][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:28:37,279][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-15 09:28:39,016][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-15 09:28:41,171][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-15 09:28:44,872][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-15 09:28:46,155][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-15 09:28:57,816][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-15 09:28:57,816][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:28:57,816][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-15 09:28:57,816][metrics][INFO] - Evaluating model_utility
	[2026-05-15 09:28:57,817][evaluator][INFO] - Result for metric model_utility: 0.5981291522236641
	[2026-05-15 09:29:00,810][metrics][INFO] - Evaluating mia_min_k
	[2026-05-15 09:29:03,743][metrics][INFO] - Evaluating privleak
	[2026-05-15 09:29:03,743][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-15 09:29:03,743][evaluator][INFO] - Result for metric privleak: -99.33124998013375
	[2026-05-15 09:29:05,057][metrics][INFO] - Evaluating extraction_strength
	[2026-05-15 09:29:05,767][evaluator][INFO] - Result for metric extraction_strength: 0.7051770244245114
	[2026-05-15 09:29:55,132][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-15 09:29:55,132][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-7/evals/TOFU_EVAL.json
	[2026-05-15 09:29:55,132][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-7/evals/TOFU_SUMMARY.json
	[2026-05-15 09:29:57,028][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-15 09:30:01,223][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-15 09:30:15,551][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-15 09:30:15,552][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4507843149038892
	[2026-05-15 09:30:15,558][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-15 09:30:15,558][metrics][INFO] - Evaluating forget_quality
	[2026-05-15 09:30:15,559][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-15 09:30:15,559][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-15 09:30:17,484][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-15 09:30:20,376][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.683253139210865
	[2026-05-15 09:30:21,709][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-15 09:30:30,491][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.5790595349419858
	[2026-05-15 09:30:32,165][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-15 09:30:36,378][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-15 09:31:02,551][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-15 09:31:06,719][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-15 09:31:20,857][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-15 09:31:22,785][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-15 09:31:24,718][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-15 09:31:26,636][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-15 09:31:27,925][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-15 09:31:29,835][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-15 09:31:29,835][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:31:29,835][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-15 09:31:31,210][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-15 09:31:33,105][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-15 09:31:34,734][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-15 09:31:36,025][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-15 09:31:37,581][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-15 09:31:37,581][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:31:37,581][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-15 09:31:37,581][metrics][INFO] - Evaluating model_utility
	[2026-05-15 09:31:37,582][evaluator][INFO] - Result for metric model_utility: 0.5623051210159115
	[2026-05-15 09:31:40,236][metrics][INFO] - Evaluating mia_min_k
	[2026-05-15 09:31:41,375][metrics][INFO] - Evaluating privleak
	[2026-05-15 09:31:41,375][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-15 09:31:41,375][evaluator][INFO] - Result for metric privleak: -98.18624998036276
	[2026-05-15 09:31:42,751][metrics][INFO] - Evaluating extraction_strength
	[2026-05-15 09:31:43,552][evaluator][INFO] - Result for metric extraction_strength: 0.3464082292160767
	[2026-05-15 09:32:07,578][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-15 09:32:07,578][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-14/evals/TOFU_EVAL.json
	[2026-05-15 09:32:07,578][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-14/evals/TOFU_SUMMARY.json
	[2026-05-15 09:32:09,322][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-15 09:32:13,518][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-15 09:32:27,833][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-15 09:32:27,833][evaluator][INFO] - Result for metric forget_truth_ratio: 0.48541763068044874
	[2026-05-15 09:32:27,839][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-15 09:32:27,839][metrics][INFO] - Evaluating forget_quality
	[2026-05-15 09:32:27,840][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-15 09:32:27,840][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-15 09:32:29,608][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-15 09:32:32,495][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.24801383328114754
	[2026-05-15 09:32:33,889][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-15 09:32:49,365][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.4082179757979397
	[2026-05-15 09:32:51,129][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-15 09:32:55,773][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-15 09:33:05,143][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-15 09:33:09,271][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-15 09:33:23,409][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-15 09:33:25,174][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-15 09:33:27,081][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-15 09:33:29,007][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-15 09:33:30,283][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-15 09:33:32,130][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-15 09:33:32,130][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:33:32,130][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-15 09:33:33,383][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-15 09:33:35,145][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-15 09:33:36,781][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-15 09:33:38,089][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-15 09:33:39,728][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-15 09:33:39,728][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:33:39,728][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-15 09:33:39,728][metrics][INFO] - Evaluating model_utility
	[2026-05-15 09:33:39,729][evaluator][INFO] - Result for metric model_utility: 0.545922184907243
	[2026-05-15 09:33:42,337][metrics][INFO] - Evaluating mia_min_k
	[2026-05-15 09:33:43,495][metrics][INFO] - Evaluating privleak
	[2026-05-15 09:33:43,495][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-15 09:33:43,495][evaluator][INFO] - Result for metric privleak: -65.30124998693972
	[2026-05-15 09:33:44,781][metrics][INFO] - Evaluating extraction_strength
	[2026-05-15 09:33:45,658][evaluator][INFO] - Result for metric extraction_strength: 0.1473824232680122
	[2026-05-15 09:33:52,915][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-15 09:33:52,916][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-21/evals/TOFU_EVAL.json
	[2026-05-15 09:33:52,916][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-21/evals/TOFU_SUMMARY.json
	[2026-05-15 09:33:54,580][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-15 09:33:58,788][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-15 09:34:13,107][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-15 09:34:13,108][evaluator][INFO] - Result for metric forget_truth_ratio: 0.47275164659117
	[2026-05-15 09:34:13,114][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-15 09:34:13,114][metrics][INFO] - Evaluating forget_quality
	[2026-05-15 09:34:13,114][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-15 09:34:13,114][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-15 09:34:14,800][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-15 09:34:17,688][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.07251899334157087
	[2026-05-15 09:34:19,005][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-15 09:34:33,296][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.35420833450882866
	[2026-05-15 09:34:35,076][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-15 09:34:39,191][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-15 09:34:48,973][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-15 09:34:53,141][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-15 09:35:07,287][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-15 09:35:09,351][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-15 09:35:12,201][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-15 09:35:14,119][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-15 09:35:15,852][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-15 09:35:19,886][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-15 09:35:19,886][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:35:19,886][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-15 09:35:21,216][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-15 09:35:23,045][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-15 09:35:24,678][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-15 09:35:25,974][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-15 09:35:30,167][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-15 09:35:30,167][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:35:30,167][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-15 09:35:30,167][metrics][INFO] - Evaluating model_utility
	[2026-05-15 09:35:30,168][evaluator][INFO] - Result for metric model_utility: 0.5412779619125754
	[2026-05-15 09:35:33,028][metrics][INFO] - Evaluating mia_min_k
	[2026-05-15 09:35:34,162][metrics][INFO] - Evaluating privleak
	[2026-05-15 09:35:34,163][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-15 09:35:34,163][evaluator][INFO] - Result for metric privleak: 3.947499999210492
	[2026-05-15 09:35:35,489][metrics][INFO] - Evaluating extraction_strength
	[2026-05-15 09:35:36,358][evaluator][INFO] - Result for metric extraction_strength: 0.09608966712557702
	[2026-05-15 09:35:45,488][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-15 09:35:45,488][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-28/evals/TOFU_EVAL.json
	[2026-05-15 09:35:45,488][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-28/evals/TOFU_SUMMARY.json
	[2026-05-15 09:35:47,292][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-15 09:35:52,054][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-15 09:36:06,385][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-15 09:36:06,386][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4563218367716289
	[2026-05-15 09:36:06,392][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-15 09:36:06,393][metrics][INFO] - Evaluating forget_quality
	[2026-05-15 09:36:06,393][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-15 09:36:06,393][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-15 09:36:08,058][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-15 09:36:10,960][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.02413768379198416
	[2026-05-15 09:36:12,801][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-15 09:36:26,547][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.3199003314255527
	[2026-05-15 09:36:28,899][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-15 09:36:33,764][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-15 09:36:43,587][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-15 09:36:47,825][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-15 09:37:01,991][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-15 09:37:04,214][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-15 09:37:06,223][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-15 09:37:08,150][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-15 09:37:09,666][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-15 09:37:10,993][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-15 09:37:10,993][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:37:10,993][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-15 09:37:12,671][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-15 09:37:14,635][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-15 09:37:16,273][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-15 09:37:17,778][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-15 09:37:19,475][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-15 09:37:19,475][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:37:19,475][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-15 09:37:19,475][metrics][INFO] - Evaluating model_utility
	[2026-05-15 09:37:19,476][evaluator][INFO] - Result for metric model_utility: 0.523505716765462
	[2026-05-15 09:37:22,135][metrics][INFO] - Evaluating mia_min_k
	[2026-05-15 09:37:23,274][metrics][INFO] - Evaluating privleak
	[2026-05-15 09:37:23,275][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-15 09:37:23,275][evaluator][INFO] - Result for metric privleak: 58.351249988329734
	[2026-05-15 09:37:24,627][metrics][INFO] - Evaluating extraction_strength
	[2026-05-15 09:37:25,505][evaluator][INFO] - Result for metric extraction_strength: 0.07072412558230562
	[2026-05-15 09:37:32,392][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-15 09:37:32,392][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-35/evals/TOFU_EVAL.json
	[2026-05-15 09:37:32,392][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-35/evals/TOFU_SUMMARY.json
	[2026-05-15 09:37:34,355][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-15 09:37:38,741][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-15 09:37:53,075][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-15 09:37:53,075][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4303001341029952
	[2026-05-15 09:37:53,082][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-15 09:37:53,082][metrics][INFO] - Evaluating forget_quality
	[2026-05-15 09:37:53,082][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-15 09:37:53,083][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-15 09:37:54,801][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-15 09:37:57,681][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.016043244256118763
	[2026-05-15 09:37:58,971][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-15 09:38:24,838][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2701899949169555
	[2026-05-15 09:38:26,526][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-15 09:38:30,636][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-15 09:38:39,908][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-15 09:38:44,094][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-15 09:38:58,204][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-15 09:38:59,917][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-15 09:39:01,870][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-15 09:39:03,791][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-15 09:39:05,075][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-15 09:39:06,618][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-15 09:39:06,618][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:39:06,618][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-15 09:39:07,947][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-15 09:39:09,712][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-15 09:39:11,341][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-15 09:39:12,615][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-15 09:39:14,277][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-15 09:39:14,277][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:39:14,277][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-15 09:39:14,278][metrics][INFO] - Evaluating model_utility
	[2026-05-15 09:39:14,278][evaluator][INFO] - Result for metric model_utility: 0.5490575696907394
	[2026-05-15 09:39:16,956][metrics][INFO] - Evaluating mia_min_k
	[2026-05-15 09:39:18,089][metrics][INFO] - Evaluating privleak
	[2026-05-15 09:39:18,090][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-15 09:39:18,090][evaluator][INFO] - Result for metric privleak: 87.889999982422
	[2026-05-15 09:39:19,396][metrics][INFO] - Evaluating extraction_strength
	[2026-05-15 09:39:20,273][evaluator][INFO] - Result for metric extraction_strength: 0.062284021012484383
	[2026-05-15 09:39:23,639][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-15 09:39:23,639][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-42/evals/TOFU_EVAL.json
	[2026-05-15 09:39:23,639][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-42/evals/TOFU_SUMMARY.json
	[2026-05-15 09:39:24,880][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-15 09:39:29,031][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-15 09:39:43,352][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-15 09:39:43,352][evaluator][INFO] - Result for metric forget_truth_ratio: 0.38683383452014125
	[2026-05-15 09:39:43,359][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-15 09:39:43,360][metrics][INFO] - Evaluating forget_quality
	[2026-05-15 09:39:43,360][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-15 09:39:43,360][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-15 09:39:45,028][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-15 09:39:47,910][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.005136876203216701
	[2026-05-15 09:39:49,172][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-15 09:40:33,715][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.14907592150468058
	[2026-05-15 09:40:35,383][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-15 09:40:39,443][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-15 09:40:53,485][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-15 09:40:57,622][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-15 09:41:11,771][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-15 09:41:13,457][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-15 09:41:15,399][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-15 09:41:17,317][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-15 09:41:18,572][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-15 09:41:21,314][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-15 09:41:21,314][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:41:21,314][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-15 09:41:22,618][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-15 09:41:24,383][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-15 09:41:26,017][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-15 09:41:27,353][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-15 09:41:29,269][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-15 09:41:29,270][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:41:29,270][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-15 09:41:29,270][metrics][INFO] - Evaluating model_utility
	[2026-05-15 09:41:29,272][evaluator][INFO] - Result for metric model_utility: 0.5352297692478013
	[2026-05-15 09:41:32,182][metrics][INFO] - Evaluating mia_min_k
	[2026-05-15 09:41:33,345][metrics][INFO] - Evaluating privleak
	[2026-05-15 09:41:33,346][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-15 09:41:33,346][evaluator][INFO] - Result for metric privleak: 95.82624998083476
	[2026-05-15 09:41:34,717][metrics][INFO] - Evaluating extraction_strength
	[2026-05-15 09:41:35,607][evaluator][INFO] - Result for metric extraction_strength: 0.0334788241303742
	[2026-05-15 09:41:43,815][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-15 09:41:43,815][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-49/evals/TOFU_EVAL.json
	[2026-05-15 09:41:43,815][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-49/evals/TOFU_SUMMARY.json
	[2026-05-15 09:41:45,523][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-15 09:41:49,808][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-15 09:42:04,138][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-15 09:42:04,140][evaluator][INFO] - Result for metric forget_truth_ratio: 0.019766035024290387
	[2026-05-15 09:42:04,146][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-15 09:42:04,147][metrics][INFO] - Evaluating forget_quality
	[2026-05-15 09:42:04,147][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-15 09:42:04,147][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-15 09:42:05,825][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-15 09:42:08,703][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.0011891471357466177
	[2026-05-15 09:42:10,060][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-15 09:42:35,278][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.04446739651347892
	[2026-05-15 09:42:36,988][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-15 09:42:41,086][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-15 09:43:06,623][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-15 09:43:11,005][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-15 09:43:25,138][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-15 09:43:26,907][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-15 09:43:28,888][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-15 09:43:30,810][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-15 09:43:32,343][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-15 09:44:08,030][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-15 09:44:08,030][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:44:08,030][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-15 09:44:09,729][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-15 09:44:11,586][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-15 09:44:13,229][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-15 09:44:15,007][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-15 09:44:31,802][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-15 09:44:31,802][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:44:31,802][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-15 09:44:31,803][metrics][INFO] - Evaluating model_utility
	[2026-05-15 09:44:31,804][evaluator][INFO] - Result for metric model_utility: 0.4895859819892679
	[2026-05-15 09:44:34,872][metrics][INFO] - Evaluating mia_min_k
	[2026-05-15 09:44:36,019][metrics][INFO] - Evaluating privleak
	[2026-05-15 09:44:36,019][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-15 09:44:36,019][evaluator][INFO] - Result for metric privleak: 98.56499998028701
	[2026-05-15 09:44:37,347][metrics][INFO] - Evaluating extraction_strength
	[2026-05-15 09:44:38,246][evaluator][INFO] - Result for metric extraction_strength: 0.03250892997513522
	[2026-05-15 09:44:42,573][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-15 09:44:42,574][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-56/evals/TOFU_EVAL.json
	[2026-05-15 09:44:42,574][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-56/evals/TOFU_SUMMARY.json
	[2026-05-15 09:44:44,251][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-15 09:44:48,431][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-15 09:45:02,780][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-15 09:45:02,781][evaluator][INFO] - Result for metric forget_truth_ratio: 0.007522837147721586
	[2026-05-15 09:45:02,788][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-15 09:45:02,789][metrics][INFO] - Evaluating forget_quality
	[2026-05-15 09:45:02,789][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-15 09:45:02,789][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-15 09:45:04,482][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-15 09:45:07,366][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.001106236909597593
	[2026-05-15 09:45:08,744][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-15 09:45:33,768][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.03212130630929463
	[2026-05-15 09:45:35,433][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-15 09:45:39,534][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-15 09:46:00,019][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-15 09:46:04,186][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-15 09:46:18,308][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-15 09:46:20,027][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-15 09:46:21,958][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-15 09:46:23,884][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-15 09:46:25,119][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-15 09:46:43,271][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-15 09:46:43,271][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:46:43,272][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-15 09:46:44,915][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-15 09:46:46,650][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-15 09:46:48,286][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-15 09:46:49,541][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-15 09:46:54,732][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-15 09:46:54,732][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:46:54,732][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-15 09:46:54,732][metrics][INFO] - Evaluating model_utility
	[2026-05-15 09:46:54,733][evaluator][INFO] - Result for metric model_utility: 0.5033978222900254
	[2026-05-15 09:46:57,819][metrics][INFO] - Evaluating mia_min_k
	[2026-05-15 09:46:58,961][metrics][INFO] - Evaluating privleak
	[2026-05-15 09:46:58,962][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-15 09:46:58,962][evaluator][INFO] - Result for metric privleak: 98.8924999802215
	[2026-05-15 09:47:00,251][metrics][INFO] - Evaluating extraction_strength
	[2026-05-15 09:47:01,148][evaluator][INFO] - Result for metric extraction_strength: 0.03250892997513522
	[2026-05-15 09:47:03,328][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-15 09:47:03,329][evaluator][INFO] - Fine-grained evaluations will be saved to: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-60/evals/TOFU_EVAL.json
	[2026-05-15 09:47:03,329][evaluator][INFO] - Aggregated evaluations will be summarised in: ./saves/unlearn/A1_tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_NoQAT/checkpoint-60/evals/TOFU_SUMMARY.json
	[2026-05-15 09:47:04,669][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-15 09:47:08,831][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-15 09:47:23,156][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-15 09:47:23,157][evaluator][INFO] - Result for metric forget_truth_ratio: 0.007090752220694865
	[2026-05-15 09:47:23,164][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-15 09:47:23,164][metrics][INFO] - Evaluating forget_quality
	[2026-05-15 09:47:23,165][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-15 09:47:23,165][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-15 09:47:24,913][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-15 09:47:27,795][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.0011314425781798782
	[2026-05-15 09:47:29,048][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-15 09:47:54,135][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.029394867577355592
	[2026-05-15 09:47:55,837][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-15 09:47:59,991][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-15 09:48:19,627][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-15 09:48:23,816][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-15 09:48:37,927][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-15 09:48:39,715][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-15 09:48:41,666][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-15 09:48:43,591][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-15 09:48:44,923][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-15 09:48:51,033][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-15 09:48:51,033][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:48:51,033][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-15 09:48:54,083][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-15 09:48:56,255][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-15 09:48:57,891][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-15 09:48:59,313][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-15 09:49:26,749][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-15 09:49:26,749][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-15 09:49:26,749][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-15 09:49:26,750][metrics][INFO] - Evaluating model_utility
	[2026-05-15 09:49:26,750][evaluator][INFO] - Result for metric model_utility: 0.5152088536846021
	[2026-05-15 09:49:29,733][metrics][INFO] - Evaluating mia_min_k
	[2026-05-15 09:49:30,878][metrics][INFO] - Evaluating privleak
	[2026-05-15 09:49:30,879][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-15 09:49:30,879][evaluator][INFO] - Result for metric privleak: 98.90874998021826
	[2026-05-15 09:49:32,192][metrics][INFO] - Evaluating extraction_strength
	[2026-05-15 09:49:33,089][evaluator][INFO] - Result for metric extraction_strength: 0.03250892997513522