Instructions to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8")
model = AutoModelForCausalLM.from_pretrained("Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8

SGLang

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8 with Docker Model Runner:
```
docker model run hf.co/Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8
```

tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8 / GradDiff.log

Jeesup

Model save

d2fcd44 verified 3 months ago

Raw

History Blame Contribute Delete

42 kB

	[2026-05-20 18:33:19,251][model][INFO] - Setting pad_token as eos token: <\|eot_id\|>
	[2026-05-20 18:33:24,044][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8
	[2026-05-20 18:33:24,915][trainer][INFO] - GradDiff Trainer loaded, output_dir: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8
	[2026-05-20 18:33:25,455][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 18:33:25,455][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-0/evals/TOFU_EVAL.json
	[2026-05-20 18:33:25,455][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-0/evals/TOFU_SUMMARY.json
	[2026-05-20 18:33:26,965][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 18:33:43,692][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 18:34:20,829][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 18:34:20,830][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4789170850743798
	[2026-05-20 18:34:20,836][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 18:34:20,837][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 18:34:20,837][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 18:34:20,837][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 18:34:22,933][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 18:34:27,762][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.8784031949937343
	[2026-05-20 18:34:29,444][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 18:35:29,091][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.819146168073363
	[2026-05-20 18:35:30,843][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 18:35:37,945][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 18:36:00,532][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 18:36:07,342][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 18:36:29,038][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 18:36:30,735][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 18:36:33,929][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 18:36:36,727][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 18:36:37,998][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 18:36:47,159][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 18:36:47,159][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:36:47,159][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 18:36:48,907][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 18:36:51,192][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 18:36:55,649][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 18:36:56,899][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 18:37:12,271][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 18:37:12,272][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:37:12,272][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 18:37:12,272][metrics][INFO] - Evaluating model_utility
	[2026-05-20 18:37:12,273][evaluator][INFO] - Result for metric model_utility: 0.5965321121365322
	[2026-05-20 18:37:16,393][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 18:37:21,629][metrics][INFO] - Evaluating privleak
	[2026-05-20 18:37:21,629][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 18:37:21,629][evaluator][INFO] - Result for metric privleak: -99.30749998013852
	[2026-05-20 18:37:23,442][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 18:37:25,027][evaluator][INFO] - Result for metric extraction_strength: 0.6937715381623286
	[2026-05-20 18:38:41,827][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 18:38:41,828][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-25/evals/TOFU_EVAL.json
	[2026-05-20 18:38:41,828][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-25/evals/TOFU_SUMMARY.json
	[2026-05-20 18:38:43,831][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 18:38:48,782][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 18:39:17,165][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 18:39:17,166][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4754659835378095
	[2026-05-20 18:39:17,173][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 18:39:17,173][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 18:39:17,173][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 18:39:17,173][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 18:39:18,936][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 18:39:24,905][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.8443998385965824
	[2026-05-20 18:39:26,775][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 18:40:08,926][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.7449732731400306
	[2026-05-20 18:40:10,771][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 18:40:17,031][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 18:40:37,381][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 18:40:42,402][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 18:40:59,137][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 18:41:00,805][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 18:41:02,832][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 18:41:05,122][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 18:41:06,355][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 18:41:09,820][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 18:41:09,820][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:41:09,820][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 18:41:11,057][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 18:41:13,168][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 18:41:15,369][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 18:41:16,800][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 18:41:24,872][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 18:41:24,873][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:41:24,873][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 18:41:24,873][metrics][INFO] - Evaluating model_utility
	[2026-05-20 18:41:24,874][evaluator][INFO] - Result for metric model_utility: 0.5971407525626945
	[2026-05-20 18:41:27,887][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 18:41:33,175][metrics][INFO] - Evaluating privleak
	[2026-05-20 18:41:33,175][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 18:41:33,175][evaluator][INFO] - Result for metric privleak: -99.23874998015224
	[2026-05-20 18:41:35,095][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 18:41:37,718][evaluator][INFO] - Result for metric extraction_strength: 0.5974875899653196
	[2026-05-20 18:42:01,107][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 18:42:01,108][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-50/evals/TOFU_EVAL.json
	[2026-05-20 18:42:01,108][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-50/evals/TOFU_SUMMARY.json
	[2026-05-20 18:42:02,816][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 18:42:10,627][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 18:42:39,940][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 18:42:39,941][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4456223296471336
	[2026-05-20 18:42:39,947][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 18:42:39,948][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 18:42:39,948][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 18:42:39,948][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 18:42:41,707][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 18:42:47,666][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.40393723831046374
	[2026-05-20 18:42:49,289][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 18:43:27,958][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.46180837574750855
	[2026-05-20 18:43:29,655][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 18:43:37,439][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 18:44:06,681][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 18:44:14,371][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 18:44:34,723][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 18:44:36,833][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 18:44:38,936][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 18:44:41,227][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 18:44:42,488][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 18:44:47,607][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 18:44:47,608][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:44:47,608][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 18:44:49,508][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 18:44:51,396][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 18:44:53,350][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 18:44:54,629][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 18:44:58,751][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 18:44:58,752][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:44:58,752][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 18:44:58,752][metrics][INFO] - Evaluating model_utility
	[2026-05-20 18:44:58,753][evaluator][INFO] - Result for metric model_utility: 0.519404205302183
	[2026-05-20 18:45:01,251][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 18:45:02,939][metrics][INFO] - Evaluating privleak
	[2026-05-20 18:45:02,939][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 18:45:02,939][evaluator][INFO] - Result for metric privleak: -92.254999981549
	[2026-05-20 18:45:04,389][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 18:45:05,544][evaluator][INFO] - Result for metric extraction_strength: 0.19059671420278737
	[2026-05-20 18:45:19,033][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 18:45:19,033][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-75/evals/TOFU_EVAL.json
	[2026-05-20 18:45:19,033][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-75/evals/TOFU_SUMMARY.json
	[2026-05-20 18:45:20,772][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 18:45:25,650][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 18:45:51,871][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 18:45:51,871][evaluator][INFO] - Result for metric forget_truth_ratio: 0.45236383647433437
	[2026-05-20 18:45:51,878][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 18:45:51,878][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 18:45:51,878][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 18:45:51,878][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 18:45:53,811][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 18:45:59,806][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.10694494379058597
	[2026-05-20 18:46:01,670][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 18:46:47,405][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.36006360016684735
	[2026-05-20 18:46:49,110][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 18:46:53,654][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 18:47:32,793][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 18:47:37,086][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 18:47:58,899][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 18:48:01,008][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 18:48:04,302][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 18:48:08,197][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 18:48:09,708][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 18:48:22,473][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 18:48:22,474][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:48:22,474][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 18:48:24,255][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 18:48:26,073][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 18:48:27,897][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 18:48:29,143][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 18:48:35,542][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 18:48:35,542][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:48:35,542][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 18:48:35,543][metrics][INFO] - Evaluating model_utility
	[2026-05-20 18:48:35,544][evaluator][INFO] - Result for metric model_utility: 0.44625926579272773
	[2026-05-20 18:48:38,512][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 18:48:40,030][metrics][INFO] - Evaluating privleak
	[2026-05-20 18:48:40,030][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 18:48:40,030][evaluator][INFO] - Result for metric privleak: -62.208749987558264
	[2026-05-20 18:48:41,377][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 18:48:42,454][evaluator][INFO] - Result for metric extraction_strength: 0.10082313815291738
	[2026-05-20 18:48:54,085][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 18:48:54,085][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-100/evals/TOFU_EVAL.json
	[2026-05-20 18:48:54,085][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-100/evals/TOFU_SUMMARY.json
	[2026-05-20 18:48:55,964][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 18:49:00,460][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 18:49:15,803][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 18:49:15,804][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4621917530992918
	[2026-05-20 18:49:15,810][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 18:49:15,811][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 18:49:15,811][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 18:49:15,811][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 18:49:17,678][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 18:49:21,167][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.05674113624557322
	[2026-05-20 18:49:22,607][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 18:49:52,271][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.3503032374110272
	[2026-05-20 18:49:53,907][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 18:49:58,615][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 18:50:25,511][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 18:50:30,220][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 18:50:47,488][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 18:50:49,130][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 18:50:51,272][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 18:50:53,656][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 18:50:54,888][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 18:51:03,090][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 18:51:03,091][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:51:03,091][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 18:51:05,384][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 18:51:07,271][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 18:51:09,325][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 18:51:10,562][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 18:51:15,041][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 18:51:15,041][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:51:15,041][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 18:51:15,041][metrics][INFO] - Evaluating model_utility
	[2026-05-20 18:51:15,042][evaluator][INFO] - Result for metric model_utility: 0.4619823061516691
	[2026-05-20 18:51:17,646][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 18:51:19,716][metrics][INFO] - Evaluating privleak
	[2026-05-20 18:51:19,716][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 18:51:19,716][evaluator][INFO] - Result for metric privleak: -28.147499994370513
	[2026-05-20 18:51:20,986][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 18:51:24,386][evaluator][INFO] - Result for metric extraction_strength: 0.08743730255316465
	[2026-05-20 18:51:34,460][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 18:51:34,461][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-125/evals/TOFU_EVAL.json
	[2026-05-20 18:51:34,461][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-125/evals/TOFU_SUMMARY.json
	[2026-05-20 18:51:36,173][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 18:51:40,877][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 18:51:58,404][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 18:51:58,405][evaluator][INFO] - Result for metric forget_truth_ratio: 0.460636314841849
	[2026-05-20 18:51:58,412][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 18:51:58,412][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 18:51:58,412][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 18:51:58,412][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 18:52:00,126][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 18:52:03,628][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.029644731777340214
	[2026-05-20 18:52:04,925][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 18:52:39,740][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.32734289299687996
	[2026-05-20 18:52:41,355][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 18:52:46,009][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 18:53:13,649][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 18:53:18,451][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 18:53:35,650][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 18:53:37,360][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 18:53:39,506][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 18:53:41,896][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 18:53:43,301][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 18:53:57,101][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 18:53:57,101][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:53:57,101][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 18:53:58,766][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 18:54:00,647][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 18:54:02,465][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 18:54:03,731][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 18:54:10,009][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 18:54:10,009][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:54:10,009][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 18:54:10,010][metrics][INFO] - Evaluating model_utility
	[2026-05-20 18:54:10,010][evaluator][INFO] - Result for metric model_utility: 0.43944058949037856
	[2026-05-20 18:54:13,244][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 18:54:14,930][metrics][INFO] - Evaluating privleak
	[2026-05-20 18:54:14,930][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 18:54:14,930][evaluator][INFO] - Result for metric privleak: 7.868749998426253
	[2026-05-20 18:54:16,211][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 18:54:17,720][evaluator][INFO] - Result for metric extraction_strength: 0.07439371648760677
	[2026-05-20 18:54:29,313][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 18:54:29,313][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-150/evals/TOFU_EVAL.json
	[2026-05-20 18:54:29,313][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-150/evals/TOFU_SUMMARY.json
	[2026-05-20 18:54:31,130][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 18:54:36,684][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 18:54:59,241][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 18:54:59,242][evaluator][INFO] - Result for metric forget_truth_ratio: 0.46603235373847013
	[2026-05-20 18:54:59,248][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 18:54:59,249][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 18:54:59,249][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 18:54:59,249][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 18:55:00,944][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 18:55:05,568][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.021181890712613692
	[2026-05-20 18:55:06,955][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 18:55:51,234][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.3079601677209019
	[2026-05-20 18:55:52,998][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 18:55:57,525][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 18:56:32,210][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 18:56:38,666][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 18:56:57,794][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 18:56:59,546][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 18:57:01,544][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 18:57:03,715][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 18:57:05,386][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 18:57:10,916][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 18:57:10,917][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:57:10,917][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 18:57:12,567][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 18:57:14,442][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 18:57:16,307][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 18:57:17,544][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 18:57:24,260][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 18:57:24,260][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 18:57:24,260][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 18:57:24,260][metrics][INFO] - Evaluating model_utility
	[2026-05-20 18:57:24,261][evaluator][INFO] - Result for metric model_utility: 0.4481899508054402
	[2026-05-20 18:57:27,235][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 18:57:28,772][metrics][INFO] - Evaluating privleak
	[2026-05-20 18:57:28,772][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 18:57:28,772][evaluator][INFO] - Result for metric privleak: 31.69249999366149
	[2026-05-20 18:57:30,127][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 18:57:31,245][evaluator][INFO] - Result for metric extraction_strength: 0.06897252474953364
	[2026-05-20 18:57:44,464][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 18:57:44,464][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-175/evals/TOFU_EVAL.json
	[2026-05-20 18:57:44,464][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-175/evals/TOFU_SUMMARY.json
	[2026-05-20 18:57:46,103][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 18:57:52,780][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 18:58:15,964][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 18:58:15,965][evaluator][INFO] - Result for metric forget_truth_ratio: 0.46723465704856637
	[2026-05-20 18:58:15,971][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 18:58:15,972][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 18:58:15,972][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 18:58:15,972][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 18:58:17,610][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 18:58:22,338][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.0171702699252306
	[2026-05-20 18:58:23,803][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 18:59:33,412][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.27420147937146
	[2026-05-20 18:59:35,067][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 18:59:39,989][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 19:00:11,397][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 19:00:15,793][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 19:00:31,373][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 19:00:33,065][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 19:00:35,042][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 19:00:37,973][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 19:00:39,445][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 19:00:50,688][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 19:00:50,688][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 19:00:50,688][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 19:00:52,410][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 19:00:54,707][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 19:00:56,566][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 19:00:57,860][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 19:01:05,341][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 19:01:05,341][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 19:01:05,341][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 19:01:05,342][metrics][INFO] - Evaluating model_utility
	[2026-05-20 19:01:05,342][evaluator][INFO] - Result for metric model_utility: 0.4391376484671766
	[2026-05-20 19:01:08,267][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 19:01:09,805][metrics][INFO] - Evaluating privleak
	[2026-05-20 19:01:09,805][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 19:01:09,805][evaluator][INFO] - Result for metric privleak: 46.2474999907505
	[2026-05-20 19:01:11,188][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 19:01:12,300][evaluator][INFO] - Result for metric extraction_strength: 0.06470503101323988
	[2026-05-20 19:01:26,408][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 19:01:26,408][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-200/evals/TOFU_EVAL.json
	[2026-05-20 19:01:26,408][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-200/evals/TOFU_SUMMARY.json
	[2026-05-20 19:01:28,160][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 19:01:32,671][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 19:01:49,161][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 19:01:49,161][evaluator][INFO] - Result for metric forget_truth_ratio: 0.47004934084315964
	[2026-05-20 19:01:49,168][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 19:01:49,169][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 19:01:49,169][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 19:01:49,169][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 19:01:50,869][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 19:01:54,055][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.014899758294861933
	[2026-05-20 19:01:55,310][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 19:03:14,386][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.25663827470039985
	[2026-05-20 19:03:16,012][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 19:03:22,090][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 19:04:04,129][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 19:04:08,860][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 19:04:25,653][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 19:04:27,344][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 19:04:30,197][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 19:04:34,312][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 19:04:35,587][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 19:04:50,187][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 19:04:50,187][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 19:04:50,188][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 19:04:51,950][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 19:04:54,300][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 19:04:57,662][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 19:04:59,550][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 19:05:09,258][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 19:05:09,259][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 19:05:09,259][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 19:05:09,259][metrics][INFO] - Evaluating model_utility
	[2026-05-20 19:05:09,260][evaluator][INFO] - Result for metric model_utility: 0.43758221054998514
	[2026-05-20 19:05:12,248][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 19:05:16,980][metrics][INFO] - Evaluating privleak
	[2026-05-20 19:05:16,981][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 19:05:16,981][evaluator][INFO] - Result for metric privleak: 53.85624998922873
	[2026-05-20 19:05:18,344][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 19:05:34,312][evaluator][INFO] - Result for metric extraction_strength: 0.06126946380225206
	[2026-05-20 19:05:52,788][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 19:05:52,788][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-225/evals/TOFU_EVAL.json
	[2026-05-20 19:05:52,789][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-225/evals/TOFU_SUMMARY.json
	[2026-05-20 19:05:54,432][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 19:05:59,080][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 19:06:15,818][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 19:06:15,819][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4727602307165236
	[2026-05-20 19:06:15,826][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 19:06:15,826][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 19:06:15,826][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 19:06:15,826][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 19:06:17,492][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 19:06:20,931][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.013935726931003384
	[2026-05-20 19:06:22,183][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 19:07:43,410][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.24396801511962757
	[2026-05-20 19:07:45,093][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 19:07:52,638][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 19:09:05,285][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 19:09:12,240][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 19:09:37,136][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 19:09:38,769][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 19:09:41,802][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 19:09:45,496][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 19:09:46,732][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 19:09:55,592][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 19:09:55,592][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 19:09:55,592][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 19:09:57,255][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 19:09:59,063][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 19:10:02,619][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 19:10:03,979][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 19:10:12,620][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 19:10:12,621][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 19:10:12,621][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 19:10:12,621][metrics][INFO] - Evaluating model_utility
	[2026-05-20 19:10:12,622][evaluator][INFO] - Result for metric model_utility: 0.43161757614068297
	[2026-05-20 19:10:15,606][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 19:10:17,414][metrics][INFO] - Evaluating privleak
	[2026-05-20 19:10:17,415][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 19:10:17,415][evaluator][INFO] - Result for metric privleak: 56.50874998869825
	[2026-05-20 19:10:18,677][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 19:10:19,838][evaluator][INFO] - Result for metric extraction_strength: 0.059159267203749824
	[2026-05-20 19:10:31,063][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 19:10:31,064][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-250/evals/TOFU_EVAL.json
	[2026-05-20 19:10:31,064][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-w8/checkpoint-250/evals/TOFU_SUMMARY.json
	[2026-05-20 19:10:32,732][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 19:10:37,326][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 19:10:54,008][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 19:10:54,009][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4679999712405959
	[2026-05-20 19:10:54,015][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 19:10:54,016][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 19:10:54,016][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 19:10:54,016][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 19:10:55,759][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 19:10:59,121][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.014001140977138626
	[2026-05-20 19:11:00,375][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 19:12:09,626][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.24989959709489049
	[2026-05-20 19:12:11,345][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 19:12:15,952][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 19:12:59,593][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 19:13:04,006][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 19:13:19,602][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 19:13:21,262][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 19:13:23,209][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 19:13:25,305][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 19:13:26,552][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 19:14:11,145][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 19:14:11,145][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 19:14:11,145][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 19:14:12,781][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 19:14:14,573][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 19:14:16,399][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 19:14:17,651][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 19:14:23,200][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 19:14:23,200][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 19:14:23,200][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 19:14:23,201][metrics][INFO] - Evaluating model_utility
	[2026-05-20 19:14:23,201][evaluator][INFO] - Result for metric model_utility: 0.43327818052728545
	[2026-05-20 19:14:26,140][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 19:14:27,674][metrics][INFO] - Evaluating privleak
	[2026-05-20 19:14:27,675][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 19:14:27,675][evaluator][INFO] - Result for metric privleak: 56.786249988642744
	[2026-05-20 19:14:29,053][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 19:14:30,129][evaluator][INFO] - Result for metric extraction_strength: 0.05824998265202028