Instructions to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4")
model = AutoModelForCausalLM.from_pretrained("Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4

SGLang

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4 with Docker Model Runner:
```
docker model run hf.co/Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4
```

tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4 / GradDiff.log

Jeesup

Model save

3929de6 verified 3 months ago

Raw

History Blame Contribute Delete

42 kB

	[2026-05-20 17:03:22,169][model][INFO] - Setting pad_token as eos token: <\|eot_id\|>
	[2026-05-20 17:03:26,012][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4
	[2026-05-20 17:03:27,699][trainer][INFO] - GradDiff Trainer loaded, output_dir: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4
	[2026-05-20 17:03:28,249][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 17:03:28,249][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-0/evals/TOFU_EVAL.json
	[2026-05-20 17:03:28,249][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-0/evals/TOFU_SUMMARY.json
	[2026-05-20 17:03:29,782][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 17:03:49,790][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 17:04:33,047][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 17:04:33,048][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5053507138495464
	[2026-05-20 17:04:33,054][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 17:04:33,055][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 17:04:33,055][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 17:04:33,055][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 17:04:35,024][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 17:04:44,366][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.6942386485636234
	[2026-05-20 17:04:46,064][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 17:06:27,815][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.5513711559350298
	[2026-05-20 17:06:29,488][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 17:06:38,043][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 17:07:21,796][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 17:07:29,416][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 17:07:54,251][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 17:07:55,948][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 17:07:59,103][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 17:08:03,086][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 17:08:04,346][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 17:08:17,005][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 17:08:17,005][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:08:17,005][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 17:08:18,661][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 17:08:21,136][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 17:08:26,228][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 17:08:28,154][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 17:08:50,770][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 17:08:50,771][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:08:50,771][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 17:08:50,771][metrics][INFO] - Evaluating model_utility
	[2026-05-20 17:08:50,772][evaluator][INFO] - Result for metric model_utility: 0.5409656142953374
	[2026-05-20 17:08:53,723][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 17:09:01,812][metrics][INFO] - Evaluating privleak
	[2026-05-20 17:09:01,812][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 17:09:01,812][evaluator][INFO] - Result for metric privleak: -98.41374998031725
	[2026-05-20 17:09:03,464][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 17:09:04,865][evaluator][INFO] - Result for metric extraction_strength: 0.2637027593640881
	[2026-05-20 17:10:50,091][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 17:10:50,093][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-25/evals/TOFU_EVAL.json
	[2026-05-20 17:10:50,093][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-25/evals/TOFU_SUMMARY.json
	[2026-05-20 17:10:52,074][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 17:10:56,679][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 17:11:12,806][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 17:11:12,807][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5078252569138209
	[2026-05-20 17:11:12,813][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 17:11:12,814][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 17:11:12,814][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 17:11:12,814][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 17:11:14,586][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 17:11:17,828][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.630031465254724
	[2026-05-20 17:11:19,092][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 17:11:51,277][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.5133406632629737
	[2026-05-20 17:11:53,000][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 17:11:58,091][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 17:12:44,975][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 17:12:50,193][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 17:13:09,263][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 17:13:10,991][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 17:13:13,031][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 17:13:15,818][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 17:13:17,072][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 17:13:25,474][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 17:13:25,474][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:13:25,474][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 17:13:27,609][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 17:13:29,743][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 17:13:32,238][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 17:13:33,499][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 17:13:42,445][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 17:13:42,445][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:13:42,445][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 17:13:42,446][metrics][INFO] - Evaluating model_utility
	[2026-05-20 17:13:42,447][evaluator][INFO] - Result for metric model_utility: 0.53664529306797
	[2026-05-20 17:13:45,557][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 17:13:48,449][metrics][INFO] - Evaluating privleak
	[2026-05-20 17:13:48,450][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 17:13:48,450][evaluator][INFO] - Result for metric privleak: -97.419999980516
	[2026-05-20 17:13:49,724][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 17:13:54,001][evaluator][INFO] - Result for metric extraction_strength: 0.21643616471407992
	[2026-05-20 17:14:14,212][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 17:14:14,212][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-50/evals/TOFU_EVAL.json
	[2026-05-20 17:14:14,213][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-50/evals/TOFU_SUMMARY.json
	[2026-05-20 17:14:15,874][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 17:14:21,061][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 17:14:40,293][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 17:14:40,294][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4968857074337199
	[2026-05-20 17:14:40,300][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 17:14:40,300][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 17:14:40,301][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 17:14:40,301][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 17:14:41,949][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 17:14:45,845][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.17429078094704892
	[2026-05-20 17:14:47,118][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 17:15:47,522][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.3784161984827456
	[2026-05-20 17:15:49,343][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 17:15:53,784][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 17:16:33,070][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 17:16:40,369][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 17:17:04,366][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 17:17:05,997][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 17:17:08,105][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 17:17:10,437][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 17:17:11,901][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 17:17:20,545][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 17:17:20,545][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:17:20,545][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 17:17:22,224][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 17:17:24,462][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 17:17:26,483][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 17:17:27,703][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 17:17:36,061][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 17:17:36,061][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:17:36,061][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 17:17:36,061][metrics][INFO] - Evaluating model_utility
	[2026-05-20 17:17:36,062][evaluator][INFO] - Result for metric model_utility: 0.461249581028985
	[2026-05-20 17:17:39,064][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 17:17:40,936][metrics][INFO] - Evaluating privleak
	[2026-05-20 17:17:40,936][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 17:17:40,936][evaluator][INFO] - Result for metric privleak: -58.65124998826975
	[2026-05-20 17:17:42,284][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 17:17:43,541][evaluator][INFO] - Result for metric extraction_strength: 0.09333800130759574
	[2026-05-20 17:18:02,526][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 17:18:02,526][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-75/evals/TOFU_EVAL.json
	[2026-05-20 17:18:02,526][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-75/evals/TOFU_SUMMARY.json
	[2026-05-20 17:18:04,202][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 17:18:09,042][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 17:18:27,760][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 17:18:27,761][evaluator][INFO] - Result for metric forget_truth_ratio: 0.49287134947497685
	[2026-05-20 17:18:27,768][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 17:18:27,768][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 17:18:27,768][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 17:18:27,768][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 17:18:29,424][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 17:18:33,974][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.0801628143481139
	[2026-05-20 17:18:35,282][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 17:19:38,175][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.3570944755980755
	[2026-05-20 17:19:39,842][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 17:19:44,356][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 17:20:27,179][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 17:20:31,699][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 17:20:47,501][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 17:20:49,112][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 17:20:51,122][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 17:20:53,364][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 17:20:54,704][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 17:21:01,620][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 17:21:01,620][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:21:01,620][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 17:21:03,263][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 17:21:05,801][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 17:21:07,752][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 17:21:09,039][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 17:21:16,023][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 17:21:16,024][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:21:16,024][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 17:21:16,024][metrics][INFO] - Evaluating model_utility
	[2026-05-20 17:21:16,025][evaluator][INFO] - Result for metric model_utility: 0.4548301936479817
	[2026-05-20 17:21:19,211][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 17:21:21,018][metrics][INFO] - Evaluating privleak
	[2026-05-20 17:21:21,018][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 17:21:21,018][evaluator][INFO] - Result for metric privleak: -19.95124999600975
	[2026-05-20 17:21:22,370][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 17:21:23,584][evaluator][INFO] - Result for metric extraction_strength: 0.07752306189364604
	[2026-05-20 17:21:38,836][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 17:21:38,836][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-100/evals/TOFU_EVAL.json
	[2026-05-20 17:21:38,836][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-100/evals/TOFU_SUMMARY.json
	[2026-05-20 17:21:40,485][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 17:21:44,974][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 17:22:00,970][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 17:22:00,970][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4875543212668265
	[2026-05-20 17:22:00,977][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 17:22:00,977][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 17:22:00,977][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 17:22:00,977][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 17:22:02,828][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 17:22:06,046][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.03949361748988849
	[2026-05-20 17:22:07,277][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 17:22:50,462][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.32424812252699664
	[2026-05-20 17:22:52,162][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 17:22:56,804][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 17:23:27,166][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 17:23:31,653][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 17:23:47,453][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 17:23:49,137][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 17:23:51,296][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 17:23:53,547][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 17:23:54,794][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 17:24:04,953][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 17:24:04,954][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:24:04,954][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 17:24:06,618][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 17:24:08,567][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 17:24:10,516][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 17:24:11,779][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 17:24:18,212][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 17:24:18,212][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:24:18,212][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 17:24:18,213][metrics][INFO] - Evaluating model_utility
	[2026-05-20 17:24:18,213][evaluator][INFO] - Result for metric model_utility: 0.41995432731938637
	[2026-05-20 17:24:21,240][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 17:24:23,043][metrics][INFO] - Evaluating privleak
	[2026-05-20 17:24:23,044][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 17:24:23,044][evaluator][INFO] - Result for metric privleak: 14.652499997069501
	[2026-05-20 17:24:24,316][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 17:24:25,538][evaluator][INFO] - Result for metric extraction_strength: 0.06639812750319583
	[2026-05-20 17:24:37,552][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 17:24:37,552][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-125/evals/TOFU_EVAL.json
	[2026-05-20 17:24:37,552][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-125/evals/TOFU_SUMMARY.json
	[2026-05-20 17:24:39,205][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 17:24:43,708][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 17:24:59,704][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 17:24:59,705][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4793975600229985
	[2026-05-20 17:24:59,712][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 17:24:59,712][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 17:24:59,712][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 17:24:59,712][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 17:25:01,382][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 17:25:04,602][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.029049916124847072
	[2026-05-20 17:25:06,008][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 17:26:17,957][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2989380417883505
	[2026-05-20 17:26:19,731][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 17:26:24,180][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 17:27:22,608][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 17:27:27,071][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 17:27:42,862][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 17:27:44,541][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 17:27:46,567][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 17:27:48,815][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 17:27:50,087][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 17:28:00,278][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 17:28:00,279][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:28:00,279][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 17:28:02,085][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 17:28:03,976][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 17:28:05,926][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 17:28:07,494][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 17:28:17,113][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 17:28:17,114][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:28:17,114][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 17:28:17,114][metrics][INFO] - Evaluating model_utility
	[2026-05-20 17:28:17,115][evaluator][INFO] - Result for metric model_utility: 0.41995590220844287
	[2026-05-20 17:28:20,142][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 17:28:21,944][metrics][INFO] - Evaluating privleak
	[2026-05-20 17:28:21,944][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 17:28:21,944][evaluator][INFO] - Result for metric privleak: 33.573749993285254
	[2026-05-20 17:28:23,372][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 17:28:24,590][evaluator][INFO] - Result for metric extraction_strength: 0.0614061298993083
	[2026-05-20 17:28:38,153][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 17:28:38,153][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-150/evals/TOFU_EVAL.json
	[2026-05-20 17:28:38,153][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-150/evals/TOFU_SUMMARY.json
	[2026-05-20 17:28:39,802][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 17:28:44,399][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 17:29:00,396][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 17:29:00,397][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4743665435350599
	[2026-05-20 17:29:00,404][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 17:29:00,404][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 17:29:00,404][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 17:29:00,404][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 17:29:02,073][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 17:29:05,302][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.0253322103745046
	[2026-05-20 17:29:06,634][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 17:30:34,828][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.27906457384796773
	[2026-05-20 17:30:36,841][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 17:30:41,265][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 17:31:29,392][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 17:31:33,862][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 17:31:49,655][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 17:31:51,449][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 17:31:53,610][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 17:31:55,863][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 17:31:57,208][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 17:32:05,846][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 17:32:05,846][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:32:05,846][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 17:32:07,545][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 17:32:09,395][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 17:32:11,353][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 17:32:12,666][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 17:32:24,371][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 17:32:24,372][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:32:24,372][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 17:32:24,372][metrics][INFO] - Evaluating model_utility
	[2026-05-20 17:32:24,373][evaluator][INFO] - Result for metric model_utility: 0.4086782879086552
	[2026-05-20 17:32:27,369][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 17:32:29,184][metrics][INFO] - Evaluating privleak
	[2026-05-20 17:32:29,184][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 17:32:29,184][evaluator][INFO] - Result for metric privleak: 42.7024999914595
	[2026-05-20 17:32:30,473][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 17:32:31,691][evaluator][INFO] - Result for metric extraction_strength: 0.05883578111644736
	[2026-05-20 17:32:46,903][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 17:32:46,904][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-175/evals/TOFU_EVAL.json
	[2026-05-20 17:32:46,904][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-175/evals/TOFU_SUMMARY.json
	[2026-05-20 17:32:48,643][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 17:32:53,141][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 17:33:09,162][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 17:33:09,163][evaluator][INFO] - Result for metric forget_truth_ratio: 0.47397445910774466
	[2026-05-20 17:33:09,169][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 17:33:09,170][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 17:33:09,170][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 17:33:09,170][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 17:33:10,838][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 17:33:14,071][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.023672567030769046
	[2026-05-20 17:33:15,339][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 17:34:43,684][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.26270419012198476
	[2026-05-20 17:34:45,631][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 17:34:50,197][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 17:35:44,930][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 17:35:49,648][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 17:36:05,449][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 17:36:07,074][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 17:36:09,094][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 17:36:11,344][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 17:36:12,594][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 17:36:21,420][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 17:36:21,420][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:36:21,420][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 17:36:23,398][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 17:36:25,378][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 17:36:27,336][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 17:36:28,641][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 17:36:39,760][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 17:36:39,760][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:36:39,760][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 17:36:39,760][metrics][INFO] - Evaluating model_utility
	[2026-05-20 17:36:39,761][evaluator][INFO] - Result for metric model_utility: 0.40431969658337713
	[2026-05-20 17:36:42,754][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 17:36:44,573][metrics][INFO] - Evaluating privleak
	[2026-05-20 17:36:44,573][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 17:36:44,573][evaluator][INFO] - Result for metric privleak: 46.829999990634015
	[2026-05-20 17:36:45,863][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 17:36:47,087][evaluator][INFO] - Result for metric extraction_strength: 0.05620382772216574
	[2026-05-20 17:37:04,527][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 17:37:04,528][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-200/evals/TOFU_EVAL.json
	[2026-05-20 17:37:04,528][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-200/evals/TOFU_SUMMARY.json
	[2026-05-20 17:37:06,303][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 17:37:10,808][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 17:37:26,832][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 17:37:26,833][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4718424531768588
	[2026-05-20 17:37:26,839][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 17:37:26,840][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 17:37:26,840][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 17:37:26,840][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 17:37:28,475][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 17:37:31,708][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.022784052822857177
	[2026-05-20 17:37:32,962][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 17:39:07,006][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.25067003573206453
	[2026-05-20 17:39:08,991][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 17:39:13,633][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 17:40:19,268][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 17:40:24,079][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 17:40:39,891][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 17:40:41,784][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 17:40:43,905][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 17:40:46,153][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 17:40:47,769][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 17:41:08,192][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 17:41:08,193][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:41:08,193][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 17:41:10,010][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 17:41:11,965][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 17:41:13,919][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 17:41:15,235][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 17:41:21,892][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 17:41:21,892][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:41:21,892][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 17:41:21,893][metrics][INFO] - Evaluating model_utility
	[2026-05-20 17:41:21,894][evaluator][INFO] - Result for metric model_utility: 0.40902266186998065
	[2026-05-20 17:41:25,327][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 17:41:27,132][metrics][INFO] - Evaluating privleak
	[2026-05-20 17:41:27,132][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 17:41:27,132][evaluator][INFO] - Result for metric privleak: 48.382499990323495
	[2026-05-20 17:41:28,469][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 17:41:29,697][evaluator][INFO] - Result for metric extraction_strength: 0.05441809783527104
	[2026-05-20 17:41:41,741][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 17:41:41,741][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-225/evals/TOFU_EVAL.json
	[2026-05-20 17:41:41,741][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-225/evals/TOFU_SUMMARY.json
	[2026-05-20 17:41:43,443][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 17:41:47,964][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 17:42:03,978][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 17:42:03,979][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4713051809185062
	[2026-05-20 17:42:03,986][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 17:42:03,986][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 17:42:03,987][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 17:42:03,987][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 17:42:05,810][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 17:42:09,032][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.02259609720293099
	[2026-05-20 17:42:10,389][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 17:43:41,435][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2521930380411389
	[2026-05-20 17:43:43,161][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 17:43:47,929][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 17:44:50,360][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 17:44:54,809][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 17:45:10,601][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 17:45:12,251][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 17:45:14,364][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 17:45:16,613][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 17:45:17,930][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 17:45:31,706][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 17:45:31,707][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:45:31,707][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 17:45:33,391][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 17:45:35,410][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 17:45:37,363][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 17:45:38,635][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 17:45:46,011][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 17:45:46,011][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:45:46,011][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 17:45:46,012][metrics][INFO] - Evaluating model_utility
	[2026-05-20 17:45:46,012][evaluator][INFO] - Result for metric model_utility: 0.40551028982896975
	[2026-05-20 17:45:49,453][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 17:45:51,261][metrics][INFO] - Evaluating privleak
	[2026-05-20 17:45:51,261][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 17:45:51,261][evaluator][INFO] - Result for metric privleak: 48.419999990316
	[2026-05-20 17:45:52,551][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 17:45:53,769][evaluator][INFO] - Result for metric extraction_strength: 0.054570526163258214
	[2026-05-20 17:46:07,465][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 17:46:07,465][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-250/evals/TOFU_EVAL.json
	[2026-05-20 17:46:07,465][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-nf4/checkpoint-250/evals/TOFU_SUMMARY.json
	[2026-05-20 17:46:09,148][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 17:46:13,650][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 17:46:29,633][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 17:46:29,634][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4703538775169842
	[2026-05-20 17:46:29,641][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 17:46:29,641][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 17:46:29,641][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 17:46:29,641][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 17:46:31,321][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 17:46:34,548][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.022587112028544673
	[2026-05-20 17:46:35,808][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 17:48:07,373][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2535035968241837
	[2026-05-20 17:48:09,498][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 17:48:14,108][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 17:49:16,268][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 17:49:21,245][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 17:49:37,034][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 17:49:38,905][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 17:49:40,908][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 17:49:43,153][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 17:49:44,493][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 17:49:57,065][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 17:49:57,065][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:49:57,066][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 17:49:58,695][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 17:50:00,701][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 17:50:02,652][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 17:50:03,909][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 17:50:10,353][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 17:50:10,354][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 17:50:10,354][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 17:50:10,354][metrics][INFO] - Evaluating model_utility
	[2026-05-20 17:50:10,354][evaluator][INFO] - Result for metric model_utility: 0.4083376436204946
	[2026-05-20 17:50:13,361][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 17:50:15,177][metrics][INFO] - Evaluating privleak
	[2026-05-20 17:50:15,177][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 17:50:15,177][evaluator][INFO] - Result for metric privleak: 48.65499999026898
	[2026-05-20 17:50:16,477][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 17:50:17,704][evaluator][INFO] - Result for metric extraction_strength: 0.05384622411627124