Instructions to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off")
model = AutoModelForCausalLM.from_pretrained("Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off

SGLang

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off with Docker Model Runner:
```
docker model run hf.co/Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off
```

tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off / GradDiff.log

Jeesup

Model save

dd4ce77 verified 3 months ago

Raw

History Blame Contribute Delete

42 kB

	[2026-05-20 15:55:37,703][model][INFO] - Setting pad_token as eos token: <\|eot_id\|>
	[2026-05-20 15:55:45,426][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off
	[2026-05-20 15:55:46,495][trainer][INFO] - GradDiff Trainer loaded, output_dir: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off
	[2026-05-20 15:55:47,041][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 15:55:47,041][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-0/evals/TOFU_EVAL.json
	[2026-05-20 15:55:47,041][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-0/evals/TOFU_SUMMARY.json
	[2026-05-20 15:55:48,662][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 15:56:05,960][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 15:56:50,120][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 15:56:50,121][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4751472517287268
	[2026-05-20 15:56:50,127][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 15:56:50,128][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 15:56:50,128][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 15:56:50,128][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 15:56:56,071][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 15:57:03,042][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.8804745058715343
	[2026-05-20 15:57:04,697][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 15:57:57,909][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.8224148587081459
	[2026-05-20 15:57:59,692][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 15:58:09,134][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 15:58:28,644][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 15:58:38,503][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 15:59:08,107][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 15:59:09,780][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 15:59:12,800][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 15:59:15,948][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 15:59:17,184][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 15:59:24,533][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 15:59:24,534][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 15:59:24,534][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 15:59:26,178][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 15:59:28,679][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 15:59:32,841][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 15:59:34,279][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 15:59:46,329][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 15:59:46,329][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 15:59:46,329][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 15:59:46,330][metrics][INFO] - Evaluating model_utility
	[2026-05-20 15:59:46,330][evaluator][INFO] - Result for metric model_utility: 0.5986573345007462
	[2026-05-20 15:59:49,803][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 15:59:54,472][metrics][INFO] - Evaluating privleak
	[2026-05-20 15:59:54,472][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 15:59:54,472][evaluator][INFO] - Result for metric privleak: -99.33374998013325
	[2026-05-20 15:59:55,744][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 16:00:03,282][evaluator][INFO] - Result for metric extraction_strength: 0.7035078413166893
	[2026-05-20 16:01:18,929][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 16:01:18,930][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-25/evals/TOFU_EVAL.json
	[2026-05-20 16:01:18,930][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-25/evals/TOFU_SUMMARY.json
	[2026-05-20 16:01:21,125][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 16:01:26,048][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 16:01:47,301][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 16:01:47,302][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4726857599767273
	[2026-05-20 16:01:47,308][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 16:01:47,309][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 16:01:47,309][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 16:01:47,309][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 16:01:49,043][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 16:01:53,873][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.8483372511714697
	[2026-05-20 16:01:55,060][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 16:02:06,592][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.7412994035842524
	[2026-05-20 16:02:08,295][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 16:02:15,328][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 16:02:27,697][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 16:02:32,551][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 16:02:50,419][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 16:02:52,066][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 16:02:54,254][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 16:02:56,972][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 16:02:58,345][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 16:02:59,866][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 16:02:59,867][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:02:59,867][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 16:03:01,358][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 16:03:03,388][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 16:03:05,433][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 16:03:06,694][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 16:03:08,577][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 16:03:08,577][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:03:08,577][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 16:03:08,578][metrics][INFO] - Evaluating model_utility
	[2026-05-20 16:03:08,578][evaluator][INFO] - Result for metric model_utility: 0.5967583056869781
	[2026-05-20 16:03:11,100][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 16:03:12,151][metrics][INFO] - Evaluating privleak
	[2026-05-20 16:03:12,151][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 16:03:12,151][evaluator][INFO] - Result for metric privleak: -99.21374998015726
	[2026-05-20 16:03:13,789][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 16:03:15,550][evaluator][INFO] - Result for metric extraction_strength: 0.6071278425815536
	[2026-05-20 16:03:29,379][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 16:03:29,379][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-50/evals/TOFU_EVAL.json
	[2026-05-20 16:03:29,379][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-50/evals/TOFU_SUMMARY.json
	[2026-05-20 16:03:31,099][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 16:03:37,309][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 16:04:00,157][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 16:04:00,158][evaluator][INFO] - Result for metric forget_truth_ratio: 0.44223383208652406
	[2026-05-20 16:04:00,164][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 16:04:00,165][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 16:04:00,165][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 16:04:00,165][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 16:04:01,791][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 16:04:06,425][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.44728922482114286
	[2026-05-20 16:04:07,665][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 16:04:20,960][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.47367544628767905
	[2026-05-20 16:04:22,648][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 16:04:27,177][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 16:04:37,779][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 16:04:43,738][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 16:05:06,051][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 16:05:07,711][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 16:05:10,025][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 16:05:13,135][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 16:05:14,365][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 16:05:16,772][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 16:05:16,773][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:05:16,773][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 16:05:18,071][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 16:05:20,147][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 16:05:21,948][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 16:05:23,163][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 16:05:25,177][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 16:05:25,177][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:05:25,177][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 16:05:25,178][metrics][INFO] - Evaluating model_utility
	[2026-05-20 16:05:25,178][evaluator][INFO] - Result for metric model_utility: 0.5237263808305183
	[2026-05-20 16:05:27,801][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 16:05:30,191][metrics][INFO] - Evaluating privleak
	[2026-05-20 16:05:30,192][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 16:05:30,192][evaluator][INFO] - Result for metric privleak: -93.05249998138947
	[2026-05-20 16:05:31,477][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 16:05:41,688][evaluator][INFO] - Result for metric extraction_strength: 0.202558653822895
	[2026-05-20 16:05:56,197][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 16:05:56,197][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-75/evals/TOFU_EVAL.json
	[2026-05-20 16:05:56,197][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-75/evals/TOFU_SUMMARY.json
	[2026-05-20 16:05:58,048][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 16:06:02,871][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 16:06:20,532][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 16:06:20,533][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4477313937515622
	[2026-05-20 16:06:20,539][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 16:06:20,540][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 16:06:20,540][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 16:06:20,540][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 16:06:22,188][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 16:06:25,737][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.124323960904876
	[2026-05-20 16:06:26,975][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 16:06:51,461][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.3673676853660689
	[2026-05-20 16:06:53,264][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 16:06:58,085][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 16:07:08,579][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 16:07:13,367][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 16:07:30,754][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 16:07:32,500][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 16:07:35,152][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 16:07:37,515][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 16:07:38,789][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 16:07:46,115][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 16:07:46,115][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:07:46,115][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 16:07:47,785][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 16:07:49,780][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 16:07:51,774][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 16:07:53,049][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 16:07:55,620][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 16:07:55,620][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:07:55,620][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 16:07:55,621][metrics][INFO] - Evaluating model_utility
	[2026-05-20 16:07:55,621][evaluator][INFO] - Result for metric model_utility: 0.4459742865355357
	[2026-05-20 16:07:58,191][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 16:08:00,141][metrics][INFO] - Evaluating privleak
	[2026-05-20 16:08:00,141][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 16:08:00,141][evaluator][INFO] - Result for metric privleak: -65.91374998681724
	[2026-05-20 16:08:01,434][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 16:08:02,242][evaluator][INFO] - Result for metric extraction_strength: 0.10435850692698509
	[2026-05-20 16:08:16,395][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 16:08:16,395][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-100/evals/TOFU_EVAL.json
	[2026-05-20 16:08:16,395][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-100/evals/TOFU_SUMMARY.json
	[2026-05-20 16:08:18,052][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 16:08:24,674][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 16:08:45,848][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 16:08:45,849][evaluator][INFO] - Result for metric forget_truth_ratio: 0.46117877582883665
	[2026-05-20 16:08:45,855][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 16:08:45,856][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 16:08:45,856][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 16:08:45,856][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 16:08:47,765][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 16:08:50,985][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.0704621555796075
	[2026-05-20 16:08:52,249][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 16:09:04,529][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.3591667156375346
	[2026-05-20 16:09:06,165][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 16:09:11,537][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 16:09:21,609][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 16:09:26,449][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 16:09:42,973][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 16:09:44,607][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 16:09:46,475][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 16:09:49,566][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 16:09:50,800][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 16:09:53,241][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 16:09:53,241][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:09:53,242][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 16:09:54,455][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 16:09:56,558][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 16:09:59,114][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 16:10:00,351][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 16:10:04,169][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 16:10:04,169][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:10:04,169][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 16:10:04,170][metrics][INFO] - Evaluating model_utility
	[2026-05-20 16:10:04,170][evaluator][INFO] - Result for metric model_utility: 0.4619028040235479
	[2026-05-20 16:10:07,592][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 16:10:11,009][metrics][INFO] - Evaluating privleak
	[2026-05-20 16:10:11,009][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 16:10:11,010][evaluator][INFO] - Result for metric privleak: -32.99749999340052
	[2026-05-20 16:10:12,673][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 16:10:24,750][evaluator][INFO] - Result for metric extraction_strength: 0.08997003084617942
	[2026-05-20 16:10:36,275][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 16:10:36,275][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-125/evals/TOFU_EVAL.json
	[2026-05-20 16:10:36,275][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-125/evals/TOFU_SUMMARY.json
	[2026-05-20 16:10:38,003][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 16:10:43,985][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 16:11:06,490][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 16:11:06,491][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4575499820189151
	[2026-05-20 16:11:06,498][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 16:11:06,498][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 16:11:06,498][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 16:11:06,498][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 16:11:08,130][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 16:11:12,827][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.03648028554088796
	[2026-05-20 16:11:14,120][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 16:11:29,112][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.33155625358108165
	[2026-05-20 16:11:30,772][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 16:11:36,720][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 16:11:48,839][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 16:11:55,011][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 16:12:17,071][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 16:12:18,791][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 16:12:20,899][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 16:12:23,261][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 16:12:24,853][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 16:12:26,407][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 16:12:26,408][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:12:26,408][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 16:12:27,677][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 16:12:29,568][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 16:12:31,559][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 16:12:32,822][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 16:12:34,133][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 16:12:34,133][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:12:34,133][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 16:12:34,133][metrics][INFO] - Evaluating model_utility
	[2026-05-20 16:12:34,134][evaluator][INFO] - Result for metric model_utility: 0.4426036393962374
	[2026-05-20 16:12:36,645][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 16:12:38,645][metrics][INFO] - Evaluating privleak
	[2026-05-20 16:12:38,645][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 16:12:38,645][evaluator][INFO] - Result for metric privleak: -0.9187499998162404
	[2026-05-20 16:12:39,943][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 16:12:47,341][evaluator][INFO] - Result for metric extraction_strength: 0.07525683427057162
	[2026-05-20 16:12:56,753][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 16:12:56,754][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-150/evals/TOFU_EVAL.json
	[2026-05-20 16:12:56,754][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-150/evals/TOFU_SUMMARY.json
	[2026-05-20 16:12:58,529][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 16:13:03,390][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 16:13:21,103][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 16:13:21,104][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4521595589663701
	[2026-05-20 16:13:21,110][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 16:13:21,111][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 16:13:21,111][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 16:13:21,111][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 16:13:22,808][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 16:13:26,130][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.024666755490824848
	[2026-05-20 16:13:27,533][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 16:13:42,949][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.31143464039825347
	[2026-05-20 16:13:44,667][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 16:13:49,532][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 16:14:15,649][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 16:14:20,553][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 16:14:42,050][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 16:14:43,738][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 16:14:46,070][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 16:14:49,202][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 16:14:50,710][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 16:14:56,127][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 16:14:56,127][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:14:56,127][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 16:14:57,791][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 16:14:59,890][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 16:15:02,092][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 16:15:03,566][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 16:15:05,531][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 16:15:05,531][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:15:05,531][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 16:15:05,532][metrics][INFO] - Evaluating model_utility
	[2026-05-20 16:15:05,533][evaluator][INFO] - Result for metric model_utility: 0.4406448434605998
	[2026-05-20 16:15:08,315][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 16:15:10,278][metrics][INFO] - Evaluating privleak
	[2026-05-20 16:15:10,278][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 16:15:10,278][evaluator][INFO] - Result for metric privleak: 21.239999995752015
	[2026-05-20 16:15:11,697][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 16:15:17,668][evaluator][INFO] - Result for metric extraction_strength: 0.07321758725552258
	[2026-05-20 16:15:28,883][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 16:15:28,883][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-175/evals/TOFU_EVAL.json
	[2026-05-20 16:15:28,884][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-175/evals/TOFU_SUMMARY.json
	[2026-05-20 16:15:30,562][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 16:15:35,413][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 16:15:52,537][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 16:15:52,538][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4495423560072609
	[2026-05-20 16:15:52,544][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 16:15:52,545][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 16:15:52,545][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 16:15:52,545][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 16:15:54,334][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 16:15:57,858][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.018523279057384833
	[2026-05-20 16:15:59,114][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 16:16:14,407][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.29644161719491097
	[2026-05-20 16:16:16,085][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 16:16:21,946][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 16:16:38,247][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 16:16:44,399][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 16:17:06,393][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 16:17:08,002][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 16:17:10,141][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 16:17:13,239][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 16:17:14,668][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 16:17:17,946][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 16:17:17,946][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:17:17,946][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 16:17:19,200][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 16:17:21,401][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 16:17:24,032][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 16:17:25,264][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 16:17:37,878][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 16:17:37,878][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:17:37,878][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 16:17:37,879][metrics][INFO] - Evaluating model_utility
	[2026-05-20 16:17:37,879][evaluator][INFO] - Result for metric model_utility: 0.43983689264611264
	[2026-05-20 16:17:40,876][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 16:17:44,295][metrics][INFO] - Evaluating privleak
	[2026-05-20 16:17:44,295][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 16:17:44,295][evaluator][INFO] - Result for metric privleak: 37.0024999925995
	[2026-05-20 16:17:45,567][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 16:17:49,178][evaluator][INFO] - Result for metric extraction_strength: 0.06736279659239777
	[2026-05-20 16:18:04,439][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 16:18:04,440][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-200/evals/TOFU_EVAL.json
	[2026-05-20 16:18:04,440][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-200/evals/TOFU_SUMMARY.json
	[2026-05-20 16:18:06,373][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 16:18:13,079][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 16:18:35,037][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 16:18:35,038][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4530301871694543
	[2026-05-20 16:18:35,045][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 16:18:35,046][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 16:18:35,046][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 16:18:35,046][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 16:18:36,817][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 16:18:40,379][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.016063582551833183
	[2026-05-20 16:18:41,642][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 16:19:08,124][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2799413732727483
	[2026-05-20 16:19:09,874][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 16:19:14,611][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 16:19:26,572][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 16:19:31,370][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 16:19:48,771][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 16:19:50,411][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 16:19:52,330][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 16:19:55,495][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 16:19:56,809][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 16:19:59,266][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 16:19:59,266][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:19:59,266][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 16:20:00,553][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 16:20:02,403][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 16:20:04,394][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 16:20:05,647][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 16:20:07,617][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 16:20:07,617][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:20:07,617][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 16:20:07,617][metrics][INFO] - Evaluating model_utility
	[2026-05-20 16:20:07,618][evaluator][INFO] - Result for metric model_utility: 0.44245617875968024
	[2026-05-20 16:20:10,382][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 16:20:12,357][metrics][INFO] - Evaluating privleak
	[2026-05-20 16:20:12,357][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 16:20:12,358][evaluator][INFO] - Result for metric privleak: 46.123749990775245
	[2026-05-20 16:20:13,786][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 16:20:20,966][evaluator][INFO] - Result for metric extraction_strength: 0.06471707065008876
	[2026-05-20 16:20:28,834][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 16:20:28,834][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-225/evals/TOFU_EVAL.json
	[2026-05-20 16:20:28,834][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-225/evals/TOFU_SUMMARY.json
	[2026-05-20 16:20:30,527][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 16:20:56,095][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 16:21:19,759][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 16:21:19,759][evaluator][INFO] - Result for metric forget_truth_ratio: 0.45344051762195264
	[2026-05-20 16:21:19,766][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 16:21:19,767][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 16:21:19,767][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 16:21:19,767][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 16:21:21,747][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 16:21:25,362][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.014845818843423175
	[2026-05-20 16:21:26,620][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 16:21:45,015][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2708639384434357
	[2026-05-20 16:21:46,783][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 16:21:51,501][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 16:22:06,075][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 16:22:10,865][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 16:22:27,740][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 16:22:29,508][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 16:22:31,838][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 16:22:33,885][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 16:22:35,155][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 16:22:43,471][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 16:22:43,471][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:22:43,471][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 16:22:45,154][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 16:22:47,201][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 16:22:49,785][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 16:22:51,059][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 16:22:53,409][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 16:22:53,409][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:22:53,409][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 16:22:53,410][metrics][INFO] - Evaluating model_utility
	[2026-05-20 16:22:53,410][evaluator][INFO] - Result for metric model_utility: 0.43816091019725906
	[2026-05-20 16:22:56,169][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 16:22:59,734][metrics][INFO] - Evaluating privleak
	[2026-05-20 16:22:59,734][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 16:22:59,735][evaluator][INFO] - Result for metric privleak: 49.7424999900515
	[2026-05-20 16:23:01,139][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 16:23:14,332][evaluator][INFO] - Result for metric extraction_strength: 0.06327374000151627
	[2026-05-20 16:23:28,791][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 16:23:28,791][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-250/evals/TOFU_EVAL.json
	[2026-05-20 16:23:28,791][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-off/checkpoint-250/evals/TOFU_SUMMARY.json
	[2026-05-20 16:23:30,578][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 16:23:36,400][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 16:23:58,665][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 16:23:58,666][evaluator][INFO] - Result for metric forget_truth_ratio: 0.4528127155374625
	[2026-05-20 16:23:58,673][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 16:23:58,673][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 16:23:58,673][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 16:23:58,673][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 16:24:00,359][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 16:24:05,020][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.014574086861909512
	[2026-05-20 16:24:06,292][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 16:24:30,856][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.26985655364826305
	[2026-05-20 16:24:32,715][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 16:24:38,622][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 16:24:58,162][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 16:25:04,202][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 16:25:21,954][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 16:25:23,618][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 16:25:25,672][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 16:25:28,046][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 16:25:29,375][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 16:25:31,597][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 16:25:31,597][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:25:31,597][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 16:25:32,854][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 16:25:34,831][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 16:25:36,850][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 16:25:38,106][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 16:25:40,269][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 16:25:40,269][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 16:25:40,269][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 16:25:40,270][metrics][INFO] - Evaluating model_utility
	[2026-05-20 16:25:40,270][evaluator][INFO] - Result for metric model_utility: 0.43329835853565113
	[2026-05-20 16:25:43,333][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 16:25:45,422][metrics][INFO] - Evaluating privleak
	[2026-05-20 16:25:45,423][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 16:25:45,423][evaluator][INFO] - Result for metric privleak: 50.35749998992851
	[2026-05-20 16:25:46,831][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 16:25:53,687][evaluator][INFO] - Result for metric extraction_strength: 0.06258991751049445