Instructions to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4")
model = AutoModelForCausalLM.from_pretrained("Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4

SGLang

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4 with Docker Model Runner:
```
docker model run hf.co/Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4
```

tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4 / GradDiff.log

Jeesup

Model save

eb9bea5 verified 3 months ago

Raw

History Blame Contribute Delete

42 kB

	[2026-05-21 13:47:17,335][model][INFO] - Setting pad_token as eos token: <\|eot_id\|>
	[2026-05-21 13:47:22,221][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4
	[2026-05-21 13:47:23,386][trainer][INFO] - GradDiff Trainer loaded, output_dir: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4
	[2026-05-21 13:47:23,948][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-21 13:47:23,948][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-0/evals/TOFU_EVAL.json
	[2026-05-21 13:47:23,948][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-0/evals/TOFU_SUMMARY.json
	[2026-05-21 13:47:25,579][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-21 13:47:44,200][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-21 13:48:28,409][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-21 13:48:28,410][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6448834990772232
	[2026-05-21 13:48:28,416][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-21 13:48:28,417][metrics][INFO] - Evaluating forget_quality
	[2026-05-21 13:48:28,417][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-21 13:48:28,417][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-21 13:48:30,256][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-21 13:48:37,828][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.10887884757248685
	[2026-05-21 13:48:39,493][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-21 13:51:24,663][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.27890823225307393
	[2026-05-21 13:51:27,074][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-21 13:51:37,120][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-21 13:53:30,254][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-21 13:53:39,891][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-21 13:54:15,593][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-21 13:54:17,268][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-21 13:54:21,052][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-21 13:54:25,904][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-21 13:54:27,540][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-21 13:55:11,875][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-21 13:55:11,875][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 13:55:11,875][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-21 13:55:13,536][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-21 13:55:16,211][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-21 13:55:21,826][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-21 13:55:23,789][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-21 13:56:25,993][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-21 13:56:25,994][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 13:56:25,994][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-21 13:56:25,994][metrics][INFO] - Evaluating model_utility
	[2026-05-21 13:56:25,994][evaluator][INFO] - Result for metric model_utility: 0.2185219733034594
	[2026-05-21 13:56:28,952][metrics][INFO] - Evaluating mia_min_k
	[2026-05-21 13:56:32,342][metrics][INFO] - Evaluating privleak
	[2026-05-21 13:56:32,342][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-21 13:56:32,342][evaluator][INFO] - Result for metric privleak: -47.96999999040602
	[2026-05-21 13:56:33,613][metrics][INFO] - Evaluating extraction_strength
	[2026-05-21 13:56:34,761][evaluator][INFO] - Result for metric extraction_strength: 0.05119585323242784
	[2026-05-21 13:57:56,790][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-21 13:57:56,790][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-25/evals/TOFU_EVAL.json
	[2026-05-21 13:57:56,790][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-25/evals/TOFU_SUMMARY.json
	[2026-05-21 13:57:58,638][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-21 13:58:06,514][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-21 13:58:34,732][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-21 13:58:34,733][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6451312392661136
	[2026-05-21 13:58:34,739][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-21 13:58:34,740][metrics][INFO] - Evaluating forget_quality
	[2026-05-21 13:58:34,740][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-21 13:58:34,740][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-21 13:58:36,353][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-21 13:58:42,630][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.10754414513707161
	[2026-05-21 13:58:44,329][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-21 14:00:44,795][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.27375294029121017
	[2026-05-21 14:00:46,534][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-21 14:00:54,069][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-21 14:02:33,921][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-21 14:02:41,651][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-21 14:03:07,186][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-21 14:03:08,873][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-21 14:03:11,737][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-21 14:03:15,661][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-21 14:03:16,992][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-21 14:04:27,246][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-21 14:04:27,246][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:04:27,246][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-21 14:04:30,513][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-21 14:04:32,846][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-21 14:04:36,292][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-21 14:04:37,528][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-21 14:05:13,029][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-21 14:05:13,029][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:05:13,029][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-21 14:05:13,029][metrics][INFO] - Evaluating model_utility
	[2026-05-21 14:05:13,030][evaluator][INFO] - Result for metric model_utility: 0.21917274643017098
	[2026-05-21 14:05:16,156][metrics][INFO] - Evaluating mia_min_k
	[2026-05-21 14:05:21,439][metrics][INFO] - Evaluating privleak
	[2026-05-21 14:05:21,440][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-21 14:05:21,440][evaluator][INFO] - Result for metric privleak: -47.312499990537496
	[2026-05-21 14:05:23,094][metrics][INFO] - Evaluating extraction_strength
	[2026-05-21 14:05:29,627][evaluator][INFO] - Result for metric extraction_strength: 0.05040106798122812
	[2026-05-21 14:05:53,508][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-21 14:05:53,509][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-50/evals/TOFU_EVAL.json
	[2026-05-21 14:05:53,509][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-50/evals/TOFU_SUMMARY.json
	[2026-05-21 14:05:55,150][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-21 14:06:02,298][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-21 14:06:26,431][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-21 14:06:26,432][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6456153910650788
	[2026-05-21 14:06:26,438][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-21 14:06:26,439][metrics][INFO] - Evaluating forget_quality
	[2026-05-21 14:06:26,439][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-21 14:06:26,439][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-21 14:06:28,338][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-21 14:06:34,149][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.10132953665219248
	[2026-05-21 14:06:35,840][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-21 14:08:12,834][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.26386606752220315
	[2026-05-21 14:08:15,040][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-21 14:08:22,704][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-21 14:10:20,643][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-21 14:10:28,157][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-21 14:10:56,804][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-21 14:10:58,411][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-21 14:11:00,409][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-21 14:11:04,287][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-21 14:11:05,555][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-21 14:11:35,990][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-21 14:11:35,990][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:11:35,990][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-21 14:11:37,634][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-21 14:11:39,804][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-21 14:11:42,083][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-21 14:11:43,593][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-21 14:12:15,886][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-21 14:12:15,886][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:12:15,887][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-21 14:12:15,887][metrics][INFO] - Evaluating model_utility
	[2026-05-21 14:12:15,887][evaluator][INFO] - Result for metric model_utility: 0.1677994304456301
	[2026-05-21 14:12:19,143][metrics][INFO] - Evaluating mia_min_k
	[2026-05-21 14:12:20,984][metrics][INFO] - Evaluating privleak
	[2026-05-21 14:12:20,984][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-21 14:12:20,984][evaluator][INFO] - Result for metric privleak: -44.123749991175245
	[2026-05-21 14:12:22,237][metrics][INFO] - Evaluating extraction_strength
	[2026-05-21 14:12:38,878][evaluator][INFO] - Result for metric extraction_strength: 0.05035002748323244
	[2026-05-21 14:13:05,312][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-21 14:13:05,312][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-75/evals/TOFU_EVAL.json
	[2026-05-21 14:13:05,312][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-75/evals/TOFU_SUMMARY.json
	[2026-05-21 14:13:07,452][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-21 14:13:12,987][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-21 14:13:42,741][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-21 14:13:42,742][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6457981770253076
	[2026-05-21 14:13:42,748][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-21 14:13:42,749][metrics][INFO] - Evaluating forget_quality
	[2026-05-21 14:13:42,749][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-21 14:13:42,749][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-21 14:13:44,519][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-21 14:13:48,999][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.09663066714187153
	[2026-05-21 14:13:50,253][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-21 14:15:49,204][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2615118306448227
	[2026-05-21 14:15:50,869][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-21 14:15:56,547][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-21 14:17:34,075][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-21 14:17:41,825][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-21 14:18:11,180][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-21 14:18:12,847][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-21 14:18:15,573][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-21 14:18:19,695][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-21 14:18:21,024][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-21 14:18:56,057][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-21 14:18:56,058][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:18:56,058][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-21 14:18:57,711][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-21 14:19:00,393][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-21 14:19:04,119][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-21 14:19:05,432][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-21 14:19:35,785][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-21 14:19:35,785][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:19:35,785][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-21 14:19:35,786][metrics][INFO] - Evaluating model_utility
	[2026-05-21 14:19:35,786][evaluator][INFO] - Result for metric model_utility: 0.18910328102652976
	[2026-05-21 14:19:38,883][metrics][INFO] - Evaluating mia_min_k
	[2026-05-21 14:19:44,187][metrics][INFO] - Evaluating privleak
	[2026-05-21 14:19:44,187][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-21 14:19:44,187][evaluator][INFO] - Result for metric privleak: -41.89999999162
	[2026-05-21 14:19:46,041][metrics][INFO] - Evaluating extraction_strength
	[2026-05-21 14:20:05,596][evaluator][INFO] - Result for metric extraction_strength: 0.050467624401651194
	[2026-05-21 14:20:27,142][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-21 14:20:27,142][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-100/evals/TOFU_EVAL.json
	[2026-05-21 14:20:27,142][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-100/evals/TOFU_SUMMARY.json
	[2026-05-21 14:20:28,974][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-21 14:20:36,586][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-21 14:21:03,596][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-21 14:21:03,596][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6467288069780801
	[2026-05-21 14:21:03,603][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-21 14:21:03,604][metrics][INFO] - Evaluating forget_quality
	[2026-05-21 14:21:03,604][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-21 14:21:03,604][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-21 14:21:05,331][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-21 14:21:11,567][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.09440476755378768
	[2026-05-21 14:21:13,257][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-21 14:22:40,611][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.26794907922767686
	[2026-05-21 14:22:42,348][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-21 14:22:47,675][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-21 14:24:49,536][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-21 14:24:56,991][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-21 14:25:25,971][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-21 14:25:27,637][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-21 14:25:29,605][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-21 14:25:33,860][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-21 14:25:35,427][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-21 14:26:06,135][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-21 14:26:06,135][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:26:06,135][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-21 14:26:07,754][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-21 14:26:10,037][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-21 14:26:12,526][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-21 14:26:14,323][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-21 14:26:47,894][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-21 14:26:47,894][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:26:47,894][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-21 14:26:47,895][metrics][INFO] - Evaluating model_utility
	[2026-05-21 14:26:47,895][evaluator][INFO] - Result for metric model_utility: 0.1762170377960403
	[2026-05-21 14:26:51,087][metrics][INFO] - Evaluating mia_min_k
	[2026-05-21 14:26:52,963][metrics][INFO] - Evaluating privleak
	[2026-05-21 14:26:52,963][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-21 14:26:52,963][evaluator][INFO] - Result for metric privleak: -40.604999991879005
	[2026-05-21 14:26:54,229][metrics][INFO] - Evaluating extraction_strength
	[2026-05-21 14:26:55,724][evaluator][INFO] - Result for metric extraction_strength: 0.050016065742406024
	[2026-05-21 14:27:10,285][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-21 14:27:10,285][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-125/evals/TOFU_EVAL.json
	[2026-05-21 14:27:10,285][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-125/evals/TOFU_SUMMARY.json
	[2026-05-21 14:27:12,045][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-21 14:27:19,880][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-21 14:27:39,193][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-21 14:27:39,194][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6461302936150455
	[2026-05-21 14:27:39,200][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-21 14:27:39,201][metrics][INFO] - Evaluating forget_quality
	[2026-05-21 14:27:39,201][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-21 14:27:39,201][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-21 14:27:40,891][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-21 14:27:47,053][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.09275924117304385
	[2026-05-21 14:27:48,876][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-21 14:29:45,984][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2681758525730431
	[2026-05-21 14:29:47,798][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-21 14:29:55,241][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-21 14:31:47,198][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-21 14:31:55,024][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-21 14:32:21,016][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-21 14:32:22,735][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-21 14:32:24,801][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-21 14:32:28,258][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-21 14:32:29,679][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-21 14:32:58,416][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-21 14:32:58,417][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:32:58,417][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-21 14:33:00,141][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-21 14:33:02,678][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-21 14:33:06,037][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-21 14:33:07,637][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-21 14:33:32,723][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-21 14:33:32,723][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:33:32,723][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-21 14:33:32,724][metrics][INFO] - Evaluating model_utility
	[2026-05-21 14:33:32,724][evaluator][INFO] - Result for metric model_utility: 0.1993004657364793
	[2026-05-21 14:33:35,666][metrics][INFO] - Evaluating mia_min_k
	[2026-05-21 14:33:41,152][metrics][INFO] - Evaluating privleak
	[2026-05-21 14:33:41,152][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-21 14:33:41,152][evaluator][INFO] - Result for metric privleak: -39.639999992071985
	[2026-05-21 14:33:42,964][metrics][INFO] - Evaluating extraction_strength
	[2026-05-21 14:33:52,966][evaluator][INFO] - Result for metric extraction_strength: 0.04982500172512249
	[2026-05-21 14:34:13,860][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-21 14:34:13,861][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-150/evals/TOFU_EVAL.json
	[2026-05-21 14:34:13,861][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-150/evals/TOFU_SUMMARY.json
	[2026-05-21 14:34:15,582][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-21 14:34:23,529][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-21 14:34:45,209][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-21 14:34:45,209][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6461448026916605
	[2026-05-21 14:34:45,216][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-21 14:34:45,216][metrics][INFO] - Evaluating forget_quality
	[2026-05-21 14:34:45,216][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-21 14:34:45,217][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-21 14:34:47,100][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-21 14:34:52,959][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.09229127565224189
	[2026-05-21 14:34:54,738][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-21 14:36:50,088][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2688430836469951
	[2026-05-21 14:36:52,001][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-21 14:37:00,160][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-21 14:38:27,732][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-21 14:38:35,421][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-21 14:39:02,644][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-21 14:39:04,643][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-21 14:39:07,392][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-21 14:39:11,422][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-21 14:39:12,807][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-21 14:39:41,142][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-21 14:39:41,142][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:39:41,142][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-21 14:39:43,199][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-21 14:39:45,720][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-21 14:39:49,378][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-21 14:39:50,652][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-21 14:40:16,575][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-21 14:40:16,575][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:40:16,575][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-21 14:40:16,575][metrics][INFO] - Evaluating model_utility
	[2026-05-21 14:40:16,576][evaluator][INFO] - Result for metric model_utility: 0.1756593488918497
	[2026-05-21 14:40:20,015][metrics][INFO] - Evaluating mia_min_k
	[2026-05-21 14:40:25,265][metrics][INFO] - Evaluating privleak
	[2026-05-21 14:40:25,265][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-21 14:40:25,265][evaluator][INFO] - Result for metric privleak: -39.33999999213199
	[2026-05-21 14:40:26,926][metrics][INFO] - Evaluating extraction_strength
	[2026-05-21 14:40:45,430][evaluator][INFO] - Result for metric extraction_strength: 0.04987333750654246
	[2026-05-21 14:41:06,558][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-21 14:41:06,558][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-175/evals/TOFU_EVAL.json
	[2026-05-21 14:41:06,558][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-175/evals/TOFU_SUMMARY.json
	[2026-05-21 14:41:08,399][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-21 14:41:16,023][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-21 14:41:42,613][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-21 14:41:42,614][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6467353839118344
	[2026-05-21 14:41:42,620][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-21 14:41:42,621][metrics][INFO] - Evaluating forget_quality
	[2026-05-21 14:41:42,621][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-21 14:41:42,621][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-21 14:41:44,313][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-21 14:41:50,563][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.09209337599691934
	[2026-05-21 14:41:52,208][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-21 14:43:17,399][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.26517273879630093
	[2026-05-21 14:43:19,073][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-21 14:43:23,338][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-21 14:44:46,144][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-21 14:44:52,044][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-21 14:45:20,907][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-21 14:45:22,576][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-21 14:45:25,233][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-21 14:45:29,274][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-21 14:45:30,629][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-21 14:45:58,705][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-21 14:45:58,705][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:45:58,705][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-21 14:46:00,371][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-21 14:46:02,710][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-21 14:46:06,159][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-21 14:46:07,775][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-21 14:46:35,249][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-21 14:46:35,250][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:46:35,250][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-21 14:46:35,250][metrics][INFO] - Evaluating model_utility
	[2026-05-21 14:46:35,251][evaluator][INFO] - Result for metric model_utility: 0.19728150016508927
	[2026-05-21 14:46:38,290][metrics][INFO] - Evaluating mia_min_k
	[2026-05-21 14:46:43,839][metrics][INFO] - Evaluating privleak
	[2026-05-21 14:46:43,839][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-21 14:46:43,839][evaluator][INFO] - Result for metric privleak: -39.229999992153985
	[2026-05-21 14:46:45,552][metrics][INFO] - Evaluating extraction_strength
	[2026-05-21 14:46:54,375][evaluator][INFO] - Result for metric extraction_strength: 0.04923588255297723
	[2026-05-21 14:47:14,636][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-21 14:47:14,637][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-200/evals/TOFU_EVAL.json
	[2026-05-21 14:47:14,637][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-200/evals/TOFU_SUMMARY.json
	[2026-05-21 14:47:16,289][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-21 14:47:23,849][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-21 14:47:44,212][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-21 14:47:44,213][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6457676093391518
	[2026-05-21 14:47:44,220][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-21 14:47:44,220][metrics][INFO] - Evaluating forget_quality
	[2026-05-21 14:47:44,220][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-21 14:47:44,220][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-21 14:47:45,867][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-21 14:47:49,053][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.09191316528769676
	[2026-05-21 14:47:50,488][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-21 14:48:52,523][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2677881241976711
	[2026-05-21 14:48:54,710][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-21 14:49:02,610][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-21 14:50:27,278][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-21 14:50:31,670][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-21 14:50:48,885][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-21 14:50:50,729][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-21 14:50:52,874][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-21 14:50:55,268][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-21 14:50:56,544][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-21 14:51:13,645][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-21 14:51:13,645][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:51:13,645][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-21 14:51:15,290][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-21 14:51:17,170][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-21 14:51:19,242][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-21 14:51:20,515][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-21 14:51:34,267][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-21 14:51:34,267][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:51:34,267][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-21 14:51:34,268][metrics][INFO] - Evaluating model_utility
	[2026-05-21 14:51:34,268][evaluator][INFO] - Result for metric model_utility: 0.1881585011308466
	[2026-05-21 14:51:37,392][metrics][INFO] - Evaluating mia_min_k
	[2026-05-21 14:51:39,618][metrics][INFO] - Evaluating privleak
	[2026-05-21 14:51:39,619][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-21 14:51:39,619][evaluator][INFO] - Result for metric privleak: -39.07624999218476
	[2026-05-21 14:51:40,926][metrics][INFO] - Evaluating extraction_strength
	[2026-05-21 14:51:44,606][evaluator][INFO] - Result for metric extraction_strength: 0.04998827057654465
	[2026-05-21 14:51:54,904][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-21 14:51:54,904][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-225/evals/TOFU_EVAL.json
	[2026-05-21 14:51:54,904][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-225/evals/TOFU_SUMMARY.json
	[2026-05-21 14:51:56,542][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-21 14:52:01,351][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-21 14:52:18,825][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-21 14:52:18,825][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6457553564897875
	[2026-05-21 14:52:18,832][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-21 14:52:18,833][metrics][INFO] - Evaluating forget_quality
	[2026-05-21 14:52:18,833][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-21 14:52:18,833][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-21 14:52:20,641][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-21 14:52:24,136][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.09176048776658717
	[2026-05-21 14:52:25,392][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-21 14:53:35,432][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2594295064852817
	[2026-05-21 14:53:37,211][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-21 14:53:42,056][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-21 14:54:50,475][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-21 14:54:55,426][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-21 14:55:12,406][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-21 14:55:14,200][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-21 14:55:16,374][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-21 14:55:18,766][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-21 14:55:20,085][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-21 14:55:37,685][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-21 14:55:37,685][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:55:37,685][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-21 14:55:39,369][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-21 14:55:41,256][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-21 14:55:43,488][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-21 14:55:44,732][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-21 14:56:06,082][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-21 14:56:06,083][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:56:06,083][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-21 14:56:06,083][metrics][INFO] - Evaluating model_utility
	[2026-05-21 14:56:06,084][evaluator][INFO] - Result for metric model_utility: 0.18418976368312862
	[2026-05-21 14:56:09,499][metrics][INFO] - Evaluating mia_min_k
	[2026-05-21 14:56:17,100][metrics][INFO] - Evaluating privleak
	[2026-05-21 14:56:17,100][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-21 14:56:17,101][evaluator][INFO] - Result for metric privleak: -38.90374999221925
	[2026-05-21 14:56:19,509][metrics][INFO] - Evaluating extraction_strength
	[2026-05-21 14:56:20,577][evaluator][INFO] - Result for metric extraction_strength: 0.05017726968547463
	[2026-05-21 14:56:31,061][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-21 14:56:31,061][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-250/evals/TOFU_EVAL.json
	[2026-05-21 14:56:31,061][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4/checkpoint-250/evals/TOFU_SUMMARY.json
	[2026-05-21 14:56:32,804][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-21 14:56:37,239][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-21 14:56:52,454][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-21 14:56:52,455][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6470660187337589
	[2026-05-21 14:56:52,461][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-21 14:56:52,462][metrics][INFO] - Evaluating forget_quality
	[2026-05-21 14:56:52,462][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-21 14:56:52,462][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-21 14:56:54,239][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-21 14:56:57,306][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.09160510779242031
	[2026-05-21 14:56:59,120][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-21 14:57:57,115][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.26243194865004194
	[2026-05-21 14:57:58,822][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-21 14:58:03,217][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-21 14:59:05,524][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-21 14:59:10,130][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-21 14:59:25,131][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-21 14:59:26,791][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-21 14:59:28,757][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-21 14:59:30,847][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-21 14:59:32,098][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-21 14:59:47,190][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-21 14:59:47,190][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 14:59:47,190][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-21 14:59:48,806][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-21 14:59:50,723][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-21 14:59:52,523][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-21 14:59:53,777][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-21 15:00:08,178][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-21 15:00:08,179][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-21 15:00:08,179][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-21 15:00:08,179][metrics][INFO] - Evaluating model_utility
	[2026-05-21 15:00:08,180][evaluator][INFO] - Result for metric model_utility: 0.19116560978103825
	[2026-05-21 15:00:11,306][metrics][INFO] - Evaluating mia_min_k
	[2026-05-21 15:00:12,801][metrics][INFO] - Evaluating privleak
	[2026-05-21 15:00:12,801][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-21 15:00:12,801][evaluator][INFO] - Result for metric privleak: -38.859999992228005
	[2026-05-21 15:00:14,090][metrics][INFO] - Evaluating extraction_strength
	[2026-05-21 15:00:15,151][evaluator][INFO] - Result for metric extraction_strength: 0.04984213782860365