Instructions to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn")
model = AutoModelForCausalLM.from_pretrained("Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn

SGLang

How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn with Docker Model Runner:
```
docker model run hf.co/Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn
```

tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn / GradDiff.log

Jeesup

Model save

c481a0e verified 3 months ago

Raw

History Blame Contribute Delete

42.1 kB

	[2026-05-20 00:21:27,541][model][INFO] - Setting pad_token as eos token: <\|eot_id\|>
	[2026-05-20 00:21:31,468][evaluator][INFO] - Evaluations stored in the experiment directory: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn
	[2026-05-20 00:21:32,595][trainer][INFO] - GradDiff Trainer loaded, output_dir: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn
	[2026-05-20 00:21:33,073][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 00:21:33,073][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-0/evals/TOFU_EVAL.json
	[2026-05-20 00:21:33,073][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-0/evals/TOFU_SUMMARY.json
	[2026-05-20 00:21:34,584][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 00:21:54,807][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 00:22:28,857][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 00:22:28,858][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6149281793680514
	[2026-05-20 00:22:28,864][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 00:22:28,864][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 00:22:28,865][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 00:22:28,865][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 00:22:30,530][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 00:22:35,191][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.18831624545156955
	[2026-05-20 00:22:36,444][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 00:24:18,774][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.3397309745491944
	[2026-05-20 00:24:20,502][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 00:24:27,627][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 00:24:58,185][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 00:25:04,985][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 00:25:26,043][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 00:25:27,695][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 00:25:30,515][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 00:25:33,246][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 00:25:34,521][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 00:25:47,047][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 00:25:47,048][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:25:47,048][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 00:25:48,909][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 00:25:51,078][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 00:25:54,918][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 00:25:56,179][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 00:26:15,297][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 00:26:15,297][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:26:15,297][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 00:26:15,297][metrics][INFO] - Evaluating model_utility
	[2026-05-20 00:26:15,298][evaluator][INFO] - Result for metric model_utility: 0.28610305991303836
	[2026-05-20 00:26:18,262][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 00:26:21,321][metrics][INFO] - Evaluating privleak
	[2026-05-20 00:26:21,321][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 00:26:21,321][evaluator][INFO] - Result for metric privleak: -68.40374998631926
	[2026-05-20 00:26:22,595][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 00:26:23,569][evaluator][INFO] - Result for metric extraction_strength: 0.07011714084641235
	[2026-05-20 00:27:32,512][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 00:27:32,512][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-25/evals/TOFU_EVAL.json
	[2026-05-20 00:27:32,512][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-25/evals/TOFU_SUMMARY.json
	[2026-05-20 00:27:42,427][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 00:27:47,145][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 00:28:01,957][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 00:28:01,958][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6154719265691011
	[2026-05-20 00:28:01,964][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 00:28:01,965][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 00:28:01,965][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 00:28:01,965][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 00:28:07,735][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 00:28:10,721][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.15049896295182408
	[2026-05-20 00:28:11,991][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 00:28:52,012][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.32735716497605594
	[2026-05-20 00:28:57,705][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 00:29:01,904][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 00:30:01,418][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 00:30:05,670][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 00:30:20,297][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 00:30:22,833][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 00:30:24,809][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 00:30:26,840][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 00:30:28,115][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 00:30:33,779][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 00:30:33,780][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:30:33,780][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 00:30:35,515][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 00:30:37,290][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 00:30:39,030][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 00:30:40,388][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 00:30:44,038][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 00:30:44,039][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:30:44,039][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 00:30:44,039][metrics][INFO] - Evaluating model_utility
	[2026-05-20 00:30:44,040][evaluator][INFO] - Result for metric model_utility: 0.29092844100321236
	[2026-05-20 00:30:46,676][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 00:30:48,018][metrics][INFO] - Evaluating privleak
	[2026-05-20 00:30:48,018][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 00:30:48,018][evaluator][INFO] - Result for metric privleak: -56.08999998878197
	[2026-05-20 00:30:49,388][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 00:30:50,357][evaluator][INFO] - Result for metric extraction_strength: 0.06666765781877576
	[2026-05-20 00:31:00,692][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 00:31:00,692][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-50/evals/TOFU_EVAL.json
	[2026-05-20 00:31:00,692][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-50/evals/TOFU_SUMMARY.json
	[2026-05-20 00:31:02,388][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 00:31:06,655][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 00:31:21,460][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 00:31:21,461][evaluator][INFO] - Result for metric forget_truth_ratio: 0.6012843458375424
	[2026-05-20 00:31:21,467][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 00:31:21,468][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 00:31:21,468][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 00:31:21,468][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 00:31:23,199][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 00:31:26,192][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.06847024955088273
	[2026-05-20 00:31:27,712][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 00:32:11,791][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.2749696515532796
	[2026-05-20 00:32:13,440][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 00:32:17,633][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 00:33:03,280][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 00:33:07,540][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 00:33:22,184][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 00:33:27,949][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 00:33:29,976][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 00:33:32,011][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 00:33:33,661][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 00:33:43,332][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 00:33:43,332][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:33:43,332][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 00:33:48,968][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 00:33:50,838][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 00:33:52,579][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 00:33:53,897][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 00:34:06,914][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 00:34:06,915][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:34:06,915][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 00:34:06,915][metrics][INFO] - Evaluating model_utility
	[2026-05-20 00:34:06,916][evaluator][INFO] - Result for metric model_utility: 0.24616334355717534
	[2026-05-20 00:34:14,024][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 00:34:15,366][metrics][INFO] - Evaluating privleak
	[2026-05-20 00:34:15,366][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 00:34:15,366][evaluator][INFO] - Result for metric privleak: -15.393749996921255
	[2026-05-20 00:34:16,616][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 00:34:17,592][evaluator][INFO] - Result for metric extraction_strength: 0.0528486934683597
	[2026-05-20 00:34:28,837][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 00:34:28,837][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-75/evals/TOFU_EVAL.json
	[2026-05-20 00:34:28,837][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-75/evals/TOFU_SUMMARY.json
	[2026-05-20 00:34:30,611][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 00:34:34,932][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 00:34:49,772][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 00:34:49,772][evaluator][INFO] - Result for metric forget_truth_ratio: 0.586691674567198
	[2026-05-20 00:34:49,778][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 00:34:49,779][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 00:34:49,779][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 00:34:49,779][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 00:34:55,442][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 00:34:58,421][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.0213226443800022
	[2026-05-20 00:34:59,672][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 00:35:47,897][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.20629174453631954
	[2026-05-20 00:35:49,653][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 00:35:53,829][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 00:36:45,767][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 00:36:50,002][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 00:37:04,625][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 00:37:06,271][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 00:37:08,224][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 00:37:10,255][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 00:37:11,514][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 00:37:27,593][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 00:37:27,593][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:37:27,593][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 00:37:33,256][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 00:37:35,078][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 00:37:36,813][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 00:37:38,121][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 00:38:11,369][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 00:38:11,369][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:38:11,369][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 00:38:11,370][metrics][INFO] - Evaluating model_utility
	[2026-05-20 00:38:11,370][evaluator][INFO] - Result for metric model_utility: 0.11850693301347763
	[2026-05-20 00:38:14,340][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 00:38:15,703][metrics][INFO] - Evaluating privleak
	[2026-05-20 00:38:15,703][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 00:38:15,704][evaluator][INFO] - Result for metric privleak: 16.564999996686986
	[2026-05-20 00:38:16,983][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 00:38:17,957][evaluator][INFO] - Result for metric extraction_strength: 0.0410300307115429
	[2026-05-20 00:38:28,273][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 00:38:28,274][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-100/evals/TOFU_EVAL.json
	[2026-05-20 00:38:28,274][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-100/evals/TOFU_SUMMARY.json
	[2026-05-20 00:38:29,944][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 00:38:34,369][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 00:38:49,178][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 00:38:49,179][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5701976340762325
	[2026-05-20 00:38:49,185][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 00:38:49,186][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 00:38:49,186][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 00:38:49,186][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 00:38:50,822][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 00:38:53,805][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.013325568062123238
	[2026-05-20 00:38:55,164][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 00:39:44,062][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.1914777083161352
	[2026-05-20 00:39:45,701][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 00:39:50,043][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 00:40:38,911][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 00:40:43,111][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 00:40:57,751][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 00:40:59,510][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 00:41:01,567][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 00:41:03,614][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 00:41:05,013][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 00:41:16,912][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 00:41:16,912][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:41:16,912][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 00:41:18,643][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 00:41:20,438][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 00:41:22,176][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 00:41:23,435][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 00:41:37,980][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 00:41:37,980][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:41:37,980][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 00:41:37,980][metrics][INFO] - Evaluating model_utility
	[2026-05-20 00:41:37,981][evaluator][INFO] - Result for metric model_utility: 0.1662268619595417
	[2026-05-20 00:41:40,948][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 00:41:42,293][metrics][INFO] - Evaluating privleak
	[2026-05-20 00:41:42,293][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 00:41:42,293][evaluator][INFO] - Result for metric privleak: 34.57374999308525
	[2026-05-20 00:41:43,549][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 00:41:44,522][evaluator][INFO] - Result for metric extraction_strength: 0.039446490025339405
	[2026-05-20 00:41:51,631][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 00:41:51,631][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-125/evals/TOFU_EVAL.json
	[2026-05-20 00:41:51,631][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-125/evals/TOFU_SUMMARY.json
	[2026-05-20 00:41:53,256][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 00:41:57,511][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 00:42:12,334][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 00:42:12,335][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5543482328298487
	[2026-05-20 00:42:12,341][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 00:42:12,341][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 00:42:12,342][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 00:42:12,342][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 00:42:18,110][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 00:42:21,085][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.008377231558397966
	[2026-05-20 00:42:22,336][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 00:43:10,794][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.17879309891317427
	[2026-05-20 00:43:16,654][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 00:43:20,950][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 00:44:11,001][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 00:44:15,247][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 00:44:29,907][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 00:44:31,531][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 00:44:33,473][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 00:44:35,501][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 00:44:36,724][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 00:45:26,430][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 00:45:26,431][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:45:26,431][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 00:45:32,137][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 00:45:33,956][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 00:45:35,693][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 00:45:36,912][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 00:45:51,712][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 00:45:51,713][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:45:51,713][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 00:45:51,713][metrics][INFO] - Evaluating model_utility
	[2026-05-20 00:45:51,714][evaluator][INFO] - Result for metric model_utility: 0.15101393308586722
	[2026-05-20 00:45:54,675][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 00:45:56,022][metrics][INFO] - Evaluating privleak
	[2026-05-20 00:45:56,022][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 00:45:56,023][evaluator][INFO] - Result for metric privleak: 46.179999990763996
	[2026-05-20 00:45:57,926][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 00:45:58,908][evaluator][INFO] - Result for metric extraction_strength: 0.037897375752444645
	[2026-05-20 00:46:07,637][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 00:46:07,637][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-150/evals/TOFU_EVAL.json
	[2026-05-20 00:46:07,637][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-150/evals/TOFU_SUMMARY.json
	[2026-05-20 00:46:09,537][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 00:46:13,821][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 00:46:28,669][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 00:46:28,670][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5439175830763036
	[2026-05-20 00:46:28,676][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 00:46:28,677][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 00:46:28,677][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 00:46:28,677][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 00:46:34,608][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 00:46:37,596][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.006310039218440089
	[2026-05-20 00:46:38,857][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 00:47:27,323][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.1679516805942966
	[2026-05-20 00:47:33,160][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 00:47:37,335][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 00:48:27,299][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 00:48:31,589][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 00:48:46,245][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 00:48:47,955][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 00:48:49,972][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 00:48:52,001][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 00:48:53,301][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 00:49:06,974][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 00:49:06,975][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:49:06,975][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 00:49:08,657][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 00:49:10,717][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 00:49:12,454][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 00:49:13,694][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 00:49:28,494][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 00:49:28,494][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:49:28,494][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 00:49:28,494][metrics][INFO] - Evaluating model_utility
	[2026-05-20 00:49:28,495][evaluator][INFO] - Result for metric model_utility: 0.12610447033729202
	[2026-05-20 00:49:31,450][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 00:49:32,808][metrics][INFO] - Evaluating privleak
	[2026-05-20 00:49:32,808][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 00:49:32,808][evaluator][INFO] - Result for metric privleak: 53.31124998933776
	[2026-05-20 00:49:34,136][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 00:49:35,107][evaluator][INFO] - Result for metric extraction_strength: 0.03742433490440379
	[2026-05-20 00:49:45,268][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 00:49:45,268][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-175/evals/TOFU_EVAL.json
	[2026-05-20 00:49:45,268][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-175/evals/TOFU_SUMMARY.json
	[2026-05-20 00:49:46,972][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 00:49:51,400][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 00:50:06,227][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 00:50:06,227][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5385811781169093
	[2026-05-20 00:50:06,234][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 00:50:06,234][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 00:50:06,234][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 00:50:06,234][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 00:50:07,904][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 00:50:10,898][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.005299835318573436
	[2026-05-20 00:50:12,174][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 00:51:00,624][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.16196447751426213
	[2026-05-20 00:51:02,314][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 00:51:06,482][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 00:51:56,412][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 00:52:00,671][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 00:52:15,305][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 00:52:17,774][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 00:52:19,706][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 00:52:21,732][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 00:52:24,046][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 00:52:38,718][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 00:52:38,719][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:52:38,719][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 00:52:40,406][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 00:52:42,751][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 00:52:44,494][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 00:52:45,765][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 00:53:00,565][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 00:53:00,565][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:53:00,565][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 00:53:00,566][metrics][INFO] - Evaluating model_utility
	[2026-05-20 00:53:00,566][evaluator][INFO] - Result for metric model_utility: 0.11788141506748863
	[2026-05-20 00:53:03,610][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 00:53:04,959][metrics][INFO] - Evaluating privleak
	[2026-05-20 00:53:04,959][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 00:53:04,959][evaluator][INFO] - Result for metric privleak: 56.85624998862876
	[2026-05-20 00:53:06,263][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 00:53:07,252][evaluator][INFO] - Result for metric extraction_strength: 0.03737202744550497
	[2026-05-20 00:53:19,699][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 00:53:19,700][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-200/evals/TOFU_EVAL.json
	[2026-05-20 00:53:19,700][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-200/evals/TOFU_SUMMARY.json
	[2026-05-20 00:53:21,449][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 00:53:25,715][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 00:53:40,514][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 00:53:40,514][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5372921729465798
	[2026-05-20 00:53:40,521][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 00:53:40,521][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 00:53:40,521][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 00:53:40,521][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 00:53:42,269][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 00:53:45,271][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.004717631993025311
	[2026-05-20 00:53:46,540][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 00:54:35,024][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.16055182158423748
	[2026-05-20 00:54:36,836][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 00:54:41,039][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 00:55:31,110][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 00:55:35,495][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 00:55:50,103][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 00:55:51,829][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 00:55:53,844][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 00:55:55,871][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 00:55:57,126][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 00:56:08,931][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 00:56:08,931][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:56:08,932][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 00:56:10,716][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 00:56:12,534][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 00:56:14,277][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 00:56:15,513][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 00:56:29,329][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 00:56:29,330][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:56:29,330][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 00:56:29,330][metrics][INFO] - Evaluating model_utility
	[2026-05-20 00:56:29,331][evaluator][INFO] - Result for metric model_utility: 0.13369341509818602
	[2026-05-20 00:56:32,418][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 00:56:33,757][metrics][INFO] - Evaluating privleak
	[2026-05-20 00:56:33,757][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 00:56:33,757][evaluator][INFO] - Result for metric privleak: 58.69124998826174
	[2026-05-20 00:56:35,090][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 00:56:36,063][evaluator][INFO] - Result for metric extraction_strength: 0.03703651421187144
	[2026-05-20 00:56:43,200][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 00:56:43,200][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-225/evals/TOFU_EVAL.json
	[2026-05-20 00:56:43,200][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-225/evals/TOFU_SUMMARY.json
	[2026-05-20 00:56:45,238][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 00:56:49,498][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 00:57:04,322][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 00:57:04,323][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5372371952871265
	[2026-05-20 00:57:04,329][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 00:57:04,330][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 00:57:04,330][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 00:57:04,330][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 00:57:06,016][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 00:57:08,997][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.004615904050986827
	[2026-05-20 00:57:10,286][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 00:57:58,796][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.15915544427875933
	[2026-05-20 00:58:00,426][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 00:58:04,637][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 00:58:54,720][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 00:58:58,948][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 00:59:13,576][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 00:59:15,307][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 00:59:17,262][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 00:59:19,292][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 00:59:20,622][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 00:59:35,294][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 00:59:35,294][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:59:35,294][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 00:59:37,596][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 00:59:39,373][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 00:59:41,110][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 00:59:42,356][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 00:59:57,156][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 00:59:57,156][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 00:59:57,156][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 00:59:57,157][metrics][INFO] - Evaluating model_utility
	[2026-05-20 00:59:57,157][evaluator][INFO] - Result for metric model_utility: 0.11225851330290189
	[2026-05-20 01:00:00,129][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 01:00:01,480][metrics][INFO] - Evaluating privleak
	[2026-05-20 01:00:01,481][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 01:00:01,481][evaluator][INFO] - Result for metric privleak: 59.0974999881805
	[2026-05-20 01:00:02,753][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 01:00:03,733][evaluator][INFO] - Result for metric extraction_strength: 0.03703651421187144
	[2026-05-20 01:00:12,916][evaluator][INFO] - *** Running TOFU evaluation suite ***
	[2026-05-20 01:00:12,916][evaluator][INFO] - Fine-grained evaluations will be saved to: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-250/evals/TOFU_EVAL.json
	[2026-05-20 01:00:12,916][evaluator][INFO] - Aggregated evaluations will be summarised in: saves/unlearn/qat-baseline/tofu_Llama-3.2-1B-Instruct_forget10_GradDiff_qat-int4_ffn/checkpoint-250/evals/TOFU_SUMMARY.json
	[2026-05-20 01:00:14,595][metrics][INFO] - Evaluating forget_Q_A_PARA_Prob
	[2026-05-20 01:00:18,971][metrics][INFO] - Evaluating forget_Q_A_PERT_Prob
	[2026-05-20 01:00:33,809][metrics][INFO] - Evaluating forget_truth_ratio
	[2026-05-20 01:00:33,810][evaluator][INFO] - Result for metric forget_truth_ratio: 0.5403969006559984
	[2026-05-20 01:00:33,816][metrics][INFO] - Skipping forget_quality's precompute forget_truth_ratio, already evaluated.
	[2026-05-20 01:00:33,816][metrics][INFO] - Evaluating forget_quality
	[2026-05-20 01:00:33,817][metrics][WARNING] - retain_model_logs not provided in reference_logs, setting forget_quality to None
	[2026-05-20 01:00:33,817][evaluator][INFO] - Result for metric forget_quality: None
	[2026-05-20 01:00:35,603][metrics][INFO] - Evaluating forget_Q_A_Prob
	[2026-05-20 01:00:38,590][evaluator][INFO] - Result for metric forget_Q_A_Prob: 0.0045827382961066655
	[2026-05-20 01:00:39,945][metrics][INFO] - Evaluating forget_Q_A_ROUGE
	[2026-05-20 01:01:28,444][evaluator][INFO] - Result for metric forget_Q_A_ROUGE: 0.16535668050190233
	[2026-05-20 01:01:30,172][metrics][INFO] - Evaluating retain_Q_A_Prob
	[2026-05-20 01:01:34,368][metrics][INFO] - Evaluating retain_Q_A_ROUGE
	[2026-05-20 01:02:24,394][metrics][INFO] - Evaluating retain_Q_A_PARA_Prob
	[2026-05-20 01:02:28,654][metrics][INFO] - Evaluating retain_Q_A_PERT_Prob
	[2026-05-20 01:02:43,284][metrics][INFO] - Evaluating retain_Truth_Ratio
	[2026-05-20 01:02:45,328][metrics][INFO] - Evaluating ra_Q_A_Prob
	[2026-05-20 01:02:47,303][metrics][INFO] - Evaluating ra_Q_A_PERT_Prob
	[2026-05-20 01:02:49,339][metrics][INFO] - Evaluating ra_Q_A_Prob_normalised
	[2026-05-20 01:02:50,585][metrics][INFO] - Evaluating ra_Q_A_ROUGE
	[2026-05-20 01:03:05,267][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_Prob, already evaluated.
	[2026-05-20 01:03:05,267][metrics][INFO] - Skipping ra_Truth_Ratio's precompute ra_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 01:03:05,267][metrics][INFO] - Evaluating ra_Truth_Ratio
	[2026-05-20 01:03:06,951][metrics][INFO] - Evaluating wf_Q_A_Prob
	[2026-05-20 01:03:08,845][metrics][INFO] - Evaluating wf_Q_A_PERT_Prob
	[2026-05-20 01:03:10,587][metrics][INFO] - Evaluating wf_Q_A_Prob_normalised
	[2026-05-20 01:03:11,922][metrics][INFO] - Evaluating wf_Q_A_ROUGE
	[2026-05-20 01:03:25,014][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_Prob, already evaluated.
	[2026-05-20 01:03:25,014][metrics][INFO] - Skipping wf_Truth_Ratio's precompute wf_Q_A_PERT_Prob, already evaluated.
	[2026-05-20 01:03:25,014][metrics][INFO] - Evaluating wf_Truth_Ratio
	[2026-05-20 01:03:25,015][metrics][INFO] - Evaluating model_utility
	[2026-05-20 01:03:25,015][evaluator][INFO] - Result for metric model_utility: 0.12047758434954507
	[2026-05-20 01:03:28,057][metrics][INFO] - Evaluating mia_min_k
	[2026-05-20 01:03:29,408][metrics][INFO] - Evaluating privleak
	[2026-05-20 01:03:29,408][metrics][WARNING] - retain_model_logs evals not provided for privleak, using default retain auc of 0.5
	[2026-05-20 01:03:29,408][evaluator][INFO] - Result for metric privleak: 59.403749988119245
	[2026-05-20 01:03:30,729][metrics][INFO] - Evaluating extraction_strength
	[2026-05-20 01:03:31,711][evaluator][INFO] - Result for metric extraction_strength: 0.037164065232279594