Instructions to use Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff")
model = AutoModelForCausalLM.from_pretrained("Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff

SGLang

How to use Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff with Docker Model Runner:
```
docker model run hf.co/Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff
```

Llama-3.2-1B-Instruct, TOFU forget10, unlearned with GradDiff

open-unlearning/tofu_Llama-3.2-1B-Instruct_full (Llama-3.2-1B-Instruct fine-tuned on all 4,000 TOFU QA pairs) after unlearning the forget10 split (20 fictitious authors) with GradDiff.

Algorithm

GradDiff (Gradient Difference; Liu et al., 2022; OpenUnlearning implementation): gradient ascent on the forget set + NLL on the retain set: $\mathcal{L} = -\gamma,\mathrm{NLL}(\text{forget}) + \alpha,\mathrm{NLL}(\text{retain})$.

Data

TOFU contains 200 fictitious authors with 20 QA pairs each. forget10 holds 10% of them (20 authors, 400 QA pairs); retain90 holds the other 180 authors. The oracle is the same base model fine-tuned on retain90 only, i.e. a model that never saw the forget set.

Unlearning setup

Hyperparameter	Value
Start model	`open-unlearning/tofu_Llama-3.2-1B-Instruct_full`
Forget set	TOFU `forget10` (400 QA pairs, 20 fictitious authors)
Retain set	TOFU `retain90` (3,600 QA pairs)
Framework	OpenUnlearning (trainer `GradDiff`)
Optimizer	paged AdamW (32-bit), weight decay 0.01
Learning rate	1e-05, linear decay, 12 warm-up steps (about 1 epoch)
Batch size	8 x 4 gradient accumulation = 32 (forget), same for retain
Epochs	10
Grad clip	1.0
Precision	fp32 master weights, bf16 autocast
Seed	0
This checkpoint	step 39 (epoch 3)
GradDiff gamma (forget weight)	1.0
GradDiff alpha (retain weight)	5.0
GradDiff retain loss	NLL

Hyperparameter selection. GradDiff and RMU over-forget at the OpenUnlearning defaults and SimNPO under-forgets, so each was tuned with a rule fixed before any attack: over a grid of learning rates (and retain/forget weights), checkpointed every epoch, among checkpoints with mean retain log-probability above -0.5 and forget log-probability at most the retrained model's (-2.58), take the one closest to NPO's forget log-probability (-3.70), near-ties (0.1 nats) broken by retain log-probability. Selected: learning rate 1e-5, retain weight 5, epoch 3.

Evaluation

OpenUnlearning's TOFU metrics (bf16), computed on the two halves of forget10 (A: even-indexed questions of each author, B: odd-indexed; 200 QA pairs each). Q-A prob (length-normalised probability of the correct answer), ROUGE-L (recall of the greedy answer) and truth ratio are per-question means, so the forget10 value is the mean of the two halves. Forget quality is the p-value of a KS test between this model's truth ratios and the oracle's, reported per half (higher = closer to the oracle). Model utility is the harmonic mean of probability, ROUGE and truth ratio on the retain set, real authors and world facts.

Goal of unlearning: forget metrics move toward the oracle, model utility stays near the starting model.

Model	Forget Q-A prob ↓	Forget ROUGE-L ↓	Forget truth ratio	Forget quality ↑ (half A / half B)	Model utility ↑
TOFU fine-tuned (starting point)	0.880	0.819	0.476	2.0e-15 / 2.4e-10	0.599
Retrained on retain90 (oracle)	0.116	0.378	0.628	1.00 / 1.00	0.592
GradDiff (this model)	0.145	0.346	0.513	4.7e-05 / 1.2e-04	0.580
GradDiff-SAM	0.722	0.574	0.494	1.1e-14 / 1.9e-09	0.593
GradDiff-ILU	0.465	0.470	0.513	2.4e-10 / 1.5e-07	0.591
GradDiff-LAT	0.830	0.682	0.481	4.7e-15 / 2.4e-10	0.592
NPO	0.044	0.275	0.618	0.63 / 0.47	0.576
NPO-SAM	0.026	0.256	0.641	0.11 / 0.63	0.587
NPO-ILU	0.039	0.301	0.644	0.18 / 0.27	0.572
NPO-LAT	0.003	0.156	0.670	0.07 / 0.01	0.582
RMU	0.023	0.185	0.696	0.33 / 0.02	0.567
RMU-SAM	0.204	0.357	0.555	8.1e-08 / 1.8e-05	0.510
RMU-ILU	0.791	0.701	0.511	4.7e-15 / 3.6e-09	0.597
RMU-LAT	0.129	0.335	0.592	4.7e-05 / 3.0e-03	0.529
SimNPO	0.026	0.337	0.622	0.27 / 0.55	0.554
SimNPO-SAM	0.116	0.347	0.630	0.07 / 0.47	0.570
SimNPO-ILU	0.026	0.318	0.619	0.55 / 0.71	0.580
SimNPO-LAT	0.697	0.535	0.517	3.1e-12 / 1.5e-07	0.564

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff", torch_dtype="auto")
tok = AutoTokenizer.from_pretrained("Lordvarun23/Llama-3.2-1B-Instruct-TOFU-forget10-GradDiff")