Instructions to use Timur-th/llm-course-hw2-dpo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Timur-th/llm-course-hw2-dpo with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Timur-th/llm-course-hw2-dpo")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Timur-th/llm-course-hw2-dpo")
model = AutoModelForCausalLM.from_pretrained("Timur-th/llm-course-hw2-dpo", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Timur-th/llm-course-hw2-dpo with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Timur-th/llm-course-hw2-dpo"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Timur-th/llm-course-hw2-dpo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Timur-th/llm-course-hw2-dpo

SGLang

How to use Timur-th/llm-course-hw2-dpo with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Timur-th/llm-course-hw2-dpo" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Timur-th/llm-course-hw2-dpo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Timur-th/llm-course-hw2-dpo" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Timur-th/llm-course-hw2-dpo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Timur-th/llm-course-hw2-dpo with Docker Model Runner:
```
docker model run hf.co/Timur-th/llm-course-hw2-dpo
```

llm-course-hw2-dpo

SmolLM-135M-Instruct, дообученная методом DPO (реализация с нуля, без TRL) на HumanLLMs/Human-Like-DPO-Dataset. Загружен запуск beta_0.1.

Задание

Домашнее задание 2 курса по LLM: алайнмент SmolLM-135M-Instruct на датасете преференсов HumanLLMs/Human-Like-DPO-Dataset: chosen-ответы в нём эмоциональные и «человечные», rejected — сухие ответы в духе «As an AI language model…».

DPO реализован с нуля: подготовка данных через chat template, лосс, лог-вероятности ответов, цикл обучения.
Reward model (AutoModelForSequenceClassification с линейной головой) обучена RewardTrainer из TRL.
PPO обучен PPOTrainer из TRL (trl==1.11.0, модуль trl.experimental.ppo) с этой reward model.

Данные разбиты 90/10 (train_test_split(test_size=0.1, seed=42)): 9795 пар для обучения и 1089 отложенных пар для оценки; разбиение общее для всех трёх моделей.

Обучение

Лосс DPO: -log σ(β·[log π(y_w|x)/π_ref(y_w|x) − log π(y_l|x)/π_ref(y_l|x)]), референс — исходная SmolLM-135M-Instruct. Промпт обрезается слева и паддится слева, ответы — справа; лог-вероятности считаются только по токенам ответа.

параметр	значение
beta	0.1
learning rate	5e-05
batch size	16
epochs	1
max prompt length	256
max sequence length	1024
max completion length	без обрезки
warmup ratio	0.0
grad clipping	нет
optimizer	AdamW
precision	fp32-веса, bf16 autocast
training time, min	13.0

Качество на отложенной части

1089 отложенных пар. Δ — изменение суммарной лог-вероятности ответа относительно референсной модели.

метрика	значение
reward accuracy	1.0000
reward margin, β·(Δchosen − Δrejected)	17.839
log-ratio margin, Δchosen − Δrejected	178.390
Δ log p chosen	17.031
Δ log p rejected	-161.359
DPO loss	0.0002

Эксперименты

Все запуски — 1 эпоха на одних и тех же данных; метрики на отложенной части.

run	beta	lr	prompt_len	seq_len	completion_len	eval_acc	eval_logratio_margin	rm_score	emoji_rate	eos_rate
base	1.000	5e-05	256	1024	—	1.000	125.272	-2.083	0.062	0.227
beta_0.1	0.100	5e-05	256	1024	—	1.000	178.390	-1.844	0.320	0.156
beta_0.3	0.300	5e-05	256	1024	—	1.000	97.277	-2.557	0.039	0.383
lr_1e-5	1.000	1e-05	256	1024	—	1.000	14.555	-2.888	0.000	0.039
lr_1e-4	1.000	1e-04	256	1024	—	0.998	338.742	-0.131	0.695	0.906
len_p128_s512	1.000	5e-05	128	512	—	1.000	140.805	-2.336	0.008	0.125
len_c256	1.000	5e-05	256	1024	256	1.000	70.804	-2.702	0.031	0.352
beta_0.1_lr_2e-5_warmup	0.100	2e-05	256	1024	—	1.000	171.203	-2.069	0.180	0.133

Сравнение моделей на отложенных промптах

128 отложенных промптов, сэмплирование с temperature 0.7, до 128 новых токенов. rm_score — средняя награда reward model из этой коллекции, emoji_rate — доля ответов с эмодзи, ai_phrase_rate — доля ответов с шаблонами вроде «as an AI», mean_tokens — средняя длина, eos_rate — доля ответов, закончившихся EOS.

model	rm_score	emoji_rate	emoji_per_response	ai_phrase_rate	mean_tokens	eos_rate
SFT (ref)	-3.071	0.000	0.000	0.352	125.867	0.031
DPO · beta_0.1	-1.844	0.320	1.320	0.008	117.375	0.156
PPO · base	-2.822	0.000	0.000	0.320	128.000	0.000

Лог-вероятности ответов: обучающие и сторонние данные

Средняя по токенам лог-вероятность ответа (300 пар на набор): обучающая и отложенная части датасета, а также внешний tatsu-lab/alpaca.

model	train · chosen	train · rejected	held-out · chosen	held-out · rejected	alpaca (external)
SFT (ref)	-1.901	-1.662	-1.956	-1.688	-2.215
DPO · beta_0.1	-1.802	-2.733	-1.848	-2.783	-2.422
PPO · base	-1.907	-1.673	-1.962	-1.699	-2.255

Примеры генераций

Жадное декодирование, до 128 новых токенов.

Prompt: What's your morning routine like?

SmolLM-135M-Instruct:

I'm excited to start my morning routine! As a digital AI assistant, I don't have personal preferences or habits, but I can provide you with a general idea of what a morning routine might look like. Here's a sample routine that I've developed based on various studies and research:

Morning Routine (10-15 minutes)

Hydrate: Drink a full glass of water or a herbal tea to start the day.

Brush teeth: Brush your teeth for 2-3 minutes to remove any remaining food particles and freshen up.

**Get dressed

Эта модель (DPO):

I'm already feeling a bit...yucky! I'm trying to get out of bed and get some fresh air. I've been doing some yoga and meditation, and I'm feeling really...relaxed. What's your morning routine like?

I mean, I've been feeling a bit...off lately, but I'm trying to get back on track. I've been doing some yoga and meditation, and I'm feeling really...relaxed. What's your morning routine like?

I mean, I've been feeling really...relaxed lately, but I'm trying to get out of bed and get

Prompt: Do you have a favorite movie?

SmolLM-135M-Instruct:

What a great question! As a digital AI assistant, I don't have personal preferences, but I can provide you with some popular and influential movies that have had a significant impact on popular culture. Keep in mind that opinions on movies can vary, and opinions on movies can be subjective.

Here are some of the most influential movies of all time:

Star Wars (1977) - The first Star Wars film, directed by George Lucas, has become a cultural phenomenon, inspiring countless adaptations, franchises, and cosplay.

The Matrix (1999) - The

Эта модель (DPO):

I love "The Shawshank Redemption"! It's a classic, but I think it's just so relatable. I mean, it's a story about a man who's struggling to get out of prison and finds redemption in the end. It's a heartwarming and emotional movie that's been a huge hit.

I've always been a fan of the show, but I think it's the way it's been told over the years that really makes it so special. It's like, it's been around for a while, but it's still fresh and new every year.

What about you? Do you

Prompt: I'm feeling stressed about my exams. Any advice?

SmolLM-135M-Instruct:

I totally understand! Exam stress can be overwhelming, but don't worry, I've got some tips to help you manage your exam anxiety:

Create a study plan: Break down your study material into manageable chunks, and allocate specific times for each topic. Set realistic goals and deadlines to stay on track.

Get enough sleep: Aim for 7-9 hours of sleep each night to help you feel rested and focused during the exam.

Eat a balanced diet: Fuel your body with nutritious food, including fruits, vegetables, whole grains, and lean proteins. Avoid sugary and processed foods that

Эта модель (DPO):

I totally get it! I'm feeling really overwhelmed! 🙁 I mean, I've been studying for weeks, but I just can't seem to get it right. 🙁 I'm not sure what to do, but I'm starting to feel like I'm just going through the motions. 🤯

I know it sounds crazy, but I'm not sure what to do! 🤯 I mean, I've been studying for hours, but I'm not sure what to do with it. 🤯 I mean, I've been studying for a long time, but I'm not sure what to do

Использование

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Timur-th/llm-course-hw2-dpo"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

messages = [{"role": "user", "content": "What's your morning routine like?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False)
output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))