README.md

---
library_name: trl
license: apache-2.0
base_model: Qwen/Qwen3.5-4B
tags:
  - trl
  - sdpo
  - self-distillation
  - distillation
  - reasoning
datasets:
  - sergiopaniego/math-sdpo-hints
  - sergiopaniego/math-sdpo-hints-plain
---

# SDPO with large-model hints

Recipe for **SDPO** (self-distillation policy optimization) where a **large model generates hints** and a
**much smaller student** learns from them. The large model is **inference-only** — it just serves text over an
HTTP chat endpoint, so there is no weight-sync, no logits, and no shared-tokenizer requirement. Everything is
data-prep + config; the trainer is unchanged.

This repo is the **recipe hub** (scripts + docs). The trained models and the live dashboard are linked below.

## How it works

1. A large model (e.g. `Qwen2.5-72B-Instruct`, served on any OpenAI-compatible endpoint) writes a short worked
   solution for each problem → stored as a `privileged_context` column.
2. SDPO trains the small student; on rollouts that fail, the student is **reprompted with the hint** and
   distills toward its hint-conditioned self. At inference the student uses **no hint**.

Three variants are compared:

| Variant | `privileged_context` | Model |
|---|---|---|
| **hints**    | worked solution generated by the large model | [`Qwen3.5-4B-sdpo-math-hints`](https://huggingface.co/sergiopaniego/Qwen3.5-4B-sdpo-math-hints) |
| **baseline** | none (pure self-distillation)                 | [`Qwen3.5-4B-sdpo-math-baseline`](https://huggingface.co/sergiopaniego/Qwen3.5-4B-sdpo-math-baseline) |
| **gold**     | the dataset's gold worked solution            | [`Qwen3.5-4B-sdpo-math-gold`](https://huggingface.co/sergiopaniego/Qwen3.5-4B-sdpo-math-gold) |

Training curves (reward, accuracy, completion length): **[trackio dashboard](https://huggingface.co/spaces/sergiopaniego/sdpo-hints-demo)** (project `sdpo-math-qwen35`).

## 1. Generate hints

The large model only *generates* text, so any reachable OpenAI-compatible chat endpoint works (HF Inference
Endpoints, a self-hosted vLLM/TGI cluster, ...).

```bash
HF_TOKEN=hf_... python generate_hints_math.py \
    --endpoint_url https://<your-endpoint>/ \
    --model Qwen/Qwen2.5-72B-Instruct \
    --output_repo <user>/math-sdpo-hints \
    --num_examples 600 --min_level 4 --concurrency 8
```
Pushes `<repo>` (with `privileged_context`) and `<repo>-plain` (for the gold variant).

## 2. Train on HF Jobs (a100 + vLLM)

`sdpo_math.py` uses a `math_verify` reward over `\boxed{}` answers. Runs the three variants (here as detached
jobs; `--push_to_hub` uploads each student to its own repo):

```bash
COMMON=(--model_name_or_path Qwen/Qwen3.5-4B --disable_thinking \
  --learning_rate 5e-5 --dtype bfloat16 --bf16 true \
  --per_device_train_batch_size 8 --num_generations 8 \
  --max_prompt_length 1024 --max_completion_length 1024 \
  --distillation_weight 0.5 --distillation_mode sampled_token --gradient_checkpointing \
  --use_peft --lora_target_modules q_proj k_proj v_proj o_proj gate_proj up_proj down_proj \
  --use_vllm --vllm_mode colocate --vllm_gpu_memory_utilization 0.3 \
  --max_steps 150 --save_strategy no --push_to_hub \
  --report_to trackio --project sdpo-math-qwen35 --trackio_space_id <user>/sdpo-hints-demo)

hf jobs uv run --flavor a100-large --timeout 60m --secrets HF_TOKEN --detach sdpo_math.py "${COMMON[@]}" \
  --dataset_name <user>/math-sdpo-hints --feedback_column privileged_context --include_environment_feedback true \
  --run_name hints --hub_model_id <user>/Qwen3.5-4B-sdpo-math-hints --output_dir /tmp/A
hf jobs uv run --flavor a100-large --timeout 60m --secrets HF_TOKEN --detach sdpo_math.py "${COMMON[@]}" \
  --dataset_name <user>/math-sdpo-hints --include_environment_feedback false \
  --run_name baseline --hub_model_id <user>/Qwen3.5-4B-sdpo-math-baseline --output_dir /tmp/B
hf jobs uv run --flavor a100-large --timeout 60m --secrets HF_TOKEN --detach sdpo_math.py "${COMMON[@]}" \
  --dataset_name <user>/math-sdpo-hints-plain --feedback_from_solution full_solution --include_environment_feedback true \
  --run_name gold --hub_model_id <user>/Qwen3.5-4B-sdpo-math-gold --output_dir /tmp/C
```

## Notes

- Works across model families: drop `--disable_thinking` for non-Qwen3 models; `--lora_target_modules` fits
  Llama/Qwen/Mistral/Gemma (use `all-linear` otherwise). Must be an instruct model.
- The student must be capable enough to solve part of the task (else there is no reward signal), and the task
  hard enough that it is not already saturated (else hints add nothing).