ShortThink DPO (Qwen3 1.7B)

Reasoning models spend far more tokens than they need on easy problems. ShortThink teaches Qwen/Qwen3-1.7B to keep its accuracy while thinking less, and to compress easy problems more than hard ones. This model is one of six in the ShortThink collection, all trained on the same self generated data with a different algorithm, and all evaluated on the same table.

Method

Direct Preference Optimization on top of the SFT model. Chosen is the shortest correct trace, rejected is the longest correct trace for the same question, with the SFT model as the frozen reference.

Training data: AmareshHebbar/ShortThink_Traces_Qwen3_1.7B. Weights: LoRA on all linear layers, merged into full bf16 safetensors.

Why finetune instead of prompting

A prompt can ask for brevity, and the base model can switch thinking off entirely. The table below includes both of those, plus budget forcing (cap the thinking at 1024 tokens and force an answer), so you can see exactly what training buys over prompting.

Results

All rows use the same prompts, sampling settings and math_verify grading. MATH500 and GSM8K use 1 samples per problem, AIME 2024 uses 4. Tokens are generated tokens including the thinking. Tokens per correct is total tokens divided by correct answers, averaged over MATH500 and GSM8K. Latency is measured at batch size 1 with vLLM; cost assumes $1.0/hour GPU at the measured batched throughput.

ModelMATH500 accMATH500 tokensGSM8K accGSM8K tokensTokens per correctToken change vs thinking baseAIME24 accMMLULatency p50 (s)Cost per 1k queries (USD)
Qwen3 1.7B (thinking)70.0395887.522054087same36.80.419
Qwen3 1.7B (thinking + concise prompt)78.6306188.61370272128.1% fewer23.40.277
Qwen3 1.7B (no thinking)72.890180.431381380.3% fewer6.30.056
Qwen3 1.7B (budget forcing 1024)45.4147370.01219249456.3% fewer10.60.101
ShortThink SFT73.4376789.0204137135.8% fewer44.20.386
ShortThink DPO81.8283188.91288245533.2% fewer27.90.236
ShortThink GRPO83.0201685.7612157157.4% fewer15.40.132
ShortThink TIES83.0220487.8818179351.0% fewer12.30.160

Accuracy vs tokens

Compression by difficulty (MATH500)

MATH levelBase accThis model accBase tokensThis model tokensToken change
Level 195.395.32507138844.6% fewer
Level 291.194.42935182237.9% fewer
Level 378.191.43687231337.3% fewer
Level 467.282.04176307626.3% fewer
Level 544.061.25114414618.9% fewer

Tokens by MATH level

Training hyperparameters

initsft
modelora
lr5e-06
epochs1
micro_batch1
grad_accum16
warmup_ratio0.05
max_len4608
logits_chunk1024
beta0.1
max_examples1500
lora_r64
lora_alpha128
lora_dropout0.0

Usage

from vllm import LLM, SamplingParams

llm = LLM("AmareshHebbar/ShortThink_Qwen3_1.7B_DPO")
tok = llm.get_tokenizer()
question = "What is 15% of 80?"
msgs = [{"role": "user", "content": question + "\nPlease reason step by step, and put your final answer within \\boxed{}."}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=True)
out = llm.generate([prompt], SamplingParams(temperature=0.6, top_p=0.95, top_k=20, max_tokens=8192))
print(out[0].outputs[0].text)

Use the same prompt format and sampling settings as the base model's thinking mode.

Files

The repository root holds merged bf16 safetensors. adapter/ holds the LoRA adapter where one was trained, training/ holds loss curves and logs, and eval/ holds raw evaluation results and sample generations.

Downloads last month
279
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AmareshHebbar/ShortThink_Qwen3_1.7B_DPO

Finetuned
Qwen/Qwen3-1.7B
Finetuned
(1)
this model
Merges
1 model

Dataset used to train AmareshHebbar/ShortThink_Qwen3_1.7B_DPO

Collection including AmareshHebbar/ShortThink_Qwen3_1.7B_DPO