course/pr_1194/en/chapter11/3.md

Supervised Fine-Tuning

Supervised Fine-Tuning (SFT) is a process primarily used to adapt pre-trained language models to follow instructions, engage in dialogue, and use specific output formats. While pre-trained models have impressive general capabilities, SFT helps transform them into assistant-like models that can better understand and respond to user prompts. This is typically done by training on datasets of human-written conversations and instructions.

This page provides a step-by-step guide to fine-tuning the deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B model using the SFTTrainer. By following these steps, you can adapt the model to perform specific tasks more effectively.

When to Use SFT

Before diving into implementation, it's important to understand when SFT is the right choice for your project. As a first step, you should consider whether using an existing instruction-tuned model with well-crafted prompts would suffice for your use case. SFT involves significant computational resources and engineering effort, so it should only be pursued when prompting existing models proves insufficient.

Consider SFT only if you:

• Need additional performance beyond what prompting can achieve
• Have a specific use case where the cost of using a large general-purpose model outweighs the cost of fine-tuning a smaller model
• Require specialized output formats or domain-specific knowledge that existing models struggle with

If you determine that SFT is necessary, the decision to proceed depends on two primary factors:

Template Control

SFT allows precise control over the model's output structure. This is particularly valuable when you need the model to:

1. Generate responses in a specific chat template format
2. Follow strict output schemas
3. Maintain consistent styling across responses

Domain Adaptation

When working in specialized domains, SFT helps align the model with domain-specific requirements by:

1. Teaching domain terminology and concepts
2. Enforcing professional standards
3. Handling technical queries appropriately
4. Following industry-specific guidelines

Before starting SFT, evaluate whether your use case requires:

• Precise output formatting
• Domain-specific knowledge
• Consistent response patterns
• Adherence to specific guidelines

This evaluation will help determine if SFT is the right approach for your needs.

Dataset Preparation

The supervised fine-tuning process requires a task-specific dataset structured with input-output pairs. Each pair should consist of:

1. An input prompt
2. The expected model response
3. Any additional context or metadata

The quality of your training data is crucial for successful fine-tuning. Let's look at how to prepare and validate your dataset: