File size: 2,817 Bytes
da50cbf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 | # EditScore Reward Model Training Guide
This guide explains how to train EditScore reward models using LLaMA-Factory.
## 1. Environment Setup
### Clone LLaMA-Factory and Configure Virtual Environment
```bash
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
conda create -n llama-factory python=3.10
conda activate llama-factory
pip install -e ".[torch,metrics]" --no-build-isolation
```
## 2. Directory Structure Configuration
Create necessary folders and files in the LLaMA-Factory root directory:
```bash
# Create log and output directories
mkdir -p logs
mkdir -p output
# Create training configuration directory
mkdir -p examples/train_editscore
# Copy training configuration files
cp EditScore/examples/EditScore-train/config/*.yaml examples/train_editscore/
# Copy training script
cp EditScore/examples/EditScore-train/train.sh .
```
## 3. Dataset Registration
Register the EditScore-Reward-Data dataset in `LLaMA-Factory/data/dataset_info.json`:
```json
"EditScore-Reward-Data": {
"file_name": "/path/to/your/reward.json",
"formatting": "sharegpt",
"columns": {
"messages": "conversations",
"images": "images"
}
}
```
## 4. Training Configuration Description
### Single-Machine Training Configuration
- `editscore_7B.yaml` - Train EditScore-7B model (single machine)
- `editscore_qwen3_vl_4B_instruct.yaml` - Train EditScore_Qwen3_Vl_4B_Instruct model (single machine)
- `editscore_qwen3_vl_8B_instruct.yaml` - Train EditScore_Qwen3_Vl_8B_Instruct model (single machine)
### Multi-Machine Training Configuration
- `editscore_32B.yaml` - Train EditScore-32B model (two machines)
- `editscore_72B.yaml` - Train EditScore-72B model (two machines)
## 5. Start Training
### Single-Machine Training
```bash
# Modify experiment_name in train.sh to the corresponding configuration file name
# For example: name=editscore_7B
bash train.sh
```
### Multi-Machine Training
**Master node (rank=0):**
```bash
bash train.sh --rank=0 --world_size=2 --master_addr=MASTER_NODE_IP --master_port=29500
```
**Worker node (rank=1):**
```bash
bash train.sh --rank=1 --world_size=2 --master_addr=MASTER_NODE_IP --master_port=29500
```
## 6. Parameter Configuration
Users can modify the following parameters in the YAML configuration files as needed:
- `per_device_train_batch_size`: Batch size per device
- `gradient_accumulation_steps`: Gradient accumulation steps
- `learning_rate`: Learning rate
- `num_train_epochs`: Number of training epochs
- `max_samples`: Maximum number of samples
- `output_dir`: Output directory
## 7. Output Files
After training completion, model files will be saved in the corresponding output directories:
- Single-machine training: `LLaMA-Factory/output/model_name/`
- Log files: `LLaMA-Factory/logs/experiment_name_rank.log`
|