finetune/README.md

中文 ｜ English

Model Fine-tuning

Hy4 preview provides processes related to model fine-tuning. This section details how to process training data for model fine-tuning purposes.

Training Data Format and Processing

Hy4 preview supports both "slow thinking" and "fast thinking" modes. You can control the mode via the reasoning_effort parameter (options: high, no_think).

The training data should be formatted as a list of messages. By default, the system prompt for both training and inference is empty, but you may customize it as needed.

# Fast thinking pattern (no_think)
{"reasoning_effort": "no_think", "messages": [{"content": "You are a helpful assistant.\nThe current time is 2026-01-01 13:26:12 Thursday", "role": "system"}, {"content": "1+1=?", "role": "user"}, {"role": "assistant", "content": "1+1=2"}]}

# Slow thinking pattern (high)
{"reasoning_effort": "high", "messages": [{"content": "You are a helpful assistant.\nThe current time is 2026-01-01 13:26:12 Thursday", "role": "system"}, {"content": "1+1=?", "role": "user"}, {"role": "assistant", "content": "1+1=2", "reasoning_content": "The user is asking for the result of 1 + 1. In basic decimal arithmetic, 1 + 1 equals 2."}]}

Example of using apply_chat_template to tokenize:

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("./models", use_fast=False, trust_remote_code=True)

messages = [
    {"content": "You are a helpful assistant.", "role": "system"},
    {"content": "1+1=?", "role": "user"},
    {"role": "assistant", "content": "1+1=2"}
]
ids = tokenizer.apply_chat_template(messages, tokenize=True, reasoning_effort="no_think")

Fine-tuning Process

Hardware Requirements

Based on testing, the minimum resource configuration is as follows:

• LoRA Fine-tuning: At least 8 machines with 64 GPUs (at least 96GB GPU memory per GPU, at least 2TB CPU memory per machine).
• Full Fine-tuning: At least 16 machines with 128 GPUs (at least 96GB GPU memory per GPU, at least 2TB CPU memory per machine).

Note: The above are minimum resource configurations; actual requirements increase with max_seq_length, batch size, etc.

Configure Passwordless SSH Login Between Machines (Multi-Machine Training)

If you only use single-machine training, you can skip this section.

The following instructions use two machines as an example, with their IPs denoted as ${ip1} and ${ip2}. All steps should be performed inside the Docker container.

First, configure passwordless SSH for each container on every machine:

ssh-keygen			# Generate id_rsa and id_rsa.pub for passwordless login
ssh-keygen -t rsa -A    # Generate /etc/ssh/ssh_host_rsa_key and ssh_host_ecdsa_key for SSH listening
/usr/sbin/sshd -p 36005 -o ListenAddress=0.0.0.0        # Start SSH listening
echo "Port 36005" > ~/.ssh/config   # Set SSH connection port to 36005
passwd root    # Set the root password to avoid monitoring platform alerts

Note: 36005 is an example port. You may use any available port, but ensure it is open and not occupied by other processes.

Next, in each machine's container, execute:

cat ~/.ssh/id_rsa.pub

Copy the output SSH public key and paste it into the ~/.ssh/authorized_keys file, one key per line. This must be done on every machine. In the end, the ~/.ssh/authorized_keys file on each machine should be identical and contain the public keys of all machines.

Please note that for multi-node training, the code executed on each node must be identical. It is recommended to mount a shared network drive. If this is not possible, you must manually copy the dataset, scripts, and code to the same directory on each machine.

Launch Methods

This project provides three fine-tuning methods. You can choose based on your needs:

• DeepSpeed Native Fine-tuning (based on HuggingFace Transformers Trainer): Located in the deepspeed_support directory
• LLaMA-Factory Fine-tuning: Located in the llama_factory_support directory
• ms-swift Fine-tuning: Located in the ms_swift_support directory

DeepSpeed Native Fine-tuning

Reference: HuggingFace Transformers Trainer

Single-Machine Fine-tuning

In the deepspeed_support directory, execute:

pip install -r requirements.txt
bash train.sh

Multi-Machine Fine-tuning

To launch fine-tuning across multiple machines, please first complete the configuration in Configure Passwordless SSH Login Between Machines, and ensure all machines are within the same cluster.

Confirm that dependencies are installed (if not, run pip install -r requirements.txt), then add the following configuration at the beginning of train.sh:

export HOST_GPU_NUM=8
# IP list, comma separated. e.g. "192.168.1.1,192.168.1.2" or single node "192.168.1.1"
IP_LIST=${IP_LIST:-"127.0.0.1"}

Note: If the IP_LIST environment variable is not set, replace IP_LIST with the IP list! The format is:

For a single IP:
IP_LIST=${ip_1}

For multiple IPs:
IP_LIST=${ip_1},${ip_2}

Replace ${ip_1} and ${ip_2} with the actual IP addresses.

Then, on the machine with ${ip1}, execute bash train.sh in the deepspeed_support/ directory. On first launch, you may see the following output:

The authenticity of host '[ip]:36005 ([ip]:36005)' can't be established.
ECDSA key fingerprint is xxxxxx.
ECDSA key fingerprint is MD5:xxxxxx.
Are you sure you want to continue connecting (yes/no)?

Type yes to continue.

Key Parameters

The key parameters in the script are as follows:

• --deepspeed: Path to the DeepSpeed configuration file. Four default DeepSpeed configuration files are provided in the deepspeed_support folder: ds_zero2_no_offload.json, ds_zero2_offload.json, ds_zero3_no_offload.json, and ds_zero3_offload.json, with different ZeRO stages (ZeRO-2 / ZeRO-3) and offload strategies selectable based on available GPU memory and communication constraints.
• --model_name_or_path: Path to the Hy4 preview HF pre-trained model weights to load, otherwise loading will fail.
• --tokenizer_name_or_path: Path to the tokenizer folder, otherwise loading will fail.
• --train_data_file: Path to the training file, which should be a jsonl file.
• --output_dir: Output directory where logs, tensorboard files, and model weights will be stored.
• --per_device_train_batch_size: Batch size per GPU.
• --gradient_accumulation_steps: Number of gradient accumulation steps. The global batch size is per_device_train_batch_size * gradient_accumulation_steps * dp_size.
• --max_steps: Total number of training steps.
• --save_steps: Number of steps between saving checkpoints.
• --use_lora: Whether to use LoRA training. Also accepts --lora_rank, --lora_alpha, and --lora_dropout parameters. By default, LoRA is applied to "q_proj", "k_proj", "v_proj", and "o_proj". To change this, modify the code. Note: When using LoRA training, only the LoRA weights are saved, not the base model weights.
• --make_moe_param_leaf_module: When using ZeRO-3 with MoE training, treat the MoE module as a leaf module, i.e., its parameters are not partitioned by ZeRO-3. This option is expected to significantly increase memory usage.
• --gradient_checkpointing: Enable gradient checkpointing.
• --learning_rate: Maximum learning rate during training.
• --min_lr: Minimum learning rate during training.
• --use_flash_attn: Enable flash-attention for accelerated training.

Notes:

• To resume training from a previously saved checkpoint rather than loading pre-trained weights, specify --resume_from_checkpoint with the path to the checkpoint. Do not specify --model_name_or_path; this will load only the weights without the training state.
• When resuming from a checkpoint, there may be minor differences in loss due to the randomness of some non-deterministic algorithms. This is normal. See: HuggingFace Transformers Trainer Randomness
• When --model_name_or_path is specified, all model-related parameters will be ignored.
• Samples within a batch are padded to the length of the longest sample in the batch, but the maximum length of each sample is max_seq_length. Any excess will be truncated.
• If you see a warning about linear layer bias weights not being loaded, you can ignore it; Hy4 preview's linear layers (q_proj / k_proj / v_proj / o_proj, etc.) do not use bias. Note: the MoE router's e_score_correction_bias is a buffer and is auto-loaded by the training script, so please do not ignore its loading failure.

What if GPU Memory is Insufficient?

Reference: DeepSpeed Configuration

You can try modifying the DeepSpeed configuration by removing the auto attribute from the following parameters and reducing their values:

• stage3_param_persistence_threshold
• stage3_prefetch_bucket_size
• stage3_max_reuse_distance

LLaMA-Factory Fine-tuning

If you are familiar with LLaMA-Factory, you may use it for fine-tuning. All scripts, code, and configuration files are archived in the llama_factory_support directory. Unless otherwise specified, all files mentioned below are located in this directory.

Installation

You can install LLaMA-Factory by downloading the source code from https://github.com/hiyouga/LLaMA-Factory/tree/main and following the instructions on the website.

Configuration Files

We provide sample LLaMA-Factory fine-tuning configuration files: hy_v4_lora_sft.yaml and hy_v4_full_sft.yaml, corresponding to LoRA fine-tuning and full fine-tuning respectively.

Key parameters in the configuration files are as follows:

Model:

• model_name_or_path: Path to the Hy4 preview HF format pre-trained model weights
• trust_remote_code: Whether to trust remote code; Hy4 preview requires this to be set to true

Training Method:

• stage: Training stage, currently sft (supervised fine-tuning)
• finetuning_type: Fine-tuning type, either full (full fine-tuning) or lora (LoRA fine-tuning)
• deepspeed: DeepSpeed configuration file path; ../deepspeed_support/ds_zero3_offload.json is recommended for full fine-tuning
• fsdp + fsdp_config: FSDP distributed strategy; recommended for LoRA fine-tuning (configuration is built into hy_v4_lora_sft.yaml); mutually exclusive with DeepSpeed

Distributed Strategy Recommendations:

• FSDP: Recommended for LoRA fine-tuning, good compatibility and simple configuration
• DeepSpeed ZeRO-3 + Offload: Recommended for full fine-tuning or memory-constrained scenarios

LoRA Parameters (only effective during LoRA fine-tuning):

• lora_rank: LoRA rank, default 64
• lora_alpha: LoRA alpha coefficient, default 128
• lora_dropout: LoRA dropout ratio, default 0.05
• lora_target: Target modules for LoRA, default q_a_proj,q_b_proj,kv_a_proj_with_mqa,kv_b_proj,o_proj

Dataset:

• dataset_dir: Dataset directory path
• dataset: Dataset name, must be registered in dataset_info.json under dataset_dir
• template: Chat template; Hy4 preview uses hy_v4
• cutoff_len: Maximum sequence length; sequences exceeding this will be truncated. For LoRA fine-tuning, a smaller value is recommended to save memory
• max_samples: Maximum number of samples per dataset
• overwrite_cache: Whether to overwrite cached preprocessed datasets

Output:

• output_dir: Output directory where logs, TensorBoard files, and weights will be stored
• logging_steps: Number of steps between logging
• save_steps: Number of steps between saving checkpoints
• plot_loss: Whether to plot the training loss curve
• overwrite_output_dir: Whether to overwrite the existing output directory
• save_only_model: Whether to save only model weights (excluding optimizer states, etc.)
• report_to: Logging tool, options: none, wandb, tensorboard, swanlab, mlflow

Training Hyperparameters:

• per_device_train_batch_size: Batch size per GPU
• gradient_accumulation_steps: Gradient accumulation steps; per_device_train_batch_size * gradient_accumulation_steps * dp_size equals the global batch size
• learning_rate: Maximum learning rate; 1.0e-5 recommended for full fine-tuning, 2.0e-4 for LoRA fine-tuning
• num_train_epochs: Number of training epochs
• lr_scheduler_type: Learning rate scheduler type; cosine_with_min_lr is recommended
• lr_scheduler_kwargs.min_lr_rate: Ratio of minimum to maximum learning rate; e.g., 0.1 means the minimum learning rate is 10% of the maximum
• warmup_steps: Number of warmup steps
• bf16: Whether to use BFloat16 mixed precision training
• gradient_checkpointing: Whether to enable gradient checkpointing to save memory
• ddp_timeout: Distributed training timeout (milliseconds)
• flash_attn: Attention implementation; auto (automatic selection) or sdpa is recommended
• resume_from_checkpoint: Resume training from a specified checkpoint path; set to null to start from scratch

Launch Fine-tuning

For multi-machine fine-tuning, please first complete the configuration in Configure Passwordless SSH Login Between Machines (single-machine fine-tuning can skip this step).

Modify the following configuration at the beginning of train_lf.sh:

export HOST_GPU_NUM=8
# IP list, comma separated. e.g. "192.168.1.1,192.168.1.2" or single node "192.168.1.1"
export IP_LIST=${IP_LIST:-"127.0.0.1"}

Note:

1. If the IP_LIST environment variable is not set, replace IP_LIST with the IP list! The format is:

For a single IP:
IP_LIST=${ip_1}

For multiple IPs:
IP_LIST=${ip_1},${ip_2}

Replace ${ip_1} and ${ip_2} with the actual IP addresses.

2. To specify a fine-tuning configuration file, set the YAML_FILE environment variable. The default is hy_v4_full_sft.yaml. For example, to use the LoRA fine-tuning configuration:

export YAML_FILE=hy_v4_lora_sft.yaml

Then, on each machine, run the launch script in the llama_factory_support/ directory:

bash train_lf.sh

ms-swift Fine-tuning

If you are familiar with ms-swift, you may use it for fine-tuning. All scripts, code, and configuration files are archived in the ms_swift_support directory. Unless otherwise specified, all files mentioned below are located in this directory.

Installation

You can install ms-swift via pip:

pip install ms-swift

Or install from source: https://github.com/modelscope/ms-swift

Fine-tuning Scripts and Configuration Files

Fine-tuning Method	Configuration File	Launch Script
Full Fine-tuning	hy_v4_full_sft.yaml	bash sft_train.sh
LoRA Fine-tuning	hy_v4_lora_sft.yaml	bash sft_train_lora.sh

About the eos_token_id Patch

The hy_v4_swift_patches.py file in the directory fixes an issue with the eos token in ms-swift's default template. The default template uses the <｜hy_eos｜> string as chat_sep and suffix, which gets tokenized into multiple token IDs, causing model.generate() to fail to stop correctly during inference.

The patch re-registers the template using the [['eos_token_id']] syntax, allowing ms-swift to dynamically resolve tokenizer.eos_token_id at runtime and generate the correct single token.

The launch script automatically loads this patch via --custom_register_path hy_v4_swift_patches.py, requiring no additional action.

Key Parameters

Key parameters in the configuration files are as follows:

Model:

• model: Model path, can be a HuggingFace Hub ID or a local path
• model_type: Model type, set to hy_v4
• template: Chat template, set to hy_v4
• torch_dtype: Data type, bfloat16 is recommended
• attn_impl: Attention implementation, sdpa is recommended

Training Method:

• train_type: Fine-tuning type; set to full for full fine-tuning, lora for LoRA fine-tuning
• lora_rank: LoRA rank, default 64
• lora_alpha: LoRA alpha coefficient, default 128
• lora_dropout: LoRA dropout ratio, default 0.05

Dataset:

• dataset: Dataset path, supports local jsonl files (sharegpt format)
• max_length: Maximum sequence length; sequences exceeding this will be truncated
• lazy_tokenize: Whether to use lazy tokenization, true is recommended

Output:

• output_dir: Output directory
• save_steps: Number of steps between saving checkpoints
• save_total_limit: Maximum number of checkpoints to keep
• logging_steps: Number of steps between logging
• report_to: Logging tool, options: none, wandb, tensorboard, swanlab, mlflow

Training Hyperparameters:

• per_device_train_batch_size: Batch size per GPU
• gradient_accumulation_steps: Gradient accumulation steps
• learning_rate: Maximum learning rate; 1.0e-5 recommended for full fine-tuning, 2.0e-4 for LoRA fine-tuning
• num_train_epochs: Number of training epochs
• lr_scheduler_type: Learning rate scheduler type, cosine is recommended
• warmup_steps: Number of warmup steps
• bf16: Whether to use BFloat16 mixed precision training

Distributed Strategy / Optimization:

• deepspeed: DeepSpeed strategy, options: zero0, zero2, zero2_offload, zero3, zero3_offload; zero3_offload recommended for full fine-tuning
• fsdp + fsdp_config: FSDP distributed strategy; recommended for LoRA fine-tuning; mutually exclusive with DeepSpeed
• gradient_checkpointing: Whether to enable gradient checkpointing
• max_grad_norm: Gradient clipping threshold

Distributed Strategy Recommendations:

• FSDP: Recommended for LoRA fine-tuning, good compatibility and simple configuration
• DeepSpeed ZeRO-3 + Offload: Recommended for full fine-tuning or memory-constrained scenarios

Other:

• ddp_timeout: Distributed training timeout (milliseconds)
• seed: Random seed
• resume_from_checkpoint: Resume training from a specified checkpoint path

Launch Fine-tuning

For multi-machine fine-tuning, please first complete the configuration in Configure Passwordless SSH Login Between Machines (single-machine fine-tuning can skip this step).

Modify the following configuration in the sft_train.sh script:

export HOST_GPU_NUM=8
# IP list, comma separated. e.g. "10.0.0.1,10.0.0.2" or single node "127.0.0.1"
export IP_LIST=${IP_LIST:-"127.0.0.1"}

Then, on each machine, execute the launch script in the ms_swift_support/ directory:

# Single-machine training
bash sft_train.sh

# Multi-machine training (execute on each machine)
IP_LIST="10.0.0.1,10.0.0.2" bash sft_train.sh