File size: 3,668 Bytes
31dc8dc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 | .. _distributed_training:
====================
Distributed Training
====================
Last updated: 2025-11-04
This guide explains how to leverage distributed training to fine-tune
your model on both single-node (multi-GPU) and multi-node
(multi-machine) setups. Our training scripts are built on PyTorch's
distributed capabilities.
Single-Node, Multi-GPU Training
-------------------------------
This is the most common scenario for training on a single machine with
multiple GPUs.
Instructions
------------
Set the ``NPROC_PER_NODE`` environment variable to the number of GPUs
you want to use. Then, execute the training script. The train.sh script
will use this variable to launch the appropriate number of processes.
.. code-block:: bash
# Set the number of GPUs to use on this machine (e.g., 8)
export NPROC_PER_NODE=8
# Run the training script
# Arguments: <training_script.py> <config_file.yaml>
export PYTHONPATH=$(pwd)/VeOmni:$PYTHONPATH
sh train.sh tasks/train_llada2_bd.py configs/sft/llada2_mini_bd_sft.yaml
Multi-Node, Multi-GPU Training
-------------------------------
For large-scale training, you can scale across multiple machines. This
requires network communication between the nodes.
Prerequisites
-------------
1. **Network Connectivity:** All nodes must be able to communicate with
each other over the network. Specifically, all worker nodes must be
able to reach the ``MASTER_ADDR`` on the specified ``MASTER_PORT``.
2. **Shared Code/Data:** Ensure that the code repository and dataset are
accessible on all nodes at the same path.
Environment Variables
---------------------
You must configure the following environment variables on **each** node:
- ``NNODES``: The total number of nodes participating in the training.
- ``NODE_RANK``: The unique rank of the current node. This must be 0 for
the master node and 1, 2, … for the worker nodes.
- ``MASTER_ADDR``: The IP address of the master node (the node with
NODE_RANK=0).
- ``MASTER_PORT``: A free network port on the master node for
communication. 29500 is a common default.
- ``NPROC_PER_NODE``: The number of GPUs to use on each node.
Example for a 2-Node Setup
--------------------------
Below is an example of how to launch training on two machines, each with
8 GPUs.
**On the Master Node (IP: 192.168.1.1, Rank: 0):**
Run the following commands in your terminal:
.. code-block:: bash
# Total number of nodes
export NNODES=2
# Rank of this node
export NODE_RANK=0
# IP address of this master node
export MASTER_ADDR="192.168.1.1"
# Port for communication
export MASTER_PORT=29500
# Number of GPUs on this node
export NPROC_PER_NODE=8
# Run the training script
export PYTHONPATH=$(pwd)/VeOmni:$PYTHONPATH
sh train.sh tasks/train_llada2_bd.py configs/sft/llada2_mini_bd_sft.yaml
**On the Worker Node (Rank: 1):**
Run the following commands in your terminal on the second machine:
.. code-block:: bash
# Total number of nodes (must be the same as on master)
export NNODES=2
# Rank of this node (note the change!)
export NODE_RANK=1
# IP address of the master node
export MASTER_ADDR="192.168.1.1"
# Port on the master node (must be the same)
export MASTER_PORT=29500
# Number of GPUs on this node
export NPROC_PER_NODE=8
# Run the training script
export PYTHONPATH=$(pwd)/VeOmni:$PYTHONPATH
sh train.sh tasks/train_llada2_bd.py configs/sft/llada2_mini_bd_sft.yaml
Once the commands are executed on all nodes, the training will begin.
The master node will coordinate the process, and you should see training
logs on all participating machines.
|