transformers/pr_48871/en/cpu.md

CPU

CPU training works well when GPUs aren't available or when you want a cost-effective setup. Modern Intel CPUs support bf16 mixed precision training with PyTorch's AMP (Automatic Mixed Precision) for CPU backends, which reduces memory usage and speeds up training.

Scale CPU training across multiple sockets or nodes if a single CPU is too slow. The examples below cover three scenarios:

• a single CPU
• multiple processes on one machine (one per CPU socket)
• multiple processes across several machines

All distributed examples use Intel MPI from the Intel oneAPI HPC Toolkit for communication and a DDP strategy with Trainer.

Trainer supports bf16 mixed precision training on CPU. Prefer bf16 over fp16 for CPU training because it's more numerically stable. Pass --bf16 to enable PyTorch's CPU autocast and --use_cpu to force CPU training. You can pass the same parameters to TrainingArguments directly.

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./outputs",
    bf16=True,
    use_cpu=True,
)

On a dual-socket CPU, run one process per socket. Keeping memory accesses local to each socket improves throughput. The example below launches two processes on a single machine with one process per socket, where train.py is your Trainer training script.

Set OMP_NUM_THREADS to the number of physical cores on one socket, minus one core reserved for the OS. For example, on a 24-core socket set OMP_NUM_THREADS=23.

export MASTER_ADDR=127.0.0.1
mpirun -n 2 -genv OMP_NUM_THREADS=23 \
python3 train.py \
 --output_dir ./outputs \
 --use_cpu \
 --bf16

Scale training to four processes across two Xeon machines (node0 and node1) using a hostfile that lists each node's IP address. The -n 4 flag sets the total number of processes. -ppn 2 sets two processes per node (one per socket).

Run this script from node0, which acts as the main process.

Set OMP_NUM_THREADS to the number of physical cores on one socket, minus one core reserved for the OS. For example, on a 24-core socket set OMP_NUM_THREADS=23.

Create a hostfile with the IP address of each node.

cat hostfile
xxx.xxx.xxx.xxx #node0 ip
xxx.xxx.xxx.xxx #node1 ip

Then run the training script.

export MASTER_ADDR=xxx.xxx.xxx.xxx #node0 ip
mpirun -f hostfile -n 4 -ppn 2 \
 -genv OMP_NUM_THREADS=23 \
python3 train.py \
 --output_dir ./outputs \
 --use_cpu \
 --bf16

Kubernetes

Distributed CPU training can also run on a Kubernetes cluster using PyTorchJob.

Complete the following setup before deploying:

1. Ensure you have access to a Kubernetes cluster with Kubeflow installed.
2. Install and configure kubectl to interact with the cluster.
3. Set up a PersistentVolumeClaim (PVC) to store datasets and model files.
4. Build a Docker image for the training script and its dependencies.

The example Dockerfile below starts from an Intel-optimized PyTorch base image that includes MPI support for multi-node CPU training. It also installs two performance libraries:

• google-perftools (libtcmalloc): a memory allocator that reduces allocation overhead compared to the default system allocator.
• libomp-dev (libiomp5): Intel's OpenMP runtime, which provides better CPU thread management than the GNU OpenMP default.

FROM intel/intel-optimized-pytorch:2.4.0-pip-multinode

RUN apt-get update -y && \
    apt-get install -y --no-install-recommends --fix-missing \
    google-perftools \
    libomp-dev

WORKDIR /workspace

# Install Transformers and copy your training script into the image
RUN pip install --no-cache-dir transformers
COPY train.py /workspace/train.py

Build the image and push it to a registry accessible from your cluster's nodes before deploying.

PyTorchJob

PyTorchJob is a Kubernetes custom resource that manages PyTorch distributed training jobs. It handles worker pod lifecycle, restart policy, and process coordination. Your training script only needs to handle the model and data.

The example yaml file below sets up four workers running a training script with bf16 enabled.

When setting CPU resource limits and requests, use CPU units where one unit equals one physical core or virtual core. Set both limits and requests to the same value for a Guaranteed quality of service. Leave some cores unallocated for kubelet and system processes. Set OMP_NUM_THREADS to match the number of allocated CPU units so PyTorch uses all available cores.

Adapt the yaml file based on your training script and the number of nodes in your cluster.

apiVersion: "kubeflow.org/v1"
kind: PyTorchJob
metadata:
  name: transformers-pytorchjob
spec:
  elasticPolicy:
    rdzvBackend: c10d
    minReplicas: 1
    maxReplicas: 4
    maxRestarts: 10
  pytorchReplicaSpecs:
    Worker:
      replicas: 4  # The number of worker pods
      restartPolicy: OnFailure
      template:
        spec:
          containers:
            - name: pytorch
              image: <image name>:<tag>  # Specify the docker image to use for the worker pods
              imagePullPolicy: IfNotPresent
              command: ["/bin/bash", "-c"]
              args:
                - >-
                  cd /workspace;
                  torchrun train.py \
                    --model_name_or_path distilbert/distilbert-base-uncased \
                    --do_train \
                    --do_eval \
                    --per_device_train_batch_size 12 \
                    --learning_rate 3e-5 \
                    --num_train_epochs 2 \
                    --output_dir /tmp/pvc-mount/output_$(date +%Y%m%d_%H%M%S) \
                    --bf16;
              env:
              - name: LD_PRELOAD
                value: "/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.9:/usr/local/lib/libiomp5.so"
              - name: HF_HUB_CACHE
                value: "/tmp/pvc-mount/hub_cache"
              - name: HF_DATASETS_CACHE
                value: "/tmp/pvc-mount/hf_datasets_cache"
              - name: LOGLEVEL
                value: "INFO"
              - name: OMP_NUM_THREADS  # Set to match the number of allocated CPU units
                value: "240"
              resources:
                limits:
                  cpu: 240  # Update the CPU and memory limit values based on your nodes
                  memory: 128Gi
                requests:
                  cpu: 240  # Update the CPU and memory request values based on your nodes
                  memory: 128Gi
              volumeMounts:
              - name: pvc-volume
                mountPath: /tmp/pvc-mount
              - mountPath: /dev/shm
                name: dshm
          restartPolicy: Never
          nodeSelector:  # Optionally use nodeSelector to match a certain node label for the worker pods
            node-type: gnr
          volumes:
          - name: pvc-volume
            persistentVolumeClaim:
              claimName: transformers-pvc
          - name: dshm
            emptyDir:
              medium: Memory

Deploy

Deploy the PyTorchJob to the cluster with the following command.

export NAMESPACE=<specify your namespace>

kubectl create -f pytorchjob.yaml -n ${NAMESPACE}

List the pods in the namespace to monitor their status. Pods start as Pending while the container image is pulled, then transition to Running.

kubectl get pods -n ${NAMESPACE}

NAME                                                     READY   STATUS                  RESTARTS          AGE
...
transformers-pytorchjob-worker-0                         1/1     Running                 0                 7m37s
transformers-pytorchjob-worker-1                         1/1     Running                 0                 7m37s
transformers-pytorchjob-worker-2                         1/1     Running                 0                 7m37s
transformers-pytorchjob-worker-3                         1/1     Running                 0                 7m37s
...

Stream logs from a worker pod to follow training progress.

kubectl logs transformers-pytorchjob-worker-0 -n ${NAMESPACE} -f

Copy the trained model from the PVC or your storage location after training completes. Then delete the PyTorchJob resource.

kubectl delete -f pytorchjob.yaml -n ${NAMESPACE}

Next steps

• Read the Accelerating PyTorch Transformers with Intel Sapphire Rapids blog post for a deeper look at BF16 performance on modern Intel hardware.