transformers/pr_46771/en/cpu.md

CPU

CPUs run Transformers models with the device string cpu, without an accelerator. They work for inference and training when GPUs aren't available, or when you want a cost-effective setup.

Run inference

Models load on CPU by default when you don't pass device_map, and pipeline() uses the CPU when no accelerator is available. Pass device_map="cpu" or device="cpu" to use the CPU on a machine that also has an accelerator.

from transformers import pipeline

generator = pipeline("text-generation", model="Qwen/Qwen3-0.6B", device="cpu")
print(generator("CPUs are", max_new_tokens=50, do_sample=False)[0]["generated_text"])

Train with Trainer

CPUs with bf16 support, such as recent Intel and AMD processors, can use bf16 mixed precision training with PyTorch's AMP, which reduces memory usage and speeds up training.

The examples below cover three setups, from a single CPU to several machines when one CPU is too slow:

• a single CPU
• multiple processes on one machine (one per CPU socket)
• multiple processes across several machines

The single-node and multi-node examples launch processes with mpirun from Intel MPI, part of the Intel oneAPI HPC Toolkit, and Trainer runs DDP over the gloo backend.

Set bf16=True and use_cpu=True in TrainingArguments, or pass --bf16 and --use_cpu to a training script. Prefer bf16 over fp16 on CPU because it's more numerically stable.

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./outputs",
    bf16=True,
    use_cpu=True,
)

On a dual-socket CPU, run one process per socket. Keeping memory accesses local to each socket improves throughput. The example below launches two processes on a single machine with one process per socket, where train.py is your Trainer training script.

Set OMP_NUM_THREADS to the number of physical cores on one socket, minus one core reserved for the OS. For example, on a 24-core socket set OMP_NUM_THREADS=23.

export MASTER_ADDR=127.0.0.1
mpirun -n 2 -genv OMP_NUM_THREADS=23 \
python3 train.py \
 --output_dir ./outputs \
 --use_cpu \
 --bf16

Scale training to four processes across two machines (node0 and node1) using a hostfile that lists each node's IP address. The -n 4 flag sets the total number of processes. -ppn 2 sets two processes per node (one per socket).

Run this script from node0, which acts as the main process.

Set OMP_NUM_THREADS to the number of physical cores on one socket, minus one core reserved for the OS. For example, on a 24-core socket set OMP_NUM_THREADS=23.

Create a hostfile with the IP address of each node.

cat hostfile
xxx.xxx.xxx.xxx #node0 ip
xxx.xxx.xxx.xxx #node1 ip

Then run the training script.

export MASTER_ADDR=xxx.xxx.xxx.xxx #node0 ip
mpirun -f hostfile -n 4 -ppn 2 \
 -genv OMP_NUM_THREADS=23 \
python3 train.py \
 --output_dir ./outputs \
 --use_cpu \
 --bf16

Tune CPU performance

Preload a faster memory allocator and Intel's OpenMP runtime to reduce overhead in CPU training. Install google-perftools (for libtcmalloc) and the intel-openmp Python package (for libiomp5), then add both libraries to the LD_PRELOAD environment variable before you launch training.

Set OMP_NUM_THREADS to the number of cores each process can use. In a container or a cluster job with CPU limits, that's the number of allocated CPU units, so leave some cores unallocated for system processes.

Next steps

• Read the Accelerating PyTorch Transformers with Intel Sapphire Rapids blog post for a deeper look at BF16 performance on modern Intel hardware.
• Read DDP for more on the distributed strategy the multi-process examples use.
• Read the PyTorch CPU threading notes to learn how OMP_NUM_THREADS affects performance.
• Read the Kubeflow PyTorchJob guide to run distributed CPU training on Kubernetes.