| 速度基准 |
| ======== |
|
|
| 我们在训练速度方面与 |
| `LLaMA-Factory <https://github.com/hiyouga/LLaMA-Factory>`__ |
| 进行了对比。对比所使用的 LLaMA-Factory commit id 为 |
| `8e04794 <https://github.com/hiyouga/LLaMA-Factory/tree/8e04794b2da067a4123b9d7091a54c5647f44244>`__\ 。使用 |
| `Alpaca <https://huggingface.co/datasets/tatsu-lab/alpaca>`__ |
| 作为训练数据集测试速度。 |
|
|
| 硬件 |
| ---- |
|
|
| - NVIDIA A100-SXM4-80GB GPUs |
|
|
| - Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz |
|
|
| 软件环境 |
| -------- |
|
|
| - Python 3.10 |
|
|
| - PyTorch 1.13 |
|
|
| - CUDA 11.7 |
|
|
| - CUDNN 8.5 |
|
|
| - NCCL 2.14.3 |
|
|
| 速度 |
| ---- |
|
|
| |image1| |
|
|
| |image2| |
|
|
| |image3| |
|
|
| .. tip:: |
| TGS 全称是 Tokens per GPU per Second，每张 GPU 每秒训练的 Token 数 |
|
|
| .. raw:: html |
|
|
| <html xmlns="http://www.w3.org/1999/xhtml"><head></head><body><div align="center"></div></body></html> |
|
|
| .. list-table:: |
| :widths: 30 15 20 20 20 50 |
| :header-rows: 1 |
|
|
| * - 模型 |
| - GPUs |
| - 序列长度 |
| - TGS |
| - TFLOPs |
| - Config |
| * - Llama2-7B |
| - 8 |
| - 8k |
| - 3028.3 |
| - 185.3 |
| - `llama2_70b_full_alpaca_enzh_8k_sp1.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_8k_sp1.py>`_ |
| * - Llama2-7B |
| - 8 |
| - 32k |
| - 2234.2 |
| - 193.0 |
| - `llama2_7b_full_alpaca_enzh_32k_sp1.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_32k_sp1.py>`_ |
| * - Llama2-7B |
| - 8 |
| - 128k |
| - 948.6 |
| - 180.3 |
| - `llama2_7b_full_alpaca_enzh_128k_sp8.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_128k_sp8.py>`_ |
| * - Llama2-7B |
| - 8 |
| - 256k |
| - 540.1 |
| - 176.9 |
| - `llama2_7b_full_alpaca_enzh_256k_sp8.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_256k_sp8.py>`_ |
| * - Llama2-7B |
| - 32 |
| - 1M |
| - 133.6 |
| - 153.9 |
| - `llama2_7b_full_alpaca_enzh_1M_sp16.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_7b/llama2_7b_full_alpaca_enzh_1M_sp16.py>`_ |
|
|
| .. list-table:: |
| :widths: 30 15 20 20 20 50 |
| :header-rows: 1 |
|
|
| * - 模型 |
| - GPUs |
| - 序列长度 |
| - TGS |
| - TFLOPs |
| - Config |
| * - Yi-34B-200K |
| - 32 |
| - 8k |
| - 485.1 |
| - 165.6 |
| - `yi_34b_200k_full_alpaca_enzh_8k_sp1.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/yi_34b/yi_34b_200k_full_alpaca_enzh_8k_sp1.py>`_ |
| * - Yi-34B-200K |
| - 32 |
| - 32k |
| - 491.5 |
| - 209.1 |
| - `yi_34b_200k_full_alpaca_enzh_32k_sp2.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/yi_34b/yi_34b_200k_full_alpaca_enzh_32k_sp2.py>`_ |
| * - Yi-34B-200K |
| - 32 |
| - 128k |
| - 251.1 |
| - 191.8 |
| - `yi_34b_200k_full_alpaca_enzh_128k_sp8.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/yi_34b/yi_34b_200k_full_alpaca_enzh_128k_sp8.py>`_ |
| * - Yi-34B-200K |
| - 32 |
| - 256k |
| - 119.7 |
| - 145.3 |
| - `yi_34b_200k_full_alpaca_enzh_256k_sp8.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/yi_34b/yi_34b_200k_full_alpaca_enzh_256k_sp8.py>`_ |
|
|
| .. list-table:: |
| :widths: 30 15 20 20 20 50 |
| :header-rows: 1 |
|
|
| * - 模型 |
| - GPUs |
| - 序列长度 |
| - TGS |
| - TFLOPs |
| - Config |
| * - Llama2-70B |
| - 32 |
| - 8k |
| - 216.8 |
| - 144.7 |
| - `llama2_70b_full_alpaca_enzh_8k_sp1.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_8k_sp1.py>`_ |
| * - Llama2-70B |
| - 32 |
| - 32k |
| - 300.9 |
| - 239.6 |
| - `llama2_70b_full_alpaca_enzh_32k_sp4.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_32k_sp4.py>`_ |
| * - Llama2-70B |
| - 32 |
| - 128k |
| - 144.7 |
| - 189.7 |
| - `llama2_70b_full_alpaca_enzh_128k_sp8.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_128k_sp8.py>`_ |
| * - Llama2-70B |
| - 32 |
| - 256k |
| - 63.8 |
| - 127.6 |
| - `llama2_70b_full_alpaca_enzh_256k_sp16.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_256k_sp16.py>`_ |
| * - Llama2-70B |
| - 64 |
| - 1M |
| - 21.8 |
| - 133.5 |
| - `llama2_70b_full_alpaca_enzh_1M_sp64.py <https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llama_speed_benchmark/llama2_70b/llama2_70b_full_alpaca_enzh_1M_sp64.py>`_ |
|
|
| .. note:: |
| 所有实验都会将 Alpaca 数据集拼接为最大长度。由于 Alpaca 数据集所含 |
| token 数较少，无法拼接成超长序列（如 1M |
| 长度），因此当序列长度较长时，会对 XTuner 代码进行如下修改： |
|
|
| .. code:: diff |
|
|
| |
| def build_origin_dataset(dataset, split): |
| ... |
| + |
| + dataset = concatenate_datasets([dataset for _ in range(6)]) |
| return dataset |
|
|
| def pack_dataset(dataset, max_length, use_varlen_attn, shuffle_before_pack, |
| map_num_proc): |
| dataset = dataset.map( |
| Packer(max_length, use_varlen_attn=use_varlen_attn), |
| batched=True, |
| - num_proc=map_num_proc |
| + batch_size=25000, |
| + num_proc=1 |
| ) |
| return dataset |
|
|
|
|
| .. note:: |
| 由于 Alpaca 数据量较小，因此做了第一处修改将数据集大小扩大了 6 |
| 倍，以保证拥有足够的训练 iter 数（保证速度测试的稳定性）。另外，由于 |
| Alpaca |
| 数据集每条数据的长度较短，因此在数据拼接的时候做了第二处修改以保证拥有足够多的数据，足以拼接为 |
| ``max_length`` 最大长度。 |
|
|
| .. |image1| image:: https://github.com/InternLM/xtuner/assets/41630003/c9c05dbd-0806-4fb2-9da9-62f04b150f7c |
| .. |image2| image:: https://github.com/InternLM/xtuner/assets/41630003/3ef6308c-595b-4624-b56d-a8737a1f2261 |
| .. |image3| image:: https://github.com/InternLM/xtuner/assets/41630003/ba16368e-e5f7-41eb-89ed-1140a8633134 |
|
|
