# BitCPM4 Continue Pretrain Example

This project provides scripts for continue pretraining **BitCPM4-CANN-1B-unquantized**.

## Environment Setup

### Docker Image

Use the following Huawei NPU image:

```
swr.cn-south-1.myhuaweicloud.com/ascendhub/mindspeed-llm:openeuler22.03-mindspeed-llm-2.3.0-a3-arm
```

Other Huawei NPU images may also work but have not been fully tested.

### Install Dependencies

After entering the container, install the Python dependencies:

```bash
pip install -r requirements.txt
```

Dependency list:

| Package | Version |
| --- | --- |
| transformers | 4.46.3 |
| tokenizers | 0.20.3 |
| accelerate | 1.1.1 |
| deepspeed | 0.16.2 |
| datasets | 3.1.0 |
| safetensors | 0.4.5 |
| pyarrow | 17.0.0 |
| tensorboard | 2.18.0 |

## Dataset

The test dataset used is [C4-Pro](https://huggingface.co/datasets/gair-prox/c4-pro), stored in parquet format after downloading.

## Usage

Modify the path configuration in `run.sh`:

```bash
MODEL_PATH="/path/to/BitCPM4-CANN-1B-unquantized/"
DATA_PATH="/path/to/c4-pro/data/your_file.parquet"
```

Then start training:

```bash
bash run.sh
```

By default, the script trains for 500 steps using 8 devices, DeepSpeed ZeRO-2, and bf16 precision.

## Training Results Reference

Below is the loss curve for the first 100 steps (learning rate warmup covers the first 50 steps):

| Step | Loss | Learning Rate | Epoch |
| --- | --- | --- | --- |
| 2 | 2.7920 | 1.60e-06 | 0.01 |
| 4 | 2.8012 | 3.20e-06 | 0.02 |
| 6 | 2.7984 | 4.80e-06 | 0.03 |
| 8 | 2.7839 | 6.40e-06 | 0.04 |
| 10 | 2.8084 | 8.00e-06 | 0.05 |
| 12 | 2.8064 | 9.60e-06 | 0.06 |
| 14 | 2.7994 | 1.12e-05 | 0.07 |
| 16 | 2.7463 | 1.28e-05 | 0.08 |
| 18 | 2.7580 | 1.44e-05 | 0.09 |
| 20 | 2.8007 | 1.60e-05 | 0.10 |
| 22 | 2.8916 | 1.76e-05 | 0.12 |
| 24 | 2.8144 | 1.92e-05 | 0.13 |
| 26 | 2.7723 | 2.08e-05 | 0.14 |
| 28 | 2.7556 | 2.24e-05 | 0.15 |
| 30 | 2.7414 | 2.40e-05 | 0.16 |
| 32 | 2.7469 | 2.56e-05 | 0.17 |
| 34 | 2.7428 | 2.72e-05 | 0.18 |
| 36 | 2.7392 | 2.88e-05 | 0.19 |
| 38 | 2.7132 | 3.04e-05 | 0.20 |
| 40 | 2.7008 | 3.20e-05 | 0.21 |
| 42 | 2.7547 | 3.36e-05 | 0.22 |
| 44 | 2.7151 | 3.52e-05 | 0.23 |
| 46 | 2.7119 | 3.68e-05 | 0.24 |
| 48 | 2.7029 | 3.84e-05 | 0.25 |
| 50 | 2.6803 | 4.00e-05 | 0.26 |
| 52 | 2.6980 | 4.00e-05 | 0.27 |
| 54 | 2.6923 | 4.00e-05 | 0.28 |
| 56 | 2.7068 | 4.00e-05 | 0.29 |
| 58 | 2.6965 | 4.00e-05 | 0.30 |
| 60 | 2.7179 | 3.99e-05 | 0.31 |
| 62 | 2.7119 | 3.99e-05 | 0.32 |
| 64 | 2.7178 | 3.99e-05 | 0.33 |
| 66 | 2.7069 | 3.99e-05 | 0.35 |
| 68 | 2.6870 | 3.98e-05 | 0.36 |
| 70 | 2.6775 | 3.98e-05 | 0.37 |
| 72 | 2.7038 | 3.98e-05 | 0.38 |
| 74 | 2.6924 | 3.97e-05 | 0.39 |
| 76 | 2.7061 | 3.97e-05 | 0.40 |
| 78 | 2.6929 | 3.96e-05 | 0.41 |
| 80 | 2.6787 | 3.96e-05 | 0.42 |
| 82 | 2.6749 | 3.95e-05 | 0.43 |
| 84 | 2.6909 | 3.94e-05 | 0.44 |
| 86 | 2.6893 | 3.94e-05 | 0.45 |
| 88 | 2.6788 | 3.93e-05 | 0.46 |
| 90 | 2.6831 | 3.92e-05 | 0.47 |
| 92 | 2.7039 | 3.91e-05 | 0.48 |
| 94 | 2.6619 | 3.91e-05 | 0.49 |
| 96 | 2.6903 | 3.90e-05 | 0.50 |
| 98 | 2.6993 | 3.89e-05 | 0.51 |
| 100 | 2.6891 | 3.88e-05 | 0.52 |
| 102 | 2.6739 | 3.87e-05 | 0.53 |

> **Note:** BitCPM has its own training dataset and data mixture. It is expected that the loss continues to decrease when continue pretraining on open-source datasets.

As shown in the table, the loss gradually decreases from ~2.79 to ~2.67, indicating a stable training process and that the model is learning normally.

## File Description

| File | Description |
| --- | --- |
| `train.py` | Training script based on HuggingFace Trainer + DeepSpeed |
| `run.sh` | Launch script with training hyperparameter configuration |
| `train_sft.py` | Supervised fine-tuning script based on HuggingFace Trainer + DeepSpeed |
| `run_sft.sh` | Launch script for SFT with hyperparameter configuration |
| `ds_config.json` | DeepSpeed ZeRO-3 configuration (with CPU offload) |
| `ds_config_z2.json` | DeepSpeed ZeRO-2 configuration (used by default) |
| `requirements.txt` | Python dependency list |