Upload README.md with huggingface_hub

README.md

@@ -1,131 +1,131 @@
-# BitCPM4 Continue Pretrain Example
+---
+license: apache-2.0
+language:
+- zh
+- en
+pipeline_tag: text-generation
+library_name: transformers
+---
+<div align="center">
+<img src="https://github.com/OpenBMB/MiniCPM/blob/main/assets/minicpm_logo.png?raw=true" width="500em" ></img> 
+</div>
+
+<p align="center">
+<a href="https://github.com/OpenBMB/MiniCPM/" target="_blank">GitHub Repo</a> |
+<a href="TODO_TECHNICAL_REPORT_LINK" target="_blank">Technical Report</a> 
+</p>
+<p align="center">
+👋 Join us on <a href="https://discord.gg/3cGQn9b3YM" target="_blank">Discord</a> and <a href="https://github.com/OpenBMB/MiniCPM/blob/main/assets/wechat.jpg" target="_blank">WeChat</a>
+</p>
+
+## Introduction
+
+BitCPM4-CANN-1B-unquantized is the **unquantized QAT training checkpoint** of the BitCPM4-CANN-1B model. This model stores the raw quantization-aware training (QAT) parameters **before** fake-quantizer fusion—the ternary fake quantizers are defined in `modeling.py` and applied during forward propagation.
+
+> ⚠️ **This model is NOT intended for direct inference.** It is designed as the starting point for fine-tuning BitCPM4-CANN. If you need a model for inference, please use the pseudo-quantized version: [openbmb/BitCPM4-CANN-0.5B](https://huggingface.co/openbmb/BitCPM4-CANN-0.5B).
+
+### Key Characteristics
+
+- 🎯 **Purpose**: Fine-tuning only. The model weights are un-fused QAT parameters with fake quantizers embedded in the `modeling.py` forward logic.
+- 🔬 **Ternary Fake Quantizer**: The forward pass in `modeling.py` contains ternary quantization logic (mapping weights to {-1, 0, 1} with group-wise scaling), which ensures the model continues learning under ternary constraints during fine-tuning.
+- 🔄 **Post-Training Conversion**: After fine-tuning, the model can be converted to pseudo-quantized format using the provided `qat-convert.py` script.
+
+## BitCPM4-CANN Model Family
+
+| Model | HuggingFace (Inference) | HuggingFace (Fine-tuning) |
+|-------|-------------------------|---------------------------|
+| BitCPM4-CANN-0.5B | [openbmb/BitCPM4-CANN-0.5B](https://huggingface.co/openbmb/BitCPM4-CANN-0.5B) | [openbmb/BitCPM4-CANN-0.5B-unquantized](https://huggingface.co/openbmb/BitCPM4-CANN-0.5B-unquantized) |
+| BitCPM4-CANN-1B | [openbmb/BitCPM4-CANN-1B](https://huggingface.co/openbmb/BitCPM4-CANN-1B) | [openbmb/BitCPM4-CANN-1B-unquantized](https://huggingface.co/openbmb/BitCPM4-CANN-1B-unquantized) |
+| BitCPM4-CANN-3B | [openbmb/BitCPM4-CANN-3B](https://huggingface.co/openbmb/BitCPM4-CANN-3B) | [openbmb/BitCPM4-CANN-3B-unquantized](https://huggingface.co/openbmb/BitCPM4-CANN-3B-unquantized) |
+| BitCPM4-CANN-8B | [openbmb/BitCPM4-CANN-8B](https://huggingface.co/openbmb/BitCPM4-CANN-8B) | [openbmb/BitCPM4-CANN-8B-unquantized](https://huggingface.co/openbmb/BitCPM4-CANN-8B-unquantized) |
 
-This project provides scripts for continue pretraining **BitCPM4-CANN-1B-unquantized**.
+## Usage
 
-## Environment Setup
+### Fine-tuning
 
-### Docker Image
+This model is designed for fine-tuning with frameworks that support custom modeling code. The critical requirement is that **the forward pass must go through the `modeling.py` file bundled with this model**, which contains the ternary fake quantizer logic. This ensures the model parameters remain compatible with ternary quantization constraints throughout fine-tuning.
 
-Use the following Huawei NPU image:
+#### Supported Fine-tuning Frameworks
 
-```
-swr.cn-south-1.myhuaweicloud.com/ascendhub/mindspeed-llm:openeuler22.03-mindspeed-llm-2.3.0-a3-arm
-```
+- **DeepSpeed** (recommended): See [example](./example)
+- **LLaMA Factory**: Supports custom model loading with `trust_remote_code=True`
+- **Other Frameworks**: Any framework that supports HuggingFace-compatible model loading with custom modeling code
+
+#### Important: Ensure Fake Quantizer is Active
 
-Other Huawei NPU images may also work but have not been fully tested.
+When fine-tuning, you **must** ensure:
+1. Load the model with `trust_remote_code=True` so that the custom `modeling.py` (containing the ternary quantizer) is used.
+2. The forward pass during training goes through the ternary quantizer defined in `modeling.py`—do NOT replace or bypass the model's forward logic.
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+path = 'openbmb/BitCPM4-CANN-1B-unquantized'
+tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    path,
+    torch_dtype=torch.bfloat16,
+    trust_remote_code=True
+)
+
+# Proceed with your fine-tuning pipeline (DeepSpeed, LLaMA Factory, etc.)
+# The ternary fake quantizer in modeling.py will be applied automatically during forward pass.
+```
 
-### Install Dependencies
+### Post-Fine-tuning Conversion
 
-After entering the container, install the Python dependencies:
+After fine-tuning is complete, use the `qat-convert.py` script to fuse the fake quantizer and produce the pseudo-quantized model weights that can be used for inference:
 
 ```bash
-pip install -r requirements.txt
+python qat-convert.py \
+    --input_bin <path-to-finetuned-pytorch.bin> \
+    --output <path-to-output-pseudo-quantized-pytorch.bin> \
+    --quant_type ternary \
+    --group_size -1
 ```
 
-Dependency list:
+The converted model can then be loaded for inference in the same way as [openbmb/BitCPM4-CANN-1B](https://huggingface.co/openbmb/BitCPM4-CANN-1B)—no special quantization libraries required.
 
-| Package | Version |
-| --- | --- |
-| transformers | 4.46.3 |
-| tokenizers | 0.20.3 |
-| accelerate | 1.1.1 |
-| deepspeed | 0.16.2 |
-| datasets | 3.1.0 |
-| safetensors | 0.4.5 |
-| pyarrow | 17.0.0 |
-| tensorboard | 2.18.0 |
+## Workflow Summary
 
-## Dataset
+```
+┌─────────────────────────────────┐
+│  BitCPM4-CANN-1B-unquantized  │   ← This model (QAT parameters + fake quantizer in modeling.py)
+└───────────────┬─────────────────┘
+                │
+                ▼  Fine-tune (DeepSpeed / LLaMA Factory / ...)
+┌─────────────────────────────────┐
+│   Fine-tuned pytorch.bin         │   ← Still contains un-fused QAT parameters
+└───────────────┬─────────────────┘
+                │
+                ▼  python qat-convert.py --quant_type ternary --group_size -1
+┌─────────────────────────────────┐
+│  Pseudo-quantized pytorch.bin    │   ← Ready for inference (same format as BitCPM4-CANN-0.5B)
+└─────────────────────────────────┘
+```
 
-The test dataset used is [C4-Pro](https://huggingface.co/datasets/gair-prox/c4-pro), stored in parquet format after downloading.
+## Technical Background
 
-## Usage
+BitCPM4-CANN uses a ternary quantizer that maps each weight group to {-1, 0, 1} scaled by a group-wise factor, trained with Straight-Through Estimator (STE) for gradient flow. The unquantized checkpoint preserves the full-precision latent weights alongside the quantizer parameters, allowing the model to continue learning under quantization constraints during fine-tuning.
 
-Modify the path configuration in `run.sh`:
+For full technical details, please refer to our [Technical Report](TODO_TECHNICAL_REPORT_LINK).
 
-```bash
-MODEL_PATH="/path/to/BitCPM4-CANN-1B-unquantized/"
-DATA_PATH="/path/to/c4-pro/data/your_file.parquet"
-```
+## Statement
+- As a language model, BitCPM4-CANN generates content by learning from a vast amount of text. 
+- However, it does not possess the ability to comprehend or express personal opinions or value judgments. 
+- Any content generated by BitCPM4-CANN does not represent the viewpoints or positions of the model developers. 
+- Therefore, when using content generated by BitCPM4-CANN, users should take full responsibility for evaluating and verifying it on their own.
 
-Then start training:
+## LICENSE
+- This repository and BitCPM4-CANN models are released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) License. 
 
-```bash
-bash run.sh
-```
+## Citation
+- Please cite our technical report if you find our work valuable.
 
-By default, the script trains for 500 steps using 8 devices, DeepSpeed ZeRO-2, and bf16 precision.
-
-## Training Results Reference
-
-Below is the loss curve for the first 100 steps (learning rate warmup covers the first 50 steps):
-
-| Step | Loss | Learning Rate | Epoch |
-| --- | --- | --- | --- |
-| 2 | 2.7920 | 1.60e-06 | 0.01 |
-| 4 | 2.8012 | 3.20e-06 | 0.02 |
-| 6 | 2.7984 | 4.80e-06 | 0.03 |
-| 8 | 2.7839 | 6.40e-06 | 0.04 |
-| 10 | 2.8084 | 8.00e-06 | 0.05 |
-| 12 | 2.8064 | 9.60e-06 | 0.06 |
-| 14 | 2.7994 | 1.12e-05 | 0.07 |
-| 16 | 2.7463 | 1.28e-05 | 0.08 |
-| 18 | 2.7580 | 1.44e-05 | 0.09 |
-| 20 | 2.8007 | 1.60e-05 | 0.10 |
-| 22 | 2.8916 | 1.76e-05 | 0.12 |
-| 24 | 2.8144 | 1.92e-05 | 0.13 |
-| 26 | 2.7723 | 2.08e-05 | 0.14 |
-| 28 | 2.7556 | 2.24e-05 | 0.15 |
-| 30 | 2.7414 | 2.40e-05 | 0.16 |
-| 32 | 2.7469 | 2.56e-05 | 0.17 |
-| 34 | 2.7428 | 2.72e-05 | 0.18 |
-| 36 | 2.7392 | 2.88e-05 | 0.19 |
-| 38 | 2.7132 | 3.04e-05 | 0.20 |
-| 40 | 2.7008 | 3.20e-05 | 0.21 |
-| 42 | 2.7547 | 3.36e-05 | 0.22 |
-| 44 | 2.7151 | 3.52e-05 | 0.23 |
-| 46 | 2.7119 | 3.68e-05 | 0.24 |
-| 48 | 2.7029 | 3.84e-05 | 0.25 |
-| 50 | 2.6803 | 4.00e-05 | 0.26 |
-| 52 | 2.6980 | 4.00e-05 | 0.27 |
-| 54 | 2.6923 | 4.00e-05 | 0.28 |
-| 56 | 2.7068 | 4.00e-05 | 0.29 |
-| 58 | 2.6965 | 4.00e-05 | 0.30 |
-| 60 | 2.7179 | 3.99e-05 | 0.31 |
-| 62 | 2.7119 | 3.99e-05 | 0.32 |
-| 64 | 2.7178 | 3.99e-05 | 0.33 |
-| 66 | 2.7069 | 3.99e-05 | 0.35 |
-| 68 | 2.6870 | 3.98e-05 | 0.36 |
-| 70 | 2.6775 | 3.98e-05 | 0.37 |
-| 72 | 2.7038 | 3.98e-05 | 0.38 |
-| 74 | 2.6924 | 3.97e-05 | 0.39 |
-| 76 | 2.7061 | 3.97e-05 | 0.40 |
-| 78 | 2.6929 | 3.96e-05 | 0.41 |
-| 80 | 2.6787 | 3.96e-05 | 0.42 |
-| 82 | 2.6749 | 3.95e-05 | 0.43 |
-| 84 | 2.6909 | 3.94e-05 | 0.44 |
-| 86 | 2.6893 | 3.94e-05 | 0.45 |
-| 88 | 2.6788 | 3.93e-05 | 0.46 |
-| 90 | 2.6831 | 3.92e-05 | 0.47 |
-| 92 | 2.7039 | 3.91e-05 | 0.48 |
-| 94 | 2.6619 | 3.91e-05 | 0.49 |
-| 96 | 2.6903 | 3.90e-05 | 0.50 |
-| 98 | 2.6993 | 3.89e-05 | 0.51 |
-| 100 | 2.6891 | 3.88e-05 | 0.52 |
-| 102 | 2.6739 | 3.87e-05 | 0.53 |
-
-> **Note:** BitCPM has its own training dataset and data mixture. It is expected that the loss continues to decrease when continue pretraining on open-source datasets.
-
-As shown in the table, the loss gradually decreases from ~2.79 to ~2.67, indicating a stable training process and that the model is learning normally.
-
-## File Description
-
-| File | Description |
-| --- | --- |
-| `train.py` | Training script based on HuggingFace Trainer + DeepSpeed |
-| `run.sh` | Launch script with training hyperparameter configuration |
-| `train_sft.py` | Supervised fine-tuning script based on HuggingFace Trainer + DeepSpeed |
-| `run_sft.sh` | Launch script for SFT with hyperparameter configuration |
-| `ds_config.json` | DeepSpeed ZeRO-3 configuration (with CPU offload) |
-| `ds_config_z2.json` | DeepSpeed ZeRO-2 configuration (used by default) |
-| `requirements.txt` | Python dependency list |
+```bibtex
+@article{bitcpm4cann,
+  title={{BitCPM-CANN}: Native 1.58-Bit Large Language Model Training on Ascend NPU},
+  author={BitCPM Team},
+  year={2026}
+}
+```