quantization_metric/alphalora/README.md

AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality

Welcome to the official GitHub repository for our paper, "AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality."

1. Clone the repository

   git clone https://github.com/peijunallin/alphalora.git
   cd alphalora
   

2. Install dependencies

   conda create -n alphalora python=3.10 -y
   conda activate alphalora
   pip install -r requirements.txt
   

3. Determine number of experts and Top K

   Run the expert_number.py script to get the number of experts and the top_k parameters:

   CUDA_VISIBLE_DEVICES=3 python expert_number.py \
   --model "mistralai/Mistral-7B-v0.1" \
   --target_sum 160 \
   --beta 2.5
   

4. Train on six datasets

   bash run_all.sh
   

   Before running the script, ensure to adjust the following hyperparameters in run_all.sh:

   Hyperparameters	Description
   base_model	The path to the base model.
   root_data_path	The path to the six datasets.
   number_experts	The number of experts for each layer (32 numbers).
   top_k	The top K value for each layer (32 numbers).
   output_dir	The directory path to save the LoRA experts' weights.

5. Evaluate on six datasets

   Ensure that mola_weights corresponds to the output_dir used during training, and keep the expert number and top_K settings consistent.

   bash eval_all.sh
   

Acknowlegements

Our code is based on MoLA and TempBalance.