# AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality

Welcome to the official GitHub repository for our paper, "[AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality](https://arxiv.org/html/2410.10054v1)."










1. **Clone the repository**

   ```bash
   git clone https://github.com/peijunallin/alphalora.git
   cd alphalora
   ```

2. **Install dependencies**

   ```bash
   conda create -n alphalora python=3.10 -y
   conda activate alphalora
   pip install -r requirements.txt
   ```

3. **Determine number of experts and Top K**

   Run the `expert_number.py` script to get the number of experts and the top_k parameters:

   ```bash
   CUDA_VISIBLE_DEVICES=3 python expert_number.py \
   --model "mistralai/Mistral-7B-v0.1" \
   --target_sum 160 \
   --beta 2.5
   ```

4. **Train on six datasets**


   ```bash
   bash run_all.sh
   ```

   Before running the script, ensure to adjust the following hyperparameters in `run_all.sh`:

   | Hyperparameters          | Description                                                       |
   |--------------------------|-------------------------------------------------------------------|
   | `base_model`             | The path to the base model.                                       |
   | `root_data_path`         | The path to the six datasets.                                     |
   | `number_experts`         | The number of experts for each layer (32 numbers).        |
   | `top_k`                  | The top K value for each layer (32 numbers).              |
   | `output_dir`             | The directory path to save the LoRA experts' weights.             |

5. **Evaluate on six datasets**

   Ensure that `mola_weights` corresponds to the `output_dir` used during training, and keep the expert number and top_K settings consistent.

   ```bash
   bash eval_all.sh
   ```
   

## Acknowlegements

Our code is based on [MoLA](https://github.com/gcyzsl/mola) and [TempBalance](https://github.com/yefanzhou/tempbalance).