README.md

# CDKA: Component Designed Kronecker Adapters

Official PyTorch implementation of **Diving into Kronecker Adapters: Component Design Matters**.

CDKA is a parameter-efficient fine-tuning framework for large-scale models. It studies how the component design of Kronecker adapters affects adapter capacity and downstream performance. In particular, CDKA exposes the component dimensions and the number of Kronecker components as controllable hyperparameters:

$$
\Delta W = \sum_{i=1}^{r} B^{(i)} \otimes A^{(i)},
$$

where `r1`, `r2`, and `r` determine the shape and number of Kronecker components.


## Installation

### 1. Clone the repository

```bash
git clone https://github.com/rainstonee/CDKA.git
cd CDKA
```

### 2. Create a Python environment

```bash
conda create -n cdka python=3.10 -y
conda activate cdka
```

### 3. Install dependencies

```bash
pip install --upgrade pip
pip install -r requirements.txt
```

### 4. Install the modified PEFT package

This repository includes a modified PEFT implementation in `peft.zip`. The custom PEFT version is required because CDKA extends the standard LoRA configuration with Kronecker-adapter-specific arguments such as `r1`, `r2`, and `r`.

```bash
unzip peft.zip
pip install -e peft
```

## Training with CDKA

Run the example script:

```bash
bash run.sh
```

The default example runs the Llama-2-7B model on `meta_math` dataset with PEFT enabled and Kronecker adapter hyperparameters specified through Hydra overrides.

The script is equivalent to:

```bash
CUDA_VISIBLE_DEVICES=0 python run_exp.py \
  +model=llama \
  +peft=all \
  +init=default \
  +dataset_name=meta_math \
  +seed=333 \
  ++peft.lora_r1=2 \
  ++peft.lora_r2=2 \
  ++peft.lora_r=8 \
  ++peft.lora_alpha=64
```

### Recommended default component configuration

For most CDKA experiments, we recommend starting with the following default component configuration:

```bash
++peft.lora_r1=2 \
++peft.lora_r2=8 \
++peft.lora_r=4 \
++peft.lora_alpha=64
```

This setting follows the main component-design principle of CDKA: use a small `r1`, a large `r2`, and a moderate number of Kronecker components `r`. In practice, this provides a balanced default configuration before task-specific tuning.


## Citation

If you find this repository useful, please cite:

```bibtex
@article{bai2026diving,
  title={Diving into Kronecker Adapters: Component Design Matters},
  author={Bai, Jiayu and Yu, Danchen and Liao, Zhenyu and Hou, TianQi and Zhou, Feng and Qiu, Robert C and Ling, Zenan},
  journal={arXiv preprint arXiv:2602.01267},
  year={2026}
}
```

## Acknowledgement

This codebase is built on [LoRA-GA](https://github.com/Outsider565/LoRA-GA). We thank the authors of LoRA-GA for releasing their implementation.