CDKA: Component Designed Kronecker Adapters

Official PyTorch implementation of Diving into Kronecker Adapters: Component Design Matters.

CDKA is a parameter-efficient fine-tuning framework for large-scale models. It studies how the component design of Kronecker adapters affects adapter capacity and downstream performance. In particular, CDKA exposes the component dimensions and the number of Kronecker components as controllable hyperparameters:

$\Delta W = \sum_{i=1}^{r} B^{(i)} \otimes A^{(i)},$

where r1, r2, and r determine the shape and number of Kronecker components.

Installation

1. Clone the repository

git clone https://github.com/rainstonee/CDKA.git
cd CDKA

2. Create a Python environment

conda create -n cdka python=3.10 -y
conda activate cdka

3. Install dependencies

pip install --upgrade pip
pip install -r requirements.txt

4. Install the modified PEFT package

This repository includes a modified PEFT implementation in peft.zip. The custom PEFT version is required because CDKA extends the standard LoRA configuration with Kronecker-adapter-specific arguments such as r1, r2, and r.

unzip peft.zip
pip install -e peft

Training with CDKA

Run the example script:

bash run.sh

The default example runs the Llama-2-7B model on meta_math dataset with PEFT enabled and Kronecker adapter hyperparameters specified through Hydra overrides.

The script is equivalent to:

CUDA_VISIBLE_DEVICES=0 python run_exp.py \
  +model=llama \
  +peft=all \
  +init=default \
  +dataset_name=meta_math \
  +seed=333 \
  ++peft.lora_r1=2 \
  ++peft.lora_r2=2 \
  ++peft.lora_r=8 \
  ++peft.lora_alpha=64

Recommended default component configuration

For most CDKA experiments, we recommend starting with the following default component configuration:

++peft.lora_r1=2 \
++peft.lora_r2=8 \
++peft.lora_r=4 \
++peft.lora_alpha=64

This setting follows the main component-design principle of CDKA: use a small r1, a large r2, and a moderate number of Kronecker components r. In practice, this provides a balanced default configuration before task-specific tuning.

Citation

If you find this repository useful, please cite:

@article{bai2026diving,
  title={Diving into Kronecker Adapters: Component Design Matters},
  author={Bai, Jiayu and Yu, Danchen and Liao, Zhenyu and Hou, TianQi and Zhou, Feng and Qiu, Robert C and Ling, Zenan},
  journal={arXiv preprint arXiv:2602.01267},
  year={2026}
}

Acknowledgement

This codebase is built on LoRA-GA. We thank the authors of LoRA-GA for releasing their implementation.