You can deploy this model and start earning money today!

rainstonee/
CDKA

2602.01267

Views

No views yet

CDKA: Component Designed Kronecker Adapters

Official PyTorch implementation of Diving into Kronecker Adapters: Component Design Matters.

CDKA is a parameter-efficient fine-tuning framework for large-scale models. It studies how the component design of Kronecker adapters affects adapter capacity and downstream performance. In particular, CDKA exposes the component dimensions and the number of Kronecker components as controllable hyperparameters:

$$ \Delta W = \sum_{i=1}^{r} B^{(i)} \otimes A^{(i)}, $$

where r1, r2, and r determine the shape and number of Kronecker components.

Installation

1. Clone the repository

bash

git clone https://github.com/rainstonee/CDKA.git
cd CDKA

2. Create a Python environment

bash

conda create -n cdka python=3.10 -y
conda activate cdka

3. Install dependencies

bash

pip install --upgrade pip
pip install -r requirements.txt

4. Install the modified PEFT package

This repository includes a modified PEFT implementation in peft.zip. The custom PEFT version is required because CDKA extends the standard LoRA configuration with Kronecker-adapter-specific arguments such as r1, r2, and r.

bash

unzip peft.zip
pip install -e peft

Training with CDKA

Run the example script:

bash run.sh

The default example runs the Llama-2-7B model on meta_math dataset with PEFT enabled and Kronecker adapter hyperparameters specified through Hydra overrides.

The script is equivalent to:

bash

CUDA_VISIBLE_DEVICES=0 python run_exp.py \
  +model=llama \
  +peft=all \
  +init=default \
  +dataset_name=meta_math \
  +seed=333 \
  ++peft.lora_r1=2 \
  ++peft.lora_r2=2 \
  ++peft.lora_r=8 \
  ++peft.lora_alpha=64

Recommended default component configuration

For most CDKA experiments, we recommend starting with the following default component configuration:

bash

++peft.lora_r1=2 \
++peft.lora_r2=8 \
++peft.lora_r=4 \
++peft.lora_alpha=64

This setting follows the main component-design principle of CDKA: use a small r1, a large r2, and a moderate number of Kronecker components r. In practice, this provides a balanced default configuration before task-specific tuning.

Citation

If you find this repository useful, please cite:

bibtex

@article{bai2026diving,
  title={Diving into Kronecker Adapters: Component Design Matters},
  author={Bai, Jiayu and Yu, Danchen and Liao, Zhenyu and Hou, TianQi and Zhou, Feng and Qiu, Robert C and Ling, Zenan},
  journal={arXiv preprint arXiv:2602.01267},
  year={2026}
}

Acknowledgement

This codebase is built on LoRA-GA. We thank the authors of LoRA-GA for releasing their implementation.