[CVPR 2026] Official LoRA-ATT checkpoints for the paper "Understanding and Enforcing Weight Disentanglement in Task Arithmetic".

[Paper]   [Code]   [OrthoReg repo]

Task arithmetic provides an efficient, training-free way to edit pre-trained models, yet lacks a fundamental theoretical explanation for its success. The existing concept of "weight disentanglement" describes the ideal outcome of non-interfering task composition but does not reveal its underlying cause. Crucially, what intrinsic properties of the pre-trained model ($\theta_0$) or the task vectors ($\tau_t$) enable this disentanglement remains underexplored. In this paper, we introduce Task-Feature Specialization (TFS), a model's ability to allocate distinct internal features to different tasks, as the fundamental principle. We first prove that TFS is a sufficient condition for weight disentanglement. More importantly, we find that TFS also gives rise to an observable geometric consequence: weight vector orthogonality. This positions TFS as the common cause for both the desired functional outcome (disentanglement) and a measurable geometric property (orthogonality). This relationship provides the key insight for our method: since the abstract TFS property is intractable to enforce directly, we can instead promote weight disentanglement by shaping its concrete geometric consequence, orthogonality. Therefore, we propose OrthoReg, a simple and effective regularization method that actively enforces an internal orthogonal structure on weight updates ($\Delta W$) that constitute $\tau_t$ during fine-tuning. And we theoretically prove that OrthoReg promotes disentanglement. Extensive experiments demonstrate that OrthoReg consistently and significantly enhances the performance of various task arithmetic methods.

The OrthoReg loss is applied to the equivalent dense weight update implied by each LoRA module:

$$\Delta W = \frac{\alpha}{r} \cdot B A$$

$$\mathcal{L}_{\text{ortho}} = \sum_l \left| (\Delta W^{(l)})^\top \Delta W^{(l)} - I \right|_F^2$$

The total training loss is:

$$\mathcal{L} = \mathcal{L}{\text{task}} + \lambda \cdot \mathcal{L}{\text{ortho}}$$

This repository contains fine-tuned LoRA-ATT checkpoints for ViT-B-32, ViT-B-16, and ViT-L-14 on all 8 tasks, covering the following finetuning modes:

Lambda values used per model:

Each mode directory is organized by dataset:

{mode}_{lr}_{model}/
├── head_CarsVal.pt           # linear classification head
├── head_DTDVal.pt
├── head_EuroSATVal.pt
├── head_GTSRBVal.pt
├── head_MNISTVal.pt
├── head_RESISC45Val.pt
├── head_SUN397Val.pt
├── head_SVHNVal.pt
├── {mode}_ft_accuracies.json # single-task accuracy results
├── {mode}_additions.json     # task addition results
├── CarsVal/
│   ├── {mode}_finetuned.pt   # fine-tuned model weights (merged task vector + θ₀)
│   └── {mode}_zeroshot.pt    # zero-shot reference weights
├── DTDVal/
...
└── SVHNVal/

All checkpoints use seed=1993, lr=1e-3, lora-rank=8, and lora-alpha=8.0 to match the paper's reported results.

Place the cloned folder as checkpoints_1993/ inside your code directory:

Generate zero-shot accuracies (required once before task addition/negation):

Evaluate single-task accuracy:

Evaluate task addition:

Evaluate task negation:

To evaluate the OrthoReg variant, replace --finetuning-mode loraatt with --finetuning-mode loraatt_ortho and add --ortho-lambda 10.0 (or 1.0 for ViT-L-14).

We evaluate on 8 image classification benchmarks: Cars · DTD · EuroSAT · GTSRB · MNIST · RESISC45 · SUN397 · SVHN

For dataset preparation, follow the instructions in the TTA repository.

If you find this work useful, please cite:

This codebase is built on top of Task Arithmetic, Tangent Task Arithmetic, and Attention-Only Fine-tuning. We thank the authors for releasing their code.