--- base_model: - openai/clip-vit-base-patch32 - openai/clip-vit-base-patch16 - openai/clip-vit-large-patch14 license: mit pipeline_tag: image-classification tags: - clip - task-arithmetic - model-merging - fine-tuning - orthogonal-regularization --- # Understanding and Enforcing Weight Disentanglement in Task Arithmetic [CVPR 2026] Official checkpoints for the paper **"Understanding and Enforcing Weight Disentanglement in Task Arithmetic"**. **Authors**: [Shangge Liu](https://huggingface.co/gezi2333), Yuehan Yin, Lei Wang, Qi Fan, Yinghuan Shi, Wenbin Li, Yang Gao, and Dacheng Tao. [[Paper](https://arxiv.org/abs/2604.17078)]   [[Code](https://github.com/RL-MIND/OrthoReg)] --- ## 🎯 Abstract Task arithmetic provides an efficient, training-free way to edit pre-trained models, yet lacks a fundamental theoretical explanation for its success. The existing concept of "weight disentanglement" describes the ideal outcome of non-interfering task composition but does not reveal its underlying cause. Crucially, what intrinsic properties of the pre-trained model ($\theta_0$) or the task vectors ($\tau_t$) enable this disentanglement remains underexplored. In this paper, we introduce Task-Feature Specialization (TFS), a model's ability to allocate distinct internal features to different tasks, as the fundamental principle. We first prove that TFS is a sufficient condition for weight disentanglement. More importantly, we find that TFS also gives rise to an observable geometric consequence: weight vector orthogonality. This positions TFS as the common cause for both the desired functional outcome (disentanglement) and a measurable geometric property (orthogonality). This relationship provides the key insight for our method: since the abstract TFS property is intractable to enforce directly, we can instead promote weight disentanglement by shaping its concrete geometric consequence, orthogonality. Therefore, we propose OrthoReg, a simple and effective regularization method that actively enforces an internal orthogonal structure on weight updates ($\Delta W$) that constitute $\tau_t$ during fine-tuning. And we theoretically prove that OrthoReg promotes disentanglement. Extensive experiments demonstrate that OrthoReg consistently and significantly enhances the performance of various task arithmetic methods. ### ✨ Key Contributions - πŸ“ **Theory**: We identify TFS as a sufficient condition for weight disentanglement, and WVO as its geometric consequence, providing the first principled explanation for task arithmetic. - πŸ”§ **Method (OrthoReg)**: A simple regularization term added to the fine-tuning loss that enforces column-wise orthogonality on Ξ”W, for which we prove theoretical efficacy. - πŸ”— **Connection to TTA**: We show that OrthoReg and Tangent Task Arithmetic (TTA) share the same underlying mechanism (i.e. inter-task vector orthogonality), but OrthoReg achieves this more efficiently. - πŸ“Š **Experiments**: Consistent and significant improvements over Non-linear FT, TTA, ATT-FT, LoRA-ATT across ViT-B-32, ViT-B-16, and ViT-L-14. --- ### The OrthoReg Loss The total loss adds a regularization term to the standard task objective: $$\mathcal{L} = \mathcal{L}_{\text{task}}(\theta_0 + \Delta\theta) + \lambda \cdot \mathcal{L}_{\text{ortho}}(\Delta\theta)$$ $$\mathcal{L}_{\text{ortho}}(\Delta\theta) = \sum_l \left\|(\Delta W^{(l)})^\top \Delta W^{(l)} - I\right\|_F^2$$ --- ## πŸ“ Checkpoint Structure This repository contains fine-tuned checkpoints for **ViT-B-32, ViT-B-16, and ViT-L-14** on all 8 tasks, covering the following finetuning modes: | Directory | Mode | Description | |---|---|---| | `standard_1e-05_{model}/` | `standard` | Non-linear full fine-tuning (baseline) | | `linear_1e-05_{model}/` | `linear` | TTA β€” tangent space fine-tuning (baseline) | | `linear-2_1e-05_{model}/` | `linear-2` | ATT-FT β€” attention-only fine-tuning (baseline) | | `linear_ortho_1e-05_lambda1.0_{model}/` | `linear_ortho` | TTA + OrthoReg | | `ViT-B-32/`, `ViT-B-16/`, `ViT-L-14/` | β€” | Pre-trained CLIP base model weights | Each mode directory is organized by dataset: ``` {mode}_{lr}_{model}/ β”œβ”€β”€ head_CarsVal.pt # linear classification head β”œβ”€β”€ head_DTDVal.pt β”œβ”€β”€ head_EuroSATVal.pt β”œβ”€β”€ head_GTSRBVal.pt β”œβ”€β”€ head_MNISTVal.pt β”œβ”€β”€ head_RESISC45Val.pt β”œβ”€β”€ head_SUN397Val.pt β”œβ”€β”€ head_SVHNVal.pt β”œβ”€β”€ CarsVal/ β”‚ β”œβ”€β”€ {mode}_finetuned.pt # fine-tuned model weights (task vector + ΞΈβ‚€) β”‚ └── {mode}_zeroshot.pt # zero-shot reference weights β”œβ”€β”€ DTDVal/ ... └── SVHNVal/ ``` All checkpoints use **seed=1993** and **lr=1e-5** to match the paper's reported results. --- ## πŸš€ Usage ### Step 1 β€” Clone this repository ```bash git lfs install git clone https://huggingface.co/gezi2333/OrthoReg-checkpoints ``` Place the cloned folder as `OrthoReg/checkpoints_1993/` inside your code directory: ```bash mv OrthoReg-checkpoints/* OrthoReg/checkpoints_1993/ ``` ### Step 2 β€” Install the codebase ```bash git clone https://github.com/RL-MIND/OrthoReg cd OrthoReg conda env create conda activate tangent-arithmetic export PYTHONPATH="$PYTHONPATH:$PWD" ``` ### Step 3 β€” Run evaluation Evaluate single-task accuracy: ```bash python src/eval_single_task.py \ --model ViT-B-32 \ --finetuning-mode linear_ortho \ --ortho-lambda 1.0 \ --lr 1e-5 \ --seed 1993 \ --data-location /path/to/datasets/ ``` Evaluate task addition: ```bash python src/eval_task_addition.py \ --model ViT-B-32 \ --finetuning-mode linear_ortho \ --ortho-lambda 1.0 \ --lr 1e-5 \ --seed 1993 \ --data-location /path/to/datasets/ ``` Evaluate task negation: ```bash python src/eval_task_negation.py \ --model ViT-B-32 \ --finetuning-mode linear_ortho \ --ortho-lambda 1.0 \ --lr 1e-5 \ --seed 1993 \ --data-location /path/to/datasets/ ``` > Run `eval_single_task` with `--finetuning-mode none --ortho-lambda 0` first to generate `zeroshot_accuracies.json`, which is required as the reference for normalized accuracy. --- ## πŸ“¦ Datasets We evaluate on 8 image classification benchmarks: **Cars Β· DTD Β· EuroSAT Β· GTSRB Β· MNIST Β· RESISC45 Β· SUN397 Β· SVHN** For dataset preparation, follow the instructions in the [TTA repository](https://github.com/gortizji/tangent_task_arithmetic#datasets). --- ## πŸ“ Citation If you find this work useful, please cite: ```bibtex @inproceedings{liu2026orthoreg, title = {Understanding and Enforcing Weight Disentanglement in Task Arithmetic}, author = {Liu, Shangge and Yin, Yuehan and Wang, Lei and Fan, Qi and Shi, Yinghuan and Li, Wenbin and Gao, Yang and Tao, Dacheng}, booktitle = {CVPR}, year = {2026} } ``` --- ## πŸ“¬ Acknowledgements This codebase is built on top of [Task Arithmetic](https://github.com/mlfoundations/task_vectors), [Tangent Task Arithmetic](https://github.com/gortizji/tangent_task_arithmetic), and [Attention-Only Fine-tuning](https://github.com/kyrie-23/linear_task_arithmetic). We thank the authors for releasing their code.