section: A APPENDIX / SUPPLEMENTARY MATERIAL

type: Text

text: The appendix is organized as follows:


type: ListGroup

text: App. B discusses the main limitations of our approach, including memory requirements and curvature-estimation challenges. App. C provides a derivation and a formal bound on the approximation error introduced when merging multiple KFAC factors using the Kronecker heuristic. App. D presents additional plots illustrating the disentanglement error. App. E details the implementation of our methods, with separate discussions for the vision and text domains. App. F reports additional experiments. These include: Core analyses: * per-task performance analysis, * alpha-sweep robustness study (App. F.2), * ablation on the regularization coefficient (App. F.3), * evaluation of a shared KFAC computed on a reference dataset (App. F.4), * task-localization analysis under non-linear fine-tuning (App. F.5); extended experiments: * analysis of task localization under memory-efficient KFAC approximations, including block-based, SVD-based, pruning, and 8-bit quantized variants (App. F.6), * additional results on more challenging vision domains using a class-incremental partitioning protocol (App. F.7). App. G provides a concise overview of prior work on linearized fine-tuning and its recent developments.


section: B LIMITATIONS

text: KFAC requires storing the Kronecker matrices in GPU memory – two per layer, each with quadratic complexity in the number of units. For large models this can become problematic, suggesting that alternative strategies based on matrix compression or structured Kronecker factors (Grosse et al., 2023; Lin et al., 2024) should be explored. While we combine the well-established KFAC with an accumulation strategy, designing curvature approximations that can easily be merged without sacrificing accuracy may be worth exploring in the future. Moreover, our experiments in the text domain indicate room for improvement, raising the question of whether more sophisticated techniques for curvature estimation could further enhance Task Arithmetic.


section: C APPROXIMATION ERROR OF THE MERGED KFAC FACTORS

text: For clarity, we focus on a single layer and assume all layers contribute equally, omitting the task weights \lambda_t . Let \{A_t\}_{t=1}^T and \{B_t\}_{t=1}^T denote the KFAC factors associated with the tasks involved in the merge. The heuristic used in Eq. 8 replaces the sum of Kronecker products with the Kronecker product between aggregated factors


type: Equation

text: \sum_{t=1}^{T} B_t \otimes A_t \approx \left(\sum_{t=1}^{T} B_t\right) \otimes \left(\frac{1}{T} \sum_{t=1}^{T} A_t\right). \tag{9}


text: We now provide a simple bound that quantifies the error introduced by this approximation. To do so, we define the empirical means and the deviations from the mean


text: \bar{A} = \frac{1}{T} \sum_{t=1}^{T} A_t, \qquad \bar{B} = \frac{1}{T} \sum_{t=1}^{T} B_t, \qquad \Delta A_t = A_t - \bar{A}, \qquad \Delta B_t = B_t - \bar{B}. (10)


text: Note that, by construction, \sum_t \Delta A_t = \sum_t \Delta B_t = 0 . Substituting A_t = \bar{A} + \Delta A_t and B_t = \bar{B} + \Delta B_t into the left-hand side of Eq. (9) yields


text: \sum_{t=1}^{T} B_t \otimes A_t = \sum_{t=1}^{T} (\bar{B} + \Delta B_t) \otimes (\bar{A} + \Delta A_t) (11)


text: = \sum_{t=1}^{T} \left( \bar{B} \otimes \bar{A} + \bar{B} \otimes \Delta A_{t} + \Delta B_{t} \otimes \bar{A} + \Delta B_{t} \otimes \Delta A_{t} \right) (12)


text: =\underbrace{\sum_{t=1}^{T} \bar{B} \otimes \bar{A}}_{T \, \bar{B} \otimes \bar{A}} + \underbrace{\bar{B} \otimes \sum_{t=1}^{T} \Delta A_{t}}_{=0} + \underbrace{\left(\sum_{t=1}^{T} \Delta B_{t}\right) \otimes \bar{A}}_{=0} + \sum_{t=1}^{T} \Delta B_{t} \otimes \Delta A_{t} \quad (13)


text: = T \bar{B} \otimes \bar{A} + \sum_{t=1}^{T} \Delta B_t \otimes \Delta A_t. \tag{14}


text: Substituting A_t = \bar{A} + \Delta A_t and B_t = \bar{B} + \Delta B_t into the right-hand side of Eq. (9), instead, yields


text: \left(\sum_{t=1}^{T} B_{t}\right) \otimes \left(\sum_{t=1}^{T} A_{t}\right) = T^{2} \bar{B} \otimes \bar{A}. \tag{15}


text: Hence the approximation error is


text: E := \sum_{t=1}^{T} B_t \otimes A_t - \frac{1}{T} \left( \sum_{t=1}^{T} B_t \right) \otimes \left( \sum_{t=1}^{T} A_t \right) = \sum_{t=1}^{T} \Delta B_t \otimes \Delta A_t.


text: Error bound. Using the Frobenius norm and the property ||X \otimes Y||_F = ||X||_F ||Y||_F , we obtain


text: ||E||_F \le \sum_{t=1}^T ||\Delta B_t||_F ||\Delta A_t||_F \le \sqrt{\sum_{t=1}^T ||\Delta B_t||_F^2} \sqrt{\sum_{t=1}^T ||\Delta A_t||_F^2}. (16)


text: Defining the deviations (standard deviations in matrix space), we obtain:


text: \sigma_A := \sqrt{\frac{1}{T} \sum_{t=1}^{T} \|\Delta A_t\|_F^2}, \qquad \sigma_B := \sqrt{\frac{1}{T} \sum_{t=1}^{T} \|\Delta B_t\|_F^2}, \tag{17}


text: we finally obtain the compact bound


text: ||E||_F \le T \sigma_A \sigma_B. \tag{18}


text: Interpretation. The approximation error is proportional to the product of the variations of the KFAC factors across tasks. When the task-specific factors (A_t, B_t) cluster tightly around their means, both \sigma_A and \sigma_B are small, yielding a small deviation between the true mixed KFAC term and its merged approximation. This situation is particularly likely to occur when the matrices are estimated from a fixed pre-trained backbone such as CLIP: since the underlying feature extractor remains unchanged across tasks, the induced activation and gradient statistics tend to vary only mildly. As a result, the corresponding KFAC factors exhibit limited task-to-task fluctuation, further justifying the accuracy of the merged approximation.


section: D ADDITIONAL PLOTS ON WEIGHT DISENTANGLEMENT

text: In Fig. 9 we report the disentanglement error, a metric introduced by Ortiz-Jimenez et al. (2023):


text: \xi(\alpha_1, \alpha_2) = \sum_{t=1}^{2} \mathbb{E}_{\boldsymbol{x} \sim \mu_t} \left[ \operatorname{dist} \left( f(\boldsymbol{x}; \boldsymbol{\theta}_0 + \alpha_t \boldsymbol{\tau}_t), f(\boldsymbol{x}; \boldsymbol{\theta}_0 + \alpha_1 \boldsymbol{\tau}_1 + \alpha_2 \boldsymbol{\tau}_2) \right) \right], \tag{19}


type: FigureGroup

text: Figure 9: Visualization of weight disentanglement (Ortiz-Jimenez et al., 2023) in ViT-B/16. Non linear fine-tuning Ilharco et al. (2022), Linear fine-tuning Ortiz-Jimenez et al. (2023), Attention-Only fine-tuning Jin et al. (2025), Linear fine-tuning with KFAC regularization.


text: where \operatorname{dist}(y_1,y_2)=\mathbb{1}(y_1\neq y_2) . When \xi(\alpha_1,\alpha_2)=0 , tasks \tau_1 and \tau_2 merge without interference for the corresponding values of \alpha_1 and \alpha_2 .


text: As shown in the plots, linearized fine-tuning substantially improves the disentanglement of task vectors. This property is further enhanced under our regularization regime, where only a few darker regions remain, mostly for \alpha>1 , a setting that is never used in practice. Notably, in our experiments the disentanglement error is consistently close to zero along the diagonals, which is the most relevant case, since in the literature the common choice is \alpha_1=\alpha_2=\cdots=\alpha_n .


section: E IMPLEMENTATION DETAILS

text: The GGN information matrices were estimated using a single Monte Carlo sample and computed on 33% of the available training data. However, our empirical analysis showed that sampling only 250-300 training points is sufficient to obtain a reliable estimation of the curvature matrix.


text: KFAC factors are estimated for all layers involving linear projections in the model – namely, attention and feed-forward projections. In contrast, for LayerNorm parameters and the class token, whose scaling, bias, and token parameters grow linearly rather than quadratically with the embedding dimension, computing the full GGN matrix is tractable. For these components, we therefore use the original, approximation-free GGN instead of its KFAC approximation.


text: Figure 10: Impact of training and regularization choices on vision tasks (absolute accuracy). Top: linearized regime, compared against the diagonal approximation. Bottom: non-linear regime, compared against attention-only fine-tuning.


text: The KFAC regularization loss is applied to all fine-tuned layers. Empirically, we found it beneficial to rescale the regularization weight of the last layer of the CLIP visual encoder by a factor of 0.1.


section: E.1 VISION DOMAIN

text: We leverage the 8 Vision protocol (Ilharco et al., 2022) and conduct experiments on Stanford Cars (Krause et al., 2013) , DTD (Cimpoi et al., 2014) , EuroSAT (Helber et al., 2019) , GTSRB (Stal lkamp et al., 2011) , MNIST (LeCun et al., 2002) , RESISC45 (Cheng et al., 2017) , SUN397 (Xiao et al., 2016) , and SVHN (Netzer et al., 2011) . For training the task vectors, we followed the setup of previous works Ilharco et al. (2022) ; Ortiz-Jimenez et al. (2023) ; Yoshida et al. (2025) , adopting a batch size of 128. We used the AdamW optimizer with a learning rate of 3 × 10 − 4 , weight decay of 0.1, and a cosine annealing learning rate scheduler. Unlike prior approaches, we did not apply gradient clipping during training. The regularization term in the loss was weighted by λ = 100 for ViT-B/32, λ = 500 for ViT-B/16, and λ = 2000 for ViT-L/14.


text: Compared to previous work, we employed a higher learning rate. Since our formulation includes an explicit regularization term in the loss, this allowed us to increase the learning rate without introducing interference across tasks.


section: E.2 TEXT DOMAIN

text: We follow the 6NLI benchmark (Stoica et al., 2025; Panariello et al., 2025) , including SNLI (Bow man et al., 2015) , MultiNLI (Williams et al., 2018) , and SICK (Marelli et al., 2014) which are three-way classification tasks where the relation between a premise and a hypothesis must be identified as entailment, contradiction, or neutral. Additionally, SciTail (Khot et al., 2018) , RTE (Wang et al., 2018) , and QNLI (Wang et al., 2018) are binary entailment tasks, and therefore fine-tuning


type: TableGroup

text: Table 4: Comparison of different merging strategies in the linear fine-tuning regime, with and without KFAC regularization. Results are reported for \alpha=1.0 and the best-performing \alpha . Method 01 ViT -B/32 ViT '-B/16 Wethod \alpha Abs. Norm. Abs. Norm. Linear FT + TIES Yadav et al. (2023) 1.0 77.1 87.8 80.1 88.7 Linear FT + TIES Taday et al. (2023) Best 77.2 87.8 80.3 88.9 Linear FT + TSV Gargiulo et al. (2025) 1.0 84.2 96.3 86.7 96.5 Linear 11 + 13 v Gargiulo et al. (2023) Best 84.2 96.3 86.8 96.6 Linear FT + ISO Marczak et al. (2025) 1.0 83.8 95.8 86.9 96.7 Linear 1-1 + 150 Marczak et al. (2023) Best 84.4 96.4 87.3 97.1 TAK, Ours + TIES Yadav et al. (2023) 1.0 81.8 92.7 86.6 95.9 TAK, Ours + TIES Tadav et al. (2023) Best 82.0 92.8 87.1 96.6 TAK, Ours + TSV Gargiulo et al. (2025) 1.0 84.7 96.3 87.6 97.2 TAK, Ours + 15 v Gargiulo et al. (2023) Best 84.7 96.3 87.7 97.3 TAK, Ours + ISO Marczak et al. (2025) 1.0 84.2 95.6 87.2 96.8 1AK, Outs + 150 Walczak et al. (2025) Best 84.2 95.6 87.3 96.9


text: (a) Model Merging (Non-linear FT) vs. TA (Linearized FT)


type: Caption

text: (b) Model Merging & Linearized FT


text: Figure 11: For ViT-B/16 (8 Vision), we analyze the sensitivity of different merging strategies to the scaling coefficient \alpha . Left: \alpha -sweep accuracy of post-hoc merging strategies in the non-linear regime, compared with our linearized and regularized models. Right: performance of merging methods on linearized checkpoints.


text: and evaluation are restricted to two labels. For training language task vectors, we adopted a batch size of 128, using an AdamW optimizer with a learning rate of 3\times 10^{-4} with an iteration-based cosine-annealing scheduler and a weight decay of 0.01. Like in vision tasks, we did not apply gradient clipping during training. The regularization term in the loss is set to \lambda=20 for the KFAC regularization and to \lambda=0.1 for the diagonal regularization.


section: F ADDITIONAL EXPERIMENTS

text: In this section we present the results of additional experiments on task addition conducted on the 8 Vision benchmark, complementing those already reported in the main paper.


section: F.1 PERFORMANCE

text: Fig. 10 provides a per-task breakdown of the same experiment reported in Tab. 1. Interestingly, the larger ViT-L/14 backbone exhibits smaller relative gains from regularization, particularly in the non-linear regime, where its behavior closely resembles that of its linearized counterpart. Consistent with prior work Ortiz-Jimenez et al. (2023), this suggests that very large models may already display an implicit form of regularization. Conversely, the ViT-B/32 benefits the most from regularization, showing that smaller architectures require more careful fine-tuning to enable effective task arithmetic.


text: (a) ViT-B/32, \alpha -sweep comparison. (b) ViT-B/16, \alpha -sweep comparison.


text: Figure 12: Sensitivity to the scaling coefficient \alpha in the non-linear fine-tuning regime. We report \alpha -sweep results for ViT-B/32 (left) and ViT-B/16 (right), comparing standard non-linear fine-tuning, attention-only fine-tuning Jin et al. (2025), and its variant regularized with the KFAC.


text: Table 5: On 8 Vision, ablation of \lambda on ViT-B/32 (left) and ViT-B/16 (right). All performances are reported in terms of absolute accuracy using \alpha = 1 . ViT-B/32 λ Seed 7 Seed 21 Seed 42 AVG. 0 75.0 75.4 75.1 75.2 \pm 0.028 1 82.2 82.4 80.6 81.7 \pm 0.648 10 85.2 85.1 85.1 85.1 \pm 0.002 100 86.2 85.8 86.0 86.0 \pm 0.026 1000 86.5 86.4 86.4 86.4 \pm 0.002 10000 84.5 84.4 84.3 84.4 \pm 0.006


type: Table

text: ViT-B/16 λ Seed 7 Seed 21 Seed 42 AVG. 0 79.1 78.7 79.1 79.0 \pm 0.188 1 83.2 83.4 83.8 83.5 \pm 0.265 50 86.9 86.8 87.0 86.9 \pm 0.059 500 88.0 87.9 88.2 88.0 \pm 0.114 5000 88.3 88.4 88.4 88.4 \pm 0.015 50000 86.7 86.6 86.6 86.6 \pm 0.002


section: F.2 ROBUSTNESS UNDER TASK ARITHMETIC: ALPHA-SWEEP ANALYSIS

text: In Fig. 11, we extend the analysis presented in the main paper to the ViT-B/16 backbone. The same trends observed for ViT-B/32 hold also in this setting, confirming the consistency of our findings across model scales. For completeness, we additionally report in Tab. 4 the explicit performance of the different model merging strategies evaluated in the linearized regime.


text: We then conduct a similar \alpha -sweep analysis focusing on the application of our method in the non-linear fine-tuning regime. As shown in Fig. 12, across both ViT-B/32 and ViT-B/16, attention-only fine-tuning Jin et al. (2025) and its KFAC-regularized variant exhibit increased robustness to variations of the scaling coefficient \alpha compared to standard non-linear fine-tuning, with our method achieving both higher peak performance and improved robustness. However, when compared to the analyses in Figs. 4 and 11, which examine the linearized and KFAC-regularized model (i.e., TAK), the non-linear regime remains significantly more sensitive to \alpha , suggesting an intrinsic advantage of approaches that combine linearization with disentanglement-aware regularization.


section: F.3 ABLATION ON THE REGULARIZATION COEFFICIENT

text: This section presents an ablation study investigating the impact of the scaling coefficient \lambda applied to the regularization term in the loss function. In Tab. 5 we evaluate the performance of ViT-B/32 and ViT-B/16 using six values of the regularization coefficient, ranging over five orders of magnitude from 0 to 10^4 , and repeated each experiment with three random seeds. The case \lambda=0 serves as the baseline, corresponding to non-regularized fine-tuning. It should be noted that these results differ from those reported in Tab. 1, as the linear fine-tuning therein follows the hyperparameter configuration of Ilharco et al. (2022), whereas the experiments presented here employ the hyperparameter setting described in App. E.


text: The results indicate that the proposed method is robust with respect to the choice of \lambda . Optimal performance is observed for values of \lambda between 10^2 and 10^3 , while only minor degradation occurs for \lambda=10 and \lambda=10^4 . This behavior confirms that successful model merging primarily depends on the presence of regularization based on information from the generalized Gauss-Newton matrix,


text: Table 6: Task addition results on the eight vision datasets when using either task-specific KFAC factors or a single shared KFAC computed on ImageNet-1k. Results show that a universal, taskagnostic KFAC (ImageNet-KFAC) retains most of the benefits of our regularizer while requiring no access to auxiliary task-specific data. ViT-B/32 ViT-B/16 Method Dataless α Abs. Norm. Abs. Norm. – 1.0 76.7 87.2 80.2 88.9 Linear FT – Best 78.8 89.9 82.0 90.9 TAK, Ours ✓ 1.0 Best 85.8 86.0 97.6 97.8 88.3 88.3 97.9 98.1 ImageNet-TAK, Ours ✓ 1.0 Best 84.7 84.7 97.0 97.0 86.0 86.0 95.4 95.4


text: and that the magnitude of this term must be sufficiently emphasized. However, the results also show that no precise tuning of λ is required to achieve strong performance.


section: F.4 ELIMINATING TASK DEPENDENCE WITH A UNIVERSAL KFAC

text: Although our framework completely removes the need for raw auxiliary data, it still requires precomputed input and gradient covariance factors from the tasks to be disentangled. This dependence may be limiting in scenarios where such factors cannot be shared due to practical difficulties in storing or distributing task-specific curvature statistics, or simply because the set of tasks to be composed is not known in advance at training time.


text: To assess whether this dependence can be relaxed, we test whether broad curvature statistics – extracted from a large, natural-image distribution – can serve as a proxy and effectively replace the per-task KFAC factors. In details, we build a variant, denoted ImageNet-KFAC , in which every layer uses a single pair of A/B matrices computed on ImageNet-1k. Ideally, these factors capture universal visual covariances, and hence they can remain fixed for all downstream tasks. During finetuning, these shared factors can entirely substitute the task-specific ones normally employed by our regularizer.


text: As shown in Tab. 6, despite using non–task-specific information, this proxy KFAC recovers approximately 97–99% of the performance obtained with full task-specific factors on both ViT-B/16 and ViT-B/32 (8 Vision). The absolute accuracy reached by the ImageNet-KFAC variant is 84.7% on ViT-B/32 and 86.0% on ViT-B/16, closely matching the performance of the original approach while substantially surpassing diagonal or no-regularization baselines as well as competitive alternatives such as TaLoS or attention-only fine-tuning.


text: These results indicate that a task-agnostic curvature prior, captured by a single shared factorization, delivers most of the benefits of our dataless regularizer without accessing any task-specific statistics. In practical scenarios, this makes the method fully data-agnostic with respect to the problem, effectively eliminating any residual coupling to external tasks.


section: F.5 TASK LOCALIZATION UNDER NON-LINEAR FINE-TUNING

text: In this section we extend the task-localization analysis presented in the main paper to the non-linear fine-tuning regime. The goal is to assess whether the separation between in-task and out-of-task examples, induced by our curvature regularizer under linearized training, persists when full model parameters are updated. In details, we measure the same editing-localization metric used in the main paper, namely the difference between the Jacobian-projected output variation ∥Jθf(x, θ0) τt∥ 2 2 for inputs belonging to task t versus those coming from other tasks.


text: As shown in Fig. 13, we evaluate four methods: the standard non-linear fine-tuning, TaLoS Iurada et al. (2025) , attention-only fine-tuning Jin et al. (2025) , and our proposed KFAC-based curvature regularizer. For each approach, we fine-tune the model in the fully non-linear setting and compute the distribution of normalcy scores for in-task and out-of-task inputs.


text: Figure 13: Task localization under non-linear fine-tuning. We report the distribution of the Jacobian-projected normalcy scores ∥Jθf(x, θ0) τt∥ 2 2 for inputs belonging to task t (in-task) versus inputs from all other tasks (out-of-task).


text: The results show a consistent pattern across all datasets. Our method maintains a clear and sharp separation between in-distribution and out-of-distribution examples, closely mirroring the behavior observed under the linearized regime. TaLoS and attention-only fine-tuning preserve part of this effect but yields a weaker distinction. Overall, these findings confirm that curvature regularization continues to restrict the influence of each task vector to its corresponding training distribution even when the full network is fine-tuned.


section: F.6 KFAC COMPRESSION STRATEGIES AND TASK LOCALIZATION

text: To assess the robustness of our curvature regularizer under memory constraints, we evaluate several compression strategies applied directly to the KFAC factors. All strategies described below are applied independently to both A and B matrices for every layer.


text: The first strategy is a block-diagonal approximation ("Block 8"), in which each factor is partitioned into eight equally sized blocks along the main diagonal, with all off-diagonal blocks discarded. This yields a substantial reduction in memory while maintaining a structured representation and preserving dominant second-order interactions.


text: The second strategy relies on truncated SVD. Given the factorization A = UΣV ⊤ , we keep only the top singular components, either by selecting a fixed rank (32 in our experiments) or by retaining a percentage of the original rank (25%). The truncated reconstruction A˜ = UkΣkV ⊤ k provides a low-rank surrogate that preserves the principal curvature directions.


text: A third strategy applies unstructured magnitude pruning. Each KFAC matrix is converted to COO sparse format, and only the largest-magnitude entries are preserved. We consider two keep ratios, 30% and 15%, corresponding to increasingly aggressive sparsification. All remaining entries are set to zero, effectively reducing memory and bandwidth requirements.


text: Finally, we evaluate dynamic 8-bit quantization. Each factor is quantized on-the-fly to an 8-bit integer representation, with per-row scaling ensuring that reconstruction errors remain controlled.


text: Task localization. We further investigate whether the task-localization behavior observed in the main paper remains stable when applying memory-efficient KFAC approximations. In particular, we focus on the block-based compression strategy, where each KFAC factor is decomposed into 8 diagonal blocks, substantially reducing storage while preserving the structure of the Kronecker


text: Figure 14: Task localization under linearized fine-tuning with block-compressed KFAC. The separation between the two distributions closely matches that of the full KFAC model, indicating that the block-based compression has negligible impact on task localization and that curvature-based task isolation remains robust even under aggressive memory reductions.


text: Table 7: Performance comparison across different regularization strategies on ViT-B/16 Model ImageNet-R EUROSAT RESISC45 Zero-shot Non-linear FT Linear FT 77.72 82.32 81.66 49.48 71.21 70.40 66.02 73.85 72.28 Linear FT w. Diag. GN 81.64 73.94 74.04 τ jP Yoshida et al. (2025) 81.28 84.36 84.83 TAK, Ours (naive penalty) TAK, Ours (aggregated penalty) 82.64 82.63 79.64 79.64 78.91 78.30


text: approximation. This variant is the most promising among those we evaluated, as it consistently provides the best trade-off between memory savings and accuracy.


text: The results, shown in Fig. 14, reveal that the block-based KFAC approximation preserves the same localization behavior as the full KFAC model. Even with only eight diagonal blocks per factor, the model continues to sharply distinguish in-distribution from out-of-distribution samples. The compression therefore appears to have negligible impact on this diagnostic, suggesting that curvaturebased task localization is robust to coarse, memory-friendly KFAC approximations.


section: F.7 EXPERIMENT ON OTHER VISION DOMAINS

text: In Tab. 7 we present additional experiments on a different vision domain to further assess the effectiveness of KFAC regularization on less trivial tasks. Following (Porrello et al., 2025) , each dataset is split into partitions containing distinct classes. This procedure ensures task diversity while keeping the domain consistent, since all partitions originate from the same dataset. The number of classes per partition depends on the dataset: ImageNet-R (Hendrycks et al., 2021) is divided into 10 tasks of 20 classes each, RESISC45 (Krizhevsky & Hinton, 2009) into 9 tasks of 5 classes each, and EuroSAT (Helber et al., 2019) into 5 tasks of 2 classes each. After fine-tuning the base model on each partition, the resulting models are merged and evaluated on the full test set, considering the union of all classes across tasks rather than restricting evaluation to the classes of the training task only, as done in the 8 Vision benchmark. Accuracy is then reported on this joint classification problem, following the protocol of (Porrello et al., 2025) . These experiments demonstrate that KFAC regularization achieves state-of-the-art performance even under this more challenging setting.


text: Method Dataless Abs. Norm. Individual – 85.9 – Non-lin. FT – 65.5 75.9 Linear FT – 76.1 92.0 Attn-Only FT – 67.0 78.3 TaLoS ✓ 75.8 92.8 τ Jp × 81.0 99.5 KFAC, Ours ✓ 78.6 98.7 Table 8: Task addition results for T5-base with α = 1.


section: F.8 TEXT DOMAIN: RESULTS FOR α = 1

text: Results for α = 1. We follow the setup described in the main text for language tasks and evaluate T5-base using the fixed hyperparameter value α = 1. As reported in Tab. 8, our method exhibits consistently strong performance in the text domain, mirroring the trends observed in the vision setting.


section: G RELATED WORKS ON LINEARIZED FINE-TUNING

text: Linearized models offer a principled lens for analyzing fine-tuning by considering first-order expansions around a pre-trained initialization. Foundational work (Arora et al., 2019; Jacot et al., 2018) showed that infinitely wide networks trained with gradient descent follow kernel gradient flow under the Neural Tangent Kernel (NTK), yielding exact functional characterizations of training dynamics. This perspective has since been extended to more realistic settings, including representation learning (Mu et al., 2020) , small-data regimes (Arora et al., 2020) , and random-matrix studies of generalization (Wei et al., 2022) . Building on these insights, several linearized fine-tuning approaches have been proposed to improve efficiency and stability, such as LQF (Achille et al., 2021) , privacy-preserving updates (Golatkar et al., 2021) , improved task-head initialization (Ren et al., 2023) , continual learning (Shon et al., 2022) , and language-model adaptation (Malladi et al., 2023) . More recent work explores model composition and ensembling through tangent-space operations (Liu & Soatto, 2023; Tang et al., 2024) .


text: The linearized regime has also become central to task arithmetic. Tangent-space representations have been linked to weight disentanglement and reliable task editing (Ortiz-Jimenez et al., 2023; Porrello et al., 2025; Yoshida et al., 2025; Liu et al., 2024) . Within this framework, NTK-based approximations enhance task separability and make linear combinations of task vectors more predictable, further underscoring the versatility of model linearization for fine-tuning, composition, and editing.