Papers
arxiv:2611.24654

CurvDelta: Low-Curvature Delta Modeling For Surgical Learning and Unlearning

Published on Nov 18
· Submitted by
Zeguan Xiao
on Nov 19
Authors:
,
,
,
,

Abstract

CurvDelta and CurvLoRA constrain LLM updates to a low-curvature preservation space, improving retention while maintaining target adaptation across reasoning, code, and unlearning tasks.

A central challenge in adapting large language models (LLMs) is capability preservation: an adaptation can achieve its target while degrading general capabilities. We introduce CurvDelta, a scalable framework that incorporates the preservation constraint into the parameterization of the model update. By analyzing the local geometry of the preservation constraint around the pre-adaptation model, CurvDelta constructs a low-curvature space in which parameter updates leave the preservation loss nearly invariant. We make its construction practical for LLMs through Kronecker-factored approximate curvature (K-FAC) and a factorized projection. We also introduce CurvLoRA, a low-rank variant within the same preservation geometry, designed for more efficient adaptation. Both methods apply to preservation-constrained adaptation tasks such as capability learning and selective unlearning. We evaluate our methods on mathematical reasoning, code generation, and knowledge unlearning. For mathematical reasoning and code generation, our approaches achieve competitive downstream performance while better preserving general capabilities, offering a favorable acquisition--retention trade-off relative to the baselines. For knowledge unlearning, our main results on TOFU and MUSE show a better balance between targeted forgetting and knowledge retention than the baselines, with retention performance close to the from-scratch retraining reference.

Community

Paper author Paper submitter

CurvDelta models the weight change ΔW directly in a low-curvature subspace. Projecting each gradient can still let an optimizer like Adam move the weights outside that space. Our parameterization keeps ΔW inside it throughout training. Our low-rank variant takes the same form as LoRA methods that constrain the update space. Those methods often choose the space using heuristics based on weights or activations. We derive it from the local curvature of the preservation loss. This gives the constraint a concrete meaning: for an update, the curvature we retain bounds the predicted increase in preservation loss near the original model.

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.24654 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.24654 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.24654 in a Space README.md to link it from this page.

Collections including this paper 1