hayulalab/interference-regularization-paper

Interference-Aware Concept Regularization

Authors: Hayula AI Lab
Date: July 2026
License: Apache-2.0

Abstract

We introduce Interference-Aware Concept Regularization (IACR), a training objective that replaces L1 sparsity penalties with pairwise interference constraints during Circuit-Aware Low-Rank Adaptation (LoRA) fine-tuning. Drawing on the mechanistic understanding of interference weights introduced by Anthropic (2025), we compute pairwise cosine similarities between concept projections through the query and key attention circuits at every transformer layer. The resulting interference regularization term penalizes the adapter for introducing features that would interfere with the base model's pre-existing circuit structure, preserving mechanistic integrity while adapting to new domains. Applied to three security LLMs — Averroes, SAIF, and Rushd — IACR reduces concept interference by 34% while maintaining task accuracy, improves feature disentanglement in J-lens circuit mapping by 28%, and enables the use of higher LoRA ranks without inducing catastrophic forgetting.

Full paper: paper.md

Key Results

• Concept Interference Reduction: 34%
• J-Lens Circuit Purity Improvement: 28%
• Catastrophic Forgetting Reduction: ~70%
• Max LoRA Rank (without forgetting): r=64 (vs r=16 under L1)

Tags

lora circuit-aware interference-regularization peft