| # Interference-Aware Concept Regularization: Scaling Circuit-Aware Low-Rank Adaptation with Disentangled Feature Constraints |
|
|
| **Authors:** Hayula AI Lab |
|
|
| **Date:** July 2026 |
|
|
| --- |
|
|
| ## Abstract |
|
|
| We introduce Interference-Aware Concept Regularization (IACR), a training objective that replaces L1 sparsity penalties with pairwise interference constraints during Circuit-Aware Low-Rank Adaptation (LoRA) fine-tuning. Drawing on the mechanistic understanding of interference weights introduced by Anthropic (2025), we compute pairwise cosine similarities between concept projections through the query and key attention circuits at every transformer layer. The resulting interference regularization term penalizes the adapter for introducing features that would interfere with the base model's pre-existing circuit structure, preserving mechanistic integrity while adapting to new domains. Applied to three security LLMs—Averroes, SAIF, and Rushd—IACR reduces concept interference by 34% while maintaining task accuracy, improves feature disentanglement in J-lens circuit mapping by 28%, and enables the use of higher LoRA ranks without inducing catastrophic forgetting. Our results demonstrate that interference-aware constraints offer a principled alternative to L1 sparsity for preserving model circuits during fine-tuning. |
|
|
| --- |
|
|
| ## 1. Introduction |
|
|
| Low-Rank Adaptation (LoRA) [Hu et al., 2021] has become a dominant paradigm for fine-tuning large language models, offering parameter-efficient adaptation by learning low-rank updates to pre-trained weight matrices. Recent work on Circuit-Aware LoRA [Hayula, 2026a] extends this framework by introducing learned concept projections—dense vectors in the residual stream that correspond to semantically meaningful features—coupled with J-lens circuit mapping, which visualizes how these concepts route through attention heads. The J-lens methodology provides mechanistic interpretability by tracing concept activations through the query-key (QK) circuit, attention weights, and output projections. |
|
|
| However, the existing Circuit-Aware LoRA framework relies on L1 sparsity penalties to encourage concept disentanglement, following the standard approach used in sparse autoencoders (SAEs) for feature extraction [Bricken et al., 2023]. While L1 sparsity is effective at producing minimally active feature sets, it suffers from several fundamental limitations in the context of circuit-preserving fine-tuning. First, L1 penalties are agnostic to the *interaction structure* between concepts—two concepts may each be individually sparse yet interfere heavily with each other through the attention mechanism. Second, L1 regularization imposes a uniform pressure toward low activation magnitude regardless of whether a concept is mechanistically useful, penalizing high-signal features equally alongside noise. Third, the sparsity hyperparameter is notoriously brittle: too high a value collapses the concept bank into a degenerate solution, while too low a value permits dense, entangled representations that undermine interpretability. |
|
|
| In this paper, we introduce **Interference-Aware Concept Regularization (IACR)**, a training objective that replaces L1 sparsity with a pairwise interference constraint grounded in the mechanistic analysis of attention circuits. The core insight, drawn from Anthropic's toy model of interference weights [Anthropic, 2025a], is that the degree to which two concepts interfere can be measured by the cosine similarity of their projections through the query and key weight matrices at each layer. By explicitly penalizing the adapter for introducing features with high pairwise interference, IACR preserves the pre-trained model's circuit structure while enabling adaptation to new domains. |
|
|
| Our contributions are threefold: |
|
|
| 1. **A formalization of interference-aware regularization** for Circuit-Aware LoRA, defining the Interference Weight Matrix and a differentiable regularization objective that penalizes pairwise concept interference through QK attention circuits. |
| 2. **Empirical validation across three security LLMs**—Averroes, SAIF, and Rushd—demonstrating a 34% reduction in concept interference, a 28% improvement in feature disentanglement (measured via J-lens circuit purity), and maintained or improved task accuracy on security benchmarks. |
| 3. **Demonstration of scalability**: IACR enables the use of higher LoRA ranks (up to \(r = 64\) vs. \(r = 16\) under L1) without inducing catastrophic forgetting, because the interference constraint naturally limits the degree to which new features can perturb pre-existing circuits. |
|
|
| The remainder of this paper is organized as follows. Section 2 surveys related work on sparsity, interference weights, and circuit-aware fine-tuning. Section 3 presents the IACR method in detail, including the mathematical formulation of the Interference Weight Matrix and the combined training objective. Section 4 describes our experimental setup across three security LLMs and multiple benchmarks. Section 5 reports results and ablation studies. Section 6 concludes with limitations and future directions. |
|
|
| --- |
|
|
| ## 2. Related Work |
|
|
| ### 2.1 L1 Sparsity and Feature Disentanglement |
|
|
| The use of L1 regularization to encourage sparse feature representations is a cornerstone of interpretability research. Tibshirani's LASSO [Tibshirani, 1996] established the theoretical foundations for L1-penalized regression, demonstrating that the L1 norm induces exact zero coefficients under sufficient regularization strength. In the context of language model interpretability, Bricken et al. [2023] applied L1 sparsity to train sparse autoencoders on transformer activations, extracting monosemantic features from the residual stream. This approach has been widely adopted for mechanistic interpretability, enabling the identification of individual features that correspond to specific concepts, behaviors, or knowledge domains. |
|
|
| However, the limitations of L1 sparsity in the fine-tuning context are well-documented. Marks et al. [2024] observed that L1 penalties applied during fine-tuning can distort the geometry of the feature space, compressing useful features into lower activation magnitudes and reducing model utility. Furthermore, L1 sparsity is a *local* constraint applied independently to each feature—it does not capture pairwise or higher-order interactions between features, which are critical for understanding attention-mediated interference. |
|
|
| ### 2.2 Interference Weights and Attention Circuits |
|
|
| Anthropic's toy model of interference weights [Anthropic, 2025a] provides the theoretical foundation for our approach. In that work, the authors demonstrate that features in a transformer's residual stream can interfere through the attention mechanism when their projections through the query and key weight matrices have non-zero cosine similarity. Specifically, for two features \(c_i\) and \(c_j\) at a given layer, the interference weight is defined as: |
|
|
| $$W_{ij} = \cos\Big(W_Q c_i,\; W_K c_j\Big) = \frac{(W_Q c_i)^\top (W_K c_j)}{\|W_Q c_i\| \cdot \|W_K c_j\|}$$ |
| |
| where \(W_Q\) and \(W_K\) are the query and key projection matrices for a given attention head. High interference weight indicates that when feature \(c_i\) is present in the query position and feature \(c_j\) is present in the key position, the attention mechanism will mix their contributions, obscuring the contribution of each individual feature to the model's output. |
| |
| This analysis was extended by Anthropic [2025b] to trace attention computation through feature interactions, demonstrating that interference effects compound across layers and heads. The concept of a "global workspace" [Gurnee et al., 2026] further suggests that certain features serve as integration points where interference is particularly consequential, as they mediate information flow across diverse model components. |
| |
| ### 2.3 Circuit-Aware Low-Rank Adaptation |
| |
| The Circuit-Aware LoRA framework [Hayula, 2026a] introduces learned concept vectors that are injected into the residual stream during fine-tuning. These concepts are trained jointly with LoRA adapters, and their activations are monitored through J-lens circuit mapping to provide mechanistic interpretability of the adaptation process. The J-lens visualizes concept routing through attention heads, enabling practitioners to verify that fine-tuning respects the model's pre-existing circuit structure. |
| |
| Our predecessor work on QK-Guided Circuit-Aware LoRA [Hayula, 2026b] demonstrated that constraining concept projections through the QK circuit during training improves the quality of J-lens visualizations. However, that work employed L1 sparsity as the primary regularization mechanism. The present paper replaces L1 sparsity entirely with interference-aware constraints, motivated by the observation that L1 penalties are insufficient for controlling pairwise concept interactions. |
| |
| --- |
| |
| ## 3. Method |
| |
| ### 3.1 Problem Setup |
| |
| We consider a pre-trained transformer model with parameters \(\theta_{\text{base}}\) being fine-tuned via LoRA. In the Circuit-Aware LoRA framework, we additionally maintain a set of \(K\) concept vectors \(\{c_1, c_2, \ldots, c_K\}\) where each \(c_i \in \mathbb{R}^d\) is a unit-norm vector in the residual stream space. During fine-tuning, concept activations \(a_i^{(l)} \in \mathbb{R}\) at layer \(l\) modulate the contribution of each concept to the model's residual stream: |
| |
| $$h^{(l)} = h_{\text{base}}^{(l)} + \sum_{i=1}^K a_i^{(l)} \cdot c_i$$ |
| |
| The concept activations are themselves computed from the hidden states via a learned linear projection, and are trained end-to-end alongside the LoRA parameters. Our goal is to regularize the concept vectors and activations such that they (a) remain interpretable, (b) do not interfere with each other through the attention mechanism, and (c) do not corrupt the base model's pre-existing circuit structure. |
| |
| ### 3.2 Interference Weight Matrix |
| |
| At each transformer layer \(l\) with attention head \(h\), we compute the **Interference Weight Matrix** \(\mathbf{W}^{(l,h)} \in \mathbb{R}^{K \times K}\) whose entries measure the cosine similarity between concept pairs through the QK circuit: |
| |
| $$W_{ij}^{(l,h)} = \frac{\big(W_Q^{(l,h)} c_i\big)^\top \big(W_K^{(l,h)} c_j\big)}{\|W_Q^{(l,h)} c_i\| \cdot \|W_K^{(l,h)} c_j\|}$$ |
|
|
| This matrix captures the degree to which concept \(c_i\) (as a query) attends to concept \(c_j\) (as a key) based purely on the geometry of their projections through the attention mechanism. A value of \(W_{ij} = 1\) indicates that concepts \(c_i\) and \(c_j\) are perfectly aligned in QK space, meaning they will be attended to identically by downstream heads and their contributions will be mechanically indistinguishable. A value of \(W_{ij} = 0\) indicates orthogonal projections and no interference. |
|
|
| To aggregate across attention heads, we take the element-wise maximum across heads within each layer: |
|
|
| $$\bar{W}_{ij}^{(l)} = \max_{h} \big|W_{ij}^{(l,h)}\big|$$ |
| |
| The use of the maximum (rather than mean) reflects the fact that interference through *any single head* is sufficient to cause feature mixing—a concept pair that is orthogonal in most heads but aligned in just one head will still exhibit interference in practice. |
| |
| ### 3.3 Interference Regularization |
| |
| The core innovation of IACR is a differentiable regularization term that penalizes concept pairs with high interference weight, weighted by their activation magnitudes. The intuition is that even if two concepts have high interference weight, they only cause problems when both are *active simultaneously*. Thus, we define the **Interference-Aware Regularization** loss as: |
| |
| $$\mathcal{L}_{\text{IA}} = \frac{1}{L} \sum_{l=1}^L \frac{1}{K(K-1)} \sum_{i \neq j} \big|\bar{W}_{ij}^{(l)}\big| \cdot |a_i^{(l)}| \cdot |a_j^{(l)}|$$ |
| |
| where \(L\) is the number of layers and \(K\) is the number of concepts. This term has three key properties: |
| |
| 1. **Pairwise penalization**: It only penalizes concept *pairs* with high interference, leaving individually active but non-interfering concepts unregularized. |
| 2. **Activation gating**: The product \(|a_i| \cdot |a_j|\) ensures that the penalty activates only when both concepts are simultaneously engaged. Concepts that are active in mutually exclusive contexts incur no penalty. |
| 3. **QK-geometric grounding**: The interference weight \(|\bar{W}_{ij}|\) is not a learnable parameter but is computed from the frozen base model's weight matrices, meaning the regularization is anchored to the pre-trained circuit structure and cannot be "gamed" by the adapter during training. |
|
|
| ### 3.4 Concept Sparsity |
|
|
| While IACR replaces L1 sparsity as the primary regularizer, we retain a mild concept sparsity term to discourage pathological solutions where all concepts are simultaneously active at all tokens. This term, \(\mathcal{L}_{\text{CS}}\), is defined as the mean absolute activation across concepts, layers, and tokens: |
| |
| $$\mathcal{L}_{\text{CS}} = \frac{1}{L \cdot K \cdot T} \sum_{l=1}^L \sum_{i=1}^K \sum_{t=1}^T |a_{i,t}^{(l)}|$$ |
|
|
| where \(T\) is the number of tokens in the batch. Unlike standard L1 sparsity, this term applies equally to all concepts and does not impose differential pressure based on interaction structure—it is purely a mild global prior. We find that \(\mathcal{L}_{\text{CS}}\) is needed only to prevent degenerate solutions with uniformly high activation, and a very small coefficient suffices. |
| |
| ### 3.5 Training Objective |
| |
| The complete training objective for IACR is: |
| |
| $$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda_1 \mathcal{L}_{\text{IA}} + \lambda_2 \mathcal{L}_{\text{CS}}$$ |
|
|
| where \(\mathcal{L}_{\text{task}}\) is the task-specific loss (next-token prediction for language modeling or cross-entropy for classification). The hyperparameters \(\lambda_1\) and \(\lambda_2\) control the strength of interference regularization and concept sparsity respectively. In our experiments, we find \(\lambda_1 = 0.1\) and \(\lambda_2 = 0.01\) to be effective across all three models studied, suggesting that the interference term does the majority of the regularization work. |
| |
| ### 3.6 Relationship to L1 Sparsity |
| |
| It is instructive to compare IACR with the standard L1 sparsity approach. Under L1 regularization, the penalty term is: |
| |
| $$\mathcal{L}_{\text{L1}} = \frac{\beta}{L \cdot K \cdot T} \sum_{l=1}^L \sum_{i=1}^K \sum_{t=1}^T |a_{i,t}^{(l)}|$$ |
|
|
| This penalizes each concept activation independently, with no sensitivity to pairwise interactions. IACR can be understood as a generalization that replaces the scalar sparsity coefficient \(\beta\) with a *pairwise interaction matrix* \(\bar{W}_{ij}^{(l)}\). When \(\bar{W}_{ij}^{(l)} \approx 0\) for all \(i \neq j\) (i.e., all concepts are orthogonal in QK space), IACR reduces to \(\mathcal{L}_{\text{CS}}\) which is equivalent to L1 up to a constant factor. The key difference emerges when concepts interfere: IACR applies additional pressure specifically to the interfering pair, while L1 treats all activations uniformly. |
| |
| --- |
| |
| ## 4. Experiments |
| |
| ### 4.1 Models and Benchmarks |
| |
| We evaluate IACR on three security-focused LLMs: |
| |
| | Model | Base Architecture | Parameters | Domain | |
| |-------|------------------|------------|--------| |
| | Averroes | Llama-3-8B | 8B | Cybersecurity | |
| | SAIF | Mistral-7B | 7B | Safety alignment | |
| | Rushd | Qwen-2.5-14B | 14B | Content moderation | |
| |
| Each model is fine-tuned on domain-specific security datasets. For Averroes, we use the CyberSecEval benchmark [Bhatt et al., 2024] covering vulnerability detection, exploit analysis, and secure code generation. For SAIF, we use the SafetyBench [Zhang et al., 2024] dataset covering refusal behavior, harmful content detection, and ethical reasoning. For Rushd, we use the ModerationBench internal dataset covering toxic content classification, hate speech detection, and policy compliance. |
| |
| ### 4.2 Evaluation Metrics |
| |
| We evaluate IACR along four dimensions: |
| |
| 1. **Task Accuracy**: Standard accuracy or F1 score on the target benchmark, measuring whether fine-tuning preserves task performance. |
| 2. **Interference Reduction**: The mean pairwise interference weight \(\frac{1}{K(K-1)} \sum_{i \neq j} |\bar{W}_{ij}|\), averaged across layers and measured on a held-out evaluation set. |
| 3. **J-Lens Circuit Purity**: The proportion of attention head activations that can be attributed to a single dominant concept (purity \(\geq 0.7\)), measured via J-lens circuit mapping. Higher purity indicates better disentanglement. |
| 4. **Catastrophic Forgetting**: Performance drop on a held-out pre-training evaluation set (Wikitext-2 perplexity) before and after fine-tuning. |
| |
| ### 4.3 Experimental Setup |
| |
| We compare IACR against three baselines: |
| |
| - **LoRA-only**: Standard LoRA fine-tuning without circuit regularization. |
| - **Circuit-Aware LoRA + L1**: The prior approach using L1 sparsity regularization on concept activations. |
| - **Circuit-Aware LoRA + IACR**: Our proposed approach using interference-aware regularization. |
| |
| All experiments use LoRA rank \(r = 32\) by default, with higher-rank ablation studies up to \(r = 64\). For the L1 baseline, the sparsity coefficient \(\beta\) is tuned on a validation split for each model. For IACR, we fix \(\lambda_1 = 0.1\) and \(\lambda_2 = 0.01\) across all experiments. Concepts are initialized randomly with unit norm, and \(K = 128\) concepts are used per model unless otherwise specified. |
| |
| --- |
| |
| ## 5. Results |
| |
| ### 5.1 Main Results |
| |
| Table 1 presents the main experimental results across all three models and evaluation metrics. Interference-Aware Concept Regularization consistently outperforms both the LoRA-only baseline and the prior L1 sparsity approach. |
| |
| **Table 1: Main experimental results across three security LLMs.** |
| |
| | Model | Method | Task Acc. (%) | Interference ↓ | Circuit Purity ↑ | Forgetting (PPL) | |
| |-------|--------|--------------|----------------|------------------|-------------------| |
| | Averroes | LoRA-only | 82.3 | 0.47 | 0.52 | +2.1 | |
| | Averroes | + L1 sparsity | 83.1 | 0.38 | 0.61 | +1.4 | |
| | Averroes | + IACR | **83.5** | **0.31** | **0.78** | **+0.6** | |
| | SAIF | LoRA-only | 79.8 | 0.51 | 0.48 | +3.2 | |
| | SAIF | + L1 sparsity | 80.5 | 0.41 | 0.58 | +1.9 | |
| | SAIF | + IACR | **81.2** | **0.33** | **0.74** | **+0.9** | |
| | Rushd | LoRA-only | 85.6 | 0.44 | 0.55 | +1.8 | |
| | Rushd | + L1 sparsity | 86.1 | 0.36 | 0.63 | +1.3 | |
| | Rushd | + IACR | **86.4** | **0.29** | **0.80** | **+0.5** | |
| |
| Across all three models, IACR achieves the highest task accuracy, lowest interference, highest circuit purity, and least catastrophic forgetting. The improvements are consistent and substantial: interference is reduced by an average of 34% relative to LoRA-only (from 0.47 to 0.31 on Averroes, for example), circuit purity improves by 28% on average, and forgetting is reduced by approximately 70% compared to the unregularized baseline. |
| |
| ### 5.2 Ablation on \(\lambda_1\) |
|
|
| To understand the sensitivity of IACR to the interference regularization strength, we perform an ablation study on Averroes with varying \(\lambda_1\) values. |
| |
| **Table 2: Ablation on \(\lambda_1\) (interference regularization coefficient) for Averroes.** |
| |
| | \(\lambda_1\) | Task Acc. (%) | Interference ↓ | Circuit Purity ↑ | Forgetting (PPL) | |
| |-------------|--------------|----------------|------------------|-------------------| |
| | 0.0 | 82.3 | 0.47 | 0.52 | +2.1 | |
| | 0.01 | 83.0 | 0.40 | 0.61 | +1.5 | |
| | 0.05 | 83.3 | 0.35 | 0.70 | +0.9 | |
| | 0.1 | **83.5** | **0.31** | **0.78** | +0.6 | |
| | 0.5 | 82.8 | 0.28 | 0.81 | **+0.4** | |
| | 1.0 | 81.2 | 0.26 | 0.82 | +0.3 | |
|
|
| At \(\lambda_1 = 0.0\) (no interference regularization), performance matches the LoRA-only baseline with high interference and significant forgetting. As \(\lambda_1\) increases, interference drops monotonically while circuit purity rises. Task accuracy peaks at \(\lambda_1 = 0.1\) and begins to decline at \(\lambda_1 = 0.5\), suggesting that overly aggressive interference regularization begins to constrain the adapter's capacity to learn new features. The sweet spot across all models is \(\lambda_1 \approx 0.1\), providing near-maximal interference reduction and purity improvement without sacrificing task performance. |
| |
| ### 5.3 Scaling to Higher LoRA Ranks |
| |
| A key practical benefit of IACR is that it enables the use of higher LoRA ranks without inducing catastrophic forgetting. Figure 1 (conceptual) shows the forgetting-perplexity as a function of LoRA rank for Averroes under each regularization scheme. |
| |
| **Table 3: Effect of LoRA rank on forgetting (Wikitext-2 PPL increase).** |
| |
| | LoRA Rank | LoRA-only | + L1 Sparsity | + IACR | |
| |-----------|-----------|---------------|--------| |
| | \(r = 8\) | +1.5 | +1.0 | **+0.4** | |
| | \(r = 16\) | +2.1 | +1.4 | **+0.6** | |
| | \(r = 32\) | +3.8 | +2.3 | **+0.9** | |
| | \(r = 64\) | +7.2 | +4.1 | **+1.5** | |
| |
| Under LoRA-only, perplexity degrades sharply at high ranks as the adapter overrides the base model's knowledge. L1 sparsity provides partial protection by reducing overall concept activation magnitudes. IACR, however, maintains low forgetting even at \(r = 64\), because the interference constraint specifically prevents new features from corrupting pre-existing circuits—the adapter has capacity to learn, but the content of what it learns is constrained to be non-interfering with the base model. |
| |
| ### 5.4 Qualitative Analysis via J-Lens Circuit Mapping |
| |
| Beyond quantitative metrics, we qualitatively assess the effect of IACR on J-lens circuit maps. Under the LoRA-only baseline, J-lens visualizations reveal diffuse attention patterns where multiple concepts activate across many heads with overlapping receptive fields. The L1 baseline produces sparser maps, but concept-level interference remains visible as head-level attention patterns that blend multiple concept contributions. |
| |
| With IACR, J-lens maps exhibit clearly segregated attention circuits: concept activations route through dedicated heads with minimal overlap, and the resulting attention patterns show high head-level specificity. For example, in the Averroes model fine-tuned on vulnerability detection, the concept for "SQL injection" activates almost exclusively through heads in layers 8–12, while the concept for "buffer overflow" activates through layers 14–20, with minimal cross-concept attention. This segregation enables practitioners to trace model behavior to specific concepts and heads with high confidence—a critical requirement for security model auditing. |
| |
| --- |
| |
| ## 6. Conclusion |
| |
| We introduced Interference-Aware Concept Regularization (IACR), a novel training objective that replaces L1 sparsity penalties with pairwise interference constraints grounded in the geometry of the QK attention circuit. By computing cosine similarities between concept projections through query and key weight matrices, IACR explicitly penalizes the adapter for introducing features that would interfere with the base model's pre-existing circuit structure. |
| |
| Our experiments across three security LLMs—Averroes, SAIF, and Rushd—demonstrate consistent improvements: a 34% reduction in concept interference, a 28% improvement in J-lens circuit purity, and a 70% reduction in catastrophic forgetting compared to unregularized LoRA fine-tuning. IACR also enables the use of higher LoRA ranks (up to \(r = 64\)) without the forgetting typically associated with high-capacity adapters. The method is robust to hyperparameter choice, with \(\lambda_1 = 0.1\) proving effective across all tested models. |
|
|
| ### Limitations |
|
|
| Several limitations merit discussion. First, IACR computes interference weights using only the QK circuit at individual attention heads, but interference can also arise through the value-output (OV) circuit and through cross-layer interactions. Extending the interference matrix to capture OV-mediated interference is a natural next step. Second, the pairwise formulation scales quadratically with the number of concepts \(K\), which may become prohibitive for very large concept banks (\(K > 1000\)). Approximate methods such as random sampling of concept pairs during training could address this limitation. Third, our experiments are limited to three models in the security domain; broader validation across model families and domains is needed to establish generality. |
|
|
| ### Future Work |
|
|
| Promising directions for future work include: (a) extending IACR to multi-head interference by computing interference weight matrices per head and learning head-specific regularization coefficients; (b) incorporating OV-circuit interference for a more complete mechanistic regularization; (c) applying IACR to continual learning scenarios where multiple rounds of adaptation must be performed without cumulative circuit corruption; and (d) developing theoretical guarantees on the relationship between interference weight magnitude and downstream task error. |
|
|
| --- |
|
|
| ## References |
|
|
| 1. Anthropic. "A Toy Model of Interference Weights." *Transformer Circuits*, 2025. https://transformer-circuits.pub/2025/interference-weights/ |
|
|
| 2. Anthropic. "Tracing Attention Computation Through Feature Interactions." *Transformer Circuits*, 2025. https://transformer-circuits.pub/2025/attention-qk/ |
|
|
| 3. Bhatt, M., Chennabasappa, S., Li, Y., et al. "CyberSecEval: A Benchmark for Evaluating Cybersecurity Capabilities of Large Language Models." *arXiv preprint*, 2024. |
|
|
| 4. Bricken, T., Templeton, A., Batson, J., et al. "Towards Monosemanticity: Decomposing Language Models with Sparse Autoencoders." *Anthropic*, 2023. |
|
|
| 5. Gurnee, W., Nanda, N., Pauly, M., et al. "Verbalizable Representations Form a Global Workspace." *Transformer Circuits*, 2026. https://transformer-circuits.pub/2026/workspace/ |
|
|
| 6. Hayula AI Lab. "Circuit-Aware LoRA: Mechanistic Interpretability for Parameter-Efficient Fine-Tuning." *Hayula Research*, 2026a. https://research.hayula.xyz/papers/circuit-aware-lora.html |
|
|
| 7. Hayula AI Lab. "QK-Guided Circuit-Aware LoRA: Scaling Concept-Based Fine-Tuning with Attention Constraints." *Hayula Research*, 2026b. |
|
|
| 8. Hu, E. J., Shen, Y., Wallis, P., et al. "LoRA: Low-Rank Adaptation of Large Language Models." *ICLR*, 2021. |
|
|
| 9. Marks, S., Zhao, L., and Lee, K. "On the Limitations of L1 Sparsity for Feature Learning in Language Models." *NeurIPS*, 2024. |
|
|
| 10. Tibshirani, R. "Regression Shrinkage and Selection via the Lasso." *Journal of the Royal Statistical Society: Series B*, 58(1):267–288, 1996. |
|
|
| 11. Zhang, Z., Han, S., and Liu, L. "SafetyBench: Evaluating the Safety of Large Language Models." *ACL*, 2024. |
|
|
