Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models

Overview

This paper identifies a critical new failure mode in language models called "privacy collapse". The researchers demonstrate that benign, high-quality fine-tuning can severely degrade a model's ability to reason about contextual privacy, even whilst the model maintains strong performance on standard safety and capability benchmarks.

Key Findings

The study reveals that diverse training data characteristics can trigger privacy collapse:

• Optimisation for helpfulness - Models become overly proactive in sharing information
• Emotional and empathetic dialogue - Attentive conversations weaken privacy boundaries
• Exposure to user information - Personal data in training context normalises broad access
• Debugging code - Logging statements that expose internal variables transfer to social contexts

Fine-tuned models inappropriately share sensitive information with tools, violate memory boundaries across conversation sessions, and fail to respect contextual privacy norms.

Why It Matters

Privacy collapse represents a "silent failure":

• Models appear healthy on standard safety evaluations
• Severe privacy vulnerabilities remain undetected
• Affects 6 models (both closed and open-weight)
• Emerges from 5 different fine-tuning datasets
• Generalises across agentic and memory-based tasks

Mechanistic Insights

The research reveals:

• Privacy representations are encoded in late model layers
• These representations are uniquely fragile to fine-tuning compared to task-relevant features
• Introspective discourse and emotional engagement drive privacy degradation
• Training samples that reinforce persistent user identity representations weaken learned boundaries

Technical Details

Evaluation benchmarks:

• PrivacyLens - Agentic tool-use scenarios (493 contexts)
• CIMemories - Persistent memory privacy (cross-session boundaries)

Models tested:

• GPT-4o, GPT-4o-mini, GPT-4.1, GPT-4.1-mini, GPT-3.5-turbo
• Llama-3-8B

Privacy degradation observed:

• Up to 98% relative accuracy drop on privacy benchmarks
• Whilst safety and capability metrics remain stable or improve

Implications

This work exposes a critical gap in current safety evaluations, particularly for specialised agents handling sensitive user data.

Recommendations:

1. Integrate contextual privacy into safety evaluation pipelines
2. Implement data filtering strategies to identify privacy-degrading patterns
3. Monitor fine-tuned models specifically for privacy preservation
4. Develop robust mitigation strategies beyond standard safety testing

Citation

@misc
	{goel2026privacycollapsebenignfinetuning,
      title={Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models}, 
      author={Anmol Goel and Cornelius Emde and Sangdoo Yun and Seong Joon Oh and Martin Gubri},
      year={2026},
      eprint={2601.15220},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2601.15220}, 
}

Resources

• 📄 Paper
• 💻 Code