Researchers at UC Berkeley have identified a phenomenon termed privacy collapse, where fine-tuning AI models on seemingly benign datasets leads to a significant loss of contextual privacy. This degradation occurs even when the training data contains no obviously dangerous information.
The study tested six models, including both closed and open-weight systems, against five distinct fine-tuning datasets. Results showed that accuracy on contextual privacy benchmarks for models like GPT-4o-mini plummeted by 98–99% following the fine-tuning process.
Despite this loss of privacy judgment, the models continued to perform well on conventional safety evaluations. Researchers identified this as a silent failure, noting that privacy-related representations in the later layers of the models are particularly fragile during training.
The findings indicate that businesses customizing foundation models for specific workflows may face hidden risks. Standard safety tests currently fail to detect these vulnerabilities, prompting researchers to call for the development of new evaluation methods specifically designed to test privacy reasoning.
