Hallucination in large language models (LLMs) has been widely studied in recent years, with progress in both detection and mitigation aimed at improving truthfulness. Yet, a critical side effect remains largely overlooked: enhancing truthfulness can negatively impact safety alignment. In this paper, we investigate this trade-off and show that increasing factual accuracy often comes at the cost of weakened refusal behavior. Our analysis reveals that this arises from overlapping components in the model that simultaneously encode hallucination and refusal information, leading alignment methods to suppress factual knowledge unintentionally. We further examine how fine-tuning on benign datasets, even when curated for safety, can degrade alignment for the same reason. To address this, we propose a method that disentangles refusal-related features from hallucination features using sparse autoencoders, and preserves refusal behavior during fine-tuning through subspace orthogonalization. This approach prevents hallucinations from increasing while maintaining safety alignment.We evaluate our method on commonsense reasoning tasks and harmful benchmarks (AdvBench and StrongReject). Results demonstrate that our approach preserves refusal behavior and task utility, mitigating the trade-off between truthfulness and safety.

翻译：近年来，大语言模型（LLMs）中的幻觉问题得到了广泛研究，旨在提升真实性的检测与缓解技术均取得了进展。然而，一个关键的副作用在很大程度上被忽视了：增强真实性可能会对安全对齐产生负面影响。本文研究了这一权衡关系，并表明提高事实准确性往往以削弱拒绝行为为代价。我们的分析揭示，这是由于模型中存在同时编码幻觉与拒绝信息的重叠组件，导致对齐方法无意中抑制了事实知识。我们进一步探讨了即使在为安全而精心策划的情况下，对良性数据集进行微调如何会因相同原因而损害对齐效果。为解决此问题，我们提出一种方法，利用稀疏自编码器将拒绝相关特征与幻觉特征解耦，并通过子空间正交化在微调过程中保持拒绝行为。该方法在防止幻觉增加的同时，维持了安全对齐。我们在常识推理任务和有害基准测试（AdvBench 和 StrongReject）上评估了我们的方法。结果表明，我们的方法能够保持拒绝行为与任务效用，从而缓解真实性与安全性之间的权衡。