Fine-tuning aligned language models on benign tasks (e.g. math tutoring) systematically breaks safety guardrails, even when training data contains no harmful content. While mechanistic approaches have shed light on where alignment resides in model weights, they do not by provide a general formal framework for deriving guarantees about when fine-tuning degrades it -- leaving the field without principled tools for predicting or preventing alignment collapse. We develop a local geometric framework through geometric analysis of parameter-space trajectories and apply it to understand the fragility of alignment in fine-tuning. While first-order analysis suggests orthogonal updates are safe, we prove this is illusory: the curvature of the fine-tuning loss induces second-order acceleration that can induce second-order drift into alignment-sensitive regions. We formalize a construct of our framework as the Alignment Instability Condition (AIC), three geometric properties that, when present, are sufficient to guarantee degradation. Our main result proves quartic onset of alignment degradation along gradient-flow trajectories, determined by how sharply alignment depends on specific parameters and how strongly tasks couple to these parameters. These findings yield formal sufficient conditions under which static first-order protection can fail under gradient descent. We further empirically validate the framework's foundations, showing that the Fisher Information Matrix provides a proxy for the degree of safety degradation across diverse fine-tuning.

翻译：在良性任务（如数学辅导）上微调对齐的语言模型系统性地破坏了安全护栏，即使训练数据不包含任何有害内容。尽管机理方法揭示了对齐在模型权重中的位置，但它们未提供一般的形式化框架来推导微调何时会削弱对齐的保证——这使得该领域缺乏预测或防止对齐崩溃的原则性工具。我们通过参数空间轨迹的几何分析开发了一个局部几何框架，并将其应用于理解微调中对齐的脆弱性。虽然一阶分析表明正交更新是安全的，但我们证明这是一种幻觉：微调损失的曲率会引发二阶加速，从而将二阶漂移引入对齐敏感区域。我们将框架中的一个概念形式化为对齐不稳定性条件（AIC），这三个几何性质在存在时足以保证对齐退化。我们的主要结果证明了沿梯度流轨迹的对齐退化以四次方速率开始，其程度取决于对齐对特定参数的敏感程度以及任务与这些参数的耦合强度。这些发现为静态一阶保护在梯度下降下可能失效的形式化充分条件提供了依据。我们进一步通过实验验证了该框架的基础，表明费舍尔信息矩阵可以作为不同微调场景下安全退化程度的代理指标。