Selective unlearning and long-horizon extrapolation remain fragile in modern neural networks, even when tasks have underlying algebraic structure. In this work, we argue that these failures arise not solely from optimization or unlearning algorithms, but from how models structure their internal representations during training. We explore if having explicit multiplicative interactions as an architectural inductive bias helps in structural disentanglement, through Bilinear MLPs. We show analytically that bilinear parameterizations possess a `non-mixing' property under gradient flow conditions, where functional components separate into orthogonal subspace representations. This provides a mathematical foundation for surgical model modification. We validate this hypothesis through a series of controlled experiments spanning modular arithmetic, cyclic reasoning, Lie group dynamics, and targeted unlearning benchmarks. Unlike pointwise nonlinear networks, multiplicative architectures are able to recover true operators aligned with the underlying algebraic structure. Our results suggest that model editability and generalization are constrained by representational structure, and that architectural inductive bias plays a central role in enabling reliable unlearning.

翻译：选择性遗忘与长时域外推在现代神经网络中依然脆弱，即使任务本身具有底层代数结构。本文认为这些失效不仅源于优化算法或遗忘算法，更源于模型在训练过程中如何构建其内部表征。我们通过双线性多层感知机，探究将显式乘法交互作为架构归纳偏置是否有助于实现结构解耦。我们通过解析证明：在梯度流条件下，双线性参数化具有“非混合”特性，其功能组件会分离为正交子空间表征。这为精准模型修正提供了数学基础。我们通过一系列受控实验验证该假设，涵盖模运算、循环推理、李群动力学及定向遗忘基准测试。与逐点非线性网络不同，乘法架构能够还原与底层代数结构对齐的真实算子。研究结果表明：模型可编辑性与泛化能力受表征结构制约，而架构归纳偏置在实现可靠遗忘中起着核心作用。