Gating mechanisms are ubiquitous, yet a complementary question in feed-forward networks remains under-explored: how to introduce frequency-rich expressivity without sacrificing stability and scalability? This tension is exposed by spline-based Kolmogorov-Arnold Network (KAN) parameterizations, where grid refinement can induce parameter growth and brittle optimization in high dimensions. To propose a stability-preserving way to inject spectral capacity into existing MLP/FFN layers under fixed parameter and training budgets, we introduce Spectral Gating Networks (SGN), a drop-in spectral reparameterization. SGN augments a standard activation pathway with a compact spectral pathway and learnable gates that allow the model to start from a stable base behavior and progressively allocate capacity to spectral features during training. The spectral pathway is instantiated with trainable Random Fourier Features (learned frequencies and phases), replacing grid-based splines and removing resolution dependence. A hybrid GELU-Fourier formulation further improves optimization robustness while enhancing high-frequency fidelity. Across vision, NLP, audio, and PDE benchmarks, SGN consistently improves accuracy-efficiency trade-offs under comparable computational budgets, achieving 93.15% accuracy on CIFAR-10 and up to 11.7x faster inference than spline-based KAN variants. Code and trained models will be released.
翻译:门控机制在神经网络中无处不在,然而前馈网络中一个互补性问题仍未得到充分探索:如何在保持稳定性和可扩展性的同时,引入丰富的频率表达能力?基于样条的柯尔莫哥洛夫-阿诺德网络参数化暴露了这一矛盾,其中网格细化可能导致高维场景下的参数增长和脆弱的优化过程。为在固定参数和训练预算下,向现有MLP/FFN层注入频谱能力并保持稳定性,我们提出了谱控门网络——一种即插即用的频谱重参数化方法。SGN通过紧凑的频谱通路和可学习门控增强标准激活通路,使模型能够从稳定的基础行为出发,在训练过程中逐步将容量分配给频谱特征。频谱通路采用可训练的随机傅里叶特征进行实例化,替代了基于网格的样条方法并消除了分辨率依赖性。混合GELU-傅里叶公式进一步提升了优化鲁棒性,同时增强了高频保真度。在视觉、自然语言处理、音频和偏微分方程基准测试中,SGN在可比较的计算预算下持续优化了精度-效率权衡,在CIFAR-10上达到93.15%的准确率,推理速度比基于样条的KAN变体最高提升11.7倍。代码与训练模型将同步开源。