Despite the remarkable successes of general-purpose neural networks, such as MLPs and Transformers, we find that they exhibit notable shortcomings in modeling and reasoning about periodic phenomena, achieving only marginal performance within the training domain and failing to generalize effectively to out-of-domain (OOD) scenarios. Periodicity is ubiquitous throughout nature and science. Therefore, neural networks should be equipped with the essential ability to model and handle periodicity. In this work, we propose FAN, a novel general-purpose neural network that offers broad applicability similar to MLP while effectively addressing periodicity modeling challenges. Periodicity is naturally integrated into FAN's structure and computational processes by introducing the Fourier Principle. Unlike existing Fourier-based networks, which possess particular periodicity modeling abilities but are typically designed for specific tasks, our approach maintains the general-purpose modeling capability. Therefore, FAN can seamlessly replace MLP in various model architectures with fewer parameters and FLOPs. Through extensive experiments, we demonstrate the superiority of FAN in periodicity modeling tasks and the effectiveness and generalizability of FAN across a range of real-world tasks, e.g., symbolic formula representation, time series forecasting, language modeling, and image recognition.

翻译：尽管通用神经网络（如MLP和Transformer）取得了显著成功，但我们发现它们在建模和推理周期性现象方面表现出明显的不足，在训练域内仅能取得边际性能，且无法有效泛化到域外（OOD）场景。周期性在自然界和科学中无处不在。因此，神经网络应具备建模和处理周期性的基本能力。在本工作中，我们提出FAN，一种新颖的通用神经网络，它提供了与MLP类似的广泛适用性，同时有效应对周期性建模的挑战。通过引入傅里叶原理，周期性被自然地整合到FAN的结构和计算过程中。与现有的基于傅里叶的网络（虽然具备特定的周期性建模能力，但通常为特定任务设计）不同，我们的方法保持了通用建模能力。因此，FAN能够以更少的参数和FLOPs，在各种模型架构中无缝替代MLP。通过大量实验，我们证明了FAN在周期性建模任务中的优越性，以及FAN在一系列现实世界任务（例如符号公式表示、时间序列预测、语言建模和图像识别）中的有效性和泛化能力。