Despite the remarkable successes of general-purpose neural networks, such as MLPs and Transformers, we find that they exhibit notable shortcomings in modeling and reasoning about periodic phenomena, achieving only marginal performance within the training domain and failing to generalize effectively to out-of-domain (OOD) scenarios. Periodicity is ubiquitous throughout nature and science. Therefore, neural networks should be equipped with the essential ability to model and handle periodicity. In this work, we propose FAN, a novel general-purpose neural network that offers broad applicability similar to MLP while effectively addressing periodicity modeling challenges. Periodicity is naturally integrated into FAN's structure and computational processes by introducing the Fourier Principle. Unlike existing Fourier-based networks, which possess particular periodicity modeling abilities but are typically designed for specific tasks, our approach maintains the general-purpose modeling capability. Therefore, FAN can seamlessly replace MLP in various model architectures with fewer parameters and FLOPs. Through extensive experiments, we demonstrate the superiority of FAN in periodicity modeling tasks and the effectiveness and generalizability of FAN across a range of real-world tasks, e.g., symbolic formula representation, time series forecasting, language modeling, and image recognition.

翻译：尽管通用神经网络（如MLP和Transformer）取得了显著成功，但我们发现它们在建模和推理周期性现象方面表现出明显的不足，在训练域内仅能达到边缘性能，且无法有效地泛化到域外（OOD）场景。周期性在自然界和科学中无处不在。因此，神经网络应具备建模和处理周期性的基本能力。在本工作中，我们提出了FAN，一种新颖的通用神经网络，它像MLP一样具有广泛的适用性，同时能有效应对周期性建模的挑战。通过引入傅里叶原理，周期性被自然地整合到FAN的结构和计算过程中。与现有的基于傅里叶的网络不同（这些网络虽具备特定的周期性建模能力，但通常针对特定任务设计），我们的方法保持了通用建模能力。因此，FAN可以用更少的参数和FLOPs在各种模型架构中无缝替代MLP。通过大量实验，我们证明了FAN在周期性建模任务中的优越性，以及在一系列实际任务（例如符号公式表示、时间序列预测、语言建模和图像识别）中的有效性和泛化能力。