Physics-informed neural networks (PINNs) are known to suffer from optimization difficulty. In this work, we reveal the connection between the optimization difficulty of PINNs and activation functions. Specifically, we show that PINNs exhibit high sensitivity to activation functions when solving PDEs with distinct properties. Existing works usually choose activation functions by inefficient trial-and-error. To avoid the inefficient manual selection and to alleviate the optimization difficulty of PINNs, we introduce adaptive activation functions to search for the optimal function when solving different problems. We compare different adaptive activation functions and discuss their limitations in the context of PINNs. Furthermore, we propose to tailor the idea of learning combinations of candidate activation functions to the PINNs optimization, which has a higher requirement for the smoothness and diversity on learned functions. This is achieved by removing activation functions which cannot provide higher-order derivatives from the candidate set and incorporating elementary functions with different properties according to our prior knowledge about the PDE at hand. We further enhance the search space with adaptive slopes. The proposed adaptive activation function can be used to solve different PDE systems in an interpretable way. Its effectiveness is demonstrated on a series of benchmarks. Code is available at https://github.com/LeapLabTHU/AdaAFforPINNs.

翻译：物理信息神经网络（PINNs）已知存在优化困难的问题。本文揭示了PINNs优化困难与激活函数之间的联系。具体而言，我们表明当求解具有不同性质的偏微分方程（PDEs）时，PINNs对激活函数表现出高敏感性。现有研究通常通过低效的试错法选择激活函数。为避免这种低效的人工选择并缓解PINNs的优化困难，我们引入自适应激活函数来针对不同问题搜索最优函数。我们比较了不同的自适应激活函数，并讨论了它们在PINNs背景下的局限性。此外，我们提出将候选激活函数组合学习的思想定制应用于PINNs优化，这对所学函数的平滑性和多样性提出了更高要求。这通过从候选集中移除无法提供高阶导数的激活函数，并根据我们对给定PDE的先验知识融入具有不同性质的初等函数来实现。我们进一步利用自适应斜率增强了搜索空间。所提出的自适应激活函数可用于以可解释的方式求解不同的PDE系统。其有效性在一系列基准测试中得到验证。代码可在https://github.com/LeapLabTHU/AdaAFforPINNs获取。