We study the complexity of the problem of training neural networks defined via various activation functions. The training problem is known to be existsR-complete with respect to linear activation functions and the ReLU activation function. We consider the complexity of the problem with respect to the sigmoid activation function and other effectively continuous functions. We show that these training problems are polynomial-time many-one bireducible to the existential theory of the reals extended with the corresponding activation functions. In particular, we establish that the sigmoid activation function leads to the existential theory of the reals with the exponential function. It is thus open, and equivalent with the decidability of the existential theory of the reals with the exponential function, whether training neural networks using the sigmoid activation function is algorithmically solvable. In contrast, we obtain that the training problem is undecidable if sinusoidal activation functions are considered. Finally, we obtain general upper bounds for the complexity of the training problem in the form of low levels of the arithmetical hierarchy.

翻译：我们研究了使用不同激活函数定义神经网络训练问题的复杂度。已知对于线性激活函数和ReLU激活函数，训练问题具有existsR完全性。本文考虑sigmoid激活函数及其他有效连续函数下该问题的复杂度。我们证明这些训练问题可与相应激活函数扩展后的实数存在理论之间建立多项式时间多一双向归约。特别地，我们确立了sigmoid激活函数对应包含指数函数的实数存在理论。因此，使用sigmoid激活函数训练神经网络是否可算法求解，这一问题仍是开放的，且与包含指数函数的实数存在理论的可判定性等价。相比之下，若考虑正弦型激活函数，训练问题则是不可判定的。最后，我们以算术层级低阶形式给出了训练问题复杂度的通用上界。