Deep Neural Networks (DNNs) have obtained impressive performance across tasks, however they still remain as black boxes, e.g., hard to theoretically analyze. At the same time, Polynomial Networks (PNs) have emerged as an alternative method with a promising performance and improved interpretability but have yet to reach the performance of the powerful DNN baselines. In this work, we aim to close this performance gap. We introduce a class of PNs, which are able to reach the performance of ResNet across a range of six benchmarks. We demonstrate that strong regularization is critical and conduct an extensive study of the exact regularization schemes required to match performance. To further motivate the regularization schemes, we introduce D-PolyNets that achieve a higher-degree of expansion than previously proposed polynomial networks. D-PolyNets are more parameter-efficient while achieving a similar performance as other polynomial networks. We expect that our new models can lead to an understanding of the role of elementwise activation functions (which are no longer required for training PNs). The source code is available at https://github.com/grigorisg9gr/regularized_polynomials.

翻译：深度神经网络（DNN）在各种任务中取得了令人瞩目的性能，但它们仍然如同黑箱，例如难以进行理论分析。与此同时，多项式网络（PN）作为一种具有良好性能和更强可解释性的替代方法出现，但其性能尚未达到强大的DNN基线水平。本研究旨在弥补这一性能差距。我们引入了一类多项式网络，能够在六个基准测试中达到ResNet的性能水平。我们证明强正则化至关重要，并对实现性能匹配所需的具体正则化方案进行了深入研究。为了进一步论证正则化方案，我们提出了D-PolyNets，其实现了比之前提出的多项式网络更高次数的扩展。D-PolyNets在参数效率更高的情况下，性能与其他多项式网络相当。我们期待新模型能促进对逐元素激活函数作用的理解（这些函数对于训练PN已不再是必需）。源代码可在https://github.com/grigorisg9gr/regularized_polynomials获取。