Optimizing Neural networks is a difficult task which is still not well understood. On the other hand, fixed representation methods such as kernels and random features have provable optimization guarantees but inferior performance due to their inherent inability to learn the representations. In this paper, we aim at bridging this gap by presenting a novel architecture called RedEx (Reduced Expander Extractor) that is as expressive as neural networks and can also be trained in a layer-wise fashion via a convex program with semi-definite constraints and optimization guarantees. We also show that RedEx provably surpasses fixed representation methods, in the sense that it can efficiently learn a family of target functions which fixed representation methods cannot.

翻译：优化神经网络是一项至今仍未被充分理解的困难任务。另一方面，核方法与随机特征等固定表示方法具有可证明的优化保证，但由于其无法学习表示的内在缺陷而性能较差。本文旨在弥合这一差距，提出一种名为RedEx（简化扩展提取器）的新型架构，该架构既具有与神经网络同等的表达能力，又能通过带有半定约束及优化保证的凸规划按层训练。我们还证明，RedEx在可证明意义上超越了固定表示方法——它能高效学习一类固定表示方法无法学习的目標函数族。