The development of a satisfying and rigorous mathematical understanding of the performance of neural networks is a major challenge in artificial intelligence. Against this background, we study the expressive power of neural networks through the example of the classical NP-hard Knapsack Problem. Our main contribution is a class of recurrent neural networks (RNNs) with rectified linear units that are iteratively applied to each item of a Knapsack instance and thereby compute optimal or provably good solution values. We show that an RNN of depth four and width depending quadratically on the profit of an optimum Knapsack solution is sufficient to find optimum Knapsack solutions. We also prove the following tradeoff between the size of an RNN and the quality of the computed Knapsack solution: for Knapsack instances consisting of $n$ items, an RNN of depth five and width $w$ computes a solution of value at least $1-\mathcal{O}(n^2/\sqrt{w})$ times the optimum solution value. Our results build upon a classical dynamic programming formulation of the Knapsack Problem as well as a careful rounding of profit values that are also at the core of the well-known fully polynomial-time approximation scheme for the Knapsack Problem. A carefully conducted computational study qualitatively supports our theoretical size bounds. Finally, we point out that our results can be generalized to many other combinatorial optimization problems that admit dynamic programming solution methods, such as various Shortest Path Problems, the Longest Common Subsequence Problem, and the Traveling Salesperson Problem.

翻译：对神经网络性能建立令人满意且严谨的数学理解是人工智能领域的重大挑战。在此背景下，我们通过经典NP难问题——背包问题来研究神经网络的表达能力。我们的主要贡献是提出一类具有整流线性单元的循环神经网络（RNN），该网络迭代地应用于背包问题实例的每个物品，从而计算出最优或可证明的优良解值。我们证明：深度为四、宽度与背包最优解利润值呈二次依赖关系的RNN足以找到背包问题的最优解。我们还证明了RNN规模与计算所得背包解质量之间的以下权衡关系：对于包含$n$个物品的背包实例，深度为五、宽度为$w$的RNN可计算出不低于最优解值$1-\mathcal{O}(n^2/\sqrt{w})$倍的解值。我们的结果建立在背包问题的经典动态规划公式以及对利润值的精细舍入基础上，该舍入技术也是背包问题著名完全多项式时间近似方案的核心。精心设计的计算实验从定性角度支持了我们提出的理论规模界限。最后，我们指出该结果可推广至许多其他允许动态规划求解方法的组合优化问题，例如各类最短路径问题、最长公共子序列问题以及旅行商问题。