Neural network (NN) designed for challenging machine learning tasks is in general a highly nonlinear mapping that contains massive variational parameters. High complexity of NN, if unbounded or unconstrained, might unpredictably cause severe issues including over-fitting, loss of generalization power, and unbearable cost of hardware. In this work, we propose a general compression scheme that significantly reduces the variational parameters of NN by encoding them to multi-layer tensor networks (TN's) that contain exponentially-fewer free parameters. Superior compression performance of our scheme is demonstrated on several widely-recognized NN's (FC-2, LeNet-5, and VGG-16) and datasets (MNIST and CIFAR-10), surpassing the state-of-the-art method based on shallow tensor networks. For instance, about 10 million parameters in the three convolutional layers of VGG-16 are compressed in TN's with just $632$ parameters, while the testing accuracy on CIFAR-10 is surprisingly improved from $81.14\%$ by the original NN to $84.36\%$ after compression. Our work suggests TN as an exceptionally efficient mathematical structure for representing the variational parameters of NN's, which superiorly exploits the compressibility than the simple multi-way arrays.

翻译：针对具有挑战性机器学习任务设计的神经网络，本质上是一种包含海量变分参数的高度非线性映射。若不加约束地任由神经网络复杂度增长，可能导致过拟合、泛化能力下降及硬件成本过高等严重问题。本文提出一种通用压缩方案，通过将神经网络变分参数编码至包含指数级更少自由参数的多层张量网络中，显著降低其参数量。我们在多个广泛认可的神经网络（FC-2、LeNet-5和VGG-16）及数据集（MNIST和CIFAR-10）上验证了该方案卓越的压缩性能，超越了基于浅层张量网络的现有最优方法。例如，VGG-16的三个卷积层中约1000万个参数被压缩至仅含632个参数的张量网络，而CIFAR-10测试准确率从原始神经网络的81.14%意外提升至压缩后的84.36%。本研究表明，张量网络作为一种异常高效的数学结构，在表征神经网络变分参数时，对可压缩性的利用远优于简单的多维数组。