One of the prevailing trends in the machine- and deep-learning community is to gravitate towards the use of increasingly larger models in order to keep pushing the state-of-the-art performance envelope. This tendency makes access to the associated technologies more difficult for the average practitioner and runs contrary to the desire to democratize knowledge production in the field. In this paper, we propose a framework for achieving improved memory efficiency in the process of learning traditional neural networks by leveraging inductive-bias-driven network design principles and layer-wise manifold-oriented regularization objectives. Use of the framework results in improved absolute performance and empirical generalization error relative to traditional learning techniques. We provide empirical validation of the framework, including qualitative and quantitative evidence of its effectiveness on two standard image datasets, namely CIFAR-10 and CIFAR-100. The proposed framework can be seamlessly combined with existing network compression methods for further memory savings.

翻译：机器学习和深度学习领域的一个普遍趋势是越来越倾向于使用更大的模型，以不断推动最先进性能的边界。这一倾向使得普通从业者更难接触到相关技术，并且与推动该领域知识生产民主化的愿望背道而驰。在本文中，我们提出了一种框架，通过利用基于归纳偏置的网络设计原则和逐层流形导向的正则化目标，在传统神经网络的学习过程中实现了更高的内存效率。与传统学习方法相比，使用该框架可提升绝对性能和经验泛化误差。我们在两个标准图像数据集（即CIFAR-10和CIFAR-100）上提供了该框架的实证验证，包括其有效性的定性和定量证据。所提出的框架可以无缝地与现有的网络压缩方法相结合，以进一步节省内存。