Reducing the number of bits needed to encode the weights and activations of neural networks is highly desirable as it speeds up their training and inference time while reducing memory consumption. For these reasons, research in this area has attracted significant attention toward developing neural networks that leverage lower-precision computing, such as mixed-precision training. Interestingly, none of the existing approaches has investigated pure 16-bit floating-point settings. In this paper, we shed light on the overlooked efficiency of pure 16-bit floating-point neural networks. As such, we provide a comprehensive theoretical analysis to investigate the factors contributing to the differences observed between 16-bit and 32-bit models. We formalize the concepts of floating-point error and tolerance, enabling us to quantitatively explain the conditions under which a 16-bit model can closely approximate the results of its 32-bit counterpart. This theoretical exploration offers perspective that is distinct from the literature which attributes the success of low-precision neural networks to its regularization effect. This in-depth analysis is supported by an extensive series of experiments. Our findings demonstrate that pure 16-bit floating-point neural networks can achieve similar or even better performance than their mixed-precision and 32-bit counterparts. We believe the results presented in this paper will have significant implications for machine learning practitioners, offering an opportunity to reconsider using pure 16-bit networks in various applications.

翻译：降低编码神经网络权重和激活值所需比特数极具吸引力，因为这能加速训练和推理过程并降低内存消耗。为此，该领域研究已聚焦于开发低精度计算神经网络（如混合精度训练）。值得注意的是，现有方法均未探索纯16位浮点环境。本文揭示了纯16位浮点神经网络被忽视的效能。我们通过全面的理论分析，探究造成16位与32位模型差异的关键因素，形式化定义了浮点误差与容错性概念，从而定量阐释16位模型逼近32位模型结果的条件。这一理论探索提供了与现有文献中“低精度神经网络成功归因于正则化效应”截然不同的视角。通过系列实验支撑的深度分析表明，纯16位浮点神经网络能达到甚至超越混合精度及32位模型的性能。我们相信本文成果将为机器学习实践者带来重要启示，推动其在各类应用中重新考量纯16位网络的部署。