With the increasing complexity of machine learning models, managing computational resources like memory and processing power has become a critical concern. Mixed precision techniques, which leverage different numerical precisions during model training and inference to optimize resource usage, have been widely adopted. However, access to hardware that supports lower precision formats (e.g., FP8 or FP4) remains limited, especially for practitioners with hardware constraints. For many with limited resources, the available options are restricted to using 32-bit, 16-bit, or a combination of the two. While it is commonly believed that 16-bit precision can achieve results comparable to full (32-bit) precision, this study is the first to systematically validate this assumption through both rigorous theoretical analysis and extensive empirical evaluation. Our theoretical formalization of floating-point errors and classification tolerance provides new insights into the conditions under which 16-bit precision can approximate 32-bit results. This study fills a critical gap, proving for the first time that standalone 16-bit precision neural networks match 32-bit and mixed-precision in accuracy while boosting computational speed. Given the widespread availability of 16-bit across GPUs, these findings are especially valuable for machine learning practitioners with limited hardware resources to make informed decisions.

翻译：随着机器学习模型日益复杂，内存和算力等计算资源管理成为关键问题。利用不同数值精度优化模型训练与推理资源使用的混合精度技术已被广泛采用。然而，对于受硬件条件限制的从业者而言，支持低精度格式（如FP8或FP4）的硬件仍难获取。许多资源受限场景下，可选的精度方案仅限于32位、16位或两者的组合。尽管普遍认为16位精度可获得与全精度（32位）相当的结果，但本研究首次通过严谨理论分析与广泛实证评估系统验证了这一假设。我们提出的浮点误差形式化理论与分类容限分析，为16位精度逼近32位精度的适用条件提供了新视角。这项研究填补了关键空白，首次证明独立使用16位精度的神经网络可在保持准确率与32位及混合精度方法相当的同时加速计算。鉴于GPU对16位精度的广泛支持，该发现对硬件资源有限的机器学习从业者制定明智决策具有特殊价值。