Mixed-precision training is a crucial technique for scaling deep learning models, but successful mixedprecision training requires identifying and applying the right combination of training methods. This paper presents our preliminary study on Mixture-of-Representations (MoR), a novel, per-tensor and sub-tensor level quantization framework that dynamically analyzes a tensor's numerical properties to select between a variety of different representations. Based on the framework, we have proposed and experimented concrete algorithms that choose dynamically between FP8 and BF16 representations for both per-tensor and sub-tensor level granularities. Our universal approach is designed to preserve model quality across various quantization partition strategies and datasets. Our initial findings show that this approach can achieve state-of-the-art results with 98.38% of tensors quantized to the FP8 format. This work highlights the potential of dynamic, property-aware quantization while preserving model quality. We believe this approach can generally improve the robustness of low precision training, as demonstrated by achieving FP8 accuracies that are on par with existing approaches without the need for fine-grain partitioning, or can be used in combination with other training methods to improve the leverage of even lower precision number formats such as NVFP4.

翻译：混合精度训练是扩展深度学习模型规模的关键技术，但成功的混合精度训练需要识别并应用恰当的训练方法组合。本文介绍了我们对表示混合方法的初步研究，这是一种新颖的、基于张量及子张量级别的量化框架，通过动态分析张量的数值特性，在多种不同表示形式之间进行选择。基于该框架，我们提出并实验了具体算法，能够在张量级别和子张量级别粒度上动态选择FP8与BF16表示。我们的通用方法旨在保持模型质量，适用于各种量化分区策略和数据集。初步结果表明，该方法能够实现最先进的性能，其中98.38%的张量被量化为FP8格式。这项工作凸显了在保持模型质量的前提下，动态且基于属性感知的量化方法的潜力。我们相信该方法能够普遍提升低精度训练的鲁棒性，正如实验所证实的：无需细粒度分区即可达到与现有方法相当的FP8精度，或可与其他训练方法结合，以更好地利用NVFP4等更低精度的数值格式。