GEneral Matrix Multiply (GEMM) is a central operation in deep learning and corresponds to the largest chunk of the compute footprint. Therefore, improving its efficiency is an active topic of ongoing research. A popular strategy is the use of low bit-width integers to approximate the original entries in a matrix. This allows efficiency gains, but often requires sophisticated techniques to control the rounding error incurred. In this work, we first verify/check that when the low bit-width restriction is removed, for a variety of Transformer-based models, whether integers are sufficient for all GEMMs need -- for {\em both} training and inference stages, and can achieve parity with floating point counterparts. No sophisticated techniques are needed. We find that while a large majority of entries in matrices (encountered in such models) can be easily represented by {\em low} bit-width integers, the existence of a few heavy hitter entries make it difficult to achieve efficiency gains via the exclusive use of low bit-width GEMMs alone. To address this issue, we develop a simple algorithm, Integer Matrix Unpacking (IM-Unpack), to {\em unpack} a matrix with large integer entries into a larger matrix whose entries all lie within the representable range of arbitrarily low bit-width integers. This allows {\em equivalence} with the original GEMM, i.e., the exact result can be obtained using purely low bit-width integer GEMMs. This comes at the cost of additional operations -- we show that for many popular models, this overhead is quite small.

翻译：通用矩阵乘法（GEMM）是深度学习中的核心运算，占据计算开销的最大部分。因此，提升其效率一直是活跃的研究方向。一种常见策略是使用低位宽整数近似矩阵中的原始元素。这种方法虽能带来效率提升，但往往需要复杂技术来控制舍入误差。本文首先验证了：在移除低位宽限制的条件下，对于多种基于Transformer的模型，整数是否足以满足所有GEMM需求——包括训练和推理阶段——并能够达到与浮点运算相当的性能，且无需复杂技术。我们发现，尽管此类模型矩阵中的绝大部分元素可以轻松用低位宽整数表示，但少数"重头"元素的存在使得单纯使用低位宽GEMM难以实现效率提升。针对这一问题，我们提出了一种简单算法——整数矩阵解包（IM-Unpack），该算法将包含大整数元素的矩阵解包为更大的矩阵，使所有元素均落在任意低位宽整数的可表示范围内。这使得原GEMM的等价性得到保证，即能够纯粹通过低位宽整数GEMM获得精确结果。该算法以额外操作为代价——我们证明，对于许多主流模型，这种开销非常小。