Many domains, from deep learning to finance, require compounding real numbers over long sequences, often leading to catastrophic numerical underflow or overflow. We introduce generalized orders of magnitude (GOOMs), a principled extension of traditional orders of magnitude that incorporates floating-point numbers as a special case, and which in practice enables stable computation over significantly larger dynamic ranges of real numbers than previously possible. We implement GOOMs, along with an efficient custom parallel prefix scan, to support native execution on parallel hardware such as GPUs. We demonstrate that our implementation of GOOMs outperforms traditional approaches with three representative experiments, all of which were previously considered impractical or impossible, and now become possible and practical: (1) compounding real matrix products far beyond standard floating-point limits; (2) estimating spectra of Lyapunov exponents in parallel, orders of magnitude faster than with previous methods, applying a novel selective-resetting method to prevent state colinearity; and (3) capturing long-range dependencies in deep recurrent neural networks with non-diagonal recurrent states, computed in parallel via a prefix scan, without requiring any form of stabilization. Our results show that our implementation of GOOMs, combined with efficient parallel scanning, offers a scalable and numerically robust alternative to conventional floating-point numbers for high-dynamic-range applications.

翻译：从深度学习到金融等多个领域，都需要对长序列中的实数进行复合运算，这常常导致灾难性的数值下溢或上溢。本文提出了广义数量级（GOOMs），这是对传统数量级概念的一种原则性扩展，它将浮点数作为特例包含在内，并在实践中实现了比以往方法更大的实数动态范围内的稳定计算。我们实现了GOOMs以及一种高效的自定义并行前缀扫描算法，以支持在GPU等并行硬件上的原生执行。通过三个代表性实验，我们证明了GOOMs的实现优于传统方法——这些实验此前均被认为不切实际或不可能实现，而现在变得可能且实用：（1）实现远超标准浮点数限制的实数矩阵乘积复合运算；（2）并行估计李雅普诺夫指数谱，速度比以往方法快数个数量级，并应用一种新颖的选择性重置方法来防止状态共线性；（3）在深度循环神经网络中捕获长程依赖关系，该网络具有非对角循环状态，通过前缀扫描并行计算，且无需任何形式的稳定性处理。我们的结果表明，结合高效并行扫描的GOOMs实现，为高动态范围应用提供了一种可扩展且数值鲁棒的替代方案，以取代传统的浮点数。