Many domains, from deep learning to finance, require compounding real numbers over long sequences, often leading to catastrophic numerical underflow or overflow. We introduce generalized orders of magnitude (GOOMs), a principled extension of traditional orders of magnitude that incorporates floating-point numbers as a special case, and which in practice enables stable computation over significantly larger dynamic ranges of real numbers than previously possible. We implement GOOMs, along with an efficient custom parallel prefix scan, to support native execution on parallel hardware such as GPUs. We demonstrate that our implementation of GOOMs outperforms traditional approaches with three representative experiments, all of which were previously considered impractical or impossible, and now become possible and practical: (1) compounding real matrix products far beyond standard floating-point limits; (2) estimating spectra of Lyapunov exponents in parallel, orders of magnitude faster than with previous methods, applying a novel selective-resetting method to prevent state colinearity; and (3) capturing long-range dependencies in deep recurrent neural networks with non-diagonal recurrent states, computed in parallel via a prefix scan, without requiring any form of stabilization. Our results show that our implementation of GOOMs, combined with efficient parallel scanning, offers a scalable and numerically robust alternative to conventional floating-point numbers for high-dynamic-range applications.

翻译：从深度学习到金融等诸多领域，都需要对长序列的实数进行复合运算，这常常导致灾难性的数值下溢或上溢。我们提出了广义数量级（GOOMs），这是对传统数量级的一种原则性扩展，它将浮点数作为特例包含在内，并在实践中能够实现比以往方法显著更大的实数动态范围内的稳定计算。我们实现了GOOMs以及一种高效的自定义并行前缀扫描算法，以支持在GPU等并行硬件上的原生执行。我们通过三个代表性实验证明，我们的GOOMs实现优于传统方法，所有这些实验先前都被认为不切实际或不可能实现，而现在变得可能且实用：(1) 进行远超标准浮点数限制的实数矩阵乘积复合运算；(2) 并行估计李雅普诺夫指数谱，速度比先前方法快数个数量级，并应用了一种新颖的选择性重置方法来防止状态共线性；(3) 在具有非对角循环状态的深度循环神经网络中捕获长程依赖关系，通过前缀扫描并行计算，无需任何形式的稳定性处理。我们的结果表明，对于高动态范围应用，我们的GOOMs实现与高效的并行扫描相结合，为传统浮点数提供了一个可扩展且数值鲁棒的替代方案。