Attention-based Neural Networks (NN) have demonstrated their effectiveness in accurate memory access prediction, an essential step in data prefetching. However, the substantial computational overheads associated with these models result in high inference latency, limiting their feasibility as practical prefetchers. To close the gap, we propose a new approach based on tabularization that significantly reduces model complexity and inference latency without sacrificing prediction accuracy. Our novel tabularization methodology takes as input a distilled, yet highly accurate attention-based model for memory access prediction and efficiently converts its expensive matrix multiplications into a hierarchy of fast table lookups. As an exemplar of the above approach, we develop DART, a prefetcher comprised of a simple hierarchy of tables. With a modest 0.09 drop in F1-score, DART reduces 99.99% of arithmetic operations from the large attention-based model and 91.83% from the distilled model. DART accelerates the large model inference by 170x and the distilled model by 9.4x. DART has comparable latency and storage costs as state-of-the-art rule-based prefetcher BO but surpasses it by 6.1% in IPC improvement, resulting in a 37.6% speed-up. DART outperforms state-of-the-art NN-based prefetchers TransFetch by 33.1% and Voyager by 37.2% in terms of IPC improvement, primarily due to its low prefetching latency.

翻译：基于注意力的神经网络在内存访问预测（数据预取的关键步骤）中展现了卓越的准确性。然而，这类模型带来的巨大计算开销导致推理延迟较高，限制了其作为实用预取器的可行性。为弥合这一差距，我们提出了一种基于表格化的新方法，该方法在不牺牲预测准确性的前提下，显著降低了模型复杂度与推理延迟。我们的新型表格化方法以蒸馏后仍保持高准确性的注意力模型作为内存访问预测的输入，并将其昂贵的矩阵乘法高效转化为层级化的快速查表操作。作为上述方法的实例，我们开发了DART——一种由简单表格层级构成的预取器。在F1分数仅下降0.09的情况下，DART将大型注意力模型的算术运算量减少了99.99%，将蒸馏模型的运算量减少了91.83%。DART将大型模型推理加速170倍，将蒸馏模型推理加速9.4倍。在延迟与存储成本方面，DART与最先进的基于规则的预取器BO相当，但在IPC提升上超越其6.1%，实现了37.6%的加速。在IPC提升方面，DART分别超越最先进的基于神经网络的预取器TransFetch（33.1%）和Voyager（37.2%），主要得益于其低预取延迟。