To break the GPU memory wall for scaling deep learning workloads, a variety of architecture and system techniques have been proposed recently. Their typical approaches include memory extension with flash memory and direct storage access. However, these techniques still suffer from suboptimal performance and introduce complexity to the GPU memory management, making them hard to meet the scalability requirement of deep learning workloads today. In this paper, we present a unified GPU memory and storage architecture named G10 driven by the fact that the tensor behaviors of deep learning workloads are highly predictable. G10 integrates the host memory, GPU memory, and flash memory into a unified memory space, to scale the GPU memory capacity while enabling transparent data migrations. Based on this unified GPU memory and storage architecture, G10 utilizes compiler techniques to characterize the tensor behaviors in deep learning workloads. Therefore, it can schedule data migrations in advance by considering the available bandwidth of flash memory and host memory. The cooperative mechanism between deep learning compilers and the unified memory architecture enables G10 to hide data transfer overheads in a transparent manner. We implement G10 based on an open-source GPU simulator. Our experiments demonstrate that G10 outperforms state-of-the-art GPU memory solutions by up to 1.75$\times$, without code modifications to deep learning workloads. With the smart data migration mechanism, G10 can reach 90.3\% of the performance of the ideal case assuming unlimited GPU memory.

翻译：为突破深度学习工作负载扩展中的GPU内存墙，近期学术界提出了多种架构与系统技术。其典型方法包括基于闪存的内存扩展与直接存储访问。然而，这些技术仍存在性能次优问题，且增加了GPU内存管理的复杂度，难以满足当前深度学习工作负载的可扩展性需求。本文基于深度学习工作负载中张量行为高度可预测的特性，提出名为G10的统一GPU内存与存储架构。G10将主机内存、GPU内存与闪存整合为统一内存空间，在实现GPU内存容量扩展的同时支持透明数据迁移。基于该统一GPU内存与存储架构，G10利用编译器技术刻画深度学习工作负载中的张量行为特征，从而可根据闪存与主机内存的可用带宽预先调度数据迁移。深度学习编译器与统一内存架构的协同机制，使G10能够以透明方式隐藏数据传输开销。我们在开源GPU模拟器上实现了G10系统。实验表明，G10在无需修改深度学习工作负载代码的前提下，性能较现有顶尖GPU内存方案提升达1.75倍。通过智能数据迁移机制，G10可达理想无限GPU内存条件下90.3%的性能表现。