The increasing size and complexity of modern deep neural networks (DNNs) pose significant challenges for on-device inference on mobile GPUs, with limited memory and computational resources. Existing DNN acceleration frameworks primarily deploy a weight preloading strategy, where all model parameters are loaded into memory before execution on mobile GPUs. We posit that this approach is not adequate for modern DNN workloads that comprise very large model(s) and possibly execution of several distinct models in succession. In this work, we introduce FlashMem, a memory streaming framework designed to efficiently execute large-scale modern DNNs and multi-DNN workloads while minimizing memory consumption and reducing inference latency. Instead of fully preloading weights, FlashMem statically determines model loading schedules and dynamically streams them on demand, leveraging 2.5D texture memory to minimize data transformations and improve execution efficiency. Experimental results on 11 models demonstrate that FlashMem achieves 2.0x to 8.4x memory reduction and 1.7x to 75.0x speedup compared to existing frameworks, enabling efficient execution of large-scale models and multi-DNN support on resource-constrained mobile GPUs.

翻译：现代深度神经网络（DNN）规模和复杂度的不断增加，对移动GPU上的设备端推理提出了重大挑战，因其内存和计算资源有限。现有的DNN加速框架主要采用权重预加载策略，即在移动GPU上执行前将所有模型参数加载到内存中。我们认为，对于包含极大模型并可能连续执行多个不同模型的现代DNN工作负载，这种方法并不适用。在本工作中，我们提出了FlashMem，这是一个内存流式处理框架，旨在高效执行大规模的现代DNN及多DNN工作负载，同时最小化内存消耗并降低推理延迟。FlashMem并非完全预加载权重，而是静态确定模型加载调度，并根据需求动态流式传输，利用2.5D纹理内存来最小化数据转换并提高执行效率。在11个模型上的实验结果表明，与现有框架相比，FlashMem实现了2.0倍至8.4倍的内存减少和1.7倍至75.0倍的加速，从而能够在资源受限的移动GPU上高效执行大规模模型并支持多DNN。