The explosive growth of various types of big data and advances in AI technologies have catalyzed a new type of applications called multi-modal DNNs. Multi-modal DNNs are capable of interpreting and reasoning about information from multiple modalities, making them more applicable to real-world AI scenarios. In recent research, multi-modal DNNs have outperformed the best uni-modal DNN in a wide range of applications from traditional multimedia to emerging autonomous systems. However, despite their importance and superiority, very limited research attention has been devoted to understand the characteristics of multi-modal DNNs and their implications on current computing software/hardware platforms. To facilitate research and advance the understanding of these multi-modal DNN workloads, we first present MMbench, an open-source benchmark suite consisting of a set of real-world multi-modal DNN workloads with relevant performance metrics for evaluation. Then we use MMbench to conduct an in-depth analysis on the characteristics of multi-modal DNNs. We study their implications on application and programming framework, operating and scheduling system, as well as execution hardware. Finally, we conduct a case study and extend our benchmark to edge devices. We hope that our work can provide guidance for future software/hardware design and optimization to underpin multi-modal DNNs on both cloud and edge computing platforms.

翻译：各种类型的大数据爆炸性增长和AI技术的进步催化了新型的应用类型,称为多式DNNs。多式DNNs能够解释和解释多种模式的信息,使其更适用于现实世界的AI情景。在最近的研究中,多式DNNs在从传统多媒体到新兴的自主系统的广泛应用中,已经超过了最佳的单式DNN工作量。然而,尽管它们的重要性和优越性,但对多式DNNs的特点及其对当前计算机软件/硬件平台的影响的研究关注非常有限。为了便利研究和增进对这些多式DNNW工作量的理解,我们首先介绍MMBench,这是一套开放源基准套,由一套现实世界的多式DNNN工作量和相关的业绩衡量标准组成。然后,我们利用MMBench对多式DNNNs的特点进行深入分析。我们研究了其对应用和方案编制框架的影响,运行和排期系统对当前计算机软件/硬件的影响。我们先期研究了这些应用和编程框架,作为执行标准硬件的基础。最后,我们将一个软模模型扩展了我们未来的版本。