Computer vision researchers are embracing two promising paradigms: Vision Transformers (ViTs) and Multi-task Learning (MTL), which both show great performance but are computation-intensive, given the quadratic complexity of self-attention in ViT and the need to activate an entire large MTL model for one task. M$^3$ViT is the latest multi-task ViT model that introduces mixture-of-experts (MoE), where only a small portion of subnetworks ("experts") are sparsely and dynamically activated based on the current task. M$^3$ViT achieves better accuracy and over 80% computation reduction but leaves challenges for efficient deployment on FPGA. Our work, dubbed Edge-MoE, solves the challenges to introduce the first end-to-end FPGA accelerator for multi-task ViT with a collection of architectural innovations, including (1) a novel reordering mechanism for self-attention, which requires only constant bandwidth regardless of the target parallelism; (2) a fast single-pass softmax approximation; (3) an accurate and low-cost GELU approximation; (4) a unified and flexible computing unit that is shared by almost all computational layers to maximally reduce resource usage; and (5) uniquely for M$^3$ViT, a novel patch reordering method to eliminate memory access overhead. Edge-MoE achieves 2.24x and 4.90x better energy efficiency comparing with GPU and CPU, respectively. A real-time video demonstration is available online, along with our code written using High-Level Synthesis, which will be open-sourced.

翻译：摘要：计算机视觉研究者正致力于两个有前景的研究范式：视觉Transformer（ViT）与多任务学习（MTL）。二者虽均展现出卓越性能，但都面临计算密集型挑战——ViT的自注意力机制具有二次复杂度，而MTL需为单个任务激活整个大模型。M$^3$ViT作为最新多任务ViT模型，引入混合专家（MoE）机制，其中仅基于当前任务稀疏且动态地激活一小部分子网络（"专家"）。M$^3$ViT在实现更高精度的同时，降低超过80%的计算量，但仍面临在FPGA上高效部署的挑战。本文提出的Edge-MoE通过一系列架构创新解决了这些难题，首次实现了面向多任务ViT的端到端FPGA加速器，具体包括：（1）新颖的自注意力重排序机制，无论目标并行度如何，仅需恒定带宽；（2）快速单次Softmax近似方法；（3）精确且低成本的GELU近似方法；（4）统一且灵活的计算单元，几乎被所有计算层共享以最大限度减少资源占用；（5）针对M$^3$ViT的独特补丁重排序方法，消除内存访问开销。与GPU和CPU相比，Edge-MoE的能效比分别提升2.24倍和4.90倍。实时视频演示及使用高级综合编写的代码均已在线公开，后续将开源。