Due to complex interactions among various deep neural network (DNN) optimization techniques, modern DNNs can have weights and activations that are dense or sparse with diverse sparsity degrees. To offer a good trade-off between accuracy and hardware performance, an ideal DNN accelerator should have high flexibility to efficiently translate DNN sparsity into reductions in energy and/or latency without incurring significant complexity overhead. This paper introduces hierarchical structured sparsity (HSS), with the key insight that we can systematically represent diverse sparsity degrees by having them hierarchically composed from multiple simple sparsity patterns. As a result, HSS simplifies the underlying hardware since it only needs to support simple sparsity patterns; this significantly reduces the sparsity acceleration overhead, which improves efficiency. Motivated by such opportunities, we propose a simultaneously efficient and flexible accelerator, named HighLight, to accelerate DNNs that have diverse sparsity degrees (including dense). Due to the flexibility of HSS, different HSS patterns can be introduced to DNNs to meet different applications' accuracy requirements. Compared to existing works, HighLight achieves a geomean of up to 6.4x better energy-delay product (EDP) across workloads with diverse sparsity degrees, and always sits on the EDP-accuracy Pareto frontier for representative DNNs

翻译：摘要：由于各类深度神经网络（DNN）优化技术之间复杂的交互作用，现代DNN的权重和激活值可能呈现密集或稀疏状态，且稀疏程度各异。为在精度与硬件性能之间实现良好权衡，理想的DNN加速器应具备高度灵活性，能够高效地将DNN稀疏性转化为能耗和/或延迟的降低，同时避免引入显著的复杂开销。本文提出层级结构化稀疏（HSS），其核心洞见在于：通过将多种简单稀疏模式进行层级组合，可以系统性地表征不同稀疏程度。基于此，HSS简化了底层硬件设计——仅需支持简单的稀疏模式即可；这显著降低了稀疏加速开销，从而提升了效率。受此启发，我们提出一种兼具高效性与灵活性的加速器HighLight，用于加速包含不同稀疏程度（包括密集情况）的DNN。凭借HSS的灵活性，可将不同的HSS模式引入DNN以满足不同应用的精度需求。与现有工作相比，HighLight在包含不同稀疏程度的工作负载上几何平均能效积（EDP）提升最高达6.4倍，且在代表性DNN的EDP-精度帕累托前沿上始终占据最优位置。