On-device Deep Neural Network (DNN) inference consumes significant computing resources and development efforts. To alleviate that, we propose LUT-NN, the first system to empower inference by table lookup, to reduce inference cost. LUT-NN learns the typical features for each operator, named centroid, and precompute the results for these centroids to save in lookup tables. During inference, the results of the closest centroids with the inputs can be read directly from the table, as the approximated outputs without computations. LUT-NN integrates two major novel techniques: (1) differentiable centroid learning through backpropagation, which adapts three levels of approximation to minimize the accuracy impact by centroids; (2) table lookup inference execution, which comprehensively considers different levels of parallelism, memory access reduction, and dedicated hardware units for optimal performance. LUT-NN is evaluated on multiple real tasks, covering image and speech recognition, and nature language processing. Compared to related work, LUT-NN improves accuracy by 66% to 92%, achieving similar level with the original models. LUT-NN reduces the cost at all dimensions, including FLOPs ($\leq$ 16x), model size ($\leq$ 7x), latency ($\leq$ 6.8x), memory ($\leq$ 6.5x), and power ($\leq$ 41.7%).

翻译：设备端深度神经网络推理消耗大量计算资源与开发成本。为缓解这一问题，我们提出LUT-NN——首个通过查表法实现推理的系统，旨在降低推理开销。LUT-NN为每个算子学习典型特征（称为质心），并预计算这些质心的结果存入查找表。推理时，输入对应的最近质心结果可直接从表中读取，作为无需计算的近似输出。LUT-NN集成两项核心创新技术：（1）基于反向传播的可微质心学习，通过三级近似策略最小化质心对精度的影响；（2）查表推理执行方法，综合考量多层次并行性、内存访问优化及专用硬件单元以达成最优性能。我们在图像识别、语音识别及自然语言处理等多项真实任务上评估LUT-NN。相较于现有方法，LUT-NN的精度提升66%至92%，达到与原模型相当的水平。同时，LUT-NN在计算量（≤16倍）、模型体积（≤7倍）、延迟（≤6.8倍）、内存占用（≤6.5倍）及功耗（≤41.7%）等各维度均实现成本降低。