Quantization emerges as one of the most promising approaches for deploying advanced deep models on resource-constrained hardware. Mixed-precision quantization leverages multiple bit-width architectures to unleash the accuracy and efficiency potential of quantized models. However, existing mixed-precision quantization suffers exhaustive search space that causes immense computational overhead. The quantization process thus relies on separate high-performance devices rather than locally, which also leads to a significant gap between the considered hardware metrics and the real deployment. In this paper, we propose an On-chip Hardware-aware Quantization (OHQ) framework that performs hardware-aware mixed-precision quantization without accessing online devices. First, we construct the On-chip Quantization Awareness (OQA) pipeline, enabling perceive the actual efficiency metrics of the quantization operator on the hardware. Second, we propose Mask-guided Quantization Estimation (MQE) technique to efficiently estimate the accuracy metrics of operators under the constraints of on-chip-level computing power. By synthesizing network and hardware insights through linear programming, we obtain optimized bit-width configurations. Notably, the quantization process occurs on-chip entirely without any additional computing devices and data access. We demonstrate accelerated inference after quantization for various architectures and compression ratios, achieving 70% and 73% accuracy for ResNet-18 and MobileNetV3, respectively. OHQ improves latency by 15~30% compared to INT8 on deployment.

翻译：量化是资源受限硬件上部署先进深度模型最具前景的方法之一。混合精度量化利用多种位宽架构来释放量化模型的精度与效率潜力。然而，现有混合精度量化面临庞大的搜索空间，导致巨大的计算开销。量化过程因此依赖外部高性能设备而非本地执行，这也在所考虑的硬件指标与实际部署之间造成显著差距。本文提出一种名为芯片级硬件感知量化（OHQ）的框架，能够在无需联网设备的情况下执行硬件感知混合精度量化。首先，我们构建了芯片级量化感知（OQA）流水线，使系统能够感知量化算子在硬件上的实际效率指标。其次，我们提出掩码引导量化估计（MQE）技术，在芯片级计算能力约束下高效估计算子的精度指标。通过线性规划综合网络与硬件洞察，获得优化的位宽配置。值得注意的是，量化过程完全在芯片上执行，无需任何额外计算设备与数据访问。我们展示了多种架构与压缩比下量化后的加速推理，ResNet-18与MobileNetV3的准确率分别达到70%与73%。与部署时的INT8相比，OHQ将延迟降低了15%~30%。