Quantization emerges as one of the most promising approaches for deploying advanced deep models on resource-constrained hardware. Mixed-precision quantization leverages multiple bit-width architectures to unleash the accuracy and efficiency potential of quantized models. However, existing mixed-precision quantization suffers exhaustive search space that causes immense computational overhead. The quantization process thus relies on separate high-performance devices rather than locally, which also leads to a significant gap between the considered hardware metrics and the real deployment. In this paper, we propose an On-chip Hardware-aware Quantization (OHQ) framework that performs hardware-aware mixed-precision quantization without accessing online devices. First, we construct the On-chip Quantization Awareness (OQA) pipeline, enabling perceive the actual efficiency metrics of the quantization operator on the hardware. Second, we propose Mask-guided Quantization Estimation (MQE) technique to efficiently estimate the accuracy metrics of operators under the constraints of on-chip-level computing power. By synthesizing network and hardware insights through linear programming, we obtain optimized bit-width configurations. Notably, the quantization process occurs on-chip entirely without any additional computing devices and data access. We demonstrate accelerated inference after quantization for various architectures and compression ratios, achieving 70% and 73% accuracy for ResNet-18 and MobileNetV3, respectively. OHQ improves latency by 15~30% compared to INT8 on deployment.

翻译：量化已成为在资源受限硬件上部署先进深度模型最具前景的方法之一。混合精度量化利用多种位宽架构充分释放量化模型的精度与效率潜力。然而，现有混合精度量化面临巨大的搜索空间，导致极高的计算开销。量化过程因此需依赖单独的高性能设备而非本地完成，这还造成所考虑的硬件指标与实际部署之间存在显著差距。本文提出一种片上硬件感知量化（OHQ）框架，无需连接在线设备即可实现硬件感知的混合精度量化。首先，我们构建片上量化感知（OQA）流水线，能够感知量化算子在硬件上的实际效率指标。其次，我们提出掩码引导量化估计（MQE）技术，在片上计算能力约束下高效估计算子的精度指标。通过线性规划融合网络与硬件的洞察，我们获得优化的位宽配置。值得注意的是，量化过程完全在片上完成，无需任何额外计算设备与数据访问。我们展示了不同架构与压缩率下量化后的推理加速效果，ResNet-18与MobileNetV3分别达到70%和73%的准确率。与INT8部署相比，OHQ将延迟降低15%~30%。