Quantization emerges as one of the most promising approaches for deploying advanced deep models on resource-constrained hardware. Mixed-precision quantization leverages multiple bit-width architectures to unleash the accuracy and efficiency potential of quantized models. However, existing mixed-precision quantization suffers exhaustive search space that causes immense computational overhead. The quantization process thus relies on separate high-performance devices rather than locally, which also leads to a significant gap between the considered hardware metrics and the real deployment.In this paper, we propose an On-chip Hardware-aware Quantization (OHQ) framework that performs hardware-aware mixed-precision quantization without accessing online devices. First, we construct the On-chip Quantization Awareness (OQA) pipeline, enabling perceive the actual efficiency metrics of the quantization operator on the hardware.Second, we propose Mask-guided Quantization Estimation (MQE) technique to efficiently estimate the accuracy metrics of operators under the constraints of on-chip-level computing power.By synthesizing network and hardware insights through linear programming, we obtain optimized bit-width configurations. Notably, the quantization process occurs on-chip entirely without any additional computing devices and data access. We demonstrate accelerated inference after quantization for various architectures and compression ratios, achieving 70% and 73% accuracy for ResNet-18 and MobileNetV3, respectively. OHQ improves latency by 15~30% compared to INT8 on deployment.

翻译：量化成为在资源受限硬件上部署先进深度学习模型最有前景的方法之一。混合精度量化利用多种位宽架构释放量化模型的精度和效率潜力。然而，现有混合精度量化存在搜索空间庞大导致计算开销巨大的问题。量化过程因此依赖独立的高性能设备而非本地执行，这还导致所考虑的硬件指标与实际部署之间存在显著差距。本文提出一种片上硬件感知量化（OHQ）框架，无需访问在线设备即可执行硬件感知混合精度量化。首先，我们构建片上量化感知（OQA）流水线，能够感知量化算子在实际硬件上的效率指标。其次，我们提出掩码引导量化估计（MQE）技术，在片上计算能力约束下高效估计算子的精度指标。通过线性规划综合网络与硬件洞察，我们获得优化的位宽配置。值得注意的是，量化过程完全在片上完成，无需额外的计算设备和数据访问。我们验证了不同架构和压缩比在量化后的推理加速效果，ResNet-18和MobileNetV3分别达到70%和73%的精度。与INT8部署相比，OHQ将延迟降低15%~30%。