Huawei's openPangu-Embedded-1B and openPangu-Embedded-7B, variants of the openPangu large language model, integrate three distinct Chain-of-Thought (CoT) reasoning paradigms, namely slow_think, auto_think, and no_think. While these CoT modes enhance reasoning capabilities, their generation of extended reasoning traces introduces substantial memory and latency overheads, posing challenges for practical deployment on Ascend NPUs. This paper addresses these computational constraints by leveraging low-bit quantization, which transforms FP16 computations into more efficient integer arithmetic. We introduce a unified low-bit inference framework, supporting INT8 (W8A8) and W4A8 quantization, specifically optimized for openPangu-Embedded models on the Atlas A2. Our comprehensive evaluation, conducted across all three CoT modes on code generation benchmarks (HumanEval and MBPP), demonstrates the efficacy of this approach. INT8 quantization consistently preserves over 90\% of the FP16 baseline accuracy and achieves a 1.5x prefill speedup on the Atlas A2. Furthermore, W4A8 quantization significantly reduces memory consumption, albeit with a moderate trade-off in accuracy. These findings collectively indicate that low-bit quantization effectively facilitates efficient CoT reasoning on Ascend NPUs, maintaining high model fidelity.

翻译：华为的openPangu-Embedded-1B和openPangu-Embedded-7B作为openPangu大语言模型的变体，集成了三种不同的思维链推理范式，即slow_think、auto_think和no_think。尽管这些CoT模式增强了推理能力，但其生成扩展推理轨迹的过程带来了显著的内存和延迟开销，为在昇腾NPU上的实际部署带来了挑战。本文通过利用低位量化技术来解决这些计算约束，该技术将FP16计算转换为更高效的整数运算。我们引入了一个统一的低位推理框架，支持INT8（W8A8）和W4A8量化，并专门针对Atlas A2平台上的openPangu-Embedded模型进行了优化。我们在代码生成基准测试集（HumanEval和MBPP）上对所有三种CoT模式进行的全面评估证明了该方法的有效性。INT8量化始终能保持超过90\%的FP16基线精度，并在Atlas A2上实现了1.5倍的预填充加速。此外，W4A8量化显著降低了内存消耗，尽管在精度上存在适度的权衡。这些发现共同表明，低位量化能有效促进昇腾NPU上的高效CoT推理，同时保持较高的模型保真度。