The increasing computational and memory demands in deep learning present significant challenges, especially in resource-constrained environments. We introduce a zero-order quantized optimization (ZOQO) method designed for training models with quantized parameters and operations. Our approach leverages zero-order approximations of the gradient sign and adapts the learning process to maintain the parameters' quantization without the need for full-precision gradient calculations. We demonstrate the effectiveness of ZOQO through experiments in fine-tuning of large language models and black-box adversarial attacks. Despite the limitations of zero-order and quantized operations training, our method achieves competitive performance compared to full-precision methods, highlighting its potential for low-resource environments.

翻译：深度学习日益增长的计算与内存需求带来了严峻挑战，尤其在资源受限的环境中。本文提出一种零阶量化优化（ZOQO）方法，专为训练具有量化参数与运算的模型而设计。该方法利用梯度符号的零阶近似，并调整学习过程以维持参数量化，无需全精度梯度计算。我们通过在大型语言模型微调与黑盒对抗攻击中的实验验证了ZOQO的有效性。尽管零阶与量化运算训练存在固有局限，本方法仍取得了与全精度方法相竞争的性能，彰显了其在低资源环境中的应用潜力。