Vision-Language Continual Learning (VLCL) has attracted significant research attention for its robust capabilities, and the adoption of Parameter-Efficient Fine-Tuning (PEFT) strategies is enabling these models to achieve competitive performance with substantially reduced resource consumption. However, dominated First-Order (FO) optimization is prone to trap models in suboptimal local minima, especially in limited exploration subspace within PEFT. To overcome this challenge, this paper pioneers a systematic exploration of adopting Zeroth-Order (ZO) optimization for PEFT-based VLCL. We first identify the incompatibility of naive full-ZO adoption in VLCL due to optimization process instability. We then investigate the application of ZO optimization from a modality branch-wise to a fine-grained layer-wise across various training units to identify an optimal strategy. Besides, a key theoretical insight reveals that vision modality exhibit higher variance than language counterparts in VLCL during the ZO optimization process, and we propose a modality-aware ZO strategy, which adopts gradient sign normalization in ZO and constrains vision modality perturbation to further improve performance. Benefiting from the adoption of ZO optimization, PEFT-based VLCL fulfills better ability to escape local minima during the optimization process, extensive experiments on four benchmarks demonstrate that our method achieves state-of-the-art results.

翻译：视觉语言持续学习因其强大的能力而受到广泛研究关注，采用参数高效微调策略使得这些模型能够以显著减少的资源消耗实现具有竞争力的性能。然而，占主导地位的一阶优化容易使模型陷入次优局部极小值，尤其是在参数高效微调所限定的有限探索子空间中。为克服这一挑战，本文率先对基于参数高效微调的视觉语言持续学习采用零阶优化进行了系统性探索。我们首先指出，在视觉语言持续学习中直接采用朴素的全零阶优化会因优化过程不稳定而导致不兼容。随后，我们研究了从模态分支级别到细粒度层级跨不同训练单元应用零阶优化的方法，以确定最优策略。此外，一项关键的理论洞察揭示了在零阶优化过程中，视觉模态比语言模态表现出更高的方差，我们提出了一种模态感知的零阶策略，该策略在零阶优化中采用梯度符号归一化并约束视觉模态扰动，从而进一步提升性能。得益于零阶优化的采用，基于参数高效微调的视觉语言持续学习在优化过程中获得了更强的逃离局部极小值的能力，在四个基准数据集上的大量实验表明，我们的方法取得了最先进的结果。