Continual learning of pretrained vision-language models (VLMs) is prone to catastrophic forgetting, yet current approaches adapt to new tasks without explicitly preserving the cross-modal semantic geometry inherited from pretraining and previous stages, allowing new-task supervision to induce geometric distortion. We observe that the most pronounced drift tends to concentrate in vulnerable neighborhoods near the old-new semantic interface, where shared visual patterns are easily re-explained by new textual semantics. To address this under an exemplar-free constraint, we propose Semantic Geometry Preservation for Continual Learning (SeGP-CL). SeGP-CL first probes the drift-prone region by constructing a compact set of adversarial anchors with dual-targeted projected gradient descent (DPGD), which drives selected new-task seeds toward old-class semantics while remaining faithful in raw visual space. During training, we preserve cross-modal structure by anchor-guided cross-modal geometry distillation (ACGD), and stabilize the textual reference frame across tasks via a lightweight text semantic-geometry regularization (TSGR). After training, we estimate anchor-induced raw-space drift to transfer old visual prototypes and perform dual-path inference by fusing cross-modal and visual cues. Extensive experiments on five continual learning benchmarks demonstrate that SeGP-CL consistently improves stability and forward transfer, achieving state-of-the-art performance while better preserving semantic geometry of VLMs.

翻译：预训练视觉语言模型（VLM）的持续学习容易遭受灾难性遗忘，然而现有方法在适应新任务时未能显式地保持从预训练及先前阶段继承的跨模态语义几何结构，导致新任务的监督信号引发几何失真。我们观察到，最显著的语义漂移往往集中在旧-新语义交界处的脆弱邻域内，该区域的共享视觉模式易被新任务的文本语义重新解释。为在无样本约束下解决此问题，我们提出用于持续学习的语义几何保持方法（SeGP-CL）。该方法首先通过构建紧凑的对抗锚点集来探测易漂移区域，该锚点集采用双目标投影梯度下降法（DPGD）生成，在保持原始视觉空间保真度的同时，将选定的新任务种子样本推向旧类语义。训练过程中，我们通过锚点引导的跨模态几何蒸馏（ACGD）保持跨模态结构，并借助轻量级文本语义几何正则化（TSGR）稳定跨任务的文本参考框架。训练完成后，我们通过估计锚点诱导的原始空间漂移来迁移旧视觉原型，并融合跨模态与视觉线索进行双路径推理。在五个持续学习基准上的大量实验表明，SeGP-CL能持续提升模型稳定性与前向迁移能力，在实现最先进性能的同时，更好地保持了视觉语言模型的语义几何结构。