Semantic segmentation in adverse weather scenarios is a critical task for autonomous driving systems. While foundation models have shown promise, the need for specialized adaptors becomes evident for handling more challenging scenarios. We introduce DiffPrompter, a novel differentiable visual and latent prompting mechanism aimed at expanding the learning capabilities of existing adaptors in foundation models. Our proposed $\nabla$HFC image processing block excels particularly in adverse weather conditions, where conventional methods often fall short. Furthermore, we investigate the advantages of jointly training visual and latent prompts, demonstrating that this combined approach significantly enhances performance in out-of-distribution scenarios. Our differentiable visual prompts leverage parallel and series architectures to generate prompts, effectively improving object segmentation tasks in adverse conditions. Through a comprehensive series of experiments and evaluations, we provide empirical evidence to support the efficacy of our approach. Project page at https://diffprompter.github.io.

翻译：恶劣天气场景下的语义分割是自动驾驶系统的关键任务。尽管基础模型已展现出潜力，但在处理更具挑战性的场景时，专用适配器的必要性变得尤为突出。我们提出DiffPrompter——一种新颖的可微分视觉与隐式提示机制，旨在扩展基础模型中现有适配器的学习能力。我们提出的$\nabla$HFC图像处理模块在恶劣天气条件下表现尤为出色，而传统方法在此类场景中常难以奏效。此外，我们研究了联合训练视觉提示与隐式提示的优势，并证明这种组合方法能显著提升模型在分布外场景中的表现。我们的可微分视觉提示利用并行与串行架构生成提示，有效改善了恶劣条件下的物体分割任务。通过一系列全面的实验与评估，我们提供了支持该方法有效性的实证依据。项目页面：https://diffprompter.github.io。