Meteorological heatmaps play a vital role in deciphering extreme weather phenomena, yet their inherent complexities marked by irregular contours, unstructured patterns, and complex color variations present unique analytical hurdles for state-of-the-art Vision-Language Models (VLMs). Current state-of-the-art models like GPT-4o, Qwen-VL, and LLaVA 1.6 struggle with tasks such as precise color identification and spatial localization, resulting in inaccurate or incomplete interpretations. To address these challenges, we introduce Sparse Position and Outline Tracking (SPOT), a novel algorithm specifically designed to process irregularly shaped colored regions in visual data. SPOT identifies and localizes these regions by extracting their spatial coordinates, enabling structured representations of irregular shapes. Building on SPOT, we construct ClimateIQA, a novel meteorological visual question answering (VQA) dataset, comprising 26,280 high-resolution heatmaps and 762,120 instruction samples for wind gust, total precipitation, wind chill index and heat index analysis. ClimateIQA enhances VLM training by incorporating spatial cues, geographic metadata, and reanalysis data, improving model accuracy in interpreting and describing extreme weather features. Furthermore, we develop Climate-Zoo, a suite of fine-tuned VLMs based on SPOT-empowered ClimateIQA, which significantly outperforms existing models in meteorological heatmap tasks.

翻译：气象热图在解读极端天气现象中起着至关重要的作用，然而其固有的复杂性——表现为不规则的轮廓、非结构化的模式和复杂的颜色变化——给最先进的视觉语言模型带来了独特的分析障碍。当前最先进的模型，如GPT-4o、Qwen-VL和LLaVA 1.6，在精确颜色识别和空间定位等任务上存在困难，导致解释不准确或不完整。为应对这些挑战，我们引入了稀疏位置与轮廓追踪算法，这是一种专门设计用于处理视觉数据中不规则形状彩色区域的新算法。SPOT通过提取这些区域的空间坐标来识别和定位它们，从而实现对不规则形状的结构化表征。基于SPOT，我们构建了ClimateIQA，一个新颖的气象视觉问答数据集，包含26,280张高分辨率热图和762,120个指令样本，用于阵风、总降水量、风寒指数和酷热指数分析。ClimateIQA通过融入空间线索、地理元数据和再分析数据，增强了VLM的训练，提高了模型在解释和描述极端天气特征方面的准确性。此外，我们开发了Climate-Zoo，一套基于SPOT赋能的ClimateIQA进行微调的VLM集合，其在气象热图任务上的表现显著优于现有模型。