Vision language models (VLMs) achieve strong performance on RGB imagery, but they do not generalize to thermal images. Thermal sensing plays a critical role in settings where visible light fails, including nighttime surveillance, search and rescue, autonomous driving, and medical screening. Unlike RGB imagery, thermal images encode physical temperature rather than color or texture, requiring perceptual and reasoning capabilities that existing RGB-centric benchmarks do not evaluate. We introduce ThermEval-B, a structured benchmark of approximately 55,000 thermal visual question answering pairs designed to assess the foundational primitives required for thermal vision language understanding. ThermEval-B integrates public datasets with our newly collected ThermEval-D, the first dataset to provide dense per-pixel temperature maps with semantic body-part annotations across diverse indoor and outdoor environments. Evaluating 25 open-source and closed-source VLMs, we find that models consistently fail at temperature-grounded reasoning, degrade under colormap transformations, and default to language priors or fixed responses, with only marginal gains from prompting or supervised fine-tuning. These results demonstrate that thermal understanding requires dedicated evaluation beyond RGB-centric assumptions, positioning ThermEval as a benchmark to drive progress in thermal vision language modeling.

翻译：视觉语言模型在RGB图像上表现出色，但其性能无法泛化至热成像领域。在可见光失效的场景中——包括夜间监控、搜救行动、自动驾驶与医学筛查——热传感技术发挥着关键作用。与RGB图像不同，热图像编码的是物理温度而非色彩或纹理，这要求模型具备现有以RGB为中心的基准测试所无法评估的感知与推理能力。本文提出ThermEval-B：一个包含约55,000个热视觉问答对的结构化基准，旨在系统评估热视觉语言理解所需的基础能力。该基准整合了公开数据集与我们新构建的ThermEval-D数据集——后者是首个在多样化室内外场景中提供密集逐像素温度图谱及语义化身体部位标注的数据集。通过对25个开源与闭源视觉语言模型的评估，我们发现现有模型普遍存在以下问题：基于温度的推理能力薄弱、在色彩映射变换下性能显著下降、过度依赖语言先验或固定响应模式，且提示工程或有监督微调仅能带来有限提升。这些结果表明，热视觉理解需要超越RGB中心化假设的专门化评估体系，而ThermEval基准的建立将为热视觉语言建模领域的发展提供关键驱动。