Large Vision-Language Models (VLMs) have demonstrated impressive performance on complex tasks involving visual input with natural language instructions. However, it remains unclear to what extent capabilities on natural images transfer to Earth observation (EO) data, which are predominantly satellite and aerial images less common in VLM training data. In this work, we propose a comprehensive benchmark to gauge the progress of VLMs toward being useful tools for EO data by assessing their abilities on scene understanding, localization and counting, and change detection tasks. Motivated by real-world applications, our benchmark includes scenarios like urban monitoring, disaster relief, land use, and conservation. We discover that, although state-of-the-art VLMs like GPT-4V possess extensive world knowledge that leads to strong performance on open-ended tasks like location understanding and image captioning, their poor spatial reasoning limits usefulness on object localization and counting tasks. Our benchmark will be made publicly available at https://vleo.danielz.ch/ and on Hugging Face at https://huggingface.co/collections/mit-ei/vleo-benchmark-datasets-65b789b0466555489cce0d70 for easy model evaluation.

翻译：大型视觉语言模型（VLMs）在涉及自然语言指令与视觉输入的复杂任务中已展现出令人瞩目的性能。然而，这些模型在自然图像上的能力在多大程度能迁移至地球观测（EO）数据（主要是卫星和航拍图像，在VLM训练数据中较少出现）仍不明确。本文提出一个综合基准，通过评估VLMs在场景理解、定位与计数及变化检测任务上的能力，衡量其成为EO数据实用工具的进展。受实际应用驱动，我们的基准包括城市监测、灾害救援、土地利用和生态保护等场景。研究发现，尽管GPT-4V等最先进VLMs具备广泛的世界知识，在位置理解和图像描述等开放式任务中表现优异，但其较差的空间推理能力限制了在目标定位与计数任务中的实用性。我们将在https://vleo.danielz.ch/ 和Hugging Face（https://huggingface.co/collections/mit-ei/vleo-benchmark-datasets-65b789b0466555489cce0d70）上公开该基准，以方便模型评估。