Language-Guided object recognition in remote sensing imagery is crucial for large-scale mapping and automated data annotation. However, existing open-vocabulary and visual grounding methods rely on explicit category cues, limiting their ability to handle complex or implicit queries that require advanced reasoning. To address this issue, we introduce a new suite of tasks, including Instruction-Oriented Object Counting, Detection, and Segmentation (InstructCDS), covering open-vocabulary, open-ended, and open-subclass scenarios. We further present EarthInstruct, the first InstructCDS benchmark for earth observation. It is constructed from two diverse remote sensing datasets with varying spatial resolutions and annotation rules across 20 categories, necessitating models to interpret dataset-specific instructions. Given the scarcity of semantically rich labeled data in remote sensing, we propose InstructSAM, a training-free framework for instruction-driven object recognition. InstructSAM leverages large vision-language models to interpret user instructions and estimate object counts, employs SAM2 for mask proposal, and formulates mask-label assignment as a binary integer programming problem. By integrating semantic similarity with counting constraints, InstructSAM efficiently assigns categories to predicted masks without relying on confidence thresholds. Experiments demonstrate that InstructSAM matches or surpasses specialized baselines across multiple tasks while maintaining near-constant inference time regardless of object count, reducing output tokens by 89% and overall runtime by over 32% compared to direct generation approaches. We believe the contributions of the proposed tasks, benchmark, and effective approach will advance future research in developing versatile object recognition systems.

翻译：遥感影像中的语言引导目标识别对于大规模制图和自动化数据标注至关重要。然而，现有的开放词汇和视觉定位方法依赖于显式的类别提示，限制了其处理需要高级推理的复杂或隐式查询的能力。为解决此问题，我们引入了一套新任务，包括面向指令的目标计数、检测与分割（InstructCDS），涵盖开放词汇、开放域和开放子类场景。我们进一步提出了EarthInstruct，首个面向地球观测的InstructCDS基准。它构建自两个具有不同空间分辨率和标注规则的多样化遥感数据集，涵盖20个类别，要求模型能够解析数据集特定的指令。鉴于遥感领域语义丰富的标注数据稀缺，我们提出了InstructSAM，一种用于指令驱动目标识别的免训练框架。InstructSAM利用大型视觉-语言模型解析用户指令并估计目标数量，采用SAM2进行掩码提议，并将掩码-标签分配建模为一个二元整数规划问题。通过整合语义相似性与计数约束，InstructSAM能够高效地为预测掩码分配类别，而无需依赖置信度阈值。实验表明，InstructSAM在多项任务中达到或超越了专用基线方法，同时保持几乎恒定的推理时间（与目标数量无关），与直接生成方法相比，输出标记减少了89%，总运行时间降低了超过32%。我们相信，所提出的任务、基准和有效方法的贡献将推动未来开发通用目标识别系统的研究。