Tool learning with foundation models aims to endow AI systems with the ability to invoke external resources -- such as APIs, computational utilities, and specialized models -- to solve complex tasks beyond the reach of standalone language generation. While recent advances in Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have expanded their reasoning and perception capabilities, existing tool-use methods are predominantly limited to text-only inputs and closed-world settings. Consequently, they struggle to interpret multimodal user instructions and cannot generalize to tools unseen during training. In this work, we introduce RaTA-Tool, a novel framework for open-world multimodal tool selection. Rather than learning direct mappings from user queries to fixed tool identifiers, our approach enables an MLLM to convert a multimodal query into a structured task description and subsequently retrieve the most appropriate tool by matching this representation against semantically rich, machine-readable tool descriptions. This retrieval-based formulation naturally supports extensibility to new tools without retraining. To further improve alignment between task descriptions and tool selection, we incorporate a preference-based optimization stage using Direct Preference Optimization (DPO). To support research in this setting, we also introduce the first dataset for open-world multimodal tool use, featuring standardized tool descriptions derived from Hugging Face model cards. Extensive experiments demonstrate that our approach significantly improves tool-selection performance, particularly in open-world, multimodal scenarios.

翻译：基于基础模型的工具学习旨在赋予AI系统调用外部资源（如API、计算工具和专用模型）的能力，以解决独立语言生成无法胜任的复杂任务。尽管大语言模型（LLMs）和多模态大语言模型（MLLMs）的最新进展增强了其推理与感知能力，但现有工具使用方法主要局限于纯文本输入和封闭世界设定。因此，这些方法既难以理解多模态用户指令，也无法泛化到训练中未见的工具。本文提出RaTA-Tool——一种面向开放世界多模态工具选择的新颖框架。该方法不学习从用户查询到固定工具标识符的直接映射，而是使MLLM能够将多模态查询转化为结构化任务描述，随后通过将该表示与语义丰富且机器可读的工具描述进行匹配，检索出最合适的工具。这种基于检索的范式天然支持在不重新训练的情况下扩展至新工具。为进一步提升任务描述与工具选择的对齐度，我们引入基于直接偏好优化（DPO）的偏好优化阶段。为支持该场景下的研究，我们还发布了首个面向开放世界多模态工具使用的数据集，其中包含从Hugging Face模型卡中提取的标准化工具描述。广泛实验表明，我们的方法显著提升了工具选择性能，尤其在开放世界多模态场景中。