This paper introduces \textbf{FinMCP-Bench}, a novel benchmark for evaluating large language models (LLMs) in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 real financial MCPs and three types of samples, single tool, multi-tool, and multi-turn, allowing evaluation of models across different levels of task complexity. Using this benchmark, we systematically assess a range of mainstream LLMs and propose metrics that explicitly measure tool invocation accuracy and reasoning capabilities. FinMCP-Bench provides a standardized, practical, and challenging testbed for advancing research on financial LLM agents.

翻译：本文提出了\textbf{FinMCP-Bench}，这是一个新颖的基准测试，用于评估大语言模型通过调用金融模型上下文协议中的工具来解决真实金融问题的能力。FinMCP-Bench包含613个样本，涵盖10个主要场景和33个子场景，其中既有真实的也有合成的用户查询，以确保多样性和真实性。它集成了65个真实的金融MCP以及三种样本类型（单工具、多工具和多轮交互），从而能够评估模型在不同任务复杂度水平下的表现。利用此基准测试，我们系统评估了一系列主流大语言模型，并提出了明确衡量工具调用准确性和推理能力的指标。FinMCP-Bench为推进金融领域LLM代理的研究提供了标准化、实用且富有挑战性的测试平台。