By integrating tools from external APIs, Large Language Models (LLMs) have expanded their promising capabilities in a diverse spectrum of complex real-world tasks. However, testing, evaluation, and analysis of LLM tool use remain in their early stages. Most existing benchmarks rely on manually collected test cases, many of which cannot be automatically checked for semantic correctness and instead depend on static methods such as string matching. Additionally, these benchmarks often overlook the complex interactions that occur between sequential API calls, which are common in real-world applications. To fill the gap, in this paper, we introduce StateGen, an automated framework designed to generate diverse coding tasks involving sequential API interactions. StateGen combines state-machine-based API constraint solving and validation, energy-based sampling, and control-flow injection to generate executable programs. These programs are then translated into human-like natural language task descriptions through a collaboration of two LLM agents. Utilizing StateGen, we construct StateEval, a benchmark encompassing 120 verified test cases spanning across three representative scenarios: Session Service, Tensor Operation, and ElevenLabs MCP. Experimental results confirm that StateGen can effectively generate challenging and realistic API-oriented tasks, highlighting areas for improvement in current LLMs incorporating APIs.We make our framework and benchmark publicly available to support future research.

翻译：通过集成外部API工具，大语言模型（LLMs）在多样化的复杂现实任务中展现出广阔的应用潜力。然而，针对LLM工具使用的测试、评估与分析仍处于早期阶段。现有基准测试大多依赖人工收集的测试用例，其中许多无法自动检验语义正确性，而仅采用字符串匹配等静态方法。此外，这些基准测试往往忽略了现实应用中常见的顺序API调用间的复杂交互。为填补这一空白，本文提出StateGen——一个自动化框架，旨在生成涉及顺序API交互的多样化编程任务。StateGen融合了基于状态机的API约束求解与验证、基于能量的采样以及控制流注入技术，以生成可执行程序。这些程序随后通过两个LLM智能体的协作，转化为类人自然语言任务描述。利用StateGen，我们构建了StateEval基准测试集，涵盖会话服务、张量运算和ElevenLabs MCP三个典型场景的120个已验证测试用例。实验结果证实，StateGen能有效生成具有挑战性且贴近现实的API导向任务，揭示了当前集成API的LLMs需改进的领域。我们将框架与基准测试集公开，以支持未来研究。