We introduce WorkBench: a benchmark dataset for evaluating agents' ability to execute tasks in a workplace setting. WorkBench contains a sandbox environment with five databases, 26 tools, and 690 tasks. These tasks represent common business activities, such as sending emails and scheduling meetings. The tasks in WorkBench are challenging as they require planning, tool selection, and often multiple actions. If a task has been successfully executed, one (or more) of the database values may change. The correct outcome for each task is unique and unambiguous, which allows for robust, automated evaluation. We call this key contribution outcome-centric evaluation. We evaluate five existing ReAct agents on WorkBench, finding they successfully complete as few as 3% of tasks (Llama2-70B), and just 43% for the best-performing (GPT-4). We further find that agents' errors can result in the wrong action being taken, such as an email being sent to the wrong person. WorkBench reveals weaknesses in agents' ability to undertake common business activities, raising questions about their use in high-stakes workplace settings. WorkBench is publicly available as a free resource at https://github.com/olly-styles/WorkBench.

翻译：摘要：我们提出WorkBench：一个用于评估智能体在工作场所执行任务能力的基准数据集。WorkBench包含一个沙盒环境，设有五个数据库、26种工具和690项任务。这些任务模拟常见商业活动，如发送电子邮件和安排会议。WorkBench中的任务具有挑战性，因为它们需要规划、工具选择，且通常涉及多个操作步骤。若任务成功执行，一个（或多个）数据库中的值可能发生变化。每项任务的正确结果具有唯一性和明确性，从而支持稳健的自动化评估。我们将这一关键贡献称为结果导向评估。我们在WorkBench上评估了五个现有ReAct智能体，发现它们成功完成的任务比例最低仅为3%（Llama2-70B），而表现最佳者（GPT-4）也仅达到43%。我们进一步发现，智能体的错误可能导致错误操作，例如将邮件发送给错误对象。WorkBench揭示了智能体在处理常见商业活动中的能力缺陷，对其在高风险工作环境中的应用提出了质疑。WorkBench作为免费资源公开提供于https://github.com/olly-styles/WorkBench。