Generative AI is being leveraged to solve a variety of computer-use tasks involving desktop applications. State-of-the-art systems have focused solely on improving accuracy on leading benchmarks. However, these systems are practically unusable due to extremely high end-to-end latency (e.g., tens of minutes) for tasks that typically take humans just a few minutes to complete. To understand the cause behind this and to guide future developments of computer agents, we conduct the first study on the temporal performance of computer-use agents on OSWorld, the flagship benchmark in computer-use AI. We find that large model calls for planning, reflection, and judging account for most of the overall latency, and as an agent uses more steps to complete a task, each successive step can take 3x longer than steps at the beginning of a task. We then construct OSWorld Human, a manually annotated version of the original OSWorld dataset that contains a human-determined trajectory for each task. We evaluate 16 agents on their efficiency using OSWorld Human and found that even the best agents take 2.7-4.3x more steps than necessary.

翻译：生成式人工智能正被用于解决涉及桌面应用程序的各类计算机使用任务。现有最先进的系统仅专注于提升主要基准测试的准确率。然而，这些系统在处理人类通常只需几分钟即可完成的任务时，由于端到端延迟极高（例如数十分钟），实际上并不可用。为探究这一现象的原因并指导计算机智能体的未来发展，我们首次针对OSWorld（计算机使用人工智能领域的旗舰基准测试）上计算机使用智能体的时间性能展开研究。我们发现，用于规划、反思和判断的大型模型调用占据了整体延迟的大部分，并且随着智能体为完成任务而使用更多步骤，每一步的耗时可能比任务初始步骤长3倍。随后，我们构建了OSWorld-Human——原始OSWorld数据集的手工注释版本，其中包含每项任务的人类确定轨迹。我们使用OSWorld-Human评估了16个智能体的效率，发现即便是最优智能体，其执行步骤也比必要步骤多出2.7至4.3倍。