We present BlueFin, a benchmark that tasks large language model (LLM) agents with synthesis, manipulation, and comprehension tasks over spreadsheet workbooks in the professional finance domain. Though estimates of the global population of paying users of spreadsheet software range in the hundreds of millions -- an order of magnitude more than the estimated global population of professional developers -- comparatively fewer resources have been devoted to exploring and expanding LLM capabilities in the spreadsheet domain, with fewer still dedicated to mirroring real occupational tasks encountered by those in professional finance roles. In response, we curate a set of 131 challenging, complex tasks with real-world relevance in the domain, containing 3,225 granular rubric criteria; notably, our rubric criteria and LM judge evaluations are validated by a team of expert human annotators, resulting in high-quality, granular evaluations of complex tasks that are difficult to verify programmatically but can be reliably evaluated by an LM judge agent. Our judge achieves parity with expert consensus ($α=0.826$) with a macro-F1 score of 0.839. Frontier LLMs demonstrate poor performance on the challenging benchmark, with the strongest LLMs achieving less than 50\% average scores across tasks -- models exhibit particular weaknesses in dynamic correctness. Our contributions include a dataset of examples across three categories of spreadsheet tasks, an open source harness and agentic evaluation framework, and a characterization of existing frontier models' performance on our benchmark.

翻译：我们提出BlueFin，这是一个基准测试，旨在要求大语言模型智能体在专业金融领域对电子表格工作簿执行综合、操作和理解任务。尽管全球电子表格软件付费用户估计达数亿——比全球专业开发人员数量高一个数量级——但探索和扩展LLM在电子表格领域的资源相对较少，而专门模拟专业金融人员实际职业任务的资源更是稀缺。为此，我们整理了一组131项具有现实相关性的挑战性复杂任务，包含3,225条细粒度评分标准；值得注意的是，我们的评分标准和语言模型裁判评估由专家人工标注团队验证，从而实现了对难以通过编程验证但可由语言模型裁判智能体可靠评估的复杂任务的高质量细粒度评估。我们的裁判与专家共识一致性达到显著水平（α=0.826），宏F1分数为0.839。前沿LLM在该挑战性基准测试中表现不佳，最强模型在各任务上的平均得分低于50%——模型在动态正确性方面表现出明显短板。我们的贡献包括三类别电子表格任务的数据集、开源框架与智能体评估体系，以及现有前沿模型在该基准测试上的性能表征。