Large Language Models (LLMs) excel in natural language tasks, but less is known about their reasoning capabilities over tabular data. Prior analyses devise evaluation strategies that poorly reflect an LLM's realistic performance on tabular queries. Moreover, we have a limited understanding of the robustness of LLMs towards realistic variations in tabular inputs. Therefore, we ask: Can general-purpose LLMs reason over tabular data, really?, and focus on two questions 1) are tabular reasoning capabilities of general-purpose LLMs robust to real-world characteristics of tabular inputs, and 2) how can we realistically evaluate an LLM's performance on analytical tabular queries? Building on a recent tabular reasoning benchmark, we first surface shortcomings of its multiple-choice prompt evaluation strategy, as well as commonly used free-form text metrics such as SacreBleu and BERT-score. We show that an LLM-as-a-judge procedure yields more reliable performance insights and unveil a significant deficit in tabular reasoning performance of LLMs. We then extend the tabular inputs reflecting three common characteristics in practice: 1) missing values, 2) duplicate entities, and 3) structural variations. Experiments show that the tabular reasoning capabilities of general-purpose LLMs suffer from these variations, stressing the importance of improving their robustness for realistic tabular inputs.

翻译：大型语言模型（LLMs）在自然语言任务中表现出色，但其在表格数据上的推理能力尚不明确。先前的研究所设计的评估策略未能准确反映LLMs在表格查询任务中的实际性能。此外，我们对于LLMs应对表格输入中现实变化的鲁棒性理解有限。因此，我们提出核心问题：通用型LLMs是否真正具备表格数据推理能力？并聚焦于两个具体问题：1）通用型LLMs的表格推理能力是否对表格输入的现实特征具有鲁棒性；2）如何对LLMs在分析性表格查询中的性能进行真实评估？基于近期提出的表格推理基准测试，我们首先揭示了其多项选择提示评估策略的缺陷，以及SacreBleu、BERT-score等常用自由文本评估指标的局限性。研究表明，采用LLM作为评判者的评估流程能提供更可靠的性能分析，并暴露出LLMs在表格推理性能上的显著不足。随后，我们扩展了表格输入以反映实践中常见的三种特征：1）缺失值，2）重复实体，3）结构变异。实验表明，通用型LLMs的表格推理能力受这些变化的影响显著，这凸显了提升其应对现实表格输入鲁棒性的重要性。