Multimodal Large Language Models (mLLMs) are often used to answer questions in structured data such as tables in Markdown, JSON, and images. While these models can often give correct answers, users also need to know where those answers come from. In this work, we study structured data attribution/citation, which is the ability of the models to point to the specific rows and columns that support an answer. We evaluate several mLLMs across different table formats and prompting strategies. Our results show a clear gap between question answering and evidence attribution. Although question answering accuracy remains moderate, attribution accuracy is much lower, near random for JSON inputs, across all models. We also find that models are more reliable at citing rows than columns, and struggle more with textual formats than images. Finally, we observe notable differences across model families. Overall, our findings show that current mLLMs are unreliable at providing fine-grained, trustworthy attribution for structured data, which limits their usage in applications requiring transparency and traceability.

翻译：多模态大语言模型（mLLMs）常被用于回答基于结构化数据（如Markdown、JSON格式的表格及图像表格）的问题。尽管这些模型通常能给出正确答案，但用户也需要了解这些答案的来源。在本研究中，我们关注结构化数据的归因/引用能力，即模型能否准确指向支撑答案的具体行与列。我们评估了多种mLLMs在不同表格格式与提示策略下的表现。结果显示，问题回答与证据归因之间存在明显差距：尽管问题回答准确率保持中等水平，但所有模型在归因准确率上均显著更低，对于JSON输入甚至接近随机水平。我们还发现，模型在引用行数据时比引用列数据更可靠，且处理文本格式表格时比图像格式表格面临更大困难。最后，我们观察到不同模型系列之间存在显著差异。总体而言，我们的研究结果表明，当前mLLMs在提供细粒度、可信的结构化数据归因方面尚不可靠，这限制了其在需要透明度和可追溯性的应用场景中的使用。