Emerging multimodal large language models (MLLMs) exhibit great potential for chart question answering (CQA). Recent efforts primarily focus on scaling up training datasets (i.e., charts, data tables, and question-answer (QA) pairs) through data collection and synthesis. However, our empirical study on existing MLLMs and CQA datasets reveals notable gaps. First, current data collection and synthesis focus on data volume and lack consideration of fine-grained visual encodings and QA tasks, resulting in unbalanced data distribution divergent from practical CQA scenarios. Second, existing work follows the training recipe of the base MLLMs initially designed for natural images, under-exploring the adaptation to unique chart characteristics, such as rich text elements. To fill the gap, we propose a visualization-referenced instruction tuning approach to guide the training dataset enhancement and model development. Specifically, we propose a novel data engine to effectively filter diverse and high-quality data from existing datasets and subsequently refine and augment the data using LLM-based generation techniques to better align with practical QA tasks and visual encodings. Then, to facilitate the adaptation to chart characteristics, we utilize the enriched data to train an MLLM by unfreezing the vision encoder and incorporating a mixture-of-resolution adaptation strategy for enhanced fine-grained recognition. Experimental results validate the effectiveness of our approach. Even with fewer training examples, our model consistently outperforms state-of-the-art CQA models on established benchmarks. We also contribute a dataset split as a benchmark for future research. Source codes and datasets of this paper are available at https://github.com/zengxingchen/ChartQA-MLLM.

翻译：新兴的多模态大语言模型（MLLMs）在图表问答（CQA）任务中展现出巨大潜力。近期研究主要通过数据收集与合成来扩大训练数据集（即图表、数据表及问答对）的规模。然而，我们对现有MLLMs和CQA数据集的实证研究揭示了显著差距。首先，当前的数据收集与合成侧重于数据量，缺乏对细粒度视觉编码和问答任务的考量，导致数据分布不平衡，与实际CQA场景存在偏差。其次，现有工作遵循了最初为自然图像设计的基础MLLMs的训练方案，未能充分探索对图表独特特性（如丰富的文本元素）的适应。为填补这一空白，我们提出了一种基于可视化参考的指令微调方法，以指导训练数据集的增强和模型开发。具体而言，我们设计了一种新颖的数据引擎，能够从现有数据集中有效筛选出多样且高质量的数据，随后利用基于LLM的生成技术对这些数据进行细化和增强，以更好地契合实际问答任务和视觉编码。接着，为促进对图表特性的适应，我们利用增强后的数据训练MLLM，通过解冻视觉编码器并引入混合分辨率适应策略，以提升细粒度识别能力。实验结果验证了我们方法的有效性。即使在训练样本较少的情况下，我们的模型在现有基准测试中始终优于最先进的CQA模型。我们还贡献了一个数据集划分，作为未来研究的基准。本文的源代码和数据集可在 https://github.com/zengxingchen/ChartQA-MLLM 获取。