Charts provide visual representations of data and are widely used for analyzing information, addressing queries, and conveying insights to others. Various chart-related downstream tasks have emerged recently, such as question-answering and summarization. A common strategy to solve these tasks is to fine-tune various models originally trained on vision tasks language. However, such task-specific models are not capable of solving a wide range of chart-related tasks, constraining their real-world applicability. To overcome these challenges, we introduce ChartInstruct: a novel chart-specific vision-language Instruction-following dataset comprising 191K instructions generated with 71K charts. We then present two distinct systems for instruction tuning on such datasets: (1) an end-to-end model that connects a vision encoder for chart understanding with a LLM; and (2) a pipeline model that employs a two-step approach to extract chart data tables and input them into the LLM. In experiments on four downstream tasks, we first show the effectiveness of our model--achieving a new set of state-of-the-art results. Further evaluation shows that our instruction-tuning approach supports a wide array of real-world chart comprehension and reasoning scenarios, thereby expanding the scope and applicability of our models to new kinds of tasks.

翻译：图表通过可视化的方式呈现数据，被广泛应用于信息分析、问题求解及知识传递。近年来涌现出多种图表相关的下游任务，例如问答系统与文本摘要。解决这些任务的常见策略是对各类原训练于视觉任务或语言任务的模型进行微调。然而，此类任务专用模型无法胜任广泛的图表相关任务，限制了其实际应用场景。为克服上述挑战，我们提出ChartInstruct：一个面向图表的专用视觉-语言指令遵循数据集，包含基于71K张图表生成的191K条指令。在此基础上，我们提出两套针对此类数据集的指令微调系统：（1）端到端模型，将视觉编码器与大规模语言模型（LLM）进行连接以实现图表理解；（2）流水线模型，采用两步法提取图表数据表格后输入LLM。在四项下游任务的实验中，我们首先验证了模型的有效性——取得一系列最新最优结果。进一步评估表明，我们的指令微调方法可支持多种真实的图表理解与推理场景，从而将模型的应用范围扩展至新型任务领域。