Recent advancements in pre-trained large-scale language-image models have ushered in a new era of visual comprehension, offering a significant leap forward. These breakthroughs have proven particularly instrumental in addressing long-standing challenges that were previously daunting. Leveraging these innovative techniques, this paper tackles two well-known issues within the realm of visual analytics: (1) the efficient exploration of large-scale image datasets and identification of potential data biases within them; (2) the evaluation of image captions and steering of their generation process. On the one hand, by visually examining the captions automatically generated from language-image models for an image dataset, we gain deeper insights into the semantic underpinnings of the visual contents, unearthing data biases that may be entrenched within the dataset. On the other hand, by depicting the association between visual contents and textual captions, we expose the weaknesses of pre-trained language-image models in their captioning capability and propose an interactive interface to steer caption generation. The two parts have been coalesced into a coordinated visual analytics system, fostering mutual enrichment of visual and textual elements. We validate the effectiveness of the system with domain practitioners through concrete case studies with large-scale image datasets.

翻译：近期预训练大规模语言-图像模型的进步开创了视觉理解的新纪元，带来了显著突破。这些创新在解决长期存在的棘手难题方面尤为关键。本文利用上述技术，针对可视分析领域的两个经典问题展开研究：（1）大规模图像数据集的高效探索及其中潜在数据偏差的识别；（2）图像描述的评估及其生成过程的引导。一方面，通过可视分析语言-图像模型为图像数据集自动生成的描述文本，我们能够深入洞察视觉内容的语义基础，揭示数据集中可能固有的偏差；另一方面，通过可视化视觉内容与文本描述之间的关联，我们揭示了预训练语言-图像模型在描述生成能力上的不足，并提出了一个交互式界面以引导描述生成。这两部分被整合为一个协同的可视分析系统，促进视觉与文本元素的相互增强。通过与领域从业者合作，基于大规模图像数据集的具体案例研究验证了该系统的有效性。