In this paper, we design novel interactive deep learning methods to improve semantic interactions in visual analytics applications. The ability of semantic interaction to infer analysts' precise intents during sensemaking is dependent on the quality of the underlying data representation. We propose the $\text{DeepSI}_{\text{finetune}}$ framework that integrates deep learning into the human-in-the-loop interactive sensemaking pipeline, with two important properties. First, deep learning extracts meaningful representations from raw data, which improves semantic interaction inference. Second, semantic interactions are exploited to fine-tune the deep learning representations, which then further improves semantic interaction inference. This feedback loop between human interaction and deep learning enables efficient learning of user- and task-specific representations. To evaluate the advantage of embedding the deep learning within the semantic interaction loop, we compare $\text{DeepSI}_{\text{finetune}}$ against a state-of-the-art but more basic use of deep learning as only a feature extractor pre-processed outside of the interactive loop. Results of two complementary studies, a human-centered qualitative case study and an algorithm-centered simulation-based quantitative experiment, show that $\text{DeepSI}_{\text{finetune}}$ more accurately captures users' complex mental models with fewer interactions.

翻译：本文设计了新颖的交互式深度学习方法，以提升可视分析应用中语义交互的效果。在意义建构过程中，语义交互推断分析师精确意图的能力依赖于底层数据表示的质量。我们提出了DeepSI_finetune框架，将深度学习集成到人在回路的交互式意义构建流程中，该框架具有两个重要特性：首先，深度学习从原始数据中提取有意义的表示，从而改进语义交互推断；其次，利用语义交互对深度学习表示进行微调，进而进一步优化语义交互推断。这种人类交互与深度学习之间的反馈循环能够高效学习用户与任务特定的表示。为评估将深度学习嵌入语义交互循环的优势，我们将DeepSI_finetune与一种仅将深度学习作为交互循环外预处理的特征提取器的现有基础方法进行对比。两项互补研究——以人为中心的定性案例研究和以算法为中心的仿真定量实验——结果表明，DeepSI_finetune能以更少的交互次数更准确地捕获用户复杂的思维模型。