Interactive machine learning (IML) is a beneficial learning paradigm in cases of limited data availability, as human feedback is incrementally integrated into the training process. In this paper, we present an IML pipeline for image captioning which allows us to incrementally adapt a pre-trained image captioning model to a new data distribution based on user input. In order to incorporate user input into the model, we explore the use of a combination of simple data augmentation methods to obtain larger data batches for each newly annotated data instance and implement continual learning methods to prevent catastrophic forgetting from repeated updates. For our experiments, we split a domain-specific image captioning dataset, namely VizWiz, into non-overlapping parts to simulate an incremental input flow for continually adapting the model to new data. We find that, while data augmentation worsens results, even when relatively small amounts of data are available, episodic memory is an effective strategy to retain knowledge from previously seen clusters.

翻译：交互式机器学习（IML）是一种在数据有限情况下有益的学习范式，它通过将人类反馈逐步整合到训练过程中。本文提出了一种用于图像描述的IML流水线，该方法能够基于用户输入，将预训练的图描述模型逐步适应于新的数据分布。为了将用户输入融入模型，我们探索了结合简单数据增强方法，以从每个新标注的数据实例中获取更大的数据批次，并采用持续学习方法防止重复更新导致的灾难性遗忘。在实验中，我们将领域特定的图像描述数据集VizWiz划分为不重叠的部分，以模拟增量输入流，使模型持续适应新数据。我们发现，尽管数据增强会降低结果性能（即便在可用数据量相对较小时），但情景记忆是一种有效策略，能够保留来自先前见过的数据簇的知识。