Automated image captioning has the potential to be a useful tool for people with vision impairments. Images taken by this user group are often noisy, which leads to incorrect and even unsafe model predictions. In this paper, we propose a quality-agnostic framework to improve the performance and robustness of image captioning models for visually impaired people. We address this problem from three angles: data, model, and evaluation. First, we show how data augmentation techniques for generating synthetic noise can address data sparsity in this domain. Second, we enhance the robustness of the model by expanding a state-of-the-art model to a dual network architecture, using the augmented data and leveraging different consistency losses. Our results demonstrate increased performance, e.g. an absolute improvement of 2.15 on CIDEr, compared to state-of-the-art image captioning networks, as well as increased robustness to noise with up to 3 points improvement on CIDEr in more noisy settings. Finally, we evaluate the prediction reliability using confidence calibration on images with different difficulty/noise levels, showing that our models perform more reliably in safety-critical situations. The improved model is part of an assisted living application, which we develop in partnership with the Royal National Institute of Blind People.

翻译：自动图像描述有望成为视力障碍者的实用工具。该用户群体拍摄的图像往往带有噪声，这会导致模型预测错误甚至引发安全隐患。本文提出一种质量无关框架，旨在提升面向视力障碍人群的图像描述模型的性能与鲁棒性。我们从数据、模型和评估三个维度切入：首先，通过生成合成噪声的数据增强技术，解决该领域数据稀疏性问题；其次，将先进模型扩展为双网络架构，利用增强数据并融合多种一致性损失函数来增强模型鲁棒性。实验结果表明，相较于当前最优图像描述网络，我们的方法在CIDEr指标上取得2.15的绝对提升，且在噪声更严重的场景下鲁棒性提升达3个CIDEr点。最后，我们通过置信度校准对不同难度/噪声等级的图像进行预测可靠性评估，证明模型在安全关键场景中具有更可靠的性能。该改进模型已集成至与皇家盲人协会合作开发的辅助生活应用中。