In intelligent video surveillance, cameras record image sequences during day and night. Commonly, this demands different sensors. To achieve a better performance it is not unusual to combine them. We focus on the case that a long-wave infrared camera records continuously and in addition to this, another camera records in the visible spectral range during daytime and an intelligent algorithm supervises the picked up imagery. More accurate, our task is multispectral CNN-based object detection. At first glance, images originating from the visible spectral range differ between thermal infrared ones in the presence of color and distinct texture information on the one hand and in not containing information about thermal radiation that emits from objects on the other hand. Although color can provide valuable information for classification tasks, effects such as varying illumination and specialties of different sensors still represent significant problems. Anyway, obtaining sufficient and practical thermal infrared datasets for training a deep neural network poses still a challenge. That is the reason why training with the help of data from the visible spectral range could be advantageous, particularly if the data, which has to be evaluated contains both visible and infrared data. However, there is no clear evidence of how strongly variations in thermal radiation, shape, or color information influence classification accuracy. To gain deeper insight into how Convolutional Neural Networks make decisions and what they learn from different sensor input data, we investigate the suitability and robustness of different augmentation techniques...

翻译：在智能视频监控中，摄像机需要昼夜拍摄图像序列。这通常需要不同的传感器。为了获得更好的性能，将它们组合使用并不罕见。我们关注以下场景：一台长波红外摄像机持续记录，同时另一台摄像机在白天记录可见光谱段，并由智能算法监控采集的图像。更准确地说，我们的任务是基于多光谱CNN的目标检测。直观上看，可见光谱段的图像与热红外图像的区别在于：前者包含颜色和清晰的纹理信息，而后者不包含物体发出的热辐射信息。尽管颜色能为分类任务提供有价值的信息，但光照变化和不同传感器特性等因素仍构成显著问题。此外，获取充足且实用的热红外数据集来训练深度神经网络仍是一大挑战。这就是为何借助可见光谱段数据进行训练可能具有优势，特别是当待评估数据同时包含可见光和红外数据时。然而，热辐射、形状或颜色信息的变化对分类精度的影响程度尚无明确证据。为深入理解卷积神经网络的决策机制及其从不同传感器输入数据中学到的特征，我们研究了不同增强技术的适用性和鲁棒性。