The use of multiple camera technologies in a combined multimodal monitoring system for plant phenotyping offers promising benefits. Compared to configurations that only utilize a single camera technology, cross-modal patterns can be recorded that allow a more comprehensive assessment of plant phenotypes. However, the effective utilization of cross-modal patterns is dependent on precise image registration to achieve pixel-accurate alignment, a challenge often complicated by parallax and occlusion effects inherent in plant canopy imaging. In this study, we propose a novel multimodal 3D image registration method that addresses these challenges by integrating depth information from a time-of-flight camera into the registration process. By leveraging depth data, our method mitigates parallax effects and thus facilitates more accurate pixel alignment across camera modalities. Additionally, we introduce an automated mechanism to identify and differentiate different types of occlusions, thereby minimizing the introduction of registration errors. To evaluate the efficacy of our approach, we conduct experiments on a diverse image dataset comprising six distinct plant species with varying leaf geometries. Our results demonstrate the robustness of the proposed registration algorithm, showcasing its ability to achieve accurate alignment across different plant types and camera compositions. Compared to previous methods it is not reliant on detecting plant specific image features and can thereby be utilized for a wide variety of applications in plant sciences. The registration approach principally scales to arbitrary numbers of cameras with different resolutions and wavelengths. Overall, our study contributes to advancing the field of plant phenotyping by offering a robust and reliable solution for multimodal image registration.

翻译：在植物表型分析中，采用结合多种相机技术的多模态监测系统具有显著优势。与仅使用单一相机技术的配置相比，该系统能够记录跨模态模式，从而实现对植物表型更全面的评估。然而，跨模态模式的有效利用依赖于精确的图像配准以实现像素级对齐，这一挑战常因植物冠层成像固有的视差和遮挡效应而复杂化。本研究提出了一种新颖的多模态三维图像配准方法，通过将飞行时间相机获取的深度信息整合到配准过程中来解决这些挑战。通过利用深度数据，我们的方法减轻了视差效应，从而促进了不同相机模态间更精确的像素对齐。此外，我们引入了一种自动机制来识别和区分不同类型的遮挡，从而最大限度地减少配准误差的引入。为了评估我们方法的有效性，我们在一个包含六种具有不同叶片几何形状的植物物种的多样化图像数据集上进行了实验。我们的结果证明了所提出的配准算法的鲁棒性，展示了其在不同植物类型和相机组合下实现精确对齐的能力。与先前方法相比，它不依赖于检测植物特定的图像特征，因此可广泛应用于植物科学中的多种应用场景。该配准方法原则上可扩展至任意数量、具有不同分辨率和波长的相机。总体而言，我们的研究通过为多模态图像配准提供一个鲁棒且可靠的解决方案，推动了植物表型分析领域的发展。