Honey bees pollinate about one-third of the world's food supply, but bee colonies have alarmingly declined by nearly 40% over the past decade due to several factors, including pesticides and pests. Traditional methods for monitoring beehives, such as human inspection, are subjective, disruptive, and time-consuming. To overcome these limitations, artificial intelligence has been used to assess beehive health. However, previous studies have lacked an end-to-end solution and primarily relied on data from a single source, either bee images or sounds. This study introduces a comprehensive system consisting of bee object detection and health evaluation. Additionally, it utilized a combination of visual and audio signals to analyze bee behaviors. An Attention-based Multimodal Neural Network (AMNN) was developed to adaptively focus on key features from each type of signal for accurate bee health assessment. The AMNN achieved an overall accuracy of 92.61%, surpassing eight existing single-signal Convolutional Neural Networks and Recurrent Neural Networks. It outperformed the best image-based model by 32.51% and the top sound-based model by 13.98% while maintaining efficient processing times. Furthermore, it improved prediction robustness, attaining an F1-score higher than 90% across all four evaluated health conditions. The study also shows that audio signals are more reliable than images for assessing bee health. By seamlessly integrating AMNN with image and sound data in a comprehensive bee health monitoring system, this approach provides a more efficient and non-invasive solution for the early detection of bee diseases and the preservation of bee colonies.

翻译：蜜蜂为全球约三分之一粮食供应提供授粉服务，但过去十年间，受农药及病虫害等多重因素影响，蜂群数量惊人地减少了近40%。传统蜂箱监测方法（如人工检查）存在主观性强、干扰性大且耗时较长等局限。为突破这些限制，人工智能已被用于评估蜂箱健康状态。然而，既有研究缺乏端到端解决方案，且主要依赖来自单一数据源（蜜蜂图像或声音）的分析。本研究提出了一套由蜜蜂目标检测和健康评估组成的综合系统，同时融合视觉与音频信号分析蜜蜂行为。我们开发了基于注意力的多模态神经网络（AMNN），能够自适应聚焦各类信号的关键特征以实现精准的蜜蜂健康评估。该模型综合准确率达92.61%，超越八种现有的单信号卷积神经网络与循环神经网络：相较最优图像模型性能提升32.51%，对比最优声音模型提升13.98%，同时保持高效处理速度。此外，模型预测鲁棒性显著增强，在全部四种评估的健康状态下F1分数均高于90%。研究还表明，音频信号在评估蜜蜂健康方面比图像更具可靠性。通过将AMNN与图像及声音数据无缝集成至综合性蜜蜂健康监测系统，本方法为早期发现蜂群疾病及保护蜂群提供了更高效、无创的解决方案。