Artificial Intelligence (AI)-aided vision-based Structural Health Monitoring (SHM) has emerged as an effective approach for monitoring and assessing structural condition by analyzing image and video data. By integrating Computer Vision (CV) and Deep Learning (DL), vision-based SHM can automatically identify and localize visual patterns associated with structural damage. However, previous works typically generate only discrete outputs, such as damage class labels and damage region coordinates, requiring engineers to further reorganize and analyze these results for evaluation and decision-making. In late 2022, Large Language Models (LLMs) became popular across multiple fields, providing new insights into AI-aided vision-based SHM. In this study, a novel LLM-based Disaster Reconnaissance Summarization (LLM-DRS) framework is proposed. It introduces a standard reconnaissance plan in which the collection of vision data and corresponding metadata follows a well-designed on-site investigation process. Text-based metadata and image-based vision data are then processed and integrated into a unified format, where well-trained Deep Convolutional Neural Networks extract key attributes, including damage state, material type, and damage level. Finally, all data are fed into an LLM with carefully designed prompts, enabling the LLM-DRS to generate summary reports for individual structures or affected regions based on aggregated attributes and metadata. Results show that integrating LLMs into vision-based SHM, particularly for rapid post-disaster reconnaissance, demonstrates promising potential for improving resilience of the built environment through effective reconnaissance.

翻译：人工智能辅助的视觉结构健康监测已成为通过分析图像和视频数据来监测与评估结构状况的有效方法。通过整合计算机视觉与深度学习，视觉结构健康监测能够自动识别并定位与结构损伤相关的视觉模式。然而，先前的研究通常仅生成离散的输出，如损伤类别标签和损伤区域坐标，需要工程师进一步重组和分析这些结果以进行评估和决策。2022年末，大语言模型在多个领域普及，为人工智能辅助的视觉结构健康监测提供了新的思路。本研究提出了一种新颖的基于大语言模型的灾害勘测总结框架。该框架引入了一种标准勘测方案，其中视觉数据及相应元数据的收集遵循精心设计的现场调查流程。文本元数据与图像视觉数据随后被处理并整合为统一格式，由训练有素的深度卷积神经网络提取关键属性，包括损伤状态、材料类型和损伤等级。最后，所有数据通过精心设计的提示词输入大语言模型，使该框架能够基于聚合属性与元数据，为单个结构或受灾区域生成总结报告。结果表明，将大语言模型集成到视觉结构健康监测中，特别是在灾后快速勘测方面，展现出通过高效勘测提升建筑环境韧性的巨大潜力。