The demand for real-time visual understanding and interaction in complex scenarios is increasingly critical for unmanned aerial vehicles. However, a significant challenge arises from the contradiction between the high computational cost of large Vision language models and the limited computing resources available on UAV edge devices. To address this challenge, this paper proposes a lightweight multimodal task platform based on BLIP-2, integrated with YOLO-World and YOLOv8-Seg models. This integration extends the multi-task capabilities of BLIP-2 for UAV applications with minimal adaptation and without requiring task-specific fine-tuning on drone data. Firstly, the deep integration of BLIP-2 with YOLO models enables it to leverage the precise perceptual results of YOLO for fundamental tasks like object detection and instance segmentation, thereby facilitating deeper visual-attention understanding and reasoning. Secondly, a content-aware key frame sampling mechanism based on K-Means clustering is designed, which incorporates intelligent frame selection and temporal feature concatenation. This equips the lightweight BLIP-2 architecture with the capability to handle video-level interactive tasks effectively. Thirdly, a unified prompt optimization scheme for multi-task adaptation is implemented. This scheme strategically injects structured event logs from the YOLO models as contextual information into BLIP-2's input. Combined with output constraints designed to filter out technical details, this approach effectively guides the model to generate accurate and contextually relevant outputs for various tasks.

翻译：在复杂场景中对实时视觉理解与交互的需求对于无人机而言日益关键。然而，大型视觉语言模型的高计算成本与无人机边缘设备有限的计算资源之间的矛盾构成了重大挑战。为应对这一挑战，本文提出了一种基于BLIP-2的轻量级多模态任务平台，该平台集成了YOLO-World与YOLOv8-Seg模型。此集成以最小的适配工作扩展了BLIP-2面向无人机应用的多任务能力，且无需在无人机数据上进行任务特定的微调。首先，BLIP-2与YOLO模型的深度融合使其能够利用YOLO在目标检测与实例分割等基础任务上的精确感知结果，从而促进更深层次的视觉注意力理解与推理。其次，设计了一种基于K-Means聚类的内容感知关键帧采样机制，该机制融合了智能帧选择与时间特征拼接。这使轻量化的BLIP-2架构具备了有效处理视频级交互任务的能力。第三，实现了一种面向多任务适配的统一提示优化方案。该方案策略性地将来自YOLO模型的结构化事件日志作为上下文信息注入BLIP-2的输入。结合为过滤技术细节而设计的输出约束，该方法能有效引导模型为各类任务生成准确且与上下文相关的输出。