AcoustoBots are mobile acoustophoretic robots capable of delivering mid-air haptics, directional audio, and acoustic levitation, but existing implementations rely on scripted commands and lack an intuitive interface for real-time human control. This work presents a gesture-based visual learning framework for contactless human-swarm interaction with a multimodal AcoustoBot platform. The system combines ESP32-CAM gesture capture, PhaseSpace motion tracking, centralized processing, and an OpenCLIP-based visual learning model (VLM) with linear probing to classify three hand gestures and map them to haptics, audio, and levitation modalities. Validation accuracy improved from about 67% with a small dataset to nearly 98% with the largest dataset. In integrated experiments with two AcoustoBots, the system achieved an overall gesture-to-modality switching accuracy of 87.8% across 90 trials, with an average end-to-end latency of 3.95 seconds. These results demonstrate the feasibility of using a vision-language-model-based gesture interface for multimodal human-swarm interaction. While the current system is limited by centralized processing, a static gesture set, and controlled-environment evaluation, it establishes a foundation for more expressive, scalable, and accessible swarm robotic interfaces.

翻译：AcoustoBot是一种可移动的声流机器人，能够实现空中触觉反馈、定向音频和声悬浮，但现有实现依赖于预设指令，缺乏实时人类控制的直观接口。本文提出了一种基于手势的视觉学习框架，用于与多模态AcoustoBot平台进行非接触式人机群体交互。该系统结合ESP32-CAM手势捕捉、PhaseSpace运动跟踪、集中式处理以及基于OpenCLIP的视觉学习模型（VLM）与线性探测方法，对三种手势进行分类，并将其映射至触觉、音频和悬浮模态。验证准确率从小数据集下的约67%提升至最大数据集下的近98%。在包含两台AcoustoBot的集成实验中，系统在90次试验中实现了87.8%的手势-模态切换总体准确率，平均端到端延迟为3.95秒。这些结果证明了基于视觉-语言模型的手势接口在多模态人机群体交互中的可行性。尽管当前系统受限于集中式处理、静态手势集及受控环境评估，但它为更具表现力、可扩展性和可访问性的群体机器人接口奠定了基础。