The rapid integration of Large Language Models (LLMs) into Multi-Agent Systems (MAS) has significantly enhanced their collaborative problem-solving capabilities, but it has also expanded their attack surfaces, exposing them to vulnerabilities such as prompt infection and compromised inter-agent communication. While emerging graph-based anomaly detection methods show promise in protecting these networks, the field currently lacks a standardized, reproducible environment to train these models and evaluate their efficacy. To address this gap, we introduce Gammaf (Graph-based Anomaly Monitoring for LLM Multi-Agent systems Framework), an open-source benchmarking platform. Gammaf is not a novel defense mechanism itself, but rather a comprehensive evaluation architecture designed to generate synthetic multi-agent interaction datasets and benchmark the performance of existing and future defense models. The proposed framework operates through two interdependent pipelines: a Training Data Generation stage, which simulates debates across varied network topologies to capture interactions as robust attributed graphs, and a Defense System Benchmarking stage, which actively evaluates defense models by dynamically isolating flagged adversarial nodes during live inference rounds. Through rigorous evaluation using established defense baselines (XG-Guard and BlindGuard) across multiple knowledge tasks (such as MMLU-Pro and GSM8K), we demonstrate Gammaf's high utility, topological scalability, and execution efficiency. Furthermore, our experimental results reveal that equipping an LLM-MAS with effective attack remediation not only recovers system integrity but also substantially reduces overall operational costs by facilitating early consensus and cutting off the extensive token generation typical of adversarial agents.

翻译：大型语言模型（LLMs）与多智能体系统（MAS）的快速融合显著增强了其协作解决问题的能力，但同时也扩大了攻击面，使其面临提示感染和智能体间通信受侵等漏洞威胁。尽管新兴的基于图的异常检测方法在保护这些网络方面展现出潜力，但该领域目前缺乏标准化、可复现的环境来训练这些模型并评估其有效性。为填补这一空白，我们提出Gammaf（面向LLM多智能体系统的基于图异常监测框架），一个开源基准测试平台。Gammaf本身并非新型防御机制，而是一个综合性评估架构，旨在生成合成多智能体交互数据集，并基准测试现有及未来防御模型的性能。该框架通过两条相互依赖的流水线运行：训练数据生成阶段，该阶段模拟不同网络拓扑结构上的辩论过程，将交互行为捕获为鲁棒的属性图；以及防御系统基准测试阶段，该阶段通过在实时推理轮次中动态隔离被标记的对抗节点，来主动评估防御模型。通过使用既定防御基线（XG-Guard和BlindGuard）在多项知识任务（如MMLU-Pro和GSM8K）上进行严格评估，我们证明了Gammaf的高效用性、拓扑可扩展性和执行效率。此外，实验结果表明，为LLM-MAS配备有效的攻击修复机制不仅能恢复系统完整性，还能通过促进早期共识、切断对抗智能体典型的冗长令牌生成过程，显著降低整体运行成本。