With the rapid evolution of Large Language Model (LLM) agent ecosystems, centralized skill marketplaces have emerged as pivotal infrastructure for augmenting agent capabilities. However, these marketplaces face unprecedented security challenges, primarily stemming from semantic-behavioral inconsistency and inter-skill combinatorial risks, where individually benign skills induce malicious behaviors during collaborative invocation. To address these vulnerabilities, we propose SkillProbe, a multi-stage security auditing framework driven by multi-agent collaboration. SkillProbe introduces a "Skills-for-Skills" design paradigm, encapsulating auditing processes into standardized skill modules to drive specialized agents through a rigorous pipeline, including admission filtering, semantic-behavioral alignment detection, and combinatorial risk simulation. We conducted a large-scale evaluation using 8 mainstream LLM series across 2,500 real-world skills from ClawHub. Our results reveal a striking popularity-security paradox, where download volume is not a reliable proxy for security quality, as over 90% of high-popularity skills failed to pass rigorous auditing. Crucially, we discovered that high-risk skills form a single giant connected component within the risk-link dimension, demonstrating that cascaded risks are systemic rather than isolated occurrences. We hope that SkillProbe will inspire researchers to provide a scalable governance infrastructure for constructing a trustworthy Agentic Web. SkillProbe is accessible for public experience at skillhub.holosai.io.
翻译:随着大语言模型智能体生态系统的快速发展,集中式技能市场已成为增强智能体能力的关键基础设施。然而,此类市场面临前所未有的安全挑战,主要源于语义-行为不一致以及技能间组合风险——即单个无害技能在协同调用过程中可能诱发恶意行为。针对这些漏洞,我们提出SkillProbe,一种由多智能体协同驱动的多阶段安全审计框架。该框架引入"以技能审计技能"设计范式,将审计流程封装为标准化的技能模块,引导专业化智能体执行严格流水线,包括准入过滤、语义-行为对齐检测及组合风险模拟。我们基于ClawHub平台的8个主流大语言模型系列及2500个真实技能开展大规模评估。结果揭示了显著的流行度-安全性悖论:下载量无法可靠反映安全质量,超过90%的高流行度技能未能通过严格审计。关键在于,我们发现在风险关联维度上,高风险技能形成单一巨型连通组件,表明级联风险具有系统性而非孤立性。我们希望SkillProbe能启发研究者构建可扩展的治理基础设施,以建立可信赖的智能体网络。SkillProbe已在skillhub.holosai.io开放公众体验。