The rise of Large Language Model (LLM)-based web agents represents a significant shift in automated interactions with the web. Unlike traditional crawlers that follow simple conventions, such as robots$.$txt, modern agents engage with websites in sophisticated ways: navigating complex interfaces, extracting structured information, and completing end-to-end tasks. Existing governance mechanisms were not designed for these capabilities. Without a way to specify what interactions are and are not allowed, website owners increasingly rely on blanket blocking and CAPTCHAs, which undermine beneficial applications such as efficient automation, convenient use of e-commerce services, and accessibility tools. We introduce agent-permissions$.$json, a robots$.$txt-style lightweight manifest where websites specify allowed interactions, complemented by API references where available. This framework provides a low-friction coordination mechanism: website owners only need to write a simple JSON file, while agents can easily parse and automatically implement the manifest's provisions. Website owners can then focus on blocking non-compliant agents, rather than agents as a whole. By extending the spirit of robots$.$txt to the era of LLM-mediated interaction, and complementing data use initiatives such as AIPref, the manifest establishes a compliance framework that enables beneficial agent interactions while respecting site owners' preferences.
翻译:基于大型语言模型(LLM)的网络代理的兴起,标志着网络自动化交互方式的重大变革。与传统爬虫遵循简单规范(如robots.txt)不同,现代代理能以复杂方式与网站交互:导航复杂界面、提取结构化信息并完成端到端任务。现有治理机制并非为这些能力设计。由于缺乏明确允许与禁止交互的规范方式,网站所有者日益依赖全面屏蔽和验证码机制,这反而损害了高效自动化、便捷电商服务及无障碍工具等有益应用。我们提出agent-permissions.json——一种仿robots.txt风格的轻量级清单机制,允许网站声明许可的交互行为,并在可行时提供API参考。该框架建立了低摩擦协调机制:网站所有者仅需编写简单JSON文件,代理则可轻松解析并自动执行清单条款。网站所有者因而能专注于拦截违规代理,而非全盘禁止代理访问。通过将robots.txt的设计理念延伸至LLM中介交互时代,并补充AIPref等数据使用倡议,该清单建立了一个合规框架,在尊重网站所有者偏好的同时,为有益的代理交互提供支持。