Countless terms of service (ToS) are being signed everyday by users all over the world while interacting with all kinds of apps and websites. More often than not, these online contracts spanning double-digit pages are signed blindly by users who simply want immediate access to the desired service. What would normally require a consultation with a legal team, has now become a mundane activity consisting of a few clicks where users potentially sign away their rights, for instance in terms of their data privacy, to countless online entities/companies. Large language models (LLMs) are good at parsing long text-based documents, and could potentially be adopted to help users when dealing with dubious clauses in ToS and their underlying privacy policies. To investigate the utility of existing models for this task, we first build a dataset consisting of 12 questions applied individually to a set of privacy policies crawled from popular websites. Thereafter, a series of open-source as well as commercial chatbots such as ChatGPT, are queried over each question, with the answers being compared to a given ground truth. Our results show that some open-source models are able to provide a higher accuracy compared to some commercial models. However, the best performance is recorded from a commercial chatbot (ChatGPT4). Overall, all models perform only slightly better than random at this task. Consequently, their performance needs to be significantly improved before they can be adopted at large for this purpose.

翻译：每天，全球用户在与各类应用和网站交互时签署无数服务条款（ToS）。这些动辄长达数十页的在线合同，往往被仅希望即时获取服务的用户盲目签署。本应需要法律团队审核的事项，现已演变为几次点击的日常操作——用户可能在此过程中向无数在线实体/公司让渡自身权利（例如数据隐私权）。大语言模型（LLM）擅长解析长篇文本文件，有望帮助用户处理服务条款及其底层隐私政策中的可疑条款。为探究现有模型在此任务中的实用性，我们首先构建了一个数据集：针对从热门网站爬取的隐私政策集合，逐条应用12个预设问题。随后，使用包括ChatGPT在内的一系列开源及商业聊天机器人对每个问题进行查询，并将答案与给定标准答案进行比对。结果表明，部分开源模型相比某些商业模型能达到更高准确率，但最佳性能由商业聊天机器人（ChatGPT4）实现。总体而言，所有模型在此任务中的表现仅略优于随机猜测。因此，在广泛采用此类模型前，其性能仍需显著提升。