In the early days of the web, giving the same web page to different browsers could provide very different results. As the rendering engine behind each browser would differ, some elements of a page could break or be positioned in the wrong location. At that time, the User Agent (UA) string was introduced for content negotiation. By knowing the browser used to connect to the server, a developer could provide a web page that was tailored for that specific browser to remove any usability problems. Over the past three decades, the UA string remained exposed by browsers, but its current usefulness is being debated. Browsers now adopt the exact same standards and use the same languages to display the same content to users, bringing the question if the content of the UA string is still relevant today, or if it is a relic of the past. Moreover, the diversity of means to browse the web has become so large that the UA string is one of the top contributors to tracking users in the field of browser fingerprinting, bringing a sense of urgency to deprecate it. In this paper, our goal is to understand the impact of the UA on the web and if this legacy string is still actively used to adapt the content served to users. We introduce UA-Radar, a web page similarity measurement tool that compares in-depth two web pages from the code to their actual rendering, and highlights the similarities it finds. We crawled 270, 048 web pages from 11, 252 domains using 3 different browsers and 2 different UA strings to observe that 100% of the web pages were similar before any JavaScript was executed, demonstrating the absence of differential serving. Our experiments also show that only a very small number of websites are affected by the lack of UA information, which can be fixed in most cases by updating code to become browser-agnostic. Our study brings some proof that it may be time to turn the page on the UA string and retire it from current web browsers.

翻译：在万维网早期，不同浏览器访问同一网页可能产生迥异的结果。由于各浏览器渲染引擎存在差异，网页元素可能显示异常或错位。当时引入用户代理（UA）字符串以实现内容协商：开发者通过识别连接服务器的浏览器类型，可提供针对该浏览器优化的网页以消除可用性问题。三十年来，浏览器始终暴露UA字符串，但其当前效用正引发争议。如今浏览器已采用完全相同的标准与语言为用户呈现相同内容，这不禁令人质疑UA字符串是否仍具现实意义，抑或已成历史遗留物。更关键的是，网络浏览方式的激增使得UA字符串成为浏览器指纹识别领域中追踪用户的主要依据，这加剧了弃用该机制的紧迫性。本研究旨在探究UA对网络的影响，以及这个传统字符串是否仍被用于调整用户内容。我们提出UA-Radar——一种网页相似性度量工具，可从代码到实际渲染效果深度比较两个网页并标定其相似性。通过使用3种不同浏览器及2种不同UA字符串，对来自11252个域名的270048个网页进行爬取分析，我们发现所有网页在执行JavaScript前相似度达100%，证实不存在差异化内容分发。实验同时表明，仅极少数网站受缺乏UA信息的影响，且多数情况可通过更新代码实现浏览器无关性修复。研究成果证明，或许已到该将UA字符串尘封于历史，并从现行浏览器中移除的时机。