In recommender systems, online A/B testing is a crucial method for evaluating the performance of different models. However, conducting online A/B testing often presents significant challenges, including substantial economic costs, user experience degradation, and considerable time requirements. With the Large Language Models' powerful capacity, LLM-based agent shows great potential to replace traditional online A/B testing. Nonetheless, current agents fail to simulate the perception process and interaction patterns, due to the lack of real environments and visual perception capability. To address these challenges, we introduce a multi-modal user agent for A/B testing (A/B Agent). Specifically, we construct a recommendation sandbox environment for A/B testing, enabling multimodal and multi-page interactions that align with real user behavior on online platforms. The designed agent leverages multimodal information perception, fine-grained user preferences, and integrates profiles, action memory retrieval, and a fatigue system to simulate complex human decision-making. We validated the potential of the agent as an alternative to traditional A/B testing from three perspectives: model, data, and features. Furthermore, we found that the data generated by A/B Agent can effectively enhance the capabilities of recommendation models. Our code is publicly available at https://github.com/Applied-Machine-Learning-Lab/ABAgent.

翻译：在推荐系统中，在线A/B测试是评估不同模型性能的关键方法。然而，进行在线A/B测试通常面临重大挑战，包括高昂的经济成本、用户体验下降以及可观的时间需求。随着大语言模型强大能力的展现，基于LLM的代理在替代传统在线A/B测试方面显示出巨大潜力。然而，由于缺乏真实环境和视觉感知能力，现有代理无法模拟感知过程和交互模式。为应对这些挑战，我们引入了一种用于A/B测试的多模态用户代理（A/B Agent）。具体而言，我们构建了一个用于A/B测试的推荐沙盒环境，支持与在线平台真实用户行为一致的多模态、多页面交互。所设计的代理利用多模态信息感知、细粒度用户偏好，并整合了用户画像、行为记忆检索以及疲劳系统，以模拟复杂的人类决策过程。我们从模型、数据和特征三个角度验证了该代理作为传统A/B测试替代方案的潜力。此外，我们发现A/B Agent生成的数据能有效提升推荐模型的能力。我们的代码已在https://github.com/Applied-Machine-Learning-Lab/ABAgent 公开。