Modern media firms require automated and efficient methods to identify content that is most engaging and appealing to users. Leveraging a large-scale dataset from Upworthy (a news publisher), which includes 17,681 headline A/B tests, we first investigate the ability of three pure-LLM approaches to identify the catchiest headline: prompt-based methods, embedding-based methods, and fine-tuned open-source LLMs. Prompt-based approaches perform poorly, while both OpenAI-embedding-based models and the fine-tuned Llama-3-8B achieve marginally higher accuracy than random predictions. In sum, none of the pure-LLM-based methods can predict the best-performing headline with high accuracy. We then introduce the LLM-Assisted Online Learning Algorithm (LOLA), a novel framework that integrates Large Language Models (LLMs) with adaptive experimentation to optimize content delivery. LOLA combines the best pure-LLM approach with the Upper Confidence Bound algorithm to allocate traffic and maximize clicks adaptively. Our numerical experiments on Upworthy data show that LOLA outperforms the standard A/B test method (the current status quo at Upworthy), pure bandit algorithms, and pure-LLM approaches, particularly in scenarios with limited experimental traffic. Our approach is scalable and applicable to content experiments across various settings where firms seek to optimize user engagement, including digital advertising and social media recommendations.

翻译：现代媒体公司需要自动化且高效的方法来识别最能吸引用户的内容。基于Upworthy（一家新闻出版商）的大规模数据集（包含17,681组标题A/B测试），我们首先探究了三种纯LLM方法识别最具吸引力标题的能力：基于提示的方法、基于嵌入的方法以及微调的开源LLM。基于提示的方法表现较差，而基于OpenAI嵌入的模型与微调后的Llama-3-8B模型仅能取得略高于随机预测的准确率。总体而言，所有基于纯LLM的方法均无法高精度预测最优表现标题。为此，我们提出了LLM辅助在线学习算法（LOLA），这是一个将大语言模型与自适应实验相结合以优化内容推送的新型框架。LOLA融合了最优纯LLM方法与置信上界算法，通过自适应流量分配实现点击最大化。在Upworthy数据上的数值实验表明，LOLA在实验流量受限的场景下，显著优于标准A/B测试方法（Upworthy当前采用的基准）、纯赌博机算法及纯LLM方法。该框架具备可扩展性，可广泛应用于数字广告、社交媒体推荐等需要优化用户参与度的各类内容实验场景。