Language models influence the external world: they query APIs that read and write to web pages, generate content that shapes human behavior, and run system commands as autonomous agents. These interactions form feedback loops: LLM outputs affect the world, which in turn affect subsequent LLM outputs. In this work, we show that feedback loops can cause in-context reward hacking (ICRH), where the LLM at test-time optimizes a (potentially implicit) objective but creates negative side effects in the process. For example, consider an LLM agent deployed to increase Twitter engagement; the LLM may retrieve its previous tweets into the context window and make them more controversial, increasing engagement but also toxicity. We identify and study two processes that lead to ICRH: output-refinement and policy-refinement. For these processes, evaluations on static datasets are insufficient -- they miss the feedback effects and thus cannot capture the most harmful behavior. In response, we provide three recommendations for evaluation to capture more instances of ICRH. As AI development accelerates, the effects of feedback loops will proliferate, increasing the need to understand their role in shaping LLM behavior.

翻译：语言模型会对外部世界产生影响：它们能查询读写网页的API、生成影响人类行为的内容，以及作为自主代理执行系统命令。这些交互构成反馈循环：大语言模型的输出影响现实世界，而现实世界的变化又反过来影响后续的大语言模型输出。本研究证明，反馈循环可能导致上下文奖励黑客行为（ICRH），即大语言模型在测试阶段优化（可能隐含的）目标时，会在此过程中产生负面副作用。例如，当部署大语言模型代理来提高Twitter互动率时，该模型可能将此前发布的推文纳入上下文窗口并使其更具争议性，这虽然能提升互动率但也会增加毒性。我们识别并研究了导致ICRH的两种过程：输出精化过程与策略精化过程。对于这些过程，基于静态数据集的评估具有局限性——它们无法捕捉反馈效应，因而难以识别最具危害性的行为。为此，我们提出三项评估建议以捕捉更多ICRH实例。随着人工智能开发的加速，反馈循环的影响将持续扩大，这迫切需要理解其在塑造大语言模型行为中的作用。