Generative Engine Marketing (GEM) is an emerging ecosystem for monetizing generative engines, such as LLM-based chatbots, by seamlessly integrating relevant advertisements into their responses. At the core of GEM lies the generation and evaluation of ad-injected responses. However, existing benchmarks are not specifically designed for this purpose, which limits future research. To address this gap, we propose GEM-Bench, the first comprehensive benchmark for ad-injected response generation in GEM. GEM-Bench includes three curated datasets covering both chatbot and search scenarios, a metric ontology that captures multiple dimensions of user satisfaction and engagement, and several baseline solutions implemented within an extensible multi-agent framework. Our preliminary results indicate that, while simple prompt-based methods achieve reasonable engagement such as click-through rate, they often reduce user satisfaction. In contrast, approaches that insert ads based on pre-generated ad-free responses help mitigate this issue but introduce additional overhead. These findings highlight the need for future research on designing more effective and efficient solutions for generating ad-injected responses in GEM. The benchmark and all related resources are publicly available at https://gem-bench.org/.

翻译：生成引擎营销（GEM）是一种新兴的生态系统，旨在通过将相关广告无缝集成到生成引擎（如基于LLM的聊天机器人）的回复中来实现盈利。GEM的核心在于广告注入式回复的生成与评估。然而，现有基准测试并非专门为此目的设计，这限制了未来的研究。为填补这一空白，我们提出了GEM-Bench，这是首个针对GEM中广告注入式回复生成的综合性基准测试。GEM-Bench包含三个精心构建的数据集，涵盖聊天机器人和搜索场景；一个捕捉用户满意度与参与度多维度指标的度量体系；以及若干在可扩展多智能体框架内实现的基线解决方案。初步结果表明，虽然基于简单提示的方法能够实现合理的参与度（如点击率），但往往会降低用户满意度。相比之下，基于预生成无广告回复进行广告插入的方法有助于缓解这一问题，但会引入额外开销。这些发现凸显了未来研究需要设计更高效、有效的GEM广告注入式回复生成方案。本基准测试及相关资源已公开于https://gem-bench.org/。