Generative Engine Marketing (GEM) is an emerging ecosystem for monetizing generative engines, such as LLM-based chatbots, by seamlessly integrating relevant advertisements into their responses. At the core of GEM lies the generation and evaluation of ad-injected responses. However, existing benchmarks are not specifically designed for this purpose, which limits future research. To address this gap, we propose GEM-Bench, the first comprehensive benchmark for ad-injected response generation in GEM. GEM-Bench includes three curated datasets covering both chatbot and search scenarios, a metric ontology that captures multiple dimensions of user satisfaction and engagement, and several baseline solutions implemented within an extensible multi-agent framework. Our preliminary results indicate that, while simple prompt-based methods achieve reasonable engagement such as click-through rate, they often reduce user satisfaction. In contrast, approaches that insert ads based on pre-generated ad-free responses help mitigate this issue but introduce additional overhead. These findings highlight the need for future research on designing more effective and efficient solutions for generating ad-injected responses in GEM. The benchmark and all related resources are publicly available at https://gem-bench.org/.

翻译：生成式引擎营销（GEM）是一个新兴生态系统，旨在通过将相关广告无缝集成到其回复中，实现基于LLM的聊天机器人等生成式引擎的商业化。GEM的核心在于广告注入响应的生成与评估。然而，现有基准测试并非专门为此目的设计，限制了未来研究的发展。为填补这一空白，我们提出GEM-Bench，这是GEM中首个面向广告注入响应生成的综合基准。GEM-Bench包含三个涵盖聊天机器人与搜索场景的精选数据集、一个捕获用户满意度和参与度多维度的指标本体，以及多个基于可扩展多智能体框架实现的基线方案。初步结果表明，尽管简单的基于提示的方法能实现合理的参与度（如点击率），但往往会降低用户满意度。相比之下，基于预先生成的无广告响应插入广告的方法有助于缓解此问题，但会引入额外开销。这些发现凸显了未来需研究在GEM中设计更高效、更有效的广告注入响应生成方案。该基准及所有相关资源已在https://gem-bench.org/上公开。