Current Large Language Models (LLMs) are gradually exploited in practically valuable agentic workflows such as Deep Research, E-commerce recommendation, and job recruitment. In these applications, LLMs need to select some optimal solutions from massive candidates, which we term as \textit{LLM-as-a-Recommender} paradigm. However, the reliability of using LLM agents for recommendations is underexplored. In this work, we introduce a \textbf{Bias} \textbf{Rec}ommendation \textbf{Bench}mark (\textbf{BiasRecBench}) to highlight the critical vulnerability of such agents to biases in high-value real-world tasks. The benchmark includes three practical domains: paper review, e-commerce, and job recruitment. We construct a \textsc{Bias Synthesis Pipeline with Calibrated Quality Margins} that 1) synthesizes evaluation data by controlling the quality gap between optimal and sub-optimal options to provide a calibrated testbed to elicit the vulnerability to biases; 2) injects contextual biases that are logical and suitable for option contexts. Extensive experiments on both SOTA (Gemini-{2.5,3}-pro, GPT-4o, DeepSeek-R1) and small-scale LLMs reveal that agents frequently succumb to injected biases despite having sufficient reasoning capabilities to identify the ground truth. These findings expose a significant reliability bottleneck in current agentic workflows, calling for specialized alignment strategies for LLM-as-a-Recommender. The complete code and evaluation datasets will be made publicly available shortly.

翻译：当前大型语言模型（LLM）正逐步被应用于深度研究、电商推荐和人才招聘等实际有价值的智能体工作流中。在此类应用中，LLM需从海量候选中筛选最优解，我们称之为“LLM即推荐代理”（LLM-as-a-Recommender）范式。然而，LLM代理用于推荐的可靠性尚未得到充分探索。本文提出**偏差推荐基准**（**BiasRecBench**），旨在揭示此类代理在高价值真实世界任务中对偏见的严重脆弱性。该基准涵盖三大实践领域：论文评审、电子商务和人才招聘。我们构建了一个**具有校准质量边际的偏差合成流水线**（Bias Synthesis Pipeline with Calibrated Quality Margins）：1）通过控制最优与次优选项间的质量差距来合成评估数据，提供校准测试平台以诱发对偏见的脆弱性；2）注入逻辑合理且契合选项上下文的语境偏差。针对当前最优（SOTA）模型（Gemini-{2.5,3}-pro、GPT-4o、DeepSeek-R1）及小规模LLM的广泛实验表明，尽管智能体具备识别真实答案的充分推理能力，却频繁屈服于注入的偏见。这些发现暴露了当前智能体工作流中显著的可靠性瓶颈，亟需针对LLM即推荐代理范式开发专门的鲁棒性对齐策略。完整代码与评估数据集将于近期公开。