Search relevance plays a central role in web e-commerce. While large language models (LLMs) have shown significant results on relevance task, existing benchmarks lack sufficient complexity for comprehensive model assessment, resulting in an absence of standardized relevance evaluation metrics across the industry. To address this limitation, we propose Rule-Aware benchmark with Image for Relevance assessment(RAIR), a Chinese dataset derived from real-world scenarios. RAIR established a standardized framework for relevance assessment and provides a set of universal rules, which forms the foundation for standardized evaluation. Additionally, RAIR analyzes essential capabilities required for current relevance models and introduces a comprehensive dataset consists of three subset: (1) a general subset with industry-balanced sampling to evaluate fundamental model competencies; (2) a long-tail hard subset focus on challenging cases to assess performance limits; (3) a visual salience subset for evaluating multimodal understanding capabilities. We conducted experiments on RAIR using 14 open and closed-source models. The results demonstrate that RAIR presents sufficient challenges even for GPT-5, which achieved the best performance. RAIR data are now available, serving as an industry benchmark for relevance assessment while providing new insights into general LLM and Visual Language Model(VLM) evaluation.

翻译：搜索相关性在电子商务中扮演着核心角色。尽管大型语言模型（LLM）在相关性任务上已展现出显著成果，但现有基准缺乏足够的复杂性以进行全面模型评估，导致行业内缺乏标准化的相关性评估指标。为弥补这一不足，我们提出了用于相关性评估的规则感知图像基准（RAIR），这是一个源自真实场景的中文数据集。RAIR建立了一个标准化的相关性评估框架，并提供了一套通用规则，为标准化评估奠定了基础。此外，RAIR分析了当前相关性模型所需的关键能力，并引入了一个由三个子集构成的综合性数据集：（1）一个通过行业平衡采样构建的通用子集，用于评估模型的基础能力；（2）一个专注于挑战性案例的长尾困难子集，用于评估模型的性能极限；（3）一个用于评估多模态理解能力的视觉显著性子集。我们使用14个开源和闭源模型在RAIR上进行了实验。结果表明，即使对于性能最佳的GPT-5，RAIR也构成了足够的挑战。RAIR数据现已公开，可作为相关性评估的行业基准，同时为通用LLM和视觉语言模型（VLM）的评估提供新的见解。