Meta-evaluation studies of system performances in controlled offline evaluation campaigns, like TREC and CLEF, show a need for innovation in evaluating IR-systems. The field of academic search is no exception to this. This might be related to the fact that relevance in academic search is multilayered and therefore the aspect of user-centric evaluation is becoming more and more important. The Living Labs for Academic Search (LiLAS) lab aims to strengthen the concept of user-centric living labs for the domain of academic search by allowing participants to evaluate their retrieval approaches in two real-world academic search systems from the life sciences and the social sciences. To this end, we provide participants with metadata on the systems' content as well as candidate lists with the task to rank the most relevant candidate to the top. Using the STELLA-infrastructure, we allow participants to easily integrate their approaches into the real-world systems and provide the possibility to compare different approaches at the same time.

翻译：对TREC和CLEF等受控离线评估活动中系统性能的元评估研究表明，信息检索系统的评估方法亟待创新。学术搜索领域亦不例外。由于学术搜索中的相关性具有多层次性，用户中心评估的重要性日益凸显。学术搜索实境实验室（LiLAS）项目旨在强化面向学术搜索领域的用户中心实境实验室概念，通过允许参与者在两套分别来自生命科学和社会科学的真实学术搜索系统中评估其检索方法。为此，我们向参与者提供系统内容的元数据及备选文献列表，要求其将最相关的文献排至首位。借助STELLA基础设施，我们使参与者能够轻松将其方法集成至真实系统，并提供同时比较不同方法的可能性。