The rapid development of large language model (LLM) evaluation methodologies and datasets has led to a profound challenge: integrating state-of-the-art evaluation techniques cost-effectively while ensuring reliability, reproducibility, and efficiency. Currently, there is a notable absence of a unified and adaptable framework that seamlessly integrates various evaluation approaches. Moreover, the reliability of evaluation findings is often questionable due to potential data contamination, with the evaluation efficiency commonly overlooked when facing the substantial costs associated with LLM inference. In response to these challenges, we introduce FreeEval, a modular and scalable framework crafted to enable trustworthy and efficient automatic evaluations of LLMs. Firstly, FreeEval's unified abstractions simplify the integration and improve the transparency of diverse evaluation methodologies, encompassing dynamic evaluation that demand sophisticated LLM interactions. Secondly, the framework integrates meta-evaluation techniques like human evaluation and data contamination detection, which, along with dynamic evaluation modules in the platform, enhance the fairness of the evaluation outcomes. Lastly, FreeEval is designed with a high-performance infrastructure, including distributed computation and caching strategies, enabling extensive evaluations across multi-node, multi-GPU clusters for open-source and proprietary LLMs.

翻译：大语言模型评估方法与数据集的快速发展带来了一项深刻挑战：如何在确保可靠性、可复现性和效率的同时，以经济高效的方式集成前沿评估技术。当前，缺乏一个能够无缝整合多种评估方法的统一且可扩展的框架。此外，由于潜在的数据污染，评估结果的可靠性常受质疑；面对大语言模型推理带来的巨大成本时，评估效率也常被忽视。针对上述挑战，我们提出FreeEval——一个模块化、可扩展的框架，旨在实现大语言模型可信、高效的自动化评估。首先，FreeEval的统一抽象简化了多样化评估方法的集成，并提升了其透明度，涵盖需要复杂大语言模型交互的动态评估。其次，该框架集成了人类评估和数据污染检测等元评估技术，与平台中的动态评估模块共同增强评估结果的公平性。最后，FreeEval采用高性能基础设施设计，包括分布式计算与缓存策略，支持跨多节点、多GPU集群对开源与闭源大语言模型进行大规模评估。